LMMs-Lab

Feeling and building multimodal intelligence

LMMs-Lab

Feeling and building multimodal intelligence

Work

EgoLife

→

We introduce EgoLife, a project focused on building AI-powered egocentric life assistants, supported by a 300-hour multimodal dataset capturing daily activities via wearable AI glasses and third-person views. To enable long-context, life-oriented question answering, we develop EgoButler, integrating EgoGPT (a state-of-the-art vision-language model) and EgoRAG (a retrieval system for ultra-long-context queries).

EgoLife

→

We introduce EgoLife, a project focused on building AI-powered egocentric life assistants, supported by a 300-hour multimodal dataset capturing daily activities via wearable AI glasses and third-person views. To enable long-context, life-oriented question answering, we develop EgoButler, integrating EgoGPT (a state-of-the-art vision-language model) and EgoRAG (a retrieval system for ultra-long-context queries).

EgoLife

→

We introduce EgoLife, a project focused on building AI-powered egocentric life assistants, supported by a 300-hour multimodal dataset capturing daily activities via wearable AI glasses and third-person views. To enable long-context, life-oriented question answering, we develop EgoButler, integrating EgoGPT (a state-of-the-art vision-language model) and EgoRAG (a retrieval system for ultra-long-context queries).

Video-MMMU

→

We introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark that evaluates LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU features a curated collection of 300 expert-level videos and 900 human-annotated questions across 6 disciplines, evaluating knowledge acquisition through 3 cognitive stages: Perception, Comprehension, and Adaptation.

Video-MMMU

→

We introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark that evaluates LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU features a curated collection of 300 expert-level videos and 900 human-annotated questions across 6 disciplines, evaluating knowledge acquisition through 3 cognitive stages: Perception, Comprehension, and Adaptation.

Video-MMMU

→

We introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark that evaluates LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU features a curated collection of 300 expert-level videos and 900 human-annotated questions across 6 disciplines, evaluating knowledge acquisition through 3 cognitive stages: Perception, Comprehension, and Adaptation.

Multimodal-SAE

→

For the first time in multimodal research, we show that Sparse Autoencoder (SAE) features learned in a smaller LMM can be effectively interpreted by a larger LMM. Our work leverages SAEs to analyze open-semantic features, enabling feature interpretation across model scales.

Multimodal-SAE

→

For the first time in multimodal research, we show that Sparse Autoencoder (SAE) features learned in a smaller LMM can be effectively interpreted by a larger LMM. Our work leverages SAEs to analyze open-semantic features, enabling feature interpretation across model scales.

Multimodal-SAE

→

For the first time in multimodal research, we show that Sparse Autoencoder (SAE) features learned in a smaller LMM can be effectively interpreted by a larger LMM. Our work leverages SAEs to analyze open-semantic features, enabling feature interpretation across model scales.

LLaVA-OneVision

→

We present LLaVA-OneVision, a family of open large multimodal models (LMMs) developed by consolidating our insights into data, models, and visual representations in the LLaVA-NeXT blog series.

LLaVA-OneVision

→

We present LLaVA-OneVision, a family of open large multimodal models (LMMs) developed by consolidating our insights into data, models, and visual representations in the LLaVA-NeXT blog series.

LLaVA-OneVision

→

We present LLaVA-OneVision, a family of open large multimodal models (LMMs) developed by consolidating our insights into data, models, and visual representations in the LLaVA-NeXT blog series.

LongVA

→

We discovered and verified that the long context capability of language models can be directly transferred to the video domain in modality-aligned multi-modal models.

LongVA

→

We discovered and verified that the long context capability of language models can be directly transferred to the video domain in modality-aligned multi-modal models.

LongVA

→

We discovered and verified that the long context capability of language models can be directly transferred to the video domain in modality-aligned multi-modal models.

LLaVA-NeXT

→

With our iteratively development, LLaVA-NeXT gradually catches up to GPT4-V on selected benchmarks.

LLaVA-NeXT

→

With our iteratively development, LLaVA-NeXT gradually catches up to GPT4-V on selected benchmarks.

LLaVA-NeXT

→

With our iteratively development, LLaVA-NeXT gradually catches up to GPT4-V on selected benchmarks.

LLaVA-NeXT Video

→

We explore LLaVA-NeXT's capabilities in video understanding tasks, highlighting its strong performance.

LLaVA-NeXT Video

→

We explore LLaVA-NeXT's capabilities in video understanding tasks, highlighting its strong performance.

LLaVA-NeXT Video

→

We explore LLaVA-NeXT's capabilities in video understanding tasks, highlighting its strong performance.

LMMs-Eval

→

Accelerating the Development of Large Multimodal Models

LMMs-Eval

→

Accelerating the Development of Large Multimodal Models

LMMs-Eval

→

Accelerating the Development of Large Multimodal Models

About

LMMs-Lab is an open research organization founded by students and faculty from NTU, Singapore, in close collaboration with research labs and companies worldwide.

Wish to collaborate?

Github

X (Twitter)