We introduce Video-MMMU, a multi-modal, multi-disciplinary benchmark that evaluates LMMs' ability to acquire and utilize knowledge from videos. Video-MMMU features a curated collection of 300 expert-level videos and 900 human-annotated questions across 6 disciplines, evaluating knowledge acquisition through 3 cognitive stages: Perception, Comprehension, and Adaptation.

For the first time in multimodal research, we show that Sparse Autoencoder (SAE) features learned in a smaller LMM can be effectively interpreted by a larger LMM. Our work leverages SAEs to analyze open-semantic features, enabling feature interpretation across model scales.

We present LLaVA-OneVision, a family of open large multimodal models (LMMs) developed by consolidating our insights into data, models, and visual representations in the LLaVA-NeXT blog series.

We discovered and verified that the long context capability of language models can be directly transferred to the video domain in modality-aligned multi-modal models.

With our iteratively development, LLaVA-NeXT gradually catches up to GPT4-V on selected benchmarks.

We explore LLaVA-NeXT's capabilities in video understanding tasks, highlighting its strong performance.

Accelerating the Development of Large Multimodal Models