Update README.md

README.md

@@ -5,9 +5,39 @@ license: apache-2.0
 
 <br>
 
-# LLaVA-Next-Video Model Card
+## LLaVA-NeXT-Video is upgraded 🚀
 
-## Model details
+In our [LLaVA-Video blog](https://llava-vl.github.io/blog/2024-04-30-llava-next-video/) released this April, we shared two key observations: 
+- 🎬 AnyRes provides a shared and flexible representation between images and videos, and thus accommodates capability transfer between the two most common vision signals. Therefore, stronger image LMMs can naturally lead to stronger zero-shot video LMMs. 
+- 🗂️ There is a lack of high-quality language-video data, including video instruction-following data, and thus naive tuning on existing public data at that time results in performance degradation. Therefore, there is an urgent need to build high-quality video captions and QA datasets to train LMMs for improved video performance.
+
+Based on the insights, the new LLaVA-NeXT-Video in this release improves from two aspects:
+
+- 🎬 A stronger image LMMs ([LLaVA-NeXT-32B-Qwen](https://huggingface.co/lmms-lab/llava-next-qwen-32b)), which is built by initializing from Qwen-1.5 32B LLM. We further initialize our video training from this image checkpoint.
+- 🗂️ A new high-quality video dataset with 830k samples. It is combined with LLaVA-1.6 image training data, and applying the same image-video mixed training procedure leads to the new video model.
+The new model achieves the best open-source performance in several video benchmarks including [Video-MME](https://video-mme.github.io/home_page.html#leaderboard).
+
+### Resources
+- **Inference Script**:
+  ```bash
+  bash scripts/video/demo/video_demo.sh lmms-lab/LLaVA-NeXT-Video-32B-Qwen 32 2 average after grid True playground/demo/xU25MMA2N4aVtYay.mp4
+  ```
+
+### Evaluation Results
+| Model                       | NextQA-MC | video-mme(overall) |        | Egochema | Perception Test  (val) |
+|-----------------------------|-----------|--------------------|--------|----------|------------------------|
+|                             |           | w/o subs           | w subs |          |                        |
+| **Proprietary**                 |           |                    |        |          |                        |
+| GPT-4o                      | -         | 71.9               | 77.2   | 72.2     | -                      |
+| Gemini 1.5 Pro              | -         | 75.0               | 81.3   | 72.2     | -                      |
+| **Open-Source**                 |           |                    |        |          |                        |
+| VideoLLaMA 2 (8x7B)         | 76.3*     | 47.9               | 50.3   | 53.3     | 51.2*                  |
+| VILA-1.5-34B                | 67.89*    | 60.1               | 61.1   | 58.04*   | 54                     |
+| LLaVA-NeXT-Video (Qwen-32B) | 77.31     | 60.2               | 63.0   | 60.85    | 59.38                  |
+
+_*Results are reproduced by [lmms-eval](https://github.com/EvolvingLMMs-Lab/lmms-eval). Please refer to the lmms-eval to reproduce the results._
+
+### Model details
 
 **Model type:**
 <br>
@@ -23,14 +53,14 @@ LLaVA-NeXT-Video-32B-Qwen was trained in June 2024.
 <br>
 https://github.com/LLaVA-VL/LLaVA-NeXT
 
-## License
+### License
 [Qwen/Qwen1.5-32B](https://huggingface.co/Qwen/Qwen1.5-32B) license.
 
 
-## Where to send questions or comments about the model
+### Where to send questions or comments about the model
 https://github.com/LLaVA-VL/LLaVA-NeXT/issues
 
-## Intended use
+### Intended use
 **Primary intended uses:**
 <br>
 The primary use of LLaVA is research on large multimodal models and chatbots.
@@ -39,7 +69,7 @@ The primary use of LLaVA is research on large multimodal models and chatbots.
 <br>
 The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
 
-## Training dataset
+### Training dataset
 
 ### Image
 - 558K filtered image-text pairs from LAION/CC/SBU, captioned by BLIP.
@@ -50,3 +80,21 @@ The primary intended users of the model are researchers and hobbyists in compute
 ### Video
 - 830k data
 
+### Citations
+```bibtex
+@misc{zhang2024llavanextvideo,
+  title={LLaVA-NeXT: A Strong Zero-shot Video Understanding Model},
+  url={https://llava-vl.github.io/blog/2024-04-30-llava-next-video/},
+  author={Zhang, Yuanhan and Li, Bo and Liu, haotian and Lee, Yong jae and Gui, Liangke and Fu, Di and Feng, Jiashi and Liu, Ziwei and Li, Chunyuan},
+  month={April},
+  year={2024}
+}
+
+@misc{li2024llavanext-interleave,
+  title={LLaVA-NeXT: Tackling Multi-image, Video, and 3D in Large Multimodal Models},
+  url={https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/},
+  author={Li, Feng and Zhang, Renrui and Zhang, Hao and Zhang, Yuanhan and Li, Bo and Li, Wei and Ma, Zejun and Li, Chunyuan},
+  month={June},
+  year={2024}
+}
+