update readme.md

README.md

@@ -1,3 +1,50 @@
 ---
 license: llama2
+datasets:
+- liuhaotian/LLaVA-Instruct-150K
 ---
+
+# Ada-LLaVA Model Card
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+Ada-LLaVA 13B is an open-source adaptive inference framework for multimodal Large Language Models (MLLMs) that dynamically adjusts its operations based on available computational resources and latency requirements. 
+
+See the paper for more details: [Learning to Inference Adaptively for Multimodal Large Language Models]()
+
+**Model details**: https://zhuoyan-xu.github.io/ada-llava/
+
+## Model Details
+
+<!-- Provide a longer summary of what this model is. -->
+
+**Model Type:** Ada LLaVA 13B follows the [LLaVA-v1.5](https://arxiv.org/abs/2310.03744) stage-2 training pipeline, 
+with [CLIP-ViT-L-336px](https://huggingface.co/openai/clip-vit-large-patch14-336) as visual encoder (336*336 image resolution), 
+[Vicuna-v1.5-13B](https://huggingface.co/lmsys/vicuna-13b-v1.5) as base LLM and a two-layer MLP as vision-language connector, customized embedding model and MLP as latency scheduler. 
+
+It was trained with stage-2 pipeline as LLaVA:
+
+Instruction tuning: Freeze vision encoder, train the remaining model with multimodal instruction following data of tabular and non-tabular tasks.
+
+**Code Base:** We use the official code of [LLaVA-v1.5](https://github.com/haotian-liu/LLaVA) for model training and inference, 
+and the saved model checkpoint is uploaded to this repository. 
+
+**Model Date:** Ada-LLaVA 13B was trained in Oct 2024.
+
+
+## License
+
+AdaLLaVA is based on LLaVA-1.5 and thus follows its license. Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
+
+## Intended use
+
+**Primary intended uses:** The primary use of Ada LLaVA is research on multimodal large multimodal models and chatbots, especially for resource-constrain inference and deployment.
+
+**Primary intended users:** The primary intended users of the model are researchers and hobbyists in computer vision, natural language processing, machine learning, and artificial intelligence.
+
+## Training dataset
+- 665K image level instruction data from LLaVA-1.5 stage-2, see details in original [LLaVA repo](https://github.com/haotian-liu/LLaVA?tab=readme-ov-file#visual-instruction-tuning).
+
+## Limitations
+
+While Ada-LLaVA is currently limited to processing one image at a time and only applies adaptive operations in its later half of layers, future work could explore multi-image input support and extend the adaptive mechanisms throughout the entire model architecture, including the vision encoder. These improvements would make the model more versatile and applicable to a broader range of real-world scenarios.