Update readme

README.md

@@ -22,7 +22,7 @@ library_name: transformers
 
 Today (September 17th, 2024), we introduce [NVLM 1.0](https://arxiv.org/abs/2409.11402), a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks, rivaling the leading proprietary models (e.g., GPT-4o) and open-access models (e.g., Llama 3-V 405B and InternVL 2). Remarkably, NVLM 1.0 shows improved text-only performance over its LLM backbone after multimodal training. 
 
-In this repo, we are open-sourcing NVLM-1.0-D-72B (decoder-only architecture), the decoder-only model weights and code for the community.
+In this repo, we are open-sourcing NVLM-1.0-D-72B (decoder-only architecture), the decoder-only model weights and code for the community. 
 
 ## Other Resources
 [Inference Code (HF)](https://huggingface.co/nvidia/NVLM-D-72B/tree/main)   [Training Code (Coming soon)]()   [Website](https://research.nvidia.com/labs/adlr/NVLM-1/)   [Paper](https://arxiv.org/abs/2409.11402)
@@ -75,7 +75,11 @@ Results (as of September 17th, 2024) in the multimodal benchmarks are as follows
 
 ## How to use
 
-When converting Megatron checkpoint to Huggingface, we adapt [InternVL codebase](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) to support model loading and multi-GPU inference in HF. For training, please refer to [Megatron-LM (Coming soon)]().
+When converting Megatron checkpoint to Huggingface, we adapt [InternVL codebase](https://huggingface.co/OpenGVLab/InternVL2-Llama3-76B) to support model loading and multi-GPU inference in HF. 
+We also use the tokenizer from [Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct/tree/main) when adapting the tokenizer to Huggingface, as it contains extra special tokens for vision tasks, e.g., `<|vision_pad|>`. 
+We train NVLM-1.0-D-72B based on the [Qwen2-72B-Instruct](https://huggingface.co/Qwen/Qwen2-72B-Instruct/tree/main) text-only model and [InternViT-6B-448px-V1-5](https://huggingface.co/OpenGVLab/InternViT-6B-448px-V1-5) ViT model with our large-scale high-quality multimodal dataset. 
+For training code, please refer to [Megatron-LM (Coming soon)]().
+
 
 ### Prepare the environment
 
@@ -304,4 +308,4 @@ Wenliang Dai* ([email protected]), Nayeon Lee* ([email protected]), Boxin Wang* (
 
 
 ## License
-The use of this model is governed by the [cc-by-nc-4.0](https://spdx.org/licenses/CC-BY-NC-4.0)
+The use of this model is governed by the [cc-by-nc-4.0](https://spdx.org/licenses/CC-BY-NC-4.0)