Update README.md

README.md

@@ -16,7 +16,7 @@ tags:
 ---
 # AnyModal/Image-Captioning-Llama-3.2-1B
 
-**AnyModal/Image-Captioning-Llama-3.2-1B** explores the potential of combining visual feature extraction and language modeling techniques to generate descriptive captions for natural images. Built within the [AnyModal](https://github.com/ritabratamaiti/AnyModal) framework, this model integrates a Vision Transformer (ViT) encoder with the Llama 3.2-1B language model, fine-tuned on the Flickr30k dataset. The model demonstrates a promising approach to bridging visual and textual modalities.
+**AnyModal/Image-Captioning-Llama-3.2-1B** is an image captioning model built within the [AnyModal](https://github.com/ritabratamaiti/AnyModal) framework. It integrates a Vision Transformer (ViT) encoder with the Llama 3.2-1B language model and has been trained on the Flickr30k dataset. The model demonstrates the integration of pre-trained vision and language components for generating descriptive captions from natural images.
 
 ---
 
@@ -27,7 +27,7 @@ This model was trained on the [Flickr30k Dataset](https://huggingface.co/dataset
 **From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference Over Event Descriptions**  
 *Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, Svetlana Lazebnik*  
 
-The dataset comprises 31,000 images collected from Flickr, each annotated with five descriptive sentences written by human annotators. These annotations offer diverse perspectives on real-world scenes and actions, forming a robust basis for image captioning experiments.
+The dataset contains 31,000 images collected from Flickr, each annotated with five descriptive sentences written by human annotators, covering a variety of real-world scenes and events.
 
 ---
 
@@ -75,10 +75,13 @@ vision_tokenizer = vision.Projector(vision_hidden_size, llm_hidden_size, num_hid
 multimodal_model = anymodal.MultiModalModel(
     input_processor=None,
     input_encoder=vision_encoder,
-    input_tokenizer=vision.Projector(vision_hidden_size, llm_hidden_size, num_hidden=1),
+    input_tokenizer=vision_tokenizer,
     language_tokenizer=llm_tokenizer,
     language_model=llm_model,
-    prompt_text="The description of the given image is: ")
+    input_start_token="<|imstart|>",
+    input_end_token="<|imend|>",
+    prompt_text="The description of the given image is: ",
+)
 
 # Download pre-trained model weights
 if not os.path.exists("image_captioning_model"):
@@ -107,14 +110,15 @@ This model is part of the [AnyModal Image Captioning Project](https://github.com
 - **Training Script**: [train.py](https://github.com/ritabratamaiti/AnyModal/blob/main/Image%20Captioning/train.py)  
 - **Inference Script**: [inference.py](https://github.com/ritabratamaiti/AnyModal/blob/main/Image%20Captioning/inference.py)  
 
-Explore the full project repository for additional details and potential customization.
+Refer to the project repository for further implementation details and customization.
 
 ---
 
 ## Project Details
 
-- **Vision Encoder**: Uses a pre-trained Vision Transformer (ViT) model for feature extraction, offering a strong baseline for processing visual information.  
-- **Projector Network**: Maps visual features into a token space that aligns with the Llama 3.2-1B language model.  
-- **Language Model**: Utilizes Llama 3.2-1B, a pre-trained causal language model, to construct coherent and context-sensitive captions.  
+- **Vision Encoder**: Pre-trained Vision Transformer (ViT) model for visual feature extraction.  
+- **Projector Network**: Projects visual features into a token space compatible with Llama 3.2-1B.  
+- **Language Model**: Llama 3.2-1B, a pre-trained causal language model for text generation.  
+
+This implementation serves as a proof of concept, combining a ViT-based image encoder and a small language model. Future iterations could achieve improved performance by incorporating text-conditioned image encoders and larger-scale language models.
 
-While trained on the Flickr30k dataset, the model's design highlights the possibilities for integrating vision and language models for captioning tasks, showcasing the feasibility of this approach within the AnyModal framework.