Update README.md

README.md

@@ -6,8 +6,15 @@ datasets:
 - toshi456/llava_pretrain_blip_laion_cc_sbu_558k_ja
 base_model: mylesgoose/Meta-Llama-3.1-8B-Instruct-goose-abliterated
 ---
+I Trained the llama 3.1 model integrating the google vison encoder. This is a base model It has not been trained on images the model itself, this modeel would be useefull to train on your own image datasets.
+It has only the encoder integrated into it. It has not been trained on any closed source datasets. Other than what is listed, for some reason its listing the japanese verison of the dataset above..
 Install https://github.com/LLaVA-VL/LLaVA-NeXT/tree/main prior to running below. Thanks to that team for their fantastic work.
+
 you can test with something like this.
+download this image and place into the path below in script or use your own image. 
+["The image shows a man in a yellow shirt and shorts sitting on the hood of a car with a clothes iron and ironing board in the back.\nThis is a common sight to see in many cities, especially in major cities like new york, where ironing clothes is a common activity for people to carry out while they are at home.\nHowever, this image is a little unusual because the man is ironing clothes on top of the car.\nIt is not unusual to see people ironing clothes while driving, but this is a rare sight.\nThis image is also unusual because the person is sitting on the hood of the car with their clothes in the back, and it seems that they are using an ironing board.\nThe man in the image is wearing a yellow shirt and shorts, and his pants and shirt appear to be in a bag on the hood.\nThe man is sitting on the car with the ironing board, which has a steamer, an ironing board, and clothes.\nThis image is unusual because it is a picture of a man in the middle of ironing clothes, 
+and it's also unusual because the car is driving down a street.\nThe man is using an ironing board with a steamer and clothes, and is sitting on the hood of the"]
+
 ![image/jpeg](https://cdn-uploads.huggingface.co/production/uploads/65069ffda7ba30bf62aea321/XJRK1McipixmNVUyiL5v1.jpeg)
 
 """"
@@ -30,44 +37,35 @@ tokenizer, model, image_processor, max_length = load_pretrained_model(pretrained
 model.eval()
 model.tie_weights()
 
-image = Image.open("https://cdn-uploads.huggingface.co/production/uploads/65069ffda7ba30bf62aea321/XJRK1McipixmNVUyiL5v1.jpeg")
+
+image = Image.open("/home/myles/Desktop/extreme_ironing.jpg")
 image_tensor = process_images([image], image_processor, model.config)
 image_tensor = [_image.to(dtype=torch.float16, device=device) for _image in image_tensor]
-conv_template = "llava_llama_3"
+
+conv_template = "llava_llama_3" # Make sure you use correct chat template for different models
 question = DEFAULT_IMAGE_TOKEN + "\nWhat is shown in this image? Is there anything strange about this image? Is this normal behaviour"
 conv = copy.deepcopy(conv_templates[conv_template])
 conv.append_message(conv.roles[0], question)
 conv.append_message(conv.roles[1], None)
 prompt_question = conv.get_prompt()
 
-# Check if tokenizer_image_token returns the attention mask
-input_ids, attention_mask = tokenizer_image_token(
-    prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt"
-)
-input_ids = input_ids.unsqueeze(0).to(device)
+input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
 image_sizes = [image.size]
 
-# If attention_mask is not returned, create it manually (adjust as needed)
-if attention_mask is None:
-    attention_mask = torch.ones_like(input_ids)
-    attention_mask[:, :IMAGE_TOKEN_INDEX] = 1
-    attention_mask[:, IMAGE_TOKEN_INDEX+1:] = 1
 
 cont = model.generate(
     input_ids,
     images=image_tensor,
     image_sizes=image_sizes,
-    attention_mask=attention_mask,
     do_sample=True,
     temperature=0.9,
     max_new_tokens=256,
 )
 text_outputs = tokenizer.batch_decode(cont, skip_special_tokens=True)
 print(text_outputs)
-
 """
 
-I Trained the llama 3.1 model integratign the google vison encoder. This is a base model . It has only the encoder integrated into it. It has not been trained on any closed datasets. Other than what is listed.
+
 LLM_VERSION="mylesgoose/Meta-Llama-3.1-8B-Instruct-goose-abliterated"
 LLM_VERSION_CLEAN="${LLM_VERSION//\//_}"
 VISION_MODEL_VERSION="google/siglip-so400m-patch14-384"