--- license: cc datasets: - vector-institute/newsmediabias-plus language: - en metrics: - accuracy - precision base_model: - meta-llama/Llama-3.2-11B-Vision-Instruct --- # LLaMA 3.2 Multimodal News Media Bias Detector ## Model Overview The LLaMA 3.2 Multimodal News Media Bias Detector is a fine-tuned version of the LLaMA 2 Vision-Instruct model, designed to assess potential disinformation in news media articles by analyzing both text and image content. The model identifies the presence of rhetorical techniques commonly associated with disinformation and classifies articles as 'Likely' or 'Unlikely' to be disinformation. ## Model Details - **Base Model**: [meta-llama/Llama-3.2-11B-Vision-Instruct](https://huggingface.co/meta-llama/Llama-3.2-11B-Vision-Instruct) - **Architecture**: Multimodal (Text and Image) - **Model Size**: 11B parameters - **Quantization**: 4-bit quantization (`nf4`) using `bitsandbytes` for efficient inference - **Framework**: [Hugging Face Transformers](https://github.com/huggingface/transformers) - **Fine-tuning**: The model was fine-tuned using the [PEFT](https://github.com/huggingface/peft) library with LoRA adapters ## Intended Use ### Primary Use Cases - **Disinformation Detection**: Analyze news articles and accompanying images to detect potential disinformation. - **Media Analysis**: Assist researchers and analysts in studying bias and rhetorical techniques in media content. - **Educational Tool**: Serve as a resource for learning about multimodal models and disinformation detection. ### Limitations - **Not a Fact-Checker**: The model does not verify factual accuracy but assesses rhetorical techniques. - **Bias in Training Data**: The model's judgments may reflect biases present in the training data. - **Language and Domain Limitations**: Primarily trained on English-language news articles; performance may degrade on other languages or domains. ## How to Use First access through huggingface cli ``` ! huggingface-cli login ``` **Sampled Data Usage** Please use the sampled data available at either of the following repositories for testing or development purposes with the LLaMA 3.2 Multimodal News Media Bias Detector: - [Sampled Data on Hugging Face - LLaMA 3.2 Multimodal News Media Bias Detector](https://huggingface.co/vector-institute/Llama3.2-Multimodal-Newsmedia-Bias-Detector/tree/main/sampled-data) - [NewsMediaBias-Plus Dataset on Hugging Face](https://huggingface.co/datasets/vector-institute/newsmediabias-plus) These datasets are pre-configured to be compatible with the model and include a variety of news articles and images that have been annotated for potential disinformation, allowing for effective model evaluation and demonstration. ### Installation Ensure you have the required libraries installed: ``` pip install transformers accelerate bitsandbytes ``` ``` ### Loading the Model import torch from transformers import AutoModelForVision2Seq, AutoProcessor, BitsAndBytesConfig merged_model_save_path= "vector-institute/Llama3.2-Multimodal-Newsmedia-Bias-Detector" # Load quantization config if used during saving bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_use_double_quant=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16 ) # Load the merged model model = AutoModelForVision2Seq.from_pretrained( merged_model_save_path, device_map="auto", torch_dtype=torch.bfloat16, quantization_config=bnb_config ) model.eval() # Load the processor processor = AutoProcessor.from_pretrained(merged_model_save_path) import pandas as pd import os from PIL import Image # Path to your dataset CSV and image folder dataset_csv_path = 'sample_dataset.csv' image_folder_path = 'sampled_images' # Load the DataFrame df = pd.read_csv(dataset_csv_path) # Function to prepare samples from the DataFrame def prepare_samples_from_dataframe(df, image_folder_path): samples = [] for index, row in df.iterrows(): unique_id = row['unique_id'] # Replace with the column that contains the image identifier text_content = row['first_paragraph'] # Replace with your text column name # Find the image file possible_extensions = ['jpg', 'jpeg', 'png'] image_path = None for ext in possible_extensions: img_path = os.path.join(image_folder_path, f"{unique_id}.{ext}") if os.path.exists(img_path): image_path = img_path break if image_path is None: print(f"No image found for ID {unique_id}") continue image = Image.open(image_path).convert("RGB") max_size = (224, 224) image.thumbnail(max_size, Image.Resampling.LANCZOS) # Prepare the prompt text sample_text = ( "Assess the text and image below for potential disinformation (try finding deliberately misleading or biased information) by identifying the presence of rhetorical techniques listed.\n" "If you find any of the listed rhetorical techniques, then the article is likely disinformation; if not, it is likely not disinformation.\n\n" "Rhetorical Techniques Checklist:\n" "- Emotional Appeal: Uses language that intentionally invokes extreme emotions like fear or anger, aiming to distract from lack of factual backing.\n" "- Exaggeration and Hyperbole: Makes claims that are unsupported by evidence, or presents normal situations as extraordinary to manipulate perceptions.\n" "- Bias and Subjectivity: Presents information in a way that unreasonably favors one perspective, omitting key facts that might provide balance.\n" "- Repetition: Uses repeated messaging of specific points or misleading statements to embed a biased viewpoint in the reader's mind.\n" "- Specific Word Choices: Employs emotionally charged or misleading terms to sway opinions subtly, often in a manipulative manner.\n" "- Appeals to Authority: References authorities who lack relevant expertise or cites sources that do not have the credentials to be considered authoritative in the context.\n" "- Lack of Verifiable Sources: Relies on sources that either cannot be verified or do not exist, suggesting a fabrication of information.\n" "- Logical Fallacies: Engages in flawed reasoning such as circular reasoning, strawman arguments, or ad hominem attacks that undermine logical debate.\n" "- Conspiracy Theories: Propagates theories that lack proof and often contain elements of paranoia or implausible scenarios as facts.\n" "- Inconsistencies and Factual Errors: Contains multiple contradictions or factual inaccuracies that are easily disprovable, indicating a lack of concern for truth.\n" "- Selective Omission: Deliberately leaves out crucial information that is essential for a fair understanding of the topic, skewing perception.\n" "- Manipulative Framing: Frames issues in a way that leaves out alternative perspectives or possible explanations, focusing only on aspects that support a biased narrative.\n\n" f"{text_content}\n\n" "Please **only** provide your answer in the format: 'Classification: Likely' or 'Classification: Unlikely'. Do not include any additional text or explanation." ) sample = { "unique_id": unique_id, "messages": [ { "role": "user", "content": [ { "type": "text", "text": sample_text, }, { "type": "image", "image": image, } ], } ] } samples.append(sample) return samples import re def extract_assistant_reply(generated_text): lines = generated_text.strip().split('\n') for idx, line in enumerate(lines): if line.strip().lower() == 'assistant': assistant_reply = '\n'.join(lines[idx+1:]).strip() return assistant_reply return lines[-1].strip() def extract_classification(assistant_reply): match = re.search(r'Classification:\s*(Likely|Unlikely)', assistant_reply, re.IGNORECASE) if match: return match.group(1).capitalize() else: # Attempt to find 'Likely' or 'Unlikely' anywhere in the reply match = re.search(r'\b(Likely|Unlikely)\b', assistant_reply, re.IGNORECASE) if match: return match.group(1).capitalize() else: return 'Unknown' def generate_prediction(sample): # Remove the model.to(device) line # device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # model.to(device) # Prepare the input texts and images texts = processor.apply_chat_template(sample["messages"], tokenize=False) image_input = sample["messages"][0]['content'][1]['image'] # Prepare the inputs without moving them to a device inputs = processor(text=texts, images=image_input, return_tensors="pt", padding=True) # Ensure that the inputs are on the same device as the model's embeddings # This is handled internally by the model when using device_map="auto" with torch.no_grad(): outputs = model.generate( **inputs, max_new_tokens=20, do_sample=False, num_beams=1, ) generated_texts = processor.batch_decode(outputs, skip_special_tokens=True) assistant_reply = extract_assistant_reply(generated_texts[0]) return assistant_reply # Prepare the samples from the DataFrame samples = prepare_samples_from_dataframe(df, image_folder_path) # Limit to a subset if desired samples_to_infer = samples[:5] # For example, take the first 5 samples # Run inference and collect results results = [] for sample in samples_to_infer: assistant_reply = generate_prediction(sample) predicted_label = extract_classification(assistant_reply) # Collect results result = { 'unique_id': sample['unique_id'], 'assistant_reply': assistant_reply, 'predicted_label': predicted_label, # Add any other fields you need from the sample or DataFrame } results.append(result) # Display the results print(f"Sample ID: {sample['unique_id']}") print("Assistant's Reply:") print(assistant_reply) print(f"Predicted Label: {predicted_label}") print("-" * 50) results_df = pd.DataFrame(results) # Save to CSV if desired results_df.to_csv('inference_results.csv', index=False) ``` ### Example Output ``` Assistant's Reply: Classification: Likely ``` ## Training Data The model is fine-tuned on [NewsMediaBias-Plus Dataset on Hugging Face](https://huggingface.co/datasets/vector-institute/newsmediabias-plus) ## Training Procedure - **Fine-Tuning**: The model was fine-tuned using the PEFT library with LoRA adapters to efficiently adapt the base model to the disinformation detection task. - **Quantization**: 4-bit quantization was applied using `bitsandbytes` to optimize inference performance. - **Hyperparameters**: Standard hyperparameters were used, with adjustments made for optimal performance on the custom dataset. ## Evaluation The model was evaluated on a balanced test set with the following metrics: - **Accuracy**: *88%* - **Precision**: *78%* - **Recall**: *75%* *Note: Replace the above metrics with actual results from your evaluations.* ## Limitations and Biases - **Biases in Data**: The model's performance and judgments are influenced by the biases present in the training data. - **False Positives/Negatives**: The model may incorrectly classify some articles due to subtle rhetorical techniques or lack thereof. - **Ethical Considerations**: Users should be cautious when interpreting the model's outputs and consider them as part of a broader analysis. ## Ethical Considerations - **User Responsibility**: The model should not be used as the sole basis for making critical decisions. - **Data Privacy**: Ensure compliance with data privacy laws when using the model with user data. - **Misuse Prevention**: The model should not be used to unjustly label or discriminate against individuals or groups. ## License This model is licensed under CC BY-NC-SA 4.0. Developed by Dr. Shaina Raza at the Vector Institute. Contact: [shaina.raza@vectorinstitute.ai](mailto:shaina.raza@vectorinstitute.ai) **Disclaimer**: The model is provided "as is" without any warranty. The developers are not responsible for any consequences arising from the use of this model. ```