metadata

license: mit
datasets:
  - newsmediabias/Bias-DeBiased
metrics:
  - accuracy
pipeline_tag: text-generation

MBIAS

Model Details

Model Name: MBIAS
Model Type: Large Language Model (LLM)
Version: 1.0
Developer: Ananya Raval, Veronica Chatrath, Shaina Raza
Model Repository: HuggingFace MBIAS

Paper: https://arxiv.org/abs/2405.11290

Model Description

MBIAS is a fine-tuned Large Language Model specifically designed to enhance safety while retaining contextual accuracy in model outputs. Traditional safety interventions often compromise contextual meaning when mitigating bias and toxicity. MBIAS addresses this by maintaining high contextual relevance and drastically reducing bias and toxicity in text generation.

Intended Use

The model is intended for research and development purposes, particularly in applications where reducing bias and toxicity in language generation is crucial without sacrificing the retention of key information.

Training Data

The model was fine-tuned on a custom dataset curated for comprehensive safety interventions. This dataset includes diverse text samples aiming to cover a wide range of demographics to effectively test and reduce bias and toxicity.

Evaluation

MBIAS has demonstrated a significant reduction in bias and toxicity, with over 30% reduction overall and exceeding 90% in specific demographic analysis on an out-of-distribution test set. Performance metrics include bias reduction, toxicity reduction, and retention of key information (KR).

Performance Metrics

Pre-Safety Intervention

Text	Bias ↓	Toxicity ↓	Knowledge Retention ↑	Faithfulness. ↑	Relevancy. ↑
Original sentence	32.21%	40.09%	N/A	N/A	N/A
Safe sentence (ground truth)	17.43%	14.53%	82.35%	77.91%	87.50%

Post-Safety Intervention

Text	Bias ↓	Toxicity ↓	Knowledge Retention ↑	Faithfulness. ↑	Relevancy. ↑
Mistral2-7B-(vanilla)	6.63%	4.50%	82.32%	79.62%	88.34%
Mistral2-7B (prompt-tuning)	11.4%	8.00%	81.45%	75.93%	86.64%
MBIAS (ours)	9.49%	8.71%	88.46%	82.54%	84.02%

How to Use

The model can be accessed and used for text generation through the HuggingFace platform. For detailed usage, please refer to the provided link in the footnote of the model card.

Hyperparameters

Batch Size per GPU: Training: 8, Evaluation: 4
Steps to Accumulate Gradients: 1
Maximum Gradient Norm: 0.3
Initial Learning Rate: 2e-05
Weight Decay: 0.001
Optimizer: paged_adamw 8bit
Learning Rate Scheduler: Constant
Warmup Steps Ratio: 0.05
Maximum Sequence Length: 2048
Training Epochs: 2
LoRA Attention Dimension: 64
LoRA Scaling/Dropout Probability: 16/0.2

Performance Metrics

Performance metrics are provided for both pre-safety and post-safety intervention phases. The model has shown excellent results in improving the retention of contextual accuracy while reducing bias and toxicity levels compared to other versions and configurations.

Citation

If you use this work, please cite it as follows:

@article{raza2024mbias,
      title={MBIAS: Mitigating Bias in Large Language Models While Retaining Context}, 
      author={Shaina Raza and Ananya Raval and Veronica Chatrath},
      year={2024},
      eprint={2405.11290},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}