README.md

---
library_name: transformers
tags:
- climate-change
- flan-t5
- qlora
- instruction-tuning
---

# Model Card for FLAN-T5 Climate Action QLoRA

This is a QLoRA-finetuned version of FLAN-T5 specifically trained for climate action content analysis and generation. The model is optimized for processing and analyzing text related to climate change, sustainability, and environmental policies.

## Model Details

### Model Description

- **Developed by:** Kshitiz Khanal
- **Shared by:** kshitizkhanal7
- **Model type:** Instruction-tuned Language Model with QLoRA fine-tuning
- **Language(s):** English
- **License:** Apache 2.0
- **Finetuned from model:** google/flan-t5-base

### Model Sources
- **Repository:** https://huggingface.co/kshitizkhanal7/flan-t5-climate-qlora
- **Training Data:** FineWeb dataset (climate action filtered)

## Uses

### Direct Use

The model is designed for:
- Analyzing climate policies and initiatives
- Summarizing climate action documents
- Answering questions about climate change and environmental policies
- Evaluating sustainability measures
- Processing climate-related research and reports

### Downstream Use

The model can be integrated into:
- Climate policy analysis tools
- Environmental reporting systems
- Sustainability assessment frameworks
- Climate research applications
- Educational tools about climate change

### Out-of-Scope Use

The model should not be used for:
- Critical policy decisions without human oversight
- Generation of climate misinformation
- Technical climate science research without expert validation
- Commercial deployment without proper testing
- Medical or legal advice

## Bias, Risks, and Limitations

- Limited to climate-related content analysis
- May not perform well on general domain tasks
- Potential biases from web-based training data
- Should not be the sole source for critical decisions
- Performance varies on technical climate science topics

### Recommendations

- Always verify model outputs with authoritative sources
- Use human expert oversight for critical applications
- Consider the model as a supplementary tool, not a replacement for expert knowledge
- Regular evaluation of outputs for potential biases
- Use in conjunction with other data sources for comprehensive analysis

## Training Details

### Training Data
- Source: FineWeb dataset filtered for climate content
- Selection criteria: Climate-related keywords and quality metrics
- Processing: Instruction-style formatting with climate focus

### Training Procedure

#### Preprocessing
- Text cleaning and normalization
- Instruction templates for climate context
- Maximum input length: 512 tokens
- Maximum output length: 128 tokens

#### Training Hyperparameters
- Training regime: QLoRA 4-bit fine-tuning
- Epochs: 3
- Learning rate: 2e-4
- Batch size: 4
- Gradient accumulation steps: 4
- LoRA rank: 16
- LoRA alpha: 32
- Target modules: Query and Value matrices
- LoRA dropout: 0.05

## Environmental Impact

- **Hardware Type:** Single GPU
- **Hours used:** ~4 hours
- **Cloud Provider:** Local
- **Carbon Emitted:** Minimal due to QLoRA efficiency

## Technical Specifications

### Model Architecture and Objective
- Base architecture: FLAN-T5
- Objective: Climate-specific text analysis
- QLoRA adaptation for efficient fine-tuning
- 4-bit quantization for reduced memory usage

### Compute Infrastructure
- Python 3.8+
- PyTorch
- Transformers library
- bitsandbytes for quantization
- PEFT for LoRA implementation

### Hardware
Minimum requirements:
- 16GB GPU memory for inference
- 24GB GPU memory recommended for training
- CPU inference possible but slower

## Citation

If you use this model, please cite:
```bibtex
@misc{khanal2024climate,
  title={FLAN-T5 Climate Action QLoRA},
  author={Khanal, Kshitiz},
  year={2024},
  publisher={HuggingFace},
  howpublished={\url{https://huggingface.co/kshitizkhanal7/flan-t5-climate-qlora}}
}