library_name: transformers
license: apache-2.0
base_model: alex-miller/ODABert
tags:
- generated_from_trainer
model-index:
- name: wb-climate-regression
results: []
datasets:
- alex-miller/wb-climate-percentage
wb-climate-regression
This model is a fine-tuned version of alex-miller/ODABert on the alex-miller/wb-climate-percentage dataset. It achieves the following results on the evaluation set:
- Loss: 0.0309
Model description
A regression model that embeds text using a fine-tuned bert-base-multilingual-uncased and uses the AutoModelForSequenceClassification class to output predicted percentage climate finance, climate adaptation finance, and climate mitigation finance.
Intended uses & limitations
Intended to regress World Bank project development objectives and abstracts. Not yet validated against project descriptions written outside of the World Bank project API V3.
Training and evaluation data
Data was collected automatically from the World Bank project API V3. For full code on how data was gathered, see: https://github.com/akmiller01/world-bank-climate-regression/blob/main/code/wb_api_climate.py
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 2e-06
- train_batch_size: 24
- eval_batch_size: 24
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 20
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 0.0722 | 1.0 | 126 | 0.0577 |
| 0.0481 | 2.0 | 252 | 0.0391 |
| 0.04 | 3.0 | 378 | 0.0350 |
| 0.0366 | 4.0 | 504 | 0.0332 |
| 0.0341 | 5.0 | 630 | 0.0323 |
| 0.0323 | 6.0 | 756 | 0.0312 |
| 0.0304 | 7.0 | 882 | 0.0312 |
| 0.0296 | 8.0 | 1008 | 0.0310 |
| 0.0281 | 9.0 | 1134 | 0.0306 |
| 0.0268 | 10.0 | 1260 | 0.0310 |
| 0.0261 | 11.0 | 1386 | 0.0303 |
| 0.0247 | 12.0 | 1512 | 0.0305 |
| 0.0243 | 13.0 | 1638 | 0.0310 |
| 0.0233 | 14.0 | 1764 | 0.0306 |
| 0.0221 | 15.0 | 1890 | 0.0309 |
| 0.0222 | 16.0 | 2016 | 0.0307 |
| 0.0216 | 17.0 | 2142 | 0.0308 |
| 0.0211 | 18.0 | 2268 | 0.0312 |
| 0.021 | 19.0 | 2394 | 0.0310 |
| 0.0206 | 20.0 | 2520 | 0.0309 |
Framework versions
- Transformers 4.44.2
- Pytorch 2.5.0+cu121
- Datasets 3.1.0
- Tokenizers 0.19.1