|
---
|
|
language:
|
|
- nl
|
|
tags:
|
|
- text-classification
|
|
- pytorch
|
|
metrics:
|
|
- accuracy
|
|
- f1-score
|
|
extra_gated_prompt: 'Our models are intended for academic use only. If you are not
|
|
affiliated with an academic institution, please provide a rationale for using our
|
|
models. Please allow us a few business days to manually review subscriptions.
|
|
|
|
If you use our models for your work or research, please cite this paper: Sebők,
|
|
M., Máté, Á., Ring, O., Kovács, V., & Lehoczki, R. (2024). Leveraging Open Large
|
|
Language Models for Multilingual Policy Topic Classification: The Babel Machine
|
|
Approach. Social Science Computer Review, 0(0). https://doi.org/10.1177/08944393241259434'
|
|
extra_gated_fields:
|
|
Name: text
|
|
Country: country
|
|
Institution: text
|
|
E-mail: text
|
|
Use case: text
|
|
---
|
|
# xlm-roberta-large-dutch-cap-v3
|
|
## Model description
|
|
An `xlm-roberta-large` model fine-tuned on dutch training data labeled with [major topic codes](https://www.comparativeagendas.net/pages/master-codebook) from the [Comparative Agendas Project](https://www.comparativeagendas.net/).
|
|
|
|
We follow the master codebook of the Comparative Agendas Project, and all of our models use the same major topic codes.
|
|
|
|
## How to use the model
|
|
|
|
```python
|
|
from transformers import AutoTokenizer, pipeline
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-large")
|
|
pipe = pipeline(
|
|
model="poltextlab/xlm-roberta-large-dutch-cap-v3",
|
|
task="text-classification",
|
|
tokenizer=tokenizer,
|
|
use_fast=False,
|
|
token="<your_hf_read_only_token>"
|
|
)
|
|
|
|
text = "We will place an immediate 6-month halt on the finance driven closure of beds and wards, and set up an independent audit of needs and facilities."
|
|
pipe(text)
|
|
```
|
|
|
|
The translation table from the model results to CAP codes is the following:
|
|
```python
|
|
CAP_NUM_DICT = {
|
|
0: 1,
|
|
1: 2,
|
|
2: 3,
|
|
3: 4,
|
|
4: 5,
|
|
5: 6,
|
|
6: 7,
|
|
7: 8,
|
|
8: 9,
|
|
9: 10,
|
|
10: 12,
|
|
11: 13,
|
|
12: 14,
|
|
13: 15,
|
|
14: 16,
|
|
15: 17,
|
|
16: 18,
|
|
17: 19,
|
|
18: 20,
|
|
19: 21,
|
|
20: 23,
|
|
21: 999,
|
|
}
|
|
```
|
|
|
|
We have included a 999 label because our models are fine-tuned on training data containing the label 'None' in addition to the 21 CAP major policy topic codes, indicating that the given text contains no relevant policy content. We use the label 999 for these cases.
|
|
|
|
### Gated access
|
|
Due to the gated access, you must pass the `token` parameter when loading the model. In earlier versions of the Transformers package, you may need to use the `use_auth_token` parameter instead.
|
|
|
|
## Model performance
|
|
The model was evaluated on a test set of 6398 examples.<br>
|
|
Model accuracy is **0.83**.
|
|
| label | precision | recall | f1-score | support |
|
|
|:-------------|------------:|---------:|-----------:|----------:|
|
|
| 0 | 0.81 | 0.77 | 0.79 | 471 |
|
|
| 1 | 0.7 | 0.72 | 0.71 | 148 |
|
|
| 2 | 0.88 | 0.8 | 0.84 | 242 |
|
|
| 3 | 0.76 | 0.87 | 0.81 | 78 |
|
|
| 4 | 0.76 | 0.78 | 0.77 | 374 |
|
|
| 5 | 0.9 | 0.92 | 0.91 | 248 |
|
|
| 6 | 0.86 | 0.75 | 0.8 | 155 |
|
|
| 7 | 0.79 | 0.86 | 0.82 | 95 |
|
|
| 8 | 0.86 | 0.82 | 0.84 | 217 |
|
|
| 9 | 0.88 | 0.9 | 0.89 | 244 |
|
|
| 10 | 0.85 | 0.87 | 0.86 | 763 |
|
|
| 11 | 0.73 | 0.75 | 0.74 | 319 |
|
|
| 12 | 0.79 | 0.83 | 0.81 | 121 |
|
|
| 13 | 0.75 | 0.77 | 0.76 | 378 |
|
|
| 14 | 0.82 | 0.83 | 0.83 | 123 |
|
|
| 15 | 0.7 | 0.75 | 0.72 | 106 |
|
|
| 16 | 0.39 | 0.58 | 0.47 | 19 |
|
|
| 17 | 0.93 | 0.92 | 0.93 | 1136 |
|
|
| 18 | 0.86 | 0.84 | 0.85 | 903 |
|
|
| 19 | 0.64 | 0.75 | 0.69 | 72 |
|
|
| 20 | 0.86 | 0.82 | 0.84 | 186 |
|
|
| macro avg | 0.79 | 0.8 | 0.79 | 6398 |
|
|
| weighted avg | 0.84 | 0.83 | 0.83 | 6398 |
|
|
|
|
### Fine-tuning procedure
|
|
This model was fine-tuned with the following key hyperparameters:
|
|
|
|
- **Number of Training Epochs**: 10
|
|
- **Batch Size**: 8
|
|
- **Learning Rate**: 5e-06
|
|
- **Early Stopping**: enabled with a patience of 2 epochs
|
|
|
|
## Inference platform
|
|
This model is used by the [CAP Babel Machine](https://babel.poltextlab.com), an open-source and free natural language processing tool, designed to simplify and speed up projects for comparative research.
|
|
|
|
## Cooperation
|
|
Model performance can be significantly improved by extending our training sets. We appreciate every submission of CAP-coded corpora (of any domain and language) at poltextlab{at}poltextlab{dot}com or by using the [CAP Babel Machine](https://babel.poltextlab.com).
|
|
|
|
## Reference
|
|
Sebők, M., Máté, Á., Ring, O., Kovács, V., & Lehoczki, R. (2024). Leveraging Open Large Language Models for Multilingual Policy Topic Classification: The Babel Machine Approach. Social Science Computer Review, 0(0). https://doi.org/10.1177/08944393241259434
|
|
|
|
## Debugging and issues
|
|
This architecture uses the `sentencepiece` tokenizer. In order to use the model before `transformers==4.27` you need to install it manually.
|
|
|
|
If you encounter a `RuntimeError` when loading the model using the `from_pretrained()` method, adding `ignore_mismatched_sizes=True` should solve the issue. |
