File size: 2,632 Bytes
ab06fd2 f7fc860 9d15da3 f7fc860 93d1635 f7fc860 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 |
---
license: apache-2.0
datasets:
- TurkuNLP/finerweb-10bt
language:
- en
base_model:
- microsoft/deberta-v3-base
---
# Model Card for FinerWeb Line Quality Classifier
This model is a DeBERTa-v3-base classifier trained to identify high and low-quality content in web text at the line level. It was developed as part of the FinerWeb-10BT project to enhance training data quality for language models.
## Model Details
### Model Description
- **Developed by:** University of Turku (Erik Henriksson*, Otto Tarkka*, Filip Ginter) (*Equal contribution.)
- **Model type:** Line-level text quality classifier
- **Language(s) (NLP):** English
- **License:** apache-2.0
- **Finetuned from model:** microsoft/deberta-v3-base
### Model Sources
- **Paper:** https://arxiv.org/abs/2501.07314
- **Repository:** https://github.com/TurkuNLP/finerweb-10bt
## Uses
### Direct Use
The model is designed to classify text lines as either Clean (high-quality) or belonging to one of several low-quality categories. It outputs a quality score between 0 and 1 for each input line, where scores closer to 1 indicate higher quality content.
### Out-of-Scope Use
The model is specifically trained on English web text and may not perform well on other languages or specialized domains.
## Training Details
### Training Data
The model was trained on a labeled dataset of 328,472 lines from 20,000 documents sampled from FineWeb. The data preparation involved:
1. Initial line-level labeling by GPT-4o mini, which generated 547 unique descriptive labels
2. Label refinement and grouping into 9 broader categories using OpenAI's o1-preview model
3. Manual verification conducted only on a small sample (50 documents/726 lines) to assess inter-annotator agreement between human annotators and the LLM-generated labels
The final dataset consisted of 86.24% Clean lines and 13.76% lines distributed across 8 low-quality categories.
### Training Procedure
#### Training Hyperparameters
- **Training regime:** bfloat16 precision
- **Learning rate:** 1e-5
- **Batch size:** 16
- **Early stopping:** Applied with patience of 5 based on evaluation loss
- **Maximum epochs:** 5
- **Label smoothing:** 0.1 applied to cross-entropy loss
### Evaluation
### Testing Data, Factors & Metrics
#### Metrics
The model was evaluated using:
- Micro F1 score: 0.81
- Macro F1 score: 0.66
- Clean class metrics:
- Precision: 0.88
- Recall: 0.91
- F1: 0.90
## Technical Specifications
### Compute Infrastructure
#### Hardware
Computational resources for this study were provided by CSC — IT Center for Science. Training was performed on a single A100 GPU. |