---
license: apache-2.0
datasets:
- TurkuNLP/finerweb-10bt
language:
- en
base_model:
- microsoft/deberta-v3-base
---
# Model Card for FinerWeb Line Quality Classifier

This model is a DeBERTa-v3-base classifier trained to identify high and low-quality content in web text at the line level. It was developed as part of the FinerWeb-10BT project to enhance training data quality for language models.

## Model Details

### Model Description

- **Developed by:** University of Turku (Erik Henriksson*, Otto Tarkka*, Filip Ginter) (*Equal contribution.)
- **Model type:** Line-level text quality classifier
- **Language(s) (NLP):** English
- **License:** apache-2.0
- **Finetuned from model:** microsoft/deberta-v3-base

### Model Sources
- **Paper:** https://arxiv.org/abs/2501.07314
- **Repository:** https://github.com/TurkuNLP/finerweb-10bt

## Uses

### Direct Use

The model is designed to classify text lines as either Clean (high-quality) or belonging to one of several low-quality categories. It outputs a quality score between 0 and 1 for each input line, where scores closer to 1 indicate higher quality content.

### Out-of-Scope Use

The model is specifically trained on English web text and may not perform well on other languages or specialized domains.

## Training Details

### Training Data

The model was trained on a labeled dataset of 328,472 lines from 20,000 documents sampled from FineWeb. The data preparation involved:
1. Initial line-level labeling by GPT-4o mini, which generated 547 unique descriptive labels
2. Label refinement and grouping into 9 broader categories using OpenAI's o1-preview model
3. Manual verification conducted only on a small sample (50 documents/726 lines) to assess inter-annotator agreement between human annotators and the LLM-generated labels

The final dataset consisted of 86.24% Clean lines and 13.76% lines distributed across 8 low-quality categories.

### Training Procedure

#### Training Hyperparameters

- **Training regime:** bfloat16 precision
- **Learning rate:** 1e-5
- **Batch size:** 16
- **Early stopping:** Applied with patience of 5 based on evaluation loss
- **Maximum epochs:** 5
- **Label smoothing:** 0.1 applied to cross-entropy loss

### Evaluation

### Testing Data, Factors & Metrics

#### Metrics

The model was evaluated using:
- Micro F1 score: 0.81
- Macro F1 score: 0.66
- Clean class metrics:
  - Precision: 0.88
  - Recall: 0.91
  - F1: 0.90

## Technical Specifications

### Compute Infrastructure

#### Hardware
Computational resources for this study were provided by CSC — IT Center for Science. Training was performed on a single A100 GPU.