Subject Classifier built on Distilbert
Table of Contents
- Model Details
- How to Get Started With the Model
- Uses
- Risks, Limitations and Biases
- Training
- Evaluation
- Environmental Impact
Model Details
Model Description: This is the uncased DistilBERT model fine-tuned on a custom dataset that is built on the IITJEE NEET AIIMS Students Questions Data for the subject classification task.
- Developed by: The Typeform team.
- Model Type: Text Classification
- Language(s): English
- License: GNU GENERAL PUBLIC LICENSE
- Parent Model: See the distilbert base uncased model for more information about the Distilled-BERT base model.
Uses
This model can be used for text classification tasks.
Risks, Limitations and Biases
CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.
Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)).
Training
Training is done on a NVIDIA RTX 3070 AMD Ryzen 7 5800 with the following hyperparameters:
$ training.ipynb \
--model_name_or_path distilbert-base-uncased \
--do_train \
--do_eval \
--max_seq_length 512 \
--per_device_train_batch_size 4 \
--learning_rate 1e-05 \
--num_train_epochs 5 \
Evaluation
Evaluation Results
When fine-tuned on downstream tasks, this model achieves the following results:
Epochs: 5 | Train Loss: 0.001 | Train Accuracy: 0.989 | Val Loss: 0.006 | Val Accuracy: 0.950 CPU times: user 18h 19min 13s, sys: 1min 34s, total: 18h 20min 47s Wall time: 18h 20min 7s
- **Epoch = ** 5.0
- Evaluation Accuracy = 0.950
- Evaluation Loss = 0.006
- Training Accuracy = 0.989
- Training Loss = 0.001
Testing Results
| precision | recall | f1-score | support | |
|---|---|---|---|---|
| biology | 0.98 | 0.99 | 0.99 | 15988 |
| chemistry | 1.00 | 0.99 | 0.99 | 20678 |
| computer | 1.00 | 0.99 | 0.99 | 8754 |
| maths | 1.00 | 1.00 | 1.00 | 26661 |
| physics | 0.99 | 0.98 | 0.99 | 10306 |
| social sciences | 0.99 | 1.00 | 0.99 | 25695 |
| accuracy | 0.99 | 108082 | ||
| macro avg | 0.99 | 0.99 | 0.99 | 108082 |
| weighted avg | 0.99 | 0.99 | 0.99 | 108082 |
Environmental Impact
Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019). We present the hardware type based on the associated paper.
Hardware Type: 1 NVIDIA RTX 3070
Hours used: 18h 19min 13s
Carbon Emitted: (Power consumption x Time x Carbon produced based on location of power grid): Unknown