File size: 1,782 Bytes
1116bde 8f00da1 1116bde |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 |
---
language:
- en
license: apache-2.0
datasets:
- glue
metrics:
- accuracy
model-index:
- name: gpt2-finetuned-cola
results:
- task:
name: Text Classification
type: text-classification
dataset:
name: GLUE COLA
type: glue
args: cola
metrics:
- name: Accuracy
type: accuracy
value: 0.77756
---
# gpt2-finetuned-cola
<!-- Provide a quick summary of what the model is/does. -->
This model is gpt2 fine-tuned on GLUE CoLA dataset. It acheives the following results on the validation set. Though MCC is used for CoLA dataset, accuracy is reported here
- Accuracy: 0.77756
## Model Details
GPT-2 is a transformers model pretrained on a very large corpus of English data in a self-supervised fashion.
This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts.
More precisely, it was trained to guess the next word in sentences. However, it acheives very good results on Text Classification tasks.
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 3e-4
- train_batch_size: 32
- eval_batch_size: 32
- seed: 42
- optimizer: epsilon=1e-08
- num_epochs: 5.0
### Training results
|Epoch | Training Loss | Validation Loss | Training Accuracy | Validation Accuracy |
|:----:|:-------------:|:-----------------:|:---------------:|:-------------------:|
| 1 | 0.65824 | 0.55883 | 0.67957 | 0.70086 |
| 2 | 0.51774 | 0.53402 | 0.74927 | 0.75264 |
| 3 | 0.44502 | 0.50641 | 0.79172 | **0.77756** |
| 4 | 0.38996 | 0.54241 | 0.82645 | 0.77469 |
| 5 | 0.35218 | 0.55088 | 0.84516 | 0.77277 | |