hf-bert-finetuning

This model is a fine-tuned version of google-bert/bert-base-uncased on the twitter-financial-news-sentiment dataset. It achieves the following results on the evaluation set:

• Loss: 2.2672
• Accuracy: 0.805

Model description

The base model is google-bert/bert-base-uncased. It was fine-tuned to perform ternary classification (bullish/neutral/bearish) on financial tweets.

Intended uses & limitations

This model is intended to be used for demonstrating how to fine-tune a BERT model using the HuggingFace API. The outputs from the model are not meant to be used in a real production use-case (e.g. to classify whether a tweet is bearish or bullish).

Training and evaluation data

The training and evaluation dataset were taken from the twitter-financial-news-sentiment dataset on HuggingFace.

Training procedure

100 training and evaluation examples were randomly sampled from the dataset. This was used to train the BERT model for 100 epochs.

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 100

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy
0.392	4.0	500	1.1767	0.805
0.0421	8.0	1000	1.3555	0.814
0.0266	12.0	1500	1.7734	0.806
0.0066	16.0	2000	1.6149	0.818
0.0264	20.0	2500	1.4583	0.823
0.0284	24.0	3000	1.8117	0.794
0.0019	28.0	3500	1.8569	0.804
0.0336	32.0	4000	1.8200	0.801
0.0221	36.0	4500	1.8082	0.806
0.0195	40.0	5000	1.8102	0.81
0.007	44.0	5500	1.9712	0.82
0.0028	48.0	6000	1.8803	0.818
0.0017	52.0	6500	1.9739	0.82
0.0	56.0	7000	2.0171	0.821
0.019	60.0	7500	1.9017	0.805
0.0	64.0	8000	2.0914	0.801
0.0	68.0	8500	2.1453	0.799
0.0061	72.0	9000	2.2067	0.786
0.0009	76.0	9500	2.1612	0.799
0.0026	80.0	10000	2.1481	0.807
0.0	84.0	10500	2.1813	0.807
0.0	88.0	11000	2.2069	0.807
0.0	92.0	11500	2.2285	0.807
0.0	96.0	12000	2.2422	0.807
0.0004	100.0	12500	2.2672	0.805

Framework versions

• Transformers 4.40.1
• Pytorch 2.3.0
• Datasets 2.19.0
• Tokenizers 0.19.1