Tiny Mistral fine-tuned on finance dataset

This model is a fine-tuned version of the openaccess-ai-collective/tiny-mistral language model. It has been fine-tuned on a specialized finance dataset using Parameter-Efficient Fine-Tuning (PEFT) with Low-Rank Adaptation (LoRA). The model is designed to generate responses based on financial instructions and contexts.

Intended uses & limitations

This model is intended for text generation tasks specifically related to financial instructions and contexts. It can be used for generating responses based on given financial prompts.

Limitations:

The model may not perform well on financial topics not covered in the training data.
The quality of responses may vary depending on the specificity and complexity of the financial queries.
The model may generate responses that are not factually accurate or may include biases present in the training data.

Training and evaluation data

The model was fine-tuned on the gbharti/finance-alpaca dataset, which includes financial instructions and outputs. The dataset was processed to format instructions with or without additional context.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0002
train_batch_size: 4
eval_batch_size: 8
seed: 42
gradient_accumulation_steps: 8
total_train_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_steps: 100
num_epochs: 2
mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
1.3155	0.2580	500	1.3207
1.1306	0.5160	1000	1.1318
0.9935	0.7739	1500	0.9970
0.7188	1.0319	2000	0.8934
0.6962	1.2899	2500	0.8238
0.6427	1.5479	3000	0.7610
0.6014	1.8059	3500	0.7193

Framework versions

PEFT 0.12.0
Transformers 4.42.4
Pytorch 2.3.1+cu121
Datasets 2.20.0
Tokenizers 0.19.1

Adriana213
/

tiny-mistral-alpaca-finance