See axolotl config

axolotl version: 0.5.2

base_model: mistralai/Mistral-7B-v0.1
model_type: AutoModelForCausalLM
tokenizer_config: Open-Orca/Mistral-7B-OpenOrca
tokenizer_type: AutoTokenizer
tokenizer_use_fast: false
resize_token_embeddings_to_32x: false

flash_attention: true
xformers_attention:

load_in_8bit: false
load_in_4bit: false
strict: false

chat_template: chatml
datasets:
  - path: skymizer/Sonnet3.5-SlimOrcaDedupCleaned-train
    type: chat_template
    field_messages: messages

test_datasets:
  - path: skymizer/Sonnet3.5-SlimOrcaDedupCleaned-test
    type: chat_template
    field_messages: messages
    split: train

hf_use_auth_token: true
dataset_prepared_path: pretokenized/slim-orca
output_dir: ./exp_output_artifacts

sequence_len: 2048
sample_packing: true
pad_to_sequence_len: true

eval_sample_packing: false
# eval_causal_lm_metrics: ["perplexity"]

wandb_project: "axolotl_mistral_sft"
wandb_entity:
wandb_watch:
wandb_name: "mistral-7B-v0.1-q-spase-v7-on-slim-orca-longer"
wandb_log_model:

gradient_accumulation_steps: 2
micro_batch_size: 16
eval_batch_size: 1
num_epochs: 3
optimizer: adamw_torch
lr_scheduler: cosine
learning_rate: 0.000005
warmup_ratio: 0.03
cosine_min_lr_ratio: 0.05
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.95
adam_eps: 0.000001
max_grad_norm: 1.0

train_on_inputs: false
group_by_length: false
bf16: true
fp16:
tf32: false

hub_model_id: "skymizer/mistral-7B-v0.1-q-sparse-v7-on-slim-orca-longer"

save_strategy: "steps"
save_steps: 100

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
local_rank:
logging_steps: 1

eval_steps: 50
eval_table_size:
eval_max_new_tokens: 2048
debug:
deepspeed: deepspeed_configs/zero3_bf16.json
fsdp:
fsdp_config:
seed: 42

mistral-7B-v0.1-q-sparse-v7-on-slim-orca-longer

This model is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.8195

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 5e-06
• train_batch_size: 16
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 2
• total_train_batch_size: 128
• total_eval_batch_size: 4
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.95) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 35
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
11.9413	0.0026	1	11.9233
6.9838	0.1277	50	6.8541
4.4134	0.2554	100	4.1669
2.4007	0.3831	150	2.3010
1.8395	0.5109	200	1.7859
1.5379	0.6386	250	1.5368
1.4082	0.7663	300	1.3234
1.2393	0.8940	350	1.2109
1.1378	1.0217	400	1.1401
1.0708	1.1494	450	1.0649
1.0354	1.2771	500	1.0288
1.0125	1.4049	550	0.9793
0.9736	1.5326	600	0.9478
0.9564	1.6603	650	0.9224
0.8883	1.7880	700	0.8997
0.8874	1.9157	750	0.8786
0.8209	2.0434	800	0.8632
0.7871	2.1711	850	0.8536
0.7876	2.2989	900	0.8447
0.7786	2.4266	950	0.8399
0.7749	2.5543	1000	0.8320
0.7688	2.6820	1050	0.8262
0.7247	2.8097	1100	0.8234
0.7637	2.9374	1150	0.8195

Framework versions

• Transformers 4.46.3
• Pytorch 2.5.1+cu124
• Datasets 3.1.0
• Tokenizers 0.20.3