See axolotl config

axolotl version: 0.6.0

base_model: peft-internal-testing/tiny-dummy-qwen2
batch_size: 32
bf16: true
chat_template: tokenizer_default_fallback_alpaca
datasets:
- format: custom
  path: argilla/databricks-dolly-15k-curated-en
  type:
    field_input: original-instruction
    field_instruction: original-instruction
    field_output: original-response
    format: '{instruction} {input}'
    no_input_format: '{instruction}'
    system_format: '{system}'
    system_prompt: ''
device_map: auto
eval_sample_packing: false
eval_steps: 200
flash_attention: true
gpu_memory_limit: 80GiB
group_by_length: true
hub_model_id: SystemAdmin123/tiny-dummy-qwen2
hub_strategy: checkpoint
learning_rate: 0.0002
logging_steps: 10
lr_scheduler: cosine
max_steps: 2500
micro_batch_size: 4
model_type: AutoModelForCausalLM
num_epochs: 100
optimizer: adamw_bnb_8bit
output_dir: /root/.sn56/axolotl/outputs/tiny-dummy-qwen2
pad_to_sequence_len: true
resize_token_embeddings_to_32x: false
sample_packing: false
save_steps: 400
save_total_limit: 1
sequence_len: 2048
tokenizer_type: Qwen2TokenizerFast
torch_dtype: bf16
trust_remote_code: true
val_set_size: 0.1
wandb_entity: ''
wandb_mode: online
wandb_name: peft-internal-testing/tiny-dummy-qwen2-argilla/databricks-dolly-15k-curated-en
wandb_project: Gradients-On-Demand
wandb_run: your_name
wandb_runid: default
warmup_ratio: 0.05

tiny-dummy-qwen2

This model is a fine-tuned version of peft-internal-testing/tiny-dummy-qwen2 on the argilla/databricks-dolly-15k-curated-en dataset. It achieves the following results on the evaluation set:

• Loss: 10.5858

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 4
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 125
• training_steps: 2500

Training results

Training Loss	Epoch	Step	Validation Loss
No log	0.0003	1	11.9292
11.78	0.0592	200	11.7501
10.974	0.1184	400	11.0010
10.6928	0.1776	600	10.6858
10.9148	0.2368	800	10.6098
10.6606	0.2959	1000	10.5931
10.5748	0.3551	1200	10.5911
10.6436	0.4143	1400	10.5852
10.5774	0.4735	1600	10.5880
10.707	0.5327	1800	10.5812
10.5304	0.5919	2000	10.5866
10.6148	0.6511	2200	10.5842
10.4931	0.7103	2400	10.5858

Framework versions

• Transformers 4.48.1
• Pytorch 2.4.1+cu124
• Datasets 3.2.0
• Tokenizers 0.21.0