End of training

README.md

@@ -20,7 +20,7 @@ should probably proofread and complete it, then remove this comment. -->
 axolotl version: `0.6.0`
 ```yaml
 base_model: trl-internal-testing/tiny-random-LlamaForCausalLM
-batch_size: 256
+batch_size: 128
 bf16: true
 chat_template: tokenizer_default_fallback_alpaca
 datasets:
@@ -45,7 +45,7 @@ hub_strategy: checkpoint
 learning_rate: 0.0002
 logging_steps: 10
 lr_scheduler: cosine
-max_steps: 20000
+max_steps: 10000
 micro_batch_size: 32
 model_type: AutoModelForCausalLM
 num_epochs: 100
@@ -59,6 +59,8 @@ save_total_limit: 1
 sequence_len: 2048
 tokenizer_type: LlamaTokenizerFast
 torch_dtype: bf16
+training_args_kwargs:
+  hub_private_repo: true
 trust_remote_code: true
 val_set_size: 0.1
 wandb_entity: ''
@@ -76,8 +78,6 @@ warmup_ratio: 0.05
 # test-repo
 
 This model is a fine-tuned version of [trl-internal-testing/tiny-random-LlamaForCausalLM](https://huggingface.co/trl-internal-testing/tiny-random-LlamaForCausalLM) on the argilla/databricks-dolly-15k-curated-en dataset.
-It achieves the following results on the evaluation set:
-- Loss: 9.6817
 
 ## Model description
 
@@ -101,19 +101,19 @@ The following hyperparameters were used during training:
 - eval_batch_size: 32
 - seed: 42
 - distributed_type: multi-GPU
-- num_devices: 8
-- total_train_batch_size: 256
-- total_eval_batch_size: 256
+- num_devices: 4
+- total_train_batch_size: 128
+- total_eval_batch_size: 128
 - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
 - lr_scheduler_type: cosine
-- training_steps: 0
+- lr_scheduler_warmup_steps: 5
+- training_steps: 100
 
 ### Training results
 
-| Training Loss | Epoch   | Step | Validation Loss |
-|:-------------:|:-------:|:----:|:---------------:|
-| No log        | 0.3333  | 1    | 10.3756         |
-| 9.6746        | 66.6667 | 200  | 9.6817          |
+| Training Loss | Epoch  | Step | Validation Loss |
+|:-------------:|:------:|:----:|:---------------:|
+| No log        | 0.1667 | 1    | 10.3764         |
 
 
 ### Framework versions