| 05/24/2024 18:37:53 - INFO - transformers.tokenization_utils_base - loading file tokenizer.json from cache at C:\Users\itos\.cache\huggingface\hub\models--meta-llama--Meta-Llama-3-8B\snapshots\62bd457b6fe961a42a631306577e622c83876cb6\tokenizer.json | |
| 05/24/2024 18:37:53 - INFO - transformers.tokenization_utils_base - loading file added_tokens.json from cache at None | |
| 05/24/2024 18:37:53 - INFO - transformers.tokenization_utils_base - loading file special_tokens_map.json from cache at C:\Users\itos\.cache\huggingface\hub\models--meta-llama--Meta-Llama-3-8B\snapshots\62bd457b6fe961a42a631306577e622c83876cb6\special_tokens_map.json | |
| 05/24/2024 18:37:53 - INFO - transformers.tokenization_utils_base - loading file tokenizer_config.json from cache at C:\Users\itos\.cache\huggingface\hub\models--meta-llama--Meta-Llama-3-8B\snapshots\62bd457b6fe961a42a631306577e622c83876cb6\tokenizer_config.json | |
| 05/24/2024 18:37:53 - WARNING - transformers.tokenization_utils_base - Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. | |
| 05/24/2024 18:37:53 - INFO - llamafactory.data.template - Add pad token: <|end_of_text|> | |
| 05/24/2024 18:37:53 - INFO - llamafactory.data.loader - Loading dataset itos_data_ko.json... | |
| 05/24/2024 18:37:54 - INFO - transformers.configuration_utils - loading configuration file config.json from cache at C:\Users\itos\.cache\huggingface\hub\models--meta-llama--Meta-Llama-3-8B\snapshots\62bd457b6fe961a42a631306577e622c83876cb6\config.json | |
| 05/24/2024 18:37:54 - INFO - transformers.configuration_utils - Model config LlamaConfig { | |
| "_name_or_path": "meta-llama/Meta-Llama-3-8B", | |
| "architectures": [ | |
| "LlamaForCausalLM" | |
| ], | |
| "attention_bias": false, | |
| "attention_dropout": 0.0, | |
| "bos_token_id": 128000, | |
| "eos_token_id": 128001, | |
| "hidden_act": "silu", | |
| "hidden_size": 4096, | |
| "initializer_range": 0.02, | |
| "intermediate_size": 14336, | |
| "max_position_embeddings": 8192, | |
| "model_type": "llama", | |
| "num_attention_heads": 32, | |
| "num_hidden_layers": 32, | |
| "num_key_value_heads": 8, | |
| "pretraining_tp": 1, | |
| "rms_norm_eps": 1e-05, | |
| "rope_scaling": null, | |
| "rope_theta": 500000.0, | |
| "tie_word_embeddings": false, | |
| "torch_dtype": "bfloat16", | |
| "transformers_version": "4.40.2", | |
| "use_cache": true, | |
| "vocab_size": 128256 | |
| } | |
| 05/24/2024 18:37:54 - INFO - transformers.modeling_utils - loading weights file model.safetensors from cache at C:\Users\itos\.cache\huggingface\hub\models--meta-llama--Meta-Llama-3-8B\snapshots\62bd457b6fe961a42a631306577e622c83876cb6\model.safetensors.index.json | |
| 05/24/2024 18:37:54 - INFO - transformers.modeling_utils - Instantiating LlamaForCausalLM model under default dtype torch.float16. | |
| 05/24/2024 18:37:54 - INFO - transformers.generation.configuration_utils - Generate config GenerationConfig { | |
| "bos_token_id": 128000, | |
| "eos_token_id": 128001 | |
| } | |
| 05/24/2024 18:38:27 - INFO - transformers.modeling_utils - All model checkpoint weights were used when initializing LlamaForCausalLM. | |
| 05/24/2024 18:38:27 - INFO - transformers.modeling_utils - All the weights of LlamaForCausalLM were initialized from the model checkpoint at meta-llama/Meta-Llama-3-8B. | |
| If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training. | |
| 05/24/2024 18:38:27 - INFO - transformers.generation.configuration_utils - loading configuration file generation_config.json from cache at C:\Users\itos\.cache\huggingface\hub\models--meta-llama--Meta-Llama-3-8B\snapshots\62bd457b6fe961a42a631306577e622c83876cb6\generation_config.json | |
| 05/24/2024 18:38:27 - INFO - transformers.generation.configuration_utils - Generate config GenerationConfig { | |
| "bos_token_id": 128000, | |
| "do_sample": true, | |
| "eos_token_id": 128001, | |
| "max_length": 4096, | |
| "temperature": 0.6, | |
| "top_p": 0.9 | |
| } | |
| 05/24/2024 18:38:27 - INFO - llamafactory.model.utils.checkpointing - Gradient checkpointing enabled. | |
| 05/24/2024 18:38:27 - INFO - llamafactory.model.utils.attention - Using torch SDPA for faster training and inference. | |
| 05/24/2024 18:38:27 - INFO - llamafactory.model.adapter - Upcasting trainable params to float32. | |
| 05/24/2024 18:38:27 - INFO - llamafactory.model.adapter - Fine-tuning method: LoRA | |
| 05/24/2024 18:38:27 - INFO - llamafactory.model.loader - trainable params: 3407872 || all params: 8033669120 || trainable%: 0.0424 | |
| 05/24/2024 18:38:27 - INFO - transformers.trainer - Using auto half precision backend | |
| 05/24/2024 18:38:28 - INFO - transformers.trainer - ***** Running training ***** | |
| 05/24/2024 18:38:28 - INFO - transformers.trainer - Num examples = 338 | |
| 05/24/2024 18:38:28 - INFO - transformers.trainer - Num Epochs = 3 | |
| 05/24/2024 18:38:28 - INFO - transformers.trainer - Instantaneous batch size per device = 2 | |
| 05/24/2024 18:38:28 - INFO - transformers.trainer - Total train batch size (w. parallel, distributed & accumulation) = 16 | |
| 05/24/2024 18:38:28 - INFO - transformers.trainer - Gradient Accumulation steps = 8 | |
| 05/24/2024 18:38:28 - INFO - transformers.trainer - Total optimization steps = 63 | |
| 05/24/2024 18:38:28 - INFO - transformers.trainer - Number of trainable parameters = 3,407,872 | |
| 05/24/2024 18:42:07 - INFO - llamafactory.extras.callbacks - {'loss': 2.9767, 'learning_rate': 4.9227e-05, 'epoch': 0.24} | |
| 05/24/2024 18:46:13 - INFO - llamafactory.extras.callbacks - {'loss': 2.9754, 'learning_rate': 4.6956e-05, 'epoch': 0.47} | |
| 05/24/2024 18:50:53 - INFO - llamafactory.extras.callbacks - {'loss': 2.9723, 'learning_rate': 4.3326e-05, 'epoch': 0.71} | |
| 05/24/2024 18:55:14 - INFO - llamafactory.extras.callbacks - {'loss': 2.7976, 'learning_rate': 3.8564e-05, 'epoch': 0.95} | |
| 05/24/2024 18:59:43 - INFO - llamafactory.extras.callbacks - {'loss': 2.7409, 'learning_rate': 3.2962e-05, 'epoch': 1.18} | |
| 05/24/2024 19:03:53 - INFO - llamafactory.extras.callbacks - {'loss': 2.4753, 'learning_rate': 2.6868e-05, 'epoch': 1.42} | |
| 05/24/2024 19:07:57 - INFO - llamafactory.extras.callbacks - {'loss': 2.5400, 'learning_rate': 2.0659e-05, 'epoch': 1.66} | |
| 05/24/2024 19:11:55 - INFO - llamafactory.extras.callbacks - {'loss': 2.4686, 'learning_rate': 1.4718e-05, 'epoch': 1.89} | |
| 05/24/2024 19:15:51 - INFO - llamafactory.extras.callbacks - {'loss': 2.4011, 'learning_rate': 9.4128e-06, 'epoch': 2.13} | |
| 05/24/2024 19:19:53 - INFO - llamafactory.extras.callbacks - {'loss': 2.4184, 'learning_rate': 5.0717e-06, 'epoch': 2.37} | |
| 05/24/2024 19:23:54 - INFO - llamafactory.extras.callbacks - {'loss': 2.3458, 'learning_rate': 1.9631e-06, 'epoch': 2.60} | |
| 05/24/2024 19:27:55 - INFO - llamafactory.extras.callbacks - {'loss': 2.3665, 'learning_rate': 2.7923e-07, 'epoch': 2.84} | |
| 05/24/2024 19:30:27 - INFO - transformers.trainer - | |
| Training completed. Do not forget to share your model on huggingface.co/models =) | |
| 05/24/2024 19:30:27 - INFO - transformers.trainer - Saving model checkpoint to saves\LLaMA3-8B\lora\train_2024-05-24-18-35-12 | |
| 05/24/2024 19:30:27 - INFO - transformers.configuration_utils - loading configuration file config.json from cache at C:\Users\itos\.cache\huggingface\hub\models--meta-llama--Meta-Llama-3-8B\snapshots\62bd457b6fe961a42a631306577e622c83876cb6\config.json | |
| 05/24/2024 19:30:27 - INFO - transformers.configuration_utils - Model config LlamaConfig { | |
| "architectures": [ | |
| "LlamaForCausalLM" | |
| ], | |
| "attention_bias": false, | |
| "attention_dropout": 0.0, | |
| "bos_token_id": 128000, | |
| "eos_token_id": 128001, | |
| "hidden_act": "silu", | |
| "hidden_size": 4096, | |
| "initializer_range": 0.02, | |
| "intermediate_size": 14336, | |
| "max_position_embeddings": 8192, | |
| "model_type": "llama", | |
| "num_attention_heads": 32, | |
| "num_hidden_layers": 32, | |
| "num_key_value_heads": 8, | |
| "pretraining_tp": 1, | |
| "rms_norm_eps": 1e-05, | |
| "rope_scaling": null, | |
| "rope_theta": 500000.0, | |
| "tie_word_embeddings": false, | |
| "torch_dtype": "bfloat16", | |
| "transformers_version": "4.40.2", | |
| "use_cache": true, | |
| "vocab_size": 128256 | |
| } | |
| 05/24/2024 19:30:27 - INFO - transformers.tokenization_utils_base - tokenizer config file saved in saves\LLaMA3-8B\lora\train_2024-05-24-18-35-12\tokenizer_config.json | |
| 05/24/2024 19:30:27 - INFO - transformers.tokenization_utils_base - Special tokens file saved in saves\LLaMA3-8B\lora\train_2024-05-24-18-35-12\special_tokens_map.json | |
| 05/24/2024 19:30:27 - INFO - transformers.modelcard - Dropping the following result as it does not have all the necessary fields: | |
| {'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}} | |