[INFO|2025-05-29 23:14:31] tokenization_utils_base.py:2034 >> loading file tokenizer.model from cache at None [INFO|2025-05-29 23:14:31] tokenization_utils_base.py:2034 >> loading file tokenizer.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/tokenizer.json [INFO|2025-05-29 23:14:31] tokenization_utils_base.py:2034 >> loading file added_tokens.json from cache at None [INFO|2025-05-29 23:14:31] tokenization_utils_base.py:2034 >> loading file special_tokens_map.json from cache at None [INFO|2025-05-29 23:14:31] tokenization_utils_base.py:2034 >> loading file tokenizer_config.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/tokenizer_config.json [INFO|2025-05-29 23:14:31] tokenization_utils_base.py:2034 >> loading file chat_template.jinja from cache at None [INFO|2025-05-29 23:14:32] tokenization_utils_base.py:2304 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. [INFO|2025-05-29 23:14:32] logging.py:157 >> Loading dataset Codes_query_filtered_330k_ns_over8_1.json... [INFO|2025-05-29 23:14:48] configuration_utils.py:696 >> loading configuration file config.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/config.json [INFO|2025-05-29 23:14:48] configuration_utils.py:768 >> Model config LlamaConfig { "_name_or_path": "deepseek-ai/deepseek-coder-7b-instruct-v1.5", "architectures": [ "LlamaForCausalLM" ], "attention_bias": false, "attention_dropout": 0.0, "bos_token_id": 100000, "eos_token_id": 100015, "head_dim": 128, "hidden_act": "silu", "hidden_size": 4096, "initializer_range": 0.02, "intermediate_size": 11008, "max_position_embeddings": 4096, "mlp_bias": false, "model_type": "llama", "num_attention_heads": 32, "num_hidden_layers": 30, "num_key_value_heads": 32, "pretraining_tp": 1, "rms_norm_eps": 1e-06, "rope_scaling": null, "rope_theta": 10000.0, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.48.2", "use_cache": true, "vocab_size": 102400 } [WARNING|2025-05-29 23:14:48] logging.py:162 >> Input length is smaller than max length. Consider increase input length. [INFO|2025-05-29 23:14:48] logging.py:157 >> Using llama3 scaling strategy and setting scaling factor to 1.0. [INFO|2025-05-29 23:14:48] logging.py:157 >> Using block diagonal attention for sequence packing without cross-attention. [INFO|2025-05-29 23:14:48] logging.py:157 >> Liger kernel has been applied to the model. [INFO|2025-05-29 23:14:48] modeling_utils.py:3904 >> loading weights file model.safetensors from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/model.safetensors.index.json [INFO|2025-05-29 23:14:48] modeling_utils.py:1582 >> Instantiating LlamaForCausalLM model under default dtype torch.bfloat16. [INFO|2025-05-29 23:14:48] configuration_utils.py:1140 >> Generate config GenerationConfig { "bos_token_id": 100000, "eos_token_id": 100015 } [INFO|2025-05-29 23:14:52] modeling_utils.py:4888 >> All model checkpoint weights were used when initializing LlamaForCausalLM. [INFO|2025-05-29 23:14:52] modeling_utils.py:4896 >> All the weights of LlamaForCausalLM were initialized from the model checkpoint at deepseek-ai/deepseek-coder-7b-instruct-v1.5. If your task is similar to the task the model of the checkpoint was trained on, you can already use LlamaForCausalLM for predictions without further training. [INFO|2025-05-29 23:14:52] configuration_utils.py:1095 >> loading configuration file generation_config.json from cache at /home/kiho/.cache/huggingface/hub/models--deepseek-ai--deepseek-coder-7b-instruct-v1.5/snapshots/2a050a4c59d687a85324d32e147517992117ed30/generation_config.json [INFO|2025-05-29 23:14:52] configuration_utils.py:1140 >> Generate config GenerationConfig { "bos_token_id": 100000, "eos_token_id": 100015 } [INFO|2025-05-29 23:14:52] logging.py:157 >> Gradient checkpointing enabled. [INFO|2025-05-29 23:14:52] logging.py:157 >> Using torch SDPA for faster training and inference. [INFO|2025-05-29 23:14:52] logging.py:157 >> Upcasting trainable params to float32. [INFO|2025-05-29 23:14:52] logging.py:157 >> Fine-tuning method: Freeze [INFO|2025-05-29 23:14:52] logging.py:157 >> Set trainable layers: .28.,.29. [INFO|2025-05-29 23:14:52] logging.py:157 >> trainable params: 404,766,720 || all params: 6,910,365,696 || trainable%: 5.8574 [INFO|2025-05-29 23:14:52] trainer.py:741 >> Using auto half precision backend [INFO|2025-05-29 23:14:52] logging.py:157 >> Found linear modules: down_proj,up_proj,q_proj,v_proj,gate_proj,o_proj,k_proj [INFO|2025-05-29 23:14:52] logging.py:157 >> Using APOLLO optimizer with args: {'rank': 256, 'proj': 'random', 'proj_type': 'std', 'update_proj_gap': 200, 'scale': 1, 'scale_type': 'channel', 'scale_front': False}. [INFO|2025-05-29 23:14:52] trainer.py:2369 >> ***** Running training ***** [INFO|2025-05-29 23:14:52] trainer.py:2370 >> Num examples = 7,871 [INFO|2025-05-29 23:14:52] trainer.py:2371 >> Num Epochs = 1 [INFO|2025-05-29 23:14:52] trainer.py:2372 >> Instantaneous batch size per device = 16 [INFO|2025-05-29 23:14:52] trainer.py:2375 >> Total train batch size (w. parallel, distributed & accumulation) = 512 [INFO|2025-05-29 23:14:52] trainer.py:2376 >> Gradient Accumulation steps = 8 [INFO|2025-05-29 23:14:52] trainer.py:2377 >> Total optimization steps = 15 [INFO|2025-05-29 23:14:52] trainer.py:2378 >> Number of trainable parameters = 404,766,720 [INFO|2025-05-29 23:16:39] logging.py:157 >> {'loss': 0.7344, 'learning_rate': 4.9454e-05, 'epoch': 0.07, 'throughput': 19884.80} [INFO|2025-05-29 23:18:18] logging.py:157 >> {'loss': 0.7113, 'learning_rate': 4.7839e-05, 'epoch': 0.13, 'throughput': 20489.52} [INFO|2025-05-29 23:19:58] logging.py:157 >> {'loss': 0.7009, 'learning_rate': 4.5225e-05, 'epoch': 0.20, 'throughput': 20677.48} [INFO|2025-05-29 23:21:37] logging.py:157 >> {'loss': 0.6704, 'learning_rate': 4.1728e-05, 'epoch': 0.26, 'throughput': 20771.66} [INFO|2025-05-29 23:23:17] logging.py:157 >> {'loss': 0.6504, 'learning_rate': 3.7500e-05, 'epoch': 0.33, 'throughput': 20829.21} [INFO|2025-05-29 23:24:56] logging.py:157 >> {'loss': 0.6217, 'learning_rate': 3.2725e-05, 'epoch': 0.39, 'throughput': 20867.75} [INFO|2025-05-29 23:26:36] logging.py:157 >> {'loss': 0.6052, 'learning_rate': 2.7613e-05, 'epoch': 0.46, 'throughput': 20897.01} [INFO|2025-05-29 23:28:15] logging.py:157 >> {'loss': 0.6302, 'learning_rate': 2.2387e-05, 'epoch': 0.52, 'throughput': 20915.80} [INFO|2025-05-29 23:29:55] logging.py:157 >> {'loss': 0.5964, 'learning_rate': 1.7275e-05, 'epoch': 0.59, 'throughput': 20930.91} [INFO|2025-05-29 23:31:35] logging.py:157 >> {'loss': 0.6233, 'learning_rate': 1.2500e-05, 'epoch': 0.65, 'throughput': 20943.31} [INFO|2025-05-29 23:33:14] logging.py:157 >> {'loss': 0.5964, 'learning_rate': 8.2717e-06, 'epoch': 0.72, 'throughput': 20954.49} [INFO|2025-05-29 23:34:54] logging.py:157 >> {'loss': 0.6106, 'learning_rate': 4.7746e-06, 'epoch': 0.78, 'throughput': 20962.99} [INFO|2025-05-29 23:36:33] logging.py:157 >> {'loss': 0.5916, 'learning_rate': 2.1614e-06, 'epoch': 0.85, 'throughput': 20970.47} [INFO|2025-05-29 23:38:13] logging.py:157 >> {'loss': 0.5893, 'learning_rate': 5.4631e-07, 'epoch': 0.91, 'throughput': 20976.44} [INFO|2025-05-29 23:39:52] logging.py:157 >> {'loss': 0.6298, 'learning_rate': 0.0000e+00, 'epoch': 0.98, 'throughput': 20983.82} [INFO|2025-05-29 23:39:52] trainer.py:3910 >> Saving model checkpoint to saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1/checkpoint-15 [INFO|2025-05-29 23:39:52] configuration_utils.py:420 >> Configuration saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1/checkpoint-15/config.json [INFO|2025-05-29 23:39:52] configuration_utils.py:909 >> Configuration saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1/checkpoint-15/generation_config.json [INFO|2025-05-29 23:40:13] modeling_utils.py:2996 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 3 checkpoint shards. You can find where each parameters has been saved in the index located at saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1/checkpoint-15/model.safetensors.index.json. [INFO|2025-05-29 23:40:13] tokenization_utils_base.py:2491 >> tokenizer config file saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1/checkpoint-15/tokenizer_config.json [INFO|2025-05-29 23:40:13] tokenization_utils_base.py:2500 >> Special tokens file saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1/checkpoint-15/special_tokens_map.json [INFO|2025-05-29 23:40:13] trainer.py:2643 >> Training completed. Do not forget to share your model on huggingface.co/models =) [INFO|2025-05-29 23:40:13] trainer.py:3910 >> Saving model checkpoint to saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1 [INFO|2025-05-29 23:40:13] configuration_utils.py:420 >> Configuration saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1/config.json [INFO|2025-05-29 23:40:13] configuration_utils.py:909 >> Configuration saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1/generation_config.json [INFO|2025-05-29 23:40:34] modeling_utils.py:2996 >> The model is bigger than the maximum size per checkpoint (5GB) and is going to be split in 3 checkpoint shards. You can find where each parameters has been saved in the index located at saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1/model.safetensors.index.json. [INFO|2025-05-29 23:40:34] tokenization_utils_base.py:2491 >> tokenizer config file saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1/tokenizer_config.json [INFO|2025-05-29 23:40:34] tokenization_utils_base.py:2500 >> Special tokens file saved in saves/DeepSeek-Coder-7B-Instruct/freeze/deepseek_nlx_8_1/special_tokens_map.json [WARNING|2025-05-29 23:40:34] logging.py:162 >> No metric eval_loss to plot. [WARNING|2025-05-29 23:40:34] logging.py:162 >> No metric eval_accuracy to plot. [INFO|2025-05-29 23:40:34] modelcard.py:449 >> Dropping the following result as it does not have all the necessary fields: {'task': {'name': 'Causal Language Modeling', 'type': 'text-generation'}}