Hugging Face's logo

asas-ai
/
bloom_7B_8bit_qlora_arc

TensorBoard
Generated from Trainer
Model card Files Metrics Community
SaiedAlshahrani commited on
Commit
5a93b4f
·
1 Parent(s): a6605fd

Upload files manually

Browse files
Files changed (43) hide show
  1. README.md +54 -0
  2. adapter_config.json +23 -0
  3. adapter_model.bin +3 -0
  4. checkpoint-1760/README.md +34 -0
  5. checkpoint-1760/adapter_config.json +23 -0
  6. checkpoint-1760/adapter_model.bin +3 -0
  7. checkpoint-1760/optimizer.pt +3 -0
  8. checkpoint-1760/rng_state.pth +3 -0
  9. checkpoint-1760/scheduler.pt +3 -0
  10. checkpoint-1760/special_tokens_map.json +6 -0
  11. checkpoint-1760/tokenizer.json +3 -0
  12. checkpoint-1760/tokenizer_config.json +11 -0
  13. checkpoint-1760/trainer_state.json +67 -0
  14. checkpoint-1760/training_args.bin +3 -0
  15. checkpoint-1980/README.md +34 -0
  16. checkpoint-1980/adapter_config.json +23 -0
  17. checkpoint-1980/adapter_model.bin +3 -0
  18. checkpoint-1980/optimizer.pt +3 -0
  19. checkpoint-1980/rng_state.pth +3 -0
  20. checkpoint-1980/scheduler.pt +3 -0
  21. checkpoint-1980/special_tokens_map.json +6 -0
  22. checkpoint-1980/tokenizer.json +3 -0
  23. checkpoint-1980/tokenizer_config.json +11 -0
  24. checkpoint-1980/trainer_state.json +73 -0
  25. checkpoint-1980/training_args.bin +3 -0
  26. checkpoint-2200/README.md +34 -0
  27. checkpoint-2200/adapter_config.json +23 -0
  28. checkpoint-2200/adapter_model.bin +3 -0
  29. checkpoint-2200/optimizer.pt +3 -0
  30. checkpoint-2200/rng_state.pth +3 -0
  31. checkpoint-2200/scheduler.pt +3 -0
  32. checkpoint-2200/special_tokens_map.json +6 -0
  33. checkpoint-2200/tokenizer.json +3 -0
  34. checkpoint-2200/tokenizer_config.json +11 -0
  35. checkpoint-2200/trainer_state.json +79 -0
  36. checkpoint-2200/training_args.bin +3 -0
  37. runs/Sep06_11-15-08_nfsixhvmvf/events.out.tfevents.1693998946.nfsixhvmvf.111.0 +3 -0
  38. runs/Sep06_11-17-15_nfsixhvmvf/events.out.tfevents.1693999043.nfsixhvmvf.111.1 +3 -0
  39. runs/Sep06_11-20-19_nfsixhvmvf/events.out.tfevents.1693999226.nfsixhvmvf.111.2 +3 -0
  40. runs/Sep06_11-20-59_nfsixhvmvf/events.out.tfevents.1693999263.nfsixhvmvf.111.3 +3 -0
  41. runs/Sep06_11-21-56_nfsixhvmvf/events.out.tfevents.1693999359.nfsixhvmvf.246.0 +3 -0
  42. runs/Sep06_11-38-22_nfsixhvmvf/events.out.tfevents.1694000340.nfsixhvmvf.356.0 +3 -0
  43. training_args.bin +3 -0
README.md ADDED
@@ -0,0 +1,54 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: asas-ai/bloom_7B_8bit
3
+ tags:
4
+ - generated_from_trainer
5
+ model-index:
6
+ - name: bloom_7B_8bit_qlora_arc
7
+ results: []
8
+ ---
9
+
10
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
11
+ should probably proofread and complete it, then remove this comment. -->
12
+
13
+ # bloom_7B_8bit_qlora_arc
14
+
15
+ This model is a fine-tuned version of [asas-ai/bloom_7B_8bit](https://huggingface.co/asas-ai/bloom_7B_8bit) on an unknown dataset.
16
+
17
+ ## Model description
18
+
19
+ More information needed
20
+
21
+ ## Intended uses & limitations
22
+
23
+ More information needed
24
+
25
+ ## Training and evaluation data
26
+
27
+ More information needed
28
+
29
+ ## Training procedure
30
+
31
+ ### Training hyperparameters
32
+
33
+ The following hyperparameters were used during training:
34
+ - learning_rate: 0.0002
35
+ - train_batch_size: 4
36
+ - eval_batch_size: 8
37
+ - seed: 42
38
+ - gradient_accumulation_steps: 4
39
+ - total_train_batch_size: 16
40
+ - optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
41
+ - lr_scheduler_type: constant
42
+ - lr_scheduler_warmup_ratio: 0.03
43
+ - training_steps: 2200
44
+
45
+ ### Training results
46
+
47
+
48
+
49
+ ### Framework versions
50
+
51
+ - Transformers 4.33.0
52
+ - Pytorch 2.0.1+cu117
53
+ - Datasets 2.4.0
54
+ - Tokenizers 0.12.1
adapter_config.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "auto_mapping": null,
3
+ "base_model_name_or_path": "asas-ai/bloom_7B_8bit",
4
+ "bias": "none",
5
+ "fan_in_fan_out": false,
6
+ "inference_mode": true,
7
+ "init_lora_weights": true,
8
+ "layers_pattern": null,
9
+ "layers_to_transform": null,
10
+ "lora_alpha": 16,
11
+ "lora_dropout": 0.1,
12
+ "modules_to_save": null,
13
+ "peft_type": "LORA",
14
+ "r": 64,
15
+ "revision": null,
16
+ "target_modules": [
17
+ "query_key_value",
18
+ "dense",
19
+ "dense_h_to_4h",
20
+ "dense_4h_to_h"
21
+ ],
22
+ "task_type": "CAUSAL_LM"
23
+ }
adapter_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:af96addcc3df02d705130db5585ac543f7422c7ede444303bc4142422acd06c4
3
+ size 503404621
checkpoint-1760/README.md ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ ---
4
+ ## Training procedure
5
+
6
+
7
+ The following `bitsandbytes` quantization config was used during training:
8
+ - quant_method: bitsandbytes
9
+ - load_in_8bit: True
10
+ - load_in_4bit: False
11
+ - llm_int8_threshold: 6.0
12
+ - llm_int8_skip_modules: None
13
+ - llm_int8_enable_fp32_cpu_offload: False
14
+ - llm_int8_has_fp16_weight: False
15
+ - bnb_4bit_quant_type: fp4
16
+ - bnb_4bit_use_double_quant: False
17
+ - bnb_4bit_compute_dtype: float32
18
+
19
+ The following `bitsandbytes` quantization config was used during training:
20
+ - quant_method: bitsandbytes
21
+ - load_in_8bit: True
22
+ - load_in_4bit: False
23
+ - llm_int8_threshold: 6.0
24
+ - llm_int8_skip_modules: None
25
+ - llm_int8_enable_fp32_cpu_offload: False
26
+ - llm_int8_has_fp16_weight: False
27
+ - bnb_4bit_quant_type: fp4
28
+ - bnb_4bit_use_double_quant: False
29
+ - bnb_4bit_compute_dtype: float32
30
+ ### Framework versions
31
+
32
+ - PEFT 0.6.0.dev0
33
+
34
+ - PEFT 0.6.0.dev0
checkpoint-1760/adapter_config.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "auto_mapping": null,
3
+ "base_model_name_or_path": "asas-ai/bloom_7B_8bit",
4
+ "bias": "none",
5
+ "fan_in_fan_out": false,
6
+ "inference_mode": true,
7
+ "init_lora_weights": true,
8
+ "layers_pattern": null,
9
+ "layers_to_transform": null,
10
+ "lora_alpha": 16,
11
+ "lora_dropout": 0.1,
12
+ "modules_to_save": null,
13
+ "peft_type": "LORA",
14
+ "r": 64,
15
+ "revision": null,
16
+ "target_modules": [
17
+ "query_key_value",
18
+ "dense",
19
+ "dense_h_to_4h",
20
+ "dense_4h_to_h"
21
+ ],
22
+ "task_type": "CAUSAL_LM"
23
+ }
checkpoint-1760/adapter_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c877e22c0e0ab661a02dbcdaca7f7a9c50f6ad48f10a4ed3be145e883498b92f
3
+ size 503404621
checkpoint-1760/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d83ded641f493273906d28768b3fb9c0592125226e2c115b4d67385c7a4bee79
3
+ size 1006781317
checkpoint-1760/rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0200b04cdc2112bb321a998147aa5826bba7fcd27ec175903e3e83291579152f
3
+ size 14575
checkpoint-1760/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:85c5ff61dc870ad27d377b69c3d781f714f8b44a93915b84c28547ea26cc839f
3
+ size 627
checkpoint-1760/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "</s>",
5
+ "unk_token": "<unk>"
6
+ }
checkpoint-1760/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f6efc66e73f1fd69da4f436e48befb519fdff3fe18910850c1d41bd862293a5
3
+ size 14500443
checkpoint-1760/tokenizer_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "bos_token": "<s>",
4
+ "clean_up_tokenization_spaces": false,
5
+ "eos_token": "</s>",
6
+ "model_max_length": 1000000000000000019884624838656,
7
+ "pad_token": "<pad>",
8
+ "padding_side": "left",
9
+ "tokenizer_class": "BloomTokenizer",
10
+ "unk_token": "<unk>"
11
+ }
checkpoint-1760/trainer_state.json ADDED
@@ -0,0 +1,67 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 12.59391771019678,
5
+ "eval_steps": 500,
6
+ "global_step": 1760,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 1.57,
13
+ "learning_rate": 0.0002,
14
+ "loss": 1.6688,
15
+ "step": 220
16
+ },
17
+ {
18
+ "epoch": 3.15,
19
+ "learning_rate": 0.0002,
20
+ "loss": 0.9324,
21
+ "step": 440
22
+ },
23
+ {
24
+ "epoch": 4.72,
25
+ "learning_rate": 0.0002,
26
+ "loss": 0.416,
27
+ "step": 660
28
+ },
29
+ {
30
+ "epoch": 6.3,
31
+ "learning_rate": 0.0002,
32
+ "loss": 0.2126,
33
+ "step": 880
34
+ },
35
+ {
36
+ "epoch": 7.87,
37
+ "learning_rate": 0.0002,
38
+ "loss": 0.1331,
39
+ "step": 1100
40
+ },
41
+ {
42
+ "epoch": 9.45,
43
+ "learning_rate": 0.0002,
44
+ "loss": 0.1011,
45
+ "step": 1320
46
+ },
47
+ {
48
+ "epoch": 11.02,
49
+ "learning_rate": 0.0002,
50
+ "loss": 0.0861,
51
+ "step": 1540
52
+ },
53
+ {
54
+ "epoch": 12.59,
55
+ "learning_rate": 0.0002,
56
+ "loss": 0.0725,
57
+ "step": 1760
58
+ }
59
+ ],
60
+ "logging_steps": 220,
61
+ "max_steps": 2200,
62
+ "num_train_epochs": 16,
63
+ "save_steps": 220,
64
+ "total_flos": 8.220211524717773e+16,
65
+ "trial_name": null,
66
+ "trial_params": null
67
+ }
checkpoint-1760/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b89f37cc2ed67a4a3cb46729dd6f7609d2e1cb7f3b1103cc262e661a64a7c9b
3
+ size 4091
checkpoint-1980/README.md ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ ---
4
+ ## Training procedure
5
+
6
+
7
+ The following `bitsandbytes` quantization config was used during training:
8
+ - quant_method: bitsandbytes
9
+ - load_in_8bit: True
10
+ - load_in_4bit: False
11
+ - llm_int8_threshold: 6.0
12
+ - llm_int8_skip_modules: None
13
+ - llm_int8_enable_fp32_cpu_offload: False
14
+ - llm_int8_has_fp16_weight: False
15
+ - bnb_4bit_quant_type: fp4
16
+ - bnb_4bit_use_double_quant: False
17
+ - bnb_4bit_compute_dtype: float32
18
+
19
+ The following `bitsandbytes` quantization config was used during training:
20
+ - quant_method: bitsandbytes
21
+ - load_in_8bit: True
22
+ - load_in_4bit: False
23
+ - llm_int8_threshold: 6.0
24
+ - llm_int8_skip_modules: None
25
+ - llm_int8_enable_fp32_cpu_offload: False
26
+ - llm_int8_has_fp16_weight: False
27
+ - bnb_4bit_quant_type: fp4
28
+ - bnb_4bit_use_double_quant: False
29
+ - bnb_4bit_compute_dtype: float32
30
+ ### Framework versions
31
+
32
+ - PEFT 0.6.0.dev0
33
+
34
+ - PEFT 0.6.0.dev0
checkpoint-1980/adapter_config.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "auto_mapping": null,
3
+ "base_model_name_or_path": "asas-ai/bloom_7B_8bit",
4
+ "bias": "none",
5
+ "fan_in_fan_out": false,
6
+ "inference_mode": true,
7
+ "init_lora_weights": true,
8
+ "layers_pattern": null,
9
+ "layers_to_transform": null,
10
+ "lora_alpha": 16,
11
+ "lora_dropout": 0.1,
12
+ "modules_to_save": null,
13
+ "peft_type": "LORA",
14
+ "r": 64,
15
+ "revision": null,
16
+ "target_modules": [
17
+ "query_key_value",
18
+ "dense",
19
+ "dense_h_to_4h",
20
+ "dense_4h_to_h"
21
+ ],
22
+ "task_type": "CAUSAL_LM"
23
+ }
checkpoint-1980/adapter_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9da1ec11be660a1a30bed83115969c4f358b55bc44b34b1733ca2df169f6ab0c
3
+ size 503404621
checkpoint-1980/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8de02ebec49b3db967ae527713abb78d9a0e503faf567e4828b1855cc8d68f87
3
+ size 1006781317
checkpoint-1980/rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:24d07816a86d28d5815c62309cf4ec0bb3ef4d1e17314b42ff617b0a398b52cb
3
+ size 14575
checkpoint-1980/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0b7b3cad452dc4595e700cca73b74b2842931f81f5ad69a77dcc8aa064e6bd19
3
+ size 627
checkpoint-1980/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "</s>",
5
+ "unk_token": "<unk>"
6
+ }
checkpoint-1980/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f6efc66e73f1fd69da4f436e48befb519fdff3fe18910850c1d41bd862293a5
3
+ size 14500443
checkpoint-1980/tokenizer_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "bos_token": "<s>",
4
+ "clean_up_tokenization_spaces": false,
5
+ "eos_token": "</s>",
6
+ "model_max_length": 1000000000000000019884624838656,
7
+ "pad_token": "<pad>",
8
+ "padding_side": "left",
9
+ "tokenizer_class": "BloomTokenizer",
10
+ "unk_token": "<unk>"
11
+ }
checkpoint-1980/trainer_state.json ADDED
@@ -0,0 +1,73 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 14.168157423971378,
5
+ "eval_steps": 500,
6
+ "global_step": 1980,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 1.57,
13
+ "learning_rate": 0.0002,
14
+ "loss": 1.6688,
15
+ "step": 220
16
+ },
17
+ {
18
+ "epoch": 3.15,
19
+ "learning_rate": 0.0002,
20
+ "loss": 0.9324,
21
+ "step": 440
22
+ },
23
+ {
24
+ "epoch": 4.72,
25
+ "learning_rate": 0.0002,
26
+ "loss": 0.416,
27
+ "step": 660
28
+ },
29
+ {
30
+ "epoch": 6.3,
31
+ "learning_rate": 0.0002,
32
+ "loss": 0.2126,
33
+ "step": 880
34
+ },
35
+ {
36
+ "epoch": 7.87,
37
+ "learning_rate": 0.0002,
38
+ "loss": 0.1331,
39
+ "step": 1100
40
+ },
41
+ {
42
+ "epoch": 9.45,
43
+ "learning_rate": 0.0002,
44
+ "loss": 0.1011,
45
+ "step": 1320
46
+ },
47
+ {
48
+ "epoch": 11.02,
49
+ "learning_rate": 0.0002,
50
+ "loss": 0.0861,
51
+ "step": 1540
52
+ },
53
+ {
54
+ "epoch": 12.59,
55
+ "learning_rate": 0.0002,
56
+ "loss": 0.0725,
57
+ "step": 1760
58
+ },
59
+ {
60
+ "epoch": 14.17,
61
+ "learning_rate": 0.0002,
62
+ "loss": 0.0713,
63
+ "step": 1980
64
+ }
65
+ ],
66
+ "logging_steps": 220,
67
+ "max_steps": 2200,
68
+ "num_train_epochs": 16,
69
+ "save_steps": 220,
70
+ "total_flos": 9.239634211022438e+16,
71
+ "trial_name": null,
72
+ "trial_params": null
73
+ }
checkpoint-1980/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b89f37cc2ed67a4a3cb46729dd6f7609d2e1cb7f3b1103cc262e661a64a7c9b
3
+ size 4091
checkpoint-2200/README.md ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ ---
4
+ ## Training procedure
5
+
6
+
7
+ The following `bitsandbytes` quantization config was used during training:
8
+ - quant_method: bitsandbytes
9
+ - load_in_8bit: True
10
+ - load_in_4bit: False
11
+ - llm_int8_threshold: 6.0
12
+ - llm_int8_skip_modules: None
13
+ - llm_int8_enable_fp32_cpu_offload: False
14
+ - llm_int8_has_fp16_weight: False
15
+ - bnb_4bit_quant_type: fp4
16
+ - bnb_4bit_use_double_quant: False
17
+ - bnb_4bit_compute_dtype: float32
18
+
19
+ The following `bitsandbytes` quantization config was used during training:
20
+ - quant_method: bitsandbytes
21
+ - load_in_8bit: True
22
+ - load_in_4bit: False
23
+ - llm_int8_threshold: 6.0
24
+ - llm_int8_skip_modules: None
25
+ - llm_int8_enable_fp32_cpu_offload: False
26
+ - llm_int8_has_fp16_weight: False
27
+ - bnb_4bit_quant_type: fp4
28
+ - bnb_4bit_use_double_quant: False
29
+ - bnb_4bit_compute_dtype: float32
30
+ ### Framework versions
31
+
32
+ - PEFT 0.6.0.dev0
33
+
34
+ - PEFT 0.6.0.dev0
checkpoint-2200/adapter_config.json ADDED
@@ -0,0 +1,23 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "auto_mapping": null,
3
+ "base_model_name_or_path": "asas-ai/bloom_7B_8bit",
4
+ "bias": "none",
5
+ "fan_in_fan_out": false,
6
+ "inference_mode": true,
7
+ "init_lora_weights": true,
8
+ "layers_pattern": null,
9
+ "layers_to_transform": null,
10
+ "lora_alpha": 16,
11
+ "lora_dropout": 0.1,
12
+ "modules_to_save": null,
13
+ "peft_type": "LORA",
14
+ "r": 64,
15
+ "revision": null,
16
+ "target_modules": [
17
+ "query_key_value",
18
+ "dense",
19
+ "dense_h_to_4h",
20
+ "dense_4h_to_h"
21
+ ],
22
+ "task_type": "CAUSAL_LM"
23
+ }
checkpoint-2200/adapter_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:af96addcc3df02d705130db5585ac543f7422c7ede444303bc4142422acd06c4
3
+ size 503404621
checkpoint-2200/optimizer.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:768ace7e3570341505534a8c4ddad40f2c8245df574bbf95552e8bafceaaf5d9
3
+ size 1006781317
checkpoint-2200/rng_state.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c796ce6e152e9f3b62b3cf6befca4b59bca03bc90534489e320f83a57d55f495
3
+ size 14575
checkpoint-2200/scheduler.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f10a983aa914555fea6e5c0db8d7ddbaebbe7e28546c78ee0e93ac76cbc28436
3
+ size 627
checkpoint-2200/special_tokens_map.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": "<s>",
3
+ "eos_token": "</s>",
4
+ "pad_token": "</s>",
5
+ "unk_token": "<unk>"
6
+ }
checkpoint-2200/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8f6efc66e73f1fd69da4f436e48befb519fdff3fe18910850c1d41bd862293a5
3
+ size 14500443
checkpoint-2200/tokenizer_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "bos_token": "<s>",
4
+ "clean_up_tokenization_spaces": false,
5
+ "eos_token": "</s>",
6
+ "model_max_length": 1000000000000000019884624838656,
7
+ "pad_token": "<pad>",
8
+ "padding_side": "left",
9
+ "tokenizer_class": "BloomTokenizer",
10
+ "unk_token": "<unk>"
11
+ }
checkpoint-2200/trainer_state.json ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "best_metric": null,
3
+ "best_model_checkpoint": null,
4
+ "epoch": 15.742397137745975,
5
+ "eval_steps": 500,
6
+ "global_step": 2200,
7
+ "is_hyper_param_search": false,
8
+ "is_local_process_zero": true,
9
+ "is_world_process_zero": true,
10
+ "log_history": [
11
+ {
12
+ "epoch": 1.57,
13
+ "learning_rate": 0.0002,
14
+ "loss": 1.6688,
15
+ "step": 220
16
+ },
17
+ {
18
+ "epoch": 3.15,
19
+ "learning_rate": 0.0002,
20
+ "loss": 0.9324,
21
+ "step": 440
22
+ },
23
+ {
24
+ "epoch": 4.72,
25
+ "learning_rate": 0.0002,
26
+ "loss": 0.416,
27
+ "step": 660
28
+ },
29
+ {
30
+ "epoch": 6.3,
31
+ "learning_rate": 0.0002,
32
+ "loss": 0.2126,
33
+ "step": 880
34
+ },
35
+ {
36
+ "epoch": 7.87,
37
+ "learning_rate": 0.0002,
38
+ "loss": 0.1331,
39
+ "step": 1100
40
+ },
41
+ {
42
+ "epoch": 9.45,
43
+ "learning_rate": 0.0002,
44
+ "loss": 0.1011,
45
+ "step": 1320
46
+ },
47
+ {
48
+ "epoch": 11.02,
49
+ "learning_rate": 0.0002,
50
+ "loss": 0.0861,
51
+ "step": 1540
52
+ },
53
+ {
54
+ "epoch": 12.59,
55
+ "learning_rate": 0.0002,
56
+ "loss": 0.0725,
57
+ "step": 1760
58
+ },
59
+ {
60
+ "epoch": 14.17,
61
+ "learning_rate": 0.0002,
62
+ "loss": 0.0713,
63
+ "step": 1980
64
+ },
65
+ {
66
+ "epoch": 15.74,
67
+ "learning_rate": 0.0002,
68
+ "loss": 0.0721,
69
+ "step": 2200
70
+ }
71
+ ],
72
+ "logging_steps": 220,
73
+ "max_steps": 2200,
74
+ "num_train_epochs": 16,
75
+ "save_steps": 220,
76
+ "total_flos": 1.0258908883550208e+17,
77
+ "trial_name": null,
78
+ "trial_params": null
79
+ }
checkpoint-2200/training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b89f37cc2ed67a4a3cb46729dd6f7609d2e1cb7f3b1103cc262e661a64a7c9b
3
+ size 4091
runs/Sep06_11-15-08_nfsixhvmvf/events.out.tfevents.1693998946.nfsixhvmvf.111.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ec5e5bd8956237a4faa6c84f9a91fe1bd669ff5e0bf36e0c7d27475703c42a4a
3
+ size 4657
runs/Sep06_11-17-15_nfsixhvmvf/events.out.tfevents.1693999043.nfsixhvmvf.111.1 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:427f5e84df17304c1f15b1cd4fa67f7fa1af3a531e29d03cee42990ac1ee615c
3
+ size 4658
runs/Sep06_11-20-19_nfsixhvmvf/events.out.tfevents.1693999226.nfsixhvmvf.111.2 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4a989a0b6ba185aebc3de74fefdf41818fe75eaf7f7fa60d284793f1442ce075
3
+ size 40
runs/Sep06_11-20-59_nfsixhvmvf/events.out.tfevents.1693999263.nfsixhvmvf.111.3 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0e63e807ec456b497fcbf91b8ac0825011fe37d0b6c35ac325c958ade4caea6a
3
+ size 40
runs/Sep06_11-21-56_nfsixhvmvf/events.out.tfevents.1693999359.nfsixhvmvf.246.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0ded344ef1563f6a37eaa7b5691a81cec71c298274e575ded354c37a7afcc7c9
3
+ size 4969
runs/Sep06_11-38-22_nfsixhvmvf/events.out.tfevents.1694000340.nfsixhvmvf.356.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:89de9f768e8bb58417ff108a38c1290975aeaf5a7ddc894d34052b8cfdd7d862
3
+ size 6582
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b89f37cc2ed67a4a3cb46729dd6f7609d2e1cb7f3b1103cc262e661a64a7c9b
3
+ size 4091