Hugging Face's logo

legraphista
/
granite-20b-code-instruct-IMat-GGUF

Text Generation
GGUF
code
granite
quantized
GGUF
quantization
imat
imatrix
static
16bit
8bit
6bit
5bit
4bit
3bit
2bit
1bit
Eval Results
conversational
Model card Files Community
legraphista commited on
Commit
f2b6c73
•
1 Parent(s): 31914d3

Upload imatrix.log with huggingface_hub

Browse files
Files changed (1) hide show
  1. imatrix.log +159 -0
imatrix.log ADDED
@@ -0,0 +1,159 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ llama_model_loader: loaded meta data with 32 key-value pairs and 628 tensors from granite-20b-code-instruct-IMat-GGUF/granite-20b-code-instruct.Q8_0.gguf.hardlink.gguf (version GGUF V3 (latest))
2
+ llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
3
+ llama_model_loader: - kv 0: general.architecture str = starcoder
4
+ llama_model_loader: - kv 1: general.type str = model
5
+ llama_model_loader: - kv 2: general.name str = Granite 20b Code Instruct
6
+ llama_model_loader: - kv 3: general.finetune str = code-instruct
7
+ llama_model_loader: - kv 4: general.basename str = granite
8
+ llama_model_loader: - kv 5: general.size_label str = 20B
9
+ llama_model_loader: - kv 6: general.license str = apache-2.0
10
+ llama_model_loader: - kv 7: general.base_model.count u32 = 1
11
+ llama_model_loader: - kv 8: general.base_model.0.name str = Granite 20b Code Base
12
+ llama_model_loader: - kv 9: general.base_model.0.organization str = Ibm Granite
13
+ llama_model_loader: - kv 10: general.base_model.0.repo_url str = https://huggingface.co/ibm-granite/gr...
14
+ llama_model_loader: - kv 11: general.tags arr[str,3] = ["code", "granite", "text-generation"]
15
+ llama_model_loader: - kv 12: general.datasets arr[str,8] = ["bigcode/commitpackft", "TIGER-Lab/M...
16
+ llama_model_loader: - kv 13: starcoder.context_length u32 = 8192
17
+ llama_model_loader: - kv 14: starcoder.embedding_length u32 = 6144
18
+ llama_model_loader: - kv 15: starcoder.feed_forward_length u32 = 24576
19
+ llama_model_loader: - kv 16: starcoder.block_count u32 = 52
20
+ llama_model_loader: - kv 17: starcoder.attention.head_count u32 = 48
21
+ llama_model_loader: - kv 18: starcoder.attention.head_count_kv u32 = 1
22
+ llama_model_loader: - kv 19: starcoder.attention.layer_norm_epsilon f32 = 0.000010
23
+ llama_model_loader: - kv 20: general.file_type u32 = 7
24
+ llama_model_loader: - kv 21: tokenizer.ggml.model str = gpt2
25
+ llama_model_loader: - kv 22: tokenizer.ggml.pre str = refact
26
+ llama_model_loader: - kv 23: tokenizer.ggml.tokens arr[str,49152] = ["<|endoftext|>", "<fim_prefix>", "<f...
27
+ llama_model_loader: - kv 24: tokenizer.ggml.token_type arr[i32,49152] = [3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, ...
28
+ llama_model_loader: - kv 25: tokenizer.ggml.merges arr[str,48891] = ["Ġ Ġ", "ĠĠ ĠĠ", "ĠĠĠĠ ĠĠ...
29
+ llama_model_loader: - kv 26: tokenizer.ggml.bos_token_id u32 = 0
30
+ llama_model_loader: - kv 27: tokenizer.ggml.eos_token_id u32 = 0
31
+ llama_model_loader: - kv 28: tokenizer.ggml.unknown_token_id u32 = 0
32
+ llama_model_loader: - kv 29: tokenizer.ggml.padding_token_id u32 = 0
33
+ llama_model_loader: - kv 30: tokenizer.chat_template str = {% for message in messages %}\n{% if m...
34
+ llama_model_loader: - kv 31: general.quantization_version u32 = 2
35
+ llama_model_loader: - type f32: 419 tensors
36
+ llama_model_loader: - type q8_0: 209 tensors
37
+ llm_load_vocab: special tokens cache size = 19
38
+ llm_load_vocab: token to piece cache size = 0.2826 MB
39
+ llm_load_print_meta: format = GGUF V3 (latest)
40
+ llm_load_print_meta: arch = starcoder
41
+ llm_load_print_meta: vocab type = BPE
42
+ llm_load_print_meta: n_vocab = 49152
43
+ llm_load_print_meta: n_merges = 48891
44
+ llm_load_print_meta: vocab_only = 0
45
+ llm_load_print_meta: n_ctx_train = 8192
46
+ llm_load_print_meta: n_embd = 6144
47
+ llm_load_print_meta: n_layer = 52
48
+ llm_load_print_meta: n_head = 48
49
+ llm_load_print_meta: n_head_kv = 1
50
+ llm_load_print_meta: n_rot = 128
51
+ llm_load_print_meta: n_swa = 0
52
+ llm_load_print_meta: n_embd_head_k = 128
53
+ llm_load_print_meta: n_embd_head_v = 128
54
+ llm_load_print_meta: n_gqa = 48
55
+ llm_load_print_meta: n_embd_k_gqa = 128
56
+ llm_load_print_meta: n_embd_v_gqa = 128
57
+ llm_load_print_meta: f_norm_eps = 1.0e-05
58
+ llm_load_print_meta: f_norm_rms_eps = 0.0e+00
59
+ llm_load_print_meta: f_clamp_kqv = 0.0e+00
60
+ llm_load_print_meta: f_max_alibi_bias = 0.0e+00
61
+ llm_load_print_meta: f_logit_scale = 0.0e+00
62
+ llm_load_print_meta: n_ff = 24576
63
+ llm_load_print_meta: n_expert = 0
64
+ llm_load_print_meta: n_expert_used = 0
65
+ llm_load_print_meta: causal attn = 1
66
+ llm_load_print_meta: pooling type = 0
67
+ llm_load_print_meta: rope type = 0
68
+ llm_load_print_meta: rope scaling = linear
69
+ llm_load_print_meta: freq_base_train = 10000.0
70
+ llm_load_print_meta: freq_scale_train = 1
71
+ llm_load_print_meta: n_ctx_orig_yarn = 8192
72
+ llm_load_print_meta: rope_finetuned = unknown
73
+ llm_load_print_meta: ssm_d_conv = 0
74
+ llm_load_print_meta: ssm_d_inner = 0
75
+ llm_load_print_meta: ssm_d_state = 0
76
+ llm_load_print_meta: ssm_dt_rank = 0
77
+ llm_load_print_meta: ssm_dt_b_c_rms = 0
78
+ llm_load_print_meta: model type = ?B
79
+ llm_load_print_meta: model ftype = Q8_0
80
+ llm_load_print_meta: model params = 20.07 B
81
+ llm_load_print_meta: model size = 20.00 GiB (8.56 BPW)
82
+ llm_load_print_meta: general.name = Granite 20b Code Instruct
83
+ llm_load_print_meta: BOS token = 0 '<|endoftext|>'
84
+ llm_load_print_meta: EOS token = 0 '<|endoftext|>'
85
+ llm_load_print_meta: UNK token = 0 '<|endoftext|>'
86
+ llm_load_print_meta: PAD token = 0 '<|endoftext|>'
87
+ llm_load_print_meta: LF token = 145 'Ä'
88
+ llm_load_print_meta: EOT token = 0 '<|endoftext|>'
89
+ llm_load_print_meta: max token length = 512
90
+ ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
91
+ ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
92
+ ggml_cuda_init: found 1 CUDA devices:
93
+ Device 0: NVIDIA GeForce RTX 4090, compute capability 8.9, VMM: yes
94
+ llm_load_tensors: ggml ctx size = 0.55 MiB
95
+ llm_load_tensors: offloading 52 repeating layers to GPU
96
+ llm_load_tensors: offloading non-repeating layers to GPU
97
+ llm_load_tensors: offloaded 53/53 layers to GPU
98
+ llm_load_tensors: CPU buffer size = 498.00 MiB
99
+ llm_load_tensors: CUDA0 buffer size = 20292.38 MiB
100
+ ....................................................................................................
101
+ llama_new_context_with_model: n_ctx = 512
102
+ llama_new_context_with_model: n_batch = 512
103
+ llama_new_context_with_model: n_ubatch = 512
104
+ llama_new_context_with_model: flash_attn = 0
105
+ llama_new_context_with_model: freq_base = 10000.0
106
+ llama_new_context_with_model: freq_scale = 1
107
+ llama_kv_cache_init: CUDA0 KV buffer size = 13.00 MiB
108
+ llama_new_context_with_model: KV self size = 13.00 MiB, K (f16): 6.50 MiB, V (f16): 6.50 MiB
109
+ llama_new_context_with_model: CUDA_Host output buffer size = 0.19 MiB
110
+ llama_new_context_with_model: CUDA0 compute buffer size = 120.00 MiB
111
+ llama_new_context_with_model: CUDA_Host compute buffer size = 25.01 MiB
112
+ llama_new_context_with_model: graph nodes = 1933
113
+ llama_new_context_with_model: graph splits = 2
114
+
115
+ system_info: n_threads = 25 (n_threads_batch = 25) / 32 | AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | FMA = 1 | NEON = 0 | SVE = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |
116
+ compute_imatrix: tokenizing the input ..
117
+ compute_imatrix: tokenization took 113.128 ms
118
+ compute_imatrix: computing over 152 chunks with batch_size 512
119
+ compute_imatrix: 1.30 seconds per pass - ETA 3.28 minutes
120
+ [1]5.5316,[2]4.3038,[3]4.5649,[4]5.0288,[5]5.6039,[6]5.7152,[7]5.0016,[8]5.9597,[9]5.9637,
121
+ save_imatrix: stored collected data after 10 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
122
+ [10]6.7819,[11]7.0343,[12]6.4546,[13]7.0672,[14]7.6034,[15]8.3912,[16]8.5493,[17]9.1657,[18]9.4702,[19]9.7523,
123
+ save_imatrix: stored collected data after 20 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
124
+ [20]9.9486,[21]10.4114,[22]10.0740,[23]10.0213,[24]10.3530,[25]10.5995,[26]10.6331,[27]10.4010,[28]10.5028,[29]10.7926,
125
+ save_imatrix: stored collected data after 30 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
126
+ [30]11.0959,[31]11.0268,[32]11.1389,[33]11.3398,[34]11.6580,[35]11.7255,[36]11.4972,[37]10.8076,[38]10.3152,[39]10.2085,
127
+ save_imatrix: stored collected data after 40 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
128
+ [40]10.1415,[41]10.0520,[42]10.1053,[43]9.9861,[44]10.0245,[45]10.0257,[46]10.1471,[47]10.1358,[48]10.3092,[49]10.5322,
129
+ save_imatrix: stored collected data after 50 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
130
+ [50]10.4987,[51]10.8835,[52]11.2237,[53]11.6686,[54]11.9040,[55]12.2365,[56]12.2093,[57]12.1756,[58]12.1845,[59]12.3144,
131
+ save_imatrix: stored collected data after 60 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
132
+ [60]12.3865,[61]12.2209,[62]12.2149,[63]12.2819,[64]12.3703,[65]12.4174,[66]12.4922,[67]12.6439,[68]12.7639,[69]12.7716,
133
+ save_imatrix: stored collected data after 70 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
134
+ [70]12.8490,[71]12.8158,[72]12.6971,[73]12.6002,[74]12.4168,[75]12.3231,[76]12.4371,[77]12.5235,[78]12.4813,[79]12.4734,
135
+ save_imatrix: stored collected data after 80 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
136
+ [80]12.4798,[81]12.3699,[82]12.2741,[83]12.2127,[84]12.1877,[85]12.1598,[86]12.0773,[87]12.0180,[88]11.9234,[89]11.8501,
137
+ save_imatrix: stored collected data after 90 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
138
+ [90]11.7759,[91]11.7479,[92]11.6583,[93]11.5726,[94]11.4962,[95]11.4289,[96]11.4031,[97]11.3468,[98]11.3321,[99]11.2733,
139
+ save_imatrix: stored collected data after 100 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
140
+ [100]11.2175,[101]11.2073,[102]11.1154,[103]11.0428,[104]11.0297,[105]11.0836,[106]11.1179,[107]11.1815,[108]11.2633,[109]11.1341,
141
+ save_imatrix: stored collected data after 110 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
142
+ [110]11.0276,[111]10.9195,[112]10.7971,[113]10.6767,[114]10.5720,[115]10.4712,[116]10.3728,[117]10.3827,[118]10.3708,[119]10.3970,
143
+ save_imatrix: stored collected data after 120 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
144
+ [120]10.5022,[121]10.6177,[122]10.7352,[123]10.7968,[124]10.9187,[125]10.9587,[126]10.9877,[127]11.0106,[128]11.0613,[129]11.0879,
145
+ save_imatrix: stored collected data after 130 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
146
+ [130]11.0612,[131]11.0532,[132]11.0083,[133]11.0206,[134]11.0336,[135]11.0436,[136]11.0618,[137]11.0628,[138]11.0782,[139]11.0803,
147
+ save_imatrix: stored collected data after 140 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
148
+ [140]11.1238,[141]11.1233,[142]11.1360,[143]11.1245,[144]11.0680,[145]11.0480,[146]11.1301,[147]11.2386,[148]11.3118,[149]11.4027,
149
+ save_imatrix: stored collected data after 150 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
150
+ [150]11.4866,[151]11.5899,[152]11.6361,
151
+ save_imatrix: stored collected data after 152 chunks in granite-20b-code-instruct-IMat-GGUF/imatrix.dat
152
+
153
+ llama_print_timings: load time = 11508.22 ms
154
+ llama_print_timings: sample time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
155
+ llama_print_timings: prompt eval time = 173482.28 ms / 77824 tokens ( 2.23 ms per token, 448.60 tokens per second)
156
+ llama_print_timings: eval time = 0.00 ms / 1 runs ( 0.00 ms per token, inf tokens per second)
157
+ llama_print_timings: total time = 184251.37 ms / 77825 tokens
158
+
159
+ Final estimate: PPL = 11.6361 +/- 0.19854