Lakoc
/

gpt2_512h_16l_add_head14

gpt2-multi-head

Model card Files Files and versions Community

Lakoc commited on Dec 28, 2023

Commit

0f8c627

·

1 Parent(s): 7477aeb

Upload config

Files changed (1) hide show

config.json +1 -1

config.json CHANGED Viewed

@@ -17,7 +17,7 @@
   "model_type": "gpt2-multi-head",
   "n_embd": 512,
   "n_head": 8,
-  "n_inner": null,
   "n_layer": 16,
   "n_positions": 256,
   "reorder_and_upcast_attn": false,

   "model_type": "gpt2-multi-head",
   "n_embd": 512,
   "n_head": 8,
+  "n_inner": 2048,
   "n_layer": 16,
   "n_positions": 256,
   "reorder_and_upcast_attn": false,