quintic
/

llama3.1-sae-k-192-layer-23-29

Model card Files Files and versions Community

quintic commited on Aug 11

Commit

3e7ee84

•

1 Parent(s): 3337c78

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -4,6 +4,6 @@ license: mit
 Data: c4 and codeparrot, about 1:1 sample-wise but 1:4 token-wise mix. Significantly biased for codes (python, go, java, javascript, c, c++).
 Params:
-- batch size 64 * 2048 = 131072 tokens
 - lr automatically according to EAI sae codebase
 - auxk_alpha 0.03

 Data: c4 and codeparrot, about 1:1 sample-wise but 1:4 token-wise mix. Significantly biased for codes (python, go, java, javascript, c, c++).
 Params:
+- batch size 64 * 2048 * 8 = 1048576 tokens
 - lr automatically according to EAI sae codebase
 - auxk_alpha 0.03