OrionStarAI
/

Orion-MoE8x7B

Text Generation

Model card Files Files and versions Community

renillhuang commited on Nov 27, 2024

Commit

6c4fdf2

·

verified ·

1 Parent(s): 0b157c9

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -69,7 +69,7 @@ tags:
     |Embedding Tying    | False       |
     |Position embedding | RoPE        |
     |seq_len            | 8192        |
-    |Vocabulary Size    | 1136664     |
 - Training hyper-parameters
   - We use the AdamW optimizer with hyperparameters set to 𝛽1 = 0.9, 𝛽2 = 0.95, and a weight decay of 0.1.

     |Embedding Tying    | False       |
     |Position embedding | RoPE        |
     |seq_len            | 8192        |
+    |Vocabulary Size    | 113664      |
 - Training hyper-parameters
   - We use the AdamW optimizer with hyperparameters set to 𝛽1 = 0.9, 𝛽2 = 0.95, and a weight decay of 0.1.