Commit
•
6ef4fd5
1
Parent(s):
7441644
Update README.md (#1)
Browse files- Update README.md (ead4174fb7b5768c8d05c34ea66795da0989ffd9)
Co-authored-by: cheng <[email protected]>
README.md
CHANGED
|
@@ -18,7 +18,8 @@ More architecture details are shown below:
|
|
| 18 |
| ------------- |:-------------:|
|
| 19 |
| Num Attention Head |36 |
|
| 20 |
| Num Hidden Layer |36 |
|
| 21 |
-
| Hidden Size |2304 |
|
|
|
|
| 22 |
| Num MoE Layer per Block| 8|
|
| 23 |
| Num MoE Layer per Token| 2|
|
| 24 |
|Vocab Size|44 |
|
|
|
|
| 18 |
| ------------- |:-------------:|
|
| 19 |
| Num Attention Head |36 |
|
| 20 |
| Num Hidden Layer |36 |
|
| 21 |
+
| Hidden Size |2304 |
|
| 22 |
+
| FFN Hidden Size |7680 |
|
| 23 |
| Num MoE Layer per Block| 8|
|
| 24 |
| Num MoE Layer per Token| 2|
|
| 25 |
|Vocab Size|44 |
|