Update README.md
Browse files
README.md
CHANGED
|
@@ -33,4 +33,25 @@ Final checkpoint: RWKV-4-Pile-1B5-20220903-8040.pth : Trained on the Pile for 33
|
|
| 33 |
* SC2016 acc 68.73%
|
| 34 |
* Hellaswag acc_norm 52.48%
|
| 35 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 36 |
## Note: 4 / 4a / 4b models ARE NOT compatible. Use RWKV-4 unless you know what you are doing.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 33 |
* SC2016 acc 68.73%
|
| 34 |
* Hellaswag acc_norm 52.48%
|
| 35 |
|
| 36 |
+
### Instruct-test models: only useful if you construct your prompt following dataset templates
|
| 37 |
+
|
| 38 |
+
RWKV-4-Pile-1B5-Instruct-test1-20230124.pth
|
| 39 |
+
instruct-tuned on https://huggingface.co/datasets/bigscience/xP3all/viewer/en/train
|
| 40 |
+
|
| 41 |
+
RWKV-4-Pile-1B5-Instruct-test2-20230209.pth
|
| 42 |
+
instruct-tuned on https://huggingface.co/datasets/Muennighoff/flan & NIv2
|
| 43 |
+
|
| 44 |
+
### Chinese models
|
| 45 |
+
|
| 46 |
+
RWKV-4-Pile-1B5-EngChn-testNovel-xxx for writing Chinese novels (trained on 200G Chinese novels.)
|
| 47 |
+
|
| 48 |
+
RWKV-4-Pile-1B5-EngChn-testxxx for Chinese Q&A (trained on 10G Chinese text. only for testing purposes.)
|
| 49 |
+
|
| 50 |
## Note: 4 / 4a / 4b models ARE NOT compatible. Use RWKV-4 unless you know what you are doing.
|
| 51 |
+
|
| 52 |
+
RWKV-4b-Pile-1B5-20230217-7954.pth (--my_testing 'a') with tiny amt of QKV attention to improve performance
|
| 53 |
+
* Pile loss 1.9947
|
| 54 |
+
* LAMBADA ppl 5.82, acc 62.35%
|
| 55 |
+
* PIQA acc 72.52%
|
| 56 |
+
* SC2016 acc 68.89%
|
| 57 |
+
* Hellaswag acc_norm 54.32%
|