|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
# TinyLlama + Japanese |
|
|
|
A continual pretraining model of TinyLlama 1.1B with a few Japanese texts. |
|
|
|
|
|
### Base Model |
|
|
|
[TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T](https://huggingface.co/TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T) |
|
|
|
### Tokenizers |
|
|
|
(elyza/ELYZA-japanese-Llama-2-7b)[https://huggingface.co/elyza/ELYZA-japanese-Llama-2-7b] |
|
|
|
|
|
### Training Dataset |
|
|
|
Around 9B tokens in total. |
|
|
|
- izumi-lab/wikipedia-ja-20230720 |
|
- if001/oscar_2023_filtered |
|
|
|
### Validation Dataset |
|
|
|
- izumi-lab/wikinews-ja-20230728 |
|
- izumi-lab/wikinews-en-20230728 |
|
- if001/aozorabunko-clean-sin |
|
|
|
|
|
### Evaluation |
|
|
|
We did not perform. |
|
|
|
|
|
### Acknowledgement |
|
|
|
We acknowledge those who prepared valuable datasets and [lit-gpt](https://github.com/Lightning-AI/lit-gpt). |
|
|
|
|
|
|
|
|
