RichardErkhov commited on
Commit
439a7bd
·
verified ·
1 Parent(s): 2ae3b7c

uploaded readme

Browse files
Files changed (1) hide show
  1. README.md +145 -0
README.md ADDED
@@ -0,0 +1,145 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Quantization made by Richard Erkhov.
2
+
3
+ [Github](https://github.com/RichardErkhov)
4
+
5
+ [Discord](https://discord.gg/pvy7H8DZMG)
6
+
7
+ [Request more models](https://github.com/RichardErkhov/quant_request)
8
+
9
+
10
+ llama-3-youko-8b - bnb 8bits
11
+ - Model creator: https://huggingface.co/rinna/
12
+ - Original model: https://huggingface.co/rinna/llama-3-youko-8b/
13
+
14
+
15
+
16
+
17
+ Original model description:
18
+ ---
19
+ thumbnail: https://github.com/rinnakk/japanese-pretrained-models/blob/master/rinna.png
20
+ license: llama3
21
+ datasets:
22
+ - mc4
23
+ - wikipedia
24
+ - EleutherAI/pile
25
+ - oscar-corpus/colossal-oscar-1.0
26
+ - cc100
27
+ language:
28
+ - ja
29
+ - en
30
+ inference: false
31
+ ---
32
+
33
+ # `Llama 3 Youko 8B (rinna/llama-3-youko-8b)`
34
+
35
+ ![rinna-icon](./rinna.png)
36
+
37
+ # Overview
38
+
39
+ We conduct continual pre-training of [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) on **22B** tokens from a mixture of Japanese and English datasets. The continual pre-training significantly improves the model's performance on Japanese tasks.
40
+
41
+ The name `youko` comes from the Japanese word [`妖狐/ようこ/Youko`](https://ja.wikipedia.org/wiki/%E5%A6%96%E7%8B%90), which is a kind of Japanese mythical creature ([`妖怪/ようかい/Youkai`](https://ja.wikipedia.org/wiki/%E5%A6%96%E6%80%AA)).
42
+
43
+
44
+ * **Library**
45
+
46
+ The model was trained using code based on [EleutherAI/gpt-neox](https://github.com/EleutherAI/gpt-neox).
47
+
48
+ * **Model architecture**
49
+
50
+ A 32-layer, 4096-hidden-size transformer-based language model. Refer to the [Llama 3 Model Card](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md) for architecture details.
51
+
52
+ * **Training: Built with Meta Llama 3**
53
+
54
+ The model was initialized with the [meta-llama/Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) model and continually trained on around **22B** tokens from a mixture of the following corpora
55
+ - [Japanese CC-100](https://huggingface.co/datasets/cc100)
56
+ - [Japanese C4](https://huggingface.co/datasets/mc4)
57
+ - [Japanese OSCAR](https://huggingface.co/datasets/oscar-corpus/colossal-oscar-1.0)
58
+ - [The Pile](https://huggingface.co/datasets/EleutherAI/pile)
59
+ - [Wikipedia](https://dumps.wikimedia.org/other/cirrussearch)
60
+ - rinna curated Japanese dataset
61
+
62
+ * **Contributors**
63
+
64
+ - [Koh Mitsuda](https://huggingface.co/mitsu-koh)
65
+ - [Kei Sawada](https://huggingface.co/keisawada)
66
+
67
+ ---
68
+
69
+ # Benchmarking
70
+
71
+ Please refer to [rinna's LM benchmark page](https://rinnakk.github.io/research/benchmarks/lm/index.html).
72
+
73
+ ---
74
+
75
+ # How to use the model
76
+
77
+ ~~~~python
78
+ import transformers
79
+ import torch
80
+
81
+ model_id = "rinna/llama-3-youko-8b"
82
+ pipeline = transformers.pipeline(
83
+ "text-generation",
84
+ model=model_id,
85
+ model_kwargs={"torch_dtype": torch.bfloat16},
86
+ device_map="auto"
87
+ )
88
+ output = pipeline(
89
+ "西田幾多郎は、",
90
+ max_new_tokens=256,
91
+ do_sample=True
92
+ )
93
+ print(output)
94
+ ~~~~
95
+
96
+ ---
97
+
98
+ # Tokenization
99
+ The model uses the original meta-llama/Meta-Llama-3-8B tokenizer.
100
+
101
+ ---
102
+
103
+ # How to cite
104
+ ```bibtex
105
+ @misc{rinna-llama-3-youko-8b,
106
+ title = {rinna/llama-3-youko-8b},
107
+ author = {Mitsuda, Koh and Sawada, Kei},
108
+ url = {https://huggingface.co/rinna/llama-3-youko-8b},
109
+ }
110
+
111
+ @inproceedings{sawada2024release,
112
+ title = {Release of Pre-Trained Models for the {J}apanese Language},
113
+ author = {Sawada, Kei and Zhao, Tianyu and Shing, Makoto and Mitsui, Kentaro and Kaga, Akio and Hono, Yukiya and Wakatsuki, Toshiaki and Mitsuda, Koh},
114
+ booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
115
+ month = {5},
116
+ year = {2024},
117
+ url = {https://arxiv.org/abs/2404.01657},
118
+ }
119
+ ```
120
+ ---
121
+
122
+ # References
123
+ ```bibtex
124
+ @article{llama3modelcard,
125
+ title={Llama 3 Model Card},
126
+ author={AI@Meta},
127
+ year={2024},
128
+ url = {https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md}
129
+ }
130
+
131
+ @software{gpt-neox-library,
132
+ title = {{GPT-NeoX: Large Scale Autoregressive Language Modeling in PyTorch}},
133
+ author = {Andonian, Alex and Anthony, Quentin and Biderman, Stella and Black, Sid and Gali, Preetham and Gao, Leo and Hallahan, Eric and Levy-Kramer, Josh and Leahy, Connor and Nestler, Lucas and Parker, Kip and Pieler, Michael and Purohit, Shivanshu and Songz, Tri and Phil, Wang and Weinbach, Samuel},
134
+ doi = {10.5281/zenodo.5879544},
135
+ month = {8},
136
+ year = {2021},
137
+ version = {0.0.1},
138
+ url = {https://www.github.com/eleutherai/gpt-neox},
139
+ }
140
+ ```
141
+ ---
142
+
143
+ # License
144
+ [Meta Llama 3 Community License](https://llama.meta.com/llama3/license/)
145
+