feat(gemma2): add gemma2 tokenizer
Browse files- gemma2/.gitattributes +1 -0
- gemma2/README.md +5 -0
- gemma2/tokenizer.json +3 -0
gemma2/.gitattributes
ADDED
|
@@ -0,0 +1 @@
|
|
|
|
|
|
|
| 1 |
+
tokenizer.json filter=lfs diff=lfs merge=lfs -text
|
gemma2/README.md
ADDED
|
@@ -0,0 +1,5 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
The `tokenizer.json` here is extracted from [google/gemma-2-2b-it](https://huggingface.co/google/gemma-2-2b-it/tree/main).
|
| 2 |
+
|
| 3 |
+
## LICENSE
|
| 4 |
+
|
| 5 |
+
gemma2
|
gemma2/tokenizer.json
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3f289bc05132635a8bc7aca7aa21255efd5e18f3710f43e3cdb96bcd41be4922
|
| 3 |
+
size 17525357
|