Duplicate from redrussianarmy/gpt2-turkish-cased

Co-authored-by: Hakan Boğan <[email protected]>

.gitattributes

@@ -0,0 +1,9 @@
+*.bin.* filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tar.gz filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,2 @@
+Pipfile
+Pipfile.lock

README.md

@@ -0,0 +1,63 @@
+---
+language: "tr"
+tags:
+- turkish
+- tr
+- gpt2-tr
+- gpt2-turkish
+---
+# 🇹🇷 Turkish GPT-2 Model
+
+In this repository I release GPT-2 model, that was trained on various texts for Turkish.
+
+The model is meant to be an entry point for fine-tuning on other texts.
+
+## Training corpora
+
+I used a Turkish corpora that is taken from oscar-corpus.
+
+It was possible to create byte-level BPE with Tokenizers library of Huggingface.
+
+With the Tokenizers library, I created a 52K byte-level BPE vocab based on the training corpora.
+
+After creating the vocab, I could train the GPT-2 for Turkish on two 2080TI over the complete training corpus (five epochs).
+
+Logs during training:
+https://tensorboard.dev/experiment/3AWKv8bBTaqcqZP5frtGkw/#scalars
+
+## Model weights
+
+Both PyTorch and Tensorflow compatible weights are available.
+
+| Model                             | Downloads
+| --------------------------------- | ---------------------------------------------------------------------------------------------------------------
+| `redrussianarmy/gpt2-turkish-cased`   | [`config.json`](https://huggingface.co/redrussianarmy/gpt2-turkish-cased/resolve/main/config.json) • [`merges.txt`](https://huggingface.co/redrussianarmy/gpt2-turkish-cased/resolve/main/merges.txt) • [`pytorch_model.bin`](https://huggingface.co/redrussianarmy/gpt2-turkish-cased/resolve/main/pytorch_model.bin) • [`special_tokens_map.json`](https://huggingface.co/redrussianarmy/gpt2-turkish-cased/resolve/main/special_tokens_map.json) • [`tf_model.h5`](https://huggingface.co/redrussianarmy/gpt2-turkish-cased/resolve/main/tf_model.h5) • [`tokenizer_config.json`](https://huggingface.co/redrussianarmy/gpt2-turkish-cased/resolve/main/tokenizer_config.json) • [`traning_args.bin`](https://huggingface.co/redrussianarmy/gpt2-turkish-cased/resolve/main/training_args.bin) • [`vocab.json`](https://huggingface.co/redrussianarmy/gpt2-turkish-cased/resolve/main/vocab.json)
+
+## Using the model
+
+The model itself can be used in this way:
+
+``` python
+from transformers import AutoTokenizer, AutoModelWithLMHead
+tokenizer = AutoTokenizer.from_pretrained("redrussianarmy/gpt2-turkish-cased")
+model = AutoModelWithLMHead.from_pretrained("redrussianarmy/gpt2-turkish-cased")
+```
+
+Here's an example that shows how to use the great Transformers Pipelines for generating text:
+
+``` python
+from transformers import pipeline
+pipe = pipeline('text-generation', model="redrussianarmy/gpt2-turkish-cased",
+                 tokenizer="redrussianarmy/gpt2-turkish-cased", config={'max_length':800})   
+text = pipe("Akşamüstü yolda ilerlerken, ")[0]["generated_text"]
+print(text)
+```
+
+### How to clone the model repo?
+```
+git lfs install
+git clone https://huggingface.co/redrussianarmy/gpt2-turkish-cased
+```
+
+## Contact (Bugs, Feedback, Contribution and more)
+For questions about the GPT2-Turkish model, just open an issue [here](https://github.com/redrussianarmy/gpt2-turkish/issues) 🤗

config.json

@@ -0,0 +1,33 @@
+{
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "gradient_checkpointing": false,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "resid_pdrop": 0.1,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "vocab_size": 50257
+}

flax_model.msgpack

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fe376584de6c19ca333b41c2cf0fdd688737edb81dfe42f57a66db2232aa957b
+size 497764120

metadata.json

@@ -0,0 +1,11 @@
+{
+    "tags":[
+        "tr",
+        "turkish",
+        "gpt2-tr",
+        "gpt2-turkish"
+    ],
+    "languages":[
+        "tr"
+    ]
+}

pytorch_model.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c11264d48e8078c36f9decc41eff62fd905111237e2cd7d60651d9f94bb946f
+size 510406550

special_tokens_map.json

@@ -0,0 +1 @@
+{"bos_token": {"content": "<|endoftext|>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "eos_token": {"content": "<|endoftext|>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}, "unk_token": {"content": "<|endoftext|>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true}}

tf_model.h5

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a4723564f63bbf7b892a5f305a7da2f615d61664796d8429bb82938eeea441f
+size 497933648

tokenizer_config.json

@@ -0,0 +1 @@
+{"errors": "replace", "unk_token": {"content": "<|endoftext|>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "bos_token": {"content": "<|endoftext|>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "eos_token": {"content": "<|endoftext|>", "single_word": false, "lstrip": false, "rstrip": false, "normalized": true, "__type": "AddedToken"}, "add_prefix_space": false, "model_max_length": 1024, "special_tokens_map_file": null, "tokenizer_file": null, "name_or_path": "tokenized_data/"}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f98dfdf111b52ba132eacb3565fd61dc1529f62b170c6ff6ed7fe5fee33eb501
+size 1775