Create README.md

README.md

@@ -0,0 +1,168 @@
+---
+base_model: https://huggingface.co/beomi/llama-2-ko-70b
+inference: false
+language:
+- en
+- ko
+model_name: Llama 2 7B Chat
+model_type: llama
+pipeline_tag: text-generation
+quantized_by: kuotient
+tags:
+  - facebook
+  - meta
+  - pytorch
+  - llama
+  - llama-2
+  - kollama
+  - llama-2-ko
+  - gptq
+license: cc-by-nc-sa-4.0
+---
+# WIP
+## Llama-2-Ko-GPTQ
+<!-- README_GPTQ.md-provided-files start -->
+## Provided files and GPTQ parameters
+Multiple quantisation parameters are provided, to allow you to choose the best one for your hardware and requirements.
+Each separate quant is in a different branch.  See below for instructions on fetching from different branches.
+All recent GPTQ files are made with AutoGPTQ, and all files in non-main branches are made with AutoGPTQ. Files in the `main` branch which were uploaded before August 2023 were made with GPTQ-for-LLaMa.
+<details>
+  <summary>Explanation of GPTQ parameters</summary>
+- Bits: The bit size of the quantised model.
+- GS: GPTQ group size. Higher numbers use less VRAM, but have lower quantisation accuracy. "None" is the lowest possible value.
+- Act Order: True or False. Also known as `desc_act`. True results in better quantisation accuracy. Some GPTQ clients have had issues with models that use Act Order plus Group Size, but this is generally resolved now.
+- Damp %: A GPTQ parameter that affects how samples are processed for quantisation. 0.01 is default, but 0.1 results in slightly better accuracy.
+- GPTQ dataset: The dataset used for quantisation. Using a dataset more appropriate to the model's training can improve quantisation accuracy. Note that the GPTQ dataset is not the same as the dataset used to train the model - please refer to the original model repo for details of the training dataset(s).
+- Sequence Length: The length of the dataset sequences used for quantisation. Ideally this is the same as the model sequence length. For some very long sequence models (16+K), a lower sequence length may have to be used.  Note that a lower sequence length does not limit the sequence length of the quantised model. It only impacts the quantisation accuracy on longer inference sequences.
+- ExLlama Compatibility: Whether this file can be loaded with ExLlama, which currently only supports Llama models in 4-bit.
+
+</details>
+# Original model card: Llama 2 ko 70b
+> 🚧 Note: this repo is under construction 🚧
+
+# **Llama-2-Ko** 🦙🇰🇷
+
+Llama-2-Ko serves as an advanced iteration of Llama 2, benefiting from an expanded vocabulary and the inclusion of a Korean corpus in its further pretraining. Just like its predecessor, Llama-2-Ko operates within the broad range of generative text models that stretch from 7 billion to 70 billion parameters. This repository focuses on the **70B** pretrained version, which is tailored to fit the Hugging Face Transformers format. For access to the other models, feel free to consult the index provided below.
+
+## Model Details
+
+**Model Developers** Junbum Lee (Beomi)
+
+**Variations** Llama-2-Ko will come in a range of parameter sizes — 7B, 13B, and 70B — as well as pretrained and fine-tuned variations.
+
+**Input** Models input text only.
+
+**Output** Models generate text only.
+
+## Usage
+
+**Use with 8bit inference**
+
+- Requires > 74GB vram (compatible with 4x RTX 3090/4090 or 1x A100/H100 80G or 2x RTX 6000 ada/A6000 48G)
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
+model_8bit = AutoModelForCausalLM.from_pretrained(
+    "beomi/llama-2-ko-70b", 
+    load_in_8bit=True,
+    device_map="auto",
+)
+tk = AutoTokenizer.from_pretrained('beomi/llama-2-ko-70b')
+pipe = pipeline('text-generation', model=model_8bit, tokenizer=tk)
+def gen(x):
+    gended = pipe(f"### Title: {x}\n\n### Contents:",  # Since it this model is NOT finetuned with Instruction dataset, it is NOT optimal prompt.
+        max_new_tokens=300,
+        top_p=0.95,
+        do_sample=True,
+    )[0]['generated_text']
+    print(len(gended))
+    print(gended)
+```
+
+**Use with bf16 inference**
+
+- Requires > 150GB vram (compatible with 8x RTX 3090/4090 or 2x A100/H100 80G or 4x RTX 6000 ada/A6000 48G)
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
+model = AutoModelForCausalLM.from_pretrained(
+    "beomi/llama-2-ko-70b", 
+    device_map="auto",
+)
+tk = AutoTokenizer.from_pretrained('beomi/llama-2-ko-70b')
+pipe = pipeline('text-generation', model=model, tokenizer=tk)
+def gen(x):
+    gended = pipe(f"### Title: {x}\n\n### Contents:",  # Since it this model is NOT finetuned with Instruction dataset, it is NOT optimal prompt.
+        max_new_tokens=300,
+        top_p=0.95,
+        do_sample=True,
+    )[0]['generated_text']
+    print(len(gended))
+    print(gended)
+```
+
+**Model Architecture** 
+
+Llama-2-Ko is an auto-regressive language model that uses an optimized transformer architecture based on Llama-2.
+
+||Training Data|Params|Content Length|GQA|Tokens|LR|
+|---|---|---|---|---|---|---|
+|Llama-2-Ko 70B|*A new mix of Korean online data*|70B|4k|✅|>20B|1e<sup>-5</sup>|
+*Plan to train upto 300B tokens
+**Vocab Expansion**
+| Model Name | Vocabulary Size | Description | 
+| --- | --- | --- |
+| Original Llama-2 | 32000 | Sentencepiece BPE |
+| **Expanded Llama-2-Ko** | 46592 | Sentencepiece BPE. Added Korean vocab and merges |
+*Note: Llama-2-Ko 70B uses `46592` not `46336`(7B), will update new 7B model soon.
+
+**Tokenizing "안녕하세요, 오늘은 날씨가 좋네요. ㅎㅎ"**
+
+| Model | Tokens |
+| --- | --- |
+| Llama-2 | `['▁', '안', '<0xEB>', '<0x85>', '<0x95>', '하', '세', '요', ',', '▁', '오', '<0xEB>', '<0x8A>', '<0x98>', '은', '▁', '<0xEB>', '<0x82>', '<0xA0>', '씨', '가', '▁', '<0xEC>', '<0xA2>', '<0x8B>', '<0xEB>', '<0x84>', '<0xA4>', '요', '.', '▁', '<0xE3>', '<0x85>', '<0x8E>', '<0xE3>', '<0x85>', '<0x8E>']` |
+| Llama-2-Ko *70B | `['▁안녕', '하세요', ',', '▁오늘은', '▁날', '씨가', '▁좋네요', '.', '▁', 'ㅎ', 'ㅎ']` |
+**Tokenizing "Llama 2: Open Foundation and Fine-Tuned Chat Models"**
+| Model | Tokens |
+| --- | --- |
+| Llama-2 | `['▁L', 'l', 'ama', '▁', '2', ':', '▁Open', '▁Foundation', '▁and', '▁Fine', '-', 'T', 'un', 'ed', '▁Ch', 'at', '▁Mod', 'els']` |
+| Llama-2-Ko 70B | `['▁L', 'l', 'ama', '▁', '2', ':', '▁Open', '▁Foundation', '▁and', '▁Fine', '-', 'T', 'un', 'ed', '▁Ch', 'at', '▁Mod', 'els']` |
+# **Model Benchmark**
+## LM Eval Harness - Korean (polyglot branch)
+- Used EleutherAI's lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness/tree/polyglot
+### TBD
+## Note for oobabooga/text-generation-webui
+Remove `ValueError` at `load_tokenizer` function(line 109 or near), in `modules/models.py`.
+```python
+diff --git a/modules/models.py b/modules/models.py
+index 232d5fa..de5b7a0 100644
+--- a/modules/models.py
++++ b/modules/models.py
+@@ -106,7 +106,7 @@ def load_tokenizer(model_name, model):
+                 trust_remote_code=shared.args.trust_remote_code,
+                 use_fast=False
+             )
+-        except ValueError:
++        except:
+             tokenizer = AutoTokenizer.from_pretrained(
+                 path_to_model,
+                 trust_remote_code=shared.args.trust_remote_code,
+```
+Since Llama-2-Ko uses FastTokenizer provided by HF tokenizers NOT sentencepiece package, 
+it is required to use `use_fast=True` option when initialize tokenizer.
+Apple Sillicon does not support BF16 computing, use CPU instead. (BF16 is supported when using NVIDIA GPU)
+## LICENSE
+- Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International Public License, under LLAMA 2 COMMUNITY LICENSE AGREEMENT
+- Full License available at: [https://huggingface.co/beomi/llama-2-ko-70b/blob/main/LICENSE](https://huggingface.co/beomi/llama-2-ko-70b/blob/main/LICENSE)
+- For Commercial Usage, contact Author.
+## Citation
+```
+@misc {l._junbum_2023,
+	author       = { {L. Junbum} },
+	title        = { llama-2-ko-70b },
+	year         = 2023,
+	url          = { https://huggingface.co/beomi/llama-2-ko-70b },
+	doi          = { 10.57967/hf/1130 },
+	publisher    = { Hugging Face }
+}
+```