Update README.md

README.md

@@ -13,6 +13,63 @@ library_name: transformers
 pipeline_tag: translation
 ---
 
+## How to use
+Just install `peft`, `transformers` and `pytorch` first.
+
+```bash
+pip install peft transformers torch
+```
+
+Then login with your huggingface token to get access to base models
+```bash
+huggingface-cli login --token <YOUR_HF_TOKEN>
+```
+
+Then load the model.
+```python
+from peft import PeftConfig, PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+peft_model_id = "ahmedheakl/arazn-gemma1.1-7B-eng-extra"
+peft_config = PeftConfig.from_pretrained(peft_model_id)
+base_model_name = peft_config.base_model_name_or_path
+base_model = AutoModelForCausalLM.from_pretrained(base_model_name)
+model = PeftModel.from_pretrained(base_model, peft_model_id)
+model = model.to("cuda")
+tokenizer = AutoTokenizer.from_pretrained(peft_model_id)
+```
+
+Then do inference
+```python
+import torch
+
+raw_prompt = """<bos><start_of_turn>user
+Translate the following code-switched Arabic-English-mixed text to English only.
+{source}<end_of_turn>
+<start_of_turn>model
+"""
+def inference(prompt) -> str:
+    prompt = raw_prompt.format(source=prompt)
+    inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+    generated_ids = model.generate(
+        **inputs,
+        use_cache=True,
+        num_return_sequences=1,
+        max_new_tokens=100,
+        do_sample=True,
+        num_beams=1,
+        temperature=0.7,
+        eos_token_id=tokenizer.eos_token_id,
+        pad_token_id=tokenizer.pad_token_id,
+    )
+    outputs = tokenizer.batch_decode(generated_ids)[0]
+    torch.cuda.empty_cache()
+    torch.cuda.synchronize()
+    return outputs.split("<start_of_turn>model\n")[-1].split("<end_of_turn>")[0]
+
+print(inference("أنا أحب الbanana")) # I like bananas.
+```
+
 **Please see paper & code for more information:**
 - https://github.com/ahmedheakl/arazn-llm
 - https://arxiv.org/abs/2406.18120