Update README.md

README.md

@@ -27,4 +27,24 @@ Imagine, if you will, a vast kingdom stretching beyond the horizon, where countl
 In this kingdom, information flows like a mighty river,...
 ```
 
+In 4-bit
+```
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+tokenizer = AutoTokenizer.from_pretrained("ontocord/phi-3-22b", trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained("ontocord/phi-3-22b", 
+    load_in_4bit=True, device_map="auto", trust_remote_code=True,  )
+with torch.no_grad():
+  print(tokenizer.batch_decode(model.generate(**tokenizer("<|user|>\nHow to explain Internet for a medieval knight?<|end|>\n<|assistant|>\n", return_tensors="pt").to('cuda'), max_new_tokens=128), use_cache=True)[0])
+
+```
+
+Will produce:
+```
+<|user|> How to explain Internet for a medieval knight?<|end|><|assistant|> Ah, noble knight, let me attempt to explain this mystical network known as the Internet, using terms and analogies from your time.
+
+Imagine a vast kingdom, stretching far beyond the horizon, where countless villages, towns, and cities are connected by roads, rivers, and paths. Each village is like a castle, filled with people who share knowledge, goods, stories, and news.
+
+Now, imagine that instead of messengers, horses, or ships, there exists a magical network of invisible threads connecting all these villages. This network is invisible to the eye, yet it allows messages, scroll
+```
 See the [Phi-3-medium-128k-instruct](https://huggingface.co/microsoft/Phi-3-medium-128k-instruct) model card for more details.