Update README.md
Browse files
README.md
CHANGED
|
@@ -38,8 +38,7 @@ Hello Orca Mini, what can you do for me?<|eot_id|>
|
|
| 38 |
<|start_header_id|>assistant<|end_header_id|>
|
| 39 |
```
|
| 40 |
|
| 41 |
-
Below shows a code example on how to use this model in default half precision (bfloat16)
|
| 42 |
-
|
| 43 |
```python
|
| 44 |
import torch
|
| 45 |
from transformers import pipeline
|
|
@@ -58,7 +57,7 @@ outputs = pipeline(messages, max_new_tokens=128, do_sample=True, temperature=0.0
|
|
| 58 |
print(outputs[0]["generated_text"][-1])
|
| 59 |
```
|
| 60 |
|
| 61 |
-
Below shows a code example on how to use this model in 4-bit format via bitsandbytes library
|
| 62 |
|
| 63 |
```python
|
| 64 |
import torch
|
|
@@ -86,7 +85,7 @@ print(outputs[0]["generated_text"][-1])
|
|
| 86 |
|
| 87 |
```
|
| 88 |
|
| 89 |
-
Below shows a code example on how to use this model in 8-bit format via bitsandbytes library
|
| 90 |
|
| 91 |
```python
|
| 92 |
import torch
|
|
|
|
| 38 |
<|start_header_id|>assistant<|end_header_id|>
|
| 39 |
```
|
| 40 |
|
| 41 |
+
Below shows a code example on how to use this model in default half precision (bfloat16), it requires around ~133GB VRAM
|
|
|
|
| 42 |
```python
|
| 43 |
import torch
|
| 44 |
from transformers import pipeline
|
|
|
|
| 57 |
print(outputs[0]["generated_text"][-1])
|
| 58 |
```
|
| 59 |
|
| 60 |
+
Below shows a code example on how to use this model in 4-bit format via bitsandbytes library, it requires around ~39GB VRAM
|
| 61 |
|
| 62 |
```python
|
| 63 |
import torch
|
|
|
|
| 85 |
|
| 86 |
```
|
| 87 |
|
| 88 |
+
Below shows a code example on how to use this model in 8-bit format via bitsandbytes library, it requires around ~69GB VRAM
|
| 89 |
|
| 90 |
```python
|
| 91 |
import torch
|