EricB HF staff commited on
Commit
9f94be4
ยท
verified ยท
1 Parent(s): 6845c7b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +11 -9
README.md CHANGED
@@ -17,13 +17,15 @@ Run with [mistral.rs](https://github.com/EricLBuehler/mistral.rs). Documentation
17
  2) **Reliable** ๐Ÿ”’: Compatibility ensured with *embedded* and *checked* semantic versioning information from day 1.
18
  3) **Easy** ๐Ÿค—: Download UQFF models *easily* and *quickly* from Hugging Face, or use a local file.
19
  3) **Customizable** ๐Ÿ› ๏ธ: Make and publish your own UQFF files in minutes.
20
- ## Files
21
 
22
- |Name|Quantization type(s)|Example|
23
- |--|--|--|
24
- |phi3.5-mini-instruct-hqq4.uqff|HQQ4|`./mistralrs-server -i plain -m microsoft/Phi-3.5-mini-instruct --from-uqff EricB/Phi-3.5-mini-instruct-UQFF/phi3.5-mini-instruct-hqq4.uqff`|
25
- |phi3.5-mini-instruct-hqq8.uqff|HQQ8|`./mistralrs-server -i plain -m microsoft/Phi-3.5-mini-instruct --from-uqff EricB/Phi-3.5-mini-instruct-UQFF/phi3.5-mini-instruct-hqq8.uqff`|
26
- |phi3.5-mini-instruct-q4k.uqff|Q4K|`./mistralrs-server -i plain -m microsoft/Phi-3.5-mini-instruct --from-uqff EricB/Phi-3.5-mini-instruct-UQFF/phi3.5-mini-instruct-q4k.uqff`|
27
- |phi3.5-mini-instruct-q5k.uqff|Q5K|`./mistralrs-server -i plain -m microsoft/Phi-3.5-mini-instruct --from-uqff EricB/Phi-3.5-mini-instruct-UQFF/phi3.5-mini-instruct-q5k.uqff`|
28
- |phi3.5-mini-instruct-q6k.uqff|Q6K|`./mistralrs-server -i plain -m microsoft/Phi-3.5-mini-instruct --from-uqff EricB/Phi-3.5-mini-instruct-UQFF/phi3.5-mini-instruct-q6k.uqff`|
29
- |phi3.5-mini-instruct-q8_0.uqff|Q8_0|`./mistralrs-server -i plain -m microsoft/Phi-3.5-mini-instruct --from-uqff EricB/Phi-3.5-mini-instruct-UQFF/phi3.5-mini-instruct-q8_0.uqff`|
 
 
 
 
17
  2) **Reliable** ๐Ÿ”’: Compatibility ensured with *embedded* and *checked* semantic versioning information from day 1.
18
  3) **Easy** ๐Ÿค—: Download UQFF models *easily* and *quickly* from Hugging Face, or use a local file.
19
  3) **Customizable** ๐Ÿ› ๏ธ: Make and publish your own UQFF files in minutes.
 
20
 
21
+ ## Examples
22
+
23
+ |Quantization type(s)|Example|
24
+ |--|--|
25
+ |FP8|`./mistralrs-server -i vision-plain -m EricB/Phi-3.5-mini-instruct-UQFF -a phi3v --from-uqff phi3.5-mini-instruct-f8e4m3.uqff`|
26
+ |HQQ4|`./mistralrs-server -i vision-plain -m EricB/Phi-3.5-mini-instruct-UQFF -a phi3v --from-uqff phi3.5-mini-instruct-hqq4.uqff`|
27
+ |HQQ8|`./mistralrs-server -i vision-plain -m EricB/Phi-3.5-mini-instruct-UQFF -a phi3v --from-uqff phi3.5-mini-instruct-hqq8.uqff`|
28
+ |Q3K|`./mistralrs-server -i vision-plain -m EricB/Phi-3.5-mini-instruct-UQFF -a phi3v --from-uqff phi3.5-mini-instruct-q3k.uqff`|
29
+ |Q4K|`./mistralrs-server -i vision-plain -m EricB/Phi-3.5-mini-instruct-UQFF -a phi3v --from-uqff phi3.5-mini-instruct-q4k.uqff`|
30
+ |Q5K|`./mistralrs-server -i vision-plain -m EricB/Phi-3.5-mini-instruct-UQFF -a phi3v --from-uqff phi3.5-mini-instruct-q5k.uqff`|
31
+ |Q8_0|`./mistralrs-server -i vision-plain -m EricB/Phi-3.5-mini-instruct-UQFF -a phi3v --from-uqff phi3.5-mini-instruct-q8_0.uqff`|