Update README.md
Browse files
README.md
CHANGED
@@ -12,11 +12,17 @@ tags:
|
|
12 |
|
13 |
# MOSS Fix Autotune
|
14 |
|
15 |
-
This model repository serves as a temporary drop-in fix for an error that occurs during inference on int4 family of MOSS models, specifically, TypeError:
|
16 |
|
17 |
When loading the model with `model = AutoModelForCausalLM.from_pretrained("fnlp/moss-moon-003-sft-int4", trust_remote_code=True).half().cuda()`, the custom_autotune.py from the huggingface model files will be used. As such, it is necessary to update the file in the huggingface model repo.
|
18 |
|
19 |
-
Before the official maintainers update and fix the issue, I have applied the patch and re-uploaded the model files in this repo for convenience.
|
|
|
|
|
|
|
|
|
|
|
|
|
20 |
|
21 |
## Table of Contents
|
22 |
|
|
|
12 |
|
13 |
# MOSS Fix Autotune
|
14 |
|
15 |
+
This model repository serves as a temporary drop-in fix for an error that occurs during inference on int4 family of MOSS models, specifically, `TypeError: '<' not supported between instances of 'tuple' and 'float'` in [#159](https://github.com/OpenLMLab/MOSS/issues/159).
|
16 |
|
17 |
When loading the model with `model = AutoModelForCausalLM.from_pretrained("fnlp/moss-moon-003-sft-int4", trust_remote_code=True).half().cuda()`, the custom_autotune.py from the huggingface model files will be used. As such, it is necessary to update the file in the huggingface model repo.
|
18 |
|
19 |
+
Before the official maintainers update and fix the issue, I have applied the patch and re-uploaded the model files in this repo for convenience.
|
20 |
+
|
21 |
+
The patched model files can be used with:
|
22 |
+
|
23 |
+
```python
|
24 |
+
model = AutoModelForCausalLM.from_pretrained("DotIN13/moss-moon-003-sft-int4-fix-autotune", trust_remote_code=True).half().cuda()
|
25 |
+
```
|
26 |
|
27 |
## Table of Contents
|
28 |
|