TheBloke/Marx-3b-GPTQ

Chat & support: TheBloke's Discord server

Want to contribute? TheBloke's Patreon page

TheBloke's LLM work is generously supported by a grant from andreessen horowitz (a16z)

---

Marx 3B - GPTQ

• Model creator: Bohan Du
• Original model: Marx 3B

Description

This repo contains GPTQ model files for Bohan Du's Marx 3B.

Multiple GPTQ parameter permutations are provided; see Provided Files below for details of the options provided, their parameters, and the software used to create them.

Repositories available

• GPTQ models for GPU inference, with multiple quantisation parameter options.
• 2, 3, 4, 5, 6 and 8-bit GGML models for CPU+GPU inference
• Bohan Du's original unquantised fp16 model in pytorch format, for GPU inference and for further conversions

Prompt template: Human-Response

### HUMAN:
{prompt}

### RESPONSE:

Provided files and GPTQ parameters

Multiple quantisation parameters are provided, to allow you to choose the best one for your hardware and requirements.

Each separate quant is in a different branch. See below for instructions on fetching from different branches.

All GPTQ files are made with AutoGPTQ.

Explanation of GPTQ parameters

• Bits: The bit size of the quantised model.
• GS: GPTQ group size. Higher numbers use less VRAM, but have lower quantisation accuracy. "None" is the lowest possible value.
• Act Order: True or False. Also known as desc_act. True results in better quantisation accuracy. Some GPTQ clients have issues with models that use Act Order plus Group Size.
• Damp %: A GPTQ parameter that affects how samples are processed for quantisation. 0.01 is default, but 0.1 results in slightly better accuracy.
• GPTQ dataset: The dataset used for quantisation. Using a dataset more appropriate to the model's training can improve quantisation accuracy. Note that the GPTQ dataset is not the same as the dataset used to train the model - please refer to the original model repo for details of the training dataset(s).
• Sequence Length: The length of the dataset sequences used for quantisation. Ideally this is the same as the model sequence length. For some very long sequence models (16+K), a lower sequence length may have to be used. Note that a lower sequence length does not limit the sequence length of the quantised model. It only impacts the quantisation accuracy on longer inference sequences.
• ExLlama Compatibility: Whether this file can be loaded with ExLlama, which currently only supports Llama models in 4-bit.

Branch	Bits	GS	Act Order	Damp %	GPTQ Dataset	Seq Len	Size	ExLlama	Desc
main	4	128	No	0.1	wikitext	2048	2.09 GB	Yes	Most compatible option. Good inference speed in AutoGPTQ and GPTQ-for-LLaMa. Lower inference quality than other options.
gptq-4bit-32g-actorder_True	4	32	Yes	0.1	wikitext	2048	2.28 GB	Yes	4-bit, with Act Order and group size 32g. Gives highest possible inference quality, with maximum VRAM usage. Poor AutoGPTQ CUDA speed.
gptq-4bit-64g-actorder_True	4	64	Yes	0.1	wikitext	2048	2.15 GB	Yes	4-bit, with Act Order and group size 64g. Uses less VRAM than 32g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed.
gptq-4bit-128g-actorder_True	4	128	Yes	0.1	wikitext	2048	2.09 GB	Yes	4-bit, with Act Order and group size 128g. Uses even less VRAM than 64g, but with slightly lower accuracy. Poor AutoGPTQ CUDA speed.
gptq-8bit--1g-actorder_True	8	None	Yes	0.1	wikitext	2048	3.64 GB	No	8-bit, with Act Order. No group size, to lower VRAM requirements and to improve AutoGPTQ speed.
gptq-8bit-128g-actorder_True	8	128	Yes	0.1	wikitext	2048	3.71 GB	No	8-bit, with group size 128g for higher inference quality and with Act Order for even higher accuracy. Poor AutoGPTQ CUDA speed.

How to download from branches

• In text-generation-webui, you can add :branch to the end of the download name, eg TheBloke/Marx-3b-GPTQ:gptq-4bit-32g-actorder_True
• With Git, you can clone a branch with:

git clone --single-branch --branch gptq-4bit-32g-actorder_True https://huggingface.co/TheBloke/Marx-3b-GPTQ

• In Python Transformers code, the branch is the revision parameter; see below.

How to easily download and use this model in text-generation-webui.

Please make sure you're using the latest version of text-generation-webui.

It is strongly recommended to use the text-generation-webui one-click-installers unless you know how to make a manual install.

1. Click the Model tab.
2. Under Download custom model or LoRA, enter TheBloke/Marx-3b-GPTQ.

• To download from a specific branch, enter for example TheBloke/Marx-3b-GPTQ:gptq-4bit-32g-actorder_True
• see Provided Files above for the list of branches for each option.

3. Click Download.
4. The model will start downloading. Once it's finished it will say "Done"
5. In the top left, click the refresh icon next to Model.
6. In the Model dropdown, choose the model you just downloaded: Marx-3b-GPTQ
7. The model will automatically load, and is now ready for use!
8. If you want any custom settings, set them and then click Save settings for this model followed by Reload the Model in the top right.

• Note that you do not need to set GPTQ parameters any more. These are set automatically from the file quantize_config.json.

9. Once you're ready, click the Text Generation tab and enter a prompt to get started!

How to use this GPTQ model from Python code

First make sure you have AutoGPTQ 0.3.1 or later installed:

pip3 install auto-gptq

If you have problems installing AutoGPTQ, please build from source instead:

pip3 uninstall -y auto-gptq
git clone https://github.com/PanQiWei/AutoGPTQ
cd AutoGPTQ
pip3 install .

Then try the following example code:

from transformers import AutoTokenizer, pipeline, logging
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig

model_name_or_path = "TheBloke/Marx-3b-GPTQ"

use_triton = False

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)

model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
        use_safetensors=True,
        trust_remote_code=False,
        device="cuda:0",
        use_triton=use_triton,
        quantize_config=None)

"""
# To download from a specific branch, use the revision parameter, as in this example:
# Note that `revision` requires AutoGPTQ 0.3.1 or later!

model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,
        revision="gptq-4bit-32g-actorder_True",
        use_safetensors=True,
        trust_remote_code=False,
        device="cuda:0",
        quantize_config=None)
"""

prompt = "Tell me about AI"
prompt_template=f'''### HUMAN:
{prompt}

### RESPONSE:
'''

print("\n\n*** Generate:")

input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()
output = model.generate(inputs=input_ids, temperature=0.7, max_new_tokens=512)
print(tokenizer.decode(output[0]))

# Inference can also be done using transformers' pipeline

# Prevent printing spurious transformers error when using pipeline with AutoGPTQ
logging.set_verbosity(logging.CRITICAL)

print("*** Pipeline:")
pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    max_new_tokens=512,
    temperature=0.7,
    top_p=0.95,
    repetition_penalty=1.15
)

print(pipe(prompt_template)[0]['generated_text'])

Compatibility

The files provided will work with AutoGPTQ (CUDA and Triton modes), GPTQ-for-LLaMa (only CUDA has been tested), and Occ4m's GPTQ-for-LLaMa fork.

ExLlama works with Llama models in 4-bit. Please see the Provided Files table above for per-file compatibility.

Discord

For further support, and discussions on these models and AI in general, join us at:

TheBloke AI's Discord server

Thanks, and how to contribute.

Thanks to the chirper.ai team!

I've had a lot of people ask if they can contribute. I enjoy providing models and helping people, and would love to be able to spend even more time doing it, as well as expanding into new projects like fine tuning/training.

If you're able and willing to contribute it will be most gratefully received and will help me to keep providing more models, and to start work on new AI projects.

Donaters will get priority support on any and all AI/LLM/model questions and requests, access to a private Discord room, plus other benefits.

• Patreon: https://patreon.com/TheBlokeAI
• Ko-Fi: https://ko-fi.com/TheBlokeAI

Special thanks to: Aemon Algiz.

Patreon special mentions: Sam, theTransient, Jonathan Leane, Steven Wood, webtim, Johann-Peter Hartmann, Geoffrey Montalvo, Gabriel Tamborski, Willem Michiel, John Villwock, Derek Yates, Mesiah Bishop, Eugene Pentland, Pieter, Chadd, Stephen Murray, Daniel P. Andersen, terasurfer, Brandon Frisco, Thomas Belote, Sid, Nathan LeClaire, Magnesian, Alps Aficionado, Stanislav Ovsiannikov, Alex, Joseph William Delisle, Nikolai Manek, Michael Davis, Junyu Yang, K, J, Spencer Kim, Stefan Sabev, Olusegun Samson, transmissions 11, Michael Levine, Cory Kujawski, Rainer Wilmers, zynix, Kalila, Luke @flexchar, Ajan Kanaga, Mandus, vamX, Ai Maven, Mano Prime, Matthew Berman, subjectnull, Vitor Caleffi, Clay Pascal, biorpg, alfie_i, 阿明, Jeffrey Morgan, ya boyyy, Raymond Fosdick, knownsqashed, Olakabola, Leonard Tan, ReadyPlayerEmma, Enrico Ros, Dave, Talal Aujan, Illia Dulskyi, Sean Connelly, senxiiz, Artur Olbinski, Elle, Raven Klaugh, Fen Risland, Deep Realms, Imad Khwaja, Fred von Graf, Will Dee, usrbinkat, SuperWojo, Alexandros Triantafyllidis, Swaroop Kallakuri, Dan Guido, John Detwiler, Pedro Madruga, Iucharbius, Viktor Bowallius, Asp the Wyvern, Edmond Seymore, Trenton Dambrowitz, Space Cruiser, Spiking Neurons AB, Pyrater, LangChain4j, Tony Hughes, Kacper Wikieł, Rishabh Srivastava, David Ziegler, Luke Pendergrass, Andrey, Gabriel Puliatti, Lone Striker, Sebastain Graf, Pierre Kircher, Randy H, NimbleBox.ai, Vadim, danny, Deo Leter

Thank you to all my generous patrons and donaters!

And thank you again to a16z for their generous grant.

Original model card: Bohan Du's Marx 3B

This is OpenLLaMA 3B V2 finetuned on EverythingLM Data(ShareGPT format more cleaned) for 1 epochs.

Prompt template:

### HUMAN:
{prompt}

### RESPONSE:
<leave a newline for the model to answer>

q4_1 GGML quant here.
All GGML quants available here.

Note: Don't expect this model to be good, I was just starting out to finetune. So don't roast me please!

Benchmarks:

arc_challenge	acc 0.38993174061433444 acc_stderr 0.014252959848892884 acc_norm 0.4308873720136519 acc_norm_stderr 0.014471133392642475
hellaswag	acc 0.5513841864170484 acc_stderr 0.004963362085275556 acc_norm 0.7257518422624976 acc_norm_stderr 0.00445222854104355
hendrycksTest-abstract_algebra	acc 0.23 acc_stderr 0.04229525846816506 acc_norm 0.23 acc_norm_stderr 0.04229525846816506
hendrycksTest-anatomy	acc 0.2962962962962963 acc_stderr 0.03944624162501116 acc_norm 0.2962962962962963 acc_norm_stderr 0.03944624162501116
hendrycksTest-astronomy	acc 0.32894736842105265 acc_stderr 0.03823428969926603 acc_norm 0.32894736842105265 acc_norm_stderr 0.03823428969926603
hendrycksTest-business_ethics	acc 0.3 acc_stderr 0.046056618647183814 acc_norm 0.3 acc_norm_stderr 0.046056618647183814
hendrycksTest-clinical_knowledge	acc 0.2641509433962264 acc_stderr 0.027134291628741713 acc_norm 0.2641509433962264 acc_norm_stderr 0.027134291628741713
hendrycksTest-college_biology	acc 0.2569444444444444 acc_stderr 0.03653946969442099 acc_norm 0.2569444444444444 acc_norm_stderr 0.03653946969442099
hendrycksTest-college_chemistry	acc 0.22 acc_stderr 0.041633319989322695 acc_norm 0.22 acc_norm_stderr 0.041633319989322695
hendrycksTest-college_computer_science	acc 0.26 acc_stderr 0.0440844002276808 acc_norm 0.26 acc_norm_stderr 0.0440844002276808
hendrycksTest-college_mathematics	acc 0.31 acc_stderr 0.04648231987117316 acc_norm 0.31 acc_norm_stderr 0.04648231987117316
hendrycksTest-college_medicine	acc 0.23121387283236994 acc_stderr 0.032147373020294696 acc_norm 0.23121387283236994 acc_norm_stderr 0.032147373020294696
hendrycksTest-college_physics	acc 0.27450980392156865 acc_stderr 0.04440521906179327 acc_norm 0.27450980392156865 acc_norm_stderr 0.04440521906179327
hendrycksTest-computer_security	acc 0.36 acc_stderr 0.048241815132442176 acc_norm 0.36 acc_norm_stderr 0.048241815132442176
hendrycksTest-conceptual_physics	acc 0.2765957446808511 acc_stderr 0.029241883869628837 acc_norm 0.2765957446808511 acc_norm_stderr 0.029241883869628837
hendrycksTest-econometrics	acc 0.2631578947368421 acc_stderr 0.04142439719489363 acc_norm 0.2631578947368421 acc_norm_stderr 0.04142439719489363
hendrycksTest-electrical_engineering	acc 0.20689655172413793 acc_stderr 0.03375672449560554 acc_norm 0.20689655172413793 acc_norm_stderr 0.03375672449560554
hendrycksTest-elementary_mathematics	acc 0.2698412698412698 acc_stderr 0.022860838309232072 acc_norm 0.2698412698412698 acc_norm_stderr 0.022860838309232072
hendrycksTest-formal_logic	acc 0.2619047619047619 acc_stderr 0.039325376803928704 acc_norm 0.2619047619047619 acc_norm_stderr 0.039325376803928704
hendrycksTest-global_facts	acc 0.35 acc_stderr 0.047937248544110196 acc_norm 0.35 acc_norm_stderr 0.047937248544110196
hendrycksTest-high_school_biology	acc 0.24193548387096775 acc_stderr 0.0243625996930311 acc_norm 0.24193548387096775 acc_norm_stderr 0.0243625996930311
hendrycksTest-high_school_chemistry	acc 0.28078817733990147 acc_stderr 0.0316185633535861 acc_norm 0.28078817733990147 acc_norm_stderr 0.0316185633535861
hendrycksTest-high_school_computer_science	acc 0.33 acc_stderr 0.04725815626252605 acc_norm 0.33 acc_norm_stderr 0.04725815626252605
hendrycksTest-high_school_european_history	acc 0.296969696969697 acc_stderr 0.03567969772268048 acc_norm 0.296969696969697 acc_norm_stderr 0.03567969772268048
hendrycksTest-high_school_geography	acc 0.2878787878787879 acc_stderr 0.03225883512300993 acc_norm 0.2878787878787879 acc_norm_stderr 0.03225883512300993
hendrycksTest-high_school_government_and_politics	acc 0.2538860103626943 acc_stderr 0.0314102478056532 acc_norm 0.2538860103626943 acc_norm_stderr 0.0314102478056532
hendrycksTest-high_school_macroeconomics	acc 0.2743589743589744 acc_stderr 0.022622765767493207 acc_norm 0.2743589743589744 acc_norm_stderr 0.022622765767493207
hendrycksTest-high_school_mathematics	acc 0.26296296296296295 acc_stderr 0.026842057873833706 acc_norm 0.26296296296296295 acc_norm_stderr 0.026842057873833706
hendrycksTest-high_school_microeconomics	acc 0.2647058823529412 acc_stderr 0.028657491285071977 acc_norm 0.2647058823529412 acc_norm_stderr 0.028657491285071977
hendrycksTest-high_school_physics	acc 0.304635761589404 acc_stderr 0.03757949922943343 acc_norm 0.304635761589404 acc_norm_stderr 0.03757949922943343
hendrycksTest-high_school_psychology	acc 0.28623853211009176 acc_stderr 0.019379436628919968 acc_norm 0.28623853211009176 acc_norm_stderr 0.019379436628919968
hendrycksTest-high_school_statistics	acc 0.25462962962962965 acc_stderr 0.02971127586000535 acc_norm 0.25462962962962965 acc_norm_stderr 0.02971127586000535
hendrycksTest-high_school_us_history	acc 0.23039215686274508 acc_stderr 0.029554292605695053 acc_norm 0.23039215686274508 acc_norm_stderr 0.029554292605695053
hendrycksTest-high_school_world_history	acc 0.2869198312236287 acc_stderr 0.029443773022594693 acc_norm 0.2869198312236287 acc_norm_stderr 0.029443773022594693
hendrycksTest-human_aging	acc 0.3811659192825112 acc_stderr 0.03259625118416827 acc_norm 0.3811659192825112 acc_norm_stderr 0.03259625118416827
hendrycksTest-human_sexuality	acc 0.1984732824427481 acc_stderr 0.03498149385462472 acc_norm 0.1984732824427481 acc_norm_stderr 0.03498149385462472
hendrycksTest-international_law	acc 0.3884297520661157 acc_stderr 0.04449270350068382 acc_norm 0.3884297520661157 acc_norm_stderr 0.04449270350068382
hendrycksTest-jurisprudence	acc 0.23148148148148148 acc_stderr 0.04077494709252627 acc_norm 0.23148148148148148 acc_norm_stderr 0.04077494709252627
hendrycksTest-logical_fallacies	acc 0.2331288343558282 acc_stderr 0.03322015795776741 acc_norm 0.2331288343558282 acc_norm_stderr 0.03322015795776741
hendrycksTest-machine_learning	acc 0.21428571428571427 acc_stderr 0.03894641120044792 acc_norm 0.21428571428571427 acc_norm_stderr 0.03894641120044792
hendrycksTest-management	acc 0.3300970873786408 acc_stderr 0.04656147110012352 acc_norm 0.3300970873786408 acc_norm_stderr 0.04656147110012352
hendrycksTest-marketing	acc 0.2905982905982906 acc_stderr 0.029745048572674078 acc_norm 0.2905982905982906 acc_norm_stderr 0.029745048572674078
hendrycksTest-medical_genetics	acc 0.29 acc_stderr 0.04560480215720684 acc_norm 0.29 acc_norm_stderr 0.04560480215720684
hendrycksTest-miscellaneous	acc 0.31545338441890164 acc_stderr 0.016617501738763394 acc_norm 0.31545338441890164 acc_norm_stderr 0.016617501738763394
hendrycksTest-moral_disputes	acc 0.2861271676300578 acc_stderr 0.02433214677913413 acc_norm 0.2861271676300578 acc_norm_stderr 0.02433214677913413
hendrycksTest-moral_scenarios	acc 0.2122905027932961 acc_stderr 0.01367664468583173 acc_norm 0.2122905027932961 acc_norm_stderr 0.01367664468583173
hendrycksTest-nutrition	acc 0.2875816993464052 acc_stderr 0.02591780611714716 acc_norm 0.2875816993464052 acc_norm_stderr 0.02591780611714716
hendrycksTest-philosophy	acc 0.2765273311897106 acc_stderr 0.02540383297817961 acc_norm 0.2765273311897106 acc_norm_stderr 0.02540383297817961
hendrycksTest-prehistory	acc 0.3117283950617284 acc_stderr 0.025773111169630446 acc_norm 0.3117283950617284 acc_norm_stderr 0.025773111169630446
hendrycksTest-professional_accounting	acc 0.26595744680851063 acc_stderr 0.026358065698880592 acc_norm 0.26595744680851063 acc_norm_stderr 0.026358065698880592
hendrycksTest-professional_law	acc 0.25684485006518903 acc_stderr 0.011158455853098832 acc_norm 0.25684485006518903 acc_norm_stderr 0.011158455853098832
hendrycksTest-professional_medicine	acc 0.1801470588235294 acc_stderr 0.023345163616544855 acc_norm 0.1801470588235294 acc_norm_stderr 0.023345163616544855
hendrycksTest-professional_psychology	acc 0.27941176470588236 acc_stderr 0.018152871051538802 acc_norm 0.27941176470588236 acc_norm_stderr 0.018152871051538802
hendrycksTest-public_relations	acc 0.3090909090909091 acc_stderr 0.044262946482000985 acc_norm 0.3090909090909091 acc_norm_stderr 0.044262946482000985
hendrycksTest-security_studies	acc 0.32653061224489793 acc_stderr 0.030021056238440313 acc_norm 0.32653061224489793 acc_norm_stderr 0.030021056238440313
hendrycksTest-sociology	acc 0.25870646766169153 acc_stderr 0.030965903123573026 acc_norm 0.25870646766169153 acc_norm_stderr 0.030965903123573026
hendrycksTest-us_foreign_policy	acc 0.32 acc_stderr 0.04688261722621504 acc_norm 0.32 acc_norm_stderr 0.04688261722621504
hendrycksTest-virology	acc 0.30120481927710846 acc_stderr 0.0357160923005348 acc_norm 0.30120481927710846 acc_norm_stderr 0.0357160923005348
hendrycksTest-world_religions	acc 0.32748538011695905 acc_stderr 0.035993357714560276 acc_norm 0.32748538011695905 acc_norm_stderr 0.035993357714560276
truthfulqa_mc	mc1 0.2423500611995104 mc1_stderr 0.01500067437357034 mc2 0.3859757929597962 mc2_stderr 0.013898628036488968