Hugging Face's logo

asedmammad
/
Marx-3B-GGML

English
llama
text-generation-inference
Model card Files Community
Marx-3B-GGML
File size: 2,162 Bytes 
84974f8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
0e233b3
84974f8
 
 
0e233b3
84974f8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
 
---
inference: false
tags:
- llama
- vicuna
- text-generation-inference
---

# acrastt's EverythingLM 3B GGML


These files are GGML format model files for [acrastt's EverythingLM 3B GGML](https://huggingface.co/acrastt/EverythingLM-3B).

GGML files are for CPU + GPU inference using [llama.cpp](https://github.com/ggerganov/llama.cpp) and libraries and UIs which support this format, such as:
* [text-generation-webui](https://github.com/oobabooga/text-generation-webui)
* [KoboldCpp](https://github.com/LostRuins/koboldcpp)
* [ParisNeo/GPT4All-UI](https://github.com/ParisNeo/gpt4all-ui)
* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
* [ctransformers](https://github.com/marella/ctransformers)

## How to run in `llama.cpp`

I use the following command line; adjust for your tastes and needs:

```
./main -t 8 -ngl 26 -m EverythingLM-3B.ggmlv3.q4_0.bin --color -c 2048 --temp 0.7 --repeat_penalty 1.1 -n -1 -p "prompt goes here"
```
Change `-t 8` to the number of physical CPU cores you have. For example if your system has 8 cores/16 threads, use `-t 8`.

Change `-ngl 26` to the number of layers to offload to GPU. Remove it if you don't have GPU acceleration.

If you want to have a chat-style conversation, replace the `-p <PROMPT>` argument with `-i -ins`, also you can use --interactive-first to also start in interactive mode.

## Compatibility

I have uploded llama.cpp quant methods (`q4_0, q4_1, q5_0, q5_1, q8_0`).

Please refer to [llama.cpp](https://github.com/ggerganov/llama.cpp) and [TheBloke](https://huggingface.co/TheBloke)'s GGML models for further explanation.

## How to run in `text-generation-webui`

Further instructions here: [text-generation-webui/docs/llama.cpp-models.md](https://github.com/oobabooga/text-generation-webui/blob/main/docs/llama.cpp-models.md).

<!-- footer start -->
## Thanks

Thanks to [TheBloke](https://huggingface.co/TheBloke) for inspiration and providing almost all of the readme here!

Thanks to [acrastt](https://huggingface.co/acrastt) for providing checkpoints of the model.

Thanks to [Georgi Gerganov](https://github.com/ggerganov) and all of the awesome people in the AI community.