Triangle104/Epos-8b-Q6_K-GGUF

This model was converted to GGUF format from P0x0/Epos-8b using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

---

Model details:

Epos-8B is a fine-tuned version of the base model Llama-3.1-8B from Meta, optimized for storytelling, dialogue generation, and creative writing. The model specializes in generating rich narratives, immersive prose, and dynamic character interactions, making it ideal for creative tasks.

    Model Details






    


    Model Description

Epos-8B is an 8 billion parameter language model fine-tuned for storytelling and narrative tasks. Inspired by the grandeur of epic tales, it is designed to produce high-quality, engaging content that evokes the depth and imagination of ancient myths and modern storytelling traditions.

Developed by: P0x0 Funded by: P0x0 Shared by: P0x0 Model type: Transformer-based Language Model Language(s) (NLP): Primarily English License: Apache 2.0 Finetuned from model: meta-llama/Llama-3.1-8B

    Model Sources

Repository: Epos-8B on Hugging Face GGUF Repository: Epos-8B-GGUF (TO BE ADDED)

    Uses






    


    Direct Use

Epos-8B is ideal for:

Storytelling: Generate detailed, immersive, and engaging narratives. Dialogue Creation: Create realistic and dynamic character interactions for stories or games.

    How to Get Started with the Model

To run the quantized version of the model, you can use KoboldCPP, which allows you to run quantized GGUF models locally.

    Steps:

Download KoboldCPP. Follow the setup instructions provided in the repository. Download the GGUF variant of Epos-8B from Epos-8B-GGUF. Load the model in KoboldCPP and start generating!

Alternatively, integrate the model directly into your code with the following snippet:

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("P0x0/Epos-8B") model = AutoModelForCausalLM.from_pretrained("P0x0/Epos-8B")

input_text = "Once upon a time in a distant land..." inputs = tokenizer(input_text, return_tensors="pt") outputs = model.generate(**inputs)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo Triangle104/Epos-8b-Q6_K-GGUF --hf-file epos-8b-q6_k.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo Triangle104/Epos-8b-Q6_K-GGUF --hf-file epos-8b-q6_k.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo Triangle104/Epos-8b-Q6_K-GGUF --hf-file epos-8b-q6_k.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo Triangle104/Epos-8b-Q6_K-GGUF --hf-file epos-8b-q6_k.gguf -c 2048