Triangle104/Phi-3.5-mini-TitanFusion-0.1-Q6_K-GGUF

This model was converted to GGUF format from bunnycore/Phi-3.5-mini-TitanFusion-0.1 using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

---

Model details:

This is a merged pre-trained language model created using the TIES merge method. It is based on the microsoft/Phi-3.5-mini-instruct model and incorporates the knowledge and capabilities of the nbeerbower/phi3.5-gutenberg-4B and ArliAI/Phi-3.5-mini-3.8B-ArliAI-RPMax-v1.1 models.

Capabilities: Roleplay: The model can engage in role-playing scenarios, taking on different personas and responding to prompts in a character-appropriate manner. Creative Writing: It can assist in creative writing tasks, such as brainstorming ideas, generating plotlines, or developing characters. Reasoning: The model can reason about information and draw conclusions based on the data it has been trained on. This is a merge of pre-trained language models created using mergekit.

Merge Details Merge Method This model was merged using the TIES merge method using microsoft/Phi-3.5-mini-instruct as a base.

Models Merged The following models were included in the merge:

nbeerbower/phi3.5-gutenberg-4B ArliAI/Phi-3.5-mini-3.8B-ArliAI-RPMax-v1.1 Configuration The following YAML configuration was used to produce this model:

models:

• model: ArliAI/Phi-3.5-mini-3.8B-ArliAI-RPMax-v1.1 parameters: weight: 1
• model: nbeerbower/phi3.5-gutenberg-4B parameters: weight: 1

merge_method: ties base_model: microsoft/Phi-3.5-mini-instruct parameters: density: 1 normalize: true int8_mask: true dtype: bfloat16

---

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo Triangle104/Phi-3.5-mini-TitanFusion-0.1-Q6_K-GGUF --hf-file phi-3.5-mini-titanfusion-0.1-q6_k.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo Triangle104/Phi-3.5-mini-TitanFusion-0.1-Q6_K-GGUF --hf-file phi-3.5-mini-titanfusion-0.1-q6_k.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo Triangle104/Phi-3.5-mini-TitanFusion-0.1-Q6_K-GGUF --hf-file phi-3.5-mini-titanfusion-0.1-q6_k.gguf -p "The meaning to life and the universe is"

or

./llama-server --hf-repo Triangle104/Phi-3.5-mini-TitanFusion-0.1-Q6_K-GGUF --hf-file phi-3.5-mini-titanfusion-0.1-q6_k.gguf -c 2048