metadata

license: mit
datasets:
  - avaliev/chat_doctor
language:
  - en
base_model: prithivMLmods/Llama-Doctor-3.2-3B-Instruct
pipeline_tag: text-generation
library_name: transformers
tags:
  - Llama-3.2
  - 3B
  - Llama-Doctor
  - Instruct
  - Llama-Cpp
  - meta
  - pytorch
  - safetensors
  - llama-cpp
  - gguf-my-repo

Triangle104/Llama-Doctor-3.2-3B-Instruct-Q4_K_S-GGUF

This model was converted to GGUF format from prithivMLmods/Llama-Doctor-3.2-3B-Instruct using llama.cpp via the ggml.ai's GGUF-my-repo space. Refer to the original model card for more details on the model.

Model details:

The Llama-Doctor-3.2-3B-Instruct model is designed for text generation tasks, particularly in contexts where instruction-following capabilities are needed. This model is a fine-tuned version of the base Llama-3.2-3B-Instruct model and is optimized for understanding and responding to user-provided instructions or prompts. The model has been trained on a specialized dataset, avaliev/chat_doctor, to enhance its performance in providing conversational or advisory responses, especially in medical or technical fields. Key Use Cases:

Conversational AI: Engage in dialogue, answering questions, or providing responses based on user instructions.
Text Generation: Generate content, summaries, explanations, or solutions to problems based on given prompts.
Instruction Following: Understand and execute instructions, potentially in complex or specialized domains like medical, technical, or academic fields.

The model leverages a PyTorch-based architecture and comes with various files such as configuration files, tokenizer files, and special tokens maps to facilitate smooth deployment and interaction. Intended Applications:

Chatbots for customer support or virtual assistants.
Medical Consultation Tools for generating advice or answering medical queries (given its training on the chat_doctor dataset).
Content Creation tools, helping generate text based on specific instructions.
Problem-solving Assistants that offer explanations or answers to user queries, particularly in instructional contexts.

Use with llama.cpp

Install llama.cpp through brew (works on Mac and Linux)

brew install llama.cpp

Invoke the llama.cpp server or the CLI.

CLI:

llama-cli --hf-repo Triangle104/Llama-Doctor-3.2-3B-Instruct-Q4_K_S-GGUF --hf-file llama-doctor-3.2-3b-instruct-q4_k_s.gguf -p "The meaning to life and the universe is"

Server:

llama-server --hf-repo Triangle104/Llama-Doctor-3.2-3B-Instruct-Q4_K_S-GGUF --hf-file llama-doctor-3.2-3b-instruct-q4_k_s.gguf -c 2048

Note: You can also use this checkpoint directly through the usage steps listed in the Llama.cpp repo as well.

Step 1: Clone llama.cpp from GitHub.

git clone https://github.com/ggerganov/llama.cpp

Step 2: Move into the llama.cpp folder and build it with LLAMA_CURL=1 flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux).

cd llama.cpp && LLAMA_CURL=1 make

Step 3: Run inference through the main binary.

./llama-cli --hf-repo Triangle104/Llama-Doctor-3.2-3B-Instruct-Q4_K_S-GGUF --hf-file llama-doctor-3.2-3b-instruct-q4_k_s.gguf -p "The meaning to life and the universe is"

./llama-server --hf-repo Triangle104/Llama-Doctor-3.2-3B-Instruct-Q4_K_S-GGUF --hf-file llama-doctor-3.2-3b-instruct-q4_k_s.gguf -c 2048