SandLogic Technology - Quantized Llama-3.2-1B-Instruct-Medical-GGUF

Model Description

We have quantized the Llama-3.2-1B-Instruct-Medical-GGUF model into two variants:

1. Q5_KM
2. Q4_KM

These quantized models offer improved efficiency while maintaining performance in medical-related tasks.

Discover our full range of quantized language models by visiting our SandLogic Lexicon GitHub. To learn more about our company and services, check out our website at SandLogic.

Original Model Information

• Base Model: Meta Llama 3.2 1B Instruct
• Developer: Meta (base model)
• Model Type: Multilingual large language model (LLM)
• Architecture: Auto-regressive language model with optimized transformer architecture
• Parameters: 1 billion
• Training Approach: Supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF)

Fine-tuning Details

• Dataset: bigbio/med_qa
• Languages: English, simplified Chinese, and traditional Chinese
• Dataset Size:
  • English: 12,723 questions
  • Simplified Chinese: 34,251 questions
  • Traditional Chinese: 14,123 questions
• Data Type: Free-form multiple-choice OpenQA for medical problems, collected from professional medical board exams

Model Capabilities

This model is optimized for medical-related dialogue and tasks, including:

• Answering medical questions
• Summarizing medical information
• Assisting with medical problem-solving

Intended Use in Medical Domain

1. Medical Education: Assisting medical students in exam preparation and learning
2. Clinical Decision Support: Providing quick references for healthcare professionals
3. Patient Education: Explaining medical concepts in simple terms for patients
4. Medical Literature Review: Summarizing and extracting key information from medical texts
5. Differential Diagnosis: Assisting in generating potential diagnoses based on symptoms
6. Medical Coding: Aiding in the accurate coding of medical procedures and diagnoses
7. Drug Information: Providing information on medications, their uses, and potential interactions
8. Medical Translation: Assisting with medical translations across supported languages

Quantized Variants

1. Q5_KM: 5-bit quantization using the KM method
2. Q4_KM: 4-bit quantization using the KM method

These quantized models aim to reduce model size and improve inference speed while maintaining performance as close to the original model as possible.

Usage

pip install llama-cpp-python 

Please refer to the llama-cpp-python documentation to install with GPU support.

Basic Text Completion

Here's an example demonstrating how to use the high-level API for basic text completion:

from llama_cpp import Llama

llm = Llama(
    model_path="./models/Llama-3.2-1B-Medical_Q4_KM.gguf",
    verbose=False,
    # n_gpu_layers=-1, # Uncomment to use GPU acceleration
    # n_ctx=2048, # Uncomment to increase the context window
)

output = llm.create_chat_completion(
    messages =[
    {
        "role": "system",
        "content": """ You are a helpful, respectful and honest medical assistant. Yu are developed by SandLogic Technologies 
            Always answer as helpfully as possible, while being safe. 
            Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. 
            Please ensure that your responses are socially unbiased and positive in nature. If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. 
            If you don’t know the answer to a question, please don’t share false information."""
            
        ,
    },
    {"role": "user", "content": "I have been experiencing a persistent cough for the last two weeks, along with a mild fever and fatigue. What could be the possible causes of these symptoms?"},
]
)

print(output["choices"][0]['message']['content'])

Download

You can download Llama models in gguf format directly from Hugging Face using the from_pretrained method. This feature requires the huggingface-hub package.

To install it, run: pip install huggingface-hub

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="SandLogicTechnologies/Llama-3.2-1B-Instruct-Medical-GGUF",
    filename="*Llama-3.2-1B-Medical_Q5_KM.gguf",
    verbose=False
)

By default, from_pretrained will download the model to the Hugging Face cache directory. You can manage installed model files using the huggingface-cli tool.

Ethical Considerations and Limitations

• This model is not a substitute for professional medical advice, diagnosis, or treatment
• Users should be aware of potential biases in the training data
• The model's knowledge cutoff date may limit its awareness of recent medical developments

Acknowledgements

We thank Meta for developing the original Llama-3.2-1B-Instruct model and the creators of the bigbio/med_qa dataset. Special thanks to Georgi Gerganov and the entire llama.cpp development team for their outstanding contributions.

Contact

For any inquiries or support, please contact us at [email protected] or visit our support page.

Explore More

For any inquiries or support, please contact us at [email protected] or visit our support page.