README.md

medmekk/gemma-2-2b-it.GGUF

GGUF quantized versions of google/gemma-2-2b-it

Available Formats:

• Q2_K: gemma-2-2b-it-Q2_K.gguf
• Q3_K_S: gemma-2-2b-it-Q3_K_S.gguf
• Q3_K_M: gemma-2-2b-it-Q3_K_M.gguf
• Q3_K_L: gemma-2-2b-it-Q3_K_L.gguf
• Q4_0: gemma-2-2b-it-Q4_0.gguf
• Q4_K_S: gemma-2-2b-it-Q4_K_S.gguf
• Q4_K_M: gemma-2-2b-it-Q4_K_M.gguf
• Q5_0: gemma-2-2b-it-Q5_0.gguf
• Q5_K_S: gemma-2-2b-it-Q5_K_S.gguf
• Q5_K_M: gemma-2-2b-it-Q5_K_M.gguf
• Q6_K: gemma-2-2b-it-Q6_K.gguf
• Q8_0: gemma-2-2b-it-Q8_0.gguf
• IQ3_M_IMAT: gemma-2-2b-it-IQ3_M_imat.gguf
• IQ3_XXS_IMAT: gemma-2-2b-it-IQ3_XXS_imat.gguf
• Q4_K_M_IMAT: gemma-2-2b-it-Q4_K_M_imat.gguf
• Q4_K_S_IMAT: gemma-2-2b-it-Q4_K_S_imat.gguf
• IQ4_NL_IMAT: gemma-2-2b-it-IQ4_NL_imat.gguf
• IQ4_XS_IMAT: gemma-2-2b-it-IQ4_XS_imat.gguf
• Q5_K_M_IMAT: gemma-2-2b-it-Q5_K_M_imat.gguf
• Q5_K_S_IMAT: gemma-2-2b-it-Q5_K_S_imat.gguf

Usage with llama.cpp:

# CLI:
llama-cli --hf-repo medmekk/gemma-2-2b-it.GGUF --hf-file MODEL_FILE -p "Your prompt"

# Server:
llama-server --hf-repo medmekk/gemma-2-2b-it.GGUF --hf-file MODEL_FILE -c 2048