I cant run it on vllm
hi, i want tu use your llm with docker and vllm but when i use this
vllm/vllm-openai:latest
--model bartowski/calme-3.2-instruct-78b-GGUF/calme-3.2-instruct-78b-IQ1_M.gguf
--tensor-parallel-size=2
--enforce-eager
i only get an error:
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'bartowski/calme-3.2-instruct-78b-GGUF/calme-3.2-instruct-78b-IQ1_M.gguf'. Use repo_type argument if needed.
even with
vllm/vllm-openai:latest
--model bartowski/calme-3.2-instruct-78b-GGUF
--tensor-parallel-size=2
--enforce-eager
i get:
ValueError: No supported config format found in bartowski/calme-3.2-instruct-78b-GGUF
are you able to download it locally and use it there? I'm not positive what format vllm expects for GGUF repositories