Spaces:

---

ggml-org

/

gguf-my-repo

like 1.32k

Running on A10G

I cant run 70b models on my machine but using Q2_K quants it works. Im generally new this whole open source AI stuff but from my understanding IQ offers a bit higher quality but with a inference speed tradeoff.