---
pipeline_tag: text-generation
inference: false
license: apache-2.0
library_name: transformers
tags:
- language
- granite-3.1
- abliterated
- uncensored
base_model:
- huihui-ai/granite-3.1-8b-instruct-abliterated
---
# granite-3.1-8b-instruct-abliterated-exl2
Model: [granite-3.1-8b-instruct-abliterated](https://huggingface.co/huihui-ai/granite-3.1-8b-instruct-abliterated)  
Made by: [huihui-ai](https://huggingface.co/huihui-ai)  
Granite 3 authors: [Granite Team, IBM](https://huggingface.co/ibm-granite)

## Quants
[4bpw h6 (main)](https://huggingface.co/cgus/granite-3.1-8b-instruct-abliterated-exl2/tree/main)  
[4.5bpw h6](https://huggingface.co/cgus/granite-3.1-8b-instruct-abliterated-exl2/tree/4.5bpw-h6)  
[5bpw h6](https://huggingface.co/cgus/granite-3.1-8b-instruct-abliterated-exl2/tree/5bpw-h6)  
[6bpw h6](https://huggingface.co/cgus/granite-3.1-8b-instruct-abliterated-exl2/tree/6bpw-h6)  
[8bpw h8](https://huggingface.co/cgus/granite-3.1-8b-instruct-abliterated-exl2/tree/8bpw-h8)  

## Quantization notes
Made with exllamav2 0.2.7 with default dataset. This model requires exllamav2 0.2.7 or newer.  
Exl2 quants require to be fully loaded into GPU VRAM, RAM offloading isn't supported natively.  
Additionally it requires Nvidia RTX on Windows or Nvidia RTX/AMD ROCm on Linux.  
These quants can be used with TabbyAPI or Text-Generation-WebUI.  
# Original model card
# huihui-ai/granite-3.1-8b-instruct-abliterated


This is an uncensored version of [ibm-granite/granite-3.1-8b-instruct](https://huggingface.co/ibm-granite/granite-3.1-8b-instruct) created with abliteration (see [remove-refusals-with-transformers](https://github.com/Sumandora/remove-refusals-with-transformers) to know more about it).  
This is a crude, proof-of-concept implementation to remove refusals from an LLM model without using TransformerLens.    

## Use with ollama

You can use [huihui_ai/granite3.1-dense-abliterated](https://ollama.com/huihui_ai/granite3.1-dense-abliterated) directly, 
```
ollama run huihui_ai/granite3.1-dense-abliterated
```