---
language: en
license: apache-2.0
tags:
- text-classification
- question-classification
- LoRA
- quantization
datasets:
- squad
- glue
model_name: question-classification-lora-quant
base_model: google/gemma-2b-it
widget:
- text: "What is the capital of France?"
- text: "This is a beautiful day."
metrics:
- accuracy
- f1
- precision
- recall
---

# Model Card: Question Classification using LoRA with Quantization

## Model Overview

This model is a fine-tuned version of [google/gemma-2b-it](https://huggingface.co/google/gemma-2b-it) designed to classify text into two categories: **QUESTION** or **NOT_QUESTION**. It was fine-tuned on a custom dataset that combines the **SQuAD** dataset (containing questions) and the **GLUE SST-2** dataset (containing general non-question sentences).

### Model Architecture

- Base Model: `google/gemma-2b-it`
- Fine-tuning Method: LoRA (Low-Rank Adaptation) with k-bit quantization (4-bit quantization with NF4).
- Configurations:
  - Quantization: 4-bit quantization using `BitsAndBytesConfig`
  - Adapter (LoRA) settings:
    - Rank: 64
    - LoRA Alpha: 32
    - Dropout: 0.05
    - Target Modules: `q_proj`, `k_proj`, `v_proj`, `o_proj`

## Dataset

The model was trained using a combination of two datasets:
- **SQuAD v1.1** (Question dataset)
- **GLUE SST-2** (Non-question dataset)

Each dataset was preprocessed to contain a label:
- **QUESTION**: For SQuAD questions
- **NOT_QUESTION**: For non-question sentences from GLUE SST-2.

### Data Preprocessing
- A random removal probability (`P_remove = 0.3`) was applied to remove some of the questions containing a question mark (`?`), to increase the model's robustness.
- Both datasets were balanced with an equal number of samples (`N=100` for training and testing).

## Model Performance

- **Metrics Evaluated**:
  - Accuracy
  - F1 Score
  - Precision
  - Recall
- These metrics were computed on a balanced test dataset containing both question and non-question examples.

## How to Use

You can use this model to classify whether a given text is a question or not. Here’s how you can use it:

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("your_model_name")
model = AutoModelForSequenceClassification.from_pretrained("your_model_name")

inputs = tokenizer("What is the capital of France?", return_tensors="pt")
outputs = model(**inputs)
predictions = torch.argmax(outputs.logits, axis=1)

label = "QUESTION" if predictions == 1 else "NOT_QUESTION"
print(f"Predicted Label: {label}")
```

## Limitations

- The model was trained on English data only, so it may not perform well on non-English languages.
- Since it is fine-tuned on specific datasets (SQuAD and GLUE SST-2), performance may vary with out-of-domain data.
- The model assumes well-formed input sentences, so performance may degrade with informal or very short text.

## Intended Use

This model is intended for text classification tasks where distinguishing between questions and non-questions is needed. Potential use cases include:
- Improving chatbot or virtual assistant interactions.
- Enhancing query detection for search engines.

## License

This model follows the same license as [google/gemma-2b-it](https://huggingface.co/google/gemma-2b-it). Please refer to the original license for any usage restrictions.