metadata

language: en
license: apache-2.0
tags:
  - text-classification
  - question-classification
  - LoRA
  - quantization
datasets:
  - squad
  - glue
model_name: question-classification-lora-quant
base_model: google/gemma-2b-it
widget:
  - text: What is the capital of France?
  - text: This is a beautiful day.
metrics:
  - accuracy
  - f1
  - precision
  - recall

Model Card: Question Classification using LoRA with Quantization

Model Overview

This model is a fine-tuned version of google/gemma-2b-it designed to classify text into two categories: QUESTION or NOT_QUESTION. It was fine-tuned on a custom dataset that combines the SQuAD dataset (containing questions) and the GLUE SST-2 dataset (containing general non-question sentences).

Model Architecture

Base Model: google/gemma-2b-it
Fine-tuning Method: LoRA (Low-Rank Adaptation) with k-bit quantization (4-bit quantization with NF4).
Configurations:
- Quantization: 4-bit quantization using BitsAndBytesConfig
- Adapter (LoRA) settings:
  - Rank: 64
  - LoRA Alpha: 32
  - Dropout: 0.05
  - Target Modules: q_proj, k_proj, v_proj, o_proj

Dataset

The model was trained using a combination of two datasets:

SQuAD v1.1 (Question dataset)
GLUE SST-2 (Non-question dataset)

Each dataset was preprocessed to contain a label:

QUESTION: For SQuAD questions
NOT_QUESTION: For non-question sentences from GLUE SST-2.

Data Preprocessing

A random removal probability (P_remove = 0.3) was applied to remove some of the questions containing a question mark (?), to increase the model's robustness.
Both datasets were balanced with an equal number of samples (N=100 for training and testing).

Model Performance

Metrics Evaluated:
- Accuracy
- F1 Score
- Precision
- Recall
These metrics were computed on a balanced test dataset containing both question and non-question examples.

How to Use

You can use this model to classify whether a given text is a question or not. Here’s how you can use it:

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("your_model_name")
model = AutoModelForSequenceClassification.from_pretrained("your_model_name")

inputs = tokenizer("What is the capital of France?", return_tensors="pt")
outputs = model(**inputs)
predictions = torch.argmax(outputs.logits, axis=1)

label = "QUESTION" if predictions == 1 else "NOT_QUESTION"
print(f"Predicted Label: {label}")

Limitations

The model was trained on English data only, so it may not perform well on non-English languages.
Since it is fine-tuned on specific datasets (SQuAD and GLUE SST-2), performance may vary with out-of-domain data.
The model assumes well-formed input sentences, so performance may degrade with informal or very short text.

Intended Use

This model is intended for text classification tasks where distinguishing between questions and non-questions is needed. Potential use cases include:

Improving chatbot or virtual assistant interactions.
Enhancing query detection for search engines.

License

This model follows the same license as google/gemma-2b-it. Please refer to the original license for any usage restrictions.