Menda-3b-Optim-100: Optimized GRPO-Tuned Qwen2.5 Model

Menda-3b-Optim-100 is a fine-tuned version of Qwen2.5-3B-Instruct, trained with an optimized GRPO (Guided Reinforcement from Preference Optimization) methodology for 100 steps. This model shows significantly improved performance on reasoning benchmarks and achieves the highest MMLU score among all Menda-3B checkpoints.

Model Details

• Base Model: Qwen/Qwen2.5-3B-Instruct
• Training Method: Optimized GRPO with enhanced reward functions
• Training Steps: 100
• Parameters: 3 billion
• Context Length: 32K tokens
• Training Data: GSM8K (mathematical reasoning)
• Chat Template: Uses the Qwen2 chat template

Optimization Improvements

This model uses several key optimizations over the standard GRPO approach:

1. Higher Learning Rate: 2e-5 (4x higher than standard)
2. Improved Scheduler: Cosine with restarts
3. Enhanced Reward Functions:
   • Continuous correctness rewards with partial credit
   • Multi-component reasoning quality assessment
   • Format validation with both strict and soft checks
4. Adjusted Batch Processing: Optimized gradient accumulation

Benchmark Results

Menda-3b-Optim-100 has been evaluated on several standard benchmarks:

Benchmark	Task Type	Accuracy
GSM8K	Mathematical Reasoning	70.0%
OpenBookQA	Knowledge-based QA	20.0% (40.0% normalized)

MMLU Performance

MMLU Category	Score
Overall	70.35%
Humanities	76.15%
Social Sciences	76.67%
STEM	61.58%
Other	71.54%

Key Strengths

• Highest MMLU Score: This checkpoint achieves the highest overall MMLU score (70.35%) among all Menda-3B checkpoints.
• Strong Mathematical Reasoning: Excellent 70% performance on GSM8K, demonstrating strong mathematical problem-solving capabilities.
• Balanced Performance: Maintains strong performance across diverse knowledge domains.
• Efficient Training: Achieves superior results with minimal training (only 100 steps).
• Subject-Specific Excellence: Perfect 100% on Logical Fallacies, Medical Genetics, Professional Psychology, and College Biology.

Chat Format

This model uses the standard Qwen2 chat template. For best results when using the model directly, format your prompts as follows:

<|im_start|>system
You are a helpful AI assistant.<|im_end|>
<|im_start|>user
Your question here<|im_end|>
<|im_start|>assistant

When using the model through the Hugging Face Transformers library, the chat template will be applied automatically when using the chat_template functionality.

Usage Examples

Basic Usage with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "weathermanj/Menda-3b-Optim-100"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

prompt = "Explain the concept of machine learning in simple terms."
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_length=300)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Chat Usage with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "weathermanj/Menda-3b-Optim-100"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Give me a short introduction to large language models."}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)

Training Configuration

The model was trained using the optimized GRPO methodology with the following configuration:

• LoRA Rank: 128
• Learning Rate: 2e-5
• Optimizer: AdamW (8-bit)
• Batch Size: 1 per device
• Gradient Accumulation Steps: 8
• Scheduler: Cosine with restarts
• Training Samples: 100 examples from GSM8K

License

This model inherits the license of the base Qwen2.5-3B-Instruct model. Please refer to the Qwen2 license for details.