# Llama-3-3B CodeSearchNet Fine-tuned

This repository hosts a **Llama 3 (3B) model** fine-tuned on the **CodeSearchNet dataset**, which contains code in six programming languages.

## 📝 Model Details

- **Base Model**: Llama 3 (3B)
- **Fine-tuning Dataset**: CodeSearchNet
- **Languages Covered**: Python, Java, JavaScript, PHP, Ruby, Go
- **Training Method**: Supervised fine-tuning (SFT) with a contrastive loss objective for code search tasks
- **Tokenization**: Llama 3 tokenizer with additional tokens for code-specific keywords
- **Frameworks Used**: Hugging Face `transformers`, PyTorch, PEFT (for LoRA-based tuning)

## 📚 Dataset

The model is trained on the **CodeSearchNet** dataset, which contains:
- Function-level code snippets
- Paired natural language descriptions
- Multiple programming languages for multi-language search support

### **Dataset Sources**
- [CodeSearchNet Dataset](https://github.com/github/CodeSearchNet)
- Contains ~2M code snippets from open-source repositories

## 🚀 Training Setup

- **Hardware**: NVIDIA A100 GPUs
- **Batch Size**: 16
- **Learning Rate**: 2e-5 with cosine annealing
- **Max Sequence Length**: 512
- **Fine-tuning Duration**: 3 epochs

## 🔍 Intended Use

- **Code Search**: Retrieve relevant code snippets given a natural language query
- **Code Completion**: Provide context-aware code suggestions
- **Code-to-Text Generation**: Explain code functionality in natural language
- **Multi-language Code Retrieval**: Search across different programming languages