library_name: transformers
datasets:
- bigcode/the-stack-v2
- modularStarEncoder/SynthCode2Code2NL-neardedup
license: bigcode-openrail-m
base_model:
- modularStarEncoder/ModularStarEncoder
ModularStarEncoder-1B Fine-Tuned model
ModularStarEncoder-finetuned is an encoder built on top of ModularStarEncoder-1B Pre-trained on SynthCode2Code2NL. ModularStarEncoder fine-tuned is an encoder for various retrieval tasks, enabling the end user to select the model size that meets their memory and computational constraints. We built ModularStarEncoder on top of StarCoder-2, reducing its size from 15B to 1B parameters in bfloat16.
The model is finetuned with CLIP objective
- Paper: Link
- Languages: English, Go, Ruby, Python, Java, C++, PHP, C, JavaScript
How to use
from transformers import AutoModel
from transformers import AutoTokenizer
#import the model
model = AutoModel.from_pretrained("andreagurioli1995/ModularStarEncoder-finetuned", trust_remote_code=True)
#import the tokenizer, the tokenizer applies LEFT padding!
tokenizer = AutoTokenizer.from_pretrained("andreagurioli1995/ModularStarEncoder-finetuned")
language = "yourlanguagelowercased"
#instruction in case of code embedding in a code language
instruction_code = f"Represent this {language} code snippet for retrieval:"
#instruction in case of code embedding in English
instruction_natural_language = "Represent this code description for retrieving supporting snippets of code:"
code_snippet = "your code to embed here"
#You should follow this pattern to embed a snippet of code or natural language queries
sentence = f"{tokenizer.sep_token}{instruction_code}{tokenizer.sep_token}{code_snippet)}{tokenizer.cls_token}"
#Tokenizing your sentence
tokenized_sensence = tokenizer(sentence, return_tensors="pt",truncation=True, max_length=2048)
#Embedding the tokenized sentence
embedded_sentence = model(**sentence)
You will get as an output three elements:
- projected_pooled_normalized: a list of the projected, pooled, and normalized embeddings from the five exit points;
- raw_hidden_states: raw representation from all the hidden states of the model, without pooling, normalization, and projection
- attentions: attention scores from the encoder
Training
We fine-tuned ModularStarEncoder with a batch size of 2048 contrastive samples for 20,000 training steps. The pre-training and fine-tuning were conducted on 512 NVIDIA Ampere (64GB) GPUs using the Leonardo supercomputer, requiring 450,000 GPU working hours.
| Hyperparameter | Value |
|---|---|
| Hidden size | 1024 |
| Max. position embeddings | 2048 |
| Num. of attention heads | 12 |
| Num. of key values heads | 4 |
| Num. of hidden layers | 36 |
| Attention | GQA |
| Num. of parameters | ≈1B |
| Loss function | CLIP loss |
| Multi-layer loss | yes |
Evaluation
Here we briefly show our codeSearchNet (codeXGLUE) results between different layers:
- (* size and corresponding projection head present in this model)
Licence
The model is licensed under the BigCode OpenRAIL-M v1 license agreement. You can find the full agreement here.