YAML Metadata
Warning:
The pipeline tag "text2text-generation" is not in the official list: text-classification, token-classification, table-question-answering, question-answering, zero-shot-classification, translation, summarization, feature-extraction, text-generation, fill-mask, sentence-similarity, text-to-speech, text-to-audio, automatic-speech-recognition, audio-to-audio, audio-classification, audio-text-to-text, voice-activity-detection, depth-estimation, image-classification, object-detection, image-segmentation, text-to-image, image-to-text, image-to-image, image-to-video, unconditional-image-generation, video-classification, reinforcement-learning, robotics, tabular-classification, tabular-regression, tabular-to-text, table-to-text, multiple-choice, text-ranking, text-retrieval, time-series-forecasting, text-to-video, image-text-to-text, visual-question-answering, document-question-answering, zero-shot-image-classification, graph-ml, mask-generation, zero-shot-object-detection, text-to-3d, image-to-3d, image-feature-extraction, video-text-to-text, keypoint-detection, visual-document-retrieval, any-to-any, video-to-video, other
CodeEncoderDecoderModel-Ghost-largeπ»
A multilingual encoder-decoder model for generating docstrings from code snippets.
It is based on a custom BERT-style encoder pretrained on source code (CodeModernBERT-Ghost
) and a large-scale decoder model (GPT2-large
).
ποΈ Model Architecture
- Encoder:
Shuu12121/CodeModernBERT-Ghost
- Decoder:
openai-community/gpt2-large
- Connected via HuggingFace's
EncoderDecoderModel
with cross-attention.
π― Intended Use
- Generating docstrings (documentation comments) for functions or methods in multiple languages.
- Summarizing code for educational or review purposes.
- Assisting in automated documentation generation pipelines.
Supported languages (code input):
- Python
- Java
π¦ How to Use
from transformers import AutoTokenizer, EncoderDecoderModel
import torch
model = EncoderDecoderModel.from_pretrained("Shuu12121/CodeEncoderDecoderModel-Ghost-large").to("cuda")
encoder_tokenizer = AutoTokenizer.from_pretrained("Shuu12121/CodeEncoderDecoderModel-Ghost-large", subfolder="encoder_tokenizer")
decoder_tokenizer = AutoTokenizer.from_pretrained("Shuu12121/CodeEncoderDecoderModel-Ghost-large", subfolder="decoder_tokenizer")
if decoder_tokenizer.pad_token is None:
decoder_tokenizer.pad_token = decoder_tokenizer.eos_token
code = '''
def greet(name):
return f"Hello, {name}!"
'''
inputs = encoder_tokenizer(code, return_tensors="pt", truncation=True, padding=True, max_length=2048).to("cuda")
outputs = model.generate(
input_ids=inputs.input_ids,
attention_mask=inputs.attention_mask,
max_length=256,
num_beams=5,
early_stopping=True,
decoder_start_token_id=model.config.decoder_start_token_id,
eos_token_id=model.config.eos_token_id,
pad_token_id=model.config.pad_token_id,
no_repeat_ngram_size=2
)
docstring = decoder_tokenizer.decode(outputs[0], skip_special_tokens=True)
print(docstring)
π§ͺ Training Details
- Task: Code-to-docstring generation
- Dataset: CodeXGLUE: Code-to-Text β using subsets of Python, Java, JavaScript, Go, Ruby, PHP
- Loss: Cross-entropy loss over tokenized docstrings
- Max input length: 2048 (encoder), max output length: 256 (decoder)
- Decoder modifications: Adapted GPT2-large with padding and cross-attention
β οΈ Limitations & Risks
- Generated documentation may be inaccurate, incomplete, or misleading. Always review generated docstrings manually.
- Formatting may not follow specific standards (e.g., Google/Numpy style in Python or full Javadoc).
- Limited context: Only considers single-function input; lacks broader project-level understanding.
- Language variance: Performance may differ depending on the programming language due to data distribution.
- β οΈ Decoder risks (GPT2-large):
GPT-2 models are known to sometimes generate inappropriate, offensive, or biased outputs, depending on the prompt.
Although this model is fine-tuned on technical datasets (code-docstring pairs), due to inherited properties fromgpt2-large
, similar risks may still be present in edge cases. Please exercise caution, especially when using the model in public or educational settings.
π License
Apache-2.0
Model weights and tokenizer artifacts are released under the same license. You are free to use, modify, and redistribute with attribution.
- Downloads last month
- 11
Inference Providers
NEW
This model isn't deployed by any Inference Provider.
π
Ask for provider support
Model tree for Shuu12121/CodeEncoderDecoderModel-Ghost-large
Base model
Shuu12121/CodeModernBERT-Ghost