|
|
--- |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- gpt2 |
|
|
- text-generation |
|
|
- pytorch |
|
|
license: mit |
|
|
--- |
|
|
|
|
|
# SchorbGPT-Medium |
|
|
|
|
|
This is a medium sized language model trained on web data. The model uses the GPT-2 architecture and tokenizer. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- Model Type: GPT-2 |
|
|
- Training Data: Web text data |
|
|
- Number of Parameters: GPT-2 medium scale |
|
|
- Context Length: 512 tokens |
|
|
- Training Framework: PyTorch |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from transformers import AutoModelForCausalLM, AutoTokenizer |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("iimaginary/schorbGPT-medium") |
|
|
model = AutoModelForCausalLM.from_pretrained("iimaginary/schorbGPT-medium") |
|
|
|
|
|
text = "Your prompt here" |
|
|
inputs = tokenizer(text, return_tensors="pt") |
|
|
outputs = model.generate(**inputs, max_length=100) |
|
|
response = tokenizer.decode(outputs[0], skip_special_tokens=True) |
|
|
``` |
|
|
|
|
|
## Performance and Model Analysis |
|
|
|
|
|
### Zero-shot Evaluation Results |
|
|
|
|
|
| Task | Metric | Value | Stderr | |
|
|
|------|--------|-------|--------| |
|
|
| WikiText | bits_per_byte | 0.9860 | N/A | |
|
|
| WikiText | byte_perplexity | 1.9806 | N/A | |
|
|
| WikiText | word_perplexity | 38.6497 | N/A | |
|
|
| ARC Easy | accuracy | 48.02% | ±1.03% | |
|
|
| ARC Easy | accuracy (normalized) | 42.17% | ±1.01% | |
|
|
| HellaSwag | accuracy | 29.06% | ±0.45% | |
|
|
| HellaSwag | accuracy (normalized) | 31.26% | ±0.46% | |
|
|
| LAMBADA | accuracy | 33.90% | ±0.66% | |
|
|
| LAMBADA | perplexity | 36.2055 | ±1.4052 | |
|
|
| PIQA | accuracy | 61.92% | ±1.13% | |
|
|
| PIQA | accuracy (normalized) | 62.46% | ±1.13% | |
|
|
| Winogrande | accuracy | 50.59% | ±1.41% | |
|
|
|
|
|
### Analysis and Comparisons |
|
|
|
|
|
#### Language Modeling Performance |
|
|
The model achieves a word perplexity of 38.65 on WikiText, which is competitive with similar-sized models. For comparison: |
|
|
- Original GPT-2 (small): ~35-40 perplexity |
|
|
- GPT-2 medium: ~30-35 perplexity |
|
|
- BERT-base: ~40-45 perplexity |
|
|
|
|
|
#### Task-Specific Analysis: |
|
|
|
|
|
1. Physical and Commonsense Reasoning: |
|
|
- PIQA: 61.92% (Random baseline: 50%) |
|
|
- Comparable to GPT-2 small/medium performance |
|
|
- Shows good physical commonsense understanding |
|
|
|
|
|
2. Science Knowledge: |
|
|
- ARC Easy: 48.02% (Random baseline: 25%) |
|
|
- Above random chance and demonstrates basic scientific knowledge |
|
|
- Similar to performance seen in early GPT-2 variants |
|
|
|
|
|
3. Linguistic Understanding: |
|
|
- LAMBADA: 33.90% accuracy with perplexity of 36.21 |
|
|
- HellaSwag: 31.26% (Random baseline: 25%) |
|
|
- Performance indicates basic linguistic and contextual understanding |
|
|
- Typical range for non-fine-tuned models of this scale |
|
|
|
|
|
4. Reasoning and Logic: |
|
|
- Winogrande: 50.59% (Random baseline: 50%) |
|
|
- At par with random chance, suggesting room for improvement in complex reasoning tasks |
|
|
- Common for base models without specific fine-tuning |
|
|
|
|
|
### Strengths and Limitations |
|
|
|
|
|
**Strengths:** |
|
|
- Strong performance on physical commonsense (PIQA) |
|
|
- Decent basic science knowledge (ARC Easy) |
|
|
- Competitive language modeling metrics |
|
|
|
|
|
**Limitations:** |
|
|
- Limited complex reasoning capabilities (Winogrande) |
|
|
- Basic linguistic understanding could be improved (LAMBADA, HellaSwag) |
|
|
- Performance typical of base models without task-specific fine-tuning |
|
|
|
|
|
## Limitations |
|
|
|
|
|
This is a base model without fine-tuning or alignment. It should be used with appropriate consideration of its capabilities and limitations. |
|
|
|
