File size: 2,783 Bytes
4e06761 4f4b56f 799a836 4f4b56f 9a8e37f 4f4b56f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 |
---
license: apache-2.0
---
# GPyT Project
GPyT is a GPT2 model trained from scratch (not fine tuned) on Python code from Github. Overall, it was ~200GB of pure
Python code, the current GPyT model is a mere 2 epochs through this data, so it may benefit greatly from continued training and/or fine-tuning.
Newlines are replaced by <N>
Input to the model is code, up to the context length of 1024, with newlines replaced by <N>
Here's a quick example of using this model:
```py
from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("Reverb/GPyT")
model = AutoModelWithLMHead.from_pretrained("Reverb/GPyT")
# copy and paste some code in here
inp = """import"""
newlinechar = "<N>"
converted = inp.replace("\n", newlinechar)
tokenized = tokenizer.encode(converted, return_tensors='pt')
resp = model.generate(tokenized)
decoded = tokenizer.decode(resp[0])
reformatted = decoded.replace("<N>","\n")
print(reformatted)
```
Should produce:
```py
import numpy as np
import pytest
import pandas as pd<N
```
---
## The Journey
The model took 6 major steps which are:
1. Data Collection
2. Raw Data Cleaning
3. Data Preprocessing
4. Building & Training the Tokenizer
5. Testing the Model on Large Dataset
6. Deploying the Final Model on HuggingFace
#### Data Collection
The data was collected from python github repositories using web scraping techniques, It took nearly a day to gather 200GB worth of data.
#### Raw Data Cleaning
200GB of python code?? sounds ridiculous! that's why we needed to clean the downloaded repositories from any non-python files such as PDF,idx..etc
#### Data Preprocessing
I tried splitting the lines of code for each repository then merged them all under one single text file named **python_text_data.txt**
#### Building & Training the Tokenizer
For this step I have used **ByteLevelBPETokenizer** and trained it then saved the model on the desktop
#### Testing the Model on Large Dataset
After training the tokenizer on a large dataset, It was time for some tests to see how good is the model before proceeding.
---
## Considerations:
> - This model is intended for educational and research use only. Do not trust model outputs.
> - Model is highly likely to regurgitate code almost exactly as it saw it. It's up to you to determine licensing if you intend to actually use the generated code.
> - All Python code was blindly pulled from github. This means included code is both Python 2 and 3, among other more subtle differences, such as tabs being 2 spaces in some cases and 4 in others...and more non-homologous things.
> - Along with the above, this means the code generated could wind up doing or suggesting just about anything. Run the generated code at own risk...it could be anything
|