|
--- |
|
license: mit |
|
datasets: |
|
- monsoon-nlp/asknyc-chatassistant-format |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
tags: |
|
- nyc |
|
- reddit |
|
--- |
|
# GPT-NYC |
|
|
|
## About |
|
|
|
GPT2-Medium fine-tuned on questions and responses from https://reddit.com/r/asknyc |
|
|
|
**2023 Update: try a larger model: [monsoon-nlp/nyc-savvy-llama2-7b](https://huggingface.co/monsoon-nlp/nyc-savvy-llama2-7b)** |
|
|
|
I filtered comments to ones with scores >= 3, and responding directly |
|
to the original post ( = ignoring responses to other commenters). |
|
|
|
I added tokens to match NYC neighborhoods, subway stations, foods, and other |
|
common terms in the original batches of questions and comments. |
|
You would be surprised what is missing from GPT tokens! |
|
|
|
Try prompting with ```question? %% ``` or ```question? - more info %%``` |
|
|
|
## Status |
|
|
|
I would like to continue by: |
|
- fine-tuning GPT2-Large with a larger dataset of questions |
|
- examining bias and toxicity |
|
- examining memorization vs. original responses |
|
- releasing a reusable benchmark |
|
|
|
## Blog |
|
|
|
https://mapmeld.medium.com/gpt-nyc-part-1-9cb698b2e3d |
|
|
|
## Notebooks |
|
|
|
### Data processing / new tokens |
|
|
|
https://colab.research.google.com/drive/13BOw0uekoAYB4jjQtaXTn6J_VHatiRLu |
|
|
|
### Fine-tuning GPT2 (small) |
|
|
|
https://colab.research.google.com/drive/1FnXcAh4H-k8dAzixkV5ieygV96ePh3lR |
|
|
|
### Fine-tuning GPT2-Medium |
|
|
|
Same code as small, but on Google Cloud to use an A100 GPU |
|
|
|
### Predictive text and probabilities |
|
|
|
Scroll to end of |
|
|
|
https://colab.research.google.com/drive/1FnXcAh4H-k8dAzixkV5ieygV96ePh3lR |
|
|
|
to see how to install git-lfs and trick ecco into loading this. |
