|
--- |
|
language: en |
|
|
|
license: mit |
|
|
|
widget: |
|
- text: "COVID-19 is" |
|
example_title: "COVID" |
|
- text: "The NBA will" |
|
example_title: "NBA" |
|
- text: "Breaking news" |
|
example_title: "Breaking" |
|
--- |
|
|
|
# NewsGPT |
|
|
|
## Model Description |
|
The model is the [gpt2](https://huggingface.co/gpt2) fine-tuned to generate news. |
|
|
|
## Training Data |
|
The model's training data consists of ~13,000,000 English articles from ~90 outlets, which each consists of a headline (title) and a subheading (description). The articles were collected from the [Sciride News Mine](http://sciride.org/news.html), after which some additional cleaning was performed on the data, such as removing duplicate articles and removing repeated "outlet tags" appearing before or after headlines such as "| Daily Mail Online". |
|
|
|
The cleaned dataset can be found on huggingface [here](https://huggingface.co/datasets/AndyReas/frontpage-news). |
|
The data was repacked before training, to avoid abrupt truncation, which altered the order of the data a bit but it is ultimately the same sentences. |
|
|
|
## How to use |
|
The model can be used with the HuggingFace pipeline like so: |
|
```python |
|
>>> from transformers import pipeline |
|
>>> generator = pipeline('text-generation', model='andyreas/gennewsgpt') |
|
>>> generator("COVID-19 is", max_length=50, num_return_sequences=2) |
|
|
|
[{'generated_text': "COVID-19 is killing more people than the coronavirus. The number of people who have been infected has more than doubled in the past decade, according to a new analysis.The study of 2,000 people by the University of California.The study by"}, |
|
{'generated_text': "COVID-19 is the worst thing to happen in Canada: A new study. A new study suggests that the COVID-19 pandemic has become the \"best thing to happen in Canada.\". But the pandemic has also been a long-term challenge for"}] |
|
``` |
|
The model's config.json file includes default parameters for text-generation, which results in the same prompt producing different outputs. |
|
These can be overwritten to generate consistent outputs by setting "do_sample" = False, like so: |
|
|
|
```python |
|
>>> generator("COVID-19 is", do_sample=False) |
|
``` |
|
|
|
or increase variance by increasing the amount of words considered during sampling, like so: |
|
|
|
```python |
|
>>> generator("COVID-19 is", do_sample=True, top_k=50) |
|
``` |
|
|
|
## Training |
|
Training ran for 1 epoch using a learning rate of 2e-6 and 50K warm-up steps out of ~800K total steps. |
|
|
|
## Bias |
|
Like any other model, GenNewsGPT is subject to bias according to the data it was trained on. |
