README.md

---
language:
- en
license: apache-2.0
tags:
- generated_from_trainer
datasets:
- postbot/multi-emails-hq
metrics:
- accuracy
widget:
- text: 'Good Morning Professor Beans,

    Hope you are doing well. I just wanted to reach out and ask if differential calculus
    will be on the exam'
  example_title: email to prof
- text: 'Hey <NAME>,


    Thank you for signing up for my weekly newsletter. Before we get started, you''ll
    have to confirm your email address.'
  example_title: newsletter
- text: 'Hi <NAME>,


    I hope this email finds you well. I wanted to reach out and ask about office hours'
  example_title: office hours
- text: 'Greetings <NAME>,


    I hope you had a splendid evening at the Company sausage eating festival. I am
    reaching out because'
  example_title: festival
- text: 'Good Morning Harold,


    I was wondering when the next'
  example_title: event
- text: URGENT - I need the TPS reports
  example_title: URGENT
- text: 'Hi Archibald,


    I hope this email finds you extremely well.'
  example_title: emails that find you
- text: 'Hello there.


    I just wanted to reach out and check in to'
  example_title: checking in
- text: 'Hello <NAME>,


    I hope this email finds you well. I wanted to reach out and see if you''ve enjoyed
    your time with us'
  example_title: work well
- text: 'Hi <NAME>,


    I hope this email finds you well. I wanted to reach out and see if we could catch
    up'
  example_title: catch up
- text: I'm <NAME> and I just moved into the area and wanted to reach out and get
    some details on where I could get groceries and
  example_title: grocery
pipeline_tag: text-generation
base_model: EleutherAI/pythia-410m-deduped
model-index:
- name: multi-emails-hq-pythia-410m-deduped-r1
  results: []
---


# emailgen-pythia-410m-deduped

[![colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/pszemraj/94b0e6b95437896f800a65ae2e5f9ab4/emailgen-pythia-410m-deduped.ipynb
)


This model is a fine-tuned version of [EleutherAI/pythia-410m-deduped](https://huggingface.co/EleutherAI/pythia-410m-deduped) on email data.
It achieves the following results on the evaluation set:
- Loss: 2.1018
- Accuracy: 0.6157
- perplexity: 8.181

## Model description

- fine-tuned on dataset of emails for 4 epochs
- intended use: "text completion" of partially written emails


## Usage example



```python
from transformers import pipeline

model_tag = "postbot/emailgen-pythia-410m-deduped"
generator = pipeline(
    "text-generation",
    model=model_tag,
)

prompt = """
Hello, 

Following up on the bubblegum shipment."""

result = generator(
    prompt,
)  # generate
print(result[0]["generated_text"])
```

---



# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_postbot__emailgen-pythia-410m-deduped)

| Metric                | Value                     |
|-----------------------|---------------------------|
| Avg.                  | 26.65   |
| ARC (25-shot)         | 27.9          |
| HellaSwag (10-shot)   | 40.04    |
| MMLU (5-shot)         | 27.35         |
| TruthfulQA (0-shot)   | 38.2   |
| Winogrande (5-shot)   | 52.09   |
| GSM8K (5-shot)        | 0.0        |
| DROP (3-shot)         | 0.99         |