File size: 4,369 Bytes
9e83889 770b96a e2bb3ae 9e83889 e2bb3ae 34f2d7f 74b1344 e2bb3ae e2a8741 2a6e06d 9bf71c1 38ffd7b 9bf71c1 2a6e06d 6128cc9 0caa7af e2bb3ae 770b96a 74b1344 e2bb3ae |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 |
---
library_name: peft
license: wtfpl
language:
- en
pipeline_tag: text-generation
---
## Model description
The togethercomputer/RedPajama-INCITE-Base-3B-v1 model finetuned for `Summary` and `Topic` generation from a dailogue. We use a sample of roughly 1000 data points from the
[Dialogsum](https://github.com/cylnlp/dialogsum) dataset for fine-tuning.
Look at the repo [llm-toys](https://github.com/kuutsav/llm-toys) for usage and other details.
Try in colab:
<a target="_blank" href="https://colab.research.google.com/drive/1MSl8IDLjs3rgEv8cPHbJLR8GHh2ucT3_">
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>
## Installation
```bash
pip install llm-toys
```
```python
from llm_toys.tasks import SummaryAndTopicGenerator
summary_topic_generator = SummaryAndTopicGenerator()
summary_topic_generator.generate_summary_and_topic(
"""
#Person1#: I'm so excited for the premiere of the latest Studio Ghibli movie!
#Person2#: What's got you so hyped?
#Person1#: Studio Ghibli movies are pure magic! The animation, storytelling, everything is incredible.
#Person2#: Which movie is it?
#Person1#: It's called "Whisper of the Wind." It's about a girl on a magical journey to save her village.
#Person2#: Sounds amazing! I'm in for the premiere.
#Person1#: Great! We're in for a visual masterpiece and a heartfelt story.
#Person2#: Can't wait to be transported to their world.
#Person1#: It'll be an unforgettable experience, for sure!
""".strip()
)
# {"summary": "#Person1# is excited for the premiere of the latest Studio Ghibli movie.
# #Person1# thinks the animation, storytelling, and heartfelt story will be unforgettable.
# #Person2# is also excited for the premiere.",
# "topic": "Studio ghibli movie"}
```
## Sample training data
```json
{
"fname": "train_664",
"dialogue": "#Person1#: Hello, Happy Time Catering Services, Vitoria speaking. How can I help you?\n#Person2#: Hello, Victoria. This is Joe Smith from country holidays. I wondered if you could do some catering for us next week, we are having a small reception. It's to launch our summer holiday advertising campaign. Will you be free?\n#Person1#: When exactly is it? Mr. Smith?\n#Person2#: April 21st, that's Thursday. Oh, sorry, no. It should be Friday.\n#Person1#: Oh, yes I can do that where will you be holding it?\n#Person2#: We thought we'd have that at head office and use the conference room, because there is enough room for everyone there.\n#Person1#: Ok. What sort of things would you like?\n#Person2#: Just a light lunch I think, so that people can eat while they move around and talk to each other. You did some thing similar for us last year. We'd be happy to have the same menu again.\n#Person1#: Right. I'll look at my diary and see what you had last time. Oh, I nearly forgot to ask you how many should I cater for?\n#Person2#: Well, I think most people will be able to come, perhaps around 30. No, let's say 35, to be sure.\n#Person1#: Right, thank you for getting in touch, Mr. Smith. I'll send you confirmation of the arrangements by the end of this week.\n#Person2#: Ok.",
"summary": "Joe Smith calls Happy Time Catering Service and wants some catering for next week. Victoria asks his requirements and will send him confirmation of the arrangements by the end of this week.",
"topic": "Catering service"
}
```
## Training params
```json
{
"batch_size": 1,
"eval_ratio": 0.05,
"eval_steps": 100,
"gradient_accumulation_steps": 4,
"learning_rate": 0.0001,
"logging_steps": 100,
"lora_alpha": 32,
"lora_dropout": 0.05,
"lora_r": 16,
"max_length": 1024,
"model_name": "togethercomputer/RedPajama-INCITE-Base-3B-v1",
"num_train_epochs": 2,
"seed": 0,
"task_type": "dialogue_summary_topic",
"use_aim": True
}
```
## Training curve
## Training procedure
The following `bitsandbytes` quantization config was used during training:
- load_in_8bit: False
- load_in_4bit: True
- llm_int8_threshold: 6.0
- llm_int8_skip_modules: None
- llm_int8_enable_fp32_cpu_offload: False
- llm_int8_has_fp16_weight: False
- bnb_4bit_quant_type: nf4
- bnb_4bit_use_double_quant: True
- bnb_4bit_compute_dtype: bfloat16
### Framework versions
- PEFT 0.4.0.dev0 |