Edit model card

๐Ÿฎ ๐Ÿฆ™ Flan-Alpaca: Instruction Tuning from Humans and Machines

Thanks to declare-lab for the training repository, contains code for extending the Stanford Alpaca synthetic instruction tuning to existing instruction-tuned models such as Flan-T5. The pretrained models and demos are available on HuggingFace ๐Ÿค— :

Model Parameters Training GPUs
Flan-Alpaca-Base 220M 1x A6000
Flan-Alpaca-Large 770M 1x A6000
Flan-Alpaca-XL 3B 1x A6000
Flan-Alpaca-XXL 11B 4x A6000 (FSDP)
Flan-Alpaca-UL2 20B 4x A100 (80G) (FSDP)

Why?

Alpaca represents an exciting new direction to approximate the performance of large language models (LLMs) like ChatGPT cheaply and easily. Concretely, they leverage an LLM such as GPT-3 to generate instructions as synthetic training data. The synthetic data which covers more than 50k tasks can then be used to finetune a smaller model. However, the original implementation is less accessible due to licensing constraints of the underlying LLaMA model. Furthermore, users have noted potential noise in the synthetic dataset. Hence, it may be better to explore a fully accessible model that is already trained on high-quality (but less diverse) instructions such as Flan-T5.

Usage

from transformers import pipeline

prompt = "Write an email about an alpaca that likes flan"
model = pipeline(model="0-hero/flan-alpaca-ul2")
model(prompt, max_length=128, do_sample=True)

Readme forked from declare-lab/flan-alpaca-xxl

Downloads last month
7
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Dataset used to train 0-hero/flan-alpaca-ul2