Update README.md
Browse filesUpdate evaluation results
README.md
CHANGED
@@ -14,13 +14,28 @@ tags:
|
|
14 |
|
15 |
Falcon-RW-1B-Instruct-OpenOrca is a 1B parameter, causal decoder-only model based on [Falcon-RW-1B](https://huggingface.co/tiiuae/falcon-rw-1b) and finetuned on the [Open-Orca/SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca) dataset.
|
16 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
17 |
**π Motivations**
|
18 |
-
1. To create a smaller, open-source, instruction-finetuned, ready-to-use model accessible for users with limited computational resources
|
19 |
2. To harness the strength of Falcon-RW-1B, a competitive model in its own right, and enhance its capabilities with instruction finetuning.
|
20 |
|
21 |
## π How to Use
|
22 |
|
23 |
-
|
24 |
|
25 |
### π Example Code
|
26 |
|
@@ -33,16 +48,16 @@ model = 'ericzzz/falcon-rw-1b-instruct-openorca'
|
|
33 |
|
34 |
tokenizer = AutoTokenizer.from_pretrained(model)
|
35 |
pipeline = transformers.pipeline(
|
36 |
-
|
37 |
model=model,
|
38 |
tokenizer=tokenizer,
|
39 |
torch_dtype=torch.bfloat16,
|
40 |
-
device_map=
|
41 |
)
|
42 |
|
43 |
-
system_message =
|
44 |
-
instruction =
|
45 |
-
prompt = f
|
46 |
|
47 |
response = pipeline(
|
48 |
prompt,
|
@@ -54,5 +69,5 @@ print(response[0]['generated_text'])
|
|
54 |
# AI, or Artificial Intelligence, refers to the ability of machines and software to perform tasks that require human intelligence, such as learning, reasoning, and problem-solving. It can be used in various fields like computer science, engineering, medicine, and more. Some common applications include image recognition, speech translation, and natural language processing.
|
55 |
```
|
56 |
|
57 |
-
## Contact
|
58 |
|
|
14 |
|
15 |
Falcon-RW-1B-Instruct-OpenOrca is a 1B parameter, causal decoder-only model based on [Falcon-RW-1B](https://huggingface.co/tiiuae/falcon-rw-1b) and finetuned on the [Open-Orca/SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca) dataset.
|
16 |
|
17 |
+
**π Evaluation Results**
|
18 |
+
|
19 |
+
Falcon-RW-1B-Instruct-OpenOrca is the #1 ranking model on [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) in ~1.5B parameters category!
|
20 |
+
|
21 |
+
| Metric | falcon-rw-1b-instruct-openorca | falcon-rw-1b |
|
22 |
+
|------------|-------------------------------:|-------------:|
|
23 |
+
| ARC | 34.56 | 35.07 |
|
24 |
+
| HellaSwag | 60.93 | 63.56 |
|
25 |
+
| MMLU | 28.77 | 25.28 |
|
26 |
+
| TruthfulQA | 37.42 | 35.96 |
|
27 |
+
| Winogrande | 60.69 | 62.04 |
|
28 |
+
| GSM8K | 1.21 | 0.53 |
|
29 |
+
| DROP | 21.94 | 4.64 |
|
30 |
+
| **Average**| **35.08** | **32.44** |
|
31 |
+
|
32 |
**π Motivations**
|
33 |
+
1. To create a smaller, open-source, instruction-finetuned, ready-to-use model accessible for users with limited computational resources (lower-end consumer GPUs).
|
34 |
2. To harness the strength of Falcon-RW-1B, a competitive model in its own right, and enhance its capabilities with instruction finetuning.
|
35 |
|
36 |
## π How to Use
|
37 |
|
38 |
+
The model operates with a structured prompt format, incorporating `<SYS>`, `<INST>`, and `<RESP>` tags to demarcate different parts of the input. The system message and instruction are placed within these tags, with the `<RESP>` tag triggering the model's response.
|
39 |
|
40 |
### π Example Code
|
41 |
|
|
|
48 |
|
49 |
tokenizer = AutoTokenizer.from_pretrained(model)
|
50 |
pipeline = transformers.pipeline(
|
51 |
+
'text-generation',
|
52 |
model=model,
|
53 |
tokenizer=tokenizer,
|
54 |
torch_dtype=torch.bfloat16,
|
55 |
+
device_map='auto',
|
56 |
)
|
57 |
|
58 |
+
system_message = 'You are a helpful assistant. Give short answers.'
|
59 |
+
instruction = 'What is AI? Give some examples.'
|
60 |
+
prompt = f'<SYS> {system_message} <INST> {instruction} <RESP> '
|
61 |
|
62 |
response = pipeline(
|
63 |
prompt,
|
|
|
69 |
# AI, or Artificial Intelligence, refers to the ability of machines and software to perform tasks that require human intelligence, such as learning, reasoning, and problem-solving. It can be used in various fields like computer science, engineering, medicine, and more. Some common applications include image recognition, speech translation, and natural language processing.
|
70 |
```
|
71 |
|
72 |
+
## π¬ Contact
|
73 |
+
For further inquiries or feedback, please contact at [email protected].
|