Update README.md
Browse files
README.md
CHANGED
@@ -6,9 +6,9 @@ license: apache-2.0
|
|
6 |
|
7 |
<!-- Provide a quick summary of what the model is/does. -->
|
8 |
|
9 |
-
|
10 |
|
11 |
-
|
12 |
|
13 |
|
14 |
### Benchmark Tests
|
@@ -16,12 +16,12 @@ DRAGON models are fine-tuned with high-quality custom instruct datasets, designe
|
|
16 |
Evaluated against the benchmark test: [RAG-Instruct-Benchmark-Tester](https://www.huggingface.co/datasets/llmware/rag_instruct_benchmark_tester)
|
17 |
Average of 2 Test Runs with 1 point for correct answer, 0.5 point for partial correct or blank / NF, 0.0 points for incorrect, and -1 points for hallucinations.
|
18 |
|
19 |
-
--**Accuracy Score**: **
|
20 |
-
--Not Found Classification:
|
21 |
-
--Boolean:
|
22 |
-
--Math/Logic:
|
23 |
-
--Complex Questions (1-5):
|
24 |
-
--Summarization Quality (1-5):
|
25 |
--Hallucinations: No hallucinations observed in test runs.
|
26 |
|
27 |
For test run results (and good indicator of target use cases), please see the files ("core_rag_test" and "answer_sheet" in this repo).
|
@@ -31,32 +31,23 @@ For test run results (and good indicator of target use cases), please see the fi
|
|
31 |
<!-- Provide a longer summary of what this model is. -->
|
32 |
|
33 |
- **Developed by:** llmware
|
34 |
-
- **Model type:**
|
35 |
- **Language(s) (NLP):** English
|
36 |
- **License:** Apache 2.0
|
37 |
-
- **Finetuned from model:**
|
38 |
|
39 |
## Uses
|
40 |
|
41 |
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
|
42 |
|
43 |
-
The intended use of DRAGON models is two-fold:
|
44 |
-
|
45 |
-
1. Provide high-quality RAG-Instruct models designed for fact-based, no "hallucination" question-answering in connection with an enterprise RAG workflow.
|
46 |
-
|
47 |
-
2. DRAGON models are fine-tuned on top of leading base foundation models, generally in the 6-7B+ range, and purposefully rolled-out across multiple base models to provide choices and "drop-in" replacements for RAG specific use cases.
|
48 |
-
|
49 |
-
3. DRAGON models were trained on the same principles as the BLING models, so generally, it should be easy to "upgrade" from a BLING model in testing to a DRAGON model in production.
|
50 |
-
|
51 |
-
|
52 |
### Direct Use
|
53 |
|
54 |
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
|
55 |
|
56 |
-
|
57 |
legal and regulatory industries with complex information sources.
|
58 |
|
59 |
-
|
60 |
without the need for a lot of complex instruction verbiage - provide a text passage context, ask questions, and get clear fact-based responses.
|
61 |
|
62 |
|
@@ -72,8 +63,8 @@ Any model can provide inaccurate or incomplete information, and should be used i
|
|
72 |
The fastest way to get started with dRAGon is through direct import in transformers:
|
73 |
|
74 |
from transformers import AutoTokenizer, AutoModelForCausalLM
|
75 |
-
tokenizer = AutoTokenizer.from_pretrained("
|
76 |
-
model = AutoModelForCausalLM.from_pretrained("
|
77 |
|
78 |
Please refer to the generation_test .py files in the Files repository, which includes 200 samples and script to test the model. The **generation_test_llmware_script.py** includes built-in llmware capabilities for fact-checking, as well as easy integration with document parsing and actual retrieval to swap out the test set for RAG workflow consisting of business documents.
|
79 |
|
|
|
6 |
|
7 |
<!-- Provide a quick summary of what the model is/does. -->
|
8 |
|
9 |
+
bling-tiny-llama-v0 is part of the BLING ("Best Little Instruct No-GPU-required...") model series, RAG-instruct trained on top of a TinyLlama-1.1b base model.
|
10 |
|
11 |
+
BLING models are fine-tuned with high-quality custom instruct datasets, designed for rapid testing and prototyping in RAG scenarios.
|
12 |
|
13 |
|
14 |
### Benchmark Tests
|
|
|
16 |
Evaluated against the benchmark test: [RAG-Instruct-Benchmark-Tester](https://www.huggingface.co/datasets/llmware/rag_instruct_benchmark_tester)
|
17 |
Average of 2 Test Runs with 1 point for correct answer, 0.5 point for partial correct or blank / NF, 0.0 points for incorrect, and -1 points for hallucinations.
|
18 |
|
19 |
+
--**Accuracy Score**: **86.0** correct out of 100
|
20 |
+
--Not Found Classification: 85.0%
|
21 |
+
--Boolean: 85.0%
|
22 |
+
--Math/Logic: 37.25%
|
23 |
+
--Complex Questions (1-5): 3 (Medium-High: multiple choice, table reading, causal)
|
24 |
+
--Summarization Quality (1-5): 3 (Coherent, extractive)
|
25 |
--Hallucinations: No hallucinations observed in test runs.
|
26 |
|
27 |
For test run results (and good indicator of target use cases), please see the files ("core_rag_test" and "answer_sheet" in this repo).
|
|
|
31 |
<!-- Provide a longer summary of what this model is. -->
|
32 |
|
33 |
- **Developed by:** llmware
|
34 |
+
- **Model type:** TinyLlama
|
35 |
- **Language(s) (NLP):** English
|
36 |
- **License:** Apache 2.0
|
37 |
+
- **Finetuned from model:** TinyLlama-1.1b
|
38 |
|
39 |
## Uses
|
40 |
|
41 |
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
|
42 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
43 |
### Direct Use
|
44 |
|
45 |
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
|
46 |
|
47 |
+
BLING is designed for enterprise automation use cases, especially in knowledge-intensive industries, such as financial services,
|
48 |
legal and regulatory industries with complex information sources.
|
49 |
|
50 |
+
BLING models have been trained for common RAG scenarios, specifically: question-answering, key-value extraction, and basic summarization as the core instruction types
|
51 |
without the need for a lot of complex instruction verbiage - provide a text passage context, ask questions, and get clear fact-based responses.
|
52 |
|
53 |
|
|
|
63 |
The fastest way to get started with dRAGon is through direct import in transformers:
|
64 |
|
65 |
from transformers import AutoTokenizer, AutoModelForCausalLM
|
66 |
+
tokenizer = AutoTokenizer.from_pretrained("bling-tiny-llama-v0", trust_remote_code=True)
|
67 |
+
model = AutoModelForCausalLM.from_pretrained("bling-tiny-llama-v0", trust_remote_code=True)
|
68 |
|
69 |
Please refer to the generation_test .py files in the Files repository, which includes 200 samples and script to test the model. The **generation_test_llmware_script.py** includes built-in llmware capabilities for fact-checking, as well as easy integration with document parsing and actual retrieval to swap out the test set for RAG workflow consisting of business documents.
|
70 |
|