File size: 18,585 Bytes
5f8b6f4 0f31be1 5f8b6f4 0f31be1 5f8b6f4 36fabc4 5f8b6f4 0f31be1 5f8b6f4 2923dd2 5f8b6f4 2923dd2 5f8b6f4 cc2cc6e 5f8b6f4 36fabc4 5f8b6f4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 |
---
license: apache-2.0
language:
- en
metrics:
- f1
- precision
- recall
tags:
- NER
- information extraction
- relation extraction
- summarization
- sentiment extraction
- question-answering
pipeline_tag: token-classification
library_name: gliner
datasets:
- knowledgator/GLINER-multi-task-synthetic-data
---
🚀 Meet the first multi-task prompt-tunable GLiNER model 🚀
**GLiNER-Multitask** is a model designed to extract various pieces of information from plain text based on a user-provided custom prompt. This versatile model leverages a bidirectional transformer encoder, similar to BERT, which ensures both high generalization and compute efficiency despite its compact size.
The `gliner-multitask-v1.0` variant achieves state-of-the-art performance on NER zero-shot benchmarks, demonstrating its robustness and flexibility. It excels not only in named entity recognition but also in handling various other information extraction tasks, making it a powerful tool for diverse natural language processing applications.
### Supported tasks:
* **Named Entity Recognition (NER)**: Identifies and categorizes entities such as names, organizations, dates, and other specific items in the text.
* **Relation Extraction**: Detects and classifies relationships between entities within the text.
* **Summarization**: Extract the most important sentences that summarize the input text, capturing the essential information.
* **Sentiment Extraction**: Identify parts of the text that signalize a positive, negative, or neutral sentiment;
* **Key-Phrase Extraction**: Identifies and extracts important phrases and keywords from the text.
* **Question-answering**: Finding an answer in the text given a question;
* **Open Information Extraction**: Extracts pieces of text given an open prompt from a user, for example, product description extraction;
* **Text classification**: Classifying text by matching labels specified in the prompt;
### Installation
To use this model, you must install the [GLiNER Python library](https://github.com/urchade/GLiNER):
```bash
pip install gliner
```
Once you've downloaded the GLiNER library, you can import the GLiNER class. You can then load this model using GLiNER.from_pretrained.
**How to use for NER:**
```python
from gliner import GLiNER
model = GLiNER.from_pretrained("knowledgator/gliner-multitask-v1.0")
text = """
Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975 to develop and sell BASIC interpreters for the Altair 8800. During his career at Microsoft, Gates held the positions of chairman, chief executive officer, president and chief software architect, while also being the largest individual shareholder until May 2014.
"""
labels = ["founder", "computer", "software", "position", "date"]
entities = model.predict_entities(text, labels)
for entity in entities:
print(entity["text"], "=>", entity["label"])
```
### Performance:
| Model | Dataset | Precision | Recall | F1 Score | F1 Score (Decimal) |
|------------------------------------|--------------------|-----------|--------|----------|--------------------|
| knowledgator/gliner-multitask-v0.5 | CrossNER_AI | 51.00% | 51.11% | 51.05% | 0.5105 |
| | CrossNER_literature | 72.65% | 65.62% | 68.96% | 0.6896 |
| | CrossNER_music | 74.91% | 73.70% | 74.30% | 0.7430 |
| | CrossNER_politics | 78.84% | 77.71% | 78.27% | 0.7827 |
| | CrossNER_science | 69.20% | 65.48% | 67.29% | 0.6729 |
| | mit-movie | 61.29% | 52.59% | 56.60% | 0.5660 |
| | mit-restaurant | 50.65% | 38.13% | 43.51% | 0.4351 |
| | **Average** | | | | **0.6276** |
| knowledgator/gliner-multitask-v1.0 | CrossNER_AI | 67.15% | 56.10% | 61.13% | 0.6113 |
| | CrossNER_literature | 71.60% | 64.74% | 68.00% | 0.6800 |
| | CrossNER_music | 73.57% | 69.29% | 71.36% | 0.7136 |
| | CrossNER_politics | 77.54% | 76.52% | 77.03% | 0.7703 |
| | CrossNER_science | 74.54% | 66.00% | 70.01% | 0.7001 |
| | mit-movie | 61.86% | 42.02% | 50.04% | 0.5004 |
| | mit-restaurant | 58.87% | 36.67% | 45.19% | 0.4519 |
| | **Average** | | | | **0.6325** |
| knowledgator/gliner-llama-multitask-1B-v1.0 | CrossNER_AI | 63.24% | 55.60% | 59.17% | 0.5917 |
| | CrossNER_literature | 69.74% | 60.10% | 64.56% | 0.6456 |
| | CrossNER_music | 74.03% | 67.22% | 70.46% | 0.7046 |
| | CrossNER_politics | 76.96% | 71.64% | 74.20% | 0.7420 |
| | CrossNER_science | 73.79% | 63.73% | 68.39% | 0.6839 |
| | mit-movie | 56.89% | 46.70% | 51.30% | 0.5130 |
| | mit-restaurant | 48.45% | 38.13% | 42.67% | 0.4267 |
| | **Average** | | | | **0.6153** |
---
**How to use for relation extraction:**
```python
text = """
Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975 to develop and sell BASIC interpreters for the Altair 8800. During his career at Microsoft, Gates held the positions of chairman, chief executive officer, president and chief software architect, while also being the largest individual shareholder until May 2014.
"""
labels = ["Microsoft <> founder", "Microsoft <> inception date", "Bill Gates <> held position"]
entities = model.predict_entities(text, labels)
for entity in entities:
print(entity["label"], "=>", entity["text"])
```
### Construct relations extraction pipeline with [utca](https://github.com/Knowledgator/utca)
First of all, we need import neccessary components of the library and initalize predictor - GLiNER model and construct pipeline that combines NER and realtions extraction:
```python
from utca.core import RenameAttribute
from utca.implementation.predictors import (
GLiNERPredictor,
GLiNERPredictorConfig
)
from utca.implementation.tasks import (
GLiNER,
GLiNERPreprocessor,
GLiNERRelationExtraction,
GLiNERRelationExtractionPreprocessor,
)
predictor = GLiNERPredictor( # Predictor manages the model that will be used by tasks
GLiNERPredictorConfig(
model_name = "knowledgator/gliner-multitask-v1.0", # Model to use
device = "cuda:0", # Device to use
)
)
pipe = (
GLiNER( # GLiNER task produces classified entities that will be at the "output" key.
predictor=predictor,
preprocess=GLiNERPreprocessor(threshold=0.7) # Entities threshold
)
| RenameAttribute("output", "entities") # Rename output entities from GLiNER task to use them as inputs in GLiNERRelationExtraction
| GLiNERRelationExtraction( # GLiNERRelationExtraction is used for relation extraction.
predictor=predictor,
preprocess=(
GLiNERPreprocessor(threshold=0.5) # Relations threshold
| GLiNERRelationExtractionPreprocessor()
)
)
)
```
To run pipeline we need to specify entity types and relations with their parameters:
```python
r = pipe.run({
"text": text, # Text to process
"labels": ["organisation", "founder", "position", "date"],
"relations": [{ # Relation parameters
"relation": "founder", # Relation label. Required parameter.
"pairs_filter": [("organisation", "founder")], # Optional parameter. It specifies possible members of relations by their entity labels.
"distance_threshold": 100, # Optional parameter. It specifies the max distance between spans in the text (i.e., the end of the span that is closer to the start of the text and the start of the next one).
}, {
"relation": "inception date",
"pairs_filter": [("organisation", "date")],
}, {
"relation": "held position",
"pairs_filter": [("founder", "position")],
}]
})
print(r["output"])
```
### Performance:
| Model | Dataset | Precision | Recall | F1 Score |
|:-----------------------|------------:|---------:|-----------:|-----------:|
| knowledgator/gliner-llama-multitask-1B-v1.0 | CrossRe | 0.606472 | 0.511444 | 0.554919 |
| | DocRed | 0.707483 | 0.589355 | 0.643039 |
| knowledgator/gliner-multitask-v0.5 | CrossRe | 0.585319 | 0.800176 | 0.676088 |
| | DocRed | 0.713392 | 0.772826 | 0.74192 |
|knowledgator/gliner-multitask-v1.0 | CrossRe | 0.760653 | 0.738556 | 0.749442 |
| | DocRed | 0.770644 | 0.761373 | 0.76598 |
---
**How to use for open information extraction:**
```python
prompt = """Find all positive aspects about the product:\n"""
text = """
I recently purchased the Sony WH-1000XM4 Wireless Noise-Canceling Headphones from Amazon and I must say, I'm thoroughly impressed. The package arrived in New York within 2 days, thanks to Amazon Prime's expedited shipping.
The headphones themselves are remarkable. The noise-canceling feature works like a charm in the bustling city environment, and the 30-hour battery life means I don't have to charge them every day. Connecting them to my Samsung Galaxy S21 was a breeze, and the sound quality is second to none.
I also appreciated the customer service from Amazon when I had a question about the warranty. They responded within an hour and provided all the information I needed.
However, the headphones did not come with a hard case, which was listed in the product description. I contacted Amazon, and they offered a 10% discount on my next purchase as an apology.
Overall, I'd give these headphones a 4.5/5 rating and highly recommend them to anyone looking for top-notch quality in both product and service.
"""
input_ = prompt+text
labels = ["match"]
matches = model.predict_entities(input_, labels)
for match in matches:
print(match["text"], "=>", match["score"])
```
### Performance:
*Dataset: WiRe57_343-manual-oie*
| Model | Precision | Recall | F1 Score |
|:-----------------------|------------:|---------:|-----------:|
| knowledgator/gliner-llama-multitask-1B-v1.0 | 0.9047 | 0.2794 | 0.4269 |
| knowledgator/gliner-multitask-v0.5 | 0.9278 | 0.2779 | 0.4287 |
| knowledgator/gliner-multitask-v1.0 | 0.8775 | 0.2733 | 0.4168 |
---
**How to use for question-answering:**
```python
question = "Who was the CEO of Microsoft?"
text = """
Microsoft was founded by Bill Gates and Paul Allen on April 4, 1975, to develop and sell BASIC interpreters for the Altair 8800. During his career at Microsoft, Gates held the positions of chairman, chief executive officer, president and chief software architect, while also being the largest individual shareholder until May 2014.
"""
labels = ["answer"]
input_ = question+text
answers = model.predict_entities(input_, labels)
for answer in answers:
print(answer["text"], "=>", answer["score"])
```
### Performance:
*Dataset: SQuAD 2.0*
| Model | Precision | Recall | F1 Score |
|:-----------------------|------------:|---------:|-----------:|
| knowledgator/gliner-llama-multitask-1B-v1.0 | 0.578296 | 0.795821 | 0.669841 |
| knowledgator/gliner-multitask-v0.5 | 0.429213 | 0.94378 | 0.590072 |
| knowledgator/gliner-multitask-v1.0 | 0.601354 | 0.874784 | 0.712745 |
---
**How to use for summarization:**
With threshold parameters, you can control how much information you want to extract.
```python
prompt = "Summarize the given text, highlighting the most important information:\n"
text = """
Several studies have reported its pharmacological activities, including anti-inflammatory, antimicrobial, and antitumoral effects.
The effect of E-anethole was studied in the osteosarcoma MG-63 cell line, and the antiproliferative activity was evaluated by an MTT assay.
It showed a GI50 value of 60.25 μM with apoptosis induction through the mitochondrial-mediated pathway. Additionally, it induced cell cycle arrest at the G0/G1 phase, up-regulated the expression of p53, caspase-3, and caspase-9, and down-regulated Bcl-xL expression.
Moreover, the antitumoral activity of anethole was assessed against oral tumor Ca9-22 cells, and the cytotoxic effects were evaluated by MTT and LDH assays.
It demonstrated a LD50 value of 8 μM, and cellular proliferation was 42.7% and 5.2% at anethole concentrations of 3 μM and 30 μM, respectively.
It was reported that it could selectively and in a dose-dependent manner decrease cell proliferation and induce apoptosis, as well as induce autophagy, decrease ROS production, and increase glutathione activity. The cytotoxic effect was mediated through NF-kB, MAP kinases, Wnt, caspase-3 and -9, and PARP1 pathways. Additionally, treatment with anethole inhibited cyclin D1 oncogene expression, increased cyclin-dependent kinase inhibitor p21WAF1, up-regulated p53 expression, and inhibited the EMT markers.
"""
labels = ["summary"]
input_ = prompt+text
threshold = 0.1
summaries = model.predict_entities(input_, labels, threshold=threshold)
for summary in summaries:
print(summary["text"], "=>", summary["score"])
```
---
**How to use for text classification:**
With threshold parameters, you can control recall and precision of text classification.
```python
prompt = "Classify text into the following classes: positive review, negative review"
text = """
"I recently purchased the Sony WH-1000XM4 Wireless Noise-Canceling Headphones from Amazon and I must say, I'm thoroughly impressed. The package arrived in New York within 2 days, thanks to Amazon Prime's expedited shipping.
"""
labels = ["match"]
input_ = prompt+text
threshold = 0.5
classes = model.predict_entities(input_, labels, threshold=threshold)
for label in classes:
print(label["text"], "=>", label["score"])
```
### Performance:
| Model Name | Dataset | Micro F1 Score |
|-----------------------|-----------|----------------|
| knowledgator/gliner-multitask-v1.0 | Emotion | 0.322 |
| | AG News | 0.7436 |
| | IMDb | 0.7907 |
| knowledgator/gliner-llama-multitask-1B-v1.0 | Emotion | 0.3475 |
| | AG News | 0.7436 |
| | IMDb | 0.7907 |
---
### Extensive NER Benchmarks:
Our multitask model demonstrates comparable performance on different zero-shot benchmarks to dedicated models to NER task (all labels were lowecased in this testing):
| Dataset | Precision | Recall | F1 Score | F1 Score (Decimal) |
|------------------------|-----------|--------|----------|--------------------|
| ACE 2004 | 53.25% | 23.20% | 32.32% | 0.3232 |
| ACE 2005 | 43.25% | 18.00% | 25.42% | 0.2542 |
| AnatEM | 51.75% | 25.98% | 34.59% | 0.3459 |
| Broad Tweet Corpus | 69.54% | 72.50% | 70.99% | 0.7099 |
| CoNLL 2003 | 68.33% | 68.43% | 68.38% | 0.6838 |
| CrossNER_AI | 67.15% | 56.10% | 61.13% | 0.6113 |
| CrossNER_literature | 71.60% | 64.74% | 68.00% | 0.6800 |
| CrossNER_music | 73.57% | 69.29% | 71.36% | 0.7136 |
| CrossNER_politics | 77.54% | 76.52% | 77.03% | 0.7703 |
| CrossNER_science | 74.54% | 66.00% | 70.01% | 0.7001 |
| FabNER | 69.28% | 62.62% | 65.78% | 0.6578 |
| FindVehicle | 49.75% | 51.25% | 50.49% | 0.5049 |
| GENIA_NER | 60.98% | 46.91% | 53.03% | 0.5303 |
| HarveyNER | 24.27% | 35.66% | 28.88% | 0.2888 |
| MultiNERD | 54.33% | 89.34% | 67.57% | 0.6757 |
| Ontonotes | 27.26% | 36.64% | 31.26% | 0.3126 |
| PolyglotNER | 33.54% | 64.29% | 44.08% | 0.4408 |
| TweetNER7 | 44.77% | 38.67% | 41.50% | 0.4150 |
| WikiANN en | 56.33% | 57.09% | 56.71% | 0.5671 |
| WikiNeural | 71.70% | 86.60% | 78.45% | 0.7845 |
| bc2gm | 64.71% | 51.68% | 57.47% | 0.5747 |
| bc4chemd | 69.24% | 50.08% | 58.12% | 0.5812 |
| bc5cdr | 79.22% | 69.19% | 73.87% | 0.7387 |
| mit-movie | 61.86% | 42.02% | 50.04% | 0.5004 |
| mit-restaurant | 58.87% | 36.67% | 45.19% | 0.4519 |
| ncbi | 68.72% | 54.86% | 61.01% | 0.6101 |
### Join Our Discord
Connect with our community on Discord for news, support, and discussion about our models. Join [Discord](https://discord.gg/dkyeAgs9DG).
### Citation:
```
@misc{stepanov2024gliner,
title={GLiNER multi-task: Generalist Lightweight Model for Various Information Extraction Tasks},
author={Ihor Stepanov and Mykhailo Shtopko},
year={2024},
eprint={2406.12925},
archivePrefix={arXiv},
primaryClass={id='cs.LG' full_name='Machine Learning' is_active=True alt_name=None in_archive='cs' is_general=False description='Papers on all aspects of machine learning research (supervised, unsupervised, reinforcement learning, bandit problems, and so on) including also robustness, explanation, fairness, and methodology. cs.LG is also an appropriate primary category for applications of machine learning methods.'}
}
``` |