File size: 3,407 Bytes
dc66226 4ef057a 8856f7b 4ef057a 8856f7b 4402889 405ade6 e6354c6 405ade6 506ba7d 405ade6 506ba7d 405ade6 fe46f77 3099d8a 405ade6 506ba7d 405ade6 c44c487 fe46f77 405ade6 3099d8a c44c487 3099d8a c44c487 3099d8a fe46f77 4402889 405ade6 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 |
---
license: mit
datasets:
- squad_v2
- squad
language:
- en
library_name: transformers
pipeline_tag: question-answering
tags:
- deberta
- deberta-v3
- mdeberta
- question-answering
- squad
- squad_v2
model-index:
- name: sjrhuschlee/mdeberta-v3-base-squad2
results:
- task:
type: question-answering
name: Question Answering
dataset:
name: squad_v2
type: squad_v2
config: squad_v2
split: validation
metrics:
- type: exact_match
value: 80.3841
name: Exact Match
- type: f1
value: 83.8713
name: F1
- task:
type: question-answering
name: Question Answering
dataset:
name: squad
type: squad
config: plain_text
split: validation
metrics:
- type: exact_match
value: 83.7843
name: Exact Match
- type: f1
value: 90.9635
name: F1
---
# mdeberta-v3-base for Extractive QA
This is the [mdeberta-v3-base](https://huggingface.co/microsoft/mdeberta-v3-base) model, fine-tuned using the [SQuAD2.0](https://huggingface.co/datasets/squad_v2) dataset. It's been trained on question-answer pairs, including unanswerable questions, for the task of Extractive Question Answering.
## Overview
**Language model:** mdeberta-v3-base
**Language:** English
**Downstream-task:** Extractive QA
**Training data:** SQuAD 2.0
**Eval data:** SQuAD 2.0
**Infrastructure**: 1x NVIDIA T4
### Model Usage
```python
import torch
from transformers import(
AutoModelForQuestionAnswering,
AutoTokenizer,
pipeline
)
model_name = "sjrhuschlee/mdeberta-v3-base-squad2"
# a) Using pipelines
nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
qa_input = {
'question': 'Where do I live?',
'context': 'My name is Sarah and I live in London'
}
res = nlp(qa_input)
# {'score': 0.984, 'start': 30, 'end': 37, 'answer': ' London'}
# b) Load model & tokenizer
model = AutoModelForQuestionAnswering.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
question = 'Where do I live?'
context = 'My name is Sarah and I live in London'
encoding = tokenizer(question, context, return_tensors="pt")
start_scores, end_scores = model(
encoding["input_ids"],
attention_mask=encoding["attention_mask"],
return_dict=False
)
all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())
answer_tokens = all_tokens[torch.argmax(start_scores):torch.argmax(end_scores) + 1]
answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens))
# 'London'
```
### Metrics
```bash
# Squad v2
{
"eval_HasAns_exact": 77.7327935222672,
"eval_HasAns_f1": 84.7172608568916,
"eval_HasAns_total": 5928,
"eval_NoAns_exact": 83.02775441547519,
"eval_NoAns_f1": 83.02775441547519,
"eval_NoAns_total": 5945,
"eval_best_exact": 80.38406468457846,
"eval_best_exact_thresh": 0.0,
"eval_best_f1": 83.87129810154576,
"eval_best_f1_thresh": 0.0,
"eval_exact": 80.38406468457846,
"eval_f1": 83.87129810154582,
"eval_runtime": 221.4855,
"eval_samples": 12054,
"eval_samples_per_second": 54.423,
"eval_steps_per_second": 2.271,
"eval_total": 11873
}
# Squad
{
"eval_exact_match": 83.78429517502366,
"eval_f1": 90.96354972948996,
"eval_runtime": 197.0364,
"eval_samples": 10715,
"eval_samples_per_second": 54.381,
"eval_steps_per_second": 2.269
}
``` |