--- base_model: - manueldeprada/FactCC datasets: - divyapatel4/Microsoft-PeNS language: - en license: apache-2.0 base_model_relatin: - finetune --- # FactCC model for PENS dataset **The model has been fine-tuned on the PENS dataset to better adapt to the task of factuality assessment for news headlines.** Original paper: [Evaluating the Factual Consistency of Abstractive Text Summarization](https://arxiv.org/abs/1910.12840) PENS paper: [PENS: A Dataset and Generic Framework for Personalized News Headline Generation](https://aclanthology.org/2021.acl-long.7) Related paper: [Fact-Preserved Personalized News Headline Generation](https://ieeexplore.ieee.org/abstract/document/10415680) Example on how to calculate the FactCC score : ```python from transformers import BertForSequenceClassification, BertTokenizer model_path = 'THEATLAS/FactCC-PENS' tokenizer = BertTokenizer.from_pretrained(model_path) model = BertForSequenceClassification.from_pretrained(model_path) text='''The US has "passed the peak" on new coronavirus cases, the White House reported. They predict that some states would reopen this month. The US has over 637,000 confirmed Covid-19 cases and over 30,826 deaths, the highest for any country in the world.''' wrong_summary = '''The pandemic has almost not affected the US''' input_dict = tokenizer(text, wrong_summary, max_length=512, padding='max_length', truncation='only_first', return_tensors='pt') logits = model(**input_dict).logits probs = torch.nn.functional.softmax(logits, dim=1) fact_scores = probs[0][0].item() print(f"fact_scores: {fact_scores}") ``` --- **The following introduction is copied from the manueldeprada/FactCC repository.** This is a more modern implementation of the model and code from [the original github repo](https://github.com/salesforce/factCC) This model is trained to predict whether a summary is factual with respect to the original text. Basic usage: ```python from transformers import BertForSequenceClassification, BertTokenizer model_path = 'THEATLAS/FactCC-PENS' tokenizer = BertTokenizer.from_pretrained(model_path) model = BertForSequenceClassification.from_pretrained(model_path) text='''The US has "passed the peak" on new coronavirus cases, the White House reported. They predict that some states would reopen this month. The US has over 637,000 confirmed Covid-19 cases and over 30,826 deaths, the highest for any country in the world.''' wrong_summary = '''The pandemic has almost not affected the US''' input_dict = tokenizer(text, wrong_summary, max_length=512, padding='max_length', truncation='only_first', return_tensors='pt') logits = model(**input_dict).logits pred = logits.argmax(dim=1) model.config.id2label[pred.item()] # prints: INCORRECT ``` It can also be used with a pipeline. Beware that since pipelines are not thought to be used with pair of sentences, and you have to use this double-list hack: ```bash >>> from transformers import pipeline >>> pipe=pipeline(model="THEATLAS/FactCC-PENS") >>> pipe([[[text1,summary1]],[[text2,summary2]]],truncation='only_first',padding='max_length') # output [{'label': 'INCORRECT', 'score': 0.9979124665260315}, {'label': 'CORRECT', 'score': 0.879124665260315}] ```