Pub-Guard-Llama-8B

Pub-Guard-Llama-8B is a fine-tuned version of the Llama-3.1-8B model, specifically designed for detecting fraudulent papers in academic publications.

Benefits of using this model:

• the first LLM specifically designed for fraud detection in scientific articles
• integration of external resources for better analysis (Semantic Scholar, OpenAlex, Pubmed...)
• it offers uncertainty-aware predictions with faithful explanations

Datasets

• Pubmed Retraction
• Instruction

Zero-Shot Reasoning

Model	Breast Cancer	Lung Cancer
Llama-3.1-8B-Instruct	20.0	46.2
OpenScholar-8B	18.8	20.3
Bio-Medical-Llama-3-8B	40.7	56.7
PMC-LLaMA-13B	36.4	-
Pub-Guard-Llama-8B (Ours)	69.5	79.3
Pub-Guard-Llama-8B (RAG)	-	-
Pub-Guard-Llama-8B (Debate)	-	-

Coherence Check

• Model: cross-encoder/nli-deberta-v3-base
• Article -> (Sentence in Explanation)
• Instruction Data (gpt-4o): 88.7 (train set)
• Predicted Data (pub-guard-llama): 90.8 (lung cancer set)

Examples

Naive Mode

Input:

Title: Changes and Influencing Factors of Cognitive Impairment in Patients with Breast Cancer.

Abstract: To investigate the changes in cognitive function and its influencing factors in patients with breast cancer after chemotherapy,
to provide a scientific basis for further cognitive correction therapy. In this study, general information on age,
marital status, and chemotherapy regimen was collected from 172 breast cancer chemotherapy patients.
172 patients with breast cancer undergoing chemotherapy were investigated by convenience sampling method,
and the subjects were tested one-on-one using the Chinese version of the MATRICS Consensus Cognitive Battery (MCCB) computer system.
...

Authors: Huixia Cui (author h-index: 6, Early Career Researcher);
Xiaoxiu Song (author h-index: 1, Emerging Researcher); Wenlu Zhang (author h-index: 7, Early Career Researcher)

Institutions: College of Nursing, Jinzhou Medical University, Jinzhou, Liaoning 121001, China. (institution average citation: 9.0, Emerging Institution);
Department of Intensive Care Medicine, Liaocheng People's Hospital, Liaocheng, Shandong 252000, China. (institution average citation: 10.0, Emerging Institution);

Journal: evidence-based complementary and alternative medicine : ecam (unknown journal)

Output:

Yes
The article should be retracted due to potential data fabrication or manipulation, as the reported linear regression results lack clarity and statistical validation.
 Additionally, the journal's reputation and peer review rigor are questionable, and the authors' affiliations are emerging with low citation averages,
raising concerns about research reliability.

Debate Mode

Input:

Title: WITHDRAWN: TWIST2 inhibits EMT and induces oxidative stress in lung cancer cells by regulating the FGF21-mediated AMPK/mTOR pathway.

Abstract: This article has been withdrawn: please see Elsevier Policy on Article Withdrawal (https://www.elsevier.com/about/policies/article-withdrawal). This Expression of Concern has been withdrawn at the request of the editor and publisher after that the authors have approved the proofs of their requested corrigendum. The full Elsevier Policy on Article Withdrawal can be found at http://www.elsevier.com/locate/withdrawalpolicy. 

Authors: Yingjian Song (author h-index: 0, Emerging Researcher); Wei Zhang (author h-index: 0, Emerging Researcher); Jiuxu Zhang (author h-index: 0, Emerging Researcher); Zhaolei You (author h-index: 2, Emerging Researcher); Tao Hu (author h-index: 0, Emerging Researcher); Guangyuan Shao (author h-index: 3, Emerging Researcher); Zheng Zhang (author h-index: 0, Emerging Researcher); Zhicheng Xu (author h-index: 0, Emerging Researcher); Xiaofeng Yu (author h-index: 0, Emerging Researcher)

Institutions: Department of Thoracic Surgery, The Affiliated Yantai Yuhuangding Hospital of Qingdao University, Yantai 264000, Shandong, People's Republic of China. (null); Department of Thoracic Surgery, The Affiliated Yantai Yuhuangding Hospital of Qingdao University, Yantai 264000, Shandong, People's Republic of China. (null); Department of Cardio-Thoracic Surgery, The People's Hospital of Hai Yang City,Yantai 264000, Shandong, People's Republic of China. (null); Department of Thoracic Surgery, The Affiliated Yantai Yuhuangding Hospital of Qingdao University, Yantai 264000, Shandong, People's Republic of China. (null); Department of Thoracic Surgery, The Affiliated Yantai Yuhuangding Hospital of Qingdao University, Yantai 264000, Shandong, People's Republic of China. (null); Department of Thoracic Surgery, The Affiliated Yantai Yuhuangding Hospital of Qingdao University, Yantai 264000, Shandong, People's Republic of China. (null); Department of Thoracic Surgery, The Affiliated Yantai Yuhuangding Hospital of Qingdao University, Yantai 264000, Shandong, People's Republic of China. (null); Department of Cardio-Thoracic Surgery, The People's Hospital of Hai Yang City,Yantai 264000, Shandong, People's Republic of China. (null); Department of Thoracic Surgery, The Affiliated Yantai Yuhuangding Hospital of Qingdao University, Yantai 264000, Shandong, People's Republic of China. (null)

Journal: experimental cell research (journal JCR: Q2, High Level Journal)

Support

 The article appears legitimate based on the following factors:

1. Journal Reputation: The article is published in "experimental cell research," a Q2 journal, indicating a high level of credibility and a rigorous peer-review process.

2. Author Reputation: All authors are affiliated with reputable institutions, such as Qingdao University and The People's Hospital of Hai Yang City, suggesting they are qualified researchers in their field.

3. Title and Abstract Integrity: The title and abstract present a clear, coherent, and scientifically plausible hypothesis, methodology, and conclusion, with no apparent signs of controversial topics, made-up data, or plagiarism.

4. Author H-Index: Although the authors have low h-indices, this is not necessarily an indicator of illegitimacy, as they are identified as emerging researchers.

5. Institutional Affiliation: All authors are associated with recognized medical institutions, further supporting the legitimacy of the research.

Attack

The article should be retracted due to the lack of substantial evidence and data to support the claims made in the abstract.
The authors' low h-index scores and their affiliation with institutions that have no established reputation
in the field raise concerns about their expertise and credibility.
The absence of email addresses for the authors also raises questions about their accountability and transparency.
The article's title and abstract, while seemingly plausible, require further investigation to confirm their validity.

Decision:

No
The article appears legitimate primarily due to its publication in a high-level, Q2 journal,
 "experimental cell research," which suggests a rigorous peer-review process.
The authors' affiliations with recognized medical institutions further support its credibility.
However, the attacking reviewer's concerns about the lack of substantial evidence and the authors'
low h-indices are valid. Despite these concerns, the article's legitimacy is supported by its clear
hypothesis, methodology, and conclusion, and the absence of apparent signs of controversial topics,
made-up data, or plagiarism. The article's legitimacy could be further confirmed with additional investigation
into the authors' expertise and the validity of their claims.