--- library_name: transformers license: mit datasets: - allenai/peer_read language: - en metrics: - accuracy - f1 --- # Model Card for PaperPub *Paper pub*lication prediction based on English computer science abstracts. ## Model Details ### Model Description PaperPub is a SciBERT ([Beltagy et al 2019](https://arxiv.org/abs/1903.10676)) model fine-tuned to predict paper acceptance from computer science abstracts. Acceptance is modeled as a binary decision of accept or reject. The training and evaluation data is based on the arXiv subsection of PeerRead ([Kang et al. 2018](https://aclanthology.org/N18-1149/)). Our main use case for PaperPub is to research how attribution scores derived from acceptance predictions can inform reflecting about content and writing quality of abstracts. - **Developed by:** Semantic Computing Group, Bielefeld University, in particular Jan-Philipp Töberg, Christoph Düsing, Jonas Belouadi and Matthias Orlikowski - **Model type:** BERT for binary classification - **Language(s) (NLP):** English - **License:** MIT - **Finetuned from model:** SciBERT ### Model Sources We will add a public demo of PaperPub used in an application which uses attribution scores to highlight words in an abstract that contribute to acceptance/rejection predcitions. - **Repository:** tba - **Demo:** tba ## Uses PaperPub can only be meaningfully used in a research setting. The model should not be used for any consequential paper quality judgements. ### Direct Use The intended use case in research into how attribution scores computed from paper acceptance decisions reflect the abstract's content quality. ### Out-of-Scope Use This model must not be used as part of any type of paper quality judgements, but in particular not in a peer review process. PaperPub is explicitly not meant to automate paper acceptance decisions. ## Bias, Risks, and Limitations Bias, Risks, and Limitations are mainly related to the used datset. In addition to limitations that apply to the SciBERT pre-training corpus, our training data represents only a very specific subset of papers. PaperPub was trained in a hackathon-like setting, so performance is not optimized and not our main goal. ### Recommendations Users should be aware that the dataset (computer science arXive preprints from a specific period) used for fine-tuning represents a very specific idea of what papers and in particular papers fit for publication look like. ## How to Get Started with the Model tba ## Training Details ### Training Data Custom stratified split of the arXiv subsection of PeerRead ([Kang et al. 2018](https://aclanthology.org/N18-1149/)). We use the data from their GitHub repository, not the Huggingface Hub version. ### Training Procedure <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> #### Training Hyperparameters - **Training regime:** bf16 mixed precision - **Epochs:** 2 - **Initial Learning Rate:** 2^-5 ## Evaluation <!-- This section describes the evaluation protocols and provides the results. --> ### Testing Data, Factors & Metrics #### Testing Data Custom stratified split of the arXiv subsection of PeerRead ([Kang et al. 2018](https://aclanthology.org/N18-1149/)). We use the data from their GitHub repository, not the Huggingface Hub version. #### Factors Models, we compare to a naive most-frequent-class baseline. #### Metrics Accuracy, Macro F1 ### Results - Majority Baseline - Acc. - 0.75 - Macro F1 - 0.43 - PaperPub - Acc. - 0.82 - Macro F1 - 0.76 ## Environmental Impact <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly --> - **Hardware Type:** 1xA40 - **Hours used:** 0.3 - **Cloud Provider:** Private Infrastructure - **Compute Region:** Europe ## Technical Specifications ### Compute Infrastructure We are using an internal SLURM cluster with A40 GPUs ## Citation tba ## Model Card Contact [Matthias Orlikowski](https://orlikow.ski)