---
library_name: transformers
tags:
- requirements-engineering
- domain-specific-lm
- concept-extraction
---

# Model Card for BERT4RE

## Model Details

### Model Description

BERT4RE is a domain-specific language model (LM) designed to support various requirements engineering (RE) tasks, including requirements classification, detection of language issues, identification of domain concepts, and establishment of requirements traceability links. BERT4RE is retrained from the generic BERTbase model using publicly available RE-related texts.

- **Developed by:** Muideen Ajagbe, Liping Zhao
- **Shared by [optional]:** Alberto Rodriguez
- **Model type:** Domain-specific language model
- **Language(s) (NLP):** English
- **License:** MIT
- **Finetuned from model:** BERTbase

### Model Sources

- **Paper [optional]:** https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9920081

## Uses

### Direct Use

BERT4RE can be used directly for various RE tasks such as requirements classification, detection of language issues, and identification of domain concepts from requirements text.

### Downstream Use

BERT4RE can be fine-tuned for specific RE tasks, providing enhanced performance over generic models like BERTbase. An example is its application in a multiclass classification task to identify nine different requirements concepts.

### Out-of-Scope Use

The model may not perform well for tasks outside the domain of requirements engineering or on datasets with significantly different characteristics from the training data.

## Bias, Risks, and Limitations

### Recommendations

Users should be aware of the biases and limitations inherent in the model due to the nature of the training data. Further evaluation and fine-tuning may be necessary for specific applications.

## Training Details

### Training Data

The training data consists of four requirements-related datasets:

- **PROMISE NFR Dataset:** Contains 625 requirements from 15 software development projects.
- **PURE Dataset:** Contains 522,444 lexical words and 865,551 tokens, with 29,000 unique words extracted.
- **App Review Datasets:** Contains four million app reviews with over three million unique words.
- **Google Playstore App Reviews:** Contains 600,000 app reviews with more than two million unique words.

### Training Procedure

The retraining process follows the procedures used to train the original BERTbase model, involving the Masking LM (MLM) and Next Sentence Prediction (NSP) tasks. The model was retrained on Google Cloud TPU v3-8 with the following hyperparameters:

- **Batch size:** 128 sequences
- **Tokens per batch:** 3200 tokens (128 * 25 subword tokens)
- **Total sequence blocks:** 1.3M
- **Epochs:** 40
- **Optimization:** Adam with a learning rate of 1e-4, a weight decay of 0.01, and a dropout rate of 0.1

## Evaluation
Please see cited paper for evaluation details.


## Environmental Impact

- **Hardware Type:** Google Cloud TPU v3-8
- **Cloud Provider:** Google Cloud

## Technical Specifications [optional]

### Model Architecture and Objective

BERT4RE is based on the BERTbase architecture with 12 Transformer blocks, 768 hidden size, and 12 self-attention heads.

### Compute Infrastructure

#### Hardware

Google Cloud TPU v3-8 with a single 8-core processor.


## Citation [optional]

**BibTeX:**

@INPROCEEDINGS{9920081,
  author={Ajagbe, Muideen and Zhao, Liping},
  booktitle={2022 IEEE 30th International Requirements Engineering Conference (RE)}, 
  title={Retraining a BERT Model for Transfer Learning in Requirements Engineering: A Preliminary Study}, 
  year={2022},
  volume={},
  number={},
  pages={309-315},
  keywords={Deep learning;Bit error rate;Transfer learning;Natural language processing;Requirements engineering;Task analysis;Requirements Engineering;Requirements Classification;Language Models;BERT;Domain-Specific Language Models;Transfer Learning;Deep Learning;Machine Learning;Natural Language Processing},
  doi={10.1109/RE54965.2022.00046}}
  
## Model Card Authors [optional]
Muideen Ajagbe, Liping Zhao

## Model Card Contact

Alberto Rodriguez (arodri39@nd.edu)