PyTorch
megatron-bert
ligeti commited on
Commit
6d61ae5
1 Parent(s): 3b7ca88

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +45 -7
README.md CHANGED
@@ -44,6 +44,7 @@ python bin/PhaStyle.py \
44
 
45
  ```
46
 
 
47
  ### Datasets Used:
48
 
49
  - **BACPHLIP (without E. coli)**: 1,868 training sequences and 246 validation sequences.
@@ -53,18 +54,55 @@ python bin/PhaStyle.py \
53
  Each dataset was processed using **512bp segment lengths** to simulate fragmented metagenomic assemblies.
54
 
55
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
56
 
57
- ## Performance
58
 
59
- ProkBERT PhaStyle outperforms state-of-the-art models, especially in generalization and speed. It has been benchmarked on **short fragments** (512bp) and **phages from unseen environments**, demonstrating its robustness for both environmental and clinical datasets.
 
 
 
 
 
 
 
60
 
61
- ### Key Metrics:
62
- - **Balanced Accuracy**: 0.94 (on 1022bp fragments from the *Escherichia* dataset)
63
- - **MCC (Matthews Correlation Coefficient)**: 0.91
64
- - **Sensitivity**: 0.97
65
- - **Specificity**: 0.91
66
 
 
67
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
68
 
69
  ## Limitations
70
 
 
44
 
45
  ```
46
 
47
+
48
  ### Datasets Used:
49
 
50
  - **BACPHLIP (without E. coli)**: 1,868 training sequences and 246 validation sequences.
 
54
  Each dataset was processed using **512bp segment lengths** to simulate fragmented metagenomic assemblies.
55
 
56
  ---
57
+ ## Performance Results
58
+
59
+ The performance of ProkBERT PhaStyle was evaluated on various datasets, including *Escherichia* and EXTREMOPHILE phages, using segment lengths of 512bp and 1022bp. The results are summarized below:
60
+
61
+ ### Performance on *Escherichia* Dataset (512bp and 1022bp segments)
62
+
63
+ | Method | Balanced Accuracy | MCC | Sensitivity | Specificity |
64
+ |--------------------------|-------------------|-------|-------------|-------------|
65
+ | **ProkBERT-mini (512bp)** | 0.91 | 0.83 | 0.94 | 0.89 |
66
+ | ProkBERT-mini-long (512bp)| 0.90 | 0.82 | 0.96 | 0.85 |
67
+ | ProkBERT-mini-c (512bp) | 0.89 | 0.80 | 0.95 | 0.84 |
68
+ | DNABERT-2-117M (512bp) | 0.84 | 0.72 | 0.95 | 0.74 |
69
+ | Nuc. Trans.-50m (512bp) | 0.85 | 0.72 | 0.92 | 0.78 |
70
+ | **ProkBERT-mini (1022bp)**| **0.94** | **0.88** | **0.97** | **0.91** |
71
+ | ProkBERT-mini-long (1022bp)| 0.94 | 0.89 | 0.97 | 0.91 |
72
 
73
+ ### Performance on EXTREMOPHILE Dataset (512bp and 1022bp segments)
74
 
75
+ | Method | Balanced Accuracy | MCC | Sensitivity | Specificity |
76
+ |--------------------------|-------------------|-------|-------------|-------------|
77
+ | **ProkBERT-mini (512bp)** | 0.93 | 0.83 | 0.99 | 0.87 |
78
+ | ProkBERT-mini-long (512bp)| 0.93 | 0.82 | **1.00** | 0.86 |
79
+ | ProkBERT-mini-c (512bp) | 0.92 | 0.80 | 0.99 | 0.84 |
80
+ | DNABERT-2-117M (512bp) | 0.89 | 0.74 | 0.99 | 0.79 |
81
+ | **ProkBERT-mini (1022bp)**| **0.96** | **0.91** | **1.00** | **0.93** |
82
+ | ProkBERT-mini-long (1022bp)| 0.96 | 0.90 | 1.00 | 0.92 |
83
 
84
+ These tables highlight the high accuracy, MCC, and generalization capability of ProkBERT models, particularly on challenging datasets like *Escherichia* and extremophile phages. The ProkBERT-mini and ProkBERT-mini-long models consistently performed well on both datasets.
 
 
 
 
85
 
86
+ For more detailed results, including additional metrics, please refer to the original research paper.
87
  ---
88
+ ## Inference Speed and Running Times
89
+
90
+ The computational performance of ProkBERT PhaStyle was evaluated using 1,000 randomly selected sequences from the BACPHLIP dataset. The evaluation was performed on a consistent hardware setup with NVIDIA Tesla A100 GPUs. The execution times and inference speeds of various models are summarized below:
91
+
92
+ ### Execution Times (in seconds)
93
+
94
+ | Model | Execution Time (seconds) | Inference Speed (MB/sec) |
95
+ |--------------------------|--------------------------|--------------------------|
96
+ | **ProkBERT-mini-long** | **132** | **0.52** |
97
+ | ProkBERT-mini | 141 | 0.49 |
98
+ | ProkBERT-mini-c | 146 | 0.47 |
99
+ | DNABERT-2-117M | 248 | 0.25 |
100
+ | Nucleotide Transformer-50m| 342 | 0.18 |
101
+ | Nucleotide Transformer-500m| 502 | 0.12 |
102
+ | DeePhage | 159 | 0.43 |
103
+ | PhaTYP | 2,718 | 0.03 |
104
+ | BACPHLIP | 7,125 | 0.01 |
105
+
106
 
107
  ## Limitations
108