fm4bio-ning
commited on
Commit
•
021cbab
1
Parent(s):
b460f23
Update README.md
Browse files
README.md
CHANGED
@@ -10,7 +10,7 @@ By leveraging MoE layers, AIDO.Protein efficiently scales to 16 billion paramete
|
|
10 |
|
11 |
## Model Architecture Details
|
12 |
ADIO.Protein is a transformer encoder-only architecture with the dense MLP layer in each transformer block replaced by a sparse MoE layer. It uses single amino acid tokenization and is optimized using a masked languange modeling (MLM) training objective. For each token, 2 experts will be selectively activated by the top-2 rounting mechiansim.
|
13 |
-
<center><img src="
|
14 |
More architecture details are shown below:
|
15 |
|
16 |
|Model Arch Component | Value |
|
@@ -49,11 +49,15 @@ We encode protein sequence with single amino acid resolution with 44 vocabularie
|
|
49 |
We assess the advantages of pretraining AIDO.Protein 16B through experiments across more than 300 tasks from two important protein benchmarks, xTrimoPGLM benchmark and ProteinGym DMS benchmark, encompassing residue-level, sequence-level, and protein-protein interaction (PPI) level tasks. We further adapted our model for structure-conditioned protein sequence generation tasks
|
50 |
|
51 |
## Results
|
52 |
-
|
|
|
53 |
|
54 |
-
|
|
|
|
|
|
|
|
|
55 |
|
56 |
-
- Inverse Folding Generation
|
57 |
|
58 |
## How to Use
|
59 |
### Build any downstream models from this backbone
|
|
|
10 |
|
11 |
## Model Architecture Details
|
12 |
ADIO.Protein is a transformer encoder-only architecture with the dense MLP layer in each transformer block replaced by a sparse MoE layer. It uses single amino acid tokenization and is optimized using a masked languange modeling (MLM) training objective. For each token, 2 experts will be selectively activated by the top-2 rounting mechiansim.
|
13 |
+
<center><img src="proteinmoe_architecture.png" alt="An Overview of AIDO.Protein" style="width:70%; height:auto;" /></center>
|
14 |
More architecture details are shown below:
|
15 |
|
16 |
|Model Arch Component | Value |
|
|
|
49 |
We assess the advantages of pretraining AIDO.Protein 16B through experiments across more than 300 tasks from two important protein benchmarks, xTrimoPGLM benchmark and ProteinGym DMS benchmark, encompassing residue-level, sequence-level, and protein-protein interaction (PPI) level tasks. We further adapted our model for structure-conditioned protein sequence generation tasks
|
50 |
|
51 |
## Results
|
52 |
+
### xTrimoPGLM Benchmark
|
53 |
+
<center><img src="xtrimo_results.png" alt="An Overview of AIDO.Protein" style="width:70%; height:auto;" /></center>
|
54 |
|
55 |
+
### ProteinGym DMS Benchmark
|
56 |
+
<center><img src="dms_results.png" alt="An Overview of AIDO.Protein" style="width:70%; height:auto;" /></center>
|
57 |
+
|
58 |
+
### Inverse Folding Generation
|
59 |
+
<center><img src="inverse_folding.png" alt="An Overview of AIDO.Protein" style="width:70%; height:auto;" /></center>
|
60 |
|
|
|
61 |
|
62 |
## How to Use
|
63 |
### Build any downstream models from this backbone
|