Spaces:
Sleeping
Sleeping
Update documentation
Browse files- hexviz/pages/2_📄Documentation.py +19 -11
hexviz/pages/2_📄Documentation.py
CHANGED
@@ -25,19 +25,27 @@ tool for interpreting transformer model internals see fex ([Abnar et al.
|
|
25 |
2020](https://arxiv.org/abs/2005.00928v2)). [BERTology meets
|
26 |
biology](https://arxiv.org/abs/2006.15222) provides a thorough introduction to
|
27 |
how we can analyze Transformer protein models through the lens of attention,
|
28 |
-
they show exciting findings such as:
|
29 |
-
|
30 |
-
|
31 |
-
|
32 |
-
|
|
|
33 |
|
34 |
Most existing tools for analyzing and visualizing attention patterns focus on
|
35 |
-
models trained on text.
|
36 |
-
|
37 |
-
|
38 |
-
|
39 |
-
|
40 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
41 |
structure. We hope this can enable domain experts to explore and interpret the
|
42 |
knowledge contained in pLMs.
|
43 |
|
|
|
25 |
2020](https://arxiv.org/abs/2005.00928v2)). [BERTology meets
|
26 |
biology](https://arxiv.org/abs/2006.15222) provides a thorough introduction to
|
27 |
how we can analyze Transformer protein models through the lens of attention,
|
28 |
+
they show exciting findings such as:
|
29 |
+
> Attention: (1) captures the folding
|
30 |
+
> structure of proteins, connecting amino acids that are far apart in the
|
31 |
+
> underlying sequence, but spatially close in the three-dimensional structure, (2)
|
32 |
+
> targets binding sites, a key functional component of proteins, and (3) focuses
|
33 |
+
> on progressively more complex biophysical properties with increasing layer depth
|
34 |
|
35 |
Most existing tools for analyzing and visualizing attention patterns focus on
|
36 |
+
models trained on text ([BertViz](https://github.com/jessevig/bertviz),
|
37 |
+
[exBERT], [exBERT](https://exbert.net/)). It can be hard to analyze protein
|
38 |
+
sequences using these tools as we don't have any intuitive understand about the
|
39 |
+
protein language when looking at an amino acid sequence in the same way we do
|
40 |
+
for natural language. Experts studying proteins do have an understanding of
|
41 |
+
proteins, but it is mostly in in the context of a protein's structure, not its
|
42 |
+
sequence. Can we build a tool for analyzing attention patterns that can leverage
|
43 |
+
expert's knowledge of protein structure to understand what pLMs learn?
|
44 |
+
|
45 |
+
BERTology meets biology shows how visualizing attention patterns in the context
|
46 |
+
of protein structure can facilitate novel discoveries about what models learn.
|
47 |
+
[**Hexviz**](https://huggingface.co/spaces/aksell/hexviz) builds on this, and is
|
48 |
+
a tool to simplify analyzing attention patterns in the context of protein
|
49 |
structure. We hope this can enable domain experts to explore and interpret the
|
50 |
knowledge contained in pLMs.
|
51 |
|