| # Attention Rollout -- RoBERTa | |
| In this demo, we use the RoBERTa language model (optimized for masked language modelling and finetuned | |
| for sentiment analysis). The model predicts for a given sentences whether it expresses a positive, | |
| negative or neutral sentiment. But how does it arrive at its classification? This is, surprisingly | |
| perhaps, very difficult to determine. | |
| Abnar & Zuidema (2020) proposed a method for Transformers called **Attention Rollout**, which was further | |
| refined by Chefer et al. (2021) into **Gradient-weighted Attention Rollout**. Here we compare them to | |
| another popular method called **Integrated Gradients**. | |