Synthyra

/

ESMplusplus_large

like 9

Follow

Synthyra 25

Dear Logan @lhallee ,

I would be happy for your help with retrieving attention maps.

I am using the ESMplusplus_large model, and following your instructions to return attention maps from a AutoModelForSequenceClassification model:

output = model(**tokenized, output_attentions=True)
att = output.attentions

But I am getting an empty att object.

The complete output object printed looks like this:

ESMplusplusOutput(loss=None, logits=tensor([[-0.8654, -2.0843, -4.2370, -1.7364,  0.2102,  1.1601,  6.3874]],
       device='mps:0', grad_fn=<LinearBackward0>), last_hidden_state=tensor([[[ 0.0066, -0.0075,  0.0006,  ...,  0.0018,  0.0035,  0.0136],
         [ 0.0031, -0.0062, -0.0041,  ...,  0.0021,  0.0272, -0.0221],
         [ 0.0214, -0.0131,  0.0022,  ..., -0.0161, -0.0220, -0.0021],
         ...,
         [ 0.0253, -0.0277, -0.0146,  ..., -0.0092, -0.0010,  0.0157],
         [ 0.0154, -0.0131,  0.0019,  ..., -0.0271, -0.0365, -0.0148],
         [-0.0117, -0.0267, -0.0263,  ...,  0.0057,  0.0389, -0.0058]]],
       device='mps:0'), hidden_states=None, attentions=None)

Many thanks again for your help!
Dani