Flash attention ?

by edmond - opened May 21, 2024

May 21, 2024

It says flash attention 2 is not available for this model.
But I was thinking maybe I could only use FA2 for the gemma part, do you think this is possible ?

merve

Google org May 27, 2024

@edmond you can check out TGI's flash PaliGemma implementation here it's implemented for vision head but doesn't have as much effect as flash in decoder

edmond changed discussion status to closed Jun 4, 2024

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Your need to confirm your account before you can post a new comment.

· Sign up or log in to comment