Rope for 32k
Hi!
I was just wondering if I can extend the model with RoPE at 32k and still maintain good coherence. I saw that you said we can extend to 16k. Would that be the limit to RoPE and maintain good reasoning and coherence?
Hi @GhostGate !
The model isn't even reliably hitting 16K, as I was testing earlier today and it totally broke down. I may have to remove that claim from the readme.
Even if it did reliably hit 16K, my gut reaction is that 32K would be too much of a stretch. In my testing with Mistrals, the most you could RoPE from 8K native context was 24K. Of course, Mistral 7B is old now, things have changed, and Gemma is a totally different animal.
TL;DR: 15~16K is the limit. Sorry.
Have a nice day! :)
Hi @matchaaaaa !
Thank you so much for the reply :) Sorry if it was a silly question, I am trying to ascertain what is the limit of different models, since I am trying to gather a list of models for RP, which I would like to do a proper test and review. I gotten really annoyed with people commenting about the quality of models based on half-assed character cards, bad instructions and horrendous pre-sets. I never tried Gemma 2, but I think I will try your model and give you an honest feedback, if you are interested of course.
EDIT: Have an epic day as well :D
Hi @GhostGate !
That's not a silly question!
I would love that! Honest feedback as well as that list (when it's ready) would be phenomenal. I have been saying for a long time that the community needs something like what you're putting together. I'm also interested in helping some if you're open to that.
Best of luck either way, and have a lovely day! (Woah, it rhymed) :D
Hi!
I would love to get some help :) If you would like, I could get in touch with you somewhere where it's more convenient and talk.
@matchaspeed on Discord is the easiest way to get ahold of me. Shoot me a message!
Have a wonderful day! :3