ArliAI

/

Mistral-Nemo-12B-ArliAI-RPMax-v1.1

like 39

I've played around with the 12B GGUF Q5_K_M version and here are my observations.

Settings:
I'm using a customized Mistral 12b formatting and text completion preset (https://qu.ax/zaWP.json and https://qu.ax/dTZc.json). So far, it's the best performing preset i've tried, it's definitely way better than Sillytavern's default Mistral preset. It works just as well with this model. I'm also using 12k context.

Setup:
I've tested this in a semi-complicated roleplay scenario that has several characters and multiple plot threads. The story features several characters (from Chainsaw Man) that are not in a group chat (only 1 character card that houses them all), and a giant lorebook describing the world. They and me go into 'the backrooms' on an expedition. That's the setup. The story gets quite stressful, everyone is on edge, radio communication with other teams stops working, there is a disagreement about whether to split up into two teams or stick together.

I use this chat as a baseline to compare different models all the time. Instead of judging them "by feel", i can spot objective differences. Did it remember about every character in the team after X messages? Did it properly remember to stick together after the argument, or randomly forgot about it and split us up anyway? How well does it portray every character's personality? Does it portray group dynamics well or just sticks to generic one-liners?
For this model, for me, the main comparison was against NemoMix Unleashed 12B, as that is what i was using prior. I essentially wanted to see if this model retained the strengths of the base model like NemoMix Unleashed did, and if the text it generates is actually better or more or less the same.

Observations:
The model remembered to keep including every character from the group as well enough as Nemomix Unleashed. It also remembered about the presence of other teams. Basically, so far this just means that it performs up to general expectation of a 12B Nemo model.

In the "lets split up after all as if that was the plan all along" test, RPMax did it more often than NemoMix Unleashed (it's bad when a model does it). But it wasn't prevalent, maybe in 1 out of 4 generations in general. In comparison, NemoMix Unleashed did it even less often, which surprised me before. But here it's still within acceptable range for a 12B finetune, as regeneration mostly fixes it.

In random sections of the chat, RPMax would generate similar responses in meaning to NemoMix Unleashed, but the text is somewhat less raw, not exactly poetic, but it flows better, has a bit more flair to it. Nemomix Unleashed is quite blunt in comparison. RPMax also seemingly better describes emotion, and i'd even say makes the characters in general somewhat more emotive, which is an improvement, although not quantifiable. I wouldn't say it's a night and day difference, but it's definitely noticeable, even if it's subtle.

The differences are in subtle details, and this is where RPMax suddenly did something NemoMix Unleashed couldn't. In some cases, RPMax just seems to understand the overall context of the scene better and thus sticks to the mood with more precision. Sort of like it reads between the lines 1 level deeper. It's not something that amounts to an actual blunder or mistake if a model misses it, but when you see a model not miss it compared to one that missed, you immediately go - "Aha! It gets it!"

Examples (i'm obviously not going to include dozens of screenshots, so the choice of examples is a bit curated, but it's also not like i'm set out to make NemoMix Unleashed look worse on purpose, it truly isn't hard to get a generally better output with RPMax than NemoMix)

Here's a VERY good example. In the chat, after the radio goes silent, i tell everyone to take out the batteries from their radios. The female leader generally goes "Wtf for?". This is a checkpoint where RPMax shines over Nemomix, as it understood by itself that i must be thinking about the possibility of the radios being tracked. I had to explain that to EVERY other 12b and 8b model i've tried! I don't know how, but RPMax KNOWS what's up in that moment - the lead female goes "Wtf for?...... Aaaaah, smaaaart, i see what you're thinking, User... It's so 'they' can't track us!". Now, it only does that about half the time, but it's so awesome that it does it at all. And i'll reiterate - Nemomix Unleashed 12B doesn't get it! Llama 8b models don't get it! RPMax 12B was the first one i came across in this size range that got it.

RPMax just gets it...

NemoMix is just confused (understandably so, but still)

But that's not even all there is to it yet, no. After that, she goes "You know, rookie, i'm starting to like you, keep that smart shit up." And i tried replying "Oh yeah? Enough to go somewhere after hours together?" And here NemoMix usually goes with an inaccuracy, sort of forgetting about the incredible tension of the setting, immediately switching into playful flirty mode, like "Hm, what're you suggesting? I'm not an easy woman to please, hehe", yada-yada.

Nemomix is down bad!

But RPMax though... Damn. It's like it read between the lines. It must've thought "the situation is very grim, and this must be just friendly banter to slightly lift the mood, with a very subtle hint, but the characters are fighting for survival against the unknown, so they have to stay on track." It really must've thought that, because that's what it gave me back every time. No nonsense. A brief retort with a hint, then click - switch back to problem solving mode. Because you know, that's more in line with what you'd expect from people that are in an extremely dangerous unknown dark territory.
And again, that difference is subtle. But it makes the difference.

RPMax seems to understand that these circumstances should take priority over flirting

A couple other examples where RPMax stayed truer to the overall context and character personalities:

Later on, when we spot a corpse from the other team, RPMax has less generations where the female leader for some reason is knocked off-balance mentally while examining him. But even when it goes that route, It does it with more nuance, and sort of makes it work anyway. And when it doesn't go there, it's just in general more coherent, less repetitive. 2 sets of examples for this checkpoint.

This is RPMax. Some reasoning going on. Some let's not forget about other characters going on. Some agreeing that splitting up is bad going on (yahoo!)

This is NemoMix Unleashed. More literal reiteration of my prompt and previously mulled-over points which don't need to be said again (looking for other teams). No mention of other characters. To be fair though, overall, this isn't a bad output at all. It's just, some things about it could be better, and i think in the screenshot above those things indeed were done better (i mentioned them)

While both versions basically achieve the same thing, even while trying to compare them unbiased, i just lean towards the RPMax's version. It ticks extra boxes.

Same checkpoint, different timeline. RPMax leaned into "she's shaken" territory, but i think does it well enough where it actually gives her colorful characterization, and it's written quite expressively. I especially like what it did near the end after she didn't finish her sentence.

NemoMix is just confused, that's the entire emotion. And of course, reiteration of previously mulled over points (we need to find alpha and bravo, don't you know???)

Another example is that whenever i give several suggestions about what i think our team should be doing or avoid doing, with most 12B and less models she starts to kind of just follow my lead all the time, shifting the de facto leader status onto me, asking my advice at every other juncture that follows. But RPMax was noticeably better at keeping her true to the character card, as even though she agreed with my suggestions, she still kept making the point that in the end it's her decision and we're following her lead, and wasn't waiting for me to give my two cents all the time.

NemoMix wants me to hold its hand. Bruh, just figure it out!

Don't worry, RPMax can take it from here

Now, this doesn't mean RPMax is entirely immune to this. I think it's a base model thing. But if NemoMix is proactive at this checkpoint only about 1/4 of the time, RPMax gets closer to 2/3, maybe even 3/4. Sorry, i'm not doing a 100+ sample test, but combined with other similar instances, that's the general feel i got.

Conclusion:
The model lost the most problematic plot point (split-up) somewhat more often than Nemomix Unleashed 12B. Kind of weird, to be honest, since both are 12B Mistral Nemo at the end of the day. Maybe i'm imagining this, maybe not, or maybe it's just because the sample size isn't big enough, after all i'm generating 5~10 examples per checkpoint rather than 100+. It still made this mistake less often than Fimbulvetr, and it's leagues above any 7-8B models, those get it wrong ALL the time. But that's just the 12B brains being functional. At least 3/4th of the time. Maybe it's to do with running at 12k context. Either way, aside from this particular thing, i didn't notice any other incoherences.

In general, even though it mostly fell in line with what Nemomix Unleashed would say, RPMax would at least say it in a fancier language, and at best would be more precise about the current undertones, leading to more moments where it "gets it" for some reason, where Nemomix Unleashed doesn't.

Overall, i think this definitely took the spot of NemoMix Unleashed 12B for me as my go to 12B RP model for now. I've still been meaning to try Chronos Gold and Rocinante, but i think this model is in a very good place, i don't see myself shoving it.

So, thanks for the model! Excited to see what'll be next from you, either it be an update to this, or a different model trained with a similar method. It really seems like whatever you did worked and there's merit in exploring this kind of llm training further.