Not impressed?
I'm not sure, maybe it doesn't like my usual test questions, but it performs worse than phi-4 it seems. For example this pretty easy question >>>
There are three killers in a room. Someone enters the room and kills one of them. Nobody leaves the room. How many killers are left in the room?
think
Alright, let's break this down step by step to understand the problem clearly.
First, there are initially three killers in the room. Then, someone enters the room and kills one of them. After that, it's mentioned that nobody leaves the room.
So, initially, there are three killers. When someone enters, they kill one killer. This means that one killer is removed from the group. Since nobody leaves after the killing occurs, we only need to account for the remaining individuals in the room.
Therefore, by subtracting the one killed from the initial number of killers, we can determine how many killers remain.
/think
After the person enters and kills one killer, there are two killers left in the room.
llama.cpp build: 4518, build-in template. temp 0.5-0.7, rep pen 1.1
For me the model is pretty terrible too, NeMo is still far greater in coherence and following instructions. I think it has to do with this model's terrible instruct format, I dunno... well, I hope so. It's a promising model on paper, I'd prefer to see it perform... Maybe a ChatML version would be better.
Maybe the fact that it was designed in a different language has something to do with it... IDK. I can easily prompt my current best LLMs to create a plan before going about something, and it does just fine with that.
The answer to the question would not be two, though- but three. If there are three killers in the room, let's represent them with the letter X:
| X X X |
|---|
One person enters the room, we will represent them with the letter O.
| X X X O |
|---|
Now that person Kills one of the three killers.
| X X O |
|---|
That person is now also a killer. So O now becomes X. As soon as O killed an X, he is no longer able to be a O. His status has changed.
Thus, if no one left the room- there are still 3 killers in the room at the end of the equation.
| X X X |
|---|
The question would only work if you asked how many original killers remained in the room; then it would be 2.
Really, the answer is 4, since the question does not mention dead or alive.
So there are 4 killers in the room, 3 alive (supposed alive) and 1 dead.