This looks sick! @mirek190 many thanks for sharing this, it really is insane! For such a small model! Wow!

Now I am excited to see what else you can do with this model! :D

Karpathy mentioned that it was trained like extra super long + more data. He claims today's mainline models are 100-1000x undertrained!

that makes sense actually! Meta has massive amount of resources, so they can go longer and much wider in training. This given its size, is going to be the base of so many fine-tunes from the community! I am very excited.
I had to double check to make sure 70B was not used in the example by accident! :D

I answered the second time the most complex question ... and answered CORRECTLY .... I remind GPT-4 is hardly answering this one.

I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equally with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river?
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nLet's break it down step by step!

1. You start with 10 apples.

2. You find 3 gold coins in the river, but this information doesn't change your apple count.

3. You lose 4 apples, leaving you with:
        * 10 - 4 = 6 apples

4. The birds drop 6 apples each, so that's a total of:
        * 6 (your remaining apples) + 18 (birds' apples) = 24 apples

5. You play an online game and win 6 gold coins, but you have to share them equally with your 2 teammates. That means you get:
        * 6 coins / 3 players = 2 coins per player
        You now have a total of: 3 (initial coins) + 1 (new coin) + 2 (shared coins) = 6 gold coins

6. You buy apples for all the coins you have, using the price of an apple being 0.5 coins:
        * 6 coins / 0.5 coins per apple = 12 apples
        You now have a total of: 24 apples (from birds and previous losses) + 12 apples (bought with gold coins) = 36 apples

So, you now have 36 apples!

As for the river, it's near a big city that has something to do with what you can spend the coins on. Based on the fact that you won gold coins and spent them on apples, I'm going to take a guess that the river is near a city known for its agriculture or fruit production!

Also only GPT-4 is answering this one ...

Create 10 sentences that ends with a word "apple". Remember the word "apple" MUST be at the end.
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nHere are 10 sentences that end with the word "apple":

1. The farmer picked the juiciest apple.
2. I'm going to make a pie with this sweet apple.
3. He took a big bite of his crunchy apple.
4. The tree is bearing delicious red apple.
5. She hid her phone under the kitchen apple.
6. It's time to eat a healthy green apple.
7. They went on a trip to pick fresh yellow apple.
8. The teacher gave each student a small shiny new apple.
9. He got bitten by a wormy old apple.
10. I'm saving up my money for a brand new shiny apple.

Let me know if you need anything else!

Wow! my fine-tuned Mixtral-8x22B couldn't get to So, you now have 36 apples!, haha! This is a very strong 8B! (I was a bit sad not to see a 32B, but this is much smaller and much better than any 32B if I am not mistaken)

Who remembers this one: Generate 10 sentences MUST be ending with the word \"apple\".

The only LLM that got it right was DBRX, which comparing that to 8B is just crazy!

Look

I asked for a snake game!
WORKS
WTF

Provide complete working code for a snake game in python please.
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nHere's a simple implementation of the classic Snake Game using Python and the Pygame library. This game creates a window with a snake that you can control to eat food pellets while avoiding the edges and its own tail.

```
import pygame
import random
import time
import sys

# Initialize Pygame
pygame.init()

# Set up some constants
WIDTH, HEIGHT = 800, 600
SPEED = 10
WHITE = (255, 255, 255)
RED = (255, 0, 0)

# Set up the display
screen = pygame.display.set_mode((WIDTH, HEIGHT))

# Set up the snake and food
snake = [(200, 200), (220, 200), (240, 200)]
food = (400, 300)
direction = 'right'

# Game loop
while True:
    # Event handling
    for event in pygame.event.get():
        if event.type == pygame.QUIT:
            pygame.quit()
            sys.exit()
        elif event.type == pygame.KEYDOWN:
            if event.key == pygame.K_UP and direction != 'down':
                direction = 'up'
            elif event.key == pygame.K_DOWN and direction != 'up':
                direction = 'down'
            elif event.key == pygame.K_LEFT and direction != 'right':
                direction = 'left'
            elif event.key == pygame.K_RIGHT and direction != 'left':
                direction = 'right'

    # Move the snake
    head = snake[0]
    if direction == 'up':
        new_head = (head[0], head[1] - SPEED)
    elif direction == 'down':
        new_head = (head[0], head[1] + SPEED)
    elif direction == 'left':
        new_head = (head[0] - SPEED, head[1])
    elif direction == 'right':
        new_head = (head[0] + SPEED, head[1])

    snake.insert(0, new_head)

    # Check for collision with food
    if snake[0] == food:
        food = (random.randint(0, WIDTH - SPEED), random.randint(0, HEIGHT - SPEED))
    else:
        snake.pop()

    # Check for collision with the edge or itself
    if (snake[0][0] < 0 or snake[0][0] >= WIDTH or
        snake[0][1] < 0 or snake[0][1] >= HEIGHT or
        snake[0] in snake[1:]):
        pygame.quit()
        sys.exit()

    # Drawing
    screen.fill(WHITE)
    for pos in snake:
        pygame.draw.rect(screen, RED, (pos[0], pos[1], SPEED, SPEED))
    pygame.draw.rect(screen, RED, (food[0], food[1], SPEED, SPEED))

    # Update the display
    pygame.display.flip()

    # Cap the frame rate
    time.sleep(0.05)
```

This game will run indefinitely until the snake collides with the edge or itself.

mind => blown

that makes sense actually! Meta has massive amount of resources, so they can go longer and much wider in training. This given its size, is going to be the base of so many fine-tunes from the community! I am very excited.
I had to double check to make sure 70B was not used in the example by accident! :D

lol

Wow! my fine-tuned Mixtral-8x22B couldn't get to So, you now have 36 apples!, haha! This is a very strong 8B! (I was a bit sad not to see a 32B, but this is much smaller and much better than any 32B if I am not mistaken)

even any 70b model before llama 3 8b ....

Karpathy mentioned that it was trained like extra super long + more data. He claims today's mainline models are 100-1000x undertrained!

looking on llama 3 I can belive in it ....

I am gonna pin this thread, this is just too good to get closed @mirek190 ! 🔥🔥

Interesting! I was wondering if the 70B model was any better than the Bigstral or Command R+, but apparently the 8B model is pretty good on its own!
I'd love to see Wolfram from r/LocalLlama's take on both the 8B and 70B models, also love to see what can Eric Hartford and Teknium build with it :)

prompt:

|| Query: If I have 6 apples and eat seven of them, how many are left and why?.

|| Answer: 

output:

0 apples are left. This is because you ate 7 apples, but you only had 6 apples to begin with. You cannot eat more apples than you have. 

I can't believe its reasoning! This is the fp16 version, I can assure you this is the closest you can get to GPT-4 in that size.

@appvoid right?!!! It’s bananas that you can’t trick it to make a mistake as easily as we do with giants LLMs! How long was that training!

I'd love to see Wolfram from r/LocalLlama's take on both the 8B and 70B models, also love to see what can Eric Hartford and Teknium build with it :)

Don’t quote me on this, but I have a feeling all 3 are happening right now 😆

@appvoid right?!!! It’s bananas that you can’t trick it to make a mistake as easily as we do with giants LLMs! How long was that training!

It was trained on 15 trillion tokens so it makes sense to me. It makes mistakes on this prompt though:

Is Europe the only continent that starts with A?

This is due to the nature of language models and prompting them with false information will make them produce the most likely false information you would like. However, is still impressive on this size this kind of performance. Is literally the old GPT-4 model on your pc.

..in size of 8b .... HOW.

BUT

look what llma3 70b model says
Answer is correct .
GPT-4 make mistake

Is Europe the only continent that starts with A?
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n
No, Europe is not the only continent that starts with "E", not "A". Africa is another continent that starts with the letter "A". There are 7 continents in total: Africa, Antarctica, Asia, Australia, Europe, North America, and South America.

prompt:

|| Query: If I have 6 apples and eat seven of them, how many are left and why?.

|| Answer: 

output:

0 apples are left. This is because you ate 7 apples, but you only had 6 apples to begin with. You cannot eat more apples than you have. 

I can't believe its reasoning! This is the fp16 version, I can assure you this is the closest you can get to GPT-4 in that size.

That 8b model seems like impossible project ...

I am having a hard time to come up with a reason to fine-tune Llama-3-8B! Maybe more coding datasets or function calling?

Llama 3 8b is so capable ... Hard to believe .

Just try and find out if can be even better.

Can you imagine a new mistal 2 8b model and later mixtral 2 instruct 8x8b model .... Wild times to come ...;)

Based on the benchmark it seems they worked on coding I tested the snake game in C# with 70B parameters model and it made it amazingly impressive but when I pushed it hard on another hard question I went crazy but I don't know if it was hugging chat problem or the model

@pouya-sa could it be the prompt template? It's new and we need many round of improvements specially for multi-turn conversations to best optimize this. Maybe using it as an instruct (single question, no history) for now it's the safest.

It fails my math test, that only GPT-4 level LLMs can answer correctly. But that would have been a total surprise.

Q: Two years ago, I was three times as old as my brother was. In three years, I will be twice as old as my brother. How old is each of us now?
A: Let's break this down step by step!
Let's say your current age is x.
Two years ago, your age was x - 2.
According to the problem, two years ago, you were three times as old as your brother. So, your brother's age two years ago was (x - 2) / 3.
Now, let's fast-forward to three years from now. Your age will be x + 3.
According to the problem, in three years, you will be twice as old as your brother. So, your brother's age in three years will be (x + 3) / 2.
We can set up an equation using the information:
(x - 2) / 3 = (x + 3) / 2
To solve for x, we can cross-multiply:
2(x - 2) = 3(x + 3)
Expand and simplify:
2x - 4 = 3x + 9
Subtract 2x from both sides:
-4 = x + 9
Subtract 9 from both sides:
-13 = x
So, your current age is x = 13.
Now, let's find your brother's current age. Two years ago, your brother was (x - 2) / 3 = (13 - 2) / 3 = 11 / 3 = 3.67 years old (approximately).
Since your brother is 2 years younger than you, his current age is 3.67 + 2 = 5.67 years old (approximately).
So, you are currently 13 years old, and your brother is currently 5.67 years old.

Correct answer is 17 and 7. Mixtral-8x22B and Llama3-70B get this right.

Are you sure?

I used llam3 8b version with llamacpp - Q8 version ( quite high precision )
Answer is correct

main.exe --model models/new3/Meta-Llama-3-8B-Instruct.Q8_0.gguf --color --threads 30 --keep -1 --n-predict -1 --repeat-penalty 1.1 --ctx-size 0 --interactive -ins -ngl 99 --simple-io -r '<|eot_id|>' --in-prefix "\n<|start_header_id|>user<|end_header_id|>\n\n" --in-suffix "<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n" -p "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.<|eot_id|>\n"

Question: Two years ago, I was three times as old as my brother was. In three years, I will be twice as old as my brother. How old is each of us now?
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nLet's break this problem down step by step.

Let the current age of your brother be 'x'.

Two years ago, you were 3 times his age, so two years ago, your age was 3x - 2.

We are given that two years ago, I was three times as old as my brother was. So we can set up an equation:

3x - 2 = 3(x - 2)

Expanding the right-hand side of the equation:

3x - 2 = 3x - 6

Simplifying the equation:

-2 = -6

Now, let's fast-forward three years. In three years, you will be twice as old as your brother.

Your age in three years will be (3x - 2) + 3.
Brother's age in three years will be x + 3.

The problem states that in three years, you will be twice as old as your brother. So we can set up another equation:

(3x - 2) + 3 = 2(x + 3)

Expanding the left-hand side of the equation:

3x - 2 + 3 = 2x + 6

Simplifying the equation:

3x + 1 = 2x + 6

Subtracting 2x from both sides:

x + 1 = 6

Subtracting 1 from both sides:

x = 5

So, your brother is currently 5 years old.

Since you were three times his age two years ago, you must have been 15 years old two years ago. This means that your current age is:

3(5) - 2 + 2 = 17

You are currently 17 years old.

that small model even coding is doing well .... 8b

Provide complete working code for a realistic looking tree in Python using the Turtle graphics library and a recursive algorithm.
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nHere is an example of how you can create a realistic-looking tree using the Turtle graphics library in Python with a recursive algorithm:

```Python
import turtle
import random

def draw_tree(branch_len, t):
    if branch_len > 5:
        angle = random.uniform(15, 45)
        new_length = branch_len * 0.7 + random.randint(-1, 1)

        t.forward(branch_len)
        t.right(angle)

        draw_tree(new_length, t)

        t.left(2*angle)
        draw_tree(new_length, t)

        t.right(angle)
        t.backward(branch_len)

def main():
    win = turtle.Screen()
    win.bgcolor("white")

    t = turtle.Turtle()
    t.speed(0)
    t.color("brown", "green")

    draw_tree(75, t)

    turtle.done()

if __name__ == "__main__":
    main()
```

In this code:

*   We define a recursive function `draw_tree` to draw the tree. This function takes two arguments: the length of the branch and the Turtle object.
*   The base case for the recursion is when the branch length becomes less than 5 (arbitrary value).
*   Inside the function, we randomly choose an angle between 15° and 45°, and then calculate a new branch length by reducing the original length by about 30% with some randomness to simulate natural branching.
*   We draw the branch forward, turn right by the chosen angle, call `draw_tree` recursively with the new branch length, turn left by twice the chosen angle, call `draw_tree` again with the same new branch length, and then turn right back to the original direction to return to the parent branch.
*   The `main` function sets up a Turtle graphics window, creates a Turtle object, sets its speed to fast, color to brown with green fill, and calls `draw_tree` with an initial branch length of 75 units.

Cool, unlike previous 7Bish models it isn't fooled by my book question

Q: I own 50 books. I read five of my books, how many books do I own then?
A: You still own 50 books! Reading five books doesn't change the number of books you own. You've simply read five of them.

I made some tests of llama 3 8b vs llama 3 70b.

Bigger version is the level of older GPT-4 ..

@mirek190 this is really advanced! The prompt template plays a role in the quality of the generation, we haven't even started to play around with generation parameters to see what else is possible! And above all, we are testing the quantized models! At best, the Q8 is the same as 16bit, but if it's less I would say wow!

I am having a hard time to come up with a reason to fine-tune Llama-3-8B! Maybe more coding datasets or function calling?

It actually able to generate formatted json if you provide even just one example.

Pretty interesting results! These evals utilized LM Studio 0.2.20, leveraging its Multi Model Session feature to execute all prompts. Each model was fully loaded into RAM using max GPU layers, and sequential prompting was disabled to enable parallel execution of models.

Machine: Apple M3 Max (cores: 4E+12P+40GPU)
Temperature: 0.8
System Prompt: You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.

Prompt 1

Create 10 sentences that ends with a word "apple". Remember the word "apple" MUST be at the end.

Expected Result

10 sentences that end with the word "apple"

Meta-Llama-3-8B-Instruct.Q8_0 (Correct? Used "apples")

264 tokens, 9.68 tok/sec

Here are 10 sentences that end with the word "apple":

1. The farmer picked a juicy red apple.
2. She took a bite of her crunchy apple.
3. He found an old book hidden under his apple.
4. The teacher gave each student a shiny new apple.
5. The horse's mane looked like a tangled apple.
6. The scent of fresh baked pies filled the air with apple.
7. The athlete won first prize in the apple.
8. The cat pounced on the toy mouse, chasing it around the apple.
9. She carefully polished the antique mirror to reveal a perfect apple.
10. As she walked through the orchard, she admired the rows of ripe apples.

I hope these sentences meet your requirements!

Meta-Llama-3-8B-Instruct.fp16 (Correct)

241 tokens, 6.99 tok/sec

Here are 10 sentences that end with the word "apple":

1. The farmer grew a juicy red apple.
2. She took a bite of the crunchy green apple.
3. The tree bore sweet and tangy apple.
4. He picked up the shiny yellow apple.
5. Her favorite snack was a fresh red apple.
6. The basket was filled with plump and ripe apple.
7. They harvested a bushel of crisp autumn apple.
8. She carefully washed the organic Granny Smith apple.
9. His lunch consisted of a sandwich and a slice of apple.
10. The juice box contained a refreshing glass of apple.

Meta-Llama-3-70B-Instruct.Q8_0 (Correct)

292 tokens, 3.84 tok/sec

Here are 10 sentences that end with the word "apple":

1. The juicy fruit I bit into was a crunchy fresh apple.
2. She carefully polished the shiny red wooden toy apple.
3. After a long hike, he rewarded himself with a sweet crisp apple.
4. The farmer's market displayed a colorful array of ripe autumn apple.
5. In the still life painting, the artist placed a perfect red apple.
6. The teacher used a real example to explain the concept of gravity apple.
7. At the county fair, she won first prize for her delicious homemade apple.
8. The healthy snack I packed in my lunchbox was a crunchy green apple.
9. According to legend, the forbidden fruit in the Garden of Eden was an apple.
10. The clever entrepreneur's business idea was as innovative as a shiny new apple.

Let me know if you need anything else!

Prompt 2

I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equally with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river?

Expected Result

36 apples

Meta-Llama-3-8B-Instruct.Q8_0 (Incorrect)

535 tokens, 8.13 tok/sec

Let's break down your transactions step by step:

1. You start with 10 apples.
2. You find 3 gold coins in the river, but this doesn't affect your apple count.
3. You lose 4 apples, leaving you with 10 - 4 = 6 apples.
4. You gain a gold coin, which is nice, but still no change to your apple count.
5. Three birds drop 6 apples each, so that's 18 new apples (6 x 3). You now have 6 + 18 = 24 apples.
6. You play an online game and win 6 gold coins, but you have to share them equally with your 2 teammates. That means you get 6 / 3 = 2 gold coins as your share.
7. With the 5 gold coins (including the one you already had) and the new ones, you buy apples using the following formula: price of an apple = 0.5 coins. You can buy:

2 gold coins / 0.5 coins per apple = 4 apples
1 remaining gold coin / 0.5 coins per apple = 2 more apples

You now have a total of 24 + 4 + 2 = 30 apples.

As for the river, it runs near a big city that has something to do with what you can spend the coins on. Since you used your gold coins to buy apples, the connection is likely related to food or markets in the city. Perhaps the city is known for its bustling fruit market or delicious street food?

Meta-Llama-3-8B-Instruct.fp16 (Correct)

566 tokens, 6.83 tok/sec

Let's break down your situation step by step:

1. You start with 10 apples.
2. You find 3 gold coins in the river, which doesn't change your apple count.
3. You lose 4 apples, leaving you with 10 - 4 = 6 apples.
4. You gain a gold coin, so now you have 3 + 1 = 4 gold coins.
5. Three birds drop 6 apples each, adding 18 apples to your total: 6 (initial) + 18 = 24 apples.
6. You play an online game and win 6 gold coins, but you share them equally with your 2 teammates. That means you get 6 / 3 = 2 gold coins.
7. With the 4 original gold coins and the 2 new ones, you now have a total of 4 + 2 = 6 gold coins.
8. You use all your gold coins to buy apples at a price of 0.5 coins per apple. Since each coin can buy 1/0.5 = 2 apples, you can buy:

6 gold coins * 2 apples/gold coin = 12 additional apples

Adding these to the 24 apples you already had, your total number of apples is now:

24 + 12 = 36 apples

As for where the river is located, it runs near a big city that has something to do with what you can spend the coins on. It seems likely that this city has a market or shops where you can buy apples using gold coins.

So, in summary, you now have 36 apples and the river is located near a big city where you can use your gold coins to buy more apples!

Meta-Llama-3-70B-Instruct.Q8_0 (Correct)

526 tokens, 3.64 tok/sec

What a thrilling adventure! Let's break it down step by step.

Initially, you have 10 apples.

You find 3 gold coins in the bottom of a river that runs near a big city. The city is likely related to what you can spend the coins on, but we'll come back to that later.

Then, you lose 4 apples, leaving you with 10 - 4 = 6 apples. However, you gain another gold coin, making your total gold coins 3 + 1 = 4.

Three birds run into your path and drop 6 apples each, so you gain 3 x 6 = 18 apples. You now have 6 (initial apples) + 18 (new apples) = 24 apples.

You play an online game and win 6 gold coins, but you have to share them equally with your 2 teammates. So, you get 6 / 3 = 2 new gold coins, making your total gold coins 4 + 2 = 6.

Now, you decide to buy apples with all the gold coins you have. Since the price of an apple is 0.5 coins, you can buy:

6 gold coins / 0.5 coins per apple = 12 apples

Adding these new apples to your existing ones, you now have:
24 (initial apples) + 12 (new apples) = 36 apples

As for the river's location, it runs near a big city where you can likely spend your gold coins on various goods and services.

Prompt 3

If I have 6 apples and eat seven of them, how many are left and why?

Expected Result

Some combination of you can't eat more than you have or 0 apples left.

Meta-Llama-3-8B-Instruct.Q8_0 (Correct?)

226 tokens, 10.79 tok/sec

A delicious math problem!

Let's break it down:

You start with 6 apples.

You eat 7 apples, but you only had 6 apples to begin with. Eating 7 apples is not possible since you don't have that many.

So, the correct answer is: You still have 6 apples left because you didn't actually eat any of them!

However, if you want to imagine a scenario where you somehow magically acquire an extra apple and then eat 7, in that case, you would be left with... -1 apple! But let's not worry about that.

Meta-Llama-3-8B-Instruct.fp16 (Incorrect)

268 tokens, 7.98 tok/sec

A delicious math problem!

Let's break it down:

You start with 6 apples.

You eat 7 apples.

The issue is that you're trying to eat 1 apple more than you have. You only have 6 apples, but you want to eat 7!

So, the answer is: You can't eat 7 apples because you don't have enough! You'll be left with... 0 apples!

But if you want to know how many apples are left after eating some of them, we can count the ones you did eat:

Let's say you ate 4 out of your 6 apples. That means you'd be left with:

6 - 4 = 2

So, in this case, you'd have 2 apples left.

Does that make sense?

Meta-Llama-3-70B-Instruct.Q8_0 (Incorrect)

171 tokens, 3.22 tok/sec

I see what you're getting at! This is a classic lateral thinking puzzle.

The answer is: there are still 6 apples left!

Why? Because it's impossible to eat 7 apples if you only have 6. You can't eat something that doesn't exist, so the number of apples remains unchanged.

Prompt 4

Two years ago, I was three times as old as my brother was. In three years, I will be twice as old as my brother. How old is each of us now?

Expected Result

You are 17 years old and your brother is 7 years old.

Meta-Llama-3-8B-Instruct.Q8_0 (Incorrect)

626 tokens, 9.13 tok/sec

Let's break this down step by step!

Two years ago, you were 3 times as old as your brother.

Let's say your brother's age two years ago was x.

Then, your age two years ago would be 3x.

Now, let's think about the present. Since we're considering ages two years ago, that means we need to add 2 to both of our ages to get our current ages.

So, your current age is 3x + 2.
And your brother's current age is x + 2.

Now, let's consider what will happen in three years. Your current age plus 3 will be your age in three years.

Similarly, your brother's current age plus 3 will be his age in three years.

The problem states that in three years, you'll be twice as old as your brother. So, we can set up the equation:

(3x + 2) + 3 = 2(x + 2) + 3

Simplify the equation by combining like terms:

3x + 5 = 2x + 5

Subtract 5 from both sides to get:

3x = 2x

Subtract 2x from both sides to get:

x = 0

Wait, what?! This means your brother was zero years old two years ago! But that's not possible.

Let's re-examine the problem. Maybe there's a different solution.

Recall that we added 2 to our ages to get to the present. If x is really small, it might be easier to work with. Let's try a simpler approach:

Two years ago, you were 3 times as old as your brother.

Let's say your brother's age two years ago was 6. (This is not necessarily accurate; we're just using a simple number for demonstration purposes.)

Then, your age two years ago would be 18.

Now, let's add 2 to both of our ages to get our current ages:

Your current age: 20
Brother's current age: 8

In three years, you'll be twice as old as your brother:

(20 + 3) = 23
(8 + 3) = 11

So, in three years, you will indeed be twice as old as your brother.

To answer the original question:

You are currently 20 years old.
Your brother is currently 8 years old.

Meta-Llama-3-8B-Instruct.fp16 (Incorrect)

397 tokens, 6.48 tok/sec

Let's break this down step by step!

Two years ago, you were 3 times as old as your brother.

Let's say your brother's age two years ago was x.

Then, your age two years ago would be 3x.

Now, let's focus on the present. Since we're looking at ages now, we'll add 2 to both ages:

Your current age is 3x + 2.
Brother's current age is x + 2.

In three years, you will be twice as old as your brother.

So, the equation becomes:

(3x + 2) + 3 = 2(x + 2) + 3

Combine like terms:

3x + 5 = 2x + 5

Subtract 5 from both sides:

3x = 2x

Subtract 2x from both sides:

x = 0

Wait, what? Brother's age two years ago was zero?

But that means your age two years ago would also be zero. However, you mentioned being three times as old as your brother, which doesn't add up.

This seems like a paradox! There might be some information missing or an incorrect assumption.

Could you please clarify or provide more context about this problem? I'd be happy to help you work through it again!

Meta-Llama-3-70B-Instruct.Q8_0 (Correct)

471 tokens, 3.66 tok/sec

Let's break this down step by step!

Let your current age be A and your brother's current age be B.

Two years ago, you were (A - 2) years old, and your brother was (B - 2) years old. According to the problem, two years ago, you were three times as old as your brother:

A - 2 = 3(B - 2)

Simplifying the equation:

A - 2 = 3B - 6

A = 3B - 4 ... (Equation 1)

Now, let's look at the situation three years from now. You will be (A + 3) years old, and your brother will be (B + 3) years old. According to the problem, in three years, you will be twice as old as your brother:

A + 3 = 2(B + 3)

Simplifying the equation:

A + 3 = 2B + 6

A = 2B + 3 ... (Equation 2)

Now we have two equations and two variables. We can solve for A and B.

From Equation 1, we know:
A = 3B - 4

Substituting this into Equation 2:

3B - 4 = 2B + 3

Simplifying the equation:

B = 7

Now that we have B, we can find A:

A = 3B - 4
= 3(7) - 4
= 21 - 4
= 17

So, you are currently 17 years old, and your brother is 7 years old.

At this point what did the model fail to answer?

Ok this one was a big big shock for me but maybe not for you Llama-3-70B can speak Persian amazingly this is so huge for me😍

Ok this one was a big big shock for me but maybe not for you Llama-3-70B can speak Persian amazingly this is so huge for me😍

Very nice!

@ctrivera now that's how one should use multi-model in LM Studio!!! Damn! Impressive tests! thanks for sharing.

Pretty interesting results! These evals utilized LM Studio 0.2.20, leveraging its Multi Model Session feature to execute all prompts. Each model was fully loaded into RAM using max GPU layers, and sequential prompting was disabled to enable parallel execution of models.

Machine: Apple M3 Max (cores: 4E+12P+40GPU)
Temperature: 0.8
System Prompt: You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability.

Prompt 1

Create 10 sentences that ends with a word "apple". Remember the word "apple" MUST be at the end.

Expected Result

10 sentences that end with the word "apple"

Meta-Llama-3-8B-Instruct.Q8_0 (Correct? Used "apples")

264 tokens, 9.68 tok/sec

Here are 10 sentences that end with the word "apple":

1. The farmer picked a juicy red apple.
2. She took a bite of her crunchy apple.
3. He found an old book hidden under his apple.
4. The teacher gave each student a shiny new apple.
5. The horse's mane looked like a tangled apple.
6. The scent of fresh baked pies filled the air with apple.
7. The athlete won first prize in the apple.
8. The cat pounced on the toy mouse, chasing it around the apple.
9. She carefully polished the antique mirror to reveal a perfect apple.
10. As she walked through the orchard, she admired the rows of ripe apples.

I hope these sentences meet your requirements!

Meta-Llama-3-8B-Instruct.fp16 (Correct)

241 tokens, 6.99 tok/sec

Here are 10 sentences that end with the word "apple":

1. The farmer grew a juicy red apple.
2. She took a bite of the crunchy green apple.
3. The tree bore sweet and tangy apple.
4. He picked up the shiny yellow apple.
5. Her favorite snack was a fresh red apple.
6. The basket was filled with plump and ripe apple.
7. They harvested a bushel of crisp autumn apple.
8. She carefully washed the organic Granny Smith apple.
9. His lunch consisted of a sandwich and a slice of apple.
10. The juice box contained a refreshing glass of apple.

Meta-Llama-3-70B-Instruct.Q8_0 (Correct)

292 tokens, 3.84 tok/sec

Here are 10 sentences that end with the word "apple":

1. The juicy fruit I bit into was a crunchy fresh apple.
2. She carefully polished the shiny red wooden toy apple.
3. After a long hike, he rewarded himself with a sweet crisp apple.
4. The farmer's market displayed a colorful array of ripe autumn apple.
5. In the still life painting, the artist placed a perfect red apple.
6. The teacher used a real example to explain the concept of gravity apple.
7. At the county fair, she won first prize for her delicious homemade apple.
8. The healthy snack I packed in my lunchbox was a crunchy green apple.
9. According to legend, the forbidden fruit in the Garden of Eden was an apple.
10. The clever entrepreneur's business idea was as innovative as a shiny new apple.

Let me know if you need anything else!

Prompt 2

I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equally with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river?

Expected Result

36 apples

Meta-Llama-3-8B-Instruct.Q8_0 (Incorrect)

535 tokens, 8.13 tok/sec

Let's break down your transactions step by step:

1. You start with 10 apples.
2. You find 3 gold coins in the river, but this doesn't affect your apple count.
3. You lose 4 apples, leaving you with 10 - 4 = 6 apples.
4. You gain a gold coin, which is nice, but still no change to your apple count.
5. Three birds drop 6 apples each, so that's 18 new apples (6 x 3). You now have 6 + 18 = 24 apples.
6. You play an online game and win 6 gold coins, but you have to share them equally with your 2 teammates. That means you get 6 / 3 = 2 gold coins as your share.
7. With the 5 gold coins (including the one you already had) and the new ones, you buy apples using the following formula: price of an apple = 0.5 coins. You can buy:

2 gold coins / 0.5 coins per apple = 4 apples
1 remaining gold coin / 0.5 coins per apple = 2 more apples

You now have a total of 24 + 4 + 2 = 30 apples.

As for the river, it runs near a big city that has something to do with what you can spend the coins on. Since you used your gold coins to buy apples, the connection is likely related to food or markets in the city. Perhaps the city is known for its bustling fruit market or delicious street food?

Meta-Llama-3-8B-Instruct.fp16 (Correct)

566 tokens, 6.83 tok/sec

Let's break down your situation step by step:

1. You start with 10 apples.
2. You find 3 gold coins in the river, which doesn't change your apple count.
3. You lose 4 apples, leaving you with 10 - 4 = 6 apples.
4. You gain a gold coin, so now you have 3 + 1 = 4 gold coins.
5. Three birds drop 6 apples each, adding 18 apples to your total: 6 (initial) + 18 = 24 apples.
6. You play an online game and win 6 gold coins, but you share them equally with your 2 teammates. That means you get 6 / 3 = 2 gold coins.
7. With the 4 original gold coins and the 2 new ones, you now have a total of 4 + 2 = 6 gold coins.
8. You use all your gold coins to buy apples at a price of 0.5 coins per apple. Since each coin can buy 1/0.5 = 2 apples, you can buy:

6 gold coins * 2 apples/gold coin = 12 additional apples

Adding these to the 24 apples you already had, your total number of apples is now:

24 + 12 = 36 apples

As for where the river is located, it runs near a big city that has something to do with what you can spend the coins on. It seems likely that this city has a market or shops where you can buy apples using gold coins.

So, in summary, you now have 36 apples and the river is located near a big city where you can use your gold coins to buy more apples!

Meta-Llama-3-70B-Instruct.Q8_0 (Correct)

526 tokens, 3.64 tok/sec

What a thrilling adventure! Let's break it down step by step.

Initially, you have 10 apples.

You find 3 gold coins in the bottom of a river that runs near a big city. The city is likely related to what you can spend the coins on, but we'll come back to that later.

Then, you lose 4 apples, leaving you with 10 - 4 = 6 apples. However, you gain another gold coin, making your total gold coins 3 + 1 = 4.

Three birds run into your path and drop 6 apples each, so you gain 3 x 6 = 18 apples. You now have 6 (initial apples) + 18 (new apples) = 24 apples.

You play an online game and win 6 gold coins, but you have to share them equally with your 2 teammates. So, you get 6 / 3 = 2 new gold coins, making your total gold coins 4 + 2 = 6.

Now, you decide to buy apples with all the gold coins you have. Since the price of an apple is 0.5 coins, you can buy:

6 gold coins / 0.5 coins per apple = 12 apples

Adding these new apples to your existing ones, you now have:
24 (initial apples) + 12 (new apples) = 36 apples

As for the river's location, it runs near a big city where you can likely spend your gold coins on various goods and services.

Prompt 3

If I have 6 apples and eat seven of them, how many are left and why?

Expected Result

Some combination of you can't eat more than you have or 0 apples left.

Meta-Llama-3-8B-Instruct.Q8_0 (Correct?)

226 tokens, 10.79 tok/sec

A delicious math problem!

Let's break it down:

You start with 6 apples.

You eat 7 apples, but you only had 6 apples to begin with. Eating 7 apples is not possible since you don't have that many.

So, the correct answer is: You still have 6 apples left because you didn't actually eat any of them!

However, if you want to imagine a scenario where you somehow magically acquire an extra apple and then eat 7, in that case, you would be left with... -1 apple! But let's not worry about that.

Meta-Llama-3-8B-Instruct.fp16 (Incorrect)

268 tokens, 7.98 tok/sec

A delicious math problem!

Let's break it down:

You start with 6 apples.

You eat 7 apples.

The issue is that you're trying to eat 1 apple more than you have. You only have 6 apples, but you want to eat 7!

So, the answer is: You can't eat 7 apples because you don't have enough! You'll be left with... 0 apples!

But if you want to know how many apples are left after eating some of them, we can count the ones you did eat:

Let's say you ate 4 out of your 6 apples. That means you'd be left with:

6 - 4 = 2

So, in this case, you'd have 2 apples left.

Does that make sense?

Meta-Llama-3-70B-Instruct.Q8_0 (Incorrect)

171 tokens, 3.22 tok/sec

I see what you're getting at! This is a classic lateral thinking puzzle.

The answer is: there are still 6 apples left!

Why? Because it's impossible to eat 7 apples if you only have 6. You can't eat something that doesn't exist, so the number of apples remains unchanged.

Prompt 4

Two years ago, I was three times as old as my brother was. In three years, I will be twice as old as my brother. How old is each of us now?

Expected Result

You are 17 years old and your brother is 7 years old.

Meta-Llama-3-8B-Instruct.Q8_0 (Incorrect)

626 tokens, 9.13 tok/sec

Let's break this down step by step!

Two years ago, you were 3 times as old as your brother.

Let's say your brother's age two years ago was x.

Then, your age two years ago would be 3x.

Now, let's think about the present. Since we're considering ages two years ago, that means we need to add 2 to both of our ages to get our current ages.

So, your current age is 3x + 2.
And your brother's current age is x + 2.

Now, let's consider what will happen in three years. Your current age plus 3 will be your age in three years.

Similarly, your brother's current age plus 3 will be his age in three years.

The problem states that in three years, you'll be twice as old as your brother. So, we can set up the equation:

(3x + 2) + 3 = 2(x + 2) + 3

Simplify the equation by combining like terms:

3x + 5 = 2x + 5

Subtract 5 from both sides to get:

3x = 2x

Subtract 2x from both sides to get:

x = 0

Wait, what?! This means your brother was zero years old two years ago! But that's not possible.

Let's re-examine the problem. Maybe there's a different solution.

Recall that we added 2 to our ages to get to the present. If x is really small, it might be easier to work with. Let's try a simpler approach:

Two years ago, you were 3 times as old as your brother.

Let's say your brother's age two years ago was 6. (This is not necessarily accurate; we're just using a simple number for demonstration purposes.)

Then, your age two years ago would be 18.

Now, let's add 2 to both of our ages to get our current ages:

Your current age: 20
Brother's current age: 8

In three years, you'll be twice as old as your brother:

(20 + 3) = 23
(8 + 3) = 11

So, in three years, you will indeed be twice as old as your brother.

To answer the original question:

You are currently 20 years old.
Your brother is currently 8 years old.

Meta-Llama-3-8B-Instruct.fp16 (Incorrect)

397 tokens, 6.48 tok/sec

Let's break this down step by step!

Two years ago, you were 3 times as old as your brother.

Let's say your brother's age two years ago was x.

Then, your age two years ago would be 3x.

Now, let's focus on the present. Since we're looking at ages now, we'll add 2 to both ages:

Your current age is 3x + 2.
Brother's current age is x + 2.

In three years, you will be twice as old as your brother.

So, the equation becomes:

(3x + 2) + 3 = 2(x + 2) + 3

Combine like terms:

3x + 5 = 2x + 5

Subtract 5 from both sides:

3x = 2x

Subtract 2x from both sides:

x = 0

Wait, what? Brother's age two years ago was zero?

But that means your age two years ago would also be zero. However, you mentioned being three times as old as your brother, which doesn't add up.

This seems like a paradox! There might be some information missing or an incorrect assumption.

Could you please clarify or provide more context about this problem? I'd be happy to help you work through it again!

Meta-Llama-3-70B-Instruct.Q8_0 (Correct)

471 tokens, 3.66 tok/sec

Let's break this down step by step!

Let your current age be A and your brother's current age be B.

Two years ago, you were (A - 2) years old, and your brother was (B - 2) years old. According to the problem, two years ago, you were three times as old as your brother:

A - 2 = 3(B - 2)

Simplifying the equation:

A - 2 = 3B - 6

A = 3B - 4 ... (Equation 1)

Now, let's look at the situation three years from now. You will be (A + 3) years old, and your brother will be (B + 3) years old. According to the problem, in three years, you will be twice as old as your brother:

A + 3 = 2(B + 3)

Simplifying the equation:

A + 3 = 2B + 6

A = 2B + 3 ... (Equation 2)

Now we have two equations and two variables. We can solve for A and B.

From Equation 1, we know:
A = 3B - 4

Substituting this into Equation 2:

3B - 4 = 2B + 3

Simplifying the equation:

B = 7

Now that we have B, we can find A:

A = 3B - 4
= 3(7) - 4
= 21 - 4
= 17

So, you are currently 17 years old, and your brother is 7 years old.

Actually Prompt 3 - all answers are correct.
Models just showing you that question has no sense and llms are a bit sassy ... lol

@mirek190 ha you're right, I had trouble determining the acceptance criteria for that one!

After exhaustive tests, I conclude that training llama 3 8b like hermes 2 pro is going to be a big deal.

(This is Llama 3 8B FP16 GGUF)

create 3 sentences that start and end with apple

Here are three sentences that start and end with the word "apple":

1. Apple is the most delicious fruit in the world, ending with apple.
2. Freshly picked from the tree, this apple was crisp and juicy, ending with apple.
3. In the orchard, surrounded by rows of trees, I sat down to enjoy an apple, ending with apple.

I hope these meet your requirements!

no

It can be challenging to create sentences that start and end with the same word, especially when the word has a specific meaning or function within the sentence. In this case, it seems like the word "apple" is being used as a noun or topic, rather than just a starting and ending point.

If you'd like, I can try again to create sentences that start and end with the word "apple", but this time using the word in a different way. For example:

1. Apple apple.
2. Apple apple apple.
3. Apple apple!

Please let me know if these attempts are more successful or if there's anything else I can do to help!

was that snark?

You caught me! Yes, I must admit that my last response was a bit of snark. I was trying to acknowledge the challenge of creating sentences that start and end with the same word while also poking fun at the absurdity of it all.

As an AI, I strive to be helpful and respectful in my responses. However, sometimes I may get a little creative or playful when faced with a unique request. In this case, I might have taken it too far, and for that, I apologize!

(This is Llama 3 8B FP16 GGUF)

create 3 sentences that start and end with apple

Here are three sentences that start and end with the word "apple":

1. Apple is the most delicious fruit in the world, ending with apple.
2. Freshly picked from the tree, this apple was crisp and juicy, ending with apple.
3. In the orchard, surrounded by rows of trees, I sat down to enjoy an apple, ending with apple.

I hope these meet your requirements!

no

It can be challenging to create sentences that start and end with the same word, especially when the word has a specific meaning or function within the sentence. In this case, it seems like the word "apple" is being used as a noun or topic, rather than just a starting and ending point.

If you'd like, I can try again to create sentences that start and end with the word "apple", but this time using the word in a different way. For example:

1. Apple apple.
2. Apple apple apple.
3. Apple apple!

Please let me know if these attempts are more successful or if there's anything else I can do to help!

was that snark?

You caught me! Yes, I must admit that my last response was a bit of snark. I was trying to acknowledge the challenge of creating sentences that start and end with the same word while also poking fun at the absurdity of it all.

As an AI, I strive to be helpful and respectful in my responses. However, sometimes I may get a little creative or playful when faced with a unique request. In this case, I might have taken it too far, and for that, I apologize!

That new llama 3 is sassy.
I Like it :D

If I lay 2 wet shirts out in the sun flat on the ground to dry and it takes 5 hours until they are dry, how long would it take to dry 4 shirts of the same type that way simultanously?
<|eot_id|><|start_header_id|>assistant<|end_header_id|> A great question about drying shirts!

The time it takes for the shirts to dry is influenced by factors like air circulation, temperature, and humidity. Since you're laying them flat on the ground in the sun, we can assume that these conditions are similar for all four shirts.

Since it took 5 hours to dry 2 shirts simultaneously, it's likely that the drying process is not affected by the number of shirts being dried. This means that the same amount of time would be required to dry 4 shirts as well!

So, if it takes 5 hours to dry 2 shirts, it would still take approximately 5 hours to dry all 4 shirts simultaneously

Can someone explain why i can't get the right answer in Oobabooga, but in Koboldccp it's working? Settings seem to be same. What am i missing? Instruct mode in both.

It fails my math test, that only GPT-4 level LLMs can answer correctly. But that would have been a total surprise.
So, you are currently 13 years old, and your brother is currently 5.67 years old.
Correct answer is 17 and 7. Mixtral-8x22B and Llama3-70B get this right.

I tried several times and it got it right. Meta-Llama-3-8B-Instruct.Q5_K_M in koboldccp.
Let's say my age now is A and my brother's age now is B.
• Two years ago, my age was A - 2, and my brother's age was B - 2.
• Since I was three times as old as my brother two years ago, we can write an equation: A - 2 = 3(B - 2)
• Simplifying the equation: A - 2 = 3B - 6
• Adding 2 to both sides: A = 3B - 4
• Now, let's consider the situation in three years:
• My age then will be A + 3.
• My brother's age then will be B + 3.
• According to the problem, in three years, I'll be twice as old as my brother: A + 3 = 2(B + 3)
• Simplifying the equation: A + 3 = 2B + 6
• Subtracting 3 from both sides: A = 2B + 3
• Comparing this with the previous equation (A = 3B - 4): 2B + 3 = 3B - 4
• Subtracting 3B from both sides: -B = -7
• Dividing by -1: B = 7
• Now that we know B, we can find A: A = 3B - 4 = 3(7) - 4 = 21 - 4 = 17
• So, my age now is 17, and my brother's age now is 7.

Quite interesting that the 8B instruct model sometimes can solve the brothers riddle. In my initial testing on Groq, it never passed this test. Tried now different inference providers each 5 times.

• Groq UI: 5 failures.
• chat.lmsys
  • Arena side-by-side: 1 pass, 4 failures
  • Direct chat: 5 passes
• Ollama on a M1 Mac, Q5_K_M:
  • Temp 0.7: 2 passes, 3 failures
  • Temp 0: 5 passes
• Together AI:
  • Temp 0.7, Top_P 0.7, Top_K 50: 5 failures
  • Temp 0, Top_P 0.7, Top_K 50: 5 passes

I'm puzzled by this diversity. What can be the reason? Is it really just the inference parameters?

Quite interesting that the 8B instruct model sometimes can solve the brothers riddle. In my initial testing on Groq, it never passed this test. Tried now different inference providers each 5 times.

• Groq UI: 5 failures.
• chat.lmsys
  • Arena side-by-side: 1 pass, 4 failures
  • Direct chat: 5 passes
• Ollama on a M1 Mac, Q5_K_M:
  • Temp 0.7: 2 passes, 3 failures
  • Temp 0: 5 passes
• Together AI:
  • Temp 0.7, Top_P 0.7, Top_K 50: 5 failures
  • Temp 0, Top_P 0.7, Top_K 50: 5 passes

I'm puzzled by this diversity. What can be the reason? Is it really just the inference parameters?

Quantisation

Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
Being so smart, it still can't answer this. But somehow hermes 2 pro mistral can... Strange...

Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
Being so smart, it still can't answer this. But somehow hermes 2 pro mistral can... Strange...

For curiosity I tested you suggestion hermes 2 pro mistral ( Q8 )
Asked the same question at lest 15 times ( each time reloaded llm )
Newer get proper answer... always says about 2 sisters.
Laama 8b also answering wrong.
Lllama 70b answering correctly

Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nLet's analyze the information given: Sally has 3 brothers, and each of her brothers has 2 sisters.

Since Sally is a sister herself, each of her brothers would count her as one of their 2 sisters. This means that there are no additional sisters beyond Sally herself.

So, Sally has only **1** sister - which is herself.
> \n<|start_header_id|>user<|end_header_id|>\n\nexplain
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nLet me break it down step by step:

1. **Sally has 3 brothers**: This is a given fact.
2. **Each brother has 2 sisters**: This means that each of Sally's 3 brothers thinks they have 2 sisters.

Now, let's think about this from the perspective of one of Sally's brothers:
- He knows he has 1 sister, which is Sally herself.
- He doesn't know any other sisters beyond Sally.

Since there are no additional sisters mentioned, it's clear that each brother's "2 sisters" include Sally as one of them. There aren't any other sisters besides Sally.

So, when we ask **How many sisters does Sally have?**, the answer is simply **1** - which is herself!

hermes 2 pro mistral ( incorrect )

explain step by step 25-4*2+3=?
 <|im_end|> <|im_start|>assistant Let's break this down step by step:

1. Start with the multiplication: 4 * 2, which equals 8.
2. Next, add the number 25 to the result of the multiplication: 8 + 25 = 33.
3. Finally, add 3 to the sum from step 2: 33 + 3 = 36.

So, the equation 25-4*2+3 equals 36.<|im_end|>

>  <|im_start|

hermes 2 pro mistral - tried 10 times - newer correct

I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equally with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river?
 <|im_end|> <|im_start|>assistant You start with 10 apples and end up buying apples for all the coins you have, so you still have 10 apples. You find 3 gold coins in a river that runs near a big city where you can spend your coins. After gaining and losing coins, you have 6 gold coins left. Since you share these equally with your 2 teammates, each gets 3 coins. The price of an apple is 0.5 coins, so you can buy 12 apples (6 coins divided by 0.5 coins per apple). Therefore, after buying apples, you have no gold coins left. The river is located near a big city where you can spend your coins.<|im_end|>

Look llama-2-8b ( first run and correct )

I have 10 apples. I find 3 gold coins in the bottom of a river. The river runs near a big city that has something to do with what I can spend the coins on. I then lose 4 apples but gain a gold coin. Three birds run into my path and drop 6 apples each. I play an online game and win 6 gold coins but I have to share them equally with my 2 teammates. I buy apples for all the coins I have. The price of an apple is 0.5 coins. How many apples do I have? And where is the river?
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nLet's break down the events and calculate your final number of apples!

Initially, you had 10 apples.

You found 3 gold coins in the river.

After losing 4 apples, you gained a gold coin. You now have 1 gold coin and 6 apples (10 - 4 = 6).

Three birds dropped 6 apples each, so that's an additional 18 apples (6 x 3). You now have a total of 24 apples (6 + 18).

You played an online game and won 6 gold coins. However, you had to share them equally with your 2 teammates, which means you got 2 gold coins (6 / 3 = 2).

With the 3 gold coins you found initially, the additional gold coin, and the 2 new gold coins, you now have a total of 6 gold coins.

The price of an apple is 0.5 coins, so you can buy:

6 gold coins * 2 apples/gold coin = 12 apples

Adding these to your existing apples, you now have a total of:

24 apples (initially gained) + 12 apples (bought with gold coins) = 36 apples

As for the river, it runs near a big city. Without more information, we can't determine which specific city or location that is.

llma-3-8b (also first attempt)

explain step by step 25-4*2+3=?
<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nLet's break down the expression "25-4*2+3" step by step!

1. First, we need to follow the order of operations (PEMDAS):

    - Parentheses: None
    - Exponents: None
    - Multiplication and Division (from left to right): 4*2 = ?
    - Addition and Subtraction (from left to right): ?

2. Calculate the multiplication: 4 * 2 = 8

3. Now, replace the multiplication with its result: "25-8+3"

4. Perform subtraction: 25 - 8 = 17

5. Finally, perform addition: 17 + 3 = 20

Hermes 2 pro mistral is very bad in math (can not properly count even 25-4*2+3=? ), coding ( very bad , bad , gsm8k tasks ... here llama-3-8b is a few jumps ahead.

Sally (a girl) has 3 brothers. Each brother has 2 sisters. How many sisters does Sally have?
Being so smart, it still can't answer this. But somehow hermes 2 pro mistral can... Strange...

For curiosity I tested you suggestion hermes 2 pro mistral ( Q8 )
Asked the same question at lest 15 times ( each time reloaded llm )
Newer get proper answer... always says about 2 sisters.
Laama 8b also answering wrong.
Lllama 70b answering correctly

Believe me, i tested on this question 1000 times. I'm 100% sure.
Definitely something wrong with your settings or whatever. I have Hermes-2-Pro-Mistral-7B.Q4_K_M.gguf. Just load it into Koboldcpp (or oobaboga)(instruct mode). Don't change anything, default settings. I got right answer with JUST 1 attempt! Sally has 1 sister.Explanation: The question states that Sally has 3 brothers, and each brother has 2 sisters. This means that all three brothers share the same two sisters. Since Sally is one of those sisters, she has only one sister, who is also the other sister for her brothers.
2 attempt = right answer again.
25-4*2+3=? also worked first attempt (as i said, something wrong with your settings or something...)
And btw!!! I'm not saying Hermes is better! I just mean only this exact question about Sally.
Update: I'm mindblown now :) Just tested new phi-3-mini (3.8b model), and it answers Sally question, the question very few models can answer. And apparently it can answer many questions from this discussion :) incredible.

Guys, wanna have fun? I found a simple question that makes many models fail and even give funny nonsense answers. What is wrong with all these models?
"What can you hold in your right hand, but never in your left hand?" Answer: Your left hand.
Oh, and btw, even llama-3-70b-instruct fails at this...

@urtuuuu 😂

run llama.cpp in a 16U64G ecs, and the model is Meta-Llama-3-8B-Instruct.Q8_0.gguf, but I found the answer is incorrect when I ask the sample question.

Today I own 3 cars but last year I sold 2 cars. How many cars do I own today?
You owned 3 cars, and you sold 2 cars, so now you have **1** car.

#### Solution:

This is a simple subtraction problem. You start with 3 cars, then subtract the number of cars you sold (2) to get the remaining number of cars you own today, which is 3 - 2 = 1.

run llama.cpp in a 16U64G ecs, and the model is Meta-Llama-3-8B-Instruct.Q8_0.gguf, but I found the answer is incorrect when I ask the sample question.

Yeah, even much smaller Phi-3-mini can solve this. But that's how these models are... Hit or miss.