arcee-ai

/

Arcee-Maestro-7B-Preview

like 35

Follow

Arcee AI 439

Sorry to break it to everyone, especially to those model creators who claim their ~7B model is good for coding, yet apparently never tested it for coding...

There's no such thing as ~7B model good for coding!!! Benchmarks are 💩. Real world testing always shows such claims like 7B model good for coding are blatant lies!

Wanna models that are somewhat decent for coding tasks? 32B are absolute entry point, meaning they MAY or MAY NOT figure out the problem with the code, but at the very least they will not screw it up even further (well not as much as smaller models that is)...

Can you give some coding scenarios to see how good it is? Because with this famous and pretty simple question looks like it fails, just like r1 distilled.

Explain the bug in the following code:

from time import sleep
from multiprocessing.pool import ThreadPool
 
def task():
    sleep(1)
    return 'all done'

if __name__ == '__main__':
    with ThreadPool() as pool:
        result = pool.apply_async(task())
        value = result.get()
        print(value)

Answer: Basically >>> result = pool.apply_async(task) not (task())

Everything i tried so far, non thinking models do better. Or am i using wrong temperature or something? It's 0.6

@MrDevolver DeepSeek-r1-distill-Qwen14B figured this out zero-shot (also: this was with temperature: 0.8). It's not just only about the number of parameters and the number of layers that a model has. Time-scaled inference works just as well. Now, albeit I'm using a Q5_K_M (GGUF) with the Output Tensors and Embeddings Tensors quantized to Q8_0, but still. The premise that only models with N parameters and M layers can only understand and figure out code does not always hold. This is being actively challenged by inference time-scaling as we speak. I hope this helps you guys. 🤔

AI:
The issue with the code is that task() is being called immediately when it's passed to pool.apply_async(task()). The way apply_async works is that you pass it the function object (not the result of calling the function), and it schedules the task to run in a separate process.

Here’s how the code is behaving:

1. When you call task() inside apply_async(task()), it immediately executes the sleep(1) and returns 'all done' before even entering the ThreadPool context.
2. Then, when you call result.get(), it just waits for that single result, which has already been computed synchronously.

To fix this, you should pass the function itself (task) without calling it to apply_async. Here’s the corrected version:

from time import sleep
from multiprocessing.pool import ThreadPool
 
def task():
    sleep(1)
    return 'all done'

if __name__ == '__main__':
    with ThreadPool() as pool:
        result = pool.apply_async(task)  # Pass the function, not the result of calling it
        value = result.get()
        print(value)

In this corrected version, task is scheduled to run in a separate process via the thread pool, and result.get() will wait for that asynchronous task to complete.

==== Below is the whole transcript including <think></thinking>...

Okay, let's see what the user is asking. They provided a Python script and asked what's wrong with it. I'll need to look at the code carefully.

Looking at the code, I notice they're using ThreadPool from multiprocessing.pool. The task function sleeps for 1 second and returns 'all done'. In the main block, they create a ThreadPool context manager. Then, they call pool.apply_async(task()), assign the result to 'result', get the value, and print it.

Hmm, wait a minute. I remember that apply_async is used for asynchronous execution. But when you call task() directly inside apply_async like this, task() gets executed immediately before even creating the pool. Oh right! Because function calls are evaluated when they're passed as arguments. So in this case, task() runs once right there, and then it's added to the pool again.

So that means instead of using the ThreadPool to parallelize tasks, we're actually running task twice: once synchronously when calling apply_async, and then another time through the pool. That's not what was intended. The user probably wanted to run the task asynchronously without blocking, but this code isn't doing that correctly.

The correct way is to pass the function reference to apply_async without calling it. So it should be apply_async(task), not task(). That way, the pool will handle the execution in a separate process.

I'll need to explain this clearly. The problem is that task() is called before being passed to apply_async, which defeats the purpose of using ThreadPool for parallel tasks. Instead, just pass the function so it can be executed asynchronously.

The issue with the code is that task() is being called immediately when it's passed to pool.apply_async(task()). The way apply_async works is that you pass it the function object (not the result of calling the function), and it schedules the task to run in a separate process.

Here’s how the code is behaving:

1. When you call task() inside apply_async(task()), it immediately executes the sleep(1) and returns 'all done' before even entering the ThreadPool context.
2. Then, when you call result.get(), it just waits for that single result, which has already been computed synchronously.

To fix this, you should pass the function itself (task) without calling it to apply_async. Here’s the corrected version:

from time import sleep
from multiprocessing.pool import ThreadPool
 
def task():
    sleep(1)
    return 'all done'

if __name__ == '__main__':
    with ThreadPool() as pool:
        result = pool.apply_async(task)  # Pass the function, not the result of calling it
        value = result.get()
        print(value)

In this corrected version, task is scheduled to run in a separate process via the thread pool, and result.get() will wait for that asynchronous task to complete.

Also: Here's NousResearch's DeepHermes-3-Llama-3.1-8B zero-shotting it as well. Keep it mind, this is just SFT - there is no RL for this model. @teknium and NousResearch did well. 😏

Model: DeepHermes-3-Llama-3.1-8B-Preview-OF32.EF32.IQ8_0.gguf

AI:
The issue with this code lies in how apply_async is used:

1. Incorrect Usage of apply_async:
   The line result = pool.apply_async(task()) invokes task() immediately, passing its return value ('all done') to apply_async. However, apply_async expects a callable (the function itself) as the first argument. To submit a task asynchronously, you should pass the unexecuted function instead of its result.

2. Fix:

   result = pool.apply_async(task)
   

Here's how to correct the code:

from time import sleep
from multiprocessing.pool import ThreadPool

def task():
    sleep(1)
    return 'all done'

if __name__ == '__main__':
    with ThreadPool() as pool:
        result = pool.apply_async(task)  # Pass `task` (function object) here
        value = result.get()
        print(value)

Explanation:

• When you use apply_async(func), it queues the function for asynchronous execution.
• By passing task() directly, the function is called immediately and its return value ('all done') becomes part of the argument list. This bypasses the intended asynchronous behavior.

The corrected code ensures that task is submitted as a callable object to the thread pool.

==== Below is the whole transcript including ...<think></thinking>
Okay, let's see. The user provided some Python code and wants to know what's wrong with it. Hmm. Let me look at the code.

First, they're importing sleep from time and ThreadPool from multiprocessing.pool. Then there's a task function that sleeps for 1 second and returns 'all done'. In the main block, they create a ThreadPool context using 'with', then try to apply_async by calling task(). But wait, here's a possible mistake. The pool.apply_async() expects an argument to be called as a callable with *args and **kwargs. So right now, they're directly calling task(), which executes it immediately, not passing the function itself. That would cause an error because result = pool.apply_async(task()) is actually invoking task() right away instead of submitting the unexecuted function.

Wait, let me confirm. In Python's multiprocessing.Pool or ThreadPool, apply_async takes a callable as its first argument. So if you do pool.apply_async(task()), it should work because task is passed as a reference to the function, not its result. But in this code, they have task() with parentheses, which calls the function immediately and returns 'all done'. Then applies that as the async call, but since apply_async needs the callable itself, not the return value. So the problem here is that instead of passing task (without invoking it) as the first argument to apply_async, they are passing the result of task(), which is a string.

Therefore, the correct way would be pool.apply_async(task), without parentheses. Because when you use with ThreadPool() as pool: it creates a context where pool is an instance of ThreadPool. Then, in that context, applying async would require the function call as a callable.

So the error here is that they are immediately invoking task(), which returns 'all done', and then trying to apply_async on that return value, which isn't a function. Instead, they should pass just task without parentheses so that apply_async gets the actual function object, not the result of executing it.

Another possible issue: The ThreadPool class in multiprocessing.pool is used for asynchronous execution via threads, but sometimes people confuse it with Pool from multiprocessing. But here, even if using ThreadPool correctly, the main problem is passing an already-executed task() instead of the unexecuted function.

So the code should be changed to pool.apply_async(task), not task(). That way, when apply_async is called, it passes the task function to be executed later in a thread. Then result.get() will wait for the asynchronous execution and get its value.

The issue with this code lies in how apply_async is used:

1. Incorrect Usage of apply_async:
   The line result = pool.apply_async(task()) invokes task() immediately, passing its return value ('all done') to apply_async. However, apply_async expects a callable (the function itself) as the first argument. To submit a task asynchronously, you should pass the unexecuted function instead of its result.

2. Fix:

   result = pool.apply_async(task)
   

Here's how to correct the code:

from time import sleep
from multiprocessing.pool import ThreadPool

def task():
    sleep(1)
    return 'all done'

if __name__ == '__main__':
    with ThreadPool() as pool:
        result = pool.apply_async(task)  # Pass `task` (function object) here
        value = result.get()
        print(value)

Explanation:

• When you use apply_async(func), it queues the function for asynchronous execution.
• By passing task() directly, the function is called immediately and its return value ('all done') becomes part of the argument list. This bypasses the intended asynchronous behavior.

The corrected code ensures that task is submitted as a callable object to the thread pool.