cognitivecomputations

/

Dolphin3.0-R1-Mistral-24B

like 82

Follow

Cognitive Computations 1.46k

Yes

That'd be great if you could, you know, elaborate?

@SerialKicked no it does not. The chat templates does not include tools. The original model did, but with dolphin its difficult, sine its mistral trained on chatml without tool calling in training.

/edit
I speified, I meant the original model not the original template, since Mistral did not include any tool calling capabilities in their chat_template in the first place either.

Okay, so no functional tool.call support.

Thank you for clarifying.

False. Nothing in the chat template prevents you from calling tools.

@ehartford Ill take your word for it, but I very much doubt it.
How were you able to successfully do tool calling let alone replicate OpenAI compatible tool calling?

My understanding is the following:

1. tokenizer_config.jsondoes not populate the tools list passed with the request as available tools into in the system prompt of the template. Also there is no tool role or any of the otherwise present elements of a tool calling templates for reference check this vLLM Office Hour.
   This would leave us with the option of injecting our own tool calling jinja template (see below) into i.e. vLLM. However...
2. ... the base model (mistral small v3) is trained on a completely different template (mistral) both expecting a different format for prompting available tools in the system prompt and for actually calling tools than ChatML, which is what you use for dolphin.

Ill give it an actual try using the following ChatML tool calling template on vLLM and report back:

/Update
That was an exercise in futility.

TypeError: Error: --enable-auto-tool-choice and --enable-reasoning cannot be enabled at the same time

However IMO the arguments above in general still hold.

When trying the below ChatML with cognitivecomputations/Dolphin3.0-Mistral-24B deployed with vLLM the followin example tool call gives an error, of not allowing consecutive user or assistant messages 🤷

Example OpenAI Tool call

def get_weather(location: str, unit: str):
    return f"Getting the weather for {location} in {unit}..."
tool_functions = {"get_weather": get_weather}

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City and state, e.g., 'San Francisco, CA'"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["location", "unit"]
        }
    }
}]

response = client.chat.completions.create(
    model="cognitivecomputations/Dolphin3.0-Mistral-24B",
    messages=[{"role": "user", "content": "What's the weather like in San Francisco?"}],
    tools=tools,
    tool_choice="auto"
)

print(f"Function called: {response.choices[0]}")

tool_call = response.choices[0].message.tool_calls[0].function
print(f"Function called: {tool_call.name}")
print(f"Arguments: {tool_call.arguments}")
print(f"Result: {get_weather(**json.loads(tool_call.arguments))}")

ChatML Template with tool calling

chat_template = """{%- set tools = tools if tools is defined else None -%}
{%- set date_string = date_string if date_string is defined else "1 Sep 2024" -%}

{%- set system_message = messages[0].content if messages[0].role == "system" else "" -%}
{%- if messages[0].role == "system" -%}
    {%- set messages = messages[1:] -%}
{%- endif -%}

{%- if not tool_prompt -%}
    {%- set tool_prompt = "For each function call return a json object with function name and arguments within <tool_call> </tool_call> tags with the following schema:\n<tool_call>\n{\\\"name\\\": <function-name>, \\\"arguments\\\": <args-dict>}\n</tool_call>" -%}
{%- endif -%}

{%- if system_message or tools -%}
  {{- '<|im_start|>system\n'}}
{%- endif -%}

{%- if system_message %}
  {{- system_message + "\n"}}
{%- endif -%}

{%- if tools  -%}
  {{- "You are a function-calling AI model. You are provided with function signatures within <tools> </tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions.\n" }}
  {{- "<tools>\n" }}
  {{- tools }}
  {{- "\n</tools>\n" }}
  {{- tool_prompt -}}
{%- endif -%}

{%- if system_message or tools -%}
  {{- '<|im_end|>\n'}}
{%- endif -%}

{# Main message loop #}
{%- for message in messages -%}
    {%- if message.role == "user" or message.role == "assistant" or message.role == "tool" -%}
        {%- if loop.first and message.role != "user" -%}
            {{ raise_exception("Invalid sequence: The first message role must be 'user' after 'system' if provided .") }}
        {%- endif -%}

        {%- if not loop.first and message.role in ["user", "assistant"] and message.role == loop.previtem.role -%}
            {{ raise_exception("Invalid sequence: Consecutive messages cannot have the same role ('user' or 'assistant').") }}
        {%- endif -%}

        {%- if message.role == "user" and not loop.first and loop.previtem.role != "assistant" -%}
            {{ raise_exception("Invalid sequence: A 'user' message must be preceded by an 'assistant' message.") }}
        {%- endif -%}

        {%- if message.role == "tool" and not loop.first and loop.previtem.role not in ["assistant", "tool"] -%}
            {{ raise_exception("Invalid sequence: A 'tool' message must be preceded by 'assistant' or 'tool'.") }}
        {%- endif -%}
    {%- else -%}
        {{- raise_exception("Invalid role detected: only 'user', 'assistant', or 'tool' roles are accepted.") }}
    {%- endif -%}
    {%- if message.role == "user" or (message.role == "assistant" and message.tool_calls is not defined) -%}
        {{- '<|im_start|>' + message.role + '\n' + message.content | trim + '<|im_end|>\n'}}
    {%- elif message.role == "assistant" -%}
        {{- '<|im_start|>' + message.role }}
        {%- for tool_call in message.tool_calls -%}
            {{ '\n<tool_call>\n' }}
              {%- if tool_call.function -%}
                {"name": "{{ tool_call.function.name }}", "arguments": {{ tool_call.function.arguments | tojson }} }
              {%- else -%}
                {"name": "{{ tool_call.name }}", "arguments": {{ tool_call.arguments | tojson }} }
              {%- endif -%}
            {{ '\n</tool_call>' }}
        {%- endfor -%}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" -%}
        {%- if loop.previtem and loop.previtem.role != "tool" -%}
            {{- '<|im_start|>tool\n' }}
        {%- endif -%}
        {{- '<tool_response>\n' }} 
            {{- message.content }}
        {{- '\n</tool_response>\n' }}
        {%- if loop.last or loop.nextitem.role != "tool" -%}
            {{- '<|im_end|>\n'}}
        {%- endif -%}
    {%- endif -%}
{%- endfor -%}

{# Prompt for assistant generation if needed #}
{%- if add_generation_prompt -%}
    {{- '<|im_start|>assistant\n' }}
{%- endif -%}"""

This was trained from Small24B-Base, so there's no innate Mistral/Tekken7-Instruct function call support. Kind of the principle when using a base model.

So, It uses the same function calling format as Qwen models. I tested it, and it works okay. @ehartford you would have wasted a lot less of everyone's time if you had just said so instead of acting unnecessarily dismissive.

ok I was finally able to get OpenAI compatible tool calling for single and multiple/parallel tool calls with a custom template and parser on vLLM OpenAI server on the mistral base model.

IIRC, this should work for any downstream models using the same mistral base template.

Nontheless, I hope I did not divert too much from the original model template , since I would like to fine-tune mistral v3 on downstream tasks 🤞

Will probably do a PR for the parser to vLLM if you think its a valuable contribution.