mistral.rs 流式工具调用Streaming Tool Calling实战基于 OpenAI 兼容接口的多轮 Function Call 客户端【免费下载链接】mistral.rsFast, flexible LLM inference项目地址: https://gitcode.com/GitHub_Trending/mi/mistral.rs导读本文以 mistral.rs 仓库中的 流式工具调用示例文档 为骨架完整讲解如何通过 OpenAI 兼容的/v1/chat/completions接口在streamTrue模式下完成增量式工具调用模型一边生成、客户端一边累积工具调用参数最终在finish_reasontool_calls的末帧中拿到完整的函数调用并由客户端执行工具、回填结果、进入下一轮对话。读完本文你将掌握一套可直接复制运行的「流式推理 工具调用 多轮 Agent 循环」的完整 Python 客户端实现并理解 mistral.rs 服务端在 SSE 流中组织工具调用数据的方式。一、示例定位与适用场景在 mistral.rs 中工具调用Tool Calling / Function Calling有两种典型消费方式非流式一次请求直接返回完整 JSON包含tool_calls字段见 Rust 示例mistralrs/examples/advanced/tools/main.rs流式以 SSE 逐帧返回工具调用的id、function.name、function.arguments被按增量片段分布在多个 chunk 中需要客户端自行拼接。本文的示例正是第二种其官方定位是This demonstrates how to use streaming with tool calls in mistral.rs. Tool calls are accumulated during streaming and sent in the final chunk withfinish_reasontool_calls.该模式非常契合需要边生成边展示的 Agent 场景模型推理未结束时用户已经能看到正文与思维链内容在滚动输出推理结束后工具调用参数也恰好收集完毕可直接解析执行。二、环境准备启动 mistral.rs 服务器示例基于mistralrs serve子命令提供的一个本地 OpenAI 兼容端点。文档给出的启动命令为mistralrs serve --quant q4k -p 1234 -m meta-llama/Meta-Llama-3.1-8B-Instruct参数含义参数说明--quant q4k使用 4-bit K-quant 量化加载模型显著降低显存占用-p 1234服务监听端口 1234-m meta-llama/Meta-Llama-3.1-8B-Instruct指定 HF Hub 上的模型 ID当前为 Meta-Llama-3.1-8B-Instruct服务器启动后会暴露 OpenAI 兼容的http://localhost:1234/v1/接口因此客户端只需使用openaiPython SDK 即可对接无需任何定制代码python3 examples/server/streaming_tool_calling.py示例源码位于 examples/server/streaming_tool_calling.py文档内容由 docs/scripts/render_examples.py 从该源文件自动渲染生成。说明serve子命令的完整参数模型源、量化、端口、并行度等见 mistralrs-cli/src/args/server.rs示例中的命令只使用了最小必要参数集。三、完整代码流式工具调用客户端以下是示例的完整实现与仓库源文件一致。它演示了工具声明 → 流式请求 → 增量收集工具调用 → 执行工具 → 回填多轮对话 → 直到模型给出最终正文答案。 Streaming tool calling example. This demonstrates how to use streaming with tool calls in mistral.rs. Tool calls are accumulated during streaming and sent in the final chunk with finish_reasontool_calls. import json import sys from io import StringIO from openai import OpenAI client OpenAI(api_keyfoobar, base_urlhttp://localhost:1234/v1/) MAX_TOOL_ROUNDS 10 tools [ { type: function, function: { name: run_python, description: Run some Python code, parameters: { type: object, properties: { code: { type: string, description: The Python code to evaluate. The return value is whatever was printed out from print., }, }, required: [code], }, }, } ] def run_python(code: str) - str: Execute Python code and return the output. lcls dict() glbls {open: None} # No opening of files print(fRunning:\npy\n{code}\n) old_stdout sys.stdout out StringIO() sys.stdout out try: exec(code, glbls, lcls) except Exception as e: sys.stdout old_stdout return fError: {e} sys.stdout old_stdout return out.getvalue() functions { run_python: run_python, } def do_streaming_request(messages, tools, round_num): Make a streaming request and handle the response. print(f\n{ * 60}) print(fRound {round_num}: Making streaming request...) print(f{ * 60}) stream client.chat.completions.create( modeldefault, messagesmessages, toolstools, tool_choiceauto, streamTrue, ) collected_content collected_tool_calls [] finish_reason None print(Assistant: , end, flushTrue) collected_reasoning for chunk in stream: choice chunk.choices[0] delta choice.delta # Print reasoning content in gray (if present) # The field may be reasoning_content or accessible via getattr reasoning getattr(delta, reasoning_content, None) if reasoning: # Print reasoning in gray/dim print(f\033[90m{reasoning}\033[0m, end, flushTrue) collected_reasoning reasoning # Print and collect content as it streams if delta.content: print(delta.content, end, flushTrue) collected_content delta.content # Collect tool calls from delta if delta.tool_calls: for tool_call in delta.tool_calls: # Find or create the tool call entry while len(collected_tool_calls) tool_call.index: collected_tool_calls.append( { id: None, type: function, function: {name: , arguments: }, } ) tc collected_tool_calls[tool_call.index] if tool_call.id: tc[id] tool_call.id if tool_call.function: if tool_call.function.name: tc[function][name] tool_call.function.name if tool_call.function.arguments: tc[function][arguments] tool_call.function.arguments if choice.finish_reason: finish_reason choice.finish_reason print() # newline after streaming print(f[Finish reason: {finish_reason}]) if collected_reasoning: print(f[Reasoning: {len(collected_reasoning)} chars]) return collected_content, collected_tool_calls, finish_reason, collected_reasoning def execute_tool_calls(tool_calls): Execute tool calls and return results. results [] for tool_call in tool_calls: func_name tool_call[function][name] func_args tool_call[function][arguments] print(f Calling {func_name}({func_args})) if func_name in functions: try: args json.loads(func_args) if func_args else {} result functionsfunc_name print(f - {result}) results.append((tool_call, result)) except Exception as e: error_result json.dumps({error: str(e)}) print(f - Error: {e}) results.append((tool_call, error_result)) else: error_result json.dumps({error: fUnknown function: {func_name}}) print( - Error: Unknown function) results.append((tool_call, error_result)) return results def main(): messages [ { role: user, content: Please write and run a python script to do a matmul of 2 random integer matrices. Then tell me JUST the result of the matmul., } ] print( User: Please write and run a python script to do a matmul of 2 random integer matrices. Then tell me JUST the result of the matmul. ) for round_num in range(1, MAX_TOOL_ROUNDS 1): content, tool_calls, finish_reason, reasoning do_streaming_request( messages, tools, round_num ) if finish_reason tool_calls and tool_calls: print(f\nTool calls detected ({len(tool_calls)}):) # Add assistant message with tool calls messages.append( { role: assistant, content: content if content else None, tool_calls: tool_calls, } ) # Execute tool calls and add results results execute_tool_calls(tool_calls) for tool_call, result in results: messages.append( { role: tool, tool_call_id: tool_call[id], name: tool_call[function][name], content: result, } ) print(f\nContinuing to round {round_num 1}...) else: # No tool calls, were done print(f\nFinal response received (finish_reason: {finish_reason})) if content: print(fContent: {content}) break else: print(f\nReached maximum tool rounds ({MAX_TOOL_ROUNDS})) if __name__ __main__: main()四、代码拆解流式工具调用的五个关键环节4.1 客户端与工具声明客户端通过标准的OpenAI构造器指向本地 mistral.rs 端点api_key传任意非空值服务端本地不校验client OpenAI(api_keyfoobar, base_urlhttp://localhost:1234/v1/)工具采用 OpenAI Function Calling 的 JSON Schema 格式声明。示例注册了一个run_python工具唯一参数是code待执行的 Python 源码并标记required: [code]tools [ { type: function, function: { name: run_python, description: Run some Python code, parameters: { type: object, properties: { code: {type: string, description: ...}, }, required: [code], }, }, } ]服务端对应结构体为 mistralrs-server-core/src/openai.rs 中的ToolCall含id、type、function三个字段与FunctionCallednameargumentsJSON 字符串。Rust 原生方式下可用Tool { tp: ToolType::Function, function: Function { name, description, parameters, strict } }手动构造等价工具见 mistralrs/examples/advanced/tools/main.rs。4.2 发起流式请求stream client.chat.completions.create( modeldefault, # serve 场景下固定使用 default messagesmessages, toolstools, tool_choiceauto, # 由模型自主决定是否调用工具 streamTrue, # 关键开启 SSE 流式 )其中modeldefault是 mistral.rs serve 的约定写法——服务端只加载了一个模型因此统一以default指代。tool_choiceauto表示模型可以自行选择直接回答或调用工具。4.3 增量收集正文、思维链与工具调用流式返回的每个 chunk 都包含一个choices[0].delta示例对 delta 做了三件事思维链输出getattr(delta, reasoning_content, None)兼容不同 SDK 版本对推理字段的命名存在则以灰色\033[90m打印并累计长度正文输出delta.content边收边打印同时拼入collected_content工具调用累积这是流式工具调用的核心。由于工具调用的三个字段是分片到达的客户端需要按tool_call.index建立占位条目再逐帧补全if delta.tool_calls: for tool_call in delta.tool_calls: # Find or create the tool call entry while len(collected_tool_calls) tool_call.index: collected_tool_calls.append( { id: None, type: function, function: {name: , arguments: }, } ) tc collected_tool_calls[tool_call.index] if tool_call.id: tc[id] tool_call.id if tool_call.function: if tool_call.function.name: tc[function][name] tool_call.function.name if tool_call.function.arguments: tc[function][arguments] tool_call.function.arguments # 增量拼接要点解读arguments是字符串拼接而非 JSON 合并。流式场景下模型逐 token 吐出参数 JSON服务端原样切分因此客户端必须用累加结束后再整体json.loadsindex用于并行工具调用。一个回合内模型可能同时发起多个工具调用每个调用拥有独立的递增index客户端据此为每个调用维护独立的占位符finish_reason从choice.finish_reason读取。文档明确工具调用在流结束时统一于最后一帧给出此时finish_reason tool_calls。4.4 执行工具并回填对话一旦流结束且finish_reason tool_calls示例便将收集到的工具调用分派给本地函数映射表functions并把结果以roletool的消息回填messages.append( { role: assistant, content: content if content else None, tool_calls: tool_calls, # 模型的工具调用记录 } ) # 每个工具调用一条 tool 消息 messages.append( { role: tool, tool_call_id: tool_call[id], # 必须与 assistant.tool_calls[].id 对应 name: tool_call[function][name], content: result, } )这里遵循了 OpenAI 兼容的多轮工具调用协议assistant 消息携带tool_calls随后每条tool消息通过tool_call_id关联到具体调用。执行过程中若工具抛异常示例也会把错误信息 JSON 序列化后作为工具结果返回让模型看到错误并自行修正——这是构建稳健 Agent 的关键技巧。示例中的run_python工具实现还包含一个安全细节执行环境注入glbls {open: None}屏蔽文件打开能力同时重定向sys.stdout捕获print输出作为返回值。4.5 多轮循环与终止条件for round_num in range(1, MAX_TOOL_ROUNDS 1): content, tool_calls, finish_reason, reasoning do_streaming_request(...) if finish_reason tool_calls and tool_calls: # ...执行工具、回填消息、进入下一轮 else: # 无工具调用输出最终回答并退出 break else: print(f\nReached maximum tool rounds ({MAX_TOOL_ROUNDS}))文档明确指出部分模型可能在给出最终回答前连续发起多次工具调用因此示例以MAX_TOOL_ROUNDS 10兜底防止死循环循环退出条件有两种模型直接以普通finish_reason如stop结束或达到最大轮数整个messages列表在轮次间持续累积user → assistant(tool_calls) → tool → assistant(tool_calls) → …保证上下文完整。五、服务端视角SSE 流与 finish_reason 的组织方式从仓库源码可以印证文档描述的工具调用在末帧给出这一行为流式响应的载体是 Server-Sent Events。服务端流式处理核心位于 mistralrs-server-core/src/streaming.rsbase_create_streamer将引擎产出的Response::Chunk逐帧编码为data:行推送流结束时发送[DONE]标记响应内容类型为text/event-stream见 chat_completion.rs 中ChatCompletionChunkResponseBody的 schema 标注服务端判断一轮生成是否收尾依据的是每个 choice 的finish_reason是否已填充mistralrs-server-core/src/chat_completion.rs 第 436 行附近的response.choices.iter().all(|x| x.finish_reason.is_some())。当模型决定调用工具时该值为tool_calls请求侧的工具声明tools/tool_choice会经过normalize_chat_completion_tools与validate_openai_tool_choice规范化校验同样位于 chat_completion.rs随后由聊天模板如 chat_templates/mistral_nemo_tool_call.jinja、chat_templates/deepseek_tool_call.jinja渲染进 promptassistant 消息中的tool_calls也会被转换为模板所需的IndexMap结构chat_completion.rs 第 656-688 行这正是流式客户端回填 assistant 消息后服务端能正确理解上下文的原因。六、运行效果与故障排查以文档中的示例请求让模型编写并执行两个随机整数矩阵的 matmul为例预期流程为第 1 轮流式请求模型先输出推理过程若支持则以灰色显示随后流结束于finish_reasontool_calls客户端收到一个run_python调用及其完整arguments客户端执行本地run_python捕获print输出作为工具结果第 2 轮请求模型读取工具结果后以普通finish_reasonstop结束输出矩阵乘法的最终结果主循环break程序结束。常见问题与排查方向现象可能原因请求报 404 / 连接失败服务器未启动或端口不一致确认-p 1234与客户端base_url端口相同tool_calls始终为空所选模型不支持函数调用模板可更换为示例指定的Meta-Llama-3.1-8B-Instruct等对齐过工具调用的模型arguments 是残缺 JSON客户端未做增量拼接漏掉或过早json.loads进入死循环直到MAX_TOOL_ROUNDS工具结果格式不符合模型预期如未回填tool_call_id模型反复尝试调用七、延伸阅读examples/server/streaming_tool_calling.py本文源码示例本体文档由其渲染生成examples/server/streaming.py不带工具调用的纯流式对话示例可对照理解 SSE 基本用法examples/server/tool_calling.py非流式工具调用示例对比两种模式下tool_calls的获取差异mistralrs/examples/advanced/tools/main.rsRust 原生 API 下的工具调用Tool/Function/ToolChoice构造方式mistralrs-server-core/src/openai.rs服务端 OpenAI 兼容数据结构ToolCall、FunctionCalled、Messagemistralrs-server-core/src/chat_completion.rs请求解析、工具声明校验、模板渲染与 finish_reason 收尾逻辑mistralrs-server-core/src/streaming.rsSSE 流式编码与[DONE]结束标记chat_templates/各模型家族的对话/工具调用 Jinja 模板【免费下载链接】mistral.rsFast, flexible LLM inference项目地址: https://gitcode.com/GitHub_Trending/mi/mistral.rs创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
mistral.rs 流式工具调用（Streaming Tool Calling）实战：基于 OpenAI 兼容接口的多轮 Function Call 客户端
mistral.rs 流式工具调用Streaming Tool Calling实战基于 OpenAI 兼容接口的多轮 Function Call 客户端【免费下载链接】mistral.rsFast, flexible LLM inference项目地址: https://gitcode.com/GitHub_Trending/mi/mistral.rs导读本文以 mistral.rs 仓库中的 流式工具调用示例文档 为骨架完整讲解如何通过 OpenAI 兼容的/v1/chat/completions接口在streamTrue模式下完成增量式工具调用模型一边生成、客户端一边累积工具调用参数最终在finish_reasontool_calls的末帧中拿到完整的函数调用并由客户端执行工具、回填结果、进入下一轮对话。读完本文你将掌握一套可直接复制运行的「流式推理 工具调用 多轮 Agent 循环」的完整 Python 客户端实现并理解 mistral.rs 服务端在 SSE 流中组织工具调用数据的方式。一、示例定位与适用场景在 mistral.rs 中工具调用Tool Calling / Function Calling有两种典型消费方式非流式一次请求直接返回完整 JSON包含tool_calls字段见 Rust 示例mistralrs/examples/advanced/tools/main.rs流式以 SSE 逐帧返回工具调用的id、function.name、function.arguments被按增量片段分布在多个 chunk 中需要客户端自行拼接。本文的示例正是第二种其官方定位是This demonstrates how to use streaming with tool calls in mistral.rs. Tool calls are accumulated during streaming and sent in the final chunk withfinish_reasontool_calls.该模式非常契合需要边生成边展示的 Agent 场景模型推理未结束时用户已经能看到正文与思维链内容在滚动输出推理结束后工具调用参数也恰好收集完毕可直接解析执行。二、环境准备启动 mistral.rs 服务器示例基于mistralrs serve子命令提供的一个本地 OpenAI 兼容端点。文档给出的启动命令为mistralrs serve --quant q4k -p 1234 -m meta-llama/Meta-Llama-3.1-8B-Instruct参数含义参数说明--quant q4k使用 4-bit K-quant 量化加载模型显著降低显存占用-p 1234服务监听端口 1234-m meta-llama/Meta-Llama-3.1-8B-Instruct指定 HF Hub 上的模型 ID当前为 Meta-Llama-3.1-8B-Instruct服务器启动后会暴露 OpenAI 兼容的http://localhost:1234/v1/接口因此客户端只需使用openaiPython SDK 即可对接无需任何定制代码python3 examples/server/streaming_tool_calling.py示例源码位于 examples/server/streaming_tool_calling.py文档内容由 docs/scripts/render_examples.py 从该源文件自动渲染生成。说明serve子命令的完整参数模型源、量化、端口、并行度等见 mistralrs-cli/src/args/server.rs示例中的命令只使用了最小必要参数集。三、完整代码流式工具调用客户端以下是示例的完整实现与仓库源文件一致。它演示了工具声明 → 流式请求 → 增量收集工具调用 → 执行工具 → 回填多轮对话 → 直到模型给出最终正文答案。 Streaming tool calling example. This demonstrates how to use streaming with tool calls in mistral.rs. Tool calls are accumulated during streaming and sent in the final chunk with finish_reasontool_calls. import json import sys from io import StringIO from openai import OpenAI client OpenAI(api_keyfoobar, base_urlhttp://localhost:1234/v1/) MAX_TOOL_ROUNDS 10 tools [ { type: function, function: { name: run_python, description: Run some Python code, parameters: { type: object, properties: { code: { type: string, description: The Python code to evaluate. The return value is whatever was printed out from print., }, }, required: [code], }, }, } ] def run_python(code: str) - str: Execute Python code and return the output. lcls dict() glbls {open: None} # No opening of files print(fRunning:\npy\n{code}\n) old_stdout sys.stdout out StringIO() sys.stdout out try: exec(code, glbls, lcls) except Exception as e: sys.stdout old_stdout return fError: {e} sys.stdout old_stdout return out.getvalue() functions { run_python: run_python, } def do_streaming_request(messages, tools, round_num): Make a streaming request and handle the response. print(f\n{ * 60}) print(fRound {round_num}: Making streaming request...) print(f{ * 60}) stream client.chat.completions.create( modeldefault, messagesmessages, toolstools, tool_choiceauto, streamTrue, ) collected_content collected_tool_calls [] finish_reason None print(Assistant: , end, flushTrue) collected_reasoning for chunk in stream: choice chunk.choices[0] delta choice.delta # Print reasoning content in gray (if present) # The field may be reasoning_content or accessible via getattr reasoning getattr(delta, reasoning_content, None) if reasoning: # Print reasoning in gray/dim print(f\033[90m{reasoning}\033[0m, end, flushTrue) collected_reasoning reasoning # Print and collect content as it streams if delta.content: print(delta.content, end, flushTrue) collected_content delta.content # Collect tool calls from delta if delta.tool_calls: for tool_call in delta.tool_calls: # Find or create the tool call entry while len(collected_tool_calls) tool_call.index: collected_tool_calls.append( { id: None, type: function, function: {name: , arguments: }, } ) tc collected_tool_calls[tool_call.index] if tool_call.id: tc[id] tool_call.id if tool_call.function: if tool_call.function.name: tc[function][name] tool_call.function.name if tool_call.function.arguments: tc[function][arguments] tool_call.function.arguments if choice.finish_reason: finish_reason choice.finish_reason print() # newline after streaming print(f[Finish reason: {finish_reason}]) if collected_reasoning: print(f[Reasoning: {len(collected_reasoning)} chars]) return collected_content, collected_tool_calls, finish_reason, collected_reasoning def execute_tool_calls(tool_calls): Execute tool calls and return results. results [] for tool_call in tool_calls: func_name tool_call[function][name] func_args tool_call[function][arguments] print(f Calling {func_name}({func_args})) if func_name in functions: try: args json.loads(func_args) if func_args else {} result functionsfunc_name print(f - {result}) results.append((tool_call, result)) except Exception as e: error_result json.dumps({error: str(e)}) print(f - Error: {e}) results.append((tool_call, error_result)) else: error_result json.dumps({error: fUnknown function: {func_name}}) print( - Error: Unknown function) results.append((tool_call, error_result)) return results def main(): messages [ { role: user, content: Please write and run a python script to do a matmul of 2 random integer matrices. Then tell me JUST the result of the matmul., } ] print( User: Please write and run a python script to do a matmul of 2 random integer matrices. Then tell me JUST the result of the matmul. ) for round_num in range(1, MAX_TOOL_ROUNDS 1): content, tool_calls, finish_reason, reasoning do_streaming_request( messages, tools, round_num ) if finish_reason tool_calls and tool_calls: print(f\nTool calls detected ({len(tool_calls)}):) # Add assistant message with tool calls messages.append( { role: assistant, content: content if content else None, tool_calls: tool_calls, } ) # Execute tool calls and add results results execute_tool_calls(tool_calls) for tool_call, result in results: messages.append( { role: tool, tool_call_id: tool_call[id], name: tool_call[function][name], content: result, } ) print(f\nContinuing to round {round_num 1}...) else: # No tool calls, were done print(f\nFinal response received (finish_reason: {finish_reason})) if content: print(fContent: {content}) break else: print(f\nReached maximum tool rounds ({MAX_TOOL_ROUNDS})) if __name__ __main__: main()四、代码拆解流式工具调用的五个关键环节4.1 客户端与工具声明客户端通过标准的OpenAI构造器指向本地 mistral.rs 端点api_key传任意非空值服务端本地不校验client OpenAI(api_keyfoobar, base_urlhttp://localhost:1234/v1/)工具采用 OpenAI Function Calling 的 JSON Schema 格式声明。示例注册了一个run_python工具唯一参数是code待执行的 Python 源码并标记required: [code]tools [ { type: function, function: { name: run_python, description: Run some Python code, parameters: { type: object, properties: { code: {type: string, description: ...}, }, required: [code], }, }, } ]服务端对应结构体为 mistralrs-server-core/src/openai.rs 中的ToolCall含id、type、function三个字段与FunctionCallednameargumentsJSON 字符串。Rust 原生方式下可用Tool { tp: ToolType::Function, function: Function { name, description, parameters, strict } }手动构造等价工具见 mistralrs/examples/advanced/tools/main.rs。4.2 发起流式请求stream client.chat.completions.create( modeldefault, # serve 场景下固定使用 default messagesmessages, toolstools, tool_choiceauto, # 由模型自主决定是否调用工具 streamTrue, # 关键开启 SSE 流式 )其中modeldefault是 mistral.rs serve 的约定写法——服务端只加载了一个模型因此统一以default指代。tool_choiceauto表示模型可以自行选择直接回答或调用工具。4.3 增量收集正文、思维链与工具调用流式返回的每个 chunk 都包含一个choices[0].delta示例对 delta 做了三件事思维链输出getattr(delta, reasoning_content, None)兼容不同 SDK 版本对推理字段的命名存在则以灰色\033[90m打印并累计长度正文输出delta.content边收边打印同时拼入collected_content工具调用累积这是流式工具调用的核心。由于工具调用的三个字段是分片到达的客户端需要按tool_call.index建立占位条目再逐帧补全if delta.tool_calls: for tool_call in delta.tool_calls: # Find or create the tool call entry while len(collected_tool_calls) tool_call.index: collected_tool_calls.append( { id: None, type: function, function: {name: , arguments: }, } ) tc collected_tool_calls[tool_call.index] if tool_call.id: tc[id] tool_call.id if tool_call.function: if tool_call.function.name: tc[function][name] tool_call.function.name if tool_call.function.arguments: tc[function][arguments] tool_call.function.arguments # 增量拼接要点解读arguments是字符串拼接而非 JSON 合并。流式场景下模型逐 token 吐出参数 JSON服务端原样切分因此客户端必须用累加结束后再整体json.loadsindex用于并行工具调用。一个回合内模型可能同时发起多个工具调用每个调用拥有独立的递增index客户端据此为每个调用维护独立的占位符finish_reason从choice.finish_reason读取。文档明确工具调用在流结束时统一于最后一帧给出此时finish_reason tool_calls。4.4 执行工具并回填对话一旦流结束且finish_reason tool_calls示例便将收集到的工具调用分派给本地函数映射表functions并把结果以roletool的消息回填messages.append( { role: assistant, content: content if content else None, tool_calls: tool_calls, # 模型的工具调用记录 } ) # 每个工具调用一条 tool 消息 messages.append( { role: tool, tool_call_id: tool_call[id], # 必须与 assistant.tool_calls[].id 对应 name: tool_call[function][name], content: result, } )这里遵循了 OpenAI 兼容的多轮工具调用协议assistant 消息携带tool_calls随后每条tool消息通过tool_call_id关联到具体调用。执行过程中若工具抛异常示例也会把错误信息 JSON 序列化后作为工具结果返回让模型看到错误并自行修正——这是构建稳健 Agent 的关键技巧。示例中的run_python工具实现还包含一个安全细节执行环境注入glbls {open: None}屏蔽文件打开能力同时重定向sys.stdout捕获print输出作为返回值。4.5 多轮循环与终止条件for round_num in range(1, MAX_TOOL_ROUNDS 1): content, tool_calls, finish_reason, reasoning do_streaming_request(...) if finish_reason tool_calls and tool_calls: # ...执行工具、回填消息、进入下一轮 else: # 无工具调用输出最终回答并退出 break else: print(f\nReached maximum tool rounds ({MAX_TOOL_ROUNDS}))文档明确指出部分模型可能在给出最终回答前连续发起多次工具调用因此示例以MAX_TOOL_ROUNDS 10兜底防止死循环循环退出条件有两种模型直接以普通finish_reason如stop结束或达到最大轮数整个messages列表在轮次间持续累积user → assistant(tool_calls) → tool → assistant(tool_calls) → …保证上下文完整。五、服务端视角SSE 流与 finish_reason 的组织方式从仓库源码可以印证文档描述的工具调用在末帧给出这一行为流式响应的载体是 Server-Sent Events。服务端流式处理核心位于 mistralrs-server-core/src/streaming.rsbase_create_streamer将引擎产出的Response::Chunk逐帧编码为data:行推送流结束时发送[DONE]标记响应内容类型为text/event-stream见 chat_completion.rs 中ChatCompletionChunkResponseBody的 schema 标注服务端判断一轮生成是否收尾依据的是每个 choice 的finish_reason是否已填充mistralrs-server-core/src/chat_completion.rs 第 436 行附近的response.choices.iter().all(|x| x.finish_reason.is_some())。当模型决定调用工具时该值为tool_calls请求侧的工具声明tools/tool_choice会经过normalize_chat_completion_tools与validate_openai_tool_choice规范化校验同样位于 chat_completion.rs随后由聊天模板如 chat_templates/mistral_nemo_tool_call.jinja、chat_templates/deepseek_tool_call.jinja渲染进 promptassistant 消息中的tool_calls也会被转换为模板所需的IndexMap结构chat_completion.rs 第 656-688 行这正是流式客户端回填 assistant 消息后服务端能正确理解上下文的原因。六、运行效果与故障排查以文档中的示例请求让模型编写并执行两个随机整数矩阵的 matmul为例预期流程为第 1 轮流式请求模型先输出推理过程若支持则以灰色显示随后流结束于finish_reasontool_calls客户端收到一个run_python调用及其完整arguments客户端执行本地run_python捕获print输出作为工具结果第 2 轮请求模型读取工具结果后以普通finish_reasonstop结束输出矩阵乘法的最终结果主循环break程序结束。常见问题与排查方向现象可能原因请求报 404 / 连接失败服务器未启动或端口不一致确认-p 1234与客户端base_url端口相同tool_calls始终为空所选模型不支持函数调用模板可更换为示例指定的Meta-Llama-3.1-8B-Instruct等对齐过工具调用的模型arguments 是残缺 JSON客户端未做增量拼接漏掉或过早json.loads进入死循环直到MAX_TOOL_ROUNDS工具结果格式不符合模型预期如未回填tool_call_id模型反复尝试调用七、延伸阅读examples/server/streaming_tool_calling.py本文源码示例本体文档由其渲染生成examples/server/streaming.py不带工具调用的纯流式对话示例可对照理解 SSE 基本用法examples/server/tool_calling.py非流式工具调用示例对比两种模式下tool_calls的获取差异mistralrs/examples/advanced/tools/main.rsRust 原生 API 下的工具调用Tool/Function/ToolChoice构造方式mistralrs-server-core/src/openai.rs服务端 OpenAI 兼容数据结构ToolCall、FunctionCalled、Messagemistralrs-server-core/src/chat_completion.rs请求解析、工具声明校验、模板渲染与 finish_reason 收尾逻辑mistralrs-server-core/src/streaming.rsSSE 流式编码与[DONE]结束标记chat_templates/各模型家族的对话/工具调用 Jinja 模板【免费下载链接】mistral.rsFast, flexible LLM inference项目地址: https://gitcode.com/GitHub_Trending/mi/mistral.rs创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考
