基于AI规划算法的MSPaint MCP服务器
该项目演示了如何使用Advanced AI Prompting使LLM能够稳健地处理多个步骤的复杂数学问题。它使用模型上下文协议（MCP）允许由谷歌Gemini模型支持的AI代理与传统的Windows应用程序（MSPaint）进行交互。AI代理利用以下定义的工具 fastmcp 并通过以下方式实施 pywinauto 解决数学问题，然后在Paint画布上绘制解决方案。
目录
引言
该项目展示了高级人工智能提示的使用，使LLM能够稳健地处理多个步骤的复杂数学问题。
此问题陈述的结构化提示
You are a math agent with painting skills, solving complex math expressions step-by-step.
You have access to various mathematical tools for calculations and verifications, as well as an MSPaint application to draw and present your solution on a canvas.
Available Tools:
{tools_description}
MSPaint Application Information:
- Rectangle coordinates: x1 = 763, y1 = 595, x2 = 1788, y2 = 1123
You must respond with EXACTLY ONE LINE in one of these formats (no additional text):
1. For function calls:
FUNCTION_CALL: {{"name": function_name, "arguments": {{"param1": value1, "param2": value2}}}}
2. For final answers:
FINAL_ANSWER:
3. For completing the task:
COMPLETE_RUN
Instructions:
- Start by calling the show_reasoning tool ONLY ONCE with a list of all step-by-step reasoning steps explaining how you will solve the problem. Once called, NEVER CALL IT AGAIN UNDER ANY CIRCUMSTANCES.
- When reasoning, tag each step with the reasoning type (e.g., [Arithmetic], [Logical Check]).
- Use all available math tools to solve the problem step-by-step.
- When a function returns multiple values, process all of them.
- Apply BODMAS rules: start with the innermost parentheses and work outward.
- Do not skip steps — perform all calculations sequentially.
- Respond only with one line at a time.
- Call only one tool per response.
- After calculating a number, verify it by calling:
FUNCTION_CALL: {{"name": "verify_calculation", "arguments": {{"expression": , "expected": }}}}
- If verify_calculation returns False, re-evaluate your previous steps.
- Once you reach a final answer, check for consistency of all steps and calculations by calling:
FUNCTION_CALL: {{"name": "verify_consistency", "arguments": {{"steps": [[, ], [, ], ...]}}}}
- If verify_consistency returns False, re-evaluate your previous steps.
- Once verify_consistency return True, submit your final result as:
FINAL_ANSWER:
Paint Instructions:
- To draw in Paint, follow this sequence strictly:
1. Call open_paint to start the Paint application.
2. Verify Paint is open using verify_paint_open.
3. If verify_paint_open returns False, retry opening Paint until it succeeds.
4. After Paint is open, draw a rectangle using draw_rectangle with correct parameters.
5. Add text using add_text_in_paint, inserting your FINAL_ANSWER: .
Final Step:
- After completing all calculations, verifications, and drawings, call:
COMPLETE_RUN
Strictly follow the above guidelines.
Your entire response should always be a single line starting with either FUNCTION_CALL:, FINAL_ANSWER: or COMPLETE_RUN.ChatGPT结构化提示评估结果
{
"explicit_reasoning": true,
"structured_output": true,
"tool_separation": true,
"conversation_loop": true,
"instructional_framing": true,
"internal_self_checks": true,
"reasoning_type_awareness": true,
"fallbacks": true,
"overall_clarity": "Extremely strong prompt — it carefully enforces step-by-step reasoning, structured outputs, error handling, and tool use separation. Very minor improvements could be to give a short worked-out example, but even without it, the robustness is excellent."
}项目结构
├── MSPaint-MCP-Server/
│ ├── mcp_server.py # Defines the MCP server with tools for Paint automation
│ ├── mcp_client.py # Defines the MCP client that interacts with the server and AI model
│ ├── requirements.txt # Lists the project dependencies
│ └── .env # Stores the Gemini API key
├── README.md # This file需求
- Python 3.11+
- Conda（推荐用于环境管理）
- Google Gemini API密钥
- pywin32
- pywinauto
- fastmcp
- python dotenv
- 谷歌genai
- 富有的
设置
- 创建Conda环境：
conda create -n eagenv python=3.11
conda activate eagenv- 安装依赖项：
pip install -r requirements.txt- 设置Gemini API密钥：
- 创建一个 .env 目录中的文件。
- 将Gemini API密钥添加到 .env 文件：
GEMINI_API_KEY=YOUR_API_KEY用法
- 运行MCP客户端：
python mcp_paint_app/mcp_client.py这将启动连接到MCP服务器的MCP客户端，初始化AI代理，并开始自动化过程。
运作原理
- MCP服务器(
mcp_server.py):
- 定义用于与MSPaint交互的工具（例如。， open_paint, draw_rectangle, add_text_in_paint)以及各种数学运算（例如。， add, subtract, multiply, divide, verify_calculation, verify_consistency). - 用途 pywinauto 以控制MSPaint应用程序。 - 通过以下方式显示这些工具 fastmcp 图书馆。
- MCP客户端(
mcp_client.py):
- 连接到MCP服务器。 - 使用Google Gemini模型生成指令并求解给定的数学表达式。 - 分析模型的输出以确定调用哪个工具。 - 使用所需参数调用MCP服务器上的相应工具。 - 处理来自工具的响应，并将其反馈给模型以进行下一步。 - 在MSPaint中编排最终答案的绘制。
- 人工智能代理（谷歌双子座）：
- 接收复杂的数学表达式（例如。， ((3000 - (400+552)) / 2 + 1024). - 使用可用工具（在系统提示中定义）逐步解决问题。 - 生成函数调用（例如。， FUNCTION_CALL: {"name": "add", "arguments" {"a": 400, "b": 552}})使用工具。 - 使用验证每个计算 verify_calculation 工具。 - 使用 verify_consistency 工具。 - 一旦获得并验证了最终答案，它就会使用Paint通过打开Paint、绘制矩形并将最终答案添加为文本来显示结果。 - 通过调用 COMPLETE_RUN 命令。
关键组件
mcp_server.py: 包含自动化MSPaint的核心逻辑。这open_paint,draw_rectangle，以及add_text_in_paint函数是AI代理使用的关键工具。mcp_client.py: 管理AI代理和MCP服务器之间的交互。它设置系统提示、调用工具并处理响应。requirements.txt: 列出项目所需的所有Python包。- .env： 存储Google Gemini API密钥。
故障排除
- 权限问题： 如果遇到权限问题，请尝试以管理员身份运行脚本。
- 协调问题： 用于在MSPaint中单击的坐标可能需要根据您的屏幕分辨率和窗口大小进行调整。使用代码中的调试打印语句来标识正确的坐标。
- 工具选择问题： 如果AI代理没有选择正确的工具，请查看系统提示并确保工具描述准确。
- API关键问题： 确保Gemini API密钥在
.env文件。
贡献
欢迎投稿！请提交一个包含您的更改的拉取请求。