Computor Use MCP

Computer Use MCP Server

An MCP (Model Context Protocol) server that provides computer interaction tools for automation and control. This server enables programmatic control of mouse, keyboard, and accessibility features on macOS systems.

Features

• Mouse Control: Click, double-click, right-click, move, and scroll operations
• Keyboard Input: Type text and execute keyboard shortcuts
• Timing Controls: Wait/delay functionality for automation sequencing
• Accessibility Integration: Advanced macOS Accessibility API tools for UI automation
• Cross-Language Architecture: TypeScript/Node.js server with Python tool implementations

Installation

Prerequisites

• Node.js (for the server)
• Python 3.7+ (for the tools)
• macOS (required for pyobjc accessibility features)
• GitHub CLI (gh) for license management

Setup Steps

1. Clone the repository:

   git clone https://github.com/HananiahKao/computor-use-MCP.git
   cd computor-use-MCP

1. Install Node.js dependencies:

   cd computer-use-mcp-server
   npm install

1. Build the project:

   npm run build

1. Set up Python virtual environment:

   python3 -m venv venv
   source venv/bin/activate
   pip install -r tools/requirements.txt

Add License

gh repo license add MIT

Usage

Running the Server

node build/index.js

The server communicates via stdio and should be connected to an MCP client.

Integration with MCP Clients

This server is designed to work with MCP-compatible clients. Configure your client to launch the server and communicate via the Model Context Protocol.

Available Tools

Mouse Operations

• click (x: number, y: number)

- Click the left mouse button at specified screen coordinates

• double_click (x: number, y: number)

- Double-click the left mouse button at specified coordinates

• right_click (x: number, y: number)

- Right-click at specified coordinates

• move (x: number, y: number)

- Move mouse cursor to coordinates without clicking

• scroll (dy: number, dx?: number, x?: number, y?: number, repeat?: number, interval?: number)

- Scroll at current or specified position - dy: Vertical scroll (positive: up, negative: down) - dx: Horizontal scroll (positive: right, negative: left)

Keyboard Operations

• type (text: string, delay_per_char?: number)

- Type text at current cursor position - Supports natural typing with configurable delay

• press_shortcut (keys: string[])

- Execute keyboard shortcut (e.g., ['cmd', 'c'] for copy)

Timing and Control

• wait (seconds: number)

- Pause execution for specified duration

Accessibility (AX) Tools

• ax_select (selector?: string, debug?: boolean)

- Query UI elements using CSS-like selectors - Returns JSON representation of found elements

• ax_highlight (element_path: string, duration?: number, color?: string)

- Visually highlight an accessibility element - Useful for debugging UI automation

• ax_attributes (element_path: string)

- Get all accessibility attributes for an element

• ax_wait_until (element_path: string, timeout?: number)

- Wait until an accessibility element exists

Testing/Sample Tool

• greet (name?: string)

- Return a friendly greeting message - Useful for testing server connectivity

Development

Build Commands

npm run build      # Compile TypeScript
npm run dev        # Watch mode compilation

Project Structure

computer-use-mcp-server/
├── src/index.ts       # Main server implementation
├── build/             # Compiled JavaScript output
├── tools/             # Python tool implementations
│   ├── requirements.txt
│   └── *.py
├── package.json
├── tsconfig.json
└── venv/              # Python virtual environment

Requirements

• Platform: macOS (due to pyobjc accessibility APIs)
• Python Libraries: pynput (input control), pyobjc (accessibility)
• Node.js: For the MCP server framework
• TypeScript: For type-safe development

Contributing

Contributions welcome! Please ensure code follows the project's clean code principles:

• Clear, readable function names
• Separation of concerns
• Proper error handling
• Comprehensive documentation

License

MIT License - see LICENSE file for details

Acknowledgments