headless-adapters（无头适配器）

Headless Adapters

Purpose

Schema-driven adapter for headless CLI agents. No code required - just define a JSON schema describing how to interact with the CLI.

Quick Start

1. Create a schema for your CLI agent using the Schema Creation Guide
2. Run the adapter: bunx @plaited/agent-eval-harness headless --schema./my-agent-headless.json

Any CLI agent that outputs JSON can be wrapped — no agent-specific code required.

CLI Commands

headless

Schema-driven adapter for ANY headless CLI agent.

bunx @plaited/agent-eval-harness headless --schema <path>

Options:

Schema Format:

{
  "version": 1,
  "name": "my-agent",
  "command": ["my-agent-cli"],
  "sessionMode": "stream",
  "prompt": { "flag": "-p" },
  "output": { "flag": "--output-format", "value": "stream-json" },
  "autoApprove": ["--allow-all"],
  "outputEvents": [
    {
      "match": { "path": "$.type", "value": "message" },
      "emitAs": "message",
      "extract": { "content": "$.text" }
    },
    {
      "match": { "path": "$.type", "value": "tool_use" },
      "emitAs": "tool_call",
      "extract": { "title": "$.name", "status": "'pending'", "input": "$.input" }
    },
    {
      "match": { "path": "$.type", "value": "tool_result" },
      "emitAs": "tool_call",
      "extract": { "title": "$.name", "status": "'completed'", "output": "$.content" }
    }
  ],
  "result": {
    "matchPath": "$.type",
    "matchValue": "result",
    "contentPath": "$.content"
  }
}

Session Modes:

Creating a Schema

1. Run the CLI's --help to identify prompt, output format, and auto-approve flags
2. Capture sample JSON output from the CLI
3. Map JSONPath patterns to output events (including input/output for tool calls)
4. Create the schema file
5. Test with headless command

See Schema Creation Guide for the complete workflow.

Security Considerations

Trust Boundary: CLI Output is Untrusted

The headless adapter parses JSON output from CLI agents. This output may contain content from external sources (web searches, file reads, API responses) that flows into trajectory data:

CLI Agent → JSON stdout → JSONPath extraction → ParsedUpdate → TrajectoryStep

Trajectory fields — especially tool_call.input and tool_call.output — should be treated as untrusted content by downstream consumers (graders, LLM-as-judge, analysis scripts). Do not:

• Execute trajectory content as code
• Use trajectory content in unsanitized shell commands
• Pass trajectory content to LLMs without injection-aware prompting

autoApprove Flags

The autoApprove field bypasses the CLI agent's safety confirmation prompts. Use the least permissive flags your evaluation requires:

Never run high-risk autoApprove flags outside isolated environments. Use --workspace-dir or Docker for evaluations that modify the filesystem.

Troubleshooting

Common Issues

Complete troubleshooting documentation: Troubleshooting Guide

External Resources

• AgentSkills Spec: agentskills.io

Related

• agent-eval-harness skill - Running evaluations against adapters