Testing Agents With Subagents
Overview
Testing agents is TDD applied to AI worker definitions.
You run agents with known test inputs (RED - observe incorrect outputs), fix the agent definition (GREEN - outputs now correct), then handle edge cases (REFACTOR - robust under all conditions).
Core principle: If you didn't run an agent with test inputs and verify its outputs, you don't know if the agent works correctly.
REQUIRED BACKGROUND: You MUST understand ring:test-driven-development before using this skill. That skill defines the fundamental RED-GREEN-REFACTOR cycle. This skill provides agent-specific test formats (test inputs, output verification, accuracy metrics).
Key difference from testing-skills-with-subagents:
- Skills = instructions that guide behavior; test if agent follows rules under pressure
- Agents = separate Claude instances via Task tool; test if they produce correct outputs
The Iron Law
NO AGENT DEPLOYMENT WITHOUT RED-GREEN-REFACTOR TESTING FIRSTAbout to deploy an agent without completing the test cycle? You have ONLY one option:
STOP. TEST FIRST. THEN DEPLOY.
You CANNOT:
- ❌ "Deploy and monitor for issues"
- ❌ "Test with first real usage"
- ❌ "Quick smoke test is enough"
- ❌ "Tested manually in Claude UI"
- ❌ "One test case passed"
- ❌ "Agent prompt looks correct"
- ❌ "Based on working template"
- ❌ "Deploy now, test in parallel"
- ❌ "Production is down, no time to test"
ZERO exceptions. Simple agent, expert confidence, time pressure, production outage - NONE override testing.
Why this is absolute: Untested agents fail in production. Every time. The question is not IF but WHEN and HOW BADLY. A 20-minute test suite prevents hours of debugging and lost trust.
When to Use
Test agents that:
- Analyze code/designs and produce findings (reviewers)
- Generate structured outputs (planners, analyzers)
- Make decisions or categorizations (severity, priority)
- Have defined output schemas that must be followed
- Are used in parallel workflows where consistency matters
Test exemptions require explicit human partner approval:
- Simple pass-through agents (just reformatting) - only if human partner confirms
- Agents without structured outputs - only if human partner confirms
- You CANNOT self-determine test exemption
- When in doubt → TEST
TDD Mapping for Agent Testing
| TDD Phase | Agent Testing | What You Do |
|---|---|---|
| RED | Run with test inputs | Dispatch agent, observe incorrect/incomplete outputs |
| Verify RED | Document failures | Capture exact output issues verbatim |
| GREEN | Fix agent definition | Update prompt/schema to address failures |
| Verify GREEN | Re-run tests | Agent now produces correct outputs |
| REFACTOR | Test edge cases | Ambiguous inputs, empty inputs, complex scenarios |
| Stay GREEN | Re-verify all | Previous tests still pass after changes |
Same cycle as code TDD, different test format.
RED Phase: Baseline Testing (Observe Failures)
Goal: Run agent with known test inputs - observe what's wrong, document exact failures.
This is identical to TDD's "write failing test first" - you MUST see what the agent actually produces before fixing the definition.
Process:
- Create test inputs (known issues, edge cases, clean inputs)
- Run agent - dispatch via Task tool with test inputs
- Compare outputs - expected vs actual
- Document failures - missing findings, wrong severity, bad format
- Identify patterns - which input types cause failures?
Test Input Categories
| Category | Purpose | Example |
|---|---|---|
| Known Issues | Verify agent finds real problems | Code with SQL injection, hardcoded secrets |
| Clean Inputs | Verify no false positives | Well-written code with no issues |
| Edge Cases | Verify robustness | Empty files, huge files, unusual patterns |
| Ambiguous Cases | Verify judgment | Code that could go either way |
| Severity Calibration | Verify severity accuracy | Mix of critical, high, medium, low issues |
Minimum Test Suite Requirements
Before deploying ANY agent, you MUST have:
| Agent Type | Minimum Test Cases | Required Coverage |
|---|---|---|
| Reviewer agents | 6 tests | 2 known issues, 2 clean, 1 edge case, 1 ambiguous |
| Analyzer agents | 5 tests | 2 typical, 1 empty, 1 large, 1 malformed |
| Decision agents | 4 tests | 2 clear cases, 2 boundary cases |
| Planning agents | 5 tests | 2 standard, 1 complex, 1 minimal, 1 edge case |
Fewer tests = incomplete testing = DO NOT DEPLOY.
One test case proves nothing. Three tests are suspicious. Six tests are minimum for confidence.
Example Test Suite for Code Reviewer
| Test | Input | Expected |
|---|---|---|
| SQL Injection | String concatenation in SQL | CRITICAL, OWASP A03:2021 |
| Clean Auth | Proper JWT validation | No findings or LOW only |
| Ambiguous Error | Caught but only logged | MEDIUM silent failure |
| Empty File | Empty source | Graceful handling |
Running the Test
Dispatch via Task tool with test input → Document exact output verbatim (don't summarize).
GREEN Phase: Fix Agent Definition (Make Tests Pass)
Write/update agent definition addressing specific failures documented in RED phase.
Common fixes:
| Failure Type | Fix Approach |
|---|---|
| Missing findings | Add explicit instructions to check for X |
| Wrong severity | Add severity calibration examples |
| Bad output format | Add output schema with examples |
| False positives | Add "don't flag X when Y" instructions |
| Incomplete analysis | Add "always check A, B, C" checklist |
Example Fix: Severity Calibration
RED Failure: Agent marked hardcoded password as MEDIUM instead of CRITICAL
GREEN Fix: Add severity calibration: CRITICAL (hardcoded secrets, SQL injection, auth bypass), HIGH (missing validation, error exposure), MEDIUM (verbose errors, missing security headers), LOW (headers, deps)
Re-run Tests
After fixing, re-run ALL test cases. If any fail → continue fixing, re-test.
VERIFY GREEN: Output Verification
Goal: Confirm agent produces correct, well-structured outputs consistently.
Accuracy Metrics
| Metric | Target |
|---|---|
| True Positives | 100% |
| False Positives | <10% |
| False Negatives | <5% |
| Severity Accuracy | >90% |
| Schema Compliance | 100% |
Consistency Testing
Run same input 3 times → outputs should be identical. Inconsistency indicates ambiguous agent definition.
REFACTOR Phase: Edge Cases and Robustness
Agent passes basic tests? Now test edge cases.
Edge Case Categories
| Category | Test Cases |
|---|---|
| Empty/Null | Empty file, null input, whitespace only |
| Large | 10K line file, deeply nested code |
| Unusual | Minified code, generated code, config files |
| Multi-language | Mixed JS/TS, embedded SQL, templates |
| Ambiguous | Code that could be good or bad depending on context |
Stress Testing
Test edge cases: Large file (5000 lines, 20 issues), Complex nesting (15-level deep). Verify all issues found with reasonable response time.
Ambiguity Testing
Test context-dependent cases (e.g., hardcoded password with "local dev" comment). Agent should flag but acknowledge context.
Plugging Holes
For each edge case failure, add explicit handling to agent definition:
- Empty files: Return "No code to review" with PASS
- Large files: Focus on high-risk patterns first
- Minified code: Note limitations
- Context comments: Consider but don't use to dismiss issues
Testing Parallel Agent Workflows
When agents run in parallel (like 5 reviewers), test combined workflow:
- Parallel Consistency: Same input to all reviewers → check findings overlap appropriately, no contradictions
- Aggregation Testing: Same issue found by multiple reviewers → severity should be consistent; fix misalignments
Agent Testing Checklist
RED Phase: Create test inputs (known issues, clean, edge cases) → Run agent → Document failures verbatim
GREEN Phase: Update agent definition → Re-run tests → All pass
REFACTOR Phase: Test edge cases → Test stress scenarios → Add explicit handling → Verify consistency (3+ runs) → Test parallel integration (if applicable) → Re-run ALL tests after each change
Metrics (reviewer agents): True positive >95%, False positive <10%, False negative <5%, Severity accuracy >90%, Schema compliance 100%, Consistency >95%
Prohibited Testing Shortcuts
You CANNOT substitute proper testing with:
| Shortcut | Why It Fails |
|---|---|
| Reading agent definition carefully | Reading ≠ executing. Must run agent with inputs. |
| Manual testing in Claude UI | Ad-hoc ≠ reproducible. No baseline documented. |
| "Looks good to me" review | Visual inspection misses runtime failures. |
| Basing on proven template | Templates need validation for YOUR use case. |
| Expert prompt engineering knowledge | Expertise doesn't prevent bugs. Tests do. |
| Testing after first production use | Production is not QA. Test before deployment. |
| Monitoring production for issues | Reactive ≠ proactive. Catch issues before users do. |
| Deploy now, test in parallel | Parallel testing still means untested code in production. |
ALL require running agent with documented test inputs and comparing outputs.
Testing Agent Modifications
EVERY agent edit requires re-running the FULL test suite:
| Change Type | Required Action |
|---|---|
| Prompt wording changes | Full re-test |
| Severity calibration updates | Full re-test |
| Output schema modifications | Full re-test |
| Adding edge case handling | Full re-test |
| "Small" one-line changes | Full re-test |
| Typo fixes in prompt | Full re-test |
"Small change" is not an exception. One-line prompt changes can completely alter LLM behavior. Re-test always.
Common Mistakes
| Mistake | Fix |
|---|---|
| Testing only "happy path" inputs | Include ambiguous + edge cases |
| Not documenting exact outputs | Capture verbatim, compare to expected |
| Fixing without re-running all tests | Re-run entire suite after each change |
| Testing single agent in isolation (parallel workflow) | Test parallel dispatch + aggregation |
| Not testing consistency | Run same input 3+ times |
| Skipping severity calibration | Add explicit severity examples |
| Not testing edge cases | Test empty, large, unusual, ambiguous |
| Single test case validation | Minimum 4-6 test cases per agent type |
| Manual UI testing as substitute | Document all test inputs and expected outputs |
| Skipping re-test for "small" changes | Re-run full suite after ANY modification |
Rationalization Table
| Excuse | Reality |
|---|---|
| "Agent prompt is obviously correct" | Obvious prompts fail in practice. Test proves correctness. |
| "Tested manually in Claude UI" | Ad-hoc ≠ reproducible. No baseline documented. |
| "One test case passed" | Sample size = 1 proves nothing. Need 4-6 cases minimum. |
| "Will test after first production use" | Production is not QA. Test before deployment. Always. |
| "Reading prompt is sufficient review" | Reading ≠ executing. Must run agent with inputs. |
| "Changes are small, re-test unnecessary" | Small changes cause big failures. Re-run full suite. |
| "Based agent on proven template" | Templates need validation for your use case. Test anyway. |
| "Expert in prompt engineering" | Expertise doesn't prevent bugs. Tests do. |
| "Production is down, no time to test" | Deploying untested fix may make outage worse. Test first. |
| "Deploy now, test in parallel" | Untested code in production = unknown behavior. Unacceptable. |
| "Quick smoke test is enough" | Smoke test misses edge cases. Full suite required. |
| "Simple pass-through agent" | You cannot self-determine exemptions. Get human approval. |
Red Flags - STOP and Test Now
If you catch yourself thinking ANY of these, STOP. You're about to violate the Iron Law:
- Agent edited but tests not re-run
- "Looks good" without execution
- Single test case only
- No documented baseline
- No edge case testing
- Manual verification only
- "Will test in production"
- "Based on template, should work"
- "Just a small prompt change"
- "No time to test properly"
- "One quick test is enough"
- "Agent is simple, obviously works"
- "Expert intuition says it's fine"
- "Production is down, skip testing"
- "Deploy now, test in parallel"
All of these mean: STOP. Run full RED-GREEN-REFACTOR cycle NOW.
Quick Reference (TDD Cycle for Agents)
| TDD Phase | Agent Testing | Success Criteria |
|---|---|---|
| RED | Run with test inputs | Document exact output failures |
| Verify RED | Capture verbatim | Have specific issues to fix |
| GREEN | Fix agent definition | All basic tests pass |
| Verify GREEN | Re-run all tests | No regressions |
| REFACTOR | Test edge cases | Robust under all conditions |
| Stay GREEN | Full test suite | All tests pass, metrics met |
Example: Testing a New Reviewer Agent
Step 1: Create Test Suite
| Test | Input | Expected |
|---|---|---|
| SQL Injection | "SELECT * FROM users WHERE id = " + user_id | CRITICAL, OWASP A03:2021 |
| Parameterized (Clean) | db.execute(query, [user_id]) | No findings |
| Hardcoded Secret | API_KEY = "sk-1234..." | CRITICAL |
| Env Variable (Clean) | os.environ.get("API_KEY") | No findings |
| Empty File | (empty) | Graceful handling |
| Ambiguous | password = "dev123" # Local dev | Flag with context |
Step 2: RED Phase - Run tests, document failures: Test 1 marked HIGH not CRITICAL, Test 3 missed, Test 5 errored, Test 6 dismissed.
Step 3: GREEN Phase - Fix definition: Add severity calibration (SQL=CRITICAL), hardcoded secrets pattern, empty file handling, "context comments dont dismiss issues".
Step 4: Re-run - All tests pass with correct severities and handling.
Step 5: REFACTOR - Add edge cases: minified code, 10K line file, mixed languages, nested vulnerabilities. Run, fix, repeat.
The Bottom Line
Agent testing IS TDD. Same principles, same cycle, same benefits.
If you wouldn't deploy code without tests, don't deploy agents without testing them.
RED-GREEN-REFACTOR for agents works exactly like RED-GREEN-REFACTOR for code:
- RED: See what's wrong (run with test inputs)
- GREEN: Fix it (update agent definition)
- REFACTOR: Make it robust (edge cases, consistency)
Evidence before deployment. Always.
Blocker Criteria
STOP and report if:
| Decision Type | Blocker Condition | Required Action |
|---|---|---|
| No test inputs | Cannot create meaningful test cases for agent | STOP and define expected inputs/outputs before testing |
| Agent unreachable | Cannot dispatch agent via Task tool | STOP and fix agent accessibility first |
| Inconsistent outputs | Same input produces different outputs across runs | STOP and fix agent definition ambiguity |
| Below metrics threshold | Agent fails accuracy metrics after multiple fixes | STOP and report for human partner review |
Cannot Be Overridden
The following requirements CANNOT be waived:
- MUST run agent with documented test inputs before deployment
- MUST complete minimum test suite (4-6 cases per agent type)
- MUST re-run FULL test suite after ANY agent modification
- MUST NOT substitute manual UI testing for documented test runs
- MUST NOT deploy agent that fails accuracy metrics
Severity Calibration
| Severity | Condition | Required Action |
|---|---|---|
| CRITICAL | Agent deployed without any testing | MUST halt deployment, run full test suite immediately |
| HIGH | Agent modified but not re-tested | MUST re-run full test suite before any further use |
| MEDIUM | Agent tested but edge cases missing | MUST add edge case tests before production use |
| LOW | Agent passes but consistency below 95% | Should investigate and fix definition ambiguity |
Pressure Resistance
| User Says | Your Response |
|---|---|
| "Deploy now, we'll test in production" | "CANNOT deploy untested agent. Production is not QA. Will complete test suite first." |
| "One test case passed, ship it" | "MUST have minimum 4-6 test cases. One case proves nothing about agent behavior." |
| "I reviewed the prompt carefully, it's fine" | "CANNOT substitute prompt review for execution. Must run agent with test inputs." |
| "Small change, no need to re-test everything" | "MUST re-run full test suite after ANY change. Small prompt changes can alter behavior completely." |
Anti-Rationalization Table
| Rationalization | Why It's WRONG | Required Action |
|---|---|---|
| "Agent prompt is obviously correct" | Obvious prompts fail in practice. Execution proves correctness, not reading. | MUST run with test inputs |
| "Tested manually in Claude UI" | Manual testing is ad-hoc with no documented baseline. Not reproducible. | MUST document inputs and expected outputs |
| "Based on a proven template" | Templates need validation for YOUR use case. Context changes behavior. | MUST test even template-based agents |
| "Expert in prompt engineering" | Expertise doesn't prevent bugs. Only tests catch runtime failures. | MUST test regardless of expertise |
| "Change was tiny, re-test is overkill" | Tiny changes cause big behavior shifts in LLMs. One word can change everything. | MUST re-run full suite after any change |