Domain Context
This skill implements a proven product management framework. The approach combines best practices from industry leaders and is designed for practical application in day-to-day PM work.
Input Requirements
- Context about your product, feature, or problem
- Relevant data, research, or constraints (recommended but optional)
- Clear articulation of what you're trying to achieve
A/B Test Designer
When to Use
- Testing a new feature or design variation
- Validating a hypothesis before full rollout
- Optimizing conversion rates or key metrics
- Choosing between multiple design approaches
- Need to make a data-driven decision on a change
What This Skill Does
Helps you design rigorous A/B tests with clear hypotheses, success metrics, sample size calculations, and analysis plans.
Instructions
Help me design an A/B test for [feature/change]. Include:
- Hypothesis
- Current situation and metrics - Proposed change - Expected impact and why
- Test Design
- Primary success metric - Secondary metrics - Sample size needed - Test duration - User segments to include/exclude
- Variants
- Control (A): current experience - Variant (B): new experience - Any additional variants (C, D, etc.)
- Risks and Controls
- Potential negative impacts - Guardrail metrics - When to stop the test early
- Analysis Plan
- Statistical significance threshold - How to handle edge cases - Decision criteria
Feature context: [Add context about the change you want to test]
Best Practices
- Start with a clear, falsifiable hypothesis
- Choose one primary metric to avoid multiple comparison issues
- Calculate sample size upfront based on expected effect size
- Run tests for full weekly cycles to account for day-of-week effects
- Set a minimum test duration (usually 1-2 weeks)
- Define success criteria before running the test
- Monitor guardrail metrics (revenue, errors, performance)
Example
Input: Testing new onboarding flow vs current 3-step process Output: Hypothesis (new 1-step flow will increase co...