Ab Test Setup

Description

When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," or "hypothesis." For tracking implementation, see analytics-tracking.

What this skill does

1. 1Guides hypothesis writing using a structured template: observation, change, expected outcome, audience, and success metric.
2. 2Calculates required sample size per variant based on baseline conversion rate, minimum detectable effect, and confidence level.
3. 3Selects primary, secondary, and guardrail metrics before the test launches to prevent scope creep mid-run.
4. 4Produces a pre-launch checklist covering hypothesis, tracking verification, QA on all variants, and traffic allocation.
5. 5Documents test results with context (external factors, segment differences) and a decision: implement, keep control, or iterate.

When to use

• You want to test a specific change (headline, CTA, layout) and need a statistically rigorous plan before touching the page.
• A product or marketing team is running tests but stopping early or testing multiple variables at once.
• You need to calculate whether your site has enough traffic to detect a meaningful lift within a reasonable timeframe.
• Post-test results need structured documentation so learnings carry forward to the next experiment.

Tools used

Frequently asked questions