Experiment Designer

Description

Use when planning product experiments, writing testable hypotheses, estimating sample size, prioritizing tests, or interpreting A/B outcomes with practical statistical rigor.

What this skill does

1. 1Writes testable hypotheses in If/Then/Because format with explicit intervention, measurable metric change, and causal mechanism.
2. 2Calculates required sample size per variant from baseline conversion rate, minimum detectable effect, alpha, and statistical power.
3. 3Scores and ranks experiments using ICE (Impact, Confidence, Ease) to prioritize the highest-return tests.
4. 4Defines primary metric, guardrail metrics, and stopping rules before any test launches to prevent mid-experiment changes.
5. 5Interprets results by comparing confidence intervals to decision thresholds, checking for novelty effects and segment heterogeneity.

When to use

• A conversion hypothesis is ready to test and you need a sample size estimate before starting.
• Multiple experiments are queued and you need a prioritized order based on effort and expected impact.
• A test result is statistically significant but the team is unsure whether it is large enough to ship.
• An experiment backlog needs hypothesis quality review before any test is approved to run.

Frequently asked questions