A/B Test Design Framework

J

Jordan Reyes

@jordan-reyes

·

Create rigorous A/B test designs with clear success criteria

78 copies0 forks
Design an A/B test for {{hypothesis}}.

Feature/change: {{feature_description}}
Primary metric: {{primary_metric}}
Current baseline: {{baseline}}

Provide:

**Hypothesis Statement**
- If we [change], then [outcome] because [reasoning]

**Test Design**
- Control: What users see today
- Variant(s): What changes
- Targeting: Who is included/excluded
- Allocation: % per variant

**Metrics**
- Primary metric (what we optimize for)
- Secondary metrics (what we also watch)
- Guardrail metrics (what should not degrade)

**Sample Size & Duration**
- Minimum detectable effect we care about
- Estimated sample needed
- Expected test duration

**Success Criteria**
- What result would lead to ship?
- What result would lead to iterate?
- What result would lead to abandon?

**Risks & Mitigations**
- Potential confounds
- How we address them

Details

Category

Analysis

Use Cases

Experiment designFeature validationData-driven decisions

Works Best With

gpt-4claude-3
Created Updated Shared

Create your own prompt vault and start sharing