🧪 Creative Test Framework

Statistically rigorous A/B/n testing for video ad creatives. Stop guessing which ad wins. Run proper experiments, get confidence intervals, and let the data decide.

Built by Sediman AI — because most marketers A/B test wrong.

---

Why This Exists

Most "A/B tests" in ad creative are just:

"We ran two ads for a week and the one with more clicks won."

That's not testing. That's vibing. This tool gives you:

• ✅ Statistical significance — z-tests with p-values
• ✅ Confidence intervals — know the range, not just the point estimate
• ✅ Sample size calculator — know how much data you actually need
• ✅ Multi-variant support — A/B/n tests, not just A/B
• ✅ Multiple metrics — CTR, CVR, CPA, ROAS

Install

pip install creative-test-framework

Or just grab the single file:

curl -O https://raw.githubusercontent.com/sediman-ai/creative-test-framework/main/abtest.py
python abtest.py --help

Quick Start

1. Generate sample data to play with

creative-test generate --output my_experiment.csv

2. Analyze your results

creative-test analyze my_experiment.csv --metric ctr --confidence 0.95

Output:

══════════════════════════════════════════════════════════════════════
  🧪 EXPERIMENT: my_experiment
  📊 Metric: CTR | Confidence: 95%
  👁️  Total Impressions: 420,000
══════════════════════════════════════════════════════════════════════

  Variant         Impr   Clicks    Conv    CTR%    CVR%      CPA   ROAS
  ────────────────────────────────────────────────────────────────────
  control       140,000    2,800     84   2.000%  3.000%  $38.57  2.14
  variant_a     140,000    3,500    122   2.500%  3.486%  $27.05  3.42
  variant_b     140,000    2,520     71   1.800%  2.817%  $45.83  1.87

  ⚡ STATISTICAL COMPARISONS (vs Control):
  ✅ variant_a      vs control
     Z=+4.2145  p=0.000025  lift=+25.0%
     🏆 variant_a WINS with +25.0% lift!
  ⏳ variant_b      vs control
     Not yet significant — keep collecting data

3. Calculate required sample size

creative-test plan --baseline-ctr 2.0 --mde 20 --confidence 95 --power 80

4. Export results as JSON

creative-test analyze data.csv --json results.json

CSV Format

Your CSV should have these columns:

Column	Required	Description
variant	✅	Name of the creative variant
impressions	✅	Number of impressions served
clicks	✅	Number of clicks
conversions	✅	Number of conversions
spend	✅	Total spend ($))
revenue	✅	Total revenue ($)
date	Optional	Date of observation (for time series)

Data is automatically aggregated by variant.

Features

Feature	Description
Z-test	Two-proportion z-test for statistical significance
Wilson CI	Wilson score confidence intervals
Sample size	Pre-experiment sample size calculator
Multi-variant	A/B/n support — test 3, 5, 10 variants at once
JSON export	Machine-readable results for dashboards
Zero deps	Pure Python stdlib — no numpy, scipy, or pandas needed

The Math

We use a two-proportion z-test:

z = (p₁ - p₂) / √(p̂(1-p̂)(1/n₁ + 1/n₂))

Where p̂ is the pooled proportion. Confidence intervals use the Wilson score method for better accuracy at small sample sizes.

When to Stop Testing

• ✅ p-value < 0.05 AND you've hit your pre-calculated sample size
• ❌ Don't stop early just because p < 0.05 (peeking problem)
• ❌ Don't test for less than 7 days (day-of-week effects)

Generate Variants Fast

The hard part of A/B testing isn't the math — it's creating enough variants to test. That's where Sediman comes in. Generate 50 video ad variants from text prompts in the time it takes to brew coffee.

---

License

MIT © Sediman AI

---