An A/B analysis that runs its validity checks before it computes the effect, on a 13.9M-row Criteo uplift log.
Read it top to bottom. On the left are the raw log and the config that fixes alpha, power, the MDE and the ship bar before any effect is seen. The centre spine runs in order: ingest to per-arm sums, then the 6 guardrails (SRM, balance, the specification curve), then power, then the effect (z-test, CUPED, Lin, CACE), then the sequential and uplift stages. The right shows what each stage writes. The amber path is a guardrail firing.
Spark reduces 13.9M rows to per-arm sums and the driver receives a 68 KB parquet file. Every test here (z-test, CUPED, Hotelling T²) is additive, so it runs off those sums. Full run: about 80 seconds.
3 of 6 fire: the split is under-dispersed (ratio is exactly 17:3), the file is not shuffled (13% to 100% treatment by block), and all 12 features differ. The effect is only computed after they run.
The unadjusted lift spans 1.44x across 3 duplicate-handling specs, the adjusted one spans 1.12x. Adjustment cuts specification sensitivity 3.5x. That stability, more than the 10.7% variance reduction, is why CUPED earns its place here.
The 85/15 split misses by 2 users of 13.98M (p=0.9989), because control was downsampled to hit exactly 15%. It proves the check runs.
2.22M byte-identical rows carry zero conversions, so all are kept. Raw lift then reads +59.45% to +85.37% by choice; the CUPED lift barely moves.
The first headline was +50.76%, an adjusted numerator over an unadjusted control mean; the real figure is +47.36%. A test in tests/test_stats.py locks each one.