Journal
Variance reduction without the math theatre
CUPED is a tool. It is not a personality.
Teams discover CUPED, add last week’s conversion as a covariate, and announce that the test is now “more scientific”. Sometimes variance drops. Sometimes you have just dressed the same noisy metric in a jacket. A/B Test Analytics for Apps should make the covariate earn its keep: correlation with the outcome, measured on a pre-period, and a check that the covariate is not itself affected by the treatment.
If the feature changes onboarding, yesterday’s onboarding events are a contaminated covariate. If you only have two weeks of history because the event was renamed, you do not have a CUPED story, you have a taxonomy story. Fix the name first.
We keep the algebra short in class. The operational questions are longer: who computes the residual, where it is stored, and whether the readout shows both the adjusted and unadjusted estimates. Stakeholders who only see the adjusted number will not know what happened if the method is later questioned.
Small iOS samples still stay small after CUPED. Anyone who promises otherwise is selling a seminar, not a method. Use variance reduction to shorten a test that was already almost powered, not to rescue a test that never had a chance.
If your vendor’s “stats engine” is a black box, ask what covariate it used. If they cannot say, you are not running CUPED; you are running brand copy. That is allowed, as long as the readout does not pretend otherwise.