Flagship · 11 weeks
Readout Craft
A working course in A/B Test Analytics for Apps. You leave with a pre-analysis habit, a metric tree you can defend, and a readout format that does not collapse when someone asks about leakage.
Informational fee: £1,180 for the full desk, or £410 for the first three modules as a sampler. There is no payment form on this site.
Outcomes we will hold you to
- Write a pre-analysis plan before any look at the live dashboard.
- Spot assignment leakage across iOS, Android, and a shared login.
- Choose a primary metric and two guardrails that are not restatements of each other.
- Explain CUPED in a product meeting without waving at a paper.
- Ship a readout that states sample, window, and what you still do not know.
Modules
-
Metric trees for mobile
We draw the tree for a subscription or commerce app, then prune nodes that exist only because the event catalogue is large. You will name one primary metric and refuse a second “also important” conversion.
-
Assignment, bucketing, and leakage
Sticky assignment, device graphs, and the quiet ways a WebView or server-side flag reassigns a user. Homework is a teardown of a fictional Android experiment that looks clean in the tool.
-
Sequential looks without lying
Peeking, spending alpha, and what a Friday screenshot is allowed to mean. We practise stopping rules that a statistician at a GB bank would not laugh at.
-
Variance reduction that earns its keep
CUPED and covariate choice. If a covariate is just the same metric with a lag, we say so. Spreadsheets, not a Python theatre.
-
Guardrails, holdouts, and ship gates
Crash, refund, latency, and a long-run holdout when the feature will never come back out of the binary. Includes a “do not ship” memo.
-
The readout a staff engineer will argue with
A written artefact: context, design, results, residual risks. You present it once to the cohort. Politeness is optional; completeness is not.
Instructor
Nia Cartwright taught measurement inside a London payments app for six years, then ran a two-person analytics consultancy for GB scale-ups. She still reviews assignment code when a student pastes a diff. She is not a motivational speaker.
Who this is not for
If you need a first introduction to “what is an A/B test”, this desk will feel impatient. We assume you have shipped at least one experiment and been disappointed by it.
FAQ
Is this a statistics degree in eleven weeks?
No. We cover enough inference to keep you honest, and we send you to longer texts for proofs. If you want a full CUPED derivation from first principles, this is the wrong purchase.
What software do you require?
Whatever your company already uses for flags and analytics. We do not lock the desk to one vendor. Spreadsheets are enough for the exercises.
A real limitation
We cannot give you more users. Small-sample iOS apps leave the course still small. We will teach you to stop claiming 8% lifts on 900 weekly actives, which is useful and also frustrating if your board wanted a number.
How do fees work?
Quoted above as information only. Enrolment is arranged by email. See refunds if you need to withdraw.
Voices from this desk
Module two on leakage was the only class I have taken that made me open the Android flag code the same evening. The CUPED week still feels like extra homework I only half finished.
★★★★☆Clear writing drills. I wanted more live SQL. Still used the readout template on a paywall test the week after the last seminar.