Seungman Kim, PhD
Quantitative researcher — causal inference, experimental design, measurement. Senior Research Associate, Texas Tech University Health Sciences Center.
Shipping a search ranking change on evidence that holds up
An end-to-end A/B analysis on simulated marketplace data: ratio metrics under clustered assignment, variance reduction, sequential monitoring, novelty decay, and a guardrail that is significant but not disqualifying.
The full two-week window says +3.2%. That number is inflated by a novelty response that has decayed by day eight. The durable estimate is the days 8–14 window, and that is the figure any forecast built on this launch should use. Search abandonment rose by 0.19 points, which is statistically detectable but comfortably inside the 0.5-point margin agreed before launch, so it does not block the release.
What was tested
A new ranking model for job search results, compared against the incumbent. Assignment is at the user level and persists for the full experiment, because a job seeker who sees two different rankings across sessions is no longer a clean unit.
| Element | Choice |
|---|---|
| Randomization unit | User, 50/50, sticky for 14 days |
| Analysis population | Users with at least one session in the window (388,167 of 400,000) |
| Primary metric | Applications per session, a ratio of sums |
| Guardrail | Search abandonment rate, non-inferiority margin 0.005 absolute |
| Covariate | Same metric over the 14 days before launch, used for CUPED |
| Decision rule | Fixed 14-day horizon; interim looks for harm only, against a calibrated boundary |
The primary metric is a ratio of two sums, not a mean over users, so the standard error needs
the delta method. I linearize each user to
(applications − R × sessions) / mean(sessions)
and work with those values, which keeps the unit of inference at the user where the randomization
happened. Treating each session as independent would have understated the standard error, and
heavy users would have quietly carried the result.
Before anything else: is the assignment intact
Counts came in at 193,752 treatment against 194,415 control, χ² = 1.13, p = 0.29. No sample ratio mismatch.
A clean check only means something if it could have failed. Re-running the same test on a copy where 1.5% of treatment users are dropped, which is what a redirect or logging bug looks like, gives χ² = 33.1, p < 10⁻⁸. The diagnostic has teeth at this sample size, so passing it is informative rather than decorative.
Precision before effects
Regressing out the pre-launch value of the same metric gives θ = 0.598 and cuts the variance of the linearized metric by 35.0%, shrinking the standard error to 0.81 of its unadjusted size. The point estimate moves from +3.47% to +3.22%, which is the adjustment for chance imbalance doing its job, not the method inventing an effect.
Why the headline number is the wrong number
Splitting by day shows the effect arriving large and settling down. Days 1–3 give +5.11% [+3.95%, +6.27%]. Days 8–14 give +2.54% [+1.73%, +3.36%]. The pooled two-week estimate of +3.22% is a weighted blend of a transient response and a durable one, and reporting it as the expected steady-state gain would overstate the launch by roughly a quarter.
This matters past the writeup. If a revenue forecast is built on the pooled figure, the launch underdelivers against its own business case for reasons that have nothing to do with the model.
What checking every day costs
Under a simulated null with 2,000 replicates and a look each day for fourteen days, the probability of crossing |z| > 1.96 at least once is 21.5%. Calibrating the boundary against that same null distribution puts the 5% threshold at |z| = 2.62.
On the full sample the effect is large enough that both rules agree on day one, so the distinction is academic here. On an 8% subsample, standing in for a lower-traffic surface, the naive rule would have stopped on day two and the calibrated rule waits until day four.
Who the change helps
Users in their first 30 days gain +4.46% [+3.37%, +5.55%]; returning users gain +2.56% [+1.77%, +3.35%]. The split was specified before the analysis, on the reasoning that a ranking change should matter most to people without enough history for personalization to work. It is the kind of split worth acting on: if it replicates, the ranking model is doing cold-start work, and that suggests where to invest next.
The guardrail
Abandonment moved from 0.1849 to 0.1867, an increase of 0.00185 [0.00117, 0.00254], p < 10⁻⁶. At this sample size almost any real movement is detectable, so significance is the wrong question. The pre-registered question is whether the upper bound clears the 0.005 margin, and it does. The change ships, with abandonment kept on the post-launch monitoring set rather than declared resolved.
What I would want before trusting this further
- A holdback past day 14. Two weeks separates novelty from steady state, but not steady state from slow erosion.
- Marketplace interference. Applications concentrating on fewer postings can look like a gain to seekers and a loss to employers; the seeker-side metric cannot see this.
- An employer-side counterpart metric before a full rollout, since the two sides of the marketplace can move in opposite directions.
- Replication of the segment effect on an independent window before it is used to set roadmap priorities.
Ground truth
The generator was set to a steady-state lift of +2.0%, with the day-one effect inflated 2.6× and decaying with a two-day time constant, an effect 1.8× larger for new users, and an absolute abandonment increase of 0.0015. The days 8–14 estimate of +2.54% covers the true steady state; the pooled estimate of +3.22% does not. The segment ratio recovers as 1.74 against a true 1.8. The guardrail estimate of 0.00185 covers the true 0.0015.