Discovery ranker · experiment brief
Luma Audio · seed 424242 · generated from a completed simulation
- Hypothesis
- Switching from Hybrid to Discovery Balanced will get about -2% more of our catalog in front of people, without people getting noticeably worse matches.
- Audience
- Discovery Explorer segment, plus a cold-start holdout for guardrails.
- Control strategy
- Hybrid — Blends affinity, behavioural association and popularity.
- Treatment strategy
- Discovery Balanced — Trades a little relevance for materially wider catalog exposure.
- Primary metric
- How many different items each person sees in a week.
- Guardrail metrics
- Things that must not get worse: how good the matches feel, how often people see repeats, whether they finish sessions, and how brand-new users get on.
- Event instrumentation
- Impression events with slot position and above/below-fold flag; click, save, skip and complete with item id and session id; a strategy-variant field on every event.
- Rollout approach
- Shadow-score both rankers for one week, then a 5% holdout, then 50/50 once guardrails hold for five consecutive days.
- Success threshold
- Agree the threshold before launch from your own baseline variance. TasteLab cannot supply it: these are simulation outputs, not measured effects.
- Risks
- In the simulation, attention spreads out more evenly. But a real catalog has licensing, availability and editorial rules this model knows nothing about, and brand-new users should be checked separately.
- Recommended next action
- Run the same comparison against your own catalog before designing the online test. If coverage does not move on real data, the ranker is not the constraint — retrieval is.