Probabilistic Interleaving
Instead of copying items from two fixed lists, probabilistic interleaving samples each slot from distributions built from the rankings, so small ranking differences change the credit.
-
-
Team Draft Interleaving (TDI)
Each round a coin flip decides which ranker picks first. Every item belongs to the ranker that picked it, so each click has exactly one owner.
-
Balanced Interleaving
Balanced interleaving alternates between two rankings and counts clicks in each ranking’s top k. With two rankings that differ only by a shift, one ranker can win most random clicks.
-
Relevance in Recommender Systems
A ranking metric can only reward what the labels call relevant. Judged, rated, clicked, and held-out labels measure different things, and a missing label is not a negative label.
-
Interleaving Tests
Instead of splitting users between two rankers, interleaving shows each request one blended list. The credit rule turns clicks into a preference, and some rules are biased.