Balanced Interleaving
Balanced interleaving compares two result lists for the same query. In the description given by Hofmann et al. (2011), it takes two lists and produces an outcome of −1, 0, or 1, meaning that the first list is judged worse than, equal to, or better than the second. It is one of the methods behind interleaving tests.
Building the blended list
- Pick one of the two lists at random as the starting list and place its first item at the top.
- The other list contributes its highest-ranked item that is not already in the blended list.
- The lists keep taking turns, each skipping items already placed, until the blended list has the desired length.
Crediting clicks
For each original list, find its lowest-ranked item that received a click, and let k be the smaller of the two ranks. Then count, for each list, the clicked items that it ranks at or above k. The list with more clicks in its top k is judged better, and equal counts are a tie.
A worked example, and where the bias comes from
This example was constructed for this entry. Let the first ranker return (a, b, c, d) and the second return (b, c, d, a), so the two lists contain the same items and differ only by a shift. The blended list is (a, b, c, d) when the first list starts and (b, a, c, d) when the second starts. Now suppose the user clicks exactly one item, chosen at random with no preference between the rankers:
| Clicked item | Rank in list 1 | Rank in list 2 | k | Top-k of list 1 | Top-k of list 2 | Credit |
|---|---|---|---|---|---|---|
| a | 1 | 4 | 1 | a | b | List 1 |
| b | 2 | 1 | 1 | a | b | List 2 |
| c | 3 | 2 | 2 | a, b | b, c | List 2 |
| d | 4 | 3 | 3 | a, b, c | b, c, d | List 2 |
The second list is credited in three of four cases even though neither ranker is better. A method with this property is biased: it can prefer one ranker for reasons unrelated to quality. Hofmann et al. describe the bias of balanced interleaving as previously known, and their simulations report that, for some ranker pairs, it and the document-constraint method do not select the better ranker however many queries are observed.
What to use instead
Team draft interleaving avoids this form of bias by assigning each item to the ranker that contributed it and crediting a click only to that ranker. Probabilistic interleaving adds sampling so that small ranking differences carry weight. Balanced interleaving remains useful as the baseline that the others are compared against.
Whatever the method, a click-based preference says which ranker the credited clicks favored. It does not say why, and it inherits position effects from the presentation.
References:
Discover more from Insightful Data Lab
Subscribe to get the latest posts sent to your email.