1. Definition
- Team Draft Interleaving (TDI) is an online evaluation technique for comparing two ranking algorithms (say Algorithm A vs Algorithm B).
- Like Balanced Interleaving, it builds a single mixed ranking list from both algorithms and shows it to users.
- The difference is: TDI uses a draft-pick process (like picking players for a sports team) to decide which algorithm contributes the next item to the combined list.
This ensures fair representation and reduces bias from list position.
2. Why It’s Used
- Problem with A/B testing: Splits traffic between A and B, requiring large samples.
- Problem with Balanced Interleaving: Can sometimes favor one algorithm depending on overlap/order.
- Solution – TDI: Randomizes the “drafting” process to ensure each algorithm gets equal opportunity.
3. How Team Draft Interleaving Works
- Take the top-k ranked results from Algorithm A and Algorithm B.
- Randomly select which algorithm gets the first pick.
- Alternate “draft picks”:
- If A picked first, then B picks second, then A again, etc.
- Each algorithm selects its top-most unchosen item that hasn’t yet appeared in the combined list.
- Continue until the final interleaved ranking is filled.
- Show this interleaved list to the user.
- Attribute each user click to the algorithm that drafted that item.
4. Example
Suppose we have two algorithms:
- Algorithm A: [A1, A2, A3, A4]
- Algorithm B: [B1, B2, B3, B4]
Draft process (randomly chosen):
- Round 1: A picks first → A1
- Round 2: B picks next → B1
- Round 3: A picks → A2
- Round 4: B picks → B2
- … and so on.
Final list (interleaved): [A1, B1, A2, B2, A3, B3, A4, B4]
If a user clicks B2, credit goes to Algorithm B.
5. Advantages of TDI
- Fair & unbiased: Randomized drafting prevents systematic bias.
- Efficient: Requires fewer users than A/B testing to detect differences.
- Click attribution clarity: Each item is clearly “owned” by one algorithm.
- Sensitive: Good at detecting small performance differences.
6. Limitations
- Only compares two algorithms at a time.
- Doesn’t naturally extend to >2 algorithms (though variations exist).
- Still assumes clicks = relevance, which may be noisy.
- Needs careful handling of ties and overlapping results.
7. Comparison: Balanced Interleaving vs Team Draft Interleaving
| Feature | Balanced Interleaving | Team Draft Interleaving (TDI) |
|---|---|---|
| Mixing method | Alternate items from A and B | Randomized draft pick between A & B |
| Fairness | Can sometimes favor one algorithm if overlaps exist | More balanced, avoids systematic bias |
| Randomization | Limited | Stronger (random draft order each time) |
| Click attribution | Sometimes ambiguous | Always clear (each pick belongs to A or B) |
| Use case | Simple, small tests | Preferred industry standard for fairness |
In short:
Team Draft Interleaving (TDI) is an online evaluation method for comparing two ranking algorithms. It works by letting algorithms take turns “drafting” results into a shared ranking list. Users interact with the combined list, and clicks are credited to whichever algorithm drafted the item. TDI is considered more robust and fair than Balanced Interleaving.