What are Interleaving Tests?

• Interleaving tests are an online evaluation method where results from two ranking algorithms (Model A vs Model B) are mixed (interleaved) into a single combined list.
• A single user sees one interleaved list rather than being bucketed into “A” or “B”.
• The user’s clicks (or other interactions) are then attributed back to the originating model, and we compare which model receives more credit.

Instead of splitting users (like A/B tests), we split the result list shown to each user.

---

Why Use Interleaving?

• Faster sensitivity: Needs fewer users than A/B tests because both models are tested on the same user at the same time.
• Fairness: Removes between-user variability (since the same user is exposed to both models).
• Ideal for ranking problems (search results, recommendations, news feeds).

---

How It Works

1. Input: Two ranking models produce ordered lists of items.
   • Model A: [a1, a2, a3, …]
   • Model B: [b1, b2, b3, …]
2. Interleaving algorithm mixes them into a single list, e.g.:
   • Interleaved: [a1, b1, a2, b2, …]
3. User interaction: User clicks on items.
4. Credit assignment: If user clicks an item from A, Model A gets credit; if from B, Model B gets credit.
5. Comparison: Tally credits across many users → whichever model gets more wins.

---

Popular Interleaving Strategies

1. Balanced Interleaving
   • Alternate between A and B fairly, ensuring each gets equal exposure.
2. Team Draft Interleaving (TDI)
   • Like picking teams in a draft: A and B take turns choosing items for the interleaved list.
   • Guarantees fair distribution without duplicates.
3. Probabilistic Interleaving
   • Items are selected into the list with probabilities proportional to each model’s ranking scores.

---

Example

Suppose we test two search ranking models on 1,000 queries.

• Users clicked 600 results from Model A’s contributions.
• Users clicked 400 results from Model B’s contributions.

Conclusion: Model A provides better relevance for users.

---

Pros & Cons

Advantages

• Faster, more sensitive than A/B testing.
• Each user acts as their own control.
• Great for ranking/recommendation comparisons.

Limitations

• Only works when you can interleave results (ranked outputs).
• Harder to extend beyond 2 models.
• Interpretation depends on fair credit assignment.

---

When to Use

Use interleaving when:

• Comparing ranking algorithms (search, ads, recsys).
• You need fast feedback with limited traffic.
• User interactions are click-based or engagement-based.

Not ideal when:

• Outcomes are long-term (e.g., retention, churn).
• Task isn’t ranking-based (e.g., binary classification decisions).

---

Summary

• A/B tests → split users.
• Interleaving tests → split ranked results shown to each user.
• Interleaving = more efficient and sensitive for ranking model comparisons.