Hit Rate

Hit rate at K (HR@K, also called hit ratio) is the fraction of evaluated users, or test cases, whose top-K list contains at least one relevant item. He et al. (2017) describe it as measuring whether the test item is present on the top-K list.

In leave-one-out evaluation

In the protocol of He et al., each user’s latest interaction is held out as the test item and the rest of the data is used for training. The test item is ranked among 100 randomly sampled items the user has not interacted with, and the lists are truncated at 10, so the main figure is HR@10. With exactly one relevant item per user, hit rate at K equals recall at K, because recall is 1 when the item is in the list and 0 when it is not. With several relevant items the two differ. If a user has three relevant items and one is in the top K, hit rate counts 1 for that user, while recall is 1/3.

A worked example

These ranks are hypothetical. Five users have held-out items at ranks 1, 3, 11, 7, and 25, and K = 10.

HR@10 is 3/5 = 0.6. NDCG@10 averages to 0.367. A hit at rank 1 and a hit at rank 10 both count as 1 for hit rate, whereas NDCG rewards the earlier position.

What changes the number

• K. HR@5 is never larger than HR@10 for the same lists.
• Candidate protocol. Ranking against every eligible item and ranking against a sample of distractors are different tasks. Removing competing items cannot push a fixed target down the list, so hit rate computed against sampled candidates is at least as high as against the full set for an unchanged model. The Recommender Systems article shows how large that gap can be for NDCG.
• The definition of relevant. See Relevance in Recommender Systems.
• The user population. Users with few or no prior interactions create a recommendation cold-start problem: the system has little evidence of their preferences. Their inclusion affects both the hit count and the number of evaluated users, so report the population and results for this group.

Hit rate is easy to interpret, but it cannot distinguish a good ordering inside the top K from a poor one.

References:

• He et al., Neural Collaborative Filtering (WWW 2017), arXiv:1708.05031