Jaccard index

The Jaccard index of two sets \(A\) and \(B\) is the size of their intersection divided by the size of their union: \(J(A,B) = |A \cap B| / |A \cup B|\). In scikit-learn’s documentation for jaccard_score, it is defined in this form for sets of labels. It is 1 for identical sets and 0 for sets with nothing in common, and \(1 – J\) is the Jaccard distance.

A worked example

These genre sets are hypothetical. Movie A has the genres {Action, Sci-Fi, Adventure} and movie B has {Action, Sci-Fi}. The intersection has 2 elements and the union has 3, so \(J = 2/3 = 0.667\) and the distance is 0.333. Movie C with {Drama} shares nothing with either, so its index with both is 0.

In binary classification, scikit-learn computes the index for the positive class. For true labels [0, 1, 1] and predictions [1, 1, 1], the true positive set has 2 members and the predicted positive set has 3, with 2 in common, so the score is 0.667.

Related measures

Other formulas compare the same two sets differently.

Because B is a subset of A, the overlap coefficient calls them identical, while Jaccard penalizes the difference in size. Which one is appropriate depends on whether containment should count as a match.

Edge cases and limits

• Two empty sets. The ratio is 0/0. Scikit-learn documents that the score is undefined when there are no true or predicted labels and returns 0 with a warning by default, and it lets the caller choose the value with its zero_division parameter. Other code treats two empty sets as identical. Whichever rule is used should be stated.
• Membership only. Counts and weights are ignored. A set of tags with different importances needs a weighted variant.
• No semantic similarity. Two different elements, such as Sci-Fi and Fantasy, contribute nothing to the intersection however close they are in meaning.

In recommender evaluation the index turns tag sets into a similarity, which can be converted into the distance used in Intra-List Diversity (ILD). Genre Overlap applies it to genre tags.

References:

• scikit-learn documentation, jaccard_score