Intersection over Union for Object Detection
Object detection systems must identify an object’s class and estimate its location using a bounding box. Intersection over Union, usually abbreviated as IoU, measures how closely a predicted bounding box matches the ground-truth box.
IoU is widely used for:
- Evaluating object localization
- Matching predictions to ground-truth objects
- Removing duplicate detections with non-maximum suppression
- Defining training targets and localization losses
Ground-Truth and Predicted Bounding Boxes
For each object in an annotated image, there are usually two relevant boxes:
- Ground-truth box: The correct bounding box supplied with the labeled data
- Predicted box: The bounding box produced by the object detection model
A prediction should receive a high localization score when its box closely overlaps the ground-truth box. IoU provides a standardized way to measure this overlap.
Intersection and Union
Suppose the ground-truth box is \(B_{\text{gt}}\) and the predicted box is \(B_{\text{pred}}\).
Their intersection is the region contained in both boxes:\[ B_{\text{gt}} \cap B_{\text{pred}} \]
Their union is the entire region contained in either box:\[ B_{\text{gt}} \cup B_{\text{pred}} \]
Intersection over Union is defined as:\[ \operatorname{IoU} = \frac{ \operatorname{Area} \left( B_{\text{gt}} \cap B_{\text{pred}} \right) }{ \operatorname{Area} \left( B_{\text{gt}} \cup B_{\text{pred}} \right) } \]
Because the intersection cannot be larger than the union, IoU always lies between zero and one:\[ 0 \leq \operatorname{IoU} \leq 1 \]
Interpreting the IoU Score
The IoU value indicates how similar two bounding boxes are.
| IoU | Interpretation |
|---|---|
| \(0\) | The boxes do not overlap |
| Close to \(0\) | The boxes overlap only slightly |
| \(0.5\) | The overlap is moderate |
| Close to \(1\) | The boxes align very closely |
| \(1\) | The boxes are identical |
A higher IoU means more accurate localization.
However, IoU does not measure whether the predicted class is correct. A complete object-detection evaluation must consider both classification and localization.
Computing the Intersection
Represent each box using its corner coordinates:\[ B = (x_{\min}, y_{\min}, x_{\max}, y_{\max}) \]
For two boxes \(A\) and \(B\), the intersection coordinates are:\[ x_{\min}^{I} = \max(x_{\min}^{A},x_{\min}^{B}) \]\[ y_{\min}^{I} = \max(y_{\min}^{A},y_{\min}^{B}) \]\[ x_{\max}^{I} = \min(x_{\max}^{A},x_{\max}^{B}) \]\[ y_{\max}^{I} = \min(y_{\max}^{A},y_{\max}^{B}) \]
The intersection width and height are:\[ w_I = \max \left( 0,\, x_{\max}^{I}-x_{\min}^{I} \right) \]\[ h_I = \max \left( 0,\, y_{\max}^{I}-y_{\min}^{I} \right) \]
The maximum with zero is important. If the boxes do not overlap, the raw width or height would be negative, but the intersection area must be zero.
Therefore:\[ A_I = w_I h_I \]
Computing the Union
The areas of boxes \(A\) and \(B\) are:\[ A_A = (x_{\max}^{A}-x_{\min}^{A}) (y_{\max}^{A}-y_{\min}^{A}) \]\[ A_B = (x_{\max}^{B}-x_{\min}^{B}) (y_{\max}^{B}-y_{\min}^{B}) \]
Adding these two areas counts the intersection twice. The intersection must therefore be subtracted once:\[ A_U = A_A + A_B – A_I \]
The final IoU is:\[ \operatorname{IoU}(A,B) = \frac{A_I}{A_U} \]
Numerical Example
Suppose the ground-truth box is:\[ B_{\text{gt}} = (1,1,5,5) \]
and the predicted box is:\[ B_{\text{pred}} = (3,2,7,6) \]
The intersection extends from:\[ (3,2) \]
to:\[ (5,5) \]
Its width and height are:\[ w_I = 5-3=2 \]\[ h_I = 5-2=3 \]
Therefore, the intersection area is:\[ A_I = 2 \times 3 = 6 \]
Each original box has area:\[ A_{\text{gt}} = 4 \times 4 = 16 \]\[ A_{\text{pred}} = 4 \times 4 = 16 \]
The union area is:\[ A_U = 16+16-6=26 \]
Thus:\[ \operatorname{IoU} = \frac{6}{26} \approx 0.231 \]
The predicted box overlaps the ground-truth box, but the localization is not particularly accurate.
Implementing IoU in Python
def intersection_over_union(box_a, box_b):
ax1, ay1, ax2, ay2 = box_a
bx1, by1, bx2, by2 = box_b
intersection_x1 = max(ax1, bx1)
intersection_y1 = max(ay1, by1)
intersection_x2 = min(ax2, bx2)
intersection_y2 = min(ay2, by2)
intersection_width = max(
0.0,
intersection_x2 - intersection_x1
)
intersection_height = max(
0.0,
intersection_y2 - intersection_y1
)
intersection_area = (
intersection_width * intersection_height
)
area_a = max(0.0, ax2 - ax1) * max(0.0, ay2 - ay1)
area_b = max(0.0, bx2 - bx1) * max(0.0, by2 - by1)
union_area = area_a + area_b - intersection_area
if union_area <= 0:
return 0.0
return intersection_area / union_areaThe exact area calculation may differ slightly between coordinate conventions. For continuous coordinates, width is commonly calculated as \(x_{\max}-x_{\min}\). Some integer-pixel implementations use inclusive coordinates and add one. A project should select one convention and apply it consistently.
Using an IoU Threshold
A predicted box can be considered a successful localization when its IoU with a matching ground-truth box exceeds a chosen threshold:\[ \operatorname{IoU} \geq \tau_{\text{IoU}} \]
A historically common threshold is:\[ \tau_{\text{IoU}}=0.5 \]
Under this rule:
- \(\operatorname{IoU} \geq 0.5\): acceptable localization
- \(\operatorname{IoU} < 0.5\): insufficient localization
The value \(0.5\) is a convention, not a theoretically required boundary. A higher threshold, such as \(0.75\), demands tighter localization.
Modern evaluation protocols often report performance across multiple thresholds rather than relying on only one. This distinguishes detectors that merely locate objects approximately from those that produce highly precise boxes.
IoU Does Not Evaluate Classification by Itself
A high IoU does not automatically make a detection correct. The predicted class must also match the ground-truth class.
For example, suppose a model places an accurate box around a car but labels it as a motorcycle. Its localization may have a high IoU, but the complete detection is still incorrect.
A prediction is generally treated as a true positive only when:
- Its predicted class is correct.
- Its IoU with an unmatched ground-truth box reaches the required threshold.
- It is the selected prediction matched to that object.
Additional predictions for the same object may become false positives, even if their boxes also overlap the object. This is one reason duplicate-removal techniques are important.
IoU in Non-Maximum Suppression
IoU can also compare two predicted boxes rather than comparing a prediction with ground truth.
When two high-confidence predictions have a large IoU, they may represent duplicate detections of the same object. Non-maximum suppression keeps the higher-scoring box and removes lower-scoring boxes whose IoU with it exceeds a suppression threshold.
In this context, IoU answers a different question:
Are these two predictions similar enough that they probably refer to the same object?
The same mathematical measure therefore supports both model evaluation and prediction cleanup.
Limitations of IoU
IoU is simple and useful, but it has several limitations.
No Gradient When Boxes Do Not Overlap
If two boxes do not overlap, their IoU is zero regardless of how far apart they are. IoU alone cannot distinguish between two non-overlapping boxes that are nearly touching and two boxes on opposite sides of an image.
This makes raw IoU less informative as a training loss when early predictions are poorly localized.
Sensitivity to Small Objects
A small coordinate error can cause a large decrease in IoU for a small object. The same absolute error may have a much smaller effect on a large object.
No Information About the Type of Misalignment
Two predictions can have the same IoU even though one has the wrong center and the other has the wrong scale or aspect ratio. IoU summarizes overlap using a single number.
Extensions such as Generalized IoU, Distance IoU, and Complete IoU address some of these limitations by incorporating information about enclosure, center distance, or aspect ratio.
Common Mistakes
Dividing by the Combined Area Without Subtracting the Intersection
The union is:\[ A_U=A_A+A_B-A_I \]
Without subtracting the intersection, the shared region is counted twice.
Allowing Negative Intersection Dimensions
Non-overlapping boxes can produce negative raw intersection widths or heights. Clamp both values to zero before calculating the area.
Mixing Bounding-Box Formats
A box may be represented as:\[ (x_{\min},y_{\min},x_{\max},y_{\max}) \]
or as:\[ (x_c,y_c,w,h) \]
These formats are not interchangeable. Center-based boxes must be converted before applying a corner-based IoU calculation.
Treating \(0.5\) as a Universal Rule
An IoU threshold of \(0.5\) is common, but the appropriate threshold depends on the evaluation protocol and application. Safety-critical localization may require substantially tighter overlap.
Key Takeaway
Intersection over Union measures the overlap between two bounding boxes:\[ \boxed{ \operatorname{IoU} = \frac{\text{intersection area}} {\text{union area}} } \]
An IoU of zero means no overlap, while an IoU of one means perfect alignment. It is a central tool for evaluating localization, matching detections with ground truth, and suppressing duplicate predictions. Higher IoU indicates more precise localization, but complete object-detection evaluation must also account for class correctness, confidence scores, missed objects, and duplicate detections.