Anchor Boxes in Object Detection
A basic grid-based detector assigns each object to the grid cell containing the object’s center. If every cell predicts only one bounding box, however, it cannot represent multiple objects whose centers fall inside the same cell.
Anchor boxes address this limitation by allowing every grid cell to predict several bounding boxes. Each anchor represents a different reference shape, enabling the detector to handle multiple objects and specialize in objects with different aspect ratios.
The One-Object-per-Cell Problem
Suppose an image is divided into a \(3 \times 3\) grid. A pedestrian and a car appear close together, and both object centers fall inside the same grid cell.
Without anchor boxes, that cell predicts only one vector:\[ \mathbf{y} = \begin{bmatrix} p_c & b_x & b_y & b_h & b_w & c_1 & c_2 & c_3 \end{bmatrix}^{T} \]
This vector can describe only one object:
- One objectness value
- One bounding box
- One class prediction
The cell must therefore choose between the pedestrian and the car. It has no separate output slot for the second object.
Introducing Anchor Boxes
An anchor box is a predefined reference box with a particular width and height.
For example, a detector might use two anchors:
- A tall, narrow anchor suitable for pedestrians
- A wide anchor suitable for cars
Each grid cell produces a separate prediction for every anchor.
If a cell has two anchors, its output becomes:\[ \mathbf{y} = \begin{bmatrix} \mathbf{y}^{(1)} \\ \mathbf{y}^{(2)} \end{bmatrix} \]
where:\[ \mathbf{y}^{(1)} = \begin{bmatrix} p_c^{(1)} & b_x^{(1)} & b_y^{(1)} & b_h^{(1)} & b_w^{(1)} & c_1^{(1)} & c_2^{(1)} & c_3^{(1)} \end{bmatrix}^{T} \]
and:\[ \mathbf{y}^{(2)} = \begin{bmatrix} p_c^{(2)} & b_x^{(2)} & b_y^{(2)} & b_h^{(2)} & b_w^{(2)} & c_1^{(2)} & c_2^{(2)} & c_3^{(2)} \end{bmatrix}^{T} \]
The cell can now represent two objects if they are assigned to different anchors.
Assigning an Object to a Prediction Slot
Anchor-based target assignment usually has two stages.
1. Assign the Object to a Grid Cell
The object is assigned to the cell containing its center:\[ (x_{\text{center}},y_{\text{center}}) \]
The complete bounding box may extend across many cells. Only the center determines cell responsibility.
2. Assign the Object to an Anchor
Within the responsible cell, compare the object’s dimensions with the available anchor shapes.
The object is assigned to the anchor with the greatest Intersection over Union:\[ a^* = \underset{a}{\operatorname{argmax}} \; \operatorname{IoU} \left( B_{\text{object}}, A_a \right) \]
Here:
- \(A_a\) is anchor \(a\).
- \(B_{\text{object}}\) is the ground-truth box.
- \(a^*\) is the selected anchor.
For shape comparison, the object box and anchor are conceptually placed at the same center. The IoU comparison therefore measures compatibility between their widths and heights rather than their positions in the image.
An object is assigned to a grid-cell-and-anchor pair, not merely to a grid cell.
Example: A Pedestrian and a Car
Suppose a cell contains the centers of:
- A tall pedestrian
- A wide car
Assume the detector has two anchors:
- Anchor 1: tall and narrow
- Anchor 2: short and wide
The pedestrian has the highest shape IoU with Anchor 1, while the car has the highest shape IoU with Anchor 2.
If the class order is pedestrian, car, and motorcycle, the target for Anchor 1 might be:\[ \mathbf{y}^{(1)} = \begin{bmatrix} 1 & b_x^{(1)} & b_y^{(1)} & b_h^{(1)} & b_w^{(1)} & 1 & 0 & 0 \end{bmatrix}^{T} \]
The target for Anchor 2 might be:\[ \mathbf{y}^{(2)} = \begin{bmatrix} 1 & b_x^{(2)} & b_y^{(2)} & b_h^{(2)} & b_w^{(2)} & 0 & 1 & 0 \end{bmatrix}^{T} \]
The first anchor represents the pedestrian, and the second represents the car.
Output-Tensor Dimensions
Suppose the detector uses:
- An \(S \times S\) grid
- \(B\) anchor boxes per cell
- \(C\) object classes
Each anchor predicts:\[ 5+C \]
values:\[ (p_c,b_x,b_y,b_h,b_w,c_1,\ldots,c_C) \]
The structured output tensor therefore has shape:\[ S \times S \times B \times (5+C) \]
It may also be flattened along the last two dimensions:\[ S \times S \times B(5+C) \]
For a \(3 \times 3\) grid, two anchors, and three classes:\[ S=3,\qquad B=2,\qquad C=3 \]
Each anchor predicts:\[ 5+3=8 \]
values, so the output can be represented as:\[ 3 \times 3 \times 2 \times 8 \]
or equivalently:\[ 3 \times 3 \times 16 \]
The four-dimensional representation is usually easier to interpret because it preserves a separate anchor dimension.
A Cell Containing Only One Object
Suppose the cell contains a car but no pedestrian, and the car is assigned to Anchor 2.
The target for Anchor 1 becomes:\[ p_c^{(1)}=0 \]
Its remaining bounding-box and class values are ignored:\[ \mathbf{y}^{(1)} = \begin{bmatrix} 0 & * & * & * & * & * & * & * \end{bmatrix}^{T} \]
The target for Anchor 2 contains the car:\[ \mathbf{y}^{(2)} = \begin{bmatrix} 1 & b_x & b_y & b_h & b_w & 0 & 1 & 0 \end{bmatrix}^{T} \]
The asterisks are not necessarily literal values stored in the target. They indicate components that should be masked out when computing localization and classification losses.
Why Anchors Improve Specialization
Handling several objects in one cell is one benefit of anchor boxes, but specialization is often even more important.
Without anchors, one prediction head must learn to represent every possible object shape. With anchors, different prediction slots begin from different geometric assumptions.
For example:
- A tall anchor can specialize in pedestrians.
- A wide anchor can specialize in cars.
- A small square anchor can specialize in traffic signs.
- A large rectangular anchor can specialize in buses.
The anchor provides a reference geometry, while the network learns adjustments that transform it into the final bounding box.
Predicting Offsets from Anchors
An anchor is not the final prediction. The model predicts offsets relative to it.
Let an anchor have dimensions:\[ (p_w,p_h) \]
The network predicts raw values such as:\[ (t_x,t_y,t_w,t_h) \]
A commonly used decoding approach is:\[ b_x=\sigma(t_x)+c_x \]\[ b_y=\sigma(t_y)+c_y \]\[ b_w=p_w e^{t_w} \]\[ b_h=p_h e^{t_h} \]
where:
- \((c_x,c_y)\) identifies the grid-cell location.
- \(\sigma\) is the sigmoid function.
- \((p_w,p_h)\) are the anchor dimensions.
- \((b_x,b_y,b_w,b_h)\) define the predicted box.
The precise equations vary between detector versions. Some modern implementations use different bounded transformations for width and height, but the central idea remains unchanged: the model predicts modifications to a reference box.
Target Assignment During Training
A simplified anchor-assignment procedure is:
for each ground-truth object:
locate the grid cell containing the object's center
compare the object's shape with every anchor
select the anchor with the highest shape IoU
assign the object to that cell-anchor pair
store its box and class targets in that prediction slotThe selected anchor receives a positive objectness target:\[ p_c=1 \]
Other anchors generally receive background targets or may be ignored, depending on their overlap with the object and the detector’s target-assignment rules.
Many practical implementations use an ignore region. An unselected anchor with substantial IoU against a ground-truth object may be excluded from the negative objectness loss instead of being treated as an ordinary background prediction.
This prevents the model from being heavily penalized for producing a plausible box that was not selected as the primary match.
Collision Cases
Anchor boxes improve representation capacity, but they do not eliminate every conflict.
More Objects Than Anchors
If three object centers fall in one cell but the cell has only two anchors, the simplified representation cannot assign a separate slot to every object.
Some object targets must be ignored or resolved using an implementation-specific rule.
Two Objects Prefer the Same Anchor
Two objects in the same cell may both have their highest IoU with the same anchor. If the training format allows only one object per cell-anchor pair, they compete for the same slot.
Possible strategies include:
- Assigning one object to its second-best available anchor
- Keeping the object with the stronger match
- Applying a predefined priority rule
- Using more anchors
- Using finer or multiscale prediction grids
- Adopting a more flexible target-assignment method
Small or Crowded Objects
Even with several anchors, coarse grids can struggle when many small objects are densely packed. Predictions at multiple feature-map resolutions help address this problem.
Choosing Anchor Shapes Manually
Anchor shapes can be selected manually by examining the expected objects.
A useful collection should include different:
- Scales
- Aspect ratios
- Widths
- Heights
For example, a detector might include:
- Tall and narrow anchors
- Square anchors
- Wide and short anchors
- Small anchors
- Large anchors
Manual selection can work when object shapes are predictable, but it may not match the actual dataset distribution particularly well.
Choosing Anchors with Clustering
Anchor boxes can also be derived from the dimensions of the ground-truth boxes.
A common approach is to cluster the width-height pairs:\[ (w_i,h_i) \]
A standard Euclidean distance is not always ideal because large boxes can dominate the clustering. An IoU-based distance is more closely aligned with shape matching:\[ d(B,C) = 1-\operatorname{IoU}(B,C) \]
The procedure is:
- Collect the normalized widths and heights of all ground-truth boxes.
- Choose the desired number of anchors \(K\).
- Cluster the box shapes using an IoU-based distance.
- Use the cluster centers as anchor dimensions.
The resulting anchors represent common object shapes in the data.
How Many Anchors Should Be Used?
More anchors increase the number of shapes available to each grid cell, but they also increase:
- Output-tensor size
- Memory consumption
- Prediction count
- Training complexity
- Post-processing cost
Too few anchors may provide poor coverage. Too many anchors may create many redundant predictions and worsen class imbalance.
The appropriate number depends on:
- Diversity of object shapes
- Number of prediction scales
- Density of objects
- Computational budget
- Target-assignment strategy
Anchors at Multiple Feature-Map Scales
Modern anchor-based detectors often make predictions at several spatial resolutions.
A high-resolution feature map is useful for small objects, while a low-resolution feature map has larger receptive fields and is better suited to large objects.
Different anchor sizes can be assigned to different feature maps:
| Feature map | Typical responsibility |
|---|---|
| High spatial resolution | Small objects |
| Medium spatial resolution | Medium objects |
| Low spatial resolution | Large objects |
In this design, an object is matched not only to a grid cell and anchor shape but also to an appropriate prediction scale.
Anchors and Non-Maximum Suppression
Several anchors may produce high-confidence boxes for the same object. As a result, anchor-based detectors commonly generate duplicate predictions.
After decoding the boxes:
- Remove predictions with low confidence.
- Separate predictions by class when using class-specific suppression.
- Select the highest-scoring remaining box.
- Suppress overlapping boxes whose IoU exceeds the chosen threshold.
- Repeat until no unprocessed boxes remain.
Anchor boxes increase the number of useful candidates, while non-maximum suppression reduces those candidates to the final detections.
Anchor-Based and Anchor-Free Detectors
Anchor boxes remain an important object-detection concept, but they are not required by every modern detector.
Anchor-free detectors predict objects without predefined box shapes. They may instead predict:
- Object centers
- Distances from a location to the four box boundaries
- Keypoints or corners
- Dense box coordinates at candidate locations
Anchor-free approaches reduce anchor-design decisions but introduce their own target-assignment and regression choices.
The two approaches can be summarized as follows:
| Anchor-based detector | Anchor-free detector |
|---|---|
| Starts from predefined box shapes | Predicts boxes without predefined shapes |
| Regresses offsets from anchors | Regresses centers, sides, or complete boxes |
| Requires anchor design or clustering | Requires another positive-location assignment rule |
| Often produces many overlapping candidates | Can also produce duplicate predictions |
| Naturally supports shape specialization | Avoids explicit anchor hyperparameters |
Key Takeaway
Anchor boxes allow every grid cell to make several object predictions:\[ Y \in \mathbb{R}^{S\times S\times B\times(5+C)} \]
Each ground-truth object is assigned to:
- The grid cell containing its center
- The anchor whose shape best matches its bounding box
This allows different prediction slots to specialize in different object geometries and can represent multiple objects whose centers fall in the same cell, provided they are assigned to different anchors. Anchor boxes improve detection flexibility, but they also require careful shape selection, target assignment, loss masking, and duplicate removal.