Putting It All Together: The YOLO Object Detection Algorithm

YOLO combines grid-based prediction, bounding-box regression, anchor boxes, class prediction, confidence filtering, and non-maximum suppression into one object-detection pipeline.

The neural network processes an image once and produces candidate detections across the complete image. These raw predictions are then decoded and filtered to obtain the final objects.

Detection Problem

Suppose a detector recognizes three foreground classes:

Pedestrian
Car
Motorcycle

The background does not need a separate class entry in the class vector. Background is represented by the objectness target:\[ p_c=0 \]

If an object is present, then:\[ p_c=1 \]

and the class vector identifies its category.

This distinction is important:

Objectness determines whether a prediction corresponds to an object; class scores determine which foreground class that object belongs to.

Grid and Anchor Configuration

Suppose the image is divided into an \(S\times S\) grid, and each grid cell uses \(B\) anchor boxes.

For every cell-anchor pair, the model predicts:\[ \mathbf{y} = \begin{bmatrix} p_c & b_x & b_y & b_h & b_w & c_1 & c_2 & \cdots & c_C \end{bmatrix}^{T} \]

where:

\(p_c\) is the objectness value.
\(b_x,b_y\) specify the box center.
\(b_h,b_w\) specify the box dimensions.
\(c_1,\ldots,c_C\) are class targets or class scores.
\(C\) is the number of foreground classes.

Each anchor therefore predicts:\[ 5+C \]

values.

Output-Tensor Shape

The structured output tensor has shape:\[ S\times S\times B\times(5+C) \]

It can also be flattened along its last two dimensions:\[ S\times S\times B(5+C) \]

For a \(3\times3\) grid, two anchors, and three classes:\[ S=3,\qquad B=2,\qquad C=3 \]

Each anchor predicts:\[ 5+3=8 \]

values, producing:\[ 3\times3\times2\times8 \]

or equivalently:\[ 3\times3\times16 \]

With a \(19\times19\) grid and five anchors, the output would be:\[ 19\times19\times5\times8 \]

or:\[ 19\times19\times40 \]

Constructing the Training Targets

Every ground-truth object must be assigned to a specific prediction slot.

The simplified assignment process is:

Find the grid cell containing the object’s center.
Compare the object’s shape with the available anchors.
Select the anchor with the highest shape IoU.
Store the object’s box and class targets in that cell-anchor slot.

Thus, each object is assigned to a pair:\[ (\text{grid cell},\text{anchor}) \]

Assigning an Object to a Grid Cell

Suppose a ground-truth box has center:\[ (x_{\text{center}},y_{\text{center}}) \]

The object is assigned to the grid cell containing that center.

The bounding box may extend beyond the cell or even span many cells. Responsibility is determined only by the center.

If normalized image coordinates are used, a simplified cell index is:\[ j = \left\lfloor Sx_{\text{center}} \right\rfloor \]\[ i = \left\lfloor Sy_{\text{center}} \right\rfloor \]

where \((i,j)\) identifies the responsible grid cell.

Assigning an Object to an Anchor

Let the available anchors be:\[ A_1,A_2,\ldots,A_B \]

The selected anchor is:\[ a^* = \underset{a}{\operatorname{argmax}} \; \operatorname{IoU} \left( B_{\text{gt}},A_a \right) \]

When comparing shapes, the ground-truth box and anchor are conceptually aligned at the same center. This makes the comparison depend on their widths and heights rather than their absolute image positions.

A tall object may be assigned to a tall anchor, while a wide object may be assigned to a wide anchor.

Positive Training Target

Suppose a car is assigned to Anchor 2 in cell \((i,j)\). If the class order is pedestrian, car, and motorcycle, its target is:\[ \mathbf{y}_{i,j,2} = \begin{bmatrix} 1 & b_x & b_y & b_h & b_w & 0 & 1 & 0 \end{bmatrix}^{T} \]

The objectness target is one:\[ p_c=1 \]

The box coordinates describe the car, and the class vector indicates that it is class two.

Empty Training Target

If no object is assigned to a particular cell-anchor pair, its objectness target is:\[ p_c=0 \]

The remaining components are ignored:\[ \mathbf{y}_{i,j,a} = \begin{bmatrix} 0 & * & * & * & * & * & * & * \end{bmatrix}^{T} \]

The asterisks indicate values that should not contribute to the box or classification losses.

The neural network will still output numerical values for those positions, but they do not describe a meaningful object.

Target Masking

A simplified loss for one cell-anchor pair can be written as:\[ \mathcal{L} = \mathcal{L}_{\text{obj}} + \mathbf{1}[p_c=1] \left( \lambda_{\text{box}}\mathcal{L}_{\text{box}} + \lambda_{\text{cls}}\mathcal{L}_{\text{cls}} \right) \]

The indicator ensures that box and class losses are calculated only for positive object assignments.

For negative positions:\[ p_c=0 \]

the detector is trained primarily to predict low objectness.

Practical detectors often balance positive and negative terms because the number of background predictions greatly exceeds the number of objects.

The Neural Network

The model receives an image:\[ X\in\mathbb{R}^{H\times W\times3} \]

A convolutional backbone extracts features, and a detection head produces the prediction tensor:\[ \hat{Y} \in \mathbb{R}^{S\times S\times B\times(5+C)} \]

Conceptually:\[ X \rightarrow \text{backbone} \rightarrow \text{detection head} \rightarrow \hat{Y} \]

All grid cells and anchors are predicted in one shared convolutional evaluation.

This is the main source of YOLO’s efficiency.

Raw Network Predictions

For every cell-anchor pair, the model outputs raw values such as:\[ (\hat{p}_c,t_x,t_y,t_w,t_h,\hat{c}_1,\ldots,\hat{c}_C) \]

These values must be decoded into:

An objectness probability
A bounding box in image coordinates
Class probabilities or scores

The precise decoding formulas vary between YOLO versions.

Decoding the Box Center

In an anchor-based formulation, the center can be decoded relative to the grid cell:\[ b_x = \frac{ \sigma(t_x)+j }{S} \]\[ b_y = \frac{ \sigma(t_y)+i }{S} \]

where:

\((i,j)\) is the grid-cell location.
\(\sigma\) is the sigmoid function.
\(b_x,b_y\) are normalized image coordinates.

The sigmoid constrains the within-cell offsets to values between zero and one.

Decoding Width and Height

Let Anchor \(a\) have dimensions:\[ (p_w^{(a)},p_h^{(a)}) \]

A common historical parameterization is:\[ b_w = p_w^{(a)}e^{t_w} \]\[ b_h = p_h^{(a)}e^{t_h} \]

The exponential ensures positive box dimensions.

Later detector versions may use different bounded transformations, so decoding must match the formulation used during training.

Converting to Corner Coordinates

Non-maximum suppression commonly operates on corner coordinates.

Given center coordinates and dimensions:\[ (b_x,b_y,b_w,b_h) \]

convert them using:\[ x_{\min} = b_x-\frac{b_w}{2} \]\[ y_{\min} = b_y-\frac{b_h}{2} \]\[ x_{\max} = b_x+\frac{b_w}{2} \]\[ y_{\max} = b_y+\frac{b_h}{2} \]

A predicted box may extend beyond the cell that generated it. Depending on the implementation, it may also extend beyond the image boundary and need to be clipped.

Objectness and Class Scores

A model often predicts:\[ P(\text{object}) \]

and conditional class probabilities:\[ P(c\mid\text{object}) \]

A class-specific confidence score can be computed as:\[ s_c = P(\text{object}) P(c\mid\text{object}) \]

This estimates the confidence that:

An object exists in the predicted box.
That object belongs to class \(c\).

For example:\[ P(\text{object})=0.8 \]

and:\[ P(\text{car}\mid\text{object})=0.75 \]

give:\[ s_{\text{car}} = 0.8\times0.75 = 0.6 \]

Filtering detections using only objectness can retain boxes whose class predictions are weak. Class-specific confidence is generally more useful during final post-processing.

How Many Raw Boxes Are Produced?

With an \(S\times S\) grid and \(B\) anchors, the detector produces:\[ S^2B \]

raw boxes.

For a \(3\times3\) grid with two anchors:\[ 3^2\times2=18 \]

boxes are predicted.

For a \(19\times19\) grid with five anchors:\[ 19^2\times5 = 1805 \]

candidate boxes are predicted.

Most of these should receive low confidence because most cell-anchor positions do not correspond to objects.

Confidence Filtering

The first post-processing step removes weak predictions.

For class \(c\), keep a box only if:\[ s_c\geq\tau_{\text{conf}} \]

where \(\tau_{\text{conf}}\) is a confidence threshold.

This greatly reduces the number of candidates passed to later processing.

The threshold involves a tradeoff:

A low threshold improves recall but retains more false positives.
A high threshold improves precision but may remove real objects.

Duplicate Predictions

An object can generate several high-confidence predictions from:

Neighboring grid cells
Different anchors
Different feature-map scales

These boxes may overlap substantially and refer to the same physical object.

Without additional processing, the detector could report one car several times.

Non-maximum suppression removes these duplicate detections.

Intersection over Union

For two boxes \(A\) and \(B\), Intersection over Union is:\[ \operatorname{IoU}(A,B) = \frac{ \operatorname{Area}(A\cap B) }{ \operatorname{Area}(A\cup B) } \]

A high IoU means that the boxes overlap strongly and may represent duplicate predictions.

Class-Specific Non-Maximum Suppression

For each class independently:

Collect predictions for that class.
Remove predictions below the confidence threshold.
Select the remaining box with the highest score.
Add it to the final detections.
Remove remaining boxes whose IoU with the selected box exceeds the suppression threshold.
Repeat until no boxes remain.

In pseudocode:

for each class:
    boxes = predictions for that class
    discard boxes below the confidence threshold

    while boxes remain:
        best = highest-scoring box
        keep best
        discard boxes with high IoU relative to best

for each class:
    boxes = predictions for that class
    discard boxes below the confidence threshold

    while boxes remain:
        best = highest-scoring box
        keep best
        discard boxes with high IoU relative to best

If the detector recognizes pedestrians, cars, and motorcycles, class-specific suppression is performed separately for each of the three classes.

This avoids incorrectly suppressing a pedestrian box merely because it overlaps a car box.

Class-Agnostic Suppression

Some implementations use class-agnostic non-maximum suppression, in which boxes compete regardless of their predicted class.

This can reduce duplicate cross-class predictions, but it can also suppress legitimate overlapping objects from different categories.

Class-specific suppression	Class-agnostic suppression
Processes each class independently	Processes all classes together
Preserves overlapping objects of different classes	Can remove cross-class duplicates
May retain duplicate labels for one object	More aggressive

The correct choice depends on the detector and application.

Complete Inference Pipeline

The complete simplified YOLO inference procedure is:

input image
    ↓
convolutional network
    ↓
raw grid-and-anchor predictions
    ↓
decode bounding boxes
    ↓
compute class-specific confidence scores
    ↓
discard low-confidence predictions
    ↓
apply non-maximum suppression
    ↓
return final boxes, classes, and scores

input image
    ↓
convolutional network
    ↓
raw grid-and-anchor predictions
    ↓
decode bounding boxes
    ↓
compute class-specific confidence scores
    ↓
discard low-confidence predictions
    ↓
apply non-maximum suppression
    ↓
return final boxes, classes, and scores

More formally:\[ X \rightarrow \hat{Y} \rightarrow \mathcal{B}_{\text{decoded}} \rightarrow \mathcal{B}_{\text{filtered}} \rightarrow \mathcal{B}_{\text{final}} \]

Complete Training Pipeline

Training can be summarized as:

for every labeled image:
    initialize the target tensor
    for every ground-truth object:
        find the cell containing its center
        select the best-matching anchor
        encode the box coordinates
        set objectness to one
        store the class target
    train the network against the completed tensor

for every labeled image:
    initialize the target tensor
    for every ground-truth object:
        find the cell containing its center
        select the best-matching anchor
        encode the box coordinates
        set objectness to one
        store the class target
    train the network against the completed tensor

The loss combines:

Objectness loss
Bounding-box regression loss
Classification loss

A general expression is:\[ \mathcal{L}_{\text{total}} = \lambda_{\text{obj}}\mathcal{L}_{\text{obj}} + \lambda_{\text{noobj}}\mathcal{L}_{\text{noobj}} + \lambda_{\text{box}}\mathcal{L}_{\text{box}} + \lambda_{\text{cls}}\mathcal{L}_{\text{cls}} \]

Modern implementations differ significantly in their exact losses, assignments, and decoding rules.

Important Edge Cases

Multiple Objects in One Cell

Multiple anchors allow a cell to represent several objects if the objects are assigned to different anchor slots.

Conflicts remain possible when:

There are more objects than anchors.
Several objects prefer the same anchor.
Objects are highly crowded.

Finer grids, multiple feature scales, and more flexible assignment strategies help reduce these problems.

Multiple Predictions for One Object

Several anchors or cells may detect the same object. Confidence filtering alone does not remove these duplicates, so suppression or another selection mechanism is needed.

Objects of Very Different Sizes

A single prediction grid may struggle to detect both tiny and very large objects. Modern detectors commonly predict at several resolutions.

Class Imbalance

Most predictions correspond to background. Loss weighting, negative sampling, focal-style losses, and improved assignment rules can prevent background examples from dominating training.

Predictions at Multiple Scales

Modern YOLO-style detectors commonly make predictions using several feature maps.

For example:

A high-resolution feature map detects small objects.
A medium-resolution feature map detects medium objects.
A low-resolution feature map detects large objects.

If the scales are indexed by \(m\), the detector may produce:\[ \hat{Y}^{(m)} \in \mathbb{R}^{S_m\times S_m\times B_m\times(5+C)} \]

All decoded predictions are combined before confidence filtering and suppression.

This multiscale design improves detection across a wide range of object sizes.

Modern Variations

The simplified anchor-based formulation explains the major components clearly, but current YOLO-style systems may differ in several ways.

They may use:

Anchor-free prediction heads
Decoupled classification and regression branches
Multiple feature-map scales
Distribution-based box regression
IoU-based localization losses
Dynamic target assignment
Objectness-free classification formulations
Alternative suppression methods
End-to-end duplicate removal

Therefore, “YOLO” refers to a family of related one-stage detectors rather than one fixed architecture.

Strengths of YOLO-Style Detection

Single Shared Network Evaluation

The entire image is processed through one convolutional system.

Dense Prediction

Objects are predicted across many spatial locations and scales.

Direct Box Regression

Bounding boxes are adjusted continuously rather than selected only from fixed windows.

Joint Learning

Objectness, classification, and localization are optimized together.

Efficient Inference

Shared convolutional computation makes the approach suitable for latency-sensitive applications.

Common Implementation Mistakes

Adding an Explicit Background Class Unnecessarily

In the simplified formulation, background is represented by:\[ p_c=0 \]

The class vector normally covers only foreground categories.

Calculating Box and Class Losses for Empty Anchors

When no object is assigned, bounding-box and foreground-class targets must generally be masked out.

Filtering Only by Objectness

Final confidence should usually account for both objectness and class prediction:\[ s_c = P(\text{object})P(c\mid\text{object}) \]

Applying Suppression Before Decoding Boxes

IoU must be calculated using boxes expressed in a common coordinate system.

Mixing Coordinate Conventions

Box values might be represented relative to:

A grid cell
An anchor
The complete image
The resized model input

Encoding and decoding must use matching conventions.

Running Suppression Without Considering Classes

Class-agnostic suppression may remove legitimate overlapping objects from different categories.

Key Takeaway

A simplified anchor-based YOLO detector outputs:\[ \hat{Y} \in \mathbb{R}^{S\times S\times B\times(5+C)} \]

Each grid-cell-and-anchor position predicts:\[ (p_c,b_x,b_y,b_h,b_w,c_1,\ldots,c_C) \]

During training, every object is assigned to the cell containing its center and an appropriate anchor. During inference, the network’s raw outputs are decoded into boxes, converted into class-specific confidence scores, filtered, and processed with non-maximum suppression.

The complete pipeline is:\[ \boxed{ \text{image} \rightarrow \text{dense predictions} \rightarrow \text{decoded boxes} \rightarrow \text{confidence filtering} \rightarrow \text{duplicate suppression} } \]

By combining these components in one convolutional system, YOLO provides fast, flexible, and effective object detection.