YOLO: Grid-Based Object Detection with Precise Bounding Boxes
Sliding-window detection evaluates a classifier at a fixed collection of positions, scales, and aspect ratios. Even with an efficient convolutional implementation, its predicted boxes remain constrained by the windows selected in advance.
The YOLO approach addresses this limitation by training a single convolutional network to predict bounding boxes and object classes directly from the complete image.
YOLO stands for You Only Look Once. The name emphasizes that the model processes the image with one shared network evaluation instead of running an independent classifier on every candidate crop.
Limitations of Sliding-Window Detection
A sliding-window detector evaluates predefined windows across an image. This creates two important problems.
First, the best available window may not align precisely with the object. A car might have a wide rectangular shape, while the detector evaluates only square windows.
Second, accurate localization requires testing many positions, sizes, and aspect ratios. This can become computationally expensive even when the calculations are shared convolutionally.
YOLO replaces these discrete window choices with direct bounding-box regression. The network predicts continuous box coordinates, allowing it to produce boxes that are not restricted to a predetermined collection of windows.
Dividing the Image into a Grid
Consider an input image divided into an \(S \times S\) grid. A small \(3 \times 3\) grid is useful for illustration, although practical systems generally use finer prediction grids or multiple feature-map resolutions.
Each object is assigned to the grid cell containing the object’s center.
An object belongs to the cell containing its center, even when its bounding box extends across several other cells.
This assignment rule ensures that one object is associated with one responsible grid cell.
For example, if an image contains two cars, each car’s center determines which cell is responsible for predicting it. A third cell may contain parts of both cars but is not responsible for either one if neither center lies inside it.
The Target Vector for Each Cell
In a simplified detector with three object classes, each grid cell predicts:\[ \mathbf{y} = \begin{bmatrix} p_c & b_x & b_y & b_h & b_w & c_1 & c_2 & c_3 \end{bmatrix}^{T} \]
The components have the following meanings:
| Component | Meaning |
|---|---|
| \(p_c\) | Whether an object is assigned to the cell |
| \(b_x,b_y\) | Coordinates of the object’s center |
| \(b_h,b_w\) | Height and width of its bounding box |
| \(c_1,c_2,c_3\) | Object-class targets |
Suppose the three classes are:
- Pedestrian
- Car
- Motorcycle
For a cell responsible for a car, the class target is:\[ (c_1,c_2,c_3)=(0,1,0) \]
The complete target becomes:\[ \mathbf{y} = \begin{bmatrix} 1 & b_x & b_y & b_h & b_w & 0 & 1 & 0 \end{bmatrix}^{T} \]
Cells Without Objects
If no object center is assigned to a cell, then:\[ p_c=0 \]
The remaining components do not represent a real object and should not contribute to the bounding-box or classification losses:\[ \mathbf{y} = \begin{bmatrix} 0 & * & * & * & * & * & * & * \end{bmatrix}^{T} \]
Here, each asterisk represents a value that is ignored during training.
It is important to implement this with a loss mask. The model should still be trained to predict that the cell contains no object, but it should not be penalized for its box coordinates or class probabilities when no object is assigned to that cell.
Constructing the Output Tensor
If the image is divided into an \(S \times S\) grid and every cell produces an eight-component vector, the output tensor has shape:\[ S \times S \times 8 \]
For the illustrative \(3 \times 3\) grid:\[ 3 \times 3 \times 8 \]
For a \(19 \times 19\) grid:\[ 19 \times 19 \times 8 \]
Each spatial position in this tensor corresponds to one image-grid cell, while its final dimension stores the objectness, bounding-box, and class predictions.
More generally, with \(C\) object classes and one predicted box per cell, the output depth is:\[ 5+C \]
Therefore, the complete output shape is:\[ S \times S \times (5+C) \]
This simplified representation assumes one box per cell. Detectors using multiple anchor boxes require additional predictions at every grid position.
A Single Convolutional Prediction
The detector receives an image such as:\[ X \in \mathbb{R}^{H \times W \times 3} \]
and maps it directly to a structured prediction tensor:\[ \hat{Y} \in \mathbb{R}^{S \times S \times (5+C)} \]
Conceptually:\[ X \longrightarrow \text{convolutional network} \longrightarrow \hat{Y} \]
This is not equivalent to running a separate neural network once for every grid cell. The convolutional layers compute shared features for the complete image, and a prediction head produces all grid-cell outputs together.
This shared computation is a major reason one-stage detectors can operate efficiently.
Predicting Precise Bounding Boxes
A sliding-window detector chooses among boxes specified before inference. YOLO instead predicts real-valued box coordinates.
For a cell responsible for an object, the model estimates:\[ (b_x,b_y,b_h,b_w) \]
Because these values are learned continuously, the predicted box can:
- Be positioned more precisely than a fixed window
- Have a non-square aspect ratio
- Extend beyond the boundaries of its responsible cell
- Represent objects of different sizes
The grid determines responsibility for the object, not the complete extent of its bounding box.
Encoding the Box Center
Suppose an object is assigned to grid cell \((i,j)\). Within that cell, define the upper-left corner as \((0,0)\) and the lower-right corner as \((1,1)\).
The object center is encoded relative to the responsible cell:\[ 0 \leq b_x \leq 1 \]\[ 0 \leq b_y \leq 1 \]
For example:\[ b_x=0.4,\qquad b_y=0.3 \]
means that the center is 40 percent across the cell and 30 percent down from its top-left corner.
If grid coordinates are zero-indexed, the corresponding normalized image coordinates can be written as:\[ x_{\text{center}} = \frac{j+b_x}{S} \]\[ y_{\text{center}} = \frac{i+b_y}{S} \]
The cell index gives the coarse position, while \(b_x\) and \(b_y\) provide the within-cell offset.
Modern implementations often apply a sigmoid or a related bounded transformation to the raw center predictions.
Encoding Box Width and Height
Unlike the center coordinates, the box width and height are not necessarily constrained to the cell.
An object assigned to one cell may span several cells, so it is possible for its width or height—when measured relative to the cell—to exceed one:\[ b_w > 1 \]
or:\[ b_h > 1 \]
The exact meaning of \(b_w\) and \(b_h\) depends on the chosen parameterization. They may be expressed relative to:
- The complete image
- The responsible grid cell
- A predefined anchor box
The simplified cell-relative formulation is useful for understanding the basic idea, but practical YOLO variants use carefully designed transformations to stabilize training and ensure positive dimensions.
For example, a width and height can be parameterized relative to an anchor:\[ b_w=p_w e^{t_w} \]\[ b_h=p_h e^{t_h} \]
where \(p_w\) and \(p_h\) are the anchor dimensions and \(t_w,t_h\) are unconstrained network outputs.
Different YOLO versions use different decoding formulas, so the implementation’s exact convention must be followed.
Example Target Encoding
Suppose a car is assigned to a particular cell and has the following cell-relative values:\[ b_x=0.4,\qquad b_y=0.3,\qquad b_h=0.5,\qquad b_w=0.9 \]
For the class order pedestrian, car, and motorcycle, the target vector is:\[ \mathbf{y} = \begin{bmatrix} 1 & 0.4 & 0.3 & 0.5 & 0.9 & 0 & 1 & 0 \end{bmatrix}^{T} \]
Notice that \(b_h\) represents height and \(b_w\) represents width. Maintaining this ordering consistently is essential when generating labels, calculating losses, and decoding predictions.
Training the Detector
Training requires a labeled target tensor for every image. The convolutional network predicts:\[ \hat{\mathbf{y}} = \begin{bmatrix} \hat{p}_c & \hat{b}_x & \hat{b}_y & \hat{b}_h & \hat{b}_w & \hat{c}_1 & \cdots & \hat{c}_C \end{bmatrix}^{T} \]
The total loss generally combines several components:\[ \mathcal{L} = \lambda_{\text{obj}}\mathcal{L}_{\text{obj}} + \lambda_{\text{box}}\mathcal{L}_{\text{box}} + \lambda_{\text{cls}}\mathcal{L}_{\text{cls}} \]
where:
- \(\mathcal{L}_{\text{obj}}\) measures objectness accuracy.
- \(\mathcal{L}_{\text{box}}\) measures localization accuracy.
- \(\mathcal{L}_{\text{cls}}\) measures class-prediction accuracy.
For a cell without an assigned object, only the relevant background or objectness term should normally be active. The box and object-class terms are masked out.
Modern detectors may use IoU-based localization losses rather than simple squared error:\[ \mathcal{L}_{\text{box}} = 1-\operatorname{IoU} \]
More advanced variants include Generalized IoU, Distance IoU, and Complete IoU losses.
Inference
At inference time, the model processes the image once and produces predictions across all grid locations.
For each predicted box, the output contains:
- An objectness score
- Bounding-box coordinates
- Class probabilities or class scores
A class-specific detection score can be formed as:\[ s_k = P(\text{object}) P(\text{class}=k\mid\text{object}) \]
Low-confidence boxes are removed. The remaining boxes are decoded into image coordinates and duplicate predictions are typically reduced with non-maximum suppression or a related post-processing method.
Why the Method Is Efficient
YOLO avoids independently evaluating hundreds or thousands of cropped image regions.
Instead, it:
- Processes the complete image with one convolutional network
- Shares feature computation across all grid positions
- Predicts classes and bounding boxes simultaneously
- Produces all detections through a dense output tensor
This is why YOLO is described as a one-stage detector. Region proposal and classification are not implemented as two completely separate pipelines.
The Multiple-Object Problem
In the simplified formulation, one cell predicts one object. A problem occurs when the centers of two objects fall within the same grid cell.
Only one target vector is available, so the cell cannot represent both objects independently.
Using a finer grid reduces the probability of this collision but does not eliminate it. A more effective solution is to predict multiple candidate boxes at each cell using anchor boxes.
If every cell uses \(B\) anchors, the output shape becomes:\[ S \times S \times B \times (5+C) \]
Each anchor can specialize in objects with different dimensions or aspect ratios, allowing one spatial location to produce multiple detections.
Simplified YOLO Versus Modern YOLO Systems
The grid-based explanation captures the central idea of early YOLO-style detection, but modern implementations are more sophisticated.
They commonly include:
- Multiple bounding-box predictions per location
- Anchor-based or anchor-free box parameterizations
- Predictions at several feature-map resolutions
- Specialized classification, objectness, and regression heads
- IoU-based box losses
- Feature pyramids for detecting objects at different scales
- Improved post-processing and confidence calibration
The essential principle remains the same:
A single convolutional system predicts object classes and bounding boxes jointly over the complete image.
Advantages of Grid-Based Detection
Continuous Box Coordinates
The detector is not limited to a fixed collection of windows. It can predict boxes with continuous positions, sizes, and aspect ratios.
Shared Computation
All grid positions use a common set of convolutional features rather than running independent classifiers.
End-to-End Training
Classification, objectness estimation, and localization can be optimized together.
Fast Inference
The architecture is well suited to applications that require low-latency or real-time detection.
Global Image Context
Because the network processes the complete image, its predictions can use contextual information beyond an individual cropped region.
Important Limitations
Grid-based detection also introduces challenges:
- Small objects may be difficult to represent on coarse feature maps.
- Several nearby objects may compete for the same cell or anchor.
- Class imbalance arises because most locations contain no object.
- Duplicate predictions require post-processing.
- Accurate box decoding depends on a consistent parameterization.
- Objectness, classification, and localization losses must be balanced carefully.
These limitations motivated later improvements such as anchor boxes, multiscale prediction, feature pyramids, focal-style losses, and anchor-free detection heads.
Key Takeaway
YOLO replaces repeated sliding-window classification with one convolutional model that predicts an entire grid of objectness scores, classes, and bounding boxes:\[ X \longrightarrow \hat{Y} \in \mathbb{R}^{S\times S\times(5+C)} \]
Each object is assigned to the grid cell containing its center, while its bounding box may extend beyond that cell. Direct regression produces more flexible and precise boxes than a fixed set of sliding windows, and shared convolutional computation makes the detector fast enough for real-time applications.