Image Classification with Object Localization
Image classification predicts what appears in an image. Object localization predicts both what the object is and where it appears.
This creates an intermediate problem between ordinary image classification and full object detection.
Classification with localization assumes that an image contains at most one primary object and asks the model to predict its class and bounding box.
The same ideas later extend to systems that detect multiple objects at different positions.
Classification, Localization, and Detection
These three tasks are related but distinct.
Image classification
The model predicts one class for the complete image:\[ X\rightarrow\hat{C} \]
For example:\[ X\rightarrow\text{car} \]
Classification with localization
The model predicts one object class and one bounding box:\[ X \rightarrow \left( \hat{C}, \hat{B} \right) \]
For example:\[ X \rightarrow \left( \text{car}, \text{bounding box} \right) \]
Object detection
The model predicts an unknown number of objects:\[ X \rightarrow \left\{ (\hat{C}_1,\hat{B}_1), (\hat{C}_2,\hat{B}_2), \ldots \right\} \]
The image may contain:
- Several cars
- Multiple pedestrians
- Motorcycles
- Objects from several categories
| Task | Number of objects | Output |
|---|---|---|
| Classification | Usually one dominant object | Class |
| Classification with localization | At most one target object | Class and box |
| Detection | Zero, one, or many objects | Multiple classes and boxes |
Example Classes
Consider an autonomous-driving dataset with three foreground classes:
- Pedestrian
- Car
- Motorcycle
The image may also contain none of these objects. Instead of treating background as an ordinary fourth class, the localization formulation uses a separate object-presence value.
Define:\[ p_c= \begin{cases} 1 & \text{if a target object is present}\\ 0 & \text{if no target object is present} \end{cases} \]
The subscript \(c\) is traditionally interpreted as indicating object presence or confidence.
Bounding-Box Representation
A bounding box can be represented using four numbers:\[ b_x,\quad b_y,\quad b_h,\quad b_w \]
where:
- \(b_x\) is the horizontal coordinate of the box center.
- \(b_y\) is the vertical coordinate of the box center.
- \(b_h\) is the box height.
- \(b_w\) is the box width.
Use normalized image coordinates:
- Upper-left corner: \((0,0)\)
- Lower-right corner: \((1,1)\)
Under this convention:\[ 0\le b_x\le1 \]\[ 0\le b_y\le1 \]
and normally:\[ 0<b_h\le1,\qquad 0<b_w\le1 \]
Example Bounding Box
Suppose a car is centered approximately halfway across the image and 70% of the way down.
A possible box is:\[ b_x=0.5 \]\[ b_y=0.7 \]\[ b_h=0.3 \]\[ b_w=0.4 \]
The corresponding box boundaries are:\[ x_{\min}=b_x-\frac{b_w}{2} \]\[ x_{\max}=b_x+\frac{b_w}{2} \]\[ y_{\min}=b_y-\frac{b_h}{2} \]\[ y_{\max}=b_y+\frac{b_h}{2} \]
For this example:\[ x_{\min}=0.5-\frac{0.4}{2}=0.3 \]\[ x_{\max}=0.5+\frac{0.4}{2}=0.7 \]\[ y_{\min}=0.7-\frac{0.3}{2}=0.55 \]\[ y_{\max}=0.7+\frac{0.3}{2}=0.85 \]
Converting Pixel Boxes to Normalized Coordinates
Suppose an image has width \(W\) and height \(H\), and its annotated box is:\[ (x_{\min},y_{\min},x_{\max},y_{\max}) \]
in pixels.
The normalized center coordinates are:\[ b_x = \frac{x_{\min}+x_{\max}}{2W} \]\[ b_y = \frac{y_{\min}+y_{\max}}{2H} \]
The normalized dimensions are:\[ b_w = \frac{x_{\max}-x_{\min}}{W} \]\[ b_h = \frac{y_{\max}-y_{\min}}{H} \]
Normalization makes the target representation independent of the original image dimensions.
Network Architecture
A convolutional backbone extracts image features:\[ X \xrightarrow{\text{ConvNet}} h \]
The final prediction head produces:
- One object-presence value
- Four bounding-box values
- Three class values
The output vector is:\[ \hat{Y} = \left[ \hat{p}_c, \hat{b}_x, \hat{b}_y, \hat{b}_h, \hat{b}_w, \hat{c}_1, \hat{c}_2, \hat{c}_3 \right]^T \]
This vector contains eight components.
Constructing the Target Vector
The ground-truth target is:\[ Y = \left[ p_c, b_x, b_y, b_h, b_w, c_1, c_2, c_3 \right]^T \]
The class components are one-hot encoded when an object is present.
For a car:\[ (c_1,c_2,c_3)=(0,1,0) \]
For a pedestrian:\[ (c_1,c_2,c_3)=(1,0,0) \]
For a motorcycle:\[ (c_1,c_2,c_3)=(0,0,1) \]
Only one foreground class is permitted because this formulation assumes at most one target object.
Positive Example
Suppose an image contains a car with box:\[ (b_x,b_y,b_h,b_w) = (0.5,0.7,0.3,0.4) \]
The target is:\[ Y = \left[ 1, 0.5, 0.7, 0.3, 0.4, 0, 1, 0 \right]^T \]
This tells the model:
- An object is present.
- Its center is at \((0.5,0.7)\).
- Its height is 0.3.
- Its width is 0.4.
- Its class is car.
Background Example
If none of the target objects is present:\[ p_c=0 \]
The remaining values have no meaningful target:\[ Y = \left[ 0, ?, ?, ?, ?, ?, ?, ? \right]^T \]
The question marks mean “ignore these values when computing the loss.”
They should not be treated as literal numerical labels.
Why Background Coordinates Are Ignored
If no object exists, there is no correct bounding box.
Penalizing the model for its box prediction on a background image would force it toward an arbitrary target.
Likewise, there is no meaningful foreground class when:\[ p_c=0 \]
The loss must therefore mask the localization and class terms for background images.
Simplified Squared-Error Loss
A simplified explanation can use squared error.
If an object is present:\[ p_c=1 \]
then:\[ \mathcal{L} = \sum_{j=1}^{8} (\hat{Y}_j-Y_j)^2 \]
If no object is present:\[ p_c=0 \]
then only the presence term is used:\[ \mathcal{L} = (\hat{p}_c-p_c)^2 \]
A combined expression is:\[ \mathcal{L} = (\hat{p}_c-p_c)^2 + p_c \sum_{j=2}^{8} (\hat{Y}_j-Y_j)^2 \]
Multiplication by \(p_c\) masks the remaining terms when no object is present.
A More Appropriate Multi-Task Loss
In practice, the outputs represent different prediction types:
- Object presence is binary classification.
- Bounding-box prediction is regression.
- Object class is multiclass classification.
It is usually better to use a separate loss for each task:\[ \mathcal{L} = \lambda_{\text{obj}}\mathcal{L}_{\text{obj}} + p_c \left[ \lambda_{\text{box}}\mathcal{L}_{\text{box}} + \lambda_{\text{class}}\mathcal{L}_{\text{class}} \right] \]
where the \(\lambda\) values balance the contributions.
Object-Presence Loss
Object presence can use binary cross-entropy:\[ \mathcal{L}_{\text{obj}} = – \left[ p_c\log\hat{p}_c + (1-p_c)\log(1-\hat{p}_c) \right] \]
The model typically produces \(\hat{p}_c\) using a sigmoid function:\[ \hat{p}_c=\sigma(z_{\text{obj}}) \]
Class Loss
When an object is present, the class head can use softmax:\[ \hat{c}_k = \frac{e^{z_k}} {\sum_{j=1}^{3}e^{z_j}} \]
The categorical cross-entropy loss is:\[ \mathcal{L}_{\text{class}} = – \sum_{k=1}^{3} c_k\log\hat{c}_k \]
The class loss is masked by \(p_c\), so it contributes only for positive examples.
Bounding-Box Loss
A simple box loss is coordinate mean squared error:\[ \mathcal{L}_{\text{box}} = (\hat{b}_x-b_x)^2 + (\hat{b}_y-b_y)^2 + (\hat{b}_h-b_h)^2 + (\hat{b}_w-b_w)^2 \]
Other common choices include:
- L1 loss
- Smooth L1 loss
- Intersection-over-Union loss
- Generalized IoU loss
- Distance-IoU loss
- Complete-IoU loss
Direct coordinate losses are easy to understand, while IoU-related losses align more closely with bounding-box overlap.
Intersection over Union
Intersection over Union measures overlap between the predicted box \(B_p\) and ground-truth box \(B_g\):\[ \operatorname{IoU}(B_p,B_g) = \frac{ \operatorname{area}(B_p\cap B_g) }{ \operatorname{area}(B_p\cup B_g) } \]
Its value ranges from 0 to 1:
- 0 means no overlap.
- 1 means identical boxes.
A simple IoU loss is:\[ \mathcal{L}_{\text{IoU}} = 1-\operatorname{IoU}(B_p,B_g) \]
IoU is also commonly used during evaluation.
Balancing the Loss Components
The three tasks may produce losses on different numerical scales.
Without weighting, one term may dominate training.
For example:\[ \lambda_{\text{obj}}=1 \]\[ \lambda_{\text{box}}=5 \]\[ \lambda_{\text{class}}=1 \]
would emphasize accurate localization.
The best weights depend on:
- Coordinate parameterization
- Class distribution
- Object sizes
- Chosen loss functions
- Optimization behavior
The individual losses should be monitored separately rather than only as a single total.
Bounding-Box Output Constraints
The model’s box outputs should describe valid boxes.
A sigmoid can constrain normalized center coordinates:\[ \hat{b}_x=\sigma(z_x) \]\[ \hat{b}_y=\sigma(z_y) \]
Width and height must remain positive. Possible parameterizations include:\[ \hat{b}_w=\sigma(z_w) \]\[ \hat{b}_h=\sigma(z_h) \]
or:\[ \hat{b}_w=e^{z_w} \]\[ \hat{b}_h=e^{z_h} \]
The appropriate transformation depends on whether width and height are normalized and how the detection architecture defines its boxes.
Annotation Requirements
Classification with localization requires more information than image classification.
For every positive image, the dataset needs:
- Object-presence label
- Object class
- Bounding-box coordinates
Annotation consistency matters.
Labelers must follow the same rules for:
- Tight versus loose boxes
- Occluded objects
- Truncated objects
- Object shadows
- Reflections
- Partially visible objects
Inconsistent boxes introduce noise into the regression target.
Geometric Data Augmentation
When an image undergoes a geometric transformation, its bounding box must be transformed as well.
Horizontal flip
For normalized center coordinates:\[ b_x’=1-b_x \]\[ b_y’=b_y \]\[ b_w’=b_w \]\[ b_h’=b_h \]
Scaling
The box must scale with the image.
Cropping
The box coordinates must be translated into the crop’s coordinate system and renormalized.
If the crop removes most or all of the object, the training example may need to be rejected or relabeled.
Limitation: Only One Object
The eight-component output has room for only one object:\[ \left[ p_c,b_x,b_y,b_h,b_w,c_1,c_2,c_3 \right] \]
If an image contains two cars, the target vector cannot represent both simultaneously.
This is the main distinction between classification with localization and full object detection.
To detect multiple objects, the network must produce multiple candidate predictions, usually across spatial locations, queries, or region proposals.
The influential YOLO formulation later represented detection as direct prediction of spatially distributed boxes and class probabilities from the full image. You Only Look Once: Unified, Real-Time Object Detection
Background Imbalance
In many datasets, background images or background regions greatly outnumber positive objects.
If the imbalance is severe, the network may minimize loss by predicting:\[ \hat{p}_c\approx0 \]
too often.
Possible responses include:
- Balanced sampling
- Weighted objectness loss
- Hard-negative mining
- Focal-style losses
- Careful threshold selection
The development set should reflect the intended operating environment even if the training batches are rebalanced.
Evaluation
A localized classification prediction is correct only if:
- The predicted class is correct.
- The predicted box overlaps the target sufficiently.
A common criterion is:\[ \operatorname{IoU}(B_p,B_g)\ge\tau \]
where \(\tau\) is a selected threshold.
Thus, a correct class with a poorly placed box is not a successful localization.
Useful metrics include:
- Classification accuracy
- Mean IoU
- Localization accuracy at a specified IoU threshold
- Object-presence precision and recall
- Per-class localization performance
Key Takeaway
Classification with localization extends image classification by adding four bounding-box outputs:\[ b_x,\quad b_y,\quad b_h,\quad b_w \]
For three foreground classes, the complete target can be written as:\[ Y = \left[ p_c, b_x, b_y, b_h, b_w, c_1, c_2, c_3 \right]^T \]
When an object is present, the model learns:
- Whether an object exists
- Where its bounding box is
- Which class it belongs to
When no object is present, only the object-presence prediction contributes to the loss.
A practical objective combines binary classification, box regression, and multiclass classification:\[ \mathcal{L} = \lambda_{\text{obj}}\mathcal{L}_{\text{obj}} + p_c \left[ \lambda_{\text{box}}\mathcal{L}_{\text{box}} + \lambda_{\text{class}}\mathcal{L}_{\text{class}} \right] \]
This is the foundational step from whole-image classification toward full multi-object detection.