Introduction to Computer Vision and Convolutional Neural Networks
Computer vision is one of the areas most dramatically transformed by deep learning. Modern vision systems can recognize objects, locate pedestrians and vehicles, verify identities from faces, organize visual content, analyze medical images, and generate new forms of digital art.
Many ideas developed for computer vision have also influenced other areas of machine learning. Convolutional architectures, residual connections, attention mechanisms, and representation-learning techniques have all crossed into fields such as speech processing, natural language processing, robotics, and scientific computing.
At the center of many computer vision systems is the convolutional neural network, commonly called a CNN or ConvNet.
What Is Computer Vision?
Computer vision develops systems that extract useful information from images or video.
An image can be represented as a multidimensional numerical array. For an RGB image:\[ X\in\mathbb{R}^{n_H\times n_W\times n_C} \]
where:
- \(n_H\) is the image height.
- \(n_W\) is the image width.
- \(n_C\) is the number of channels.
For an RGB image:\[ n_C=3 \]
The three channels represent red, green, and blue intensity values.
A computer vision model learns a function such as:\[ f_\theta(X)=\hat{Y} \]
The meaning of \(\hat{Y}\) depends on the task. It could be a class label, a collection of bounding boxes, a segmentation mask, an image, or another structured output.
Major Computer Vision Tasks
Computer vision includes many distinct learning problems. Three important examples are image classification, object detection, and neural style transfer.
Image Classification
Image classification assigns a label to an entire image.
For example:\[ X\longrightarrow \hat{Y} \]
where:
- \(X\) is an image.
- \(\hat{Y}\) is the predicted class.
A binary classifier might answer:\[ \text{Does this image contain a cat?} \]
A multiclass classifier might choose among:
- Cat
- Dog
- Bird
- Vehicle
- Other
The model usually produces a probability distribution:\[ \hat{y}_k=P(y=k\mid X) \]
and predicts:\[ \hat{k}=\arg\max_k\hat{y}_k \]
Classification identifies the primary category but usually does not report where the object appears.
Object Detection
Object detection combines classification with localization.
Instead of merely deciding whether a vehicle exists in an image, the model must determine:
- Which objects are present
- Where each object is located
- How confident it is about each prediction
A detector commonly represents an object with a class label and bounding box:\[ (c,b_x,b_y,b_w,b_h) \]
where:
- \(c\) is the predicted class.
- \(b_x,b_y\) specify the box position.
- \(b_w,b_h\) specify its dimensions.
Unlike ordinary image classification, object detection must handle multiple objects:\[ X \longrightarrow \left\{ (c_i,b_i,p_i) \right\}_{i=1}^{N} \]
where:
- \(N\) is the number of detected objects.
- \(c_i\) is an object category.
- \(b_i\) is a bounding box.
- \(p_i\) is a confidence score.
For autonomous systems, detecting the existence of another vehicle is not sufficient. The system must know its position and extent to support tracking, prediction, planning, and collision avoidance.
Image Segmentation
A related task is image segmentation, which assigns a label to individual pixels.
In semantic segmentation:\[ X \longrightarrow \hat{Y}, \qquad \hat{Y}\in\{1,\ldots,K\}^{n_H\times n_W} \]
Each output location specifies the predicted category of the corresponding pixel.
For example, a road scene may be divided into:
- Road
- Vehicle
- Pedestrian
- Sidewalk
- Building
- Sky
Segmentation provides a more detailed spatial description than a bounding box.
Neural Style Transfer
Neural style transfer combines the content of one image with the visual style of another.
Let:
- \(C\) be a content image.
- \(S\) be a style image.
- \(G\) be the generated image.
The objective is to preserve high-level content from \(C\) while reproducing stylistic characteristics from \(S\).
A simplified objective is:\[ J(G) = \alpha J_{\text{content}}(C,G) + \beta J_{\text{style}}(S,G) \]
where:
- \(J_{\text{content}}\) measures differences in content representations.
- \(J_{\text{style}}\) measures differences in visual style.
- \(\alpha\) and \(\beta\) control their relative importance.
This demonstrates that convolutional networks can do more than recognize objects. Their internal representations can also capture meaningful properties of texture, composition, and visual appearance.
Why Image Inputs Are Challenging
Images can contain millions of numerical values.
Consider a small RGB image with dimensions:\[ 64\times64\times3 \]
The number of input features is:\[ 64\cdot64\cdot3=12{,}288 \]
The flattened input vector would therefore be:\[ x\in\mathbb{R}^{12{,}288} \]
This is manageable for a modest fully connected network.
However, a \(64\times64\) image contains relatively little spatial detail. Real applications often require much larger images.
A One-Megapixel Example
Consider an RGB image with dimensions:\[ 1000\times1000\times3 \]
Its number of features is:\[ 1000\cdot1000\cdot3=3{,}000{,}000 \]
After flattening:\[ x\in\mathbb{R}^{3{,}000{,}000} \]
The model must process three million input values for a single image.
The Fully Connected Parameter Problem
Suppose the first hidden layer contains 1,000 units.
For a fully connected layer:\[ z^{[1]}=W^{[1]}x+b^{[1]} \]
The weight matrix must have dimensions:\[ W^{[1]} \in \mathbb{R}^{1000\times3{,}000{,}000} \]
Therefore, the number of weights is:\[ 1000\cdot3{,}000{,}000 = 3{,}000{,}000{,}000 \]
That is three billion parameters in only the first layer.
If each parameter uses a 32-bit floating-point value, the weights alone require approximately:\[ 3\times10^9\cdot4\text{ bytes} = 12\times10^9\text{ bytes} \]
This is about 12 GB using decimal units, before accounting for:
- Biases
- Additional layers
- Gradients
- Optimizer state
- Activations
- Temporary computation buffers
- Mini-batch processing
An optimizer that stores additional values for every parameter can multiply the training-memory requirement.
Why So Many Parameters Are a Problem
An enormous fully connected layer creates several difficulties.
Computational expense
Computing:\[ W^{[1]}x \]
requires billions of multiply-add operations for a single example.
Memory consumption
Training requires storing not only parameters but also gradients, activations, and optimizer statistics.
Overfitting
A model with billions of independently adjustable weights can memorize the training data unless the dataset is extremely large or strong regularization is used.
Loss of spatial structure
Flattening an image converts:\[ X\in\mathbb{R}^{n_H\times n_W\times n_C} \]
into a vector:\[ x\in\mathbb{R}^{n_Hn_Wn_C} \]
The operation obscures the natural two-dimensional relationships among nearby pixels.
But spatial relationships are central to visual understanding:
- Adjacent pixels often belong to the same object.
- Edges are defined by local intensity changes.
- Textures are composed of recurring local patterns.
- Object parts have structured spatial arrangements.
A general fully connected layer does not explicitly exploit these properties.
The Key Insight Behind Convolutional Networks
Convolutional neural networks address these difficulties by using three important ideas:
- Local connectivity
- Parameter sharing
- Hierarchical feature learning
Local Connectivity
A convolutional unit examines only a small local region of an image rather than connecting to every input pixel.
For example, a \(3\times3\) filter applied to an RGB image contains:\[ 3\cdot3\cdot3=27 \]
weights, plus an optional bias.
That same filter can be applied across the entire image.
This is much more efficient than learning a separate connection between every pixel and every hidden unit.
Parameter Sharing
A visual feature such as a vertical edge can appear anywhere in an image.
Instead of learning a different edge detector for every location, a CNN reuses the same filter across multiple positions.
If a filter is denoted by \(K\), the convolution operation can be written conceptually as:\[ Z_{i,j} = \sum_{a} \sum_{b} \sum_{c} K_{a,b,c} X_{i+a,j+b,c} +b \]
The parameters of \(K\) are shared at every valid spatial location.
This dramatically reduces the number of trainable parameters.
Hierarchical Feature Learning
Early convolutional layers tend to learn relatively simple local patterns, such as:
- Horizontal edges
- Vertical edges
- Corners
- Color transitions
- Basic textures
Deeper layers combine these patterns into more complex representations:
- Curves
- Object parts
- Faces
- Wheels
- Vehicles
- Entire objects
- Scene-level concepts
A simplified hierarchy is:\[ \text{pixels} \rightarrow \text{edges} \rightarrow \text{textures and shapes} \rightarrow \text{parts} \rightarrow \text{objects} \]
These representations are learned from data rather than manually specified.
Comparing Fully Connected and Convolutional Layers
| Property | Fully connected layer | Convolutional layer |
|---|---|---|
| Connectivity | Every output connects to every input | Outputs connect to local regions |
| Parameter sharing | No | Yes |
| Spatial structure | Usually lost after flattening | Preserved |
| Parameter count | Grows rapidly with image size | Determined mainly by filter size and count |
| Translation behavior | Must be learned separately | Naturally supports shared detection across locations |
| Typical role in vision | Final prediction layers or small inputs | Main visual feature extractor |
Parameter Count in a Convolutional Layer
Suppose a convolutional layer has:
- \(f\times f\) filters
- \(n_C^{\text{prev}}\) input channels
- \(n_C\) output channels
Each output channel has one filter with:\[ f\cdot f\cdot n_C^{\text{prev}} \]
weights.
Including one bias for each output channel, the total number of parameters is:\[ n_C \left( f^2n_C^{\text{prev}}+1 \right) \]
For example, suppose:\[ f=3,\qquad n_C^{\text{prev}}=3,\qquad n_C=64 \]
Then:\[ \text{parameters} = 64(3^2\cdot3+1) \]\[ = 64(27+1) = 1{,}792 \]
Only 1,792 parameters are required, regardless of whether the input image is \(64\times64\) or \(1000\times1000\). A larger image increases the amount of computation, but it does not increase the number of parameters in that convolutional layer.
This is a fundamental advantage of convolution.
Why Convolution Fits Visual Data
Convolution reflects several useful assumptions about images.
Nearby pixels are related
Objects and textures are usually expressed through local spatial patterns.
Features can appear in different locations
The same edge, corner, or texture may occur anywhere in an image.
Complex features can be composed from simpler ones
Edges combine into shapes, shapes combine into parts, and parts combine into objects.
These assumptions form an effective inductive bias for visual learning.
Translation Equivariance
If an object moves within an image, the corresponding feature response should also move.
Convolution naturally supports this property, known as translation equivariance.
Conceptually, if \(T_\Delta\) shifts an image and \(f\) is a convolutional operation, then:\[ f(T_\Delta X) \approx T_\Delta f(X) \]
A shifted edge produces a shifted activation map rather than requiring a completely new detector.
Translation equivariance is not the same as complete translation invariance. A CNN can preserve information about where a feature appears, which is essential for localization and detection.
Computer Vision Beyond Recognition
The influence of convolutional architectures extends beyond assigning labels to images.
CNN-based or convolution-inspired systems can support:
- Image classification
- Object detection
- Face verification
- Semantic segmentation
- Instance segmentation
- Pose estimation
- Medical-image analysis
- Image restoration
- Super-resolution
- Visual search
- Image generation
- Style transformation
- Video understanding
- Robotics perception
The shared principle is learning spatial representations efficiently.
Why Vision Research Influences Other Fields
Convolutional architectures were developed around visual data, but their core ideas are more general.
Local structure
Audio waveforms, time series, and token sequences also contain important local relationships.
Shared patterns
A useful temporal or linguistic pattern can appear at different positions in a sequence.
Hierarchical composition
Simple patterns can combine into increasingly abstract structures.
Consequently, convolutional ideas have been applied to:
- Speech recognition
- Audio classification
- Natural language processing
- Biological sequences
- Time-series forecasting
- Anomaly detection
- Scientific measurements
Even when a system does not process images, understanding convolutional architectures provides valuable insight into efficient representation learning.
From Edge Detection to Deep Vision Systems
One of the simplest uses of convolution is edge detection.
An edge is a region where image intensity changes sharply. For example:
- A vertical edge involves a strong horizontal change in intensity.
- A horizontal edge involves a strong vertical change in intensity.
Small convolutional filters can detect these changes. Deep CNNs expand this basic operation into a complete hierarchy of learned visual features.
The progression is:\[ \text{edge detection} \rightarrow \text{learned filters} \rightarrow \text{deep convolutional layers} \rightarrow \text{complex visual recognition} \]
Understanding edge detection therefore provides a natural introduction to the convolution operation itself.
Key Takeaway
Computer vision models must often process images containing millions of input values. A conventional fully connected layer applied directly to a high-resolution image can require billions of parameters, creating severe computational, memory, and overfitting problems.
Convolutional neural networks address this challenge by:
- Connecting units to local image regions
- Sharing filters across spatial locations
- Preserving spatial structure
- Learning hierarchical visual representations
- Using far fewer parameters than dense layers
These properties make convolution one of the fundamental building blocks of modern computer vision. The simplest place to see convolution in action is edge detection, where a small filter scans an image to identify sharp spatial changes.