Multitask Learning: Training One Neural Network for Multiple Tasks

Transfer learning and multitask learning both allow knowledge to be shared between related problems, but they organize the learning process differently.

In transfer learning, training is sequential:\[ \text{Task A} \longrightarrow \text{Task B} \]

A model first learns Task A and then transfers some of that knowledge to Task B.

In multitask learning, several tasks are learned simultaneously:\[ \{\text{Task }1,\text{Task }2,\ldots,\text{Task }K\} \longrightarrow \text{one shared model} \]

The model learns shared representations that can benefit multiple tasks at once.

Multitask learning trains one model to solve several related problems simultaneously, allowing useful information from one task to improve the others.

A Multitask Autonomous-Driving System

Consider a perception system for an autonomous vehicle. Given an image, the system may need to determine whether it contains:

• A pedestrian
• A vehicle
• A stop sign
• A traffic light

One image can contain several of these objects simultaneously. For example, it might contain a vehicle and a stop sign but no pedestrian or traffic light.

Its target label could therefore be:\[ y^{(i)} = \begin{bmatrix} 0\\ 1\\ 1\\ 0 \end{bmatrix} \]

The four components mean:

This is a multilabel classification problem because more than one label can be positive for the same example.

Label-Matrix Representation

Suppose the dataset contains \(m\) examples and \(K\) tasks. Each example has a label vector:\[ y^{(i)} \in \{0,1\}^{K} \]

For the autonomous-driving example:\[ K=4 \]

Stacking the label vectors horizontally gives:\[ Y = \begin{bmatrix} | & | & & |\\ y^{(1)} & y^{(2)} & \cdots & y^{(m)}\\ | & | & & | \end{bmatrix} \]

Therefore:\[ Y\in\{0,1\}^{K\times m} \]

and, with four tasks:\[ Y\in\{0,1\}^{4\times m} \]

A neural network receives an image \(x^{(i)}\) and produces:\[ \hat{y}^{(i)} = \begin{bmatrix} \hat{y}^{(i)}_1\\ \hat{y}^{(i)}_2\\ \hat{y}^{(i)}_3\\ \hat{y}^{(i)}_4 \end{bmatrix} \]

where every output is a probability:\[ \hat{y}^{(i)}_j = P\left(y^{(i)}_j=1\mid x^{(i)}\right) \]

Shared Representation and Task-Specific Outputs

A common multitask architecture contains:

1. A shared feature extractor
2. One or more task-specific output heads

For an input \(x\), the shared layers compute a representation:\[ h=f_{\theta_s}(x) \]

where \(\theta_s\) represents the shared parameters.

Each task then makes a prediction from that representation:\[ \hat{y}_j=g_{\theta_j}(h) \]

where \(\theta_j\) contains parameters specific to task \(j\).

Conceptually:

                         ┌─ Pedestrian prediction
                         ├─ Vehicle prediction
Image → Shared features ─┼─ Stop-sign prediction
                         └─ Traffic-light prediction

The shared layers might learn features such as:

• Edges and corners
• Shapes and contours
• Textures
• Object boundaries
• Perspective
• Road structure
• Spatial relationships

These features can be useful across several road-object recognition tasks.

Multilabel Classification vs. Softmax Classification

Multitask multilabel classification should not be confused with softmax multiclass classification.

Softmax multiclass classification

Softmax assumes that each example belongs to one mutually exclusive class:\[ \sum_{j=1}^{K}\hat{y}_j=1 \]

For example, an image might be classified as exactly one of:

• Cat
• Dog
• Bird
• Other

Increasing the probability of one class reduces the probabilities assigned to the others.

Multilabel classification

Multilabel classification treats every label as a separate binary decision:\[ \hat{y}_j=\sigma(z_j) \]

The probabilities do not need to sum to one. An image may simultaneously contain:

• A pedestrian
• Two vehicles
• A stop sign
• A traffic light

The key distinction is:

The Multitask Loss Function

For example \(i\) and task \(j\), the binary cross-entropy loss is:\[ \mathcal{L}_{ij} = – \left[ y_j^{(i)}\log \hat{y}_j^{(i)} + \left(1-y_j^{(i)}\right) \log\left(1-\hat{y}_j^{(i)}\right) \right] \]

The total loss can be averaged across all examples and tasks:\[ J = \frac{1}{m} \sum_{i=1}^{m} \sum_{j=1}^{K} \mathcal{L}_{ij} \]

For the four-task driving example:\[ J = \frac{1}{m} \sum_{i=1}^{m} \sum_{j=1}^{4} \mathcal{L}_{ij} \]

Minimizing this loss trains the shared model to solve all four binary prediction problems simultaneously.

Weighted Multitask Losses

Not every task should necessarily contribute equally to the total objective.

A more flexible formulation assigns a weight \(\lambda_j\) to each task:\[ J = \frac{1}{m} \sum_{i=1}^{m} \sum_{j=1}^{K} \lambda_j\mathcal{L}_{ij} \]

Task weights can account for:

• Differences in practical importance
• Different numbers of labeled examples
• Class imbalance
• Different loss scales
• Different learning difficulty
• Safety-critical requirements

For example, failing to detect a pedestrian may be more consequential than failing to classify a less important roadside object. A larger task weight could therefore be assigned to pedestrian detection.

However, increasing a weight does not automatically improve that task. Excessively large weights may destabilize optimization or harm other tasks.

Multitask Learning with Missing Labels

A dataset does not need to contain every label for every example.

Suppose the target matrix contains missing entries:\[ Y= \begin{bmatrix} 1 & 1 & ? & 0\\ 0 & 1 & ? & 1\\ ? & ? & 1 & 0\\ ? & ? & 0 & 1 \end{bmatrix} \]

A question mark means that the label is unknown—not that the object is absent.

This distinction is essential:

• \(0\): confirmed absence
• \(1\): confirmed presence
• \(?\): unknown or unannotated

Treating an unknown label as zero introduces false negative labels and can seriously damage training.

Masked Loss for Missing Labels

Define a mask:\[ M_{ij} = \begin{cases} 1, & \text{if } y_j^{(i)} \text{ is known}\\ 0, & \text{if } y_j^{(i)} \text{ is missing} \end{cases} \]

The masked multitask loss becomes:\[ J = \frac{ \displaystyle \sum_{i=1}^{m} \sum_{j=1}^{K} M_{ij}\lambda_j\mathcal{L}_{ij} }{ \displaystyle \sum_{i=1}^{m} \sum_{j=1}^{K} M_{ij} } \]

Only known labels contribute to the loss.

This allows the model to learn from datasets in which:

• Some images were labeled only for vehicles
• Others were labeled only for pedestrians
• Some have complete annotations
• Different annotation sources cover different tasks

Example Masked-Loss Implementation

import torch
import torch.nn.functional as F

def masked_multitask_loss(logits, targets, mask, task_weights=None):
    """
    logits:  [batch_size, num_tasks]
    targets: [batch_size, num_tasks]
    mask:    [batch_size, num_tasks], where 1 means known
    """
    safe_targets = torch.where(mask.bool(), targets, torch.zeros_like(targets))

    per_entry_loss = F.binary_cross_entropy_with_logits(
        logits,
        safe_targets,
        reduction="none",
    )

    if task_weights is not None:
        per_entry_loss = per_entry_loss * task_weights

    masked_loss = per_entry_loss * mask
    denominator = mask.sum().clamp_min(1.0)

    return masked_loss.sum() / denominator

import torch
import torch.nn.functional as F

def masked_multitask_loss(logits, targets, mask, task_weights=None):
    """
    logits:  [batch_size, num_tasks]
    targets: [batch_size, num_tasks]
    mask:    [batch_size, num_tasks], where 1 means known
    """
    safe_targets = torch.where(mask.bool(), targets, torch.zeros_like(targets))

    per_entry_loss = F.binary_cross_entropy_with_logits(
        logits,
        safe_targets,
        reduction="none",
    )

    if task_weights is not None:
        per_entry_loss = per_entry_loss * task_weights

    masked_loss = per_entry_loss * mask
    denominator = mask.sum().clamp_min(1.0)

    return masked_loss.sum() / denominator

Using logits directly with a numerically stable binary cross-entropy implementation is preferable to manually computing logarithms of sigmoid outputs.

Why One Network Can Outperform Separate Networks

Instead of one multitask model, it would be possible to train:

• One pedestrian detector
• One vehicle detector
• One stop-sign detector
• One traffic-light detector

However, separate networks cannot directly share what they learn.

A multitask model may perform better because its shared layers receive supervision from every task.

Shared statistical strength

Suppose one task has only 1,000 labeled examples, while 99 related tasks each have a similar amount of data.

Training the task alone provides approximately:\[ 1{,}000 \text{ examples} \]

Training it with the other tasks exposes the shared representation to approximately:\[ 100\times 1{,}000=100{,}000 \]

labeled examples across related problems.

The labels are not interchangeable, but they may collectively teach the shared network useful structure.

Representation regularization

A model trained on only one small task may learn narrow or accidental patterns. Requiring the same features to support several related tasks can encourage more broadly useful representations.

In that sense, multitask learning can act as a form of inductive bias or regularization.

Computational efficiency

A shared model may also be more efficient at inference time. Rather than running several large networks separately, the system computes the expensive shared features once and then applies relatively small task-specific heads.

When Multitask Learning Is Most Likely to Work

Multitask learning is especially promising when several conditions hold.

1. The Tasks Share Useful Features

The tasks should benefit from related representations.

Examples include:

• Detecting multiple road objects
• Predicting facial landmarks and head pose
• Performing speech recognition and speaker identification
• Predicting related medical outcomes from the same patient record
• Estimating depth, surface normals, and semantic segmentation from images
• Analyzing sentiment, topic, and intent from the same text

The more closely related the useful representations are, the greater the potential for positive transfer.

2. The Other Tasks Provide Substantial Additional Data

Consider a particular task \(T_j\). Multitask learning is most useful when the other tasks collectively provide enough relevant supervision to improve the shared representation.

If task \(T_j\) already has millions of high-quality examples, adding a tiny auxiliary task may produce little benefit.

If task \(T_j\) has limited data but many related tasks collectively have a large amount, the potential gain is much greater.

The tasks do not need exactly equal dataset sizes. The key question is:

Do the other tasks collectively provide useful information that the target task could not learn as reliably from its own data?

3. The Model Has Enough Capacity

A shared model must be large enough to represent all tasks adequately.

If the network is too small, the tasks may compete for limited capacity. Performance can then be worse than with separate models.

Possible responses include:

• Increasing the width or depth of shared layers
• Adding task-specific branches
• Sharing only earlier layers
• Using separate normalization parameters
• Introducing mixture-of-experts components
• Reducing the number of tasks grouped into one model

4. The Inputs and Operational Setting Are Compatible

Multitask learning is easiest when tasks operate on the same input or closely related inputs.

For example, a single road image naturally supports several object-presence predictions. In contrast, combining unrelated tasks with different input types may require substantially more complex architectures and may offer little useful sharing.

5. The Labels and Objectives Are Reliable

Noisy labels from one task can damage the shared representation and affect other tasks. This makes label quality especially important.

Each task should be checked for:

• Label noise
• Ambiguous definitions
• Missing-label handling
• Class imbalance
• Distribution mismatch
• Inconsistent annotation policies

Negative Transfer

Multitask learning does not always improve every task. Sometimes sharing causes negative transfer, where training tasks together makes one or more tasks worse.

This can happen even when the model is large.

Common causes include:

• Tasks requiring conflicting features
• One task dominating the gradients
• Extremely different dataset sizes
• Different noise levels
• Different input distributions
• Poorly chosen loss weights
• Conflicting optimization dynamics
• Unrelated or weakly related tasks

For example, one task may reward invariance to color while another requires precise color information. If they share all features, their gradient updates may pull the representation in conflicting directions.

Related tasks can reinforce one another, but unrelated or conflicting tasks can interfere with one another.

Detecting Negative Transfer

Always compare the multitask model against single-task baselines.

For task \(j\), define the transfer effect as:\[ \Delta_j = E_j^{\text{single}} – E_j^{\text{multi}} \]

where \(E\) is an error metric.

Then:

• \(\Delta_j>0\): multitask learning improved the task.
• \(\Delta_j=0\): little measurable effect.
• \(\Delta_j<0\): negative transfer occurred.

Average performance alone can hide regressions. A multitask system might improve easy tasks while degrading a safety-critical one.

Therefore, track:

• Per-task metrics
• Overall aggregate metrics
• Worst-task performance
• Safety or fairness constraints
• Training stability
• Inference cost

Hard and Soft Parameter Sharing

There are several ways to share information among tasks.

Hard parameter sharing

Most layers are shared, with separate output heads:\[ h=f_{\theta_s}(x), \qquad \hat{y}_j=g_{\theta_j}(h) \]

This is simple and computationally efficient.

It works well when the tasks are closely related, but it can cause interference when their needs differ.

Soft parameter sharing

Each task has its own model parameters, but the models are encouraged to learn similar representations.

For two tasks, a regularization term might be:\[ \Omega(\theta_1,\theta_2) = \|\theta_1-\theta_2\|_2^2 \]

The full objective becomes:\[ J = J_1+J_2+\lambda\Omega(\theta_1,\theta_2) \]

Soft sharing gives each task more independence but usually requires more parameters and computation.

Partial sharing

A practical compromise is to share early layers while using separate deeper branches:\[ x \rightarrow \text{shared low-level features} \rightarrow \begin{cases} \text{Task group A}\\ \text{Task group B}\\ \text{Task group C} \end{cases} \]

This is useful when tasks share low-level structure but require different high-level reasoning.

Balancing Multiple Tasks

One of the hardest practical issues is deciding how strongly each task should influence learning.

Fixed weighting

Choose constant task weights:\[ J=\sum_{j=1}^{K}\lambda_jJ_j \]

This is simple but may require tuning.

Sampling balance

Instead of modifying the loss, control how frequently examples from each task appear.

This can prevent large datasets from overwhelming smaller tasks.

Dynamic weighting

Weights can change during training according to:

• Loss magnitude
• Learning speed
• Gradient magnitude
• Task uncertainty
• Validation performance

Dynamic methods can help, but they add complexity and do not guarantee that all task conflicts will disappear.

Multitask Learning vs. Transfer Learning

The two methods share knowledge differently.

When Transfer Learning Is Preferable

Transfer learning is often preferable when:

• One target task matters most.
• The target dataset is small.
• A large pretrained model already exists.
• The source and target tasks are trained at different times.
• Joint access to all datasets is difficult.
• Training all tasks together would be operationally complex.

For example, a model trained on millions of general images can be adapted to a smaller medical-imaging dataset.

When Multitask Learning Is Preferable

Multitask learning is attractive when:

• Several tasks must be performed in the same application.
• The tasks use the same or related inputs.
• Low-level or intermediate features are likely to be shared.
• Joint labels are available or missing labels can be masked.
• One sufficiently capable model can serve all tasks.
• Shared inference reduces computational cost.

Object perception for autonomous systems is a natural example because a single image must support many related predictions.

Multitask Learning and Modern Object Detection

The simple example predicts whether each object category appears anywhere in an image. A complete object detector must do more:

• Determine which categories appear
• Locate every object
• Predict bounding boxes or masks
• Handle multiple instances of the same category
• Estimate confidence scores

These requirements introduce several related objectives:\[ J = \lambda_{\text{class}}J_{\text{class}} + \lambda_{\text{box}}J_{\text{box}} + \lambda_{\text{object}}J_{\text{object}} + \lambda_{\text{mask}}J_{\text{mask}} \]

A single detector may therefore learn multiple tasks simultaneously:

• Object classification
• Object localization
• Foreground/background prediction
• Instance segmentation

This is one reason multitask learning appears naturally in computer vision systems.

A Practical Development Workflow

Step 1: Define the tasks precisely

Specify:

• Inputs
• Outputs
• Label meanings
• Metrics
• Missing-label conventions
• Operational importance

Step 2: Build single-task baselines

Train each task independently when practical. These results establish whether sharing actually helps.

Step 3: Identify plausible shared representations

Determine which layers or features should be common and which should remain task-specific.

Step 4: Implement masked and weighted losses

Exclude missing labels and prevent one task from dominating simply because it has more annotations.

Step 5: Train a sufficiently capable model

Under-capacity can cause task competition. Monitor both aggregate and per-task training behavior.

Step 6: Compare per-task performance

Do not rely only on the total loss. Each task should be compared with its single-task baseline.

Step 7: Diagnose interference

If one task deteriorates, consider:

• Adjusting its loss weight
• Resampling its examples
• Giving it a separate branch
• Sharing fewer layers
• Grouping only compatible tasks
• Removing a harmful auxiliary task

Step 8: Evaluate deployment tradeoffs

Consider whether the shared model improves:

• Accuracy
• Latency
• Memory use
• Energy consumption
• Maintenance
• Training complexity

A multitask model may be desirable even with similar accuracy if it substantially reduces deployment cost.

Key Takeaway

Multitask learning trains one neural network to perform several related tasks simultaneously. A shared representation can allow each task to benefit from supervision provided by the others.

For multilabel tasks, use independent sigmoid outputs and binary cross-entropy:\[ J = \frac{1}{m} \sum_{i=1}^{m} \sum_{j=1}^{K} \mathcal{L}\left(y_j^{(i)},\hat{y}_j^{(i)}\right) \]

When labels are missing, introduce a mask and exclude unknown entries from the loss.

Multitask learning is most promising when:

• The tasks share useful features.
• The other tasks collectively provide substantial relevant data.
• The model has enough capacity.
• The objectives and datasets are compatible.

Its main risk is negative transfer. A large model alone does not guarantee success, so the multitask system should always be compared against per-task baselines. When the tasks are well aligned, one shared model can provide better representations, improved data efficiency, and lower deployment cost than training every task independently.