Data Mismatch, Transfer, and Multitask Learning

Suppose you have 200,000 web images and 10,000 photographs from the product camera. The deployed system will see camera images, whose lighting, composition, or labels may differ from the web collection. A gap between training and development error can then mix generalization behavior with population differences. A held-out sample from the training population provides another comparison, but does not turn those differences into a causal decomposition.

Split the data so the question is answerable

Reserve representative target-distribution data for dev (validation) and final test evaluation. You can also use a separate target-training portion for adaptation; putting every target example into dev or test would prevent that. For example, the 10,000 camera images could be divided into 6,000 for adaptation, 2,000 for dev, and 2,000 for test. These counts are illustrative. Choose the allocation from evaluation precision and training needs, and split by person, device, session, or time when related observations could otherwise leak across sets. Keep test data out of model and adaptation choices.

A training-dev set is a held-out sample from the distribution used for fitting. If fitting uses only web images, hold out web images for this comparison. If fitting uses a source–target mixture, either match that mixture in training-dev or report separate source and target holdouts. When deployment itself is a mixture, dev should reflect that mixture and its important slices. A single source–target label is not enough to describe how the data were sampled.

def error_gaps(train, train_dev, dev):
    return {
        "train_dev_minus_train_pp": round(100 * (train_dev - train), 2),
        "target_dev_minus_train_dev_pp": round(100 * (dev - train_dev), 2),
    }


print(error_gaps(0.01, 0.09, 0.10))
print(error_gaps(0.01, 0.015, 0.10))
# {'train_dev_minus_train_pp': 8.0, 'target_dev_minus_train_dev_pp': 1.0}
# {'train_dev_minus_train_pp': 0.5, 'target_dev_minus_train_dev_pp': 8.5}

Inputs are error fractions and outputs are percentage-point gaps. Evaluate the same fitted model with comparable preprocessing, prediction rules, and labels on all sets. The first case suggests investigating performance on unseen training-distribution examples. The second suggests inspecting target-specific failures. Sampling noise, label quality, task difficulty, and pipeline differences can affect either gap. A target domain can also be easier, producing a negative gap. Neither subtraction measures statistical variance or the amount of error a particular adaptation method will remove.

More source data might help either model if it improves useful coverage or representation learning; it might help little if relevant target features are absent. Test that hypothesis with learning curves and a target-dev comparison. A worse final test score likewise does not prove dev overfitting by itself. Human-reference gaps require comparable inputs and a credible baseline, as discussed in A Systematic Strategy for Improving Machine Learning Models.

Addressing mismatch

Inspect what differs before choosing an intervention: input appearance, class frequencies, label definitions, acquisition hardware, or the relation between inputs and labels. Correct a preprocessing or annotation inconsistency before treating it as a modeling problem. Appearance augmentation alone cannot repair a changed target definition.

Compare a few targeted candidates: collect or label representative target-training examples, adjust the source–target sampling mixture, or synthesize a plausible variation such as vehicle noise in speech. Collection is useful only if the new examples and labels address the failure. Synthetic noise must preserve the intended label and resemble relevant conditions. Repeatedly reusing a small noise library risks learning its particular patterns; test on real target examples and noise recordings excluded from synthesis.

Upweighting target-training examples changes the empirical objective. It can improve target performance, but large weights can also amplify noise or reduce effective coverage. Choose weights on target dev and report source performance too if it still matters. Importance weighting has a more specific interpretation under assumptions such as unchanged conditional labels and source coverage of the target inputs; arbitrary upweighting does not automatically correct a distribution shift.

Transfer learning

Transfer learning reuses knowledge learned on a source task or domain for a target one. Here the workflow is pretraining followed by target adaptation; the broader idea also includes reusing frozen features. The backbone extracts features, and the head maps them to the target outputs. Source and target need not have the same classes, and useful pretraining can be self-supervised rather than based on source labels.

Useful feature structure, source quality, and target coverage make transfer promising. Source data need not always outnumber target data, and “near” or “far” domains do not determine a layer-unfreezing rule. With few target labels, a frozen-backbone baseline is inexpensive to compare with partial and full fine-tuning. The number of trainable parameters alone cannot prove that fine-tuning will overfit: initialization, regularization, optimization, and the information in the target examples matter. Compare suitable pretrained candidates and, when feasible, a from-scratch baseline under stated tuning and compute budgets.

The following CPU example isolates freezing and unfreezing. Its small backbone is randomly initialized, so it demonstrates mechanics, not a benefit from transfer. In an actual application, load compatible pretrained weights first, follow their input preprocessing, and replace the source head for the target outputs. The learning rates below are example settings to tune, not thresholds tied to dataset size.

import torch
import torch.nn as nn
import torch.nn.functional as F


torch.manual_seed(4)
backbone = nn.Sequential(nn.Linear(4, 8), nn.BatchNorm1d(8), nn.ReLU())
head = nn.Linear(8, 3)
x = torch.randn(12, 4)
y = torch.arange(12) % 3

for parameter in backbone.parameters():
    parameter.requires_grad_(False)
backbone.eval()
head.train()
optimizer = torch.optim.SGD(head.parameters(), lr=0.01)
before = {k: v.clone() for k, v in backbone.state_dict().items()}
head_before = head.weight.detach().clone()
optimizer.zero_grad(set_to_none=True)
with torch.no_grad():
    features = backbone(x)
F.cross_entropy(head(features), y).backward()
optimizer.step()
print("backbone state unchanged", all(torch.equal(before[k], v)
      for k, v in backbone.state_dict().items()))
print("head changed", not torch.equal(head_before, head.weight))
# backbone state unchanged True
# head changed True

for parameter in backbone.parameters():
    parameter.requires_grad_(True)
# Keep BatchNorm running statistics fixed for this fine-tuning example.
backbone.eval()
optimizer = torch.optim.SGD([
    {"params": backbone.parameters(), "lr": 1e-4},
    {"params": head.parameters(), "lr": 1e-3},
])
backbone_before = backbone[0].weight.detach().clone()
optimizer.zero_grad(set_to_none=True)
F.cross_entropy(head(backbone(x)), y).backward()
optimizer.step()
print("backbone changed", not torch.equal(backbone_before, backbone[0].weight))
# backbone changed True

requires_grad_(False) prevents parameter gradients; it does not freeze BatchNorm running statistics or disable dropout. The first stage also uses eval() and checks the entire backbone state, including buffers. Calling a parent model’s train() later would switch its children back, so reapply the intended backbone mode in the training loop. When BatchNorm is configured without running statistics, evaluation mode still uses batch statistics.

The second stage enables gradients and adds backbone parameters to the optimizer. It removes the no_grad() region, while retaining evaluation-mode BatchNorm as a deliberate choice. Evaluation mode does not disable autograd, so the backbone still updates. Decide separately whether target data justify adapting running statistics. This example rebuilds stateless SGD; rebuilding Adam or momentum SGD would discard accumulated optimizer state unless you preserve it or extend the parameter groups intentionally.

Negative transfer means reuse worsens target performance against the stated comparison, not simply that the domains look different. Catastrophic forgetting refers to loss of previously learned capabilities during adaptation; test those capabilities if retaining them matters. Smaller updates, rehearsal data, regularization, or parameter-efficient adaptation may help. A method such as LoRA reduces the trainable parameter set but does not guarantee retention or better target performance.

Multitask learning

Multitask learning trains for several tasks within a shared training process. A common architecture feeds shared backbone features into separate classification and regression heads. Batches can contain labels for both tasks or alternate between task-specific datasets. The heads receive their own loss gradients; the backbone receives the weighted sum of contributions from whichever tasks are present.

One choice for a batch with missing labels is a weighted sum of per-task means: \[J=\sum_{j:n_j>0}\lambda_j\frac{1}{n_j}\sum_{i:m_{ij}=1}\mathcal L_j(\hat y_j^{(i)},y_j^{(i)}),\qquad n_j=\sum_i m_{ij}.\] The mask \(m_{ij}\) is 1 when example \(i\) has a label for task \(j\), and \(n_j\) counts those labels. The nonnegative weight \(\lambda_j\) specifies that task’s contribution. With complete labels this reduces to the usual weighted sum of task losses averaged over the batch. With missing labels, the denominator is a modeling choice: dividing every task by the full batch size would also weight tasks by label availability.

For a small calculation, two labeled classification examples with equal two-class logits have mean loss log(2), about 0.6931. One labeled regression example with prediction 0 and target 2 has squared error 4. With weights 1 and 0.1, the total is 0.6931 + 0.1 × 4 = 1.0931. The code uses stand-in head outputs to show the loss and mask behavior directly.

def multitask_loss(logits, values, classes, targets):
    class_mask = classes >= 0
    reg_mask = torch.isfinite(targets)
    terms = {}
    if class_mask.any():
        terms["classification"] = F.cross_entropy(logits[class_mask], classes[class_mask])
    if reg_mask.any():
        terms["regression"] = F.mse_loss(values[reg_mask], targets[reg_mask])
    if not terms:
        raise ValueError("batch has no labels for either task")
    weights = {"classification": 1.0, "regression": 0.1}
    total = sum(weights[name] * loss for name, loss in terms.items())
    return total, terms


logits = torch.zeros(3, 2, requires_grad=True)
values = torch.zeros(3, requires_grad=True)
classes = torch.tensor([0, -1, 1])
targets = torch.tensor([float("nan"), 2.0, float("nan")])
total, terms = multitask_loss(logits, values, classes, targets)
total.backward()
print("classification", round(terms["classification"].item(), 4))
print("regression", round(terms["regression"].item(), 4))
print("total", round(total.item(), 4))
print("missing-label gradients", logits.grad[1].tolist(), values.grad[[0, 2]].tolist())
# classification 0.6931
# regression 4.0
# total 1.0931
# missing-label gradients [0.0, 0.0] [0.0, 0.0]

Here −1 marks a missing class and NaN a missing regression target. The masks select valid rows before computing losses; multiplying a NaN loss by zero would not remove it safely. Data validation should distinguish genuinely missing targets from corrupted nonfinite values. The example assumes finite model outputs and valid class indices for labeled rows. A task with no labels contributes no term, and an entirely unlabeled batch raises an error so the caller can handle it explicitly. These means describe the observed labels; if labeling is selective, masking alone does not make them representative of all examples.

Sharing can improve data efficiency, but a feature need not serve every task and task gradients can conflict. Compare each task with a single-task baseline and monitor its own metric. Loss magnitude, gradient magnitude, task sampling, label frequency, and the chosen weights all affect which task drives the shared update. A larger dataset does not automatically make multitask learning unsuitable; sampling and weighting can control its influence, with tradeoffs that require evaluation. Even equal loss values do not imply equal gradient influence.

Classification and box regression in object detection provide a common example of shared features with distinct outputs. The relevant question is whether joint learning and shared inference help the required tasks. Adapting a pretrained language model is covered in Fine-Tuning and Alignment: SFT, LoRA, RLHF, and DPO.

End-to-end versus pipelines

A pipeline exposes intermediate stages, such as locating a meter, recognizing its digits, and parsing the reading. End-to-end training optimizes parameters across the input-to-output mapping using a final-task objective. That mapping can still contain designed modules, pretrained components, and intermediate supervision. A modular system can also be fine-tuned jointly, so these are not mutually exclusive architecture categories.

Question	Joint end-to-end training	Separately trained stages
Supervision	Final-task labels, possibly supplemented by pretraining or auxiliary labels	Stage labels or pretrained modules plus a final-system evaluation
Structure	Architecture and objectives still encode prior choices	Explicit interfaces encode a decomposition
Error investigation	Probes, ablations, and intermediate outputs can help	Stage metrics help, but interactions and error propagation remain
Potential limitation	Optimization cost, data coverage, and objective mismatch	Interface information loss and misaligned stage objectives

Abundant final-task data can make joint optimization attractive, but does not guarantee a win. End-to-end losses can still be surrogates for the operational goal, and interfaces do not always discard useful information. Conversely, reusable stage models can make a pipeline practical with limited target labels, but their preprocessing and domain assumptions must fit the target. Compare available systems under latency, reliability, labeling, and maintenance constraints rather than assigning a universal dataset-size threshold.

For meter reading, evaluate both localization and the final numeric reading. If recognition fails after an incorrect crop, test with a corrected crop to examine that interaction. An oracle crop can reveal potential headroom but is not a deployable result. Stage accuracy alone does not prove that replacing one component will improve the final system. A pretrained image-to-text model is another candidate even with few labeled meter photos; public OCR data alone do not settle the choice.

A practical fine-tuning and augmentation recipe is in Image Classification in Practice: Transfer Learning and Augmentation. Select an adaptation or system change on representative target dev data, then assess the chosen design on the reserved target test set.

References

• PyTorch: Transfer Learning for Computer Vision demonstrates feature extraction and fine-tuning.
• BatchNorm documentation describes running statistics and evaluation behavior.
• Standley et al. (2020), Which Tasks Should Be Learned Together in Multi-task Learning?, studies task groupings.
• Chen et al. (2018), GradNorm, examines balancing task influence through gradient magnitudes.