Setting Up Development and Test Sets from the Same Distribution

The way development and test sets are constructed can strongly affect how efficiently a machine-learning team makes progress.

The development set guides model selection. The test set measures the final system. If these datasets represent different targets, months of improvements on the development set may fail to transfer to the test set.

Development and test sets should generally come from the same distribution and represent the data on which the system must perform well.

The Role of the Development Set

The development set is also called:

• The dev set
• The validation set
• The holdout validation set
• The cross-validation set

The usual workflow is:

1. Train candidate models using the training set.
2. Evaluate them on the development set.
3. Select promising architectures and hyperparameters.
4. Repeat the process until development performance is satisfactory.
5. Evaluate the selected system on the test set.

The development set influences many decisions:

• Model architecture
• Hyperparameters
• Regularization
• Feature engineering
• Decision thresholds
• Error-analysis priorities
• Checkpoint selection

Even though the model does not directly learn its weights from the development set, the development process adapts to it.

The Role of the Test Set

The test set should provide a relatively unbiased estimate of the selected model’s performance.

It should normally be used after the main development decisions have been completed. If test results repeatedly influence those decisions, the test set gradually becomes another development set.

The intended separation is:\[ \text{Development set} \rightarrow \text{model selection} \]\[ \text{Test set} \rightarrow \text{final evaluation} \]

The Target Analogy

The development set and evaluation metric together define a target.

For example:\[ \text{Target} = \text{performance on the development distribution} \]

measured using:\[ \text{selected evaluation metric} \]

Once this target is defined, a team can iterate quickly:\[ \text{idea} \rightarrow \text{experiment} \rightarrow \text{development score} \rightarrow \text{next idea} \]

Machine-learning teams are often effective at improving whatever metric they repeatedly measure.

The danger is that they may optimize the wrong target.

A team can efficiently improve development performance while making little progress on the application that actually matters.

A Geographic Example

Suppose an image classifier will operate in eight regions:

• United States
• United Kingdom
• Other European countries
• South America
• India
• China
• Other Asian countries
• Australia

One possible split would place four regions entirely in the development set and the other four entirely in the test set.

For example:\[ \text{Dev} = \{\text{United States, United Kingdom, Europe, South America}\} \]\[ \text{Test} = \{\text{India, China, other Asia, Australia}\} \]

This is usually a poor design because the two sets may follow different distributions:\[ P_{\text{dev}}(X,Y) \neq P_{\text{test}}(X,Y) \]

Differences may include:

• Camera types
• Image quality
• Backgrounds
• User behavior
• Label frequencies
• Lighting conditions
• Internet connectivity
• Cultural or geographic patterns

A model selected for one collection of regions may not be the best model for the other.

A Better Geographic Split

If all eight regions are important, combine the relevant data and then divide it into development and test sets so both contain examples from all regions.

Conceptually:\[ D_{\text{target}} = D_{\text{US}} \cup D_{\text{UK}} \cup D_{\text{Europe}} \cup \cdots \cup D_{\text{Australia}} \]

Then sample:\[ D_{\text{dev}}\sim P_{\text{target}} \]

and:\[ D_{\text{test}}\sim P_{\text{target}} \]

This produces:\[ P_{\text{dev}}(X,Y) \approx P_{\text{test}}(X,Y) \approx P_{\text{target}}(X,Y) \]

Both datasets now represent the same intended application.

Why Different Distributions Waste Time

Suppose the team spends several months improving the development metric. At the end, the model is evaluated on a test set with a substantially different distribution.

Poor test performance would not necessarily mean that the team failed to optimize. It could mean that the team successfully optimized the wrong objective.

Using the target analogy:

1. The development set places the target in one location.
2. The team improves its aim.
3. The test set moves the target somewhere else.
4. Previous progress is no longer reliable.

This creates unnecessary rework because the evaluation structure—not merely the model—was incorrect.

A Lending Example

Consider a model that predicts whether a loan applicant will repay:\[ X \rightarrow \hat{Y} \]

Suppose the development set contains applications from middle-income postal codes, but the final evaluation uses applications from lower-income postal codes.

These populations may differ in important ways:

• Income distributions
• Employment patterns
• Loan purposes
• Credit histories
• Repayment behavior
• Missing-data patterns
• Access to financial services

A model selected using the first distribution may not perform well on the second.

The problem is not simply that the model generalized poorly. The development process optimized performance on a population different from the one used to judge success.

A better design would ensure that both development and test sets reflect the populations important to the intended deployment.

Define the Target Distribution First

Before splitting the data, define the population on which the system must perform well.

Questions include:

• Who will use the system?
• Where will inputs come from?
• Which devices will generate the data?
• What input quality should be expected?
• Which groups are most important?
• Which rare cases must be handled?
• Will the data distribution change over time?
• What conditions are expected after deployment?

This defines the target distribution:\[ P_{\text{target}}(X,Y) \]

Development and test data should then be sampled to represent that distribution.

Same Distribution Does Not Mean Identical Data

Development and test sets should contain different examples. “Same distribution” means that they are sampled according to the same underlying rules.

For example, both may contain:

• The same geographic mixture
• Similar class proportions
• Similar image quality
• Similar device types
• The same time period
• Comparable user populations

But no individual example should normally appear in both sets.

Development and test sets should represent the same target population without sharing examples.

Random Splitting

When examples are independent and identically distributed, a random split is often appropriate:

from sklearn.model_selection import train_test_split

dev_data, test_data = train_test_split(
    target_data,
    test_size=0.5,
    random_state=42,
    shuffle=True,
)

from sklearn.model_selection import train_test_split

dev_data, test_data = train_test_split(
    target_data,
    test_size=0.5,
    random_state=42,
    shuffle=True,
)

This works when ordinary random sampling preserves the important characteristics of the target distribution.

However, random splitting must be performed at the correct unit.

Preventing Data Leakage

Related examples should not be divided across development and test sets.

Examples include:

• Multiple images from the same user
• Frames from the same video
• Records from the same patient
• Transactions from the same account
• Documents from the same source
• Measurements from the same device

If related records appear in both sets, the test result may be overly optimistic.

The split may need to occur by:

• User
• Patient
• Device
• Geographic location
• Session
• Document source
• Time period

The objective is to preserve independence while maintaining comparable target distributions.

Stratified Splitting

If important groups are uncommon, ordinary random sampling may produce unbalanced development and test sets.

Stratification can preserve proportions for:

• Classes
• Regions
• Device types
• Demographic groups
• Input-quality categories
• Other important subgroups

For classification, a stratified split might be:

from sklearn.model_selection import train_test_split

dev_data, test_data = train_test_split(
    target_data,
    test_size=0.5,
    random_state=42,
    shuffle=True,
    stratify=target_labels,
)

from sklearn.model_selection import train_test_split

dev_data, test_data = train_test_split(
    target_data,
    test_size=0.5,
    random_state=42,
    shuffle=True,
    stratify=target_labels,
)

If several attributes must be preserved, a combined stratification category or a custom splitting procedure may be needed.

Time-Dependent Systems

Random splitting is not always appropriate.

If the model predicts future events, evaluation should reproduce that setting:\[ \text{Training time} < \text{Development time} < \text{Test time} \]

For example:

• Train on January through September
• Develop on October
• Test on November

The development and test sets still need to represent comparable future-use conditions, but chronological separation prevents future information from leaking backward.

In this case, the two sets may not be perfectly identical because real distributions change over time. The goal is to make development performance predictive of the next unseen period.

Development and Test Sets Need the Same Objective

Matching distributions is necessary but not sufficient. Both sets should also use:

• The same label definitions
• The same annotation standards
• The same evaluation metric
• Comparable preprocessing
• The same inclusion and exclusion criteria
• Consistent treatment of ambiguous examples

If the development set uses one labeling policy and the test set uses another, model rankings may still fail to transfer.

The Training Distribution Can Differ

The training set does not always need to follow exactly the same distribution as the development and test sets.

A model may benefit from additional training data collected from:

• Public datasets
• Web scraping
• Synthetic data
• Historical sources
• Different devices
• Different geographic regions

This additional data may improve learning even when it does not perfectly match deployment.

The most important requirement is:\[ P_{\text{dev}} \approx P_{\text{test}} \approx P_{\text{target}} \]

The training distribution can be broader or somewhat different, provided that the difference is recognized and managed.

Why Development and Test Must Match More Closely

The development set selects the model, and the test set evaluates the selected model.

If they represent different objectives, development results cannot reliably predict test results.

The training set has a different role. It provides information from which the model learns. A broader training distribution can sometimes be beneficial.

Weighting Multiple Regions or Groups

Simply including every region may not be sufficient. Their proportions also matter.

Suppose expected production traffic is:

The development and test sets could reflect these proportions. Alternatively, if performance in every region is equally important, the evaluation metric could give regions equal weight regardless of traffic volume.

These represent different product objectives:

Traffic-weighted objective

\[ \operatorname{Error} = \sum_r P_{\text{production}}(r) \operatorname{Error}_r \]

Equal-region objective

\[ \operatorname{Error} = \frac{1}{R} \sum_{r=1}^{R} \operatorname{Error}_r \]

The correct choice depends on what success means.

Validate the Split

After creating development and test sets, compare their properties.

Useful checks include:

• Number of examples
• Class distribution
• Geographic distribution
• Device distribution
• Input-quality distribution
• Missing-value rates
• Label prevalence
• Time coverage
• Key subgroup counts

A simple comparison table can reveal unintended mismatches before model development begins.

Exact equality is unnecessary, but large unexplained differences deserve investigation.

Common Mistakes

Splitting by convenience

Data sources are assigned to different sets simply because it is operationally easy.

Using different geographic regions

One group of regions is used for development and another for testing, even though both are important.

Ignoring production data

The evaluation data is clean and curated, while production data is noisy and informal.

Splitting related examples independently

Nearly identical examples appear in both sets, producing data leakage.

Changing the target late

The intended application changes, but the development and test distributions remain unchanged.

Using incompatible label standards

Different annotation rules create artificial differences between the datasets.

A Practical Setup Process

1. Define the intended deployment population.
2. Identify important groups and rare cases.
3. Collect or select representative examples.
4. Decide the correct splitting unit.
5. Separate related examples to prevent leakage.
6. Divide the target data into development and test sets.
7. Preserve relevant distributions through randomization or stratification.
8. Verify that both sets have comparable characteristics.
9. Define one consistent evaluation metric.
10. Revisit the setup if the application or data distribution changes.

Key Takeaway

The development set and evaluation metric define the target your team optimizes. The test set should measure performance on that same target. Construct development and test sets from the same application-relevant distribution, prevent leakage between them, and ensure that both represent the data the system must handle in real use.