Choosing the Size of Development and Test Sets
Traditional machine-learning guidance often recommends dividing data using fixed percentages such as:\[ 70\% \text{ training},\quad 30\% \text{ test} \]
or:\[ 60\% \text{ training},\quad 20\% \text{ development},\quad 20\% \text{ test} \]
These ratios can be reasonable for small datasets. With hundreds of thousands or millions of examples, however, allocating 20–30% of the data to evaluation may be unnecessary.
Development and test sets should be sized according to their purposes—not according to a universal percentage.
Use enough development data to compare models reliably and enough test data to estimate final performance confidently. Allocate the remaining useful data to training.
The Purpose of Each Dataset
The appropriate size depends on the role of each split.
Training set
The training set is used to learn model parameters:\[ W^{[1]},b^{[1]},\ldots,W^{[L]},b^{[L]} \]
More training data can help the model:
- Learn more robust patterns
- Cover additional variation
- Reduce variance
- Generalize more effectively
Development set
The development set is used to make decisions such as:
- Selecting an architecture
- Choosing hyperparameters
- Comparing experiments
- Selecting a checkpoint
- Setting decision thresholds
- Performing error analysis
It should be large enough to distinguish meaningfully between competing models.
Test set
The test set is used after model development to estimate the final system’s performance.
It should be large enough to provide the required statistical confidence and detailed enough to evaluate important subgroups or error types.
Why Traditional Ratios Were Reasonable
Suppose a dataset contains only 100 examples.
A 70/30 split produces:
- 70 training examples
- 30 test examples
A 60/20/20 split produces:
- 60 training examples
- 20 development examples
- 20 test examples
These evaluation sets are already small. Reducing them to 1% would leave only one example, which is clearly inadequate.
With 1,000 examples, allocating 20% to a development set produces 200 examples. Depending on the task, this may be a reasonable minimum for comparing models.
Fixed percentages were useful when datasets were relatively small because the resulting absolute evaluation-set sizes remained modest.
Why the Same Ratios Fail at Large Scale
Suppose the complete dataset contains:\[ 1{,}000{,}000 \]
examples.
A 60/20/20 split would produce:
- 600,000 training examples
- 200,000 development examples
- 200,000 test examples
The development and test sets may be much larger than necessary.
A 98/1/1 split instead produces:
- 980,000 training examples
- 10,000 development examples
- 10,000 test examples
In many applications, 10,000 examples may be sufficient for model comparison and final evaluation.
The percentages are small, but the absolute evaluation sets remain substantial.
Percentages Versus Absolute Sizes
The key distinction is:\[ \text{Percentage} \neq \text{Evaluation quality} \]
The usefulness of an evaluation set depends more directly on its absolute number of representative examples and the precision required.
| Total examples | Development percentage | Development examples |
|---|---|---|
| \(1{,}000\) | \(20\%\) | \(200\) |
| \(10{,}000\) | \(10\%\) | \(1{,}000\) |
| \(100{,}000\) | \(5\%\) | \(5{,}000\) |
| \(1{,}000{,}000\) | \(1\%\) | \(10{,}000\) |
| \(10{,}000{,}000\) | \(0.1\%\) | \(10{,}000\) |
As the complete dataset grows, the percentage reserved for evaluation can decrease while the absolute evaluation-set size remains constant or increases.
A small percentage of a very large dataset can still produce a large and reliable evaluation set.
How Large Should the Development Set Be?
The development set should be large enough to rank competing approaches reliably.
Suppose model A has:\[ 5.0\% \text{ development error} \]
and model B has:\[ 5.1\% \text{ development error} \]
If the development set is very small, this difference may reflect random sampling variation rather than a meaningful improvement.
The required size depends on:
- The magnitude of differences being compared
- The baseline error rate
- The number of models being evaluated
- The amount of noise in the metric
- The number of important subgroups
- The cost of choosing the wrong model
If the team only needs to distinguish 5% error from 10% error, a smaller development set may be sufficient. Distinguishing 5.0% from 5.1% requires much more data.
Statistical Precision
For a simple error rate estimated from \(n\) independent examples, the approximate standard error is:\[ \operatorname{SE} \approx \sqrt{ \frac{p(1-p)}{n} } \]
where \(p\) is the observed error rate.
Suppose:\[ p=0.05 \]
and:\[ n=10{,}000 \]
Then:\[ \operatorname{SE} \approx \sqrt{ \frac{0.05(0.95)}{10{,}000} } \approx0.00218 \]
This is approximately \(0.218\) percentage points.
A rough 95% interval is:\[ p\pm1.96\operatorname{SE} \]
which is approximately:\[ 5.0\%\pm0.43\% \]
This illustrates why the required development-set size depends on the resolution needed for decision-making.
Repeated Model Selection
A development set may be consulted hundreds or thousands of times.
Repeatedly choosing models based on the same development data can gradually overfit the development set, even though its examples are never used directly in gradient descent.
A larger development set can reduce this risk, but it does not eliminate it completely.
Other useful practices include:
- Keeping experiment records
- Avoiding decisions based on tiny score differences
- Periodically refreshing the development set
- Confirming major improvements on additional data
- Preserving a separate test set
How Large Should the Test Set Be?
The test set should be large enough to provide a sufficiently confident estimate of final performance.
The required size depends on:
- Desired confidence interval
- Expected error rate
- Importance of the decision
- Number of metrics being reported
- Number of important subgroups
- Frequency of rare events
- Regulatory or safety requirements
A consumer prototype may require less precision than a medical, financial, or safety-critical system.
Test-set size should be determined by the confidence required in the final evaluation.
Evaluating Rare but Important Cases
Overall sample size is not the only consideration.
Suppose an important condition occurs in only:\[ 0.1\% \]
of examples.
A test set containing 10,000 examples would include approximately:\[ 10{,}000\times0.001=10 \]
such examples.
Ten cases may be insufficient for a reliable subgroup estimate, even though the overall test set appears large.
In this situation, the evaluation set may need:
- More examples overall
- Deliberate oversampling of rare cases
- Separate challenge sets
- Per-group reporting
- Weighted metrics
Representative aggregate performance and reliable rare-case performance are separate requirements.
Development and Test Sets Should Share a Distribution
Development and test sets should generally come from the same target distribution.
This ensures that model selection on the development set is aligned with final evaluation on the test set:\[ P_{\text{dev}}(X,Y) \approx P_{\text{test}}(X,Y) \]
If the development set contains high-quality professional images while the test set contains blurry mobile photographs, improvements on the development set may not transfer to the test set.
The training distribution may differ when additional data sources are useful, but development and test sets should represent the application the system is intended to serve.
Is a Test Set Always Necessary?
A system can technically be developed using only:
- A training set
- A development set
This may be acceptable when an unbiased final estimate is not required.
However, the development-set result is no longer an unbiased estimate because many decisions were made using it. It represents performance on data that influenced the development process.
A separate test set is valuable when:
- A reliable final estimate is needed
- Results will be reported externally
- Deployment decisions are consequential
- Regulatory or contractual evidence is required
- Many models and hyperparameters were compared
- The development set may have been overfit
In most production settings, retaining a separate test set is reassuring and advisable.
Naming the Sets Correctly
If a dataset is repeatedly used to:
- Select models
- Tune hyperparameters
- Choose thresholds
- Select checkpoints
- Guide architecture changes
then it functions as a development set, even if it is called a test set.
A genuine test set should remain untouched during routine development and be used for final evaluation.
| How the set is used | Correct role |
|---|---|
| Parameter learning | Training set |
| Model and hyperparameter selection | Development set |
| Final unbiased evaluation | Test set |
Clear terminology helps prevent teams from believing they have an unbiased estimate when they have actually optimized toward the reported data.
Example Splits
The following are illustrative rather than universal rules.
Small dataset
For 1,000 examples:\[ 60\%/20\%/20\% \]
may be reasonable if all three sets are needed.
Medium dataset
For 100,000 examples, a possible split is:
- 95,000 training
- 2,500 development
- 2,500 test
Large dataset
For 1,000,000 examples:
- 980,000 training
- 10,000 development
- 10,000 test
Very large dataset
For 10,000,000 examples:
- 9,980,000 training
- 10,000 development
- 10,000 test
The last split reserves only \(0.2\%\) for evaluation but still provides 20,000 evaluation examples.
The correct numbers depend on the application’s statistical and operational requirements.
A Practical Sizing Process
A useful process is:
- Identify the target production distribution.
- Decide which metrics and subgroups must be measured.
- Determine how precisely models need to be compared.
- Choose a development-set size that supports those comparisons.
- Determine the confidence required for final reporting.
- Choose a test-set size that provides that confidence.
- Allocate the remaining suitable data to training.
- Reevaluate the split if the dataset or application changes.
Common Mistakes
Applying 70/30 automatically
A traditional ratio may waste substantial training data when millions of examples are available.
Making evaluation sets too small
A tiny development set produces unstable rankings and encourages overfitting to noise.
Ignoring rare subgroups
A large overall test set can still contain too few critical examples.
Tuning on the test set
Once the test results guide model changes, that set has effectively become part of development.
Using different target distributions
Development and test sets drawn from different distributions make model selection unreliable.
Confusing percentage with adequacy
A 1% test set can be enormous, while a 20% test set can still be too small.
Key Takeaway
Do not choose development and test sizes from fixed percentages alone. Make the development set large enough to compare candidate systems reliably and the test set large enough to measure final performance with the required confidence. With very large datasets, these goals can often be achieved using only a small percentage of the available data, leaving most examples for training.