Partitioning Observations to Train Objective Models

The objective question

You prepared the data and trained a model. The key question is:

How can you be confident your results are objective—i.e., not an artifact of a particular dataset slice, machine state, or accidental setup?

---

What “Objective Results” Means Here

Objective results mean your model can be:

• applied to different data, in a different environment, at a future time, by someone else,
• and still produce sensible outputs and conclusions that create real impact.

You cannot guarantee success in every possible scenario, but you are responsible for evaluating how well the model is likely to perform under other plausible circumstances.

This evaluation rests on two criteria:

• Reproducibility
• Replicability

---

Reproducibility (ME)

Reproducibility asks whether you can recreate the same results when you repeat the work under the same conditions.

To be reproducible:

1. Start from the original data
2. Use the same algorithms
3. Execute the same tasks
4. Run on the same or compatible machine
5. Reproduce the same results and conclusions

If results cannot be reproduced, it suggests unexplained (intentional or random) interactions among:

• the data,
• the algorithm,
• and the machine/runtime environment.

Common causes include:

1. Uninitialized variables in code
2. Misunderstood documentation or mis-executed steps

---

Replicability (S/HE)

Replicability asks whether someone else (or you in a different setting) can reach similar conclusions using different data in the same business context.

To be replicable:

1. Use different data in the same business context
2. Use the same algorithms
3. Execute the same tasks
4. Run on a compatible machine
5. Reach similar conclusions

If conclusions cannot be replicated, it often indicates design flaws in the task or modeling setup.

Common causes include:

1. Feature correlations not accounted for
2. Degenerate data not handled properly
3. Algorithms that only work in special scenarios (e.g., no missing values)
4. Software issues (e.g., needing a specific hotfix)

---

Common Practice: Partition the Data

A widely used approach is to split the original observations into:

• Training
• Testing
• optionally Hold-Out

Training partition

Used to:

• prepare data,
• determine model structure,
• estimate parameters.

Testing partition

Used to:

• assess model performance,
• obtain an objective estimate of error size,
• compare performance among models.

The testing partition plays the role of:

• “third-party” data, and
• “future” data (data the model did not train on).

Hold-Out partition (optional)

Often used when:

• you must report performance on an external benchmark,
• or you are participating in a data challenge.

---

When Hold-Out Is Not Needed

If you do not need to report performance to a third-party benchmark dataset, you typically do not need a hold-out partition.

Practical guidance

• Large / good data: typical allocation works (train/test).
• Small / noisy data: adding a hold-out set may waste too much data; train/test is usually enough (or you may rely more on cross-validation, though that is outside what you provided here).

---

How to Partition the Data

There are two major approaches:

A) Partition using a pre-defined group

By chronological variables

Example:

• Train a customer sentiment model using data from 2020–2022
• Apply the model to “future” data from 2023

This mimics real deployment where the future arrives after training.

By geographical variables

Example:

• Train a marketing campaign model using the U.S. coastal states
• Apply the model to U.S. inland states

This tests generalization across regions.

---

B) Partition by statistical sampling

Simple Random Sampling (SRS)

Definition:

• If you select $k$ observations from $N$, any sample of size $k$ has the same probability of being selected.
• Sampling is without replacement (once selected, an observation cannot be selected again).

(Reference in your notes: Fan, Muller, Rezucha, 1962.)

Stratified Simple Random Sampling

Key rules:

• Strata must be disjoint groups.
• Each distinct value combination of stratification variables forms a stratum.
• Perform SRS within each stratum.
• Select the same proportion from each stratum.
• Combine stratum subsamples into the final training/testing sets.

---

Python Function for Partitioning

Function:

• sklearn.model_selection.train_test_split(*arrays, **options)

Purpose:

• Allocate rows of a DataFrame into random training/testing subsets.
• Original indices are carried over.

---

Example: Simple Random Sampling (hmeq)

Dataset:

• hmeq.csv has 5,960 observations

Split:

• train_size = 0.7
• test_size = 0.3
• random_state = 60616 (seed for reproducibility)

Resulting sizes:

• Training: $5960 \times 0.70 = 4172$5
• Testing: $5960 \times 0.30 = 1788$

Issue shown in the notes:
When BAD is the label variable, the label distribution differs across partitions.

Training distribution (BAD):

• 0: 0.7955
• 1: 0.2045

Testing distribution (BAD):

• 0: 0.8121
• 1: 0.1879

Why this matters:

• The model may implicitly favor the class proportions it sees in training.
• This can degrade performance on test data if distributions are not aligned.

---

Example: Stratified Random Sampling (hmeq)

To fix label distribution mismatch, stratify on BAD:

• stratify = hmeq['BAD']
• train_size = 0.7
• random_state = 60616

Resulting distributions:

Training:

• 0: 0.8006
• 1: 0.1994

Testing:

• 0: 0.8003
• 1: 0.1997

So both partitions preserve essentially the same label distribution, matching the original dataset.

---

Stratified Sampling: The Formula Logic (as given)

Suppose:

• total observations: $n$
• $n_0$ observations with BAD = 0
• $n_1$​ observations with BAD = 1
• sampling proportion: $0 < p < 1$

Then:

Training partition:

• BAD = 0: $n_0 p$
• BAD = 1: $n_1 p$

Testing partition:

• BAD = 0: $n_0 (1-p)$
• BAD = 1: $n_1 (1-p)$

Proportion of BAD=0 in training:

$$\frac{n_0 p}{n_0 p + n_1 p} = \frac{n_0}{n_0+n_1}$$

Proportion of BAD=0 in testing:

$$\frac{n_0 (1-p)}{n_0 (1-p) + n_1 (1-p)} = \frac{n_0}{n_0+n_1}$$

Same for BAD=1. Therefore, stratified sampling maintains the label distribution across both partitions, enabling fairer evaluation.

---

Simple Random vs Stratified: Which One to Use?

This depends on the measurement level of the label variable:

If the label is categorical

Recommendation:

• Stratified random sampling
• Strata defined by label categories
• Label distribution maintained across partitions
• Caution: watch out for rare label categories (they can be hard to preserve properly)

If the label is continuous

Recommendation:

• Simple random sampling
• After splitting, check whether the target distributions in the partitions are similar