Standardization, Z-Scores, and the Standard Normal Distribution

A normal distribution is completely determined by two parameters:

• its mean, \(\mu\);
• its standard deviation, \(\sigma\).

If:\[ X\sim N(\mu,\sigma^2), \]

then knowing \(\mu\) and \(\sigma\) determines the complete theoretical probability distribution.

This does not mean that two observed datasets with the same mean and standard deviation must have identical histograms. It means that within the family of normal distributions, those two parameters uniquely determine the curve.

Standardization converts observations from any normal distribution to a common reference scale called the standard normal distribution.

The Z-Score Formula

For an observation \(x\) from a population with mean \(\mu\) and standard deviation \(\sigma\), its standardized value is:\[ \boxed{ z=\frac{x-\mu}{\sigma} } \]

The standardized value \(z\) is called a z-score.

For sample data, the corresponding calculation is often:\[ \boxed{ z=\frac{x-\bar{x}}{s} } \]

where:

• \(\bar{x}\) is the sample mean;
• \(s\) is the sample standard deviation.

The calculation has two steps:

1. subtract the mean;
2. divide by the standard deviation.

Each step serves a different purpose.

Why Subtract the Mean?

The quantity:\[ x-\mu \]

is the observation’s deviation from the mean.

If:\[ x>\mu, \]

then:\[ x-\mu>0, \]

so the observation is above average.

If:\[ x<\mu, \]

then:\[ x-\mu<0, \]

so the observation is below average.

If:\[ x=\mu, \]

then:\[ x-\mu=0. \]

Subtracting the mean recenters the distribution around zero.

Why Divide by the Standard Deviation?

The deviation \(x-\mu\) is still expressed in the original measurement units.

For example, a height deviation might be:\[ 3.6\text{ inches}. \]

Dividing by \(\sigma\) expresses this difference in standard-deviation units:\[ \frac{3.6\text{ inches}}{1.8\text{ inches}} = 2. \]

Thus, the observation lies two standard deviations above the mean.

Division by the standard deviation also puts variables with different units and spreads onto a comparable scale.

Interpreting Z-Scores

A z-score tells how many standard deviations an observation lies above or below the mean.

The sign indicates direction:

• positive: above the mean;
• negative: below the mean;
• zero: equal to the mean.

The magnitude:\[ |z| \]

measures distance from the mean in standard-deviation units.

Why Z-Scores Have No Units

Suppose height is measured in inches.

Then:\[ x-\mu \]

has units of inches, and:\[ \sigma \]

also has units of inches.

Therefore:\[ z = \frac{\text{inches}}{\text{inches}} = 1. \]

The units cancel.

A z-score is dimensionless. This makes it possible to compare standardized values from variables measured in different units.

Height Example

Suppose father heights are approximately normal with:\[ \mu=68.3\text{ inches} \]

and:\[ \sigma=1.8\text{ inches}. \]

Consider a height of:\[ x=71.9\text{ inches}. \]

The z-score is:\[ z = \frac{71.9-68.3}{1.8} = \frac{3.6}{1.8} = 2. \]

Therefore, 71.9 inches is two standard deviations above the mean.

A Below-Average Height

Consider:\[ x=65.6\text{ inches}. \]

Then:\[ z = \frac{65.6-68.3}{1.8} = \frac{-2.7}{1.8} = -1.5. \]

Thus, 65.6 inches is 1.5 standard deviations below the mean.

Converting Back to the Original Scale

Starting with:\[ z=\frac{x-\mu}{\sigma}, \]

multiply by \(\sigma\):\[ z\sigma=x-\mu. \]

Then add \(\mu\):\[ \boxed{ x=\mu+z\sigma } \]

This converts a z-score back to the original measurement scale.

For:\[ z=-1.5, \]

the corresponding height is:\[ x = 68.3+(-1.5)(1.8). \]

Therefore:\[ x = 68.3-2.7 = 65.6\text{ inches}. \]

Standardizing a Random Variable

Let \(X\) have mean \(\mu\) and standard deviation \(\sigma>0\). Define:\[ Z=\frac{X-\mu}{\sigma}. \]

Then:\[ E[Z] = E\left[ \frac{X-\mu}{\sigma} \right]. \]

Using linearity of expectation:\[ E[Z] = \frac{E[X]-\mu}{\sigma} = \frac{\mu-\mu}{\sigma} = 0. \]

Therefore:\[ \boxed{ E[Z]=0 } \]

The standardized variable has mean zero.

Standard Deviation After Standardization

Variance obeys:\[ \operatorname{Var}(aX+b) = a^2\operatorname{Var}(X). \]

Therefore:\[ \operatorname{Var}(Z) = \operatorname{Var} \left( \frac{X-\mu}{\sigma} \right) = \frac{1}{\sigma^2} \operatorname{Var}(X). \]

Since:\[ \operatorname{Var}(X)=\sigma^2, \]

we obtain:\[ \operatorname{Var}(Z)=1. \]

Thus:\[ \boxed{ \operatorname{SD}(Z)=1 } \]

Standardization transforms the mean to 0 and the standard deviation to 1.

A Detail About Sample Standardization

For observed sample values:\[ z_i=\frac{x_i-\bar{x}}{s}, \]

the standardized values have sample mean:\[ \bar{z}=0. \]

If \(s\) is calculated using the \(n-1\) denominator, the sample standard deviation of the \(z_i\) values—also calculated using \(n-1\)—is 1.

The variance convention must remain consistent. Mixing formulas with \(n\) and \(n-1\) can produce a value close to, but not exactly, 1.

The Standard Normal Distribution

If:\[ X\sim N(\mu,\sigma^2) \]

and:\[ Z=\frac{X-\mu}{\sigma}, \]

then:\[ \boxed{ Z\sim N(0,1) } \]

This is the standard normal distribution.

Its mean is:\[ \mu_Z=0 \]

and its standard deviation is:\[ \sigma_Z=1. \]

Every normal distribution can be converted to this common reference distribution.

Standard Normal Density

The probability density function of the standard normal distribution is:\[ \boxed{ \phi(z) = \frac{1}{\sqrt{2\pi}} e^{-z^2/2} } \]

The graph is symmetric around zero:

Density
   |
   |                        /\
   |                      /    \
   |                    /        \
   |                 __/          \__
   |______________ /                  \______________
                 -3 -2 -1  0  1  2  3
                           z

The density formula is usually evaluated with software, a calculator, or a statistical table rather than by direct integration.

Density Versus Cumulative Probability

The density:\[ \phi(z) \]

is the height of the standard normal curve at \(z\). It is not the probability of observing exactly \(z\).

Because the normal distribution is continuous:\[ P(Z=z)=0. \]

Probabilities correspond to areas over intervals.

The standard normal cumulative distribution function is:\[ \boxed{ \Phi(z)=P(Z\leq z) } \]

or equivalently:\[ \Phi(z) = \int_{-\infty}^{z} \phi(t)\,dt. \]

The function \(\Phi(z)\) gives the area under the standard normal curve to the left of \(z\).

Computing a Probability by Standardizing

Suppose:\[ X\sim N(\mu,\sigma^2). \]

To calculate:\[ P(X\leq x), \]

standardize \(x\):\[ z=\frac{x-\mu}{\sigma}. \]

Then:\[ \boxed{ P(X\leq x) = P\left( Z\leq \frac{x-\mu}{\sigma} \right) = \Phi\left( \frac{x-\mu}{\sigma} \right) } \]

This is why standardization is useful: probabilities for every normal distribution can be found using the same standard normal curve.

Height Probability Example

Using:\[ X\sim N(68.3,1.8^2), \]

calculate the probability that a height is at most 71.9 inches.

Standardize:\[ z = \frac{71.9-68.3}{1.8} = 2. \]

Therefore:\[ P(X\leq71.9) = P(Z\leq2) = \Phi(2). \]

From the standard normal distribution:\[ \Phi(2)\approx0.9772. \]

Thus:\[ \boxed{ P(X\leq71.9)\approx97.72\% } \]

The probability of exceeding 71.9 inches is:\[ P(X>71.9) = 1-0.9772 = 0.0228. \]

Therefore:\[ \boxed{ P(X>71.9)\approx2.28\% } \]

The empirical rule rounds this upper-tail probability to approximately 2.5%.

Probability Between Two Values

To calculate:\[ P(a\leq X\leq b), \]

standardize both endpoints:\[ z_a=\frac{a-\mu}{\sigma} \]

and:\[ z_b=\frac{b-\mu}{\sigma}. \]

Then:\[ \boxed{ P(a\leq X\leq b) = \Phi(z_b)-\Phi(z_a) } \]

For the interval from 64.7 to 71.9 inches:\[ z_a = \frac{64.7-68.3}{1.8} = -2 \]

and:\[ z_b = \frac{71.9-68.3}{1.8} = 2. \]

Therefore:\[ P(64.7\leq X\leq71.9) = \Phi(2)-\Phi(-2). \]

Using:\[ \Phi(2)\approx0.9772 \]

and:\[ \Phi(-2)\approx0.0228, \]

we obtain:\[ 0.9772-0.0228=0.9544. \]

Thus:\[ \boxed{ P(64.7\leq X\leq71.9)\approx95.44\% } \]

This is the precise normal-curve version of the empirical rule’s approximate 95%.

Symmetry of the Standard Normal Curve

Because the standard normal curve is symmetric:\[ \phi(-z)=\phi(z). \]

For cumulative probabilities:\[ \boxed{ \Phi(-z)=1-\Phi(z) } \]

For example:\[ \Phi(-2) = 1-\Phi(2) = 1-0.9772 = 0.0228. \]

Likewise:\[ P(Z>z) = P(Z<-z). \]

This symmetry simplifies many calculations.

Finding a Value from a Percentile

Sometimes the probability is known and the original value is required.

Suppose we want the 90th percentile of:\[ X\sim N(\mu,\sigma^2). \]

Find \(z_{0.90}\) such that:\[ \Phi(z_{0.90})=0.90. \]

For the standard normal distribution:\[ z_{0.90}\approx1.2816. \]

Convert to the original scale:\[ x_{0.90} = \mu+z_{0.90}\sigma. \]

For the height example:\[ x_{0.90} = 68.3+(1.2816)(1.8). \]

Therefore:\[ x_{0.90} \approx70.61\text{ inches}. \]

Under the normal model, approximately 90% of the population lies at or below 70.61 inches.

Useful Standard Normal Percentiles

The values \(-1.96\) and \(1.96\) contain approximately 95% of the standard normal distribution:\[ P(-1.96\leq Z\leq1.96)\approx0.95. \]

Comparing Values from Different Scales

Z-scores allow relative positions to be compared across variables with different units.

Suppose a student receives:

• 85 on an exam with mean 70 and standard deviation 10;
• 620 on another test with mean 500 and standard deviation 80.

For the first test:\[ z_1 = \frac{85-70}{10} = 1.5. \]

For the second:\[ z_2 = \frac{620-500}{80} = 1.5. \]

Although the raw scores differ, both are 1.5 standard deviations above their respective means.

This does not necessarily mean the performances are substantively equivalent. It means they occupy the same standardized position under their respective distributions.

Standardization and Linear Transformations

Suppose a variable is converted from one unit to another:\[ Y=a+bX \]

with \(b>0\).

Then:\[ \mu_Y=a+b\mu_X \]

and:\[ \sigma_Y=b\sigma_X. \]

Its z-score is:\[ \frac{Y-\mu_Y}{\sigma_Y} = \frac{ a+bX-(a+b\mu_X) }{ b\sigma_X } = \frac{X-\mu_X}{\sigma_X}. \]

Therefore, z-scores are unchanged by positive linear unit conversions.

A height has the same z-score whether measured in inches or centimeters.

Standardization Does Not Change Distribution Shape

Subtracting the mean and dividing by the standard deviation changes:

• location;
• scale;
• units.

It does not change:

• skewness;
• number of peaks;
• relative ordering;
• tail structure;
• outlier positions in standard-deviation units.

If the original data are right-skewed, the standardized data remain right-skewed.

If \(X\) is normal, then \(Z\) is standard normal. If \(X\) is not normal, \(Z\) has mean 0 and standard deviation 1 but is not necessarily normally distributed.

Standardized does not mean normally distributed.

Z-Scores and Unusual Observations

Under an approximately normal model:

• \(|z|<1\) is common;
• \(|z|>2\) is relatively unusual;
• \(|z|>3\) is rare.

However, a large absolute z-score does not prove that an observation is erroneous.

It may indicate:

• a valid extreme value;
• a different subgroup;
• measurement error;
• a heavy-tailed distribution;
• an inappropriate normal model;
• a genuine anomaly.

The context and distribution shape must be examined.

Population and Sample Z-Scores

It is important to distinguish two settings.

Population parameters known

Use:\[ z=\frac{x-\mu}{\sigma}. \]

This gives position relative to the population distribution.

Parameters estimated from a sample

Use:\[ z=\frac{x-\bar{x}}{s}. \]

This gives position relative to the observed sample.

The second value contains uncertainty because \(\bar{x}\) and \(s\) vary between samples.

In inferential procedures involving an estimated standard deviation, the resulting statistic may follow a \(t\)-distribution rather than a standard normal distribution.

Z-Scores Are Not Always Directly Comparable

Z-scores compare relative position only when the reference distributions are meaningful.

Potential problems include:

• strongly different distribution shapes;
• severe skewness;
• heavy tails;
• incomparable populations;
• restricted ranges;
• unreliable standard deviations;
• ordinal measurements treated as interval scales.

For example, a z-score of 2 in a heavy-tailed distribution may be much less unusual than a z-score of 2 under a normal distribution.

Common Mistakes

Saying the mean and standard deviation determine every histogram

They determine a normal distribution, not an arbitrary observed distribution.

Forgetting parentheses

The correct formula is:\[ z=\frac{x-\mu}{\sigma}, \]

not:\[ x-\frac{\mu}{\sigma}. \]

Interpreting a negative z-score as a negative measurement

A negative z-score means below the mean. The original value can still be positive.

Attaching original units to a z-score

Z-scores have no measurement units.

Assuming standardization makes data normal

It changes center and scale, not shape.

Confusing density with cumulative probability

\[ \phi(z) \]

is curve height, while:\[ \Phi(z) \]

is area to the left.

Using sample statistics as though they were known population parameters

Estimated means and standard deviations introduce additional uncertainty.

Treating large z-scores as automatic errors

They should be investigated, not automatically removed.

A Reliable Standardization Process

1. Identify the observation \(x\).
2. Identify the appropriate reference population.
3. Determine the relevant mean.
4. Determine the relevant standard deviation.
5. subtract the mean.
6. Divide by the standard deviation.
7. Interpret the sign.
8. Interpret the magnitude.
9. Use the standard normal curve only if normality is justified.
10. Convert back using \(x=\mu+z\sigma\) when necessary.

Key Takeaway

A z-score standardizes an observation by subtracting the mean and dividing by the standard deviation:\[ z=\frac{x-\mu}{\sigma}. \]

It has no units and indicates how many standard deviations an observation lies above or below the mean. If \(X\sim N(\mu,\sigma^2)\), then its standardized form follows \(N(0,1)\), allowing probabilities for any normal distribution to be calculated from one common reference curve. Standardization changes location and scale, but it does not make a non-normal distribution normal.