Oversmoothing and Undersmoothing

Oversmoothing is estimating with so much averaging that real structure disappears: a histogram with bins wide enough to flatten two distinct peaks into one hump, a rolling mean long enough to erase the spike it was meant to reveal. Undersmoothing is the opposite — so little averaging that sampling noise is rendered as though it were signal, producing peaks that would not reappear if you collected the data again.

They are the two ends of a single dial: bin width in a histogram, bandwidth in a kernel estimate, window length in a moving average. Turning it one way reduces the noise that survives into the picture while increasing the systematic distortion; turning it the other does the reverse. No setting eliminates both, which is why “the right amount of smoothing” is always relative to how much data there is and how fast the underlying quantity changes.

The practical difficulty is that both failures look plausible. An oversmoothed curve looks clean and authoritative, and nothing on it announces the structure it removed. An undersmoothed one looks detailed and rich, and its spurious peaks invite explanation — which is the more dangerous of the two, because a reader will generally supply a story for any bump that is drawn.

Two checks help. Re-estimate at several settings and see which features persist: a peak that appears at one width and vanishes at neighboring ones is sensitive to the setting, so it needs further evidence before being treated as robust structure. And split the data — a structure present in one half and absent in the other is not yet a finding.

Data-driven selection rules exist precisely to avoid choosing by eye, since choosing by eye means choosing the picture that matches what you expected to see.

See it in use in Choosing Histogram Bins for Time-Indexed Events.

References

• numpy.histogram_bin_edges