Low-pass Filtering

A low-pass filter lets low-frequency components through and attenuates high-frequency ones. In the frequency domain the ideal version is a step:

\[ H(f) = \begin{cases} 1 & |f| \leq f_c \\ 0 & |f| > f_c \end{cases} \]

where \(f_c\) is the cutoff. In the time domain the same operation is a convolution with a smoothing kernel — a moving average, a Gaussian window — which is why smoothing a time series and low-pass filtering it are the same act described in two vocabularies.

The usual families are the ideal filter (theoretical, a perfectly sharp cutoff), Butterworth (smooth roll-off, flat passband), Chebyshev (sharper cutoff at the price of passband ripple), digital FIR and IIR filters, and the moving average, which is the crudest form. Applications run from removing hiss in audio and blurring images to cleaning ECG and EEG traces, filtering unwanted radio frequencies, and extracting a long-run trend from a noisy series — the stock-price case, where the short-term fluctuations are smoothed away and the trend is left.

Two properties of that last case decide whether the result is usable, and neither is visible in the definition.

What a moving average actually does to a spectrum

import numpy as np
from scipy.signal import butter, filtfilt, lfilter, freqz

fs = 500.0
t = np.arange(0, 4, 1 / fs)
clean = np.sin(2 * np.pi * 5 * t)
signal = clean + 0.5 * np.sin(2 * np.pi * 50 * t)
"""A 5 Hz signal buried under a 50 Hz one, sampled at 500 Hz. The job of a
   low-pass filter is to keep the first and remove the second."""

def gain(b, a, f):
    w, hh = freqz(b, a, worN=np.atleast_1d(2 * np.pi * f / fs))
    return np.abs(hh)

print(f"{'frequency (Hz)':>16}{'moving avg, 10':>16}{'moving avg, 25':>16}"
      f"{'Butterworth, 4th':>19}")
bb, ab = butter(4, 20 / (fs / 2))
for f in (5, 20, 50, 60, 75, 100, 125):
    g10 = gain(np.ones(10) / 10, [1.0], f)[0]
    g25 = gain(np.ones(25) / 25, [1.0], f)[0]
    gb = gain(bb, ab, f)[0]
    print(f"{f:>16}{g10:>16.4f}{g25:>16.4f}{gb:>19.4f}")

k = 20
w = np.ones(k) / k
ww, hh = freqz(w, [1.0], worN=4096, fs=fs)
sign = np.real(hh * np.exp(1j * ww * (k - 1) * np.pi / fs))
print(f"{'frequency (Hz)':>16}{'gain':>10}{'sign after the delay':>23}")
for f in (5, 20, 30, 40, 60, 90):
    i = np.abs(ww - f).argmin()
    print(f"{ww[i]:>16.1f}{np.abs(hh[i]):>10.4f}"
          f"{('kept' if sign[i] >= 0 else 'inverted'):>23}")

from scipy.signal import group_delay

bs, a_s = butter(4, 8 / (fs / 2))
w5 = np.array([2 * np.pi * 5 / fs])
for name, (bn, an) in (("moving average, 20", (np.ones(20) / 20, [1.0])),
                       ("moving average, 60", (np.ones(60) / 60, [1.0])),
                       ("Butterworth, 20 Hz", (bb, ab)),
                       ("Butterworth, 8 Hz", (bs, a_s))):
    d = float(group_delay((bn, an), w=w5)[1][0])
    print(f"{name:>22}{d:>10.2f} samples{d / fs * 1000:>10.1f} ms")
print(f"{'filtfilt (two-pass)':>22}{0.0:>10.2f} samples{0.0:>10.1f} ms   by construction")
#   frequency (Hz)  moving avg, 10  moving avg, 25   Butterworth, 4th
#                5          0.9838          0.9005             1.0000
#               20          0.7588          0.0000             0.7071
#               50          0.0000          0.1294             0.0228
#               60          0.1597          0.0000             0.0104
#               75          0.2203          0.0623             0.0038
#              100          0.0000          0.0000             0.0009
#              125          0.1414          0.0400             0.0003
#   frequency (Hz)      gain   sign after the delay
#              5.0    0.9355                   kept
#             20.0    0.2335                   kept
#             30.0    0.1575               inverted
#             40.0    0.1915               inverted
#             60.0    0.1292                   kept
#             90.0    0.0886               inverted
#     moving average, 20      9.50 samples      19.0 ms
#     moving average, 60     29.50 samples      59.0 ms
#     Butterworth, 20 Hz     10.64 samples      21.3 ms
#      Butterworth, 8 Hz     32.63 samples      65.3 ms
#    filtfilt (two-pass)      0.00 samples       0.0 ms   by construction

The two properties

The first is that a moving average is not a gentle approximation to the ideal filter. Its frequency response is a sinc, which crosses zero and comes back, so its stopband is a set of ripples rather than a floor. The 10-point average removes 50 Hz completely and then passes 75 Hz at 0.2203 — a frequency further from the signal survives better than one closer to it. Worse, alternate lobes are negative: components in them arrive inverted, and an inverted component actively pulls the smoothed line the wrong way rather than merely adding noise to it. A Butterworth filter of modest order falls monotonically and does not do either thing.

The second is lag. The causal smoothing filters examined here use only past and present samples and delay components in the measured band, and the delay grows with the amount of smoothing: 9.5 samples for a 20-point average, 29.5 for a 60-point one, 10.64 and 32.63 samples for Butterworth filters cutting at 20 and 8 Hz. This is not a defect to be engineered away. Smoothing means combining a value with its neighbors, and a causal filter has neighbors on one side only.

Which is what makes filtfilt worth understanding rather than simply using. It runs the filter forwards and then backwards, and the two delays cancel to exactly zero. The price is that the output at time \(t\) depends on samples after \(t\). For plotting a completed history that is fine and usually correct. For a trend line that feeds a forecast, a trading rule or an alert, it is leakage: the smoothed value at each point was computed using the future, so any backtest built on it will look better than the system can be. The honest comparison for anything that runs forward in time is a causal filter, with its lag stated.

Choosing one

The cutoff encodes a decision about which timescales are signal and which are noise, so choosing the objective requires domain knowledge. Once that objective is defined, spectral analysis and suitable validation data can help tune the cutoff; data alone do not define what the application should preserve: a weekly cycle is noise to a capacity planner and the entire signal to someone staffing weekends.

After that, the choices are ordinary. Use a designed filter rather than a moving average where the stopband matters, since the sinc ripples above are avoidable for a few more lines of code. Prefer a higher order when the cutoff has to be sharp, and accept more delay and more ringing near edges. Use filtfilt for presentation and analysis of completed data, and a one-pass filter anywhere a decision is made in real time. And when a filtered series feeds a model, filter inside the training fold rather than across the whole dataset — learning filter parameters from future data or using future samples leaks information. A fixed causal filter run forward over the full timeline does not inherently leak; preserve its state across the split as deployment would. Even filtfilt can be recomputed on the observed history at each forecast origin, provided no later observations enter that origin’s features.

References

• SciPy documentation, scipy.signal.butter, scipy.signal.filtfilt, scipy.signal.freqz and scipy.signal.group_delay. docs.scipy.org