Windows (in Time-Series)

A window is the slice of history a statistic is computed over. Fixed windows cover a named period, rolling windows move forward one step at a time, expanding windows grow from a start point. Many comparative monitoring alerts use two of them: a baseline window standing for normal, and a current window standing for now.

The usual worked example runs: baseline latency over eight weeks is 200 ms, the last 24 hours is 350 ms, therefore something is wrong. Three separate decisions are missing from that sentence, and each one is a real production failure.

How long the window is

Bifet and Gavaldà describe the trade-off this way: a window that is too large makes the estimate slow to react when the data changes, and a window that is too small leaves too few readings to average out noise, so the estimate is unreliable.

In the simulation each reading is normal with mean 200 ms and standard deviation 30 ms, and the mean steps up by 18 ms at reading 200, which is 0.6 of one reading’s standard deviation. For each window length, run raises the alarm when the rolling mean crosses a gate three standard errors above 200 ms, mu + 3 * sd / sqrt(window), and repeats this on 3,000 series. It reports the mean delay after the step, the share of series that alarm before the step (false alarms), and the share in which the step is never caught.

import numpy as np

rng = np.random.default_rng(66)
mu, sd, jump = 200.0, 30.0, 18.0
horizon, change_at, reps = 400, 200, 3000


def run(window, k):
    gate = mu + k * sd / np.sqrt(window)
    delays, alarms, missed = [], 0, 0
    for _ in range(reps):
        series = mu + sd * rng.normal(size=horizon)
        series[change_at:] += jump
        roll = np.convolve(series, np.ones(window) / window, mode="valid")
        ready = np.arange(len(roll)) + window - 1
        over = roll > gate
        alarms += bool(np.any(over & (ready < change_at)))
        after = np.flatnonzero(over & (ready >= change_at))
        if len(after):
            delays.append(ready[after[0]] - change_at + 1)
        else:
            missed += 1
    return np.mean(delays), alarms / reps, missed / reps, gate


print("window      gate  mean delay  false alarm rate  never caught")
for window in (3, 7, 14, 28, 56):
    delay, false_rate, miss, gate = run(window, 3.0)
    print("%6d %9.1f %9.1f readings %8.1f%% %12.1f%%"
          % (window, gate, delay, false_rate * 100, miss * 100))

window      gate  mean delay  false alarm rate  never caught
     3     252.0      50.8 readings     21.9%          2.6%
     7     234.0      37.3 readings     14.4%          0.3%
    14     224.1      30.3 readings     10.3%          0.0%
    28     217.0      29.1 readings      6.2%          0.0%
    56     212.0      36.6 readings      3.0%          0.0%

The false-alarm column falls the whole way down, so quiet always argues for longer. The delay column turns around: 50.8 readings at a window of three, down to 29.1 at twenty-eight, back up to 36.6 at fifty-six. In this simulation, a short window has a higher threshold and waits until the rolling mean rises enough to cross it, and a long window is slow because it has to fill up first. There is a balance point, it is a property of the shift you are looking for, and nothing about eight weeks is special.

Whether the baseline moves

A rolling baseline is the default because it adapts. What it adapts to includes the incident. In the simulation, latency is 200 ms with a 25 ms standard deviation, steps to 240 ms on day 40 and never comes back. On day t the current window is the last 7 days, series[t - 7:t], and the rolling baseline is the 56 days before it, series[t - 63:t - 7]. The pinned baseline is the mean of the 40 days before the change. Both alarms fire when the current mean exceeds the baseline by three standard errors.

rng = np.random.default_rng(1001)
days, change_at = 140, 40
mu, sd, jump = 200.0, 25.0, 40.0
baseline_len, current_len = 56, 7

series = mu + sd * rng.normal(size=days)
series[change_at:] += jump
pinned = series[:change_at].mean()
gate_roll = 3 * sd * np.sqrt(1 / current_len + 1 / baseline_len)
gate_pin = 3 * sd * np.sqrt(1 / current_len + 1 / change_at)

roll_state, shown = {}, {}
for t in range(baseline_len + current_len, days):
    base = series[t - baseline_len - current_len:t - current_len].mean()
    cur = series[t - current_len:t].mean()
    roll_state[t] = cur - base > gate_roll
    shown[t] = (base, cur, cur - pinned > gate_pin)

print("   day  rolling base  current  rolling says  pinned says")
for t in (63, 65, 67, 69, 75, 90, 110, 139):
    base, cur, pin_fires = shown[t]
    print("%6d %13.1f %8.1f %13s %12s"
          % (t, base, cur, "FIRING" if roll_state[t] else "quiet",
             "FIRING" if pin_fires else "quiet"))

on = [t for t, v in roll_state.items() if v]
first_full = change_at + baseline_len + current_len
print("rolling alarm: %d of %d days firing, first day %d, last day %d"
      % (len(on), len(roll_state), min(on), max(on)))
print("pinned alarm:  %d of %d days firing" % (sum(v[2] for v in shown.values()), len(shown)))
print("first day with an all-post-change baseline slice: %d" % first_full)

   day  rolling base  current  rolling says  pinned says
    63         214.4    243.7         quiet       FIRING
    65         215.8    246.0        FIRING       FIRING
    67         216.6    247.0        FIRING       FIRING
    69         219.6    244.3         quiet       FIRING
    75         223.3    238.9         quiet       FIRING
    90         232.0    253.5         quiet       FIRING
   110         243.0    258.8         quiet       FIRING
   139         241.5    247.9         quiet       FIRING
rolling alarm: 3 of 77 days firing, first day 65, last day 68
pinned alarm:  68 of 77 days firing
first day with an all-post-change baseline slice: 103

The rolling alarm fires on three of the 77 evaluated days, all between day 65 and day 68, and is silent on the other seventy-four, while the latency stays 40 ms high for every one of them. The pinned baseline fires on 68 of the same 77 days. The baseline walked into the incident. Its slice series[t - 63:t - 7] still contains pre-change days until day 102 and holds only post-change days from day 103, which is 63 days after the change at day 40. The slice starts 63 days before day t (the 56 baseline days plus the 7 days of the current window that come after them), so it is all post-change only once that start reaches day 40. Well before that the gap between the two windows has shrunk below the gate, and once the eight-week window is full of post-incident days the comparison is the new normal against the new normal. Anyone who opens the dashboard next month sees a short spike that resolved itself.

The two designs answer different questions. A rolling baseline asks has this changed recently. A pinned baseline can detect a persistent departure from its reference. Absolute SLOs, domain bounds and suitable seasonal expected-value models can also test ongoing health; a pinned historical window is not the only way.

Whether the readings inside it are comparable

A 24-hour current window against an eight-week baseline compares one day to an average of fifty-six. That is only a fair comparison if the fifty-six days are the same kind of day. In the simulation the weekday effect runs from −46 ms to +22 ms and one day’s noise is 12 ms. Each day is compared with the mean of the 56 days before it, and then with only the days of the same weekday inside that window.

rng = np.random.default_rng(77)
names = ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"]
weekday = np.array([18.0, 16.0, 15.0, 17.0, 22.0, -42.0, -46.0])
mu, sd, days, base_len = 200.0, 12.0, 700, 56

t = np.arange(days)
series = mu + weekday[t % 7] + sd * rng.normal(size=days)

flat_hits = np.zeros(7)
like_hits = np.zeros(7)
counts = np.zeros(7)
for i in range(base_len, days):
    d = i % 7
    counts[d] += 1
    window = series[i - base_len:i]
    flat_hits[d] += abs(series[i] - window.mean()) > 3 * sd
    same = window[(np.arange(i - base_len, i) % 7) == d]
    like_hits[d] += abs(series[i] - same.mean()) > 3 * sd * np.sqrt(1 + 1 / len(same))

flat, like = flat_hits / counts * 100, like_hits / counts * 100
print("   day  effect  alarms vs 56-day mean  alarms vs same weekday")
for d in range(7):
    print("%6s %7.0f %21.1f%% %22.1f%%" % (names[d], weekday[d], flat[d], like[d]))
print("%6s %7s %21.1f%% %22.1f%%"
      % ("all", "", flat_hits.sum() / counts.sum() * 100, like_hits.sum() / counts.sum() * 100))

   day  effect  alarms vs 56-day mean  alarms vs same weekday
   Mon      18                   8.7%                    1.1%
   Tue      16                   4.3%                    0.0%
   Wed      15                   6.5%                    0.0%
   Thu      17                   3.3%                    0.0%
   Fri      22                  12.0%                    0.0%
   Sat     -42                  68.5%                    0.0%
   Sun     -46                  78.3%                    1.1%
   all                          25.9%                    0.3%

Nothing is broken on any day in that series. The rolling mean averages five busy days with two quiet ones and lands between the two regimes, so it fires on 68.5% of Saturdays and 78.3% of Sundays, and on 25.9% of all days. Compared like for like, against the same weekday inside the same window, the rate drops to 0.3%.

A window is not only a length. It is also a claim that what is inside it is comparable, and an eight-week mean quietly asserts that a Sunday and a Friday are the same thing. This simulation shows one source of false alerts in drift and latency monitoring.

So the worked example is missing three decisions: how long the windows are, whether the baseline is pinned or rolling, and what makes the days inside it comparable. 350 against 200 means nothing without them. And when the baseline period itself was abnormal, no window length repairs it — that is what a macro shift does to a monitoring system.

References

• Bifet, A., & Gavaldà, R. (2007). Learning from time-changing data with adaptive windowing. Proceedings of the 2007 SIAM International Conference on Data Mining, 443–448.
• Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., & Bouchachia, A. (2014). A survey on concept drift adaptation. ACM Computing Surveys, 46(4), 1–37.