Serial correlation: daily rows are not daily independent evidence

Last updated: 7 Oct 20265 min read

tutorial

Positive correlation across adjacent observations reduces information in a time series relative to independent rows.

Define the regular observation clock

A returns team records a daily fraction of late refunds. Yesterday’s backlog affects today’s processing, so adjacent rates may move together. The day is an observation, but it is not automatically an independent replicate. Record reporting lag, missing days, weekday schedule, policy changes and denominator before estimating uncertainty. A series with trend or seasonality can show large apparent lag correlation even when its remaining errors follow a different pattern. The interrupted-series lesson separates calendar structure from residual dependence.

Use an approximation with a narrow scope

For a stationary AR(1)-like process with positive lag-one correlation rho, the rough effective sample size of a mean is n times one minus rho divided by one plus rho. Sixty daily values with rho 0.6 carry information resembling about fifteen independent values under that approximation. It is a diagnostic, not a replacement for an interval method. Negative or long-memory dependence, a moving mean, missing calendar days and changing variance defeat the simple formula. Interval coverage needs assumptions matching the actual series.

Inspect the residual sequence

Fit planned mean and calendar effects first, then inspect residuals by date and lag. A random-looking marginal histogram can coexist with long positive runs. If an intervention started midway, check pre-period fit and transition lag rather than estimating one correlation across the level change. A conventional standard error that divides by the square root of every daily row can be much too small. A block bootstrap retains short runs of dependence when its stationarity conditions are defensible.

State what can be inferred

Report the outcome definition, window, number of observed days, estimated lag structure and sensitivity to plausible dependence lengths. A change in daily average is an association unless assignment or a credible control supports causality. Do not turn a rough effective count into an exact p-value. The project catches a policy launch where delayed refunds and a changed reporting feed both affect the apparent level.

Implementation

def ar1_effective_days(observed_days, lag_one_correlation):
    if observed_days < 2 or not 0 <= lag_one_correlation < 1:
        raise ValueError("at least two days and nonnegative AR(1) correlation required")
    return observed_days * (1 - lag_one_correlation) / (
        1 + lag_one_correlation)

assert ar1_effective_days(60, 0.6) == 15
assert ar1_effective_days(60, 0) == 60

Performance and operating cost

The approximation is O(1). Estimating correlation, testing residual structure and fitting a time-aware interval need O(n) or more work. Treating sixty correlated days as sixty independent assignments costs less computation but can make a weak change appear precise.

Common Mistakes

Using daily row count as independent sample size without checking dependence.
Applying the AR(1) approximation to trend or seasonal data as if exact.
Estimating correlation across an intervention step and calling it residual correlation.
Claiming causality from a before/after series without a credible design.