Time Series Forecasting (ARIMA)

Standard machine learning assumes observations are Independent and Identically Distributed (i.i.d.): shuffling dataset rows preserves the underlying joint probability distribution. In time series analysis, this assumption completely fails. Temporal sequences possess strong autocorrelation: the value observed at week $t$ is physically and causally bound to values at week $t-1, t-2, \dots$. Consider weekly sales figures $[8, 10, 12, 11]$. In chronological order, they reveal an unmistakable upward momentum and positive lag-1 autocorrelation ($\rho_1 > 0$). Permuting those identical numbers into $[12, 8, 11, 10]$ destroys the lag covariance structure, turning a predictable trend into meaningless random noise. In time series, temporal sequence is the signal.

Split the Signal

Real-world time series entangle multiple physical dynamics. Classical additive decomposition deconstructs raw observation $Y_t$ into three orthogonal constituent components:

$Y_t = T_t + S_t + I_t$

At week $t$, an observed sales figure of $12$ decomposes into:

• Secular Trend ($T_t = 8$): The smooth, long-term multi-month growth trajectory.
• Seasonal Cycle ($S_t = 3$): Predictable periodic oscillations (such as weekend shopping surges or holiday demand spikes).
• Stochastic Irregular Residual ($I_t = 1$): The stationary, zero-mean leftover variation ($8 + 3 + 1 = 12$).

Classical ARIMA models do not fit raw trending waves directly; they operate on the stationary irregular residual $I_t$ once deterministic trend and cyclical seasonality have been extracted.

Past Plus Error

The core engine of an ARMA(1, 1) model blends autoregressive inertia with moving average shock absorption:

$\hat{Y}_t = \phi_1 Y_{t-1} + \theta_1 \epsilon_{t-1}$

Suppose the previous week's sales reached $Y_{t-1} = 10$, and the model's previous forecast error (the shock) was $\epsilon_{t-1} = 2.0$. With estimated parameters $\phi_1 = 0.70$ and $\theta_1 = 0.30$:

• The Autoregressive (AR) term captures persistence: $0.70 \times 10 = 7.0$.
• The Moving Average (MA) term damps recent surprise: $0.30 \times 2.0 = 0.60$.
• Summing both branches produces the calibrated point forecast: $\hat{Y}_t = 7.0 + 0.60 = 7.60$.

Over multi-step forecast horizons, compounding variance expands the $95\%$ confidence cone, providing honest probabilistic bounds on future uncertainty.

Three Small Knobs

Box-Jenkins forecasting requires weak stationarity: a constant mean $\mu$, constant variance $\sigma^2$, and autocovariance depending exclusively on lag distance. Non-stationary series exhibit unit roots ($\phi = 1$) that cause statistical forecasts to drift into infinite variance. The ARIMA$(p, d, q)$ framework resolves this through three coordinated levers:

• Differencing ($d$): First-order differencing ($\nabla Y_t = Y_t - Y_{t-1}$) transforms drifting non-stationary data $[5, 7, 9]$ into a stationary flat sequence of identical increments $[2, 2]$. Establishing stationarity ($d=1$) is the mandatory foundation.
• Autoregression ($p$): Once stationary, the $p$ parameter specifies how many lagged historical observations ($\phi_1 Y_{t-1} + \dots + \phi_p Y_{t-p}$) govern the forecast.
• Moving Average ($q$): The $q$ parameter specifies how many lagged white noise error shocks ($\theta_1 \epsilon_{t-1} + \dots + \theta_q \epsilon_{t-q}$) are factored into the estimate.

Beats Repeating Yesterday

Evaluating forecast performance against a naive persistence baseline ($\hat{Y}_t = Y_{t-1}$) demonstrates the power of modeled error feedback:

• Naive persistence baseline: Simply assuming tomorrow equals today creates an inherent one-period lag. When trend accelerates or noise spikes occur, the naive forecast overshoots and oscillates, yielding a large Mean Absolute Error of $\text{MAE} = 4.10$ across twelve weeks.
• ARIMA$(1, 1, 1)$ model: By combining autoregressive momentum with moving average error damping, the model anticipates trend slope and attenuates transient noise spikes. On the exact same twelve test weeks, ARIMA drops forecast error to $\text{MAE} = 1.20$—a $71\%$ reduction in forecast variance.

Small Orders First

Model complexity in time series is governed by Occam's razor and the Box-Jenkins principle of parsimony:

• Starting at Order 0 (naive mean), validation error sits at $4.10$.
• Increasing to Order 1 (e.g., ARIMA$(1, 1, 0)$) incorporates past values, reducing error to $2.40$.
• Increasing to Order 2 (e.g., ARIMA$(1, 1, 1)$) incorporates error damping, reaching the global error minimum of $\text{MAE} = 1.20$.
• Adding excess lag terms (Order 3 and beyond, such as ARIMA$(3, 1, 3)$) introduces high-order polynomial roots that fit sample noise and cause AIC/BIC penalties to explode. Forecast error climbs back up to $1.90$ and $3.20$. Stop adding parameters the moment held-out error stops falling.

Where It Breaks

Pure univariate ARIMA models operate under a fundamental vulnerability:

• Exogenous structural breaks: Univariate models possess no causal understanding of the external world. If an unexpected macroeconomic policy shift, competitor collapse, or viral event causes an immediate $+20$ level shift at week 6 (jumping sales from 15 to 35), ARIMA assumes historical statistical stationarity.
• The lag void: The model treats the sudden jump as a transient moving average shock and crawls upward sluggishly, producing severe forecast deficits that lag real-world ground truth for six consecutive weeks.
• Operational safeguards: Production forecasting systems must incorporate external causal regressors (ARIMAX), automated changepoint detection algorithms that reset the stationarity window, and human-in-the-loop override protocols during known black swan disruptions.

The Quick Version

• Chronological sequence holds autocorrelation; shuffling destroyed time series structure.
• Additive decomposition breaks raw series into trend (8), seasonality (3), and residual noise (1).
• ARMA(1, 1) blends 70% of last value with 30% of last error to forecast 7.60.
• Differencing ($d=1$) turns drifting values [5, 7, 9] into stationary increments [2, 2].
• ARIMA drops test error from 4.10 (naive repeat) to 1.20 by actively damping shocks.
• Parsimony rule: stop adding lag terms at Order 2 where validation error hits its minimum.
• Univariate ARIMA is blind to exogenous $+20$ shocks; use ARIMAX and changepoint detectors.