Time-series forecasting

1 Problem formulation and assumptions

Time-series forecasting aims to predict future observations from an ordered sequence of measurements indexed by time. Formally, the observed series is \(\{y_t\}_{t=1}^T\), and the task is to estimate future values \(\{y_{T+1}, \dots, y_{T+H}\}\) given information available up to time \(T\).

Forecasting models often rely on structural assumptions about temporal dependence. These can include the presence of trends (systematic movement over time), seasonality (periodic patterns), and autocorrelation (correlation between values at different lags). In practice, even when such assumptions are not explicitly enforced, many modeling choices implicitly target them.

1.1 Univariate vs. multivariate forecasting

In univariate forecasting, only a single target series \(y_t\) is modeled. Multivariate forecasting extends this by using multiple related series, either as additional targets or as inputs. Multivariate settings are useful when correlated processes share common dynamics (for example, multiple products with linked demand drivers).

A related distinction is between multivariate inputs with a single target and multivariate targets where several future variables are predicted jointly. The latter can capture cross-dependencies but may require more careful modeling and more data.

1.2 Deterministic vs. probabilistic forecasting

Deterministic forecasting produces point estimates for future values, such as \(\hat{y}_{t}\) for each forecasted time step. Probabilistic forecasting instead aims to represent uncertainty by outputting predictive distributions, prediction intervals, or quantiles.

Uncertainty-aware forecasts are typically preferable when decisions depend on risk or when variability changes over time. Probabilistic methods also help diagnose model misspecification when observed outcomes repeatedly fall outside expected ranges.

1.3 Horizon types: short-term, long-term, and multi-step

The forecast horizon \(H\) governs how far into the future predictions extend. Short-term forecasts focus on near-future behavior and often benefit from strong local temporal dependence. Long-term forecasts become increasingly sensitive to uncertainty accumulation and model bias.

Multi-step forecasting refers to predicting several future steps at once. Approaches differ in how they handle intermediate future values: some models generate forecasts step-by-step (autoregressive), while others predict all horizons directly (direct multi-horizon).

1.4 Stationarity, seasonality, and autocorrelation concepts

Stationarity describes whether statistical properties of a series remain stable over time. Strict stationarity is rarely satisfied in real data; however, weak forms (stable mean and variance, or stability after transformation) are common assumptions in classical modeling.

Seasonality captures recurring periodic effects, such as daily or weekly cycles. Autocorrelation reflects the tendency of values at one time to relate to past values. Many forecasting strategies can be viewed as ways to represent and learn these dependencies, explicitly through model structure or implicitly through feature design.

2 Data preparation

Model quality frequently hinges on preparation choices, since forecasting models can be sensitive to scaling, missingness, and timestamp irregularities. Data preparation involves aligning time indices, cleaning anomalies, transforming the series, and ensuring evaluation splits follow temporal order.

A central principle is that only information available prior to the forecast point may influence features used to predict that point. This constraint shapes both preprocessing and validation.

2.1 Timestamp handling and resampling

Time-series data may arrive at irregular intervals or with inconsistent sampling rates. Timestamp handling includes converting timestamps to a consistent timezone, detecting gaps, and resampling to a uniform grid when appropriate.

Resampling decisions (aggregation vs. interpolation) affect the meaning of the target. For example, summing over intervals is often appropriate for counts, while averaging may suit continuous measurements.

2.2 Missing data and imputation strategies

Missing observations can arise from sensor downtime, logging failures, or intermittent user activity. Imputation strategies range from simple methods (forward fill, linear interpolation) to model-based approaches (using neighboring lags, or learning missingness patterns).

The choice depends on whether missingness is random or systematic. If missingness correlates with the process being measured, naive imputation can introduce bias, and the imputation method may need to incorporate uncertainty or explicit missingness indicators.

2.3 Outliers and robust preprocessing

Outliers may represent genuine extreme events or artifacts such as data entry errors. Robust preprocessing aims to reduce undue influence from aberrant points without erasing meaningful extremes.

Common tactics include clipping, robust scaling (e.g., based on medians and quantiles), or using transformations that temper heavy tails. For decision-critical applications, it can also be important to track outlier treatment separately to understand downstream effects.

2.4 Transformations: scaling, differencing, and detrending

Transformations can improve learnability and align the series with model assumptions. Scaling standardizes magnitudes so optimization and regularization behave consistently across time. Differencing removes trends by modeling changes rather than levels, which can help when the target exhibits non-stationary behavior.

Detrending and deseasonalizing can simplify the signal, although they must be applied consistently to training and then inverted or handled appropriately during forecasting.

2.4.1 Box-Cox and variance-stabilizing transforms

Some series show changing variance over time, often increasing with the level (heteroscedasticity). Box-Cox transformations aim to stabilize variance and make the distribution closer to Gaussian-like under certain conditions.

Variance-stabilizing transforms, including variants of log transforms, can be particularly helpful for strictly positive targets such as demand counts or sensor intensities. After forecasting in transformed space, results generally need to be mapped back to the original scale.

2.5 Train/validation/test splitting for time series

Unlike random splits in cross-sectional problems, time-series evaluation must respect chronological order. The usual approach holds out later segments for validation and testing, ensuring that future values do not leak into training features.

Validation design also matters because many models require hyperparameter tuning and early stopping. The split should mimic how the model will be used operationally, including how often it will be retrained.

3 Feature engineering for temporal models

Feature engineering converts raw time-series observations into representations that models can use effectively. Even when using flexible machine learning models, lags, windows, calendar signals, and covariate handling often determine whether temporal structure is captured.

Feature sets should be constructed so they use only data available at forecast creation time.

3.1 Lag features and rolling statistics

Lag features use past values \(y_{t-k}\) for selected lags \(k\). Rolling statistics summarize recent behavior over windows, such as moving averages, minima/maxima, or rolling standard deviations.

Choosing lags and window sizes is a design task: too few lags can underfit dependence; too many can increase noise and computational burden. Rolling features can also help models detect short-term shifts and volatility changes.

3.2 Calendar and event-based features

Many time series follow human or environmental cycles. Calendar features encode day-of-week, month, hour-of-day, or holiday indicators. Encoding may be done using one-hot vectors, sine/cosine periodic representations, or learned embeddings.

Event-based features reflect known external events such as promotions, outages, or scheduled releases. These can materially improve accuracy when the event timing is known.

3.3 Windowing, sequences, and sliding forecasts

Windowing creates fixed-length inputs from variable-length histories. In sequence models, a window may be treated as a sequence fed into an encoder, often with the target being the next value or a vector of future values.

Sliding forecast generation produces multiple training examples by moving the window forward through time. This increases data efficiency but must be paired with leakage-safe split boundaries and consistent target definitions.

3.4 Handling exogenous variables (covariates)

Covariates are external inputs that may influence the target, such as weather forecasts for demand or marketing spend for website traffic. Feature engineering for covariates includes aligning them to the same time grid and deciding what transformations are required.

In practice, covariates may be delayed, incomplete, or available only up to certain timestamps. Models need to be robust to these realities, either through preprocessing or through architectural choices.

3.4.1 Future-known vs. future-unknown covariates

Some covariates are known ahead of time (e.g., planned schedules, published weather forecasts within a certain lead time), while others only become available after they occur (e.g., observed campaign performance).

This distinction affects forecast creation. For future-known covariates, the model can condition directly on their future values. For future-unknown covariates, strategies include forecasting covariates separately, using proxies available at forecast time, or restricting models to historical information.

4 Baseline forecasting approaches

Baselines provide reference points that ensure more complex models deliver real improvements. A well-chosen baseline is not necessarily simplistic; it should reflect common temporal structure such as persistence and seasonality.

If a proposed method does not beat strong baselines under time-aware evaluation, its added complexity may not be justified.

4.1 Naive methods and seasonal naive baselines

The simplest baseline is persistence: predict that the next value equals the most recent observation (\(\hat{y}_{t+1}=y_t\)). Seasonal naive methods extend this by using the value from the same seasonal position in the past (e.g., one week ago).

These approaches often perform surprisingly well when the series is dominated by regular cycles or when changes between adjacent periods are small.

4.2 Moving average and exponential smoothing variants

Moving average baselines predict future values using recent averages over a fixed window. Their strength lies in averaging out noise, but they can lag sudden changes.

Exponential smoothing assigns weights that decay over time, typically offering a better bias-variance trade-off than simple moving averages. Variants include methods with trend and seasonality components, aligning the forecast with recurring temporal patterns.

4.3 Simple trend and seasonal decomposition baselines

Decomposition approaches separate the observed series into components such as trend, seasonality, and residuals. Forecasting can be performed by extrapolating the trend component and repeating or modeling seasonal patterns.

Even when decomposition is imperfect, these baselines can provide interpretable predictors and a useful starting point for more advanced approaches.

5 Classical statistical forecasting models

Classical statistical models formalize temporal dependence using parametric structures. They often provide interpretability, relatively fast training, and well-studied behaviors, especially when data is limited.

Their performance depends on how well the model assumptions match the data’s generating process.

5.1 AR, MA, and ARMA fundamentals

Autoregressive (AR) models express a value as a function of past values. Moving average (MA) models use past forecast errors. Autoregressive moving average (ARMA) combines both, enabling richer dependence structures.

These models are commonly studied in terms of autocorrelation and model identifiability. They may struggle when strong non-stationarity or complex seasonal patterns exist unless extended.

5.2 ARIMA and seasonal ARIMA concepts

ARIMA extends ARMA by incorporating differencing to address non-stationarity. The parameters typically represent autoregressive order, differencing degree, and moving average order.

Seasonal ARIMA adds seasonal differencing and seasonal AR/MA terms to model periodic effects. When properly configured, these models can capture trend and seasonality within a unified framework.

5.3 Exponential smoothing state-space intuition

Exponential smoothing methods can be interpreted through a state-space lens: latent states represent level (and optionally trend/seasonality), and observations update those states through recursive smoothing.

This view connects classical smoothing to broader probabilistic forecasting, making it easier to reason about uncertainty and to implement variants that track different components.

5.4 Prophet-style modeling overview

Prophet-style approaches are designed to handle multiple seasonality patterns and non-linear trends using additive components. The model includes changepoints to allow the trend to shift over time, which can help when growth rates change.

Although not a universal solution, these models are valued for robustness in many business forecasting settings and for providing readable component breakdowns.

6 Modern machine learning approaches

Modern machine learning frames forecasting as a supervised learning problem: learn a mapping from historical features to future targets. The underlying goal remains the same, but model flexibility and feature design become central.

These methods can handle non-linearities and complex interactions between covariates and temporal context.

6.1 Regression with lagged features

A common approach is linear or generalized regression using lagged features and rolling statistics. Regularization (such as ridge or lasso) helps manage many correlated lag features.

Even straightforward models can be competitive if feature choices capture seasonality and if the series is relatively stable.

6.2 Tree-based models for forecasting

Decision trees and ensembles such as random forests and gradient-boosted trees can learn non-linear relationships. In forecasting, trees typically use lagged values and calendar features as inputs.

Because trees handle mixed feature types and are robust to scaling, they are convenient choices. However, multi-horizon prediction still requires a strategy, such as training separate models per horizon or using direct multi-output formulations.

6.3 Support vector regression and kernel methods

Support vector regression seeks a function that fits training data while controlling complexity through margin-based objectives. Kernels allow modeling non-linear dependencies without explicitly transforming features into high-dimensional spaces.

Kernel-based methods can be effective for medium-sized datasets but may face scalability challenges for large time series due to training complexity.

6.4 Gradient boosting and categorical/calendar encoding

Gradient boosting models iteratively refine predictions, often achieving strong accuracy with careful tuning. Proper encoding of time-related inputs is important; categorical encodings for day-of-week or month can help trees and boosting methods represent periodicity.

Calendar features may be encoded as integers, one-hot vectors, or more structured periodic representations depending on the model family and data characteristics.

7 Deep learning for time-series forecasting

Deep learning approaches learn representations directly from sequences, often using historical windows as inputs. They can model complex temporal patterns and interactions, including those arising from covariates.

Deep models can require more data and careful regularization to avoid overfitting, particularly when the available history is short.

7.1 Sequence modeling with RNN/LSTM/GRU

Recurrent neural networks process sequences step-by-step, maintaining a hidden state that summarizes past information. Long short-term memory (LSTM) and gated recurrent units (GRU) add gating mechanisms to mitigate vanishing gradient issues and improve learning of longer dependencies.

For forecasting, these models may be trained to predict the next value or multiple future steps, depending on the output design.

7.2 Temporal Convolutional Networks (TCN)

Temporal convolutional networks use 1D convolutions with causal structure, ensuring that predictions at time \(t\) depend only on inputs up to \(t\). Dilated convolutions enlarge receptive fields efficiently, capturing both short and long lags.

TCNs often train effectively due to parallelizable convolution operations and can be competitive for many forecasting problems.

7.3 Transformers and attention mechanisms

Transformers use attention to relate each time step to relevant past steps, enabling flexible dependency modeling. For forecasting, they may attend over input windows and generate outputs via autoregressive generation or direct prediction heads.

When data is limited, attention models can overfit; regularization, careful window sizing, and robust validation are important.

7.4 Auto-regressive vs. direct multi-horizon deep approaches

Auto-regressive deep forecasting produces future values sequentially, using previously generated predictions as inputs for later steps. This can compound errors but matches the natural dependency structure of many processes.

Direct multi-horizon methods predict all future steps at once, typically avoiding error accumulation from step-by-step generation. The choice can depend on horizon length, data availability, and how well intermediate future values can be inferred.

8 Probabilistic forecasting and uncertainty

Probabilistic forecasting acknowledges that future outcomes are uncertain. Models can output distributions or calibrated intervals, supporting risk-sensitive planning and more informative evaluation.

Uncertainty estimation methods vary widely, ranging from explicit probabilistic heads to distribution-free approaches.

8.1 Prediction intervals and quantile forecasting

Prediction intervals specify ranges expected to contain the true value with a chosen probability. Quantile forecasting models target conditional quantiles (e.g., the 10th and 90th percentiles), which can then form an interval.

Quantile-based training can better handle asymmetry in errors than variance-only Gaussian assumptions, particularly for heavy-tailed or skewed data.

8.2 Distributional forecasting (e.g., parametric vs. nonparametric)

Distributional forecasting aims to describe the entire predictive distribution rather than only moments or intervals. Parametric methods assume a family (such as Gaussian or negative binomial) and estimate its parameters.

Nonparametric approaches may approximate arbitrary distributions using quantiles, mixtures, or flexible heads. The trade-off involves expressiveness versus the complexity of calibration and interpretability.

8.3 Conformal prediction for time series (conceptual overview)

Conformal prediction produces uncertainty sets with coverage guarantees under certain exchangeability assumptions. For time series, these assumptions are addressed through adaptations such as rolling calibration windows and time-aware resampling schemes.

While conceptual, conformal ideas are influential because they can provide robust interval validity even when the underlying model’s calibration is imperfect.

8.4 Calibration and sharpness of predictive distributions

Calibration assesses whether predicted probabilities match observed frequencies (for instance, whether nominal 90% intervals contain the truth about 90% of the time). Sharpness measures how concentrated the predictive distribution is.

Good forecasts are both calibrated and sharp. A model can be sharp but poorly calibrated (intervals too narrow) or well calibrated but uninformative (intervals too wide).

9 Model evaluation and selection

Evaluation in time-series forecasting must reflect the temporal nature of the task. Metrics, validation schemes, and diagnostic plots together determine whether a model generalizes to future data.

Selection is not only about average error; it also includes robustness across horizons and operating regimes.

9.1 Forecast accuracy metrics for time series

Common point forecast metrics include mean absolute error (MAE), root mean squared error (RMSE), and mean absolute percentage error (MAPE), though percentage metrics can be unstable when targets approach zero. For probabilistic forecasts, interval coverage and proper scoring rules (such as pinball loss for quantiles or negative log-likelihood for distributional models) are often used.

Metric choice should reflect the costs of different errors and the distributional characteristics of the series.

9.2 Error analysis by horizon and regime

Errors often grow with horizon length, and error patterns may differ across time regimes (such as volatile periods versus stable ones). Analyzing metrics by forecast step helps reveal whether the model properly captures long-range dependencies.

Regime-based analysis can involve splitting evaluation periods by season, event occurrence, or volatility level. This can identify whether the model’s weaknesses are systematic rather than random.

9.3 Cross-validation schemes: walk-forward and rolling origin

Walk-forward validation repeatedly trains on an expanding window and evaluates on the next time segment. Rolling origin validation uses a fixed-size training window that shifts forward over time.

These schemes better approximate operational conditions than random splits, though they increase computational demands. The chosen scheme should align with how the system will be retrained in production.

9.4 Hyperparameter tuning and early stopping

Hyperparameter tuning adapts model complexity and regularization to the data. For deep learning, early stopping halts training when validation performance ceases to improve, reducing overfitting.

Because validation uses future time segments, tuning should be constrained to avoid excessive reuse of a specific test set. Ideally, there is a distinct final test period reserved for unbiased reporting.

10 Forecasting workflows and deployment considerations

Deployment transforms a model from a research artifact into a reliable forecasting service. Workflows must address retraining schedules, drift, inference mode, and operational monitoring.

The goal is to keep forecasts accurate as data distributions change over time.

10.1 Re-training frequency and drift monitoring

Forecast accuracy can degrade as the underlying process evolves. Re-training frequency depends on how quickly the series changes and whether covariates or seasonal patterns shift.

Drift monitoring tracks changes in input distributions, residual behavior, or prediction error proxies. When drift is detected, pipelines may trigger retraining or model updates.

10.2 Backtesting with realistic cutoffs

Backtesting evaluates models as if they were deployed at past times. This requires using realistic forecast cutoffs, where only data available before the cutoff is used to generate forecasts.

Backtesting also clarifies how model performance evolves with repeated retraining, helping identify whether improvements are consistent or tied to specific periods.

10.3 Serving forecasts and batch vs. streaming inference

Batch inference produces forecasts on a schedule (e.g., hourly demand forecasts every night). Streaming inference updates forecasts as new data arrives, potentially requiring incremental feature updates and low-latency processing.

Both modes require consistent preprocessing and feature generation so that inputs during deployment match those used during training.

10.4 Performance monitoring and alerting

Operational monitoring includes tracking forecast error where ground truth becomes available, monitoring input quality, and verifying pipeline health. Alerting thresholds can be based on rolling error measures or predicted uncertainty behavior.

Monitoring is essential not only for accuracy but also for detecting failures such as missing data, misaligned timestamps, or broken feature extraction.

11 Common pitfalls and troubleshooting

Many failures in forecasting systems stem from evaluation mistakes, misaligned targets, or inappropriate assumptions. Troubleshooting often involves checking data leakage, verifying time indexing, and inspecting residuals.

The following pitfalls are frequent and typically diagnosable through careful analysis.

11.1 Leakage in time-series feature construction

Leakage occurs when information from the future influences features used for training or evaluation. Examples include using rolling statistics computed across the forecast boundary or normalizing with statistics that include future data.

Leakage can yield overly optimistic results and harm deployment performance. Ensuring that all transformations are fit only on past data is a key safeguard.

11.2 Overfitting to seasonality or noise

Models may memorize seasonal patterns that change over time or latch onto noise captured by overly complex features. Symptoms include excellent validation performance in one period but degradation elsewhere.

Regularization, limiting model complexity, using robust baselines for comparison, and employing time-aware validation can mitigate this issue.

11.3 Mis-specified horizon and alignment issues

Forecasting involves careful alignment between input windows and forecast targets. Off-by-one errors, incorrect shift amounts, or inconsistent definitions of horizon can silently degrade performance.

Verifying indexing conventions and plotting predicted vs. true series over time are practical checks that often reveal alignment problems.

11.4 Interpreting residuals and diagnostics

Residual analysis helps assess whether errors are structured or random. For instance, autocorrelated residuals indicate missing temporal structure, while changing residual variance suggests heteroscedasticity.

Diagnostic plots and residual-based tests guide whether additional features, transformations, or model families are warranted.

12 Reference implementations and resources

Practical forecasting work benefits from reproducible experimentation, good ecosystem support, and standard benchmark practices. Reference implementations help compare approaches fairly and reduce engineering overhead.

Documentation, version control, and dataset handling policies are especially important for time-series projects.

12.1 Reproducible experiment tracking

Reproducibility depends on recording data versions, preprocessing steps, model parameters, and evaluation protocols. Experiment tracking tools capture metrics over time and store artifacts such as trained models and configuration files.

With time series, reproducibility also requires deterministic data splitting to ensure evaluation repeatability.

12.2 Typical libraries and ecosystem overview

A forecasting ecosystem often spans classical statistics packages, gradient boosting toolkits, and deep learning frameworks. Libraries may provide utilities for lag feature generation, probabilistic objectives, and time-aware cross-validation.

Selecting tools depends on the modeling approach, required inference latency, and whether probabilistic outputs are needed.

12.3 Dataset curation and benchmark conventions

Benchmark datasets typically define consistent train/test splits, specify temporal granularity, and report standard metrics. Good dataset curation includes documenting missingness handling, outlier treatment, and how covariates are aligned.

Benchmark conventions help ensure that improvements are comparable across studies, reducing ambiguity in reported performance.