Different models can fit the same observations yet imply different mechanisms or predictions. This review examines how observation assumptions, imposed structure and learned components affect what a model can explain or predict. Forecasting, ensembles and causal effects have separate reviews.
🔒 Full text at this link requires payment or institutional access. Unmarked links may be unchecked.
- The two cultures - Breiman contrasts assumed stochastic data models with algorithms assessed by predictive accuracy. He argues for testing predictions and using a wider range of models instead of relying on goodness of fit within an assumed family. Breiman 2001.
- Explanation and prediction - Shmueli shows how explanatory and predictive goals change variable selection, fitting and validation. An underspecified model can predict better than a correctly specified one. Predictive accuracy does not establish a causal explanation. Shmueli 2010.
- Least squares and ridge - Least squares minimizes squared prediction errors; ridge adds a squared-coefficient penalty to stabilize fits with correlated predictors. docs.
- Generalized linear models - Connects an exponential-family response mean to a linear predictor through a link function. Unifies models fitted by iteratively weighted linear regression. Nelder and Wedderburn 1972.
- Generalized linear mixed models - Adds random effects to GLMs for correlated observations and overdispersion. Breslow-Clayton develops approximate inference using penalized quasi-likelihood. Breslow and Clayton 1993.
- Lasso - Constrains the sum of absolute regression coefficients, shrinking some exactly to zero. Combines estimation and variable selection. Tibshirani 1996.
- Logistic regression - Models binary class probability through a logistic link applied to a linear score. The documented implementation also supports multiclass classification and regularization. docs.
- Elastic net - Combines L1 and L2 penalties, allowing sparse coefficients while retaining ridge-style shrinkage. A useful alternative to lasso for correlated predictors. Zou and Hastie 2005; docs.
- Tests as linear models - Worked connections between regression, t-tests, correlation and ANOVA. Rank-based analogues are also discussed; some equivalences are approximate. Lindeløv 2019.
- Kaplan-Meier - Estimates a survival curve from incomplete lifetime observations using a product-limit estimator. Independent censoring is a substantive assumption, explicitly discussed in the paper. Kaplan and Meier 1958.
- Cox regression - Models covariate effects as multipliers of an unspecified baseline hazard. Handles censored event times without choosing a parametric baseline distribution. Cox 1972.
- DeepSurv - Uses a neural network for the risk function in a Cox proportional-hazards model, allowing nonlinear covariate interactions in survival prediction. Katzman et al. 2018.
- Distributional ensembles - See Ensemble Learning for quantile regression forests and NGBoost.
- Treatment effects - See Causal Inference for heterogeneous effects and meta-learners.
- Generalized additive models - Replaces a GLM's linear predictor terms with smooth functions of individual inputs. Local scoring extends additive regression to different response distributions. Hastie and Tibshirani 1986.
- MARS - Builds regression models from products of spline basis functions, selecting their number, variables and knot locations from data. Separates additive contributions from interactions. Friedman 1991.
- Intelligible classification and regression - Studies additive models whose individual feature contributions can be plotted, comparing fitting methods and the accuracy cost of restricting interactions. Lou et al. 2012.
- GAMs with autoregressive terms - Combines nonlinear covariate terms with autoregression for serially correlated outcomes. Demonstrated on temperature and daily mortality. Yang et al. 2012.
- GA2M - Extends additive models with selected pairwise interactions. FAST ranks candidate feature pairs, keeping explanations as one- and two-dimensional functions. Lou et al. 2013.
- Intelligible healthcare models - Pneumonia and readmission case studies use additive models with pairwise interactions to expose unexpected learned associations and revise models. Caruana et al. 2015.
- Neural Additive Models - Jointly trains a separate neural network for each input feature, then combines their outputs linearly. Learns flexible univariate shapes while preserving additive structure. Agarwal et al. 2021.
- How trustworthy are GAMs? - Shows that similarly accurate GAMs can learn conflicting feature shapes. Compares inductive biases, sparsity and behavior on rare subpopulations; accuracy alone does not validate an explanation. Chang et al. 2021.
- CART - Recursively partitions feature space into regions with simple predictions. Depth and pruning control complexity; individual trees can be unstable under changes to training data. docs.
- Bayesian additive regression trees - Represents a response as a sum of trees with a regularization prior that keeps individual trees weak. Bayesian backfitting samples the model and its predictive uncertainty. Chipman et al. 2010.
- TabPFN: original classifier - Pretrains a transformer on synthetic classification tasks. At inference, labeled training rows provide context for predictions without updating the model weights; the original scope is small tabular datasets. Hollmann et al. 2023.
- TabPFN: classification and regression - Extends prior-data-fitted tabular prediction to regression, categorical inputs and missing values. A pretrained transformer performs in-context prediction from labeled examples. Hollmann et al. 2025.
- TabICL - Combines representations of table rows with a transformer for in-context classification. Synthetic-task pretraining supports prediction on new tables without task-specific weight updates. Qu et al. 2025.
- Repeated measurements - At low noise, repeats can harm new-input prediction. IID priors linking inputs and responses count repeats as fresh input evidence even with exact inference; TabPFN-v3's row-count-dependent attention adds harm. Averaging often helps with many repeats. Liu et al. 2026.
- Tabular-model compression - Fits affine replacements to target-table activations. Across eight models and 171 classification datasets, halving depth costs at most 0.002 median AUROC per model; single-block regression deteriorates more sharply. Cannistraci and Vogt 2026.
- Tabular ensembles - See Ensemble Learning for random forests, XGBoost, CatBoost and TabM.
- Nearest-neighbor classification - Predicts from nearby labeled examples. Cover and Hart bound asymptotic one-neighbor error by twice Bayes error under their assumptions; this is neither a finite-sample guarantee nor a guarantee for an arbitrary learned distance. Cover and Hart 1967.
- Naive Bayes - Multiplies class-conditional feature probabilities under an independence model. Domingos and Pazzani show that correct class decisions can survive some dependence violations even when estimated probabilities are inaccurate. Domingos and Pazzani 1997.
- Support-vector networks - Constructs a linear decision surface in a nonlinear feature space and extends the approach to nonseparable training data. A foundation for kernel-based classification. Cortes and Vapnik 1995.
- SVM implementations - Linear and kernel classification, regression and outlier detection. Decision scores are not probabilities; probability estimation adds a separate fitting step. docs.
- Kalman filtering - Formulates linear filtering through state transitions and recursive error-covariance updates. Treats stationary and nonstationary prediction problems in a common framework. Kalman 1960.
- Sequence alignment - See Audio and Speech for dynamic time warping and spoken-word alignment.
- Hidden Markov models - Rabiner reviews Markov models with hidden states and probabilistic observations, with applications to speech recognition. A reference for the model's statistical structure and inference problems. Rabiner 1989.
- Infinite HMM - Uses Dirichlet processes to extend HMMs to countably many possible hidden states, integrating out transition parameters rather than fixing a finite state count in advance. Beal et al. 2001.
- SIR epidemic dynamics - Models infection and removal in a susceptible population under simplifying assumptions about contact and immunity. Epidemics can end while susceptible individuals remain. Kermack and McKendrick 1927.
- Predator-prey inference - Fits Lotka-Volterra differential equations with a statistical observation model to lynx-hare pelt counts. Demonstrates Bayesian parameter inference and posterior predictive checks. docs.
- Lorenz dynamics - Studies a deterministic nonlinear convection model whose nearby initial states can diverge. Connects nonperiodic dynamics to limits of long-range prediction. Lorenz 1963.
- Bass diffusion - Models initial product adoption using innovative and imitative behavior: purchase timing depends on previous buyers. The original study tests the model on consumer durables. Bass 1969.
- Bass model retrospective - Reviews the origins, applications and extensions of the product-diffusion model, identifying subsequent research directions. Bass 2004.
- Simple discrete dynamics - Shows how simple first-order nonlinear difference equations can produce stable equilibria, cycles or irregular fluctuations. A foundation for studying deterministic complexity. May 1976 🔒.
- DICE - Describes an integrated climate-economy model for dynamic optimization of climate policy. Documents the structure and assumptions behind its cost-benefit analysis. Nordhaus 1992.
- Bayesian calibration - Fits simulator parameters to physical observations while accounting for their remaining uncertainty. A discrepancy term represents systematic differences between the simulator and reality that adjusting parameters alone cannot remove. Kennedy and O'Hagan 2001.
- Model discrepancy and identifiability - Ignoring simulator error can bias parameters and overstate certainty. Adding a discrepancy function can improve predictions without identifying physical parameters. Recovering them requires informative assumptions about the discrepancy. Brynjarsdóttir and O'Hagan 2014 🔒.
- Hybrid scientific models - See Scientific ML for learned components constrained by physical models, and Symbolic Regression for discovering dynamical equations.
- rpart - R implementation of recursive partitioning for classification, regression and survival trees, based on CART. code.
- party - R toolbox for conditional-inference trees, forests and model-based recursive partitioning. Supports response types including censored and multivariate outcomes. code.
- mgcv - R implementation of generalized additive and additive mixed models, with smoothing-parameter estimation and several response-distribution families. code.