search_and_ranking_ml

Other biases compound this. Presentation bias means items with images or particular formats attract
attention independently of relevance. Selection bias means only items the current system retrieved
were ever seen, so nothing is learned about items it never showed. Trust bias means users click
top results partly because they trust the ranking. And the feedback loop closes on all of them: today's
model determines what data is collected, which trains tomorrow's model. Deliberate exploration —
occasionally showing items the model ranks lower — is the only reliable way to break the loop, and
it costs immediate quality in exchange for the ability to improve.


search_and_ranking_ml

Behaviour and bias:
  behavioural signals  clicks, dwell, purchases, reformulations
  the tradeoff         abundant and biased against scarce and clean
  position bias        higher items clicked more regardless of relevance
  the convergence failure  the model reproducing the ranking that generated its
                       data
  propensity weighting  observations weighted by inverse examination probability
  randomization        swapping positions in a fraction of traffic to estimate
                       the bias
  presentation bias    formatting and imagery attracting attention
  selection bias       nothing learned about items never retrieved
  trust bias           clicks reflecting confidence in the ranking itself
  the feedback loop    today's model determines tomorrow's training data
  exploration          deliberately showing lower-ranked items
  the cost             immediate quality traded for the ability to improve
  dwell time           a stronger relevance signal than click, with its own
                       biases by content type
  negative signals     quick returns to the results page; informative and
                       underused
  the judgement complement  a small clean judgement set for evaluation even when
                       training is behavioural, because a behavioural evaluation
                       inherits every bias the training data has


search_and_ranking_ml

## Evaluation and Serving

Offline evaluation on a judgement set iterates quickly and correlates imperfectly with online
outcomes. Counterfactual evaluation using logged data with propensity weights estimates what a new
ranker would have achieved on past traffic, which is stronger than a judgement set and depends on
the logged policy having had support for the actions the new policy would take. Online evaluation is
authoritative, and interleaving — mixing two rankings within one result list and attributing clicks —
is far more sensitive than a conventional split test for ranking comparisons specifically.


search_and_ranking_ml

Serving architecture is the multi-stage cascade, and each stage's job is to reduce the candidate set
cheaply enough that the next stage can afford to be more expensive. Retrieval produces thousands
from millions; a lightweight ranker reduces to hundreds; a heavy reranker orders the top tens; and
final policies apply diversity, deduplication, and business rules. The latency budget is divided
among these, and the design question at each stage is how much recall is lost for how much cost
saved.


search_and_ranking_ml

Recall at the first stage bounds everything downstream, and it is the metric most often unmeasured.
A reranker cannot recover a relevant document the retriever never returned, so an investment in
reranking quality is wasted if the retrieval stage is the binding constraint. Measuring stage-wise
recall against a superset produced by an expensive method is the diagnostic that tells you where to
spend effort, and it frequently redirects work from the sophisticated end of the pipeline to the
simple end.


search_and_ranking_ml

Evaluation and serving:
  offline judgement sets  fast iteration, imperfect correlation
  counterfactual evaluation  logged data with propensity weights
  the support requirement  the logged policy must have had a chance of taking the
                       new policy's actions
  interleaving         mixing two rankings; far more sensitive for ranking
                       comparisons
  split testing        authoritative and less sensitive
  the cascade          retrieval, light ranking, heavy reranking, policy
  the budget division  latency allocated across stages
  the stage question   how much recall lost for how much cost saved
  first-stage recall   bounds everything downstream
  the unmeasured metric  stage-wise recall is frequently not measured at all
  the diagnostic       compare each stage's output against an expensive superset
  where effort goes    frequently the simple end rather than the sophisticated one
  diversity and deduplication  applied last; a ranking of near-identical results
                       is worse than a slightly less relevant varied one
  freshness policy     time-sensitive queries need recency weighting that general
                       queries do not; detecting which is which is itself a
                       ranking problem and one of the higher-value ones to solve


speech_and_audio_ml

## Audio Representation

Audio machine learning begins with representation, and the choice determines what the model can
learn and how efficiently. A raw waveform at 16 kilohertz is 16,000 samples per second, so a
ten-second clip is 160,000 values — a far longer sequence than a comparable amount of text, which
is the structural difficulty of the domain.


speech_and_audio_ml

The spectrogram is the standard representation and converts the waveform into a time-frequency
image by taking the Fourier transform over successive overlapping windows. The window length sets
the time-frequency tradeoff directly: a short window localizes events in time and resolves
frequency poorly, and a long window does the reverse. Speech typically uses windows of twenty to
twenty-five milliseconds with ten-millisecond hops, which is chosen because the vocal tract's
configuration is approximately stationary over that duration.


speech_and_audio_ml

The mel scale reflects human pitch perception, which is approximately logarithmic above a few
hundred hertz, and warping the frequency axis to it concentrates resolution where the auditory
system has it. A mel spectrogram with sixty-four to a hundred and twenty-eight bands is the
standard input to speech and audio models, reducing the dimensionality substantially while
preserving the perceptually relevant structure.


speech_and_audio_ml

Audio representation:
  sample rate          16 kHz for speech, 44.1 or 48 kHz for music; Nyquist
                       requires twice the highest frequency of interest
  bit depth            16-bit is standard for delivery; higher for processing
                       headroom
  raw waveform         the most information and the longest sequence
  spectrogram          time-frequency; the standard representation
  window length        the time-frequency tradeoff; 20-25 ms for speech
  hop length           the frame advance; 10 ms is standard
  window function      Hann is the usual choice; reduces spectral leakage
  mel scale            approximately logarithmic; matches pitch perception
  mel spectrogram      64-128 bands; the standard model input
  log magnitude        compresses the dynamic range to match perception
  MFCC                 a decorrelating transform of the mel spectrum; the
                       classical feature, and largely superseded because neural
                       models prefer the correlated mel representation
  phase                discarded in a magnitude spectrogram; reconstruction
                       therefore requires estimating it, which is why vocoders
                       exist
  constant-Q transform  logarithmic frequency spacing; suits music where pitch
                       relationships are ratios


speech_and_audio_ml

The discarding of phase deserves emphasis because it creates the central difficulty in audio
generation. A magnitude spectrogram is a convenient and lossy representation — the phase carries
information about the fine temporal structure, and reconstructing audio from magnitude alone
requires estimating it. Classical iterative methods produce audible artifacts, and neural vocoders
that learn the mapping from spectrogram to waveform are what made high-quality synthesis
practical. Any pipeline that generates a spectrogram must address this, and it is the reason
end-to-end waveform models became attractive despite their sequence length.


speech_and_audio_ml

Q: A speech model uses a 25 ms analysis window. Doubling it to 50 ms would:
A. Improve both time and frequency resolution
B. Improve frequency resolution and degrade time resolution, and would also span multiple
   phonetic events, violating the stationarity assumption
C. Have no effect
D. Improve time resolution
Reasoning: The time-frequency uncertainty relation means a longer window resolves frequency more
finely and localizes events more coarsely. For speech, 20-25 ms is chosen because the vocal tract
configuration is roughly stationary over that interval; a 50 ms window spans a transition between
phonemes and smears them together. Answer: B.


speech_and_audio_ml

## Speech Recognition

Automatic speech recognition converts audio to text, and the architecture has shifted
substantially. Classical systems combined an acoustic model mapping audio to phonetic units, a
pronunciation lexicon mapping words to phone sequences, and a language model scoring word
sequences, with a decoder searching over the combination. Each component was trained separately
and the pipeline required linguistic resources.


speech_and_audio_ml

End-to-end models map audio directly to text, learning the acoustic, lexical, and linguistic
relationships jointly. Connectionist temporal classification handles the alignment problem — the
audio and the transcript have different lengths with no given correspondence — by summing over all
alignments, which permits training without frame-level labels. Attention-based encoder-decoder
models learn the alignment implicitly. Transducer models combine an audio encoder with a
prediction network, which permits streaming because the output can be produced incrementally.


speech_and_audio_ml

The streaming requirement divides architectures sharply. An offline model can attend to the entire
utterance, including audio after the current position, which improves accuracy substantially. A
streaming model must produce output with bounded latency and can therefore attend only to past and
a limited amount of future context. The accuracy difference is real, which is why systems
frequently run a streaming model for immediate display and rescore with an offline model when the
utterance completes.


speech_and_audio_ml

Speech recognition:
  acoustic model       audio to phonetic units; classical pipeline component
  pronunciation lexicon  words to phone sequences; requires linguistic resources
  language model       scores word sequences; still valuable for domain adaptation
                       even with end-to-end acoustic modelling
  CTC                  sums over alignments; permits training without frame labels
  attention encoder-decoder  learns alignment implicitly; strong offline
  transducer           encoder plus prediction network; the standard streaming
                       architecture
  streaming            bounded latency, limited future context; measurably less
                       accurate than offline
  two-pass             stream for display, rescore offline for the final result
  beam search          decoding; wider beams improve accuracy at latency cost
  shallow fusion       combine an external language model at decoding; the
                       practical route to domain adaptation
  self-supervised pretraining  learn representations from unlabelled audio, then
                       fine-tune on a small labelled set; the change that made
                       low-resource recognition feasible
  word error rate      the standard metric; substitutions plus deletions plus
                       insertions over reference words
  the failure modes    accented speech, code-switching, overlapping speakers,
                       domain vocabulary, and noise; performance varies far more
                       across these than aggregate figures suggest


speech_and_audio_ml

Self-supervised pretraining deserves emphasis because it changed the resource requirements. A
model pretrained on large quantities of unlabelled audio — learning to predict masked portions or
to distinguish true from distractor continuations — acquires representations capturing phonetic
structure without any transcripts. Fine-tuning on a small labelled set then reaches accuracy that
previously required orders of magnitude more transcribed data. This made recognition practical for
languages and domains where transcription is expensive, which is most of them.


speech_and_audio_ml

The variation in performance across speaker populations deserves stating because aggregate word
error rates conceal it. Systems trained predominantly on one accent, dialect, or demographic
perform substantially worse on others, with documented disparities of a factor of two or more in
error rate between speaker groups. This is a data composition problem rather than a modelling one,
and reporting per-group performance rather than an aggregate is what makes it visible.


speech_and_audio_ml

## Speaker and Audio Understanding

Speaker recognition determines who is speaking, and it divides into verification — confirming a
claimed identity — and identification — determining which of a set of known speakers. The modern
approach embeds an utterance into a fixed-dimensional vector such that utterances from the same
speaker are close, and comparison is then a distance computation. This permits enrolment with a
short sample and comparison against a large set efficiently.


speech_and_audio_ml

Speaker diarization determines who spoke when, which is required for any multi-speaker recording.
The classical pipeline segments the audio, embeds each segment, clusters the embeddings, and
assigns labels. Its persistent difficulty is overlapping speech, where two people talk
simultaneously — common in natural conversation and handled poorly by approaches assuming one
speaker per segment. End-to-end methods that predict per-speaker activity directly handle overlap
by construction.


speech_and_audio_ml

General audio understanding covers classification and detection of non-speech sound — recognizing
a dog bark, a glass break, a machine fault, a bird species. The techniques parallel image
classification with spectrograms as the input, and the distinctive difficulties are that events
overlap, that the relevant timescales vary from milliseconds to minutes, and that labelled data is
scarce relative to images.


speech_and_audio_ml

Speaker and audio understanding:
  speaker verification  confirm a claimed identity; a threshold on similarity
  speaker identification  which of a known set
  speaker embedding    a fixed vector per utterance; comparison by distance
  enrolment            a short reference sample per speaker
  diarization          who spoke when
  overlapping speech   common in conversation, handled poorly by segment-based
                       approaches
  target speaker extraction  isolate one speaker given a reference sample
  source separation    separate mixed sources; the cocktail party problem
  permutation invariant training  the separation output order is arbitrary, so
                       the loss must consider all assignments
  voice activity detection  speech versus not; the front end of most pipelines,
                       and its errors propagate through everything after it
  keyword spotting     detect specific words continuously at very low power;
                       always-on wake words
  audio event detection  what happened and when
  audio tagging        what is present, without timing
  acoustic scene       the environment; street, office, forest
  anti-spoofing        detect replayed or synthesized audio; increasingly
                       necessary as synthesis quality rises


speech_and_audio_ml

Anti-spoofing deserves attention because synthesis quality has made voice authentication
substantially less secure. A few seconds of reference audio now suffices to produce speech
convincing to a human listener and to many verification systems, which means voice alone is no
longer adequate authentication for anything consequential. Detection systems exist and are in an
adversarial contest with synthesis, and the practical response is treating voice as one factor
among several rather than as sufficient in itself.


speech_and_audio_ml

## Speech Synthesis

Text to speech has progressed through distinct generations with substantially different quality.
Concatenative synthesis assembles recorded units, which gives natural segments and audible
discontinuities at the joins and requires a large recorded corpus per voice. Parametric synthesis
generates from a model of the vocal tract, which is compact and flexible and sounds buzzy. Neural
synthesis learns the mapping end to end and reaches quality that listeners rate comparably to
recorded speech.


speech_and_audio_ml

The modern architecture is typically two stages. An acoustic model maps text — or phonemes derived
from it — to a mel spectrogram, handling pronunciation, prosody, and timing. A vocoder converts the
spectrogram to a waveform, solving the phase reconstruction problem. Separating them permits each
to be trained and improved independently, and the vocoder is reusable across voices.


speech_and_audio_ml

Prosody is where synthesis remains weakest and where the remaining gap to human speech lies.
Segmental quality — whether the phonemes sound right — is essentially solved. Prosody covers the
rhythm, stress, intonation, and emphasis that convey meaning beyond the words, and it is
underdetermined by the text: the same sentence can be spoken many ways, all correct, conveying
different emphasis and attitude. A model trained on text-to-audio pairs learns an average prosody,
which is why synthesized speech sounds competent and slightly flat.


speech_and_audio_ml

Speech synthesis:
  concatenative        recorded units; natural segments, audible joins
  parametric           vocal tract model; compact, buzzy
  neural end to end    the current approach; near-human segmental quality
  two-stage            acoustic model to spectrogram, vocoder to waveform
  vocoder              spectrogram to waveform; solves phase reconstruction
  autoregressive vocoder  highest quality, slow, sample by sample
  parallel vocoder     generates the whole waveform at once; fast, and the quality
                       gap has largely closed
  text normalization   numbers, dates, abbreviations, and symbols to spoken form;
                       unglamorous and a major source of errors
  grapheme to phoneme  pronunciation; ambiguous for homographs, requiring context
  prosody              rhythm, stress, intonation; underdetermined by the text and
                       where the remaining gap lies
  duration modelling   how long each phoneme lasts; determines the rhythm
  voice cloning        a few seconds of reference; raises consent and misuse
                       questions that the technical capability does not address
  emotional control    conditioning on a style reference or an explicit label
  streaming synthesis  produce audio before the full text is processed; required
                       for conversational latency


speech_and_audio_ml

Text normalization deserves mention because it is unglamorous and is where a substantial share of
synthesis errors originate. Converting written text to what should be spoken requires expanding
numbers according to context — a year, a quantity, a phone number, and a decimal are all read
differently — expanding abbreviations, handling currency and units, and disambiguating homographs
from context. These are largely rule-based and language-specific, they interact with the
pronunciation model, and a system with excellent acoustic quality that reads a date incorrectly is
judged on the error rather than on the quality.


speech_and_audio_ml

[user] Our voice assistant's speech recognition works well in our office and poorly in customers'
homes. Word error rate goes from 4% to 19%.
[assistant] tool_call asr.deployment_gap({"office_wer": 0.04, "field_wer": 0.19,
"training_data": "clean_read_speech", "office_conditions": {"snr_db": 35, "reverb_t60_ms": 350,
"mic_distance_cm": 30}, "field_conditions": {"snr_db": 12, "reverb_t60_ms": 700,
"mic_distance_cm": 200}, "augmentation": "none"})
[tool] {"primary_factors": ["far_field_distance", "reverberation", "background_noise"],
"training_mismatch": "clean_close_talk_vs_far_field_reverberant",
"recommendations": ["room_impulse_response_augmentation", "noise_augmentation",
"multichannel_frontend", "far_field_fine_tuning"]}
[assistant] The model was trained on clean close-talk speech and deployed at two metres in
reverberant rooms with substantial background noise. Three factors compound and each is
addressable.


speech_and_audio_ml

Distance is the largest. At two metres rather than thirty centimetres, the direct sound is roughly
sixteen times weaker while the reverberant field is unchanged, so the ratio of direct to reverberant
energy falls sharply. Reverberation smears the temporal structure that the model relies on —
each phoneme's energy persists into the following ones — and your field reverberation time is
double your office's.


speech_and_audio_ml

Background noise at 12 dB signal-to-noise is a substantial degradation from 35 dB, and the model
has never seen anything like it.


speech_and_audio_ml

Four responses, and the first two require no new data collection.


speech_and_audio_ml

Augment the training data with simulated far-field conditions. Convolve your clean training audio
with measured or simulated room impulse responses spanning realistic reverberation times, and add
noise at a range of signal-to-noise ratios. This is the highest-value change and it is
straightforward — impulse response datasets are available, and the augmentation is applied at
training time. Expect a substantial fraction of the gap to close.


speech_and_audio_ml

Add a multichannel front end if the hardware has more than one microphone. Beamforming steers
sensitivity toward the speaker and away from noise sources, typically gaining several decibels of
effective signal-to-noise ratio, and it also reduces the reverberant contribution. Even two
microphones help meaningfully.


speech_and_audio_ml

Then fine-tune on real field audio once you have some. Simulated augmentation covers the acoustics
and does not capture the actual devices, rooms, and speaking styles, and a modest amount of real
in-domain data typically outperforms a large amount of simulated.


speech_and_audio_ml

One thing to check first: verify your voice activity detection is not the problem. If it is
triggering late or cutting off early in noise, the recognizer receives truncated audio and the
word error rate rises for reasons unrelated to the acoustic model. Inspect a sample of field
recordings with the detected boundaries marked, because this failure is common and is fixed
differently from everything above.


speech_and_audio_ml

## Music, Evaluation, and Deployment

Music information retrieval has its own tasks and its own structure. Pitch and note detection
extract the musical content. Beat and tempo tracking find the rhythmic structure. Chord recognition
identifies harmony. Source separation isolates instruments or vocals, which has become good enough
to be used commercially for remixing and practice tools. Genre and mood classification categorize.
And generation produces new material, which raises the same rights questions as generative models
in other media.


speech_and_audio_ml

The distinctive property of music is that the relevant structure is hierarchical and spans
timescales from milliseconds to minutes. A model attending only locally captures timbre and misses
form; capturing structure over minutes requires either very long context or an explicit
hierarchical representation. This is why music generation produces convincing short segments more
readily than coherent long pieces.


speech_and_audio_ml

Evaluation in audio is difficult because the target is perceptual. Objective metrics correlate
imperfectly with human judgment, and the standard for synthesis quality remains a subjective
listening test with a substantial number of listeners. Automated metrics are used for iteration and
the final judgment is human, which is expensive and is why the field moves more slowly than
domains with reliable automatic evaluation.


speech_and_audio_ml

Music, evaluation, and deployment:
  pitch detection      fundamental frequency; polyphonic is far harder than
                       monophonic
  beat tracking        the rhythmic grid
  chord recognition    harmonic content over time
  source separation    isolate stems; commercially useful quality now
  music generation     convincing short segments, coherent long form remains hard
  hierarchical structure  milliseconds to minutes; the distinctive difficulty
  word error rate      the recognition standard; and it weights all errors equally
                       while a wrong entity name matters more than a wrong article
  mean opinion score   subjective listening test; the synthesis standard
  objective metrics    correlate imperfectly; useful for iteration, not for the
                       final judgment
  intelligibility      distinct from quality; a synthetic voice can be clear and
                       unnatural or natural and unclear
  latency budget       conversational turn-taking degrades above a few hundred
                       milliseconds
  on-device            privacy, latency, and offline operation; requires
                       quantization and architecture constraints
  wake word            always on at very low power; a tiny model with a strict
                       false accept budget, since the cost of a false accept is a
                       privacy incident
  streaming            partial results with bounded latency
  privacy              audio is highly sensitive; on-device processing and explicit
                       retention policies are the substantive responses


speech_and_audio_ml

The wake word constraint deserves the closing note because it is an unusual engineering problem.
The model runs continuously on battery, so its power budget is measured in milliwatts, which
constrains it to a few tens of kilobytes and a fraction of a processor core. It must have a very
low false accept rate, since a false accept means the device begins recording and transmitting
audio the user did not intend — a privacy incident rather than an inconvenience. And it must have
a low false reject rate, since a device that does not respond is perceived as broken. Meeting all
three simultaneously is what makes wake word detection a specialized problem rather than a small
instance of keyword spotting, and it is typically solved with a cascade where a tiny always-on
model gates a larger verification model that runs only on a candidate detection.


statistics_and_inference

## Probability Foundations

A probability space is a triple (Omega, F, P): a sample space of outcomes, a
sigma-algebra of measurable events, and a measure assigning each event a number in
[0, 1] with P(Omega) = 1 and countable additivity over disjoint events. Most applied
work never touches the measure-theoretic scaffolding directly, but it is what makes
conditional expectation well defined for continuous variables, and it is why a
"probability zero" event is not the same as an impossible one — a continuous random
variable takes any particular value with probability zero, yet it takes some value
every time you sample it.


statistics_and_inference

A random variable is a measurable function from the sample space to the reals. Its
distribution is fully described by the cumulative distribution function F(x) = P(X <=
x), which is right-continuous and non-decreasing with limits 0 and 1 at minus and
plus infinity. Discrete variables have a probability mass function; continuous ones
have a density f(x) = dF/dx. The density is not a probability — it can exceed 1 — and
only its integral over an interval is one. Mixed distributions, common in practice
(a spend variable that is exactly zero for non-purchasers and continuous above), have
neither a pure pmf nor a pure density, and modelling them as if they did is a
frequent source of bad likelihoods.


statistics_and_inference

Expectation is the Lebesgue integral E[X] = integral x dF(x), linear in its argument
regardless of dependence: E[aX + bY] = aE[X] + bE[Y] always, even for correlated X
and Y. Variance is not linear; Var(aX + b) = a^2 Var(X), and Var(X + Y) = Var(X) +
Var(Y) + 2Cov(X, Y), with the covariance term vanishing only under uncorrelatedness.
Independence implies zero correlation, but the converse fails: if X is standard
normal and Y = X^2, then Cov(X, Y) = E[X^3] = 0 while Y is a deterministic function
of X. Correlation measures linear association only, and forgetting that is how
analysts declare "no relationship" in the presence of a perfect parabola.


statistics_and_inference

The law of large numbers and the central limit theorem are the two pillars that make
statistics work. The weak LLN says the sample mean converges in probability to the
population mean; the strong LLN upgrades this to almost-sure convergence. The CLT
says that for iid variables with finite variance sigma^2, sqrt(n)(Xbar - mu)
converges in distribution to Normal(0, sigma^2), regardless of the shape of the
underlying distribution. The practical consequences are enormous: it is why standard
errors shrink as 1/sqrt(n), why doubling precision costs four times the data, and why
normal-theory confidence intervals work for means of decidedly non-normal data at
moderate sample sizes. It is also routinely over-applied — the CLT says nothing
useful about the sample mean of a Cauchy distribution (infinite variance, the sample
mean has the same distribution as a single observation no matter how much data you
collect), and its convergence is slow in the tails, which is exactly where risk
calculations live.


statistics_and_inference

Conditional probability is defined as P(A | B) = P(A and B) / P(B) for P(B) > 0, and
Bayes' theorem simply rearranges it: P(A | B) = P(B | A) P(A) / P(B). The theorem is
arithmetically trivial and conceptually the most misused result in the field, because
P(A | B) and P(B | A) feel interchangeable and are not. The probability that a test
is positive given disease is a property of the test; the probability of disease given
a positive test depends on how common the disease is. For a test with 99%
sensitivity and 99% specificity applied to a disease with 0.1% prevalence, a positive
result carries only about a 9% chance of disease, because the false positives drawn
from the enormous healthy population swamp the true positives from the tiny sick one.


statistics_and_inference

Key discrete distributions and what generates them:
  Bernoulli(p)          single binary trial; mean p, variance p(1-p)
  Binomial(n, p)        count of successes in n independent trials; mean np,
                        variance np(1-p)
  Poisson(lambda)       count of events in fixed time at constant rate; mean =
                        variance = lambda (equidispersion; real count data is
                        usually overdispersed, motivating negative binomial)
  Geometric(p)          trials until first success; memoryless
  NegBinomial(r, p)     trials until r-th success; Poisson-Gamma mixture, the
                        standard overdispersed count model
  Multinomial(n, p_vec) categorical counts across k cells


statistics_and_inference

Key continuous distributions:
  Uniform(a, b)         maximum entropy on a bounded interval
  Normal(mu, sigma^2)   maximum entropy for fixed mean and variance; CLT limit
  Exponential(lambda)   waiting time at constant hazard; memoryless
  Gamma(k, theta)       sum of k exponentials; conjugate prior for Poisson rate
  Beta(a, b)            distribution on [0,1]; conjugate prior for a proportion
  ChiSquare(k)          sum of k squared standard normals; sampling distribution
                        of variance
  Student-t(nu)         normal with estimated variance; heavier tails, converging
                        to normal as nu grows
  Cauchy                t with nu = 1; undefined mean, infinite variance


statistics_and_inference

Q: X and Y are random variables with Var(X) = 4, Var(Y) = 9, and Corr(X, Y) = 0.5.
What is Var(X - Y)?
A. 13   B. 7   C. 19   D. 5
Reasoning: Cov(X, Y) = Corr * sd(X) * sd(Y) = 0.5 * 2 * 3 = 3. Var(X - Y) = Var(X) +
Var(Y) - 2Cov(X, Y) = 4 + 9 - 6 = 7. The sign flip on the covariance term is what
separates this from Var(X + Y) = 19. Answer: B.


statistics_and_inference

Q: A screening test has 95% sensitivity and 90% specificity. Disease prevalence is
2%. A patient tests positive. The probability they have the disease is closest to:
A. 95%   B. 62%   C. 16%   D. 2%
Reasoning: Out of 10,000 people, 200 have the disease and 9,800 do not. True
positives: 0.95 * 200 = 190. False positives: 0.10 * 9,800 = 980. Positive predictive
value = 190 / (190 + 980) = 190 / 1170 = 16.2%. The specificity looks high but is
applied to a base 49 times larger, so false positives dominate. Answer: C.


statistics_and_inference

## Estimation

An estimator is a function of the data used to infer a population parameter. Its
quality is judged on bias (does its expectation equal the truth?), variance (how much
does it move across samples?), and consistency (does it converge to the truth as n
grows?). Mean squared error decomposes exactly as MSE = bias^2 + variance, and this
decomposition is the single most useful idea transferred from statistics into machine
learning, where it becomes the bias-variance tradeoff. An unbiased estimator is not
automatically preferable: the James-Stein estimator is biased and dominates the
unbiased sample mean in three or more dimensions, and ridge regression deliberately
accepts bias to buy a larger reduction in variance.


statistics_and_inference

Maximum likelihood estimation picks the parameter value maximizing the probability of
the observed data. Under regularity conditions the MLE is consistent, asymptotically
normal, and asymptotically efficient — it attains the Cramer-Rao lower bound, so no
unbiased estimator has smaller asymptotic variance. Its asymptotic covariance is the
inverse Fisher information, which is why the inverse Hessian of the negative
log-likelihood at the optimum gives standard errors. The regularity conditions matter:
MLE misbehaves when the parameter is on a boundary (a variance component estimated at
exactly zero), when the support depends on the parameter (the MLE for the upper limit
of a uniform is the sample maximum, which is biased and not asymptotically normal),
and when the number of parameters grows with n.


statistics_and_inference

The method of moments equates sample moments to population moments and solves. It is
usually less efficient than MLE but often has a closed form where MLE requires
numerical optimization, and it provides good starting values. Generalized method of
moments extends this to more moment conditions than parameters, weighting them
optimally, and is the workhorse of econometrics where a full likelihood is often
unavailable or untrustworthy.


statistics_and_inference

A confidence interval is a procedure, not a statement about a particular interval. A
95% CI means that the procedure, applied repeatedly to fresh samples, produces
intervals containing the true parameter 95% of the time. Once computed, a specific
interval either contains the parameter or does not; there is no probability left. The
Bayesian credible interval is the object people usually intend — given the data and a
prior, there is a 95% posterior probability the parameter lies inside. The two often
coincide numerically with flat priors and large samples, which lets the distinction be
ignored right up until it matters.


statistics_and_inference

Standard error formulas worth having memorized:
  mean, known sigma            SE = sigma / sqrt(n)
  mean, estimated sigma        SE = s / sqrt(n), use t with n-1 df
  proportion                   SE = sqrt(p(1-p)/n)
  difference of means          SE = sqrt(s1^2/n1 + s2^2/n2)  (Welch, unequal var)
  difference of proportions    SE = sqrt(p1(1-p1)/n1 + p2(1-p2)/n2)
  regression slope             SE = s_resid / (s_x * sqrt(n-1))
  correlation (Fisher z)       z = atanh(r), SE = 1/sqrt(n-3)


statistics_and_inference

The bootstrap replaces analytic standard errors with resampling. Draw B samples of
size n with replacement from the observed data, recompute the statistic on each, and
use the spread of the resulting distribution as the sampling distribution. It works
for statistics with no tractable analytic error — medians, trimmed means, ratios,
the difference between two AUCs — and requires only that the statistic be a smooth
functional of the empirical distribution. It fails at the boundary: bootstrapping the
sample maximum does not work, because resampling can never produce a value above the
observed maximum. For dependent data the naive bootstrap destroys the dependence
structure and understates uncertainty; the block bootstrap resamples contiguous
blocks to preserve it.


statistics_and_inference

[user] I have conversion rates from two variants: A converted 340 of 4,821 and B
converted 401 of 4,776. Give me the lift with a confidence interval.
[assistant] tool_call stats.prop_test({"x": [340, 401], "n": [4821, 4776], "conf": 0.95})
[tool] {"p1": 0.07053, "p2": 0.08396, "diff": 0.01343, "ci": [0.00292, 0.02394],
"z": 2.505, "p_value": 0.01224}
[assistant] Variant B converts at 8.40% versus A at 7.05% — an absolute lift of 1.34
percentage points, 95% CI [0.29, 2.39] pp. That is a relative lift of about 19%, and
the interval excludes zero (p = 0.012), so the difference is statistically
significant at the conventional threshold.


statistics_and_inference

Two cautions before you act on it. First, the relative-lift interval is much wider
than the absolute one — dividing by a small baseline amplifies uncertainty, so the
plausible range on "19% better" runs roughly from 4% to 34%. Second, this test assumes
these were the only comparison you made and the sample size was fixed in advance. If
you have been peeking at the results daily, or this is one of several metrics you
checked, the nominal p-value understates your false-positive rate and you need a
sequential or multiplicity correction.


statistics_and_inference

## Hypothesis Testing

A hypothesis test fixes a null hypothesis, computes a statistic whose distribution
under that null is known, and reports how extreme the observed value is. The p-value
is the probability, under the null, of a result at least as extreme as the one
observed. It is not the probability the null is true, not the probability the result
was due to chance, and not a measure of effect size. A p-value of 0.001 from a huge
sample can accompany a completely trivial effect, and a p-value of 0.2 from a small
one can hide a large effect the study lacked power to detect.


statistics_and_inference

The two error types trade off. A Type I error rejects a true null (rate alpha, set by
convention at 0.05); a Type II error fails to reject a false null (rate beta). Power
is 1 - beta, the probability of detecting an effect that is really there. Power
increases with sample size, effect size, and alpha, and decreases with variance. The
standard rule of thumb for a two-sample comparison of means is that detecting an
effect of d standard deviations at 80% power and alpha 0.05 requires roughly 16/d^2
observations per group — so d = 0.5 needs about 64 per group, d = 0.2 needs about
400. Running an underpowered study is worse than running none, because it not only
usually misses the effect but, on the occasions it does reach significance,
necessarily overstates the magnitude (the winner's curse, or Type M error).


statistics_and_inference

Common tests and when each applies:
  one-sample t          mean vs a fixed value, roughly normal or large n
  two-sample t (Welch)  two independent means, unequal variances (the default;
                        Student's pooled version needs equal variance and buys
                        very little)
  paired t              same units measured twice; tests mean difference, far more
                        powerful than the unpaired version when pairing is real
  Mann-Whitney U        two independent samples, ordinal or non-normal; tests
                        stochastic dominance, not means
  Wilcoxon signed-rank  paired nonparametric analogue
  chi-square            association in a contingency table, expected counts >= 5
  Fisher exact          same, valid for small counts
  ANOVA / F-test        3+ group means; significance says only "not all equal"
  Kruskal-Wallis        nonparametric ANOVA analogue
  Kolmogorov-Smirnov    whole-distribution comparison


statistics_and_inference

Multiple comparisons destroy naive error control. Running 20 independent tests at
alpha = 0.05 gives roughly a 64% chance of at least one false positive (1 - 0.95^20).
Bonferroni divides alpha by the number of tests: simple, valid under any dependence,
and very conservative. Holm-Bonferroni is uniformly more powerful and equally valid.
Benjamini-Hochberg controls the false discovery rate — the expected proportion of
rejections that are false — rather than the family-wise error rate, and is the right
choice for screening contexts like genomics or feature selection where you expect
many true effects and can tolerate some false ones.


statistics_and_inference

Q: A study with 80% power at alpha = 0.05 fails to reject the null. The correct
interpretation is:
A. The null hypothesis is true
B. There is a 20% chance the null is true
C. The data are consistent with the null, but effects smaller than the study was
   powered to detect remain plausible
D. The study should be repeated until significance is reached
Reasoning: Failing to reject is not evidence of absence; it means the data did not
provide enough evidence against the null at the chosen threshold. The 80% power was
computed for a specific alternative effect size, so smaller effects were always
likely to be missed. Option D describes p-hacking, which invalidates error control.
The honest report pairs the non-significant test with a confidence interval showing
which effect sizes remain compatible with the data. Answer: C.


statistics_and_inference

Q: You run 40 independent hypothesis tests at alpha = 0.05 and obtain 4 significant
results. Under Benjamini-Hochberg at FDR = 0.05, roughly how many of your rejections
would you expect to be false, if all 4 survive the procedure?
A. 4   B. 2   C. 0.2   D. 0
Reasoning: FDR control at 0.05 bounds the expected proportion of false discoveries
among rejections at 5%. With 4 rejections, the expected number of false ones is at
most 0.05 * 4 = 0.2. Note this is an expectation over repeated experiments, not a
guarantee about this particular set. Answer: C.


statistics_and_inference

## Regression

Ordinary least squares fits y = X beta + epsilon by minimizing the sum of squared
residuals, giving the closed form beta_hat = (X'X)^{-1} X'y. It is the best linear
unbiased estimator under the Gauss-Markov conditions: linearity in parameters, random
sampling, no perfect collinearity, zero conditional mean of the errors, and
homoskedasticity. Normality of errors is not required for unbiasedness or for the
Gauss-Markov result — it is required only for exact small-sample t and F
distributions, and the CLT covers that asymptotically.


statistics_and_inference

Each condition fails in a characteristic way. Heteroskedasticity leaves the
coefficients unbiased but breaks the standard errors, fixed by using
heteroskedasticity-consistent (White / HC3) errors, which cost nothing and should
arguably be the default. Correlated errors — time series, repeated measures, clustered
sampling — likewise leave coefficients unbiased but badly understate standard errors;
cluster-robust errors or an explicit mixed model are the remedies, and ignoring
clustering is probably the most common inference error in applied work. Endogeneity,
where a regressor correlates with the error because of omitted variables, simultaneity,
or measurement error, biases the coefficients themselves and no standard-error fix
touches it; instrumental variables, fixed effects, or a design change are required.


statistics_and_inference

Multicollinearity inflates standard errors without biasing coefficients. The variance
inflation factor VIF_j = 1/(1 - R_j^2), where R_j^2 comes from regressing predictor j
on all the others, quantifies it; values above 5 or 10 are conventional warning
thresholds. The important nuance is that multicollinearity is not a problem for
prediction at all — the fitted values are fine — it only makes individual coefficients
unstable and hard to interpret. If the goal is forecasting, high VIF can be ignored;
if the goal is attributing effect to a specific variable, it is fatal.


statistics_and_inference

Diagnostics and what they reveal:
  residuals vs fitted        nonlinearity (curvature), heteroskedasticity (funnel)
  Q-Q plot of residuals      tail behaviour, outliers
  scale-location             heteroskedasticity more clearly than resid-vs-fitted
  Cook's distance            influential points; > 4/n warrants a look
  leverage (hat values)      unusual predictor combinations; > 2p/n is high
  Durbin-Watson              first-order autocorrelation; ~2 is clean
  VIF                        collinearity
  partial residual plots     the functional form for one predictor, others held


statistics_and_inference

Generalized linear models extend OLS by pairing a distribution from the exponential
family with a link function connecting the linear predictor to the mean. Logistic
regression uses a binomial family with a logit link, so coefficients are log odds
ratios: exponentiating gives the multiplicative change in odds per unit predictor.
Poisson regression uses a log link, so coefficients are log rate ratios; because real
count data is almost always overdispersed, the Poisson standard errors are typically
too small and quasi-Poisson or negative binomial is the safer default. GLMs are fit by
iteratively reweighted least squares maximizing the likelihood, and their residuals
(deviance, Pearson) need care — the raw response residuals are not interpretable the
way OLS residuals are.


statistics_and_inference

Regularization adds a penalty to the loss. Ridge (L2) shrinks coefficients smoothly
toward zero, never reaching it, and handles collinearity gracefully by splitting the
effect among correlated predictors. Lasso (L1) drives coefficients exactly to zero,
performing selection, but among a group of correlated predictors it arbitrarily keeps
one and discards the rest, making the selected set unstable across resamples. Elastic
net mixes both and recovers grouped selection. All three require standardized
predictors, because the penalty is scale-dependent, and none of them produce valid
conventional standard errors — post-selection inference is a genuinely hard problem
and the naive p-values from a lasso-selected model are meaningless.


statistics_and_inference

[user] My logistic regression has a coefficient of 0.83 for a binary treatment
variable and the model has an AUC of 0.71. Explain both numbers to a product manager.
[assistant] tool_call stats.glm_summary({"model": "logit", "term": "treatment"})
[tool] {"estimate": 0.83, "std_error": 0.19, "z": 4.37, "p_value": 1.2e-05,
"odds_ratio": 2.293, "ci_or": [1.581, 3.326], "auc": 0.71, "n": 4200}
[assistant] Two different things, and it is worth keeping them apart.


statistics_and_inference

The coefficient 0.83 is on the log-odds scale. Exponentiating gives an odds ratio of
2.29, with a 95% CI of [1.58, 3.33]. Plain language: treated users have roughly 2.3
times the odds of converting compared to untreated, and the data are consistent with
anything from about 1.6x to 3.3x. Note that odds ratios are not risk ratios — if the
baseline conversion rate is low, say 3%, an odds ratio of 2.3 corresponds to a risk
ratio near 2.2 and they are close; but at a 40% baseline the same odds ratio is only
about a 1.5x risk ratio. Always translate to absolute percentage points before
anyone makes a business decision on it.


statistics_and_inference

The AUC of 0.71 is about ranking, not effect size. It says that given one converter
and one non-converter at random, the model scores the converter higher 71% of the
time. 0.5 is a coin flip and 1.0 is perfect. These two numbers are independent in a
useful sense: a single strong predictor can produce a large, highly significant odds
ratio while the model as a whole still ranks individuals only moderately well,
because conversion depends on much more than this one variable. A significant
treatment effect and a mediocre AUC together are exactly what you would expect from a
real but partial driver of behaviour.


statistics_and_inference

## Bayesian Inference

Bayesian inference treats parameters as random variables with distributions
expressing uncertainty. Bayes' theorem updates a prior into a posterior:
posterior proportional to likelihood times prior. Everything follows from that single
mechanic. Where frequentist inference asks "how would this procedure behave over
repeated samples?", Bayesian inference asks "given the data I actually have, what
should I believe?" — which is the question most practitioners think they are asking
anyway.


statistics_and_inference

Conjugate priors make the posterior analytically tractable by staying in the same
family as the prior. Beta-Binomial is the canonical example: with a Beta(a, b) prior
on a probability and s successes in n trials, the posterior is Beta(a + s, b + n - s).
The prior parameters read as pseudo-counts, which makes prior specification concrete —
Beta(1, 1) is uniform and worth one observation of each outcome, Beta(30, 70) encodes
a belief centred at 0.3 with the strength of 100 prior observations. Other conjugate
pairs: Gamma-Poisson for rates, Normal-Normal for a mean with known variance,
Normal-Inverse-Gamma for a mean and variance jointly, Dirichlet-Multinomial for
category probabilities.


statistics_and_inference

For non-conjugate models the posterior has no closed form and is approximated.
Markov chain Monte Carlo constructs a chain whose stationary distribution is the
posterior; Metropolis-Hastings and Gibbs sampling are the classical algorithms, and
Hamiltonian Monte Carlo with the No-U-Turn Sampler (Stan, PyMC, NumPyro) is the modern
default because it exploits gradients to move efficiently through high-dimensional
correlated posteriors. Convergence must be checked, not assumed: R-hat should be below
1.01, effective sample size should be in the hundreds at minimum, trace plots should
look like white noise rather than wandering, and divergent transitions in HMC signal
geometry the sampler cannot navigate — usually a funnel that calls for a
non-centred reparameterization. Variational inference replaces sampling with
optimization over a simpler family, trading exactness for speed; it typically
underestimates posterior variance because the reverse-KL objective is mode-seeking.


statistics_and_inference

Hierarchical models are where the Bayesian machinery earns its keep. When you have
many related groups — stores, users, experiments, sensors — a hierarchical model
estimates each group's parameter while shrinking it toward the population mean, with
the amount of shrinkage determined by the data rather than by fiat. Groups with
little data shrink heavily toward the global estimate; groups with abundant data
barely move. This partial pooling sits between the two bad extremes of complete
pooling (one estimate for everyone, ignoring real heterogeneity) and no pooling
(separate estimates, wildly noisy for small groups) and it is the principled answer
to the ubiquitous problem of ranking entities with unequal sample sizes.


statistics_and_inference

Model comparison in the Bayesian world uses different tools. The Bayes factor is the
ratio of marginal likelihoods and directly compares evidence for two models, but it is
notoriously sensitive to prior specification even where the posterior is not.
Information criteria approximate out-of-sample predictive accuracy: WAIC and PSIS-LOO
are the modern choices, both estimating leave-one-out predictive density with a
complexity penalty, and PSIS-LOO helpfully reports Pareto k diagnostics flagging
observations that dominate the estimate. Posterior predictive checks — simulating
datasets from the fitted model and comparing them to the real one on chosen summary
statistics — are the most practically informative check of all, because they reveal
what the model cannot reproduce.


statistics_and_inference

Q: A Beta(2, 8) prior is updated with 15 successes in 30 trials. The posterior mean
is closest to:
A. 0.50   B. 0.425   C. 0.20   D. 0.475
Reasoning: The posterior is Beta(2 + 15, 8 + 15) = Beta(17, 23). The mean of a
Beta(a, b) is a/(a+b) = 17/40 = 0.425. It sits between the prior mean of 0.2 and the
data's 0.5, pulled toward the data because 30 observations outweigh the prior's 10
pseudo-counts. Answer: B.


statistics_and_inference

Q: An HMC run reports R-hat = 1.34 and 180 divergent transitions. The correct next
step is:
A. Report the posterior means, since the chains did run
B. Increase the number of samples until R-hat falls
C. Diagnose the model geometry — likely reparameterize — before trusting anything
D. Switch to variational inference for speed
Reasoning: R-hat of 1.34 means the chains have not mixed and are exploring different
regions; the posterior summaries are meaningless. Divergences indicate the sampler is
failing in a region of high curvature, classically a hierarchical funnel, and simply
drawing more samples does not fix biased exploration. The standard remedy is a
non-centred parameterization, tighter priors, or both. Answer: C.


statistics_and_inference

## Experimental Design and Causal Inference

Randomization is what licenses causal claims. By assigning treatment independently of
everything else, it makes the treated and control groups exchangeable in expectation
on every covariate — observed and unobserved alike — so the difference in outcomes
estimates the average treatment effect without a model of the confounding. No
observational adjustment achieves this, because adjustment can only handle the
confounders you measured and thought of.


statistics_and_inference

The potential outcomes framework makes this precise. Each unit has outcomes Y(1) and
Y(0) under treatment and control; the individual effect is their difference, and the
fundamental problem of causal inference is that only one is ever observed. Randomized
assignment makes the observed group means unbiased for E[Y(1)] and E[Y(0)], so their
difference estimates the average treatment effect. The SUTVA assumption is doing
quiet work here: no interference between units, and only one version of the
treatment. Interference breaks A/B tests in marketplaces and social networks, where
treating one user changes the experience of untreated ones — a supply-side test that
gives some sellers better placement necessarily takes impressions from the control
sellers, so the naive comparison overstates the effect. Cluster randomization or
switchback designs are the usual mitigations.


statistics_and_inference

Practical experiment design decisions:
  unit of randomization   user > session > request; randomize at the level where
                          interference stops and where the metric is measured
  sample size             fix in advance from a power calculation on the minimum
                          detectable effect that would change a decision
  stratification          block on strong pre-treatment predictors to cut variance
  CUPED                   regress out a pre-period covariate; typically 30-50%
                          variance reduction on engagement metrics for free
  guardrail metrics       monitor for harm even when not the primary hypothesis
  sequential testing      always-valid p-values or alpha spending if you will peek
  SRM check               sample ratio mismatch flags assignment bugs before any
                          effect is interpreted


statistics_and_inference

Sample ratio mismatch deserves particular emphasis because it is the highest-yield
diagnostic in industrial experimentation. If a 50/50 split delivers 49.2% / 50.8% on
two million users, a chi-square test rejects overwhelmingly, and that means the
randomization or the logging is broken — bot filtering that correlates with treatment,
a redirect that drops slow clients, a bug where errors default to control. Any effect
estimate from an experiment with SRM is untrustworthy no matter how significant,
because the groups are no longer comparable.


statistics_and_inference

When randomization is impossible, quasi-experimental designs exploit structure.
Difference-in-differences compares the change over time in a treated group to the
change in an untreated one, differencing out both time-invariant group differences
and common time shocks; it stands or falls on the parallel trends assumption, which
should be supported by pre-period data. Regression discontinuity exploits a threshold
rule — a scholarship awarded above a test-score cutoff — and compares units just above
and just below, giving a locally randomized comparison with strong internal validity
but limited external validity. Instrumental variables use a source of variation that
affects treatment but influences the outcome only through it; the exclusion
restriction is untestable and usually the weak point. Synthetic control constructs a
weighted combination of untreated units matching the treated unit's pre-period
trajectory, and is the standard tool for a single treated unit like a state or market.


statistics_and_inference

Confounding, colliders, and mediators all bias estimates, but only confounders are
fixed by adjustment. A confounder causes both treatment and outcome and must be
controlled. A collider is caused by both, and controlling for it induces spurious
association between them — this is why conditioning on hospitalization creates
correlations between unrelated diseases (Berkson's paradox), and why "controlling for
everything available" is not a conservative strategy but an active source of bias. A
mediator lies on the causal path; controlling for it removes the very effect you
wanted to measure. Directed acyclic graphs turn these distinctions into a mechanical
procedure: draw the assumed causal structure, and the back-door criterion tells you
exactly which set of variables to adjust for.


statistics_and_inference

[user] Our A/B test shows a 3% lift in revenue per user but the control group has
51.4% of the traffic out of 1.8M users. Should I ship it?
[assistant] tool_call stats.srm_check({"observed": [925200, 874800], "expected_ratio":
[0.5, 0.5]})
[tool] {"chi_square": 1428.0, "p_value": 4.2e-313, "observed_split": [0.514, 0.486],
"verdict": "sample ratio mismatch"}
[assistant] No — do not ship, and do not interpret the 3% lift at all yet.


statistics_and_inference

You have a sample ratio mismatch. A 51.4/48.6 split on 1.8M users under an intended
50/50 assignment has a p-value on the order of 1e-313. That is not sampling noise; a
50/50 randomizer on this sample size should stay within roughly plus or minus 0.15
percentage points. Something is systematically excluding users from one arm.


statistics_and_inference

The practical implication is that your two groups are no longer exchangeable, so the
3% revenue difference could be entirely composition rather than treatment. The usual
culprits are worth checking in this order: bot or fraud filtering applied
post-assignment and correlated with the treatment, client-side assignment where the
treatment variant loads more slowly and drops users before the exposure event fires,
redirect-based implementations that lose a slice of traffic, and logging joins that
silently drop rows.


statistics_and_inference

Fix the assignment mechanism, verify a clean split on an A/A test, then rerun. If
there is schedule pressure, the one salvageable analysis is to check whether the SRM
is confined to an identifiable segment and whether the effect holds within strata
that show a clean split — but treat that as a hypothesis to re-test, not a result.


statistics_and_inference

## Resampling and Model Validation

Cross-validation estimates out-of-sample performance by partitioning the data,
training on all but one fold and evaluating on the held-out fold, then rotating.
K-fold with k = 5 or 10 is standard; leave-one-out is nearly unbiased but has high
variance and is expensive, and for most models the k = 10 compromise dominates. The
estimate is of the performance of the *procedure*, not of any particular fitted
model, and the folds' scores are not independent, so their standard deviation
understates true uncertainty.


statistics_and_inference

The dominant failure mode in applied cross-validation is leakage: any information
flowing from the validation fold into training. Standardizing features on the full
dataset before splitting leaks the validation set's mean and variance. Imputing
missing values globally leaks. Selecting features by their correlation with the target
on all the data, then cross-validating only the model fit, produces spectacularly
optimistic estimates — this is the classic error that yields 95% accuracy on pure
noise. The rule is that every data-dependent step, including preprocessing and
feature selection, must sit inside the cross-validation loop, which is precisely what
scikit-learn pipelines exist to enforce.


statistics_and_inference

Structured data needs structured splitting. Time series require forward-chaining
validation: train on the past, test on the future, never the reverse, because random
folds let the model see the future and any temporal autocorrelation converts to
inflated scores. Grouped data — multiple rows per patient, per user, per device —
require group-aware splits keeping all of an entity's rows on one side, or the model
memorizes the entity rather than the signal. Rare-outcome data benefits from
stratified folds preserving class balance, without which small folds can end up with
zero positives and undefined metrics.


statistics_and_inference

Choosing a metric matters more than choosing a model in imbalanced settings. With 1%
positives, a model predicting "negative" always achieves 99% accuracy and is useless.
Precision-recall curves and average precision are far more informative than ROC-AUC
when the positive class is rare, because ROC's false-positive-rate axis is normalized
by a huge negative population and stays flattering. For probabilistic forecasts, use
a proper scoring rule — log loss or Brier score — since these are minimized only by
the true probabilities, and follow up with a calibration plot, since a model can rank
perfectly (AUC 1.0) while its probabilities are systematically wrong by a factor of
three.


statistics_and_inference

Metric selection quick reference:
  balanced classification, ranking matters      ROC-AUC
  rare positives, ranking matters               PR-AUC / average precision
  probabilities used downstream                 log loss + calibration curve
  fixed decision threshold, costs asymmetric    expected cost, or F-beta
  regression, outliers meaningful               RMSE
  regression, outliers are noise                MAE, or Huber
  regression, relative error matters            MAPE (breaks near zero), or
                                                RMSLE
  forecasting vs a naive baseline               MASE


statistics_and_inference

Q: A model is trained with feature selection performed on the full dataset before a
10-fold cross-validation of the model fit alone. The resulting CV accuracy is:
A. An unbiased estimate of test performance
B. Optimistically biased, potentially severely
C. Pessimistically biased
D. Unbiased if the number of features is small
Reasoning: The selection step saw every fold's labels, so each validation fold
influenced which features exist. The subsequent CV therefore evaluates on data already
used, and the bias can be extreme — the canonical demonstration selects features from
pure noise and obtains high apparent accuracy. Only nesting the selection inside each
training fold gives an honest estimate. Answer: B.


statistics_and_inference

Q: For a fraud model where positives are 0.2% of transactions, the least informative
headline metric is:
A. Precision at the operating threshold   B. Recall at the operating threshold
C. Overall accuracy   D. Average precision
Reasoning: Predicting "not fraud" for everything yields 99.8% accuracy while catching
zero fraud, so accuracy carries almost no information at this base rate. The other
three all speak directly to performance on the rare class. Answer: C.


synthetic_data_and_simulation

## Why Synthesize, and What It Can and Cannot Fix

Synthetic data is generated rather than collected, and the reasons to use it fall into a few
distinct categories that call for different methods. Scarcity — the real data does not exist in
sufficient quantity, particularly for rare events. Cost — collecting or labelling is expensive
while generating is cheap. Privacy — the real data cannot be shared. Coverage — the real data
lacks the edge cases the system must handle. Control — an experiment requires knowing the ground
truth exactly, which real data never provides.


synthetic_data_and_simulation

The critical constraint is that synthetic data contains only the information put into the generator.
A generator trained on real data can reproduce and interpolate its patterns; it cannot introduce
patterns absent from its training source. This means synthetic data multiplies the utility of
existing information and does not create new information, and any claim that it solves a data
scarcity problem needs to specify which of these two things is meant.


synthetic_data_and_simulation

The corollary is that synthetic data inherits the biases of its source. A generator trained on a
dataset that underrepresents a population produces synthetic data that underrepresents it, usually
more severely, because generative models allocate capacity to the modes and smooth away the tails.
Using synthesis to address representation problems requires explicitly conditioning generation on
the underrepresented categories rather than expecting the generator to fix the imbalance.


synthetic_data_and_simulation

Motivations and limits:
  scarcity             rare events with insufficient real examples
  cost                 collection or labelling expensive relative to generation
  privacy              real data cannot be shared
  coverage             edge cases absent from real data
  control              exact ground truth for controlled experiments
  the fundamental constraint  a generator contains only the information given to
                       it
  interpolation not creation  patterns can be recombined, not invented
  bias inheritance     source biases are reproduced and usually amplified
  mode collapse toward the majority  generative models smooth away tails
  the representation fix  explicit conditioning, not expecting the generator to
                       correct imbalance
  the honest framing   synthesis multiplies existing information; state which
                       problem it is solving
  hybrid use           synthetic for pretraining and coverage, real for
                       fine-tuning and evaluation; the configuration that works
                       most reliably


synthetic_data_and_simulation

Q: A team synthesizes additional examples of a rare class from a model trained on the existing
imbalanced data, and classifier performance on that class does not improve. Why?
A. The synthesizer is broken
B. The generator learned the rare class from the same few examples the classifier already had, so
   its samples carry no information the classifier lacked; the bottleneck is information, not
   example count
C. Not enough synthetic data was generated
D. The classifier is too small
Reasoning: Generating more samples from a model fit to a handful of real examples produces
variations on those examples. The classifier already had access to that information. Real
additional examples, a generator informed by domain knowledge, or a physics-based simulator can
add information; resampling a fitted model cannot. Answer: B.


synthetic_data_and_simulation

## Generation Methods

Rule-based and procedural generation constructs data from explicit specifications: grammars,
templates, physical models, or programmatic composition. Its advantages are complete control,
guaranteed label correctness, unlimited volume, and the ability to produce configurations that have
never occurred. Its limitation is that it produces only what was specified, so the realism gap is
exactly the gap between the specification and reality.


synthetic_data_and_simulation

Simulation is procedural generation with a physical or behavioural model underneath — rendering
engines for images, physics engines for dynamics, agent models for behaviour, discrete event models
for systems. It is the dominant approach where the underlying process is well understood, and its
quality is bounded by the fidelity of the model rather than by any statistical property.


synthetic_data_and_simulation

Learned generative models fit a distribution to real data and sample from it. Diffusion models
dominate for continuous modalities, autoregressive models for sequences, and each produces samples
that resemble the training distribution closely. Their advantage is realism without needing to
specify the process; their disadvantages are that labels must come from somewhere, that
out-of-distribution generation is unreliable, and that memorization of training examples is a real
possibility with privacy consequences.


synthetic_data_and_simulation

Generation methods:
  rule-based           grammars, templates, programmatic composition
  the advantages       control, correct labels, unlimited volume, novel
                       configurations
  the limitation       only what was specified
  simulation           a physical or behavioural model underneath
  domain suitability   dominant where the process is understood
  fidelity bound       quality limited by the model, not by statistics
  learned generators   fit a distribution and sample from it
  diffusion models     the current default for continuous modalities
  autoregressive models  for sequences and text
  the label problem    generated samples need labels from somewhere
  conditional generation  labels specified as the condition; the usual solution
  out-of-distribution weakness  learned generators extrapolate unreliably
  memorization         training examples reproduced verbatim; a real privacy risk
                       requiring explicit testing
  hybrid pipelines     simulate the structure, learn the appearance; frequently
                       the best of both
  model-based versus data-driven  the choice hinges on whether the process is
                       better understood than it is observed


synthetic_data_and_simulation

## Sim-to-Real Transfer

Training in simulation and deploying in reality is the central problem in robotics and in several
perception domains, and the obstacle is the reality gap — the systematic difference between
simulated and real observations and dynamics. A policy or model that exploits any artifact of the
simulator will fail on contact with reality, and simulators have many artifacts.


synthetic_data_and_simulation

Domain randomization is the technique that works most reliably. Rather than making the simulation
match reality precisely, it randomizes the simulation's parameters — textures, lighting, friction,
masses, sensor noise, latency — across a range wide enough that reality falls somewhere inside it.
The model then cannot rely on any particular parameter value and learns features robust across the
whole range, so reality is just another sample from a distribution it has already seen.


synthetic_data_and_simulation

The complementary approach is system identification: measuring the real system's parameters and
tuning the simulator to match. It produces a more accurate simulator and is limited by what can be
measured and by the aspects of reality the simulator cannot represent at all. In practice both are
used — identify what can be identified, randomize what cannot — and the residual gap is closed by
fine-tuning on a small amount of real data.


synthetic_data_and_simulation

Sim-to-real:
  the reality gap      systematic differences in observation and dynamics
  simulator artifacts  a model will exploit them and fail on transfer
  domain randomization  randomize parameters so reality is inside the training
                       distribution
  what to randomize    appearance, physics, sensor characteristics, timing
  the randomization range  wide enough to contain reality, narrow enough to
                       remain learnable
  system identification  measure and match the real parameters
  the combination      identify what you can, randomize what you cannot
  real fine-tuning     a small amount of real data closing the residual gap
  domain adaptation    aligning feature distributions between simulated and real
  the unmodelled residual  aspects reality has that the simulator does not
                       represent at all; randomization does not help with these
  the evaluation rule  measure on real data, always; simulated evaluation of a
                       simulation-trained model measures only self-consistency
  latency and control  timing differences between simulator and hardware are a
                       frequently overlooked component of the gap
  progressive transfer  simulation, then a controlled real setting, then
                       deployment; each stage catching a different failure class


synthetic_data_and_simulation

## Privacy-Preserving Synthesis

Synthetic data is frequently proposed as a privacy solution: generate data resembling the real data
and share that instead. The reasoning is intuitive and the guarantee it provides by default is
none. A generative model fit to real records can memorize and reproduce them, and even without
verbatim reproduction, the synthetic data reflects the training data closely enough that membership
inference — determining whether a particular record was in the training set — is often successful.


synthetic_data_and_simulation

A formal guarantee requires training the generator under differential privacy, which bounds how much
any single record can influence the output. This provides a quantified guarantee at a real utility
cost, and the cost falls unevenly: common patterns survive and rare ones — which are exactly the
records most identifiable and often most interesting — are suppressed. That suppression is the
privacy mechanism working as intended and it is also a limitation on what the synthetic data can be
used for.


synthetic_data_and_simulation

Empirical privacy evaluation should accompany any synthetic release regardless of the method.
Nearest-neighbour distance between synthetic and real records detects near-duplicates. Membership
inference attacks estimate whether training membership is detectable. Attribute inference tests
whether sensitive fields can be predicted from released ones. A synthetic dataset that has not been
tested this way is a claim rather than a guarantee.


synthetic_data_and_simulation

Privacy synthesis:
  the intuition        share generated data instead of real data
  the default guarantee  none
  memorization         verbatim reproduction of training records
  membership inference  detecting whether a record was in the training set
  differential privacy  a formal bound on any single record's influence
  the utility cost     real, and it falls on rare patterns
  the uneven suppression  rare records are both the most identifiable and often
                       the most interesting
  the privacy-utility frontier  the honest way to present the tradeoff
  nearest-neighbour testing  detects near-duplicate synthetic records
  attribute inference  predicting sensitive fields from released ones
  the testing obligation  untested synthesis is a claim, not a guarantee
  regulatory position  synthetic data is not automatically outside data
                       protection scope; the determination depends on whether
                       individuals remain identifiable
  the practical recommendation  formal guarantees where the data is sensitive,
                       empirical testing always, and explicit documentation of
                       what was and was not verified


synthetic_data_and_simulation

## Evaluation and Failure Modes

Evaluating synthetic data means answering three separate questions, and conflating them is the
usual source of overclaiming. Fidelity asks whether the synthetic data resembles the real
distribution. Utility asks whether models trained on it perform well on real data. Privacy asks
whether it leaks information about individuals. High fidelity does not imply high utility — a
generator can match marginal distributions while destroying the correlations a model needs — and
high utility does not imply privacy.


synthetic_data_and_simulation

Utility is the measure that matters most for machine learning applications, and the definitive test
is direct: train on synthetic, evaluate on real, and compare against training on real. The reverse
comparison — train on real, evaluate on synthetic — measures whether the synthetic data is
representative and is a useful complementary check. Both should be run because they fail
differently.


synthetic_data_and_simulation

The failure modes worth screening for recur across domains. Mode collapse produces samples covering
only part of the distribution, which is invisible in aggregate statistics and obvious in a coverage
analysis. Distributional shift means the synthetic data differs systematically in ways that matter
downstream. Correlation destruction means the marginals are right and the joint structure is wrong.
Label noise from an imperfect generation process silently caps achievable accuracy. And the most
insidious is the feedback loop: training a generator on data that included previous synthetic
output progressively narrows the distribution over generations.


synthetic_data_and_simulation

Evaluation and failures:
  three questions      fidelity, utility, privacy; distinct and often conflated
  fidelity             resemblance to the real distribution
  utility              performance on real data after training on synthetic
  privacy              leakage about individuals
  the definitive utility test  train synthetic, evaluate real, compare with real
                       training
  the reverse test     train real, evaluate synthetic; checks representativeness
  marginal comparison  necessary and far from sufficient
  joint structure      correlations and interactions; where marginal-matching
                       generators fail
  coverage analysis    whether the full support is represented
  mode collapse        partial coverage; invisible in aggregate statistics
  distributional shift  systematic difference in a downstream-relevant direction
  label noise          an imperfect generation process capping achievable accuracy
  discriminability     whether a classifier can separate real from synthetic; a
                       useful single-number fidelity proxy
  the feedback loop    generators trained on prior synthetic output progressively
                       narrowing the distribution
  provenance tracking  recording what is synthetic and from what; the practical
                       defence against that loop and increasingly necessary as
                       generated content accumulates in the corpora that future
                       models are trained on


time_series

## Components, Stationarity, and Structure

A time series is a sequence of observations ordered in time, and the ordering is what makes
it a distinct subject. Observations are not independent — that is the entire point — so
every technique built on the independence assumption needs modification or replacement.


time_series

Decomposition separates a series into interpretable components. Trend is the long-run
direction. Seasonality is periodic variation at a fixed known frequency — day of week,
month of year, hour of day — and a series can carry several seasonal periods
simultaneously. Cyclical variation is longer-term oscillation without a fixed period, such
as business cycles, and is distinguished from seasonality precisely by that irregularity.
The remainder is what the components do not explain.


time_series

The additive versus multiplicative distinction matters practically. In an additive
decomposition the seasonal swing has constant absolute magnitude; in a multiplicative one it
scales with the level. Retail sales usually behave multiplicatively — a December uplift is
a percentage, not a fixed number of units — and applying an additive model produces
systematic errors that grow with the level. Taking logs converts a multiplicative structure
to additive, which is why log transforms are so common in this field.


time_series

Stationarity is the property most classical methods require. A series is weakly stationary
when its mean, variance, and autocovariance structure do not change over time. Most real
series are not: they trend, their variance grows with their level, their seasonal pattern
evolves. Differencing removes a trend, seasonal differencing removes a seasonal pattern,
and a log or Box-Cox transform stabilizes variance. The number of differences required is
the integration order, and over-differencing is a real error that introduces artificial
negative autocorrelation and inflates forecast variance.


time_series

Core concepts and diagnostics:
  autocorrelation (ACF)    correlation with lagged values; decays slowly for a
                           trending series, spikes at seasonal lags
  partial autocorr (PACF)  correlation at a lag after removing shorter lags;
                           identifies AR order
  white noise              no autocorrelation at any lag; what residuals should be
  random walk              each value is the last plus noise; ACF decays very
                           slowly, and the best forecast is the last observation
  ADF / KPSS tests         stationarity tests with opposite null hypotheses; run
                           both, since they disagree informatively
  differencing             y_t - y_{t-1};  removes a linear trend
  seasonal differencing    y_t - y_{t-m};  removes a seasonal pattern of period m
  Ljung-Box test           tests whether residual autocorrelation is jointly zero
  spurious regression      two independent trending series correlate strongly;
                           the classic trap that motivates stationarity


time_series

Spurious regression deserves emphasis because it produces confident nonsense. Two
independent random walks will typically show a high R-squared and a highly significant
coefficient when regressed on each other, because both are trending and the standard errors
computed under independence are badly wrong. This is why regressions on levels of
non-stationary series are untrustworthy and why the analysis is done on differences, or
within a cointegration framework when the series genuinely share a long-run relationship.


time_series

Q: A series' ACF decays very slowly and its PACF has a single large spike at lag 1. The
series most plausibly:
A. Is white noise
B. Requires differencing; it behaves like a random walk or strongly autoregressive process
C. Has strong seasonality at lag 1
D. Is over-differenced
Reasoning: Slow ACF decay indicates non-stationarity or a near-unit-root autoregressive
process; a single PACF spike at lag 1 with slow ACF decay is the signature of an AR(1) with
coefficient near one, which is effectively a random walk. Differencing once should produce a
stationary series with no remaining structure. Over-differencing shows the opposite pattern:
a large negative ACF at lag 1. Answer: B.


time_series

## Classical Forecasting Models

Exponential smoothing forecasts by weighting past observations with exponentially decaying
weights, so recent data matters more. Simple exponential smoothing handles a level with no
trend or seasonality, using one smoothing parameter. Holt's method adds a trend component
with a second parameter. Holt-Winters adds seasonality with a third, in additive or
multiplicative form. The family is organized systematically by the error, trend, and
seasonal specification, giving a set of models from which one can be selected by information
criterion.


time_series

Damped trend variants deserve specific mention because they are consistently better in
practice. An undamped linear trend extrapolates indefinitely, which produces absurd
long-horizon forecasts — a product growing at last quarter's rate for five years. Damping
the trend so it flattens toward the horizon is more realistic and empirically improves
accuracy across large forecasting competitions, and it should generally be the default.


time_series

ARIMA models the series as a combination of autoregressive terms (regression on its own
past values), differencing for integration, and moving average terms (regression on past
forecast errors). Seasonal ARIMA adds the same three components at the seasonal lag. Order
selection was traditionally done by inspecting ACF and PACF plots and remains a useful
diagnostic skill, though automated selection by information criterion is now standard and
performs comparably.


time_series

Classical model families:
  naive                 forecast equals the last value; the benchmark every model
                        must beat, and frequently does not for financial series
  seasonal naive        forecast equals the value from the same season last cycle;
                        a strong benchmark for seasonal data
  drift                 naive plus the average historical change
  simple exp smoothing  level only; one parameter
  Holt                  level and trend
  Holt-Winters          level, trend, seasonality; additive or multiplicative
  damped trend          flattens the extrapolated trend; usually better
  ARIMA(p,d,q)          autoregressive, integrated, moving average
  SARIMA                plus seasonal terms at lag m
  ARIMAX / dynamic reg  ARIMA errors around a regression on external predictors
  TBATS                 multiple and non-integer seasonal periods
  state space / structural  components as latent states; handles missing data and
                        irregular sampling naturally


time_series

The benchmark discipline is the most important practice in the field and the most
frequently skipped. Any forecasting model must be compared against naive, seasonal naive,
and a simple exponential smoothing baseline on the same evaluation protocol. Complex models
routinely fail to beat seasonal naive on real data, and a project that reports only its own
model's error has not established that the model is worth deploying. For many financial
series the naive forecast is genuinely difficult to beat, which is informative about the
series rather than about the modeller.


time_series

## Evaluation

Evaluation of forecasts requires a protocol respecting time. Random cross-validation
splits are invalid because they train on the future to predict the past, and any temporal
autocorrelation converts that leakage into an inflated score. The correct approach is
rolling-origin evaluation: train on data up to a point, forecast the following horizon,
score it, advance the origin, and repeat. This produces many forecasts at each horizon and
reflects how the model will actually be used.


time_series

The forecast horizon must match the decision. A model evaluated on one-step-ahead accuracy
tells you nothing about its twelve-step performance, and the two can rank models
differently — a model capturing short-run momentum wins at one step and loses at twelve
where the level matters more. Accuracy should be reported per horizon, not averaged into a
single number that obscures where the model is useful.


time_series

Metric choice interacts with the data's properties in ways that cause real mistakes.
Percentage errors are undefined at zero and explode near it, which makes them unusable for
intermittent demand. They are also asymmetric, penalizing over-forecasts more than
under-forecasts, which biases model selection toward under-forecasting. Scale-dependent
metrics cannot be aggregated across series with different magnitudes.


time_series

Forecast evaluation:
  rolling origin         train, forecast, advance; the only valid protocol
  per-horizon reporting  accuracy at h=1 says nothing about h=12
  MAE                    scale-dependent, robust, optimal for the median
  RMSE                   scale-dependent, penalizes large errors, optimal for the
                         mean
  MAPE                   scale-free, undefined at zero, asymmetric; avoid despite
                         its popularity
  sMAPE                  symmetric variant; still unstable near zero
  MASE                   error scaled by the naive method's in-sample error;
                         scale-free, defined at zero, interpretable — below 1 beats
                         naive. The best general default.
  pinball loss           for quantile forecasts; the proper scoring rule
  CRPS                   for full predictive distributions
  coverage               fraction of actuals within the prediction interval;
                         should match the nominal level


time_series

The choice between MAE and RMSE is not stylistic: they are minimized by different
predictions. MAE is minimized by the conditional median and RMSE by the conditional mean, so
for a skewed distribution they call for different forecasts. If the business cost of error
is roughly linear, optimize MAE; if large errors are disproportionately costly, optimize
RMSE. Reporting one and optimizing the other produces a model that is systematically
mistuned for its own evaluation.


time_series

Prediction intervals matter more than point forecasts for most decisions and are the part
most often wrong. Intervals from classical models assume the model is correctly specified
and the parameters known, so they systematically understate uncertainty — empirical coverage
of nominal 95 percent intervals is frequently near 80. Intervals should be validated against
actual coverage on held-out data, and if they are too narrow, widened empirically or
replaced with conformal methods that provide distribution-free coverage guarantees.


time_series

[user] Our demand forecast has a MAPE of 18% and the business says it's unusable. The model
looks fine to me.
[assistant] tool_call ts.forecast_audit({"metric": "mape", "value": 0.18,
"series_count": 2400, "zero_fraction": 0.31, "intermittent": true,
"benchmark_seasonal_naive_mape": 0.16, "horizon_days": 14, "eval": "random_split"})
[tool] {"issues": ["mape_unsuitable_for_intermittent", "31pct_zeros_excluded_from_metric",
"underperforms_seasonal_naive", "random_split_invalid"], "mase_model": 1.09,
"mase_seasonal_naive": 1.0, "verdict": "model_worse_than_naive_metric_hid_it"}
[assistant] The business is right and the metric is hiding it. Three problems compound.


time_series

MAPE is the wrong metric for your data. Thirty-one percent of your series-periods are zero,
and MAPE is undefined there, so those observations are silently dropped from the
calculation. You are therefore reporting 18% error on the 69% of cases where demand was
nonzero — which are the easy cases — and reporting nothing at all about the zeros, which for
intermittent demand are most of the forecasting problem.


time_series

On MASE, which is defined at zero and scaled to the naive benchmark, your model scores 1.09
against seasonal naive's 1.0. Above one means worse than the benchmark. You have a model
that is slightly worse than repeating last week's value, and MAPE presented it as an 18%
error which sounded reasonable.


time_series

Your evaluation protocol is also invalid. A random split trains on future periods to predict
past ones, and with autocorrelated demand that inflates the score. Redo it with
rolling-origin evaluation at your actual 14-day horizon, reporting accuracy per horizon
day rather than averaged.


time_series

What to do differently. Switch to MASE or to a scaled MAE as the headline metric. For
intermittent demand specifically, consider Croston's method or its variants, which model
the demand size and the inter-arrival interval separately rather than trying to forecast a
mostly-zero series directly. And check whether the business decision actually needs a point
forecast at all — for inventory it usually needs a quantile, since the cost of understocking
and overstocking differ, in which case you should be forecasting the relevant quantile
directly with pinball loss and not the mean at all.


time_series

That last point may be the whole issue. A mean forecast optimized for MAPE is close to the
wrong quantity for a stocking decision even when it is accurate.


time_series

## Machine Learning Approaches

Machine learning methods for time series reframe forecasting as supervised learning by
constructing features from the history. Lagged values, rolling statistics over various
windows, calendar features, and external regressors become the feature matrix, and any
standard regressor can be applied. Gradient-boosted trees on well-constructed features are
a very strong baseline and win a large share of forecasting competitions, particularly for
many related series where a single global model can be trained across all of them.


time_series

Feature construction for this framing has one dominant rule: every feature must be computable
at forecast time using only information available then. A rolling mean must be lagged so it
does not include the target period. A feature derived from a variable that is itself
forecast must account for that variable's forecast error. Violating this produces
spectacular backtest results and production failure, and it is the single most common error
in machine learning applied to forecasting.


time_series

The global versus local model distinction is where machine learning offers a genuine
advantage over classical methods. Classical models are fit per series, so a series with
short history cannot be modelled well. A global model trained across thousands of related
series learns shared patterns and transfers them, which is why global models dominate in
retail and demand forecasting where there are many series each with limited history. Series
identity enters as a categorical feature or an embedding.


time_series

Approaches and where each fits:
  gradient boosting     on lag and calendar features; the strongest general
                        baseline, especially for many related series
  linear with features  fast, interpretable, surprisingly competitive
  RNN / LSTM            sequential; largely superseded
  temporal convolution  dilated causal convolutions; parallel training, long
                        receptive field
  transformer variants  attention over the sequence; strong at scale, and often
                        beaten by simpler methods on modest data
  N-BEATS / N-HiTS      deep architectures with basis expansion; competitive and
                        interpretable in components
  DeepAR                probabilistic global model; outputs distributions
  foundation models     pretrained across many series, zero-shot forecasting;
                        promising and not yet dominant
  hybrid                classical model for structure, ML for the residual


time_series

The consistent empirical finding across competitions deserves stating plainly: simple
methods, properly evaluated, are hard to beat, and combining forecasts beats individual
models reliably. A simple average of several reasonable models almost always outperforms
the best individual one selected on validation data, because model selection overfits the
validation set while averaging reduces variance. The winning entries in large forecasting
competitions have repeatedly been ensembles of relatively simple methods rather than single
sophisticated ones.


time_series

## Hierarchical, Probabilistic, and Anomaly Detection

Hierarchical forecasting arises when series aggregate — products into categories into
regions into totals — and forecasts must be coherent, meaning the parts sum to the whole.
Forecasting each level independently produces incoherent results, since the sum of
independently forecast components will not equal the independently forecast total.


time_series

Three reconciliation strategies exist and the third dominates. Bottom-up forecasts the
lowest level and aggregates, which preserves detail and is noisy where individual series are
sparse. Top-down forecasts the total and disaggregates by historical proportions, which is
stable and loses series-specific information. Optimal reconciliation forecasts every level
independently and then projects the whole set onto the coherent subspace, using the
forecast error covariance to weight the adjustment — which uses information from all levels
and empirically beats both simpler approaches.


time_series

Probabilistic forecasting outputs a distribution rather than a point, and it is what most
decisions actually require. An inventory decision needs a high quantile of demand, not the
mean, because the costs of stockout and overstock are asymmetric. A capacity decision needs
the tail. Quantile regression targets specific quantiles directly using pinball loss;
distributional models output parameters of an assumed family; ensemble and simulation
approaches generate sample paths, which is the most flexible representation since any
functional of the distribution can be computed from samples. Conformal prediction provides
finite-sample coverage guarantees under exchangeability with adaptations for the temporal
setting.