deep_learning

Adam combines momentum with per-parameter step sizes scaled by a running estimate of
the gradient's second moment, making it robust to badly scaled parameters and the
default for transformers. Its bias-correction terms matter early, when the moving
averages initialized at zero would otherwise be badly underestimated. The known
weakness is that L2 penalty added to the loss interacts with the adaptive scaling in a
way that does not implement true weight decay; AdamW decouples the decay, applying it
directly to the weights, and this is now standard for any regularized transformer
training. Adam's memory cost is two extra states per parameter, so a model with P
parameters needs about 12P bytes in fp32 for optimizer state alone — the reason
Adafactor (factored second moments) and 8-bit optimizers exist.


deep_learning

Learning rate schedules and their uses:
  constant                 baselines and debugging only
  step decay               classical vision recipes; abrupt, works well with SGD
  cosine annealing         smooth decay to near zero; the transformer default
  linear warmup            first 1-5% of steps; prevents early divergence when
                           adaptive moments are still poorly estimated
  inverse sqrt             original transformer schedule, tied to step count
  one-cycle                warmup then anneal below the initial rate; fast
                           convergence for fixed short budgets
  cyclical / warm restarts periodically re-raise the rate to escape basins
  ReduceLROnPlateau        reactive; convenient but lags the ideal schedule


deep_learning

The learning rate is the single most important hyperparameter and is worth tuning
before anything else. A practical procedure is the range test: start at a very small
rate and increase it exponentially over a few hundred steps while recording loss, then
plot loss against log rate. The loss falls, reaches a minimum, and then diverges; a
good choice is roughly an order of magnitude below the divergence point, near the
steepest descent of the curve. This costs a couple of minutes and routinely beats
hours of random search.


deep_learning

Gradient clipping bounds the update norm, either per-parameter by value or globally by
the total gradient norm. Global norm clipping at 1.0 is near-universal in language
model training because loss spikes from rare bad batches — a document of repeated
tokens, a numerical edge case — otherwise produce enormous updates that destroy hours
of progress. It is cheap insurance and rarely hurts.


deep_learning

Q: Training a transformer with AdamW diverges in the first 200 steps at a learning
rate that works fine later in training. The standard fix is:
A. Switch to plain SGD
B. Add a linear warmup over the first few hundred steps
C. Reduce the batch size
D. Remove weight decay
Reasoning: Early in training Adam's second-moment estimate is based on very few
samples and is unreliable, so the adaptive denominator can be far too small, producing
enormous effective steps. Warmup ramps the rate from near zero, giving the moment
estimates time to stabilize before full-size steps are taken. This is why essentially
every transformer recipe includes warmup. Answer: B.


deep_learning

Q: You increase batch size from 256 to 1024 and training becomes markedly slower to
converge in epochs. The most likely cause is:
A. The model is too small
B. The learning rate was not scaled up with the batch size
C. Weight decay is too high
D. The data loader is the bottleneck
Reasoning: Larger batches give lower-variance gradients, so each step is more reliable
but you take four times fewer of them per epoch. Without raising the learning rate the
total distance travelled in parameter space per epoch drops sharply. The linear
scaling rule — multiply the rate by the same factor, with warmup — restores
convergence per epoch. Answer: B.


deep_learning

## Regularization and Normalization

Regularization constrains model capacity to improve generalization. Weight decay
penalizes parameter magnitude, biasing toward simpler functions; in the decoupled
AdamW form it is a direct multiplicative shrink applied each step. Dropout randomly
zeroes activations during training with probability p and rescales the survivors by
1/(1-p) so the expected activation is unchanged, which forces redundant
representations and approximates an ensemble over exponentially many subnetworks. It
must be disabled at inference, and forgetting `model.eval()` is one of the most common
sources of mysteriously poor and non-deterministic validation scores.


deep_learning

Data augmentation is usually the highest-leverage regularizer because it injects real
invariances rather than generic smoothness. For images: random crops, flips, colour
jitter, and the mixing augmentations mixup (convex combinations of inputs and labels)
and cutmix. For audio: time and frequency masking, speed perturbation. For text,
augmentation is harder because most edits change meaning, so the practical levers are
back-translation and, increasingly, synthetic paraphrase. The invariance you augment
for should be one the deployed system genuinely faces — augmenting with vertical flips
teaches invariance to a transformation that never occurs in natural photographs and
wastes capacity.


deep_learning

Early stopping halts training when validation performance stops improving, with a
patience window. It is a form of regularization because the optimization path passes
from underfit through well-fit to overfit, and stopping selects a point on that path.
It requires a validation set never used for anything else, and the stopping point must
not then be reported as the test score — that number is optimistically biased by the
selection.


deep_learning

Normalization layers standardize activations to stabilize training. Batch
normalization normalizes across the batch dimension per channel, then applies a
learned scale and shift. It substantially accelerates training but couples examples
within a batch, which causes trouble: small batches give noisy statistics, the
train/eval discrepancy between batch statistics and running averages is a persistent
bug source, and it interacts badly with sequence models of varying length. Layer
normalization normalizes across the feature dimension within each example
independently, removing all batch coupling, and is therefore the transformer standard.
RMSNorm drops the mean-centering and keeps only the scaling by root-mean-square,
saving computation with no measured quality loss, which is why most recent large
language models use it.


deep_learning

Normalization variants at a glance:
  BatchNorm     across (N, H, W) per channel; CNNs, large batches
  LayerNorm     across features per example; transformers, RNNs
  RMSNorm       LayerNorm without mean subtraction; cheaper, LLM default
  GroupNorm     across channel groups per example; small-batch vision
  InstanceNorm  per channel per example; style transfer
  WeightNorm    reparameterizes weights by direction and magnitude


deep_learning

Placement matters as much as choice. Post-norm transformers (normalization after the
residual addition, as in the original paper) need careful warmup and are unstable at
depth. Pre-norm (normalization inside the residual branch, before the sublayer) leaves
a clean identity path from output to input and trains stably at great depth, which is
why every modern large model uses it. The tradeoff is that pre-norm slightly degrades
final quality at equal depth, which some architectures address by adding a final
normalization before the output projection or by sandwiching norms on both sides.


deep_learning

## Initialization and Training Dynamics

Initialization sets the scale of the signal propagating through an untrained network.
Initialize too small and activations shrink layer over layer until the output is
constant; too large and they saturate or overflow. The principle behind the standard
schemes is variance preservation: choose the weight variance so that the variance of
activations, and of gradients, stays roughly constant across layers. Xavier/Glorot
initialization uses variance 2/(fan_in + fan_out), balancing forward and backward
passes, and suits tanh. He/Kaiming initialization uses 2/fan_in, the factor of two
compensating for ReLU zeroing half the inputs, and is correct for ReLU-family
networks. Zero initialization is fatal for weights because it makes every unit in a
layer compute identical outputs and receive identical gradients, so they never
differentiate; biases can safely start at zero.


deep_learning

Modern large models add scale-aware tweaks. Residual branches are often initialized
so their contribution starts near zero — zeroing the last layer of each block, or
scaling by 1/sqrt(2L) for L layers — so the network begins as an identity function and
grows complexity gradually, which markedly improves stability at depth. Embedding
matrices are typically initialized with a small fixed standard deviation such as 0.02
rather than a fan-based rule, since their fan_in is the vocabulary size and a
fan-based scale would be far too small.


deep_learning

Reading training curves is a diagnostic skill. Training loss falling while validation
loss rises is textbook overfitting, calling for more data, more augmentation, or more
regularization. Both flat from the start means the learning rate is wrong (usually too
low, occasionally so high the model sits in a bad region), or the data pipeline is
broken and labels are not aligned with inputs. Both falling but plateauing well above
a sensible floor suggests underfitting: more capacity, longer training, or a better
architecture. A sudden loss spike that recovers indicates a bad batch and argues for
gradient clipping; a spike that does not recover usually means numerical overflow, and
in mixed precision means the loss scale needs adjustment.


deep_learning

The first diagnostic to run on any new setup is the overfit-one-batch test: take a
single minibatch of a handful of examples and train until the loss reaches
approximately zero. A correct model with a correct loss and a correct optimizer can
always memorize a few examples. If it cannot, the bug is in the code, not the
hyperparameters — inputs disconnected from the loss, labels shuffled relative to
inputs, an activation squashing the output range, a frozen parameter group, gradients
not flowing through a detached tensor. This test takes a minute and saves days.


deep_learning

[user] My validation loss is 0.42 and training loss is 0.41 after 30 epochs, and both
have been flat for the last 15. Accuracy is 71% and I need 85%.
[assistant] tool_call ml.training_diagnostics({"train_loss": 0.41, "val_loss": 0.42,
"epochs": 30, "plateau_epochs": 15, "accuracy": 0.71, "target": 0.85})
[tool] {"gap": 0.01, "regime": "underfitting", "signals": ["train_val_gap_negligible",
"both_plateaued", "accuracy_far_below_target"]}
[assistant] You are underfitting, not overfitting — the train/validation gap is 0.01,
essentially nothing. The model is not memorizing your data; it cannot fit it in the
first place. That is good news in one respect: every regularization knob you might
have been tempted to reach for (more dropout, more weight decay, more augmentation)
would make this worse.


deep_learning

Work through these in order, cheapest first:


deep_learning

Confirm the ceiling is not in the data. Run the overfit-one-batch test — take 32
examples and train until loss approaches zero. If it will not, you have a bug, not a
capacity problem, and no amount of scaling will help.


deep_learning

Check the learning rate. A 15-epoch flat plateau is often a rate that has decayed too
far or was too low throughout. Do a range test, and if you are using a schedule that
has annealed to near zero, that alone explains the plateau.


deep_learning

Then add capacity: more layers or width, and train longer. Underfitting with a
negligible generalization gap is the one situation where a bigger model is
straightforwardly the right answer.


deep_learning

Finally, question whether 85% is achievable. If labels are noisy or the features
genuinely do not determine the target, there is an irreducible Bayes error and the
target may simply be unreachable with this input. Have a human label 100 random
examples from the features alone and see what accuracy they reach.


deep_learning

## Quantization and Efficient Inference

Quantization maps high-precision weights and activations to low-bit representations,
cutting memory and bandwidth. Since inference on modern hardware is overwhelmingly
memory-bandwidth-bound rather than compute-bound at batch size one, halving the bits
nearly halves decode latency — the arithmetic was never the bottleneck. A model in
fp16 needs 2 bytes per parameter, so a 30-billion-parameter model needs 60 GB; at 4
bits it needs about 15 GB plus overhead, which is the difference between a datacentre
card and a laptop.


deep_learning

Post-training quantization converts an already-trained model without retraining.
The naive form, round-to-nearest on a per-tensor scale, degrades badly below 8 bits
because a few outlier channels with much larger magnitudes stretch the scale and crush
the resolution available to everything else. The refinements all attack this.
Finer granularity — per-channel, then per-group of 128 or 64 or 32 weights — gives
outliers their own scale, at the cost of storing more scale parameters. Activation-aware
methods observe that not all weights matter equally: the channels that receive large
activations dominate the output error, so scaling those channels up before quantizing
(and folding the inverse scale into the preceding normalization) protects them at no
inference cost. Error-compensating methods quantize weights one column at a time and
push the resulting error into the not-yet-quantized columns using the inverse Hessian
of the layer's input covariance, so later weights partially cancel earlier mistakes.


deep_learning

Quantization-aware training simulates quantization in the forward pass while keeping
high-precision master weights, using a straight-through estimator to pass gradients
through the non-differentiable rounding step. It recovers substantially more quality
than post-training methods at very low bit widths, at the cost of a training run. For
natively low-precision architectures — ternary or binary weight models — there is no
post-hoc quantization error to recover at all, because the released checkpoint already
is the quantized model in a lossless container; the only lever is continued training
with the quantization in the loop.


deep_learning

Quantization concepts and terminology:
  bits per weight (bpw)   the real currency; includes scale/zero-point overhead
  symmetric               zero maps to zero; no zero-point, cheaper arithmetic
  asymmetric              stores a zero-point; better for skewed distributions
                          and essential at 2-3 bits where symmetric grids waste
                          levels
  group size              weights sharing one scale; 128 typical, 32 for low-bit
  calibration set         data used to observe activation statistics; a few
                          hundred representative sequences usually suffices
  mixed precision         sensitive tensors kept at higher precision; attention
                          value projections and the first/last layers commonly
  perplexity delta        the standard quality proxy for language models
  KL divergence           a stricter proxy comparing full output distributions
                          against the full-precision reference


deep_learning

Which tensors to protect is empirically consistent across architectures. The embedding
and output projection matrices are disproportionately sensitive, as are the attention
value and output projections and the down-projection of the feedforward block in early
layers. Practical low-bit recipes therefore ship mixed: a nominally "2-bit" model
typically stores several tensor classes at 3 or 4 bits, and its true bits-per-weight
lands well above the nominal figure. Reporting nominal bit width without the measured
bpw obscures exactly the tradeoff being made.


deep_learning

Beyond quantization, the main inference efficiency levers are structural. KV caching
stores past keys and values so each new token attends without recomputing history,
turning generation from quadratic to linear in total work, at the cost of a cache that
grows linearly with context and often exceeds the weights in memory for long contexts —
which is what grouped-query and multi-query attention address by sharing key/value
heads across query heads. Speculative decoding runs a small draft model to propose
several tokens and verifies them in one forward pass of the large model, accepting the
longest prefix that matches what the large model would have produced; because
verification is parallel and generation is bandwidth-bound, this yields real speedups
with mathematically identical output. Multi-token prediction heads bake the drafter
into the model itself, predicting several future tokens from shared trunk
representations.


deep_learning

Q: A 2-bit quantized model is reported at 2.0 bpw nominal but measures 2.7 bpw on
disk. The most likely explanation is:
A. The file is uncompressed
B. Group scales, zero-points, and higher-precision mixed tensors add overhead
C. The measurement is wrong
D. Padding to block boundaries
Reasoning: A group of 32 weights at 2 bits is 64 bits of payload, but the shared scale
(often fp16) adds 16 more, roughly 0.5 bpw of overhead on its own; and standard
low-bit recipes promote sensitive tensors such as attention value projections and
embeddings to 4 or more bits. Together these routinely account for a 0.5-0.8 bpw gap
between nominal and measured. Answer: B.


deep_learning

Q: At batch size 1, halving the weight precision from 16 to 8 bits roughly halves
decode latency. This is primarily because:
A. Integer arithmetic is twice as fast
B. Decode is memory-bandwidth-bound, and half the bytes move half the time
C. The model has fewer parameters
D. Cache hit rates double
Reasoning: Generating one token requires reading every weight once and performing very
little arithmetic per weight, so the operation is bandwidth-limited rather than
compute-limited. Latency tracks bytes moved, so halving bytes per weight nearly halves
the time. This is also why the speedup shrinks at large batch sizes, where weight
reads amortize across many sequences and the workload becomes compute-bound.
Answer: B.


experiment_platforms

Experimentation Platforms and Causal Measurement

## Randomization Infrastructure

An experimentation platform exists to make causal measurement routine, and most of its value is in
the infrastructure rather than in the statistics. Assignment must be deterministic given the unit
and the experiment, so that a user sees a consistent experience across sessions and devices; it must
be independent across concurrent experiments, so that overlapping tests do not confound each other;
and it must be logged at the point of assignment, so that analysis uses exactly the population that
was assigned.


experiment_platforms

Hashing the unit identifier together with an experiment-specific salt achieves the first two
properties. Determinism follows from the hash being a pure function; independence follows from
different salts producing uncorrelated partitions. Reusing a salt across experiments, or bucketing by
a value correlated with the outcome, produces correlated assignment that is difficult to detect
after the fact and that invalidates everything downstream.


experiment_platforms

The randomization unit is a design choice with consequences. Randomizing by user gives clean
independence and requires stable identification. Randomizing by session is easier and permits the
same user to see both variants, which contaminates any effect that persists across sessions.
Randomizing by cluster — an account, a household, a geography — is required when units within the
cluster influence each other, and it substantially reduces effective sample size, which must be
accounted for in the power calculation rather than discovered afterward.


experiment_platforms

Randomization infrastructure:
  determinism          consistent assignment across sessions and devices
  independence         concurrent experiments uncorrelated
  assignment logging   analysis uses exactly the assigned population
  hashing with a salt  the standard mechanism achieving the first two
  salt reuse           correlated assignment; hard to detect afterward
  the randomization unit  user, session, or cluster
  user-level           clean independence; requires stable identification
  session-level        easier; contaminates persistent effects
  cluster-level        required under within-cluster influence; reduces effective
                       sample size
  the power implication  clustering must enter the sample size calculation
  holdout groups       a persistent unexposed population measuring cumulative
                       effect across many experiments
  sample ratio mismatch  observed allocation differing from intended; the single
                       most important automated check, because it indicates a
                       bug that invalidates the result regardless of how good the
                       numbers look
  the mismatch causes  assignment errors, logging loss correlated with variant,
                       redirect failures, and bot traffic filtered unevenly


experiment_platforms

Q: An experiment intended to split traffic evenly shows fifty-two percent in treatment. The result
looks positive and significant. What should be done?
A. Ship it; the imbalance is small
B. Investigate the imbalance before interpreting anything — a ratio mismatch indicates a systematic
   problem in assignment or logging that usually also biases the outcome, so the estimate cannot be
   trusted
C. Adjust the analysis for the imbalance
D. Rerun with more traffic
Reasoning: Under correct randomization, a two-percent imbalance at any realistic sample size is
astronomically unlikely. Its presence means something is differentially affecting which units are
assigned or recorded, and whatever that mechanism is will generally also affect the metric.
Statistical adjustment does not fix an unknown selection process. Answer: B.


experiment_platforms

## Metrics Design

Metrics determine what an experiment can conclude, and their design is where most of the thinking
belongs. A primary metric captures the intended effect and is decided before the experiment runs.
Guardrail metrics capture harms the change might cause and are monitored regardless of the primary
result. Diagnostic metrics explain mechanism and support interpretation without themselves being
decision criteria.


experiment_platforms

Sensitivity and validity pull in different directions. A metric close to the change — clicks on the
modified element — moves reliably and may not reflect any real benefit. A metric close to the
business outcome — retention, revenue, lifetime value — is what matters and is too noisy and too
slow to detect realistic effects. The standard resolution is a surrogate that is both sensitive and
validated against the long-term outcome, and validating the surrogate is real work that is often
skipped, leaving a metric that is trusted because it moves rather than because it predicts.


experiment_platforms

Metric definition details matter more than they appear to. Whether a ratio is computed per user or
across all events changes what it measures and how its variance behaves. Whether users with zero
activity are included changes the denominator and can reverse a result. Whether outliers are capped
changes both the estimate and its variance substantially for heavy-tailed metrics, and choosing the
cap after seeing the data is a form of the selection problem the experiment was meant to avoid.


experiment_platforms

Metrics design:
  primary metric       the intended effect; decided in advance
  guardrail metrics    harms monitored regardless of the primary
  diagnostic metrics   mechanism and interpretation, not decisions
  the sensitivity-validity tension  proximate metrics move, distal metrics matter
  surrogate metrics    sensitive and validated against the long-term outcome
  the validation gap   surrogates trusted because they move rather than because
                       they predict
  ratio metrics        per user or across events; genuinely different quantities
  the delta method     required for variance of a ratio whose denominator is
                       random
  zero-activity users  inclusion changes the denominator and can reverse a result
  outlier capping      changes estimate and variance for heavy tails
  post-hoc cap selection  a selection problem reintroduced
  multiple metrics     the more you examine, the more will appear significant by
                       chance; decide the primary in advance and treat the rest
                       as exploratory
  novelty and primacy effects  short-term responses to change that fade; a reason
                       to run long enough and to check the effect over time rather
                       than only in aggregate


experiment_platforms

## Variance Reduction and Power

An experiment's ability to detect an effect depends on the effect size, the metric's variance, and
the sample size. Sample size is often fixed by traffic and effect size by reality, so variance
reduction is the available lever, and it can be worth more than a multiple of traffic.


experiment_platforms

Regression adjustment using pre-experiment data is the standard technique. A user's metric value
before the experiment predicts their value during it, so subtracting the predicted portion removes
variance attributable to persistent user differences while leaving the treatment effect intact. The
reduction is substantial for metrics with high between-user variance and persistent behaviour, which
describes most engagement metrics, and it commonly reduces required sample size by half or more.


experiment_platforms

Stratification and blocking achieve related gains by ensuring balance on known predictive
covariates rather than relying on randomization to produce it. Trigger analysis restricts the
population to units that could actually be affected — users who reached the modified surface —
which removes the diluting effect of unaffected units and can multiply sensitivity, at the cost of a
narrower population to which the estimate applies. The trigger condition must be defined identically
in both arms, and defining it by an event that only occurs in treatment is a common and invalidating
mistake.


experiment_platforms

Variance reduction and power:
  the three factors    effect size, variance, sample size
  the available lever  variance, when traffic and effect are fixed
  pre-experiment adjustment  removing persistent user differences
  the typical gain     half or more of the required sample size
  the requirement      pre-period data and a persistent metric
  stratification       balance on predictive covariates by design
  trigger analysis     restrict to units that could be affected
  the dilution removed  unaffected units contributing only noise
  the population narrowing  the estimate applies to triggered users only
  the trigger trap     defining it by an event that only occurs in treatment
  the symmetry requirement  the trigger must be identical in both arms
  minimum detectable effect  what the design can find; compute it before running
  the underpowered experiment  the most common design failure, and its result is
                       uninformative rather than negative
  the interpretation error  reading a non-significant underpowered result as
                       evidence of no effect


experiment_platforms

## Sequential Testing, Interference, and Special Designs

Fixed-horizon tests assume the sample size is set in advance and the result examined once. Repeatedly
checking and stopping when significance appears inflates the false positive rate dramatically —
continuous monitoring of a fixed-horizon test can reach a false positive rate several times the
nominal level. Since experimenters will look at running experiments regardless, the practical answer
is a method that permits it.


experiment_platforms

Sequential methods provide validity under continuous monitoring by adjusting the boundary over time.
Always-valid confidence sequences maintain coverage at every point, group sequential designs specify
interim analysis points with adjusted thresholds, and both trade some power at the fixed horizon for
the ability to stop early. The tradeoff is nearly always worth it in practice, because the ability
to stop a harmful experiment quickly has real value.


experiment_platforms

Interference violates the assumption that one unit's treatment does not affect another's outcome,
and it appears whenever units interact — social networks, marketplaces, shared resources, and any
system with a supply side. A marketplace experiment where treatment users book more inventory leaves
less for control users, so the measured difference overstates the effect that would occur if
everyone were treated. Cluster randomization, switchback designs alternating treatment over time
across the whole system, and explicit two-sided designs are the responses, and each measures
something subtly different from the naive comparison.


experiment_platforms

Sequential and special designs:
  fixed horizon        sample size set in advance, examined once
  peeking              repeated checking inflating false positives severalfold
  the practical reality  experimenters look regardless
  sequential methods   valid under continuous monitoring
  confidence sequences  coverage maintained at every point
  group sequential     specified interim analyses with adjusted thresholds
  the tradeoff         some power at the horizon for the ability to stop early
  interference         one unit's treatment affecting another's outcome
  where it appears     networks, marketplaces, shared resources, supply
                       constraints
  the bias direction   naive comparison overstates when treatment consumes a
                       shared resource
  cluster randomization  treating interacting units together
  switchback design    alternating the whole system over time
  the switchback assumption  no carryover between periods; check it with a
                       washout
  two-sided designs    randomizing both supply and demand
  the honest framing   under interference the naive estimate answers a different
                       question than the one asked, and stating which question is
                       being answered is more useful than adding corrections to a
                       design that does not support the interpretation


experiment_platforms

## Quasi-Experiments and Organizational Practice

Not everything can be randomized. Changes affecting all users simultaneously, decisions already
made, and interventions at the level of a whole market require methods that approximate randomization
from observational data. Difference-in-differences compares changes across treated and untreated
groups and depends on the assumption that they would have moved in parallel absent the treatment,
which should be checked in the pre-period rather than asserted. Synthetic control constructs a
weighted combination of untreated units matching the treated one's pre-period trajectory, which is
more credible for a single treated unit than any single comparison.


experiment_platforms

Regression discontinuity exploits an arbitrary threshold — a score cutoff, a date, a size boundary —
where units just above and just below are comparable in everything except treatment. Interrupted
time series compares before and after with an explicit model of the counterfactual trend. Each of
these carries assumptions that can fail, and the credible applications are those that argue for the
assumptions with evidence rather than asserting them in a methods section.


experiment_platforms

The organizational practice around experimentation determines whether it produces knowledge or
justification. A culture that runs experiments to validate decisions already made will find the
results it wants through metric selection, timing, and segment slicing. The practices that prevent
this are pre-registration of the primary metric and the decision rule, a shared platform that makes
the analysis path uniform, review of results by someone without a stake, and — most importantly —
publishing negative results as prominently as positive ones, because a body of evidence in which
only successes are recorded teaches nothing about the base rate of success.


experiment_platforms

Quasi-experiments and practice:
  when randomization is impossible  system-wide changes, past decisions, market
                       interventions
  difference-in-differences  changes compared across groups
  parallel trends      the assumption; check it in the pre-period
  synthetic control    a weighted combination matching the pre-period trajectory
  regression discontinuity  comparability near an arbitrary threshold
  interrupted time series  an explicit counterfactual trend model
  the standard          argue for the assumptions rather than asserting them
  pre-registration     primary metric and decision rule fixed in advance
  a shared platform    uniform analysis paths reducing analyst degrees of freedom
  independent review   results examined by someone without a stake
  publishing negatives  as prominently as positives
  the base rate        most changes have no detectable effect, and a team that
                       does not know this expects too much and reads noise as
                       signal
  the cumulative view  a holdout measuring the combined effect of everything
                       shipped, which frequently falls well short of the sum of
                       the individual reported effects and is the most sobering
                       and most valuable measurement an experimentation programme
                       can make


experimentation

Experimentation and Causal Inference in Practice

## Why Experiments and What They Measure

Online controlled experiments are the most reliable tool available for establishing that a
change caused an outcome, and their reliability comes entirely from randomization. Random
assignment makes treatment and control groups equivalent in expectation on every attribute —
those measured, those unmeasured, and those nobody thought of — so the difference in outcomes
estimates the causal effect without requiring a model of what else might differ. No
observational adjustment achieves this, because adjustment handles only the confounders you
measured.


experimentation

The consistent empirical finding across large experimentation programmes is humbling: most
changes have no detectable effect, a substantial minority are harmful, and a small fraction
produce the gains. Published figures from several large technology companies converge on
roughly a third of changes being positive, a third neutral, and a third negative. This is the
strongest argument for experimenting: teams cannot reliably predict which of their changes
work, and the ones that seem most obviously beneficial include a meaningful share of the
harmful ones.


experimentation

The unit of randomization must be chosen deliberately. Randomizing by user gives a consistent
experience and captures cumulative effects, and it is the default for anything a user might
notice. Randomizing by session or request gives more units and therefore more power, and it
breaks down when the treatment produces a visibly inconsistent experience or when effects
accumulate across sessions. Randomizing by cluster — geography, account, or social group — is
required when interference between units is present, at a substantial cost in effective
sample size.


experimentation

Experiment design essentials:
  randomization unit    user > session > request; choose where interference stops
  hash-based assignment deterministic from user ID and experiment ID; stable across
                        sessions and independent between experiments
  A/A test              run before launch; validates the assignment and the
                        analysis pipeline, and should show no effect
  sample ratio check    the highest-value diagnostic; a mismatch invalidates
                        everything downstream
  fixed sample size     computed in advance from the minimum detectable effect
  minimum detectable effect  the smallest change that would alter a decision;
                        driven by business value, not by statistical convenience
  randomization seed    independent across experiments to avoid carryover
  exposure logging      log when a user actually encountered the change, not when
                        they were assigned; analysing on assignment dilutes the
                        effect with users who never saw it


experimentation

The distinction between assignment and exposure deserves emphasis because it produces a
consistent bias. If a change appears on a page only ten percent of users visit, analysing all
assigned users dilutes the effect by a factor of ten and may render a real effect
undetectable. Analysing only exposed users introduces a different problem, since exposure may
itself be affected by the treatment — a change making a page load faster increases the number
of users who reach it, so the exposed populations differ. The clean approach is to trigger
the analysis on the first point where treatment and control users are still equivalent and
where the treatment becomes relevant, which requires logging that trigger explicitly.


experimentation

Q: An experiment shows a 2% lift in the primary metric with p = 0.03, and the team wants to
ship. The most important thing to check first is:
A. Whether the effect is larger in any segment
B. Sample ratio mismatch and whether the analysis was pre-specified
C. Whether the p-value would survive a Bonferroni correction
D. The confidence interval width
Reasoning: A sample ratio mismatch means the groups are not comparable and no effect estimate
from the experiment is trustworthy, regardless of significance. Pre-specification matters
because a result selected from many metrics or many analysis choices has a much higher false
positive rate than the nominal p-value suggests. Both are threshold questions that determine
whether the other analyses mean anything. Answer: B.


experimentation

## Metrics

Metric selection determines what an experimentation programme optimizes toward, and choosing
badly produces a system that reliably improves the wrong thing. The primary metric should be
a single measure directly reflecting the hypothesis, decided before the experiment runs.
Secondary metrics explain the mechanism. Guardrail metrics detect harm the primary metric
would not reveal, and they should be monitored on every experiment regardless of hypothesis.


experimentation

The tension between short-term and long-term metrics is the central difficulty. Metrics
measurable within an experiment's duration — clicks, sessions, immediate conversion — are
often poor proxies for what the business cares about, which is retention and lifetime value
over months. A change that increases clicks by making content more sensational may reduce
long-term retention, and an experiment lasting two weeks cannot see that. Approaches include
holdback groups maintained for months, surrogate metrics validated against long-term outcomes
on historical data, and explicit degradation experiments that establish the sensitivity of
long-term metrics to short-term ones.


experimentation

Metric properties worth checking:
  sensitivity          moves enough to detect realistic effects at feasible sample
                       sizes; an insensitive metric is unusable regardless of its
                       relevance
  directionality       an improvement is unambiguously good; metrics where an
                       increase can be good or bad are not decision metrics
  low variance         high variance metrics require enormous samples; the ratio
                       of variance to effect size determines feasibility
  not easily gamed     a metric optimized directly diverges from what it proxied
  computable quickly   available within the experiment window
  guardrails           latency, error rate, unsubscribes, complaints, support
                       contacts; catch harm outside the hypothesis
  count vs ratio       ratio metrics have delta-method variance and cannot be
                       treated as simple means
  capping / winsorizing  heavy-tailed metrics like revenue per user are dominated
                       by outliers; capping reduces variance greatly at a small
                       bias cost


experimentation

Ratio metrics deserve specific mention because they are analysed incorrectly often. A metric
such as clicks per session, computed as total clicks divided by total sessions, is a ratio of
two random quantities rather than an average over independent units. Treating it as a simple
mean and computing a standard error accordingly understates the variance and produces
false positives. The delta method gives the correct variance, accounting for the covariance
between numerator and denominator, and this correction matters most exactly when the treatment
affects the denominator — which is common, since many changes affect session counts.


experimentation

Overall evaluation criteria combine several metrics into a single decision quantity by
weighting them. The value is that it forces the tradeoffs to be made explicitly and in advance
rather than argued after results arrive, when the arguing is contaminated by knowing which
metrics moved. The difficulty is agreeing the weights, which is a business decision rather
than a statistical one and is best made by the people who own the outcome.


experimentation

## Power and Variance Reduction

Statistical power is the probability of detecting an effect that exists, and underpowered
experiments are the most common defect in practice. An underpowered experiment usually fails
to detect a real effect, and worse, on the occasions it does reach significance it
necessarily overstates the magnitude, because only large observed effects clear the threshold
at small sample sizes. Both failures are costly: real improvements are discarded, and the ones
that ship carry inflated expectations.


experimentation

Sample size is determined by the minimum detectable effect, the metric's variance, the
significance level, and the desired power. The relationship worth internalizing is that
required sample size scales with the inverse square of the detectable effect: halving the
effect you wish to detect requires four times the sample. This is why detecting small effects
is so expensive and why the minimum detectable effect should be set by what would change a
decision rather than by optimism.


experimentation

Variance reduction is the highest-leverage technique available, because reducing variance is
equivalent to increasing sample size at no cost in traffic or time. The most effective method
is CUPED — using pre-experiment data as a covariate. Each user's outcome is adjusted by their
pre-period behaviour, removing the component of variance attributable to users simply being
heavy or light users, which is typically the dominant source of variance in engagement
metrics.


experimentation

Power and variance techniques:
  sample size          scales as 1/MDE^2; halving the target effect quadruples
                       the requirement
  CUPED                regress out a pre-experiment covariate; commonly 30-50%
                       variance reduction on engagement metrics, sometimes more
  stratification       block on strong predictors before randomization
  post-stratification  adjust after the fact; nearly as effective and simpler
  capping              limit the influence of extreme values on heavy-tailed
                       metrics
  variance-weighted    weight units by their pre-period variance
  paired designs       where applicable, comparing a unit to itself removes
                       between-unit variance entirely
  longer duration      increases sample but also introduces novelty decay and
                       seasonality
  more traffic         the expensive option; exhaust the others first


experimentation

Novelty and primacy effects complicate duration decisions. A visible change may show an
inflated effect initially because users notice it and explore, or a depressed effect because
they must relearn a familiar interface. Both decay over days to weeks. Running an experiment
only until significance is reached therefore risks measuring the transient rather than the
steady state, and the diagnostic is to plot the effect over time: an effect trending toward
zero or reversing indicates novelty, and an effect that is flat indicates the measurement is
stable.


experimentation

[user] We want to detect a 0.5% change in conversion. Baseline is 4%, we have 200k users per
week. How long do we run?
[assistant] tool_call exp.power_analysis({"baseline_rate": 0.04,
"relative_mde": 0.005, "weekly_users": 200000, "alpha": 0.05, "power": 0.80,
"variance_reduction": 0.0})
[tool] {"absolute_mde": 0.0002, "required_n_per_arm": 15400000,
"total_required": 30800000, "weeks_required": 154, "with_cuped_40pct": 92,
"verdict": "infeasible_as_specified"}
[assistant] Not feasible as specified. A 0.5% relative change on a 4% baseline is an absolute
change of 0.02 percentage points, and detecting that at 80% power needs about 15.4 million
users per arm — roughly three years of your traffic.


experimentation

The binding constraint is that sample size scales with the inverse square of the effect size,
so small relative effects on small baselines are extraordinarily expensive. Even a 40%
variance reduction from CUPED only brings this to about 92 weeks.


experimentation

Three ways forward, and I would push hardest on the first.


experimentation

Reconsider the minimum detectable effect. The right question is not what change you hope for
but what change would alter your decision. If a 0.5% relative lift would not by itself justify
shipping and maintaining the feature, you do not need to detect it. A 3% relative effect needs
about 4 weeks; a 5% needs about 1.5 weeks. Ask what lift would make this worth doing and size
for that.


experimentation

Apply variance reduction regardless. CUPED on a pre-period conversion covariate is close to
free and typically gives 30-50% here. Also check whether capping or a more sensitive proxy
metric is available — sometimes a leading indicator with lower variance detects the same
underlying change far more cheaply, provided you have validated that it tracks conversion.


experimentation

Consider whether conversion is the right primary metric. If the change affects a specific step
in a funnel, measuring that step directly is far more sensitive than measuring end-to-end
conversion, because the intermediate metric has less noise from everything downstream. You
lose the direct business interpretation and gain the ability to detect the effect at all,
which is usually the better trade for an iteration decision.


experimentation

One thing not to do: run it until it happens to be significant. With a fixed sample size not
reached, peeking inflates your false positive rate substantially — repeatedly checking at
p < 0.05 reaches significance eventually on pure noise. If you need to monitor as it runs,
use sequential testing with always-valid p-values or an alpha-spending schedule.


experimentation

## Sequential Testing, Interference, and Pitfalls

The peeking problem is that repeatedly testing accumulating data at a fixed threshold
inflates the false positive rate dramatically. A fixed-sample test at the 5 percent level
assumes a single analysis at a predetermined sample size; checking daily and stopping when
significance appears can push the actual false positive rate above 30 percent, because random
walks cross a threshold eventually.


experimentation

Sequential methods solve this properly. Group sequential designs pre-specify a small number
of interim analyses with adjusted thresholds via an alpha-spending function. Always-valid
inference — sequential probability ratio tests, mixture sequential tests, confidence sequences
— provides bounds valid at every moment, so a practitioner may look continuously and stop
whenever the result is conclusive, at the cost of requiring somewhat more data than a fixed
design for the same power. For an organization where stakeholders will look at dashboards
daily regardless of instruction, always-valid methods are the honest choice.


experimentation

Interference violates the assumption that one unit's treatment does not affect another's
outcome, and it breaks experiments in specific settings. Marketplace experiments are the
clearest case: giving treated sellers better placement necessarily takes impressions from
control sellers, so the measured difference overstates the effect that would occur if the
change were applied to everyone. Social networks have the same problem through direct
interaction. Shared resources — a cache, a recommendation model trained on both arms, a
fixed inventory or budget — create it too.


experimentation

Interference and its mitigations:
  marketplace           treated and control compete for the same demand or supply
  network               social interaction between arms
  shared model          a model trained on pooled data learns from both arms
  shared inventory      a fixed pool depleted by both arms
  cluster randomization randomize groups within which interference occurs
  switchback            alternate treatment for the whole system over time
                        intervals; handles marketplace interference at the cost
                        of temporal confounding
  budget-split          allocate separate resource pools per arm
  two-sided designs     randomize both supply and demand
  bias direction        interference usually inflates the measured effect, so a
                        naive result is an upper bound


experimentation

Several other pitfalls recur often enough to name. Multiple comparisons across many metrics
inflate false positives, and the remedy is pre-specifying a primary metric and applying
false discovery rate control to the rest. Segment mining after the fact — searching
subgroups until one shows an effect — has no error control at all and should be treated as
hypothesis generation requiring a fresh test. Survivorship in the analysis population,
where the treatment affects who appears in the data, breaks comparability. And carryover
between sequential experiments on the same users, particularly when a prior experiment
changed behaviour persistently, contaminates the next.


experimentation

## Quasi-Experimental Methods and Practice

When randomization is impossible — a change that must apply to everyone, a market-level
intervention, a policy already implemented — quasi-experimental designs exploit structure to
approximate it, with weaker guarantees that must be stated honestly.


experimentation

Difference-in-differences compares the change over time in a treated group against the change
in an untreated one, differencing out both fixed group differences and common time shocks. It
rests on the parallel trends assumption — that the groups would have moved together absent
treatment — which is untestable for the treatment period and should be supported by showing
parallel pre-period trends. Regression discontinuity exploits a threshold rule, comparing
units just above and below a cutoff, which are plausibly similar in everything but treatment;
it has strong internal validity and applies only near the threshold. Synthetic control
constructs a weighted combination of untreated units matching the treated unit's pre-period
trajectory and is the standard approach for a single treated market. Instrumental variables
use a source of variation affecting treatment but not the outcome directly, and the exclusion
restriction is untestable and usually the weak point.


experimentation

Practice and organization:
  experiment platform    self-service assignment, logging, and analysis; the
                         infrastructure determines the volume of experimentation
  guardrails by default  every experiment monitors latency, errors, and harm
                         metrics without the team opting in
  automated SRM check    run on every experiment; block analysis on failure
  pre-registration       hypothesis, primary metric, sample size, and analysis
                         plan before launch
  standard analysis      one implementation everyone uses; ad-hoc analysis
                         introduces inconsistent methods and unreviewed choices
  institutional memory   a searchable record of past experiments and outcomes;
                         prevents repeating tests and preserves negative results
  negative results       record them; they are the majority and they are the most
                         useful part of the archive
  effect size culture    report intervals and magnitudes, not binary verdicts
  ship criteria          agreed before results; who decides and on what basis


experimentation

The organizational point that matters most is that trust in the experimentation system is
its main asset and is easily lost. A platform that produces a result someone can plausibly
dispute — because the sample ratio was off, because the metric definition was ambiguous,
because the analysis was done differently than last time — invites relitigation of every
result, and once teams believe they can argue with the data they will argue with the results
they dislike and accept the ones they like. Automated checks, a single standard analysis, and
pre-specification are what make the output credible enough to act on, and their value is
organizational as much as statistical.


feature_engineering

Feature Engineering and Data Preparation

## Exploratory Analysis and Data Understanding

Feature engineering begins with understanding what the data actually contains, and the effort
spent here has a higher return than nearly anything downstream. The recurring experience is
that datasets contain surprises — encoded missing values, duplicated records, units that
changed partway through the collection period, categories that were merged or split, columns
whose meaning differs from their name — and that a model trained without discovering them
learns the artifacts.


feature_engineering

Univariate examination establishes the basic properties of each column: its type,
distribution, range, missing rate, and cardinality. The specific things to look for are
distributions with implausible values such as negative ages or future dates, spikes at
particular values that indicate defaults or sentinels, and heavy tails that will dominate any
scale-sensitive method. Categorical columns need their cardinality checked, since a column with
thousands of levels needs different handling from one with five, and their frequency
distribution examined, since rare levels behave differently from common ones.


feature_engineering

Bivariate examination looks at relationships, both between features and between each feature
and the target. Correlation matrices identify redundancy, and the caution is that correlation
captures only linear association, so mutual information or a scatter plot may reveal
relationships that correlation misses entirely. The relationship with the target is what
guides construction, and examining it before modelling frequently suggests transformations that
a model would need substantial capacity to learn.


feature_engineering

Exploratory checklist:
  types and semantics    what each column means and how it was produced
  missing patterns       rate, and whether missingness correlates with anything
  sentinel values        -1, -999, 0, empty string, "unknown"; often encode missing
  duplicates             exact and near; check whether they are legitimate
  cardinality            categorical levels; high cardinality needs special handling
  distributions          skew, multimodality, heavy tails, spikes at round numbers
  outliers               distinguish errors from genuine extremes; the response
                         differs completely
  units and scale        verify consistency, especially across time and source
  temporal drift         plot each feature's distribution over time; changes
                         indicate upstream changes
  target relationship    plot the target against each feature; suggests
                         transformations
  target distribution    imbalance, skew, and whether a transform is warranted
  data dictionary        write one if it does not exist; the act of writing it
                         surfaces ambiguities


feature_engineering

Temporal drift plots deserve specific mention because they catch a class of problem nothing
else does. Plotting each feature's mean and missing rate by month reveals when an upstream
system changed, when a data source was added or removed, when a definition changed, and when a
collection outage occurred. A model trained across such a boundary learns from two different
data-generating processes, and its behaviour on new data depends on which one the new data
resembles. Discovering the boundary allows a decision — train on the recent period only, add an
indicator, or correct the older data — that cannot be made if the boundary is invisible.


feature_engineering

Q: A numeric column has a large spike of values at exactly -999 and is otherwise distributed
between 0 and 100. The correct interpretation is most likely:
A. Genuine negative measurements
B. A sentinel encoding missing data, which must be converted to null before any statistic is
   computed
C. An outlier to be winsorized
D. A scaling error
Reasoning: A concentrated spike at an implausible round negative value is the signature of a
missing-data sentinel from a legacy system. Treating it as a real value corrupts every
statistic, and models learn a spurious relationship at that value. Converting to null and
adding a missingness indicator is the correct handling. Answer: B.


feature_engineering

## Missing Data

Missingness is not one problem but three, and the mechanism determines what handling is
valid. Data missing completely at random has missingness independent of everything, so
complete-case analysis is unbiased and merely loses power. Data missing at random has
missingness depending on observed variables, so it can be corrected by conditioning on them.
Data missing not at random has missingness depending on the unobserved value itself, which
cannot be corrected from the data alone and requires either external information or explicit
sensitivity analysis.


feature_engineering

The third case is common and consequential. Income is missing more often for high earners,
symptom severity is missing more often for the sickest patients who were too ill to complete
the survey, and a sensor reading is missing exactly when the condition it measures exceeded the
sensor's range. In each case the missingness carries information about the missing value, and
imputing with a mean systematically distorts the distribution in a known direction.


feature_engineering

The practically important consequence is that the fact of missingness is itself a feature. A
binary indicator recording whether each value was missing preserves that information, and it
frequently turns out to be among the most predictive features in the model — which is a signal
that the missingness is informative and that imputation alone would have discarded real signal.


feature_engineering

Missing data handling:
  MCAR                   independent of everything; complete-case is unbiased
  MAR                    depends on observed variables; correctable by conditioning
  MNAR                   depends on the missing value itself; not correctable from
                         the data alone
  missingness indicator  add a binary flag; preserves informative missingness and
                         is often highly predictive
  drop rows              only when the missing fraction is small and MCAR is
                         plausible
  drop columns           when the missing rate is very high and the column is not
                         critical
  mean or median         simple, distorts the distribution, understates variance
  forward fill           for time series where the last value persists; wrong when
                         it does not
  model-based            regression or k-nearest-neighbour imputation; better and
                         requires care to avoid leakage
  multiple imputation    generate several completed datasets and pool the results;
                         propagates imputation uncertainty into the estimates
  native handling        gradient-boosted tree implementations learn a default
                         direction per split; frequently better than any imputation
  fit inside the fold    imputation parameters must be learned from training data
                         only


feature_engineering

Native missing-value handling in tree ensembles deserves emphasis because it removes a whole
category of decision. LightGBM, XGBoost, and CatBoost all learn, for each split, which
direction missing values should go, based on which reduces the loss more. This uses the
missingness informatively without requiring the analyst to choose an imputation strategy, and
it is frequently better than any imputation because it adapts per split rather than applying
one global rule. Where a tree ensemble is the model, imputing first is often actively harmful.


feature_engineering

## Encoding and Transformation

Categorical variables must be converted to numbers, and the encoding choice interacts
strongly with the model type. One-hot encoding creates a binary column per level, which is
correct for linear models and neural networks and explodes the dimensionality for
high-cardinality features. Ordinal encoding assigns integers, which imposes a false ordering on
nominal categories for models that interpret magnitude, and is entirely fine for tree ensembles
which only split on thresholds and can isolate any subset through repeated splits.


feature_engineering

Target encoding replaces each level with a statistic of the target within that level, which
compresses high-cardinality features into a single informative column. It is powerful and is
the single most leakage-prone technique in common use, because the encoding is computed from
the target. Computed naively on the full training set, each row's encoding includes its own
target value, so the model can partially read the answer from the feature. The remedies are
cross-fold encoding, where each fold's encoding is computed from the other folds, and smoothing
toward the global mean proportional to how few observations a level has.


feature_engineering

Numeric transformation addresses distributional problems. Standardization centres and scales
to unit variance and is required for any method using distances or penalties — k-nearest
neighbours, support vector machines, principal components, and regularized regression are all
scale-sensitive, while tree ensembles are entirely scale-invariant. Skewed distributions
benefit from a log or power transform, since many methods assume roughly symmetric errors and
heavy tails otherwise dominate.


feature_engineering

Encoding and transformation:
  one-hot               binary per level; correct for linear and neural models
  ordinal               integers; fine for trees, imposes false order elsewhere
  target encoding       target statistic per level; powerful, leakage-prone;
                        requires cross-fold computation and smoothing
  frequency encoding    replace with the level's count; simple and often effective
  hashing               fixed dimensionality for very high cardinality; collisions
                        are tolerable
  embeddings            learned dense representations; suit neural models and
                        very high cardinality
  rare level grouping   collapse infrequent levels into "other"; prevents
                        overfitting to levels with few observations
  standardization       zero mean, unit variance; required for distance and
                        penalty based methods
  min-max scaling       to a fixed range; sensitive to outliers
  robust scaling        median and interquartile range; resists outliers
  log transform         for right-skewed positive data; log1p handles zeros
  Box-Cox / Yeo-Johnson  fitted power transforms toward normality
  quantile transform    maps to a uniform or normal distribution; very robust,
                        loses the original scale's meaning
  binning               discretizes; loses information and can capture
                        nonlinearity for linear models


feature_engineering

The scale sensitivity table is worth committing to memory because applying the wrong
preparation is a common and silent error. Linear and logistic regression with regularization
require scaling because the penalty is applied to coefficients whose magnitude depends on the
feature scale. Neural networks require it for optimization stability. Distance-based methods
require it because an unscaled feature with a large range dominates the distance. Tree
ensembles require none of it and are unaffected, so scaling them is harmless and pointless.


feature_engineering

## Feature Construction

Constructing features is where domain knowledge enters, and it remains the highest-leverage
activity in applied machine learning on tabular data. The goal is to express in the feature
space what the model would otherwise need substantial capacity and data to discover.


feature_engineering

Temporal features are the most consistently valuable. A raw timestamp is nearly useless to a
model; the derived hour of day, day of week, month, and holiday indicator are highly
informative, and the cyclical nature must be encoded properly — hour 23 and hour 0 are
adjacent, which an integer encoding does not express, so sine and cosine of the angle is the
standard representation. Time since a previous event, counts within recent windows, and
rolling aggregates all capture behaviour that a single snapshot does not.


feature_engineering

Aggregation features summarize related records and are what turn a relational structure into
a flat feature vector. For a customer, the count, sum, mean, maximum, and recency of their
transactions each carry different information, and ratios between them — this month's spend
relative to their twelve-month average — normalize for individual scale and are frequently more
predictive than the raw values.


feature_engineering

Feature construction patterns:
  datetime decomposition  hour, day of week, month, quarter, holiday, business day
  cyclical encoding       sine and cosine so that wraparound is continuous
  time since event        recency; often the single strongest behavioural feature
  rolling aggregates      mean, max, count, standard deviation over windows;
                          multiple window lengths capture different timescales
  lag features            prior values; essential in time series
  entity aggregates       count, sum, mean, distribution statistics per group
  ratios and differences  normalize for scale; frequently more predictive than
                          the components
  interactions            products or explicit combinations; trees find these
                          automatically, linear models cannot
  domain formulas         body mass index, debt-to-income, utilization rate;
                          encode established knowledge directly
  text                    length, token counts, sentiment, embeddings
  geospatial              distance to points of interest, density within a radius,
                          region aggregates
  count-based             frequency of a value in the dataset; captures typicality


feature_engineering

The cyclical encoding point deserves illustration because it is frequently gotten wrong.
Encoding the month as an integer from one to twelve tells a linear model that December is
eleven units from January, when they are adjacent. Encoding it as the sine and cosine of the
corresponding angle places every month on a circle, so December and January are close in the
feature space. The same applies to hour of day, day of week, day of year, and wind direction —
any quantity whose maximum wraps around to its minimum.


feature_engineering

[user] My model uses a "days_since_last_purchase" feature and performs great in validation but
poorly in production.
[assistant] tool_call ml.leakage_check({"feature": "days_since_last_purchase",
"target": "will_churn_next_30d", "computed_at": "dataset_build_time",
"prediction_time": "scoring_time", "validation_auc": 0.94, "production_auc": 0.68})
[tool] {"leakage_type": "temporal", "issue": "feature_computed_after_label_period",
"detail": "days_since_last_purchase_at_build_time_reflects_post_label_behavior",
"fix": "compute_as_of_prediction_timestamp"}
[assistant] Temporal leakage. Your feature was computed at dataset build time rather than as
of the prediction time, so for historical training rows it reflects behaviour that occurred
during and after the label window.


feature_engineering

Concretely: for a customer whose churn label covers a 30-day window ending last March, the
days-since-last-purchase value in your training data was computed when you built the dataset —
which might be months later. If they purchased in April, that value is small. If they churned
and never purchased again, it is large. The feature is therefore partly a function of the
outcome, which is why validation looks excellent and production does not, since at scoring time
you only have information up to the present.


feature_engineering

The fix is to compute every feature as of the prediction timestamp for that row. For a
training example whose label window begins on a given date, every feature must use only data
available strictly before that date. This is a point-in-time correct join, and it is the single
most important correctness property in any feature pipeline built over historical data.


feature_engineering

Practically this means storing the history with timestamps and computing features by an
as-of join rather than by aggregating the current state. It is more work than the naive
approach and it is the difference between a model that works and one that does not.


feature_engineering

Two things worth checking while you are in there. Any other feature derived from behavioural
history — transaction counts, average order value, session frequency — has the same defect if
computed the same way, so audit all of them rather than fixing this one. And once corrected,
expect the validation number to fall substantially toward the production number; that drop is
the leakage being removed, not a regression, and the corrected validation figure is the one to
use for decisions.


feature_engineering

## Selection, Leakage, and Pipeline Discipline

Feature selection reduces dimensionality to improve generalization, reduce computation, and
aid interpretation. Filter methods score features independently of any model — by correlation,
mutual information, or a univariate test — and are fast and blind to interactions. Wrapper
methods evaluate subsets by training a model, which is accurate and expensive and prone to
overfitting the selection to the validation set. Embedded methods perform selection as part of
training, as L1 regularization and tree-based importance do.


feature_engineering

The caution that applies to all of them is that selection performed using the target on the
full dataset leaks, and the leak can be spectacular. Selecting the thousand features most
correlated with the target from a hundred thousand random features, then cross-validating the
model alone, produces impressively high accuracy on pure noise, because the selection saw every
fold's labels. Selection must occur inside the cross-validation loop, on each training fold
independently, for the resulting estimate to be honest.


feature_engineering

Leakage more broadly is any information in the training features that would not be available
at prediction time, and it is the defect most likely to produce a model that validates
brilliantly and fails in production. Its forms are worth enumerating because each is easy to
introduce inadvertently.


feature_engineering

Leakage sources:
  target leakage        a feature computed from or determined by the target
  temporal leakage      features using information from after the prediction time
  train-test contamination  preprocessing fitted on the full dataset; scaling,
                        imputation, encoding, and selection all leak this way
  duplicate records     the same entity in both train and test
  group leakage         multiple rows per entity split across train and test, so
                        the model memorizes the entity
  proxy features        an identifier or a downstream artifact that encodes the
                        outcome; a case number assigned only to confirmed cases
  future aggregates     a group statistic computed over the whole dataset
                        including test rows
  detection             suspiciously high performance, a single dominant feature,
                        or a large validation-to-production gap
  the fix               every data-dependent step fitted inside the training fold;
                        point-in-time correct feature computation; group-aware
                        splitting


feature_engineering

Pipeline discipline is what enforces all of this structurally rather than by vigilance. A
pipeline object encapsulating imputation, encoding, scaling, selection, and the model, fitted
as a unit inside each cross-validation fold, makes contamination impossible by construction —
every transformation learns its parameters from training data only, because that is the only
data the pipeline sees when fitting. Doing preprocessing in separate steps outside the
validation loop is the practice that produces most leakage, and it is convenient enough that it
persists despite being known to be wrong.


feature_engineering

The same pipeline must then serve inference, which is what eliminates training-serving skew.
If the transformations applied at scoring time are a separate implementation of the same logic,
they will eventually diverge — a rounding difference, a different null handling, a category
seen at serving that was not seen in training. Serializing the fitted pipeline and applying it
at inference means there is one implementation and one set of learned parameters, and the class
of bug where the model receives inputs unlike its training distribution disappears.


graph_machine_learning

## Graph Data and Tasks

Graph machine learning applies to data where relationships between entities carry information
that the entities' own attributes do not. Molecules, social networks, citation graphs, knowledge
bases, road networks, recommendation interaction graphs, program dependency graphs, and protein
interaction networks all have this character, and treating their entities as independent
observations discards exactly what makes them informative.


graph_machine_learning

The structural difficulty is that graphs lack the regularity that convolutional and sequential
architectures exploit. An image has a fixed grid where every pixel has the same number of
neighbours in the same relative positions; a sequence has a canonical order. A graph has nodes
with varying degree, no canonical ordering, and no consistent notion of relative position, so any
method must produce the same answer regardless of how the nodes are numbered — permutation
invariance for graph-level outputs and permutation equivariance for node-level ones.


graph_machine_learning

Tasks divide by the level at which prediction occurs, and the level determines both the
architecture and the evaluation design. Node-level tasks predict a property of each node.
Edge-level tasks predict whether an edge exists or what type it is. Graph-level tasks predict a
property of the whole structure. Each requires different readout mechanisms and, importantly,
different data splitting.


graph_machine_learning

Graph data and tasks:
  node features        attributes of each entity
  edge features        attributes of each relationship
  directed vs undirected  whether relationships are symmetric
  heterogeneous        several node and edge types; requires type-aware handling
  multigraph           several edges between a pair
  dynamic graph        structure or attributes changing over time
  permutation invariance  graph-level outputs must not depend on node numbering
  permutation equivariance  node-level outputs must permute with the nodes
  node classification  a property per node; semi-supervised, since unlabelled
                       nodes are still present in the structure
  link prediction      whether an edge exists; the basis of recommendation and
                       knowledge base completion
  graph classification  a property of the whole graph; molecules and programs
  community detection  groups of densely connected nodes
  transductive         the test nodes are present during training; only their
                       labels are hidden
  inductive            generalize to entirely unseen nodes or graphs; a stronger
                       and more realistic requirement


graph_machine_learning

The transductive and inductive distinction deserves emphasis because it determines what a
reported result means. In the transductive setting the full graph including test nodes is
available during training and only the labels are withheld, so the model has seen the test nodes'
features and connections. In the inductive setting the model must generalize to nodes or graphs
it has never seen. Many benchmark results are transductive, and a model achieving them may fail
entirely at the inductive task that a deployment requires — which is the case whenever new users,
new molecules, or new documents arrive after training.


graph_machine_learning

Q: A node classification model is evaluated by randomly splitting nodes into train and test sets
within a single graph, with the full graph structure available throughout. This evaluates:
A. Inductive generalization to unseen nodes
B. Transductive performance, where the model has seen the test nodes' features and connections
   during training
C. Graph-level classification
D. Nothing useful
Reasoning: The test nodes participate in message passing during training even though their
labels are hidden, so the model has used their features and neighbourhood structure. This
measures transductive performance, which is a legitimate setting and does not establish
performance on nodes arriving after training. An inductive evaluation requires withholding the
nodes entirely. Answer: B.


graph_machine_learning

## Classical Features and Node Embeddings

Before learned representations, graph machine learning used hand-designed structural features,
and these remain strong baselines and useful inputs. Node-level features include degree, various
centrality measures, clustering coefficient, and the counts of small subgraphs the node
participates in. Graph-level features include the degree distribution, the counts of small
subgraph patterns, and spectral properties of the adjacency or Laplacian matrix.


graph_machine_learning

Graph kernels compare graphs by counting shared substructures — common walks, shortest paths, or
subtrees — and feeding the resulting similarity into a kernel method. The Weisfeiler-Lehman
kernel is the most effective of these and is worth knowing because it turns out to characterize
the expressive power of the standard neural approaches, which is an unusual case of a classical
method providing the theoretical ceiling for a modern one.


graph_machine_learning

Node embedding methods learn a vector per node such that structurally similar nodes are nearby,
and the shallow methods do so by treating random walks on the graph as sentences and applying
sequence embedding techniques. The intuition is direct: nodes appearing in similar walk contexts
occupy similar structural positions. The walk strategy determines what similarity means —
breadth-first walks capture local neighbourhood similarity while depth-first walks capture
broader structural role, and interpolating between them is the main tunable in these methods.


graph_machine_learning

Classical features and embeddings:
  degree               the simplest and frequently the strongest single feature
  centrality           betweenness, closeness, eigenvector, PageRank
  clustering coefficient  the fraction of a node's neighbour pairs that are
                       adjacent
  graphlets            counts of small induced subgraphs
  spectral features    eigenvalues of the adjacency or Laplacian
  graph kernel         similarity from shared substructures
  Weisfeiler-Lehman    iterative colour refinement; the standard kernel and the
                       expressiveness ceiling of message passing
  random walk embedding  walks as sentences; nodes in similar contexts embed
                       nearby
  walk strategy        breadth-first captures neighbourhood, depth-first captures
                       structural role
  matrix factorization  factor the adjacency or a proximity matrix
  shallow embedding limits  a lookup table per node; cannot generalize to unseen
                       nodes, does not use node features, and parameters grow with
                       the graph
  the strong baseline  degree plus node features in a gradient-boosted model; it
                       beats graph neural networks more often than the literature
                       suggests


graph_machine_learning

The shallow embedding limitations are worth stating because they motivated the shift to message
passing. A method learning a lookup table of embeddings has one parameter vector per node, so
parameters grow with the graph, no embedding exists for a node not present at training time, and
node attributes are not used at all unless bolted on. Message passing addresses all three: the
parameters are shared across nodes, an unseen node's representation is computed from its
neighbourhood, and features are the input rather than an afterthought.


graph_machine_learning

## Message Passing and Architectures

Graph neural networks compute node representations by iteratively aggregating information from
neighbours. Each layer has three steps: each node's neighbours send messages, those messages are
aggregated by a permutation-invariant function, and the node updates its representation from its
previous state and the aggregate. Stacking k layers means each node's representation depends on
its k-hop neighbourhood, so depth controls the receptive field exactly as it does in convolutional
networks.


graph_machine_learning

The aggregation function must be permutation invariant since neighbours have no order, and the
choice affects what the network can distinguish. Sum aggregation preserves multiset information
including neighbour counts, which makes it the most expressive of the simple choices. Mean
aggregation normalizes by degree, discarding the count, which helps when degrees vary widely and
loses information. Max aggregation captures the presence of a distinctive neighbour and discards
multiplicity entirely.


graph_machine_learning

Architectural variants differ mainly in how messages are weighted. Graph convolutional networks
use a fixed degree-based normalization. Attention-based networks learn to weight each neighbour by
a computed relevance, which handles heterogeneous neighbourhoods better and costs more. Sampling
approaches aggregate over a fixed-size random sample of neighbours rather than all of them, which
bounds computation and permits inductive application at scale.


graph_machine_learning

Message passing and architectures:
  message              computed from the neighbour and the edge
  aggregate            permutation-invariant combination; sum, mean, max, attention
  update               new representation from the previous state and the aggregate
  k layers             a k-hop receptive field
  sum aggregation      preserves multiset structure; most expressive
  mean aggregation     degree-normalized; loses counts
  max aggregation      distinctive neighbours; loses multiplicity
  GCN                  fixed degree normalization; simple and effective
  GraphSAGE            neighbour sampling; inductive and scalable
  GAT                  learned attention weights per neighbour
  GIN                  sum aggregation with an MLP; provably as expressive as the
                       Weisfeiler-Lehman test, which is the ceiling for this family
  readout              pooling node representations to a graph representation; sum,
                       mean, max, or a learned hierarchical pooling
  edge features        incorporated into the message computation
  residual connections  as in other deep networks; necessary beyond a few layers
  jumping knowledge    combine representations from several depths; different nodes
                       need different receptive fields


graph_machine_learning

## Expressiveness and Limits

The expressive power of message passing networks is bounded, and the bound is precise: they can
distinguish two graphs only if the one-dimensional Weisfeiler-Lehman colour refinement test can.
This is a genuine limitation rather than a modelling detail — there exist structurally different
graphs that no such network can tell apart, including simple cases such as a six-cycle and two
disjoint triangles, which have identical local structure everywhere.


graph_machine_learning

The practical consequence is an inability to count substructures. Message passing cannot in
general count triangles, cycles, or cliques, which are exactly the features that matter for
molecular properties depending on ring structure and for social network analysis depending on
triadic closure. Providing those counts explicitly as node features is the pragmatic remedy and
is frequently more effective than architectural sophistication.


graph_machine_learning

Over-smoothing is the depth limitation. As layers are stacked, node representations aggregate
from progressively larger neighbourhoods and converge toward each other, so a deep network
produces nearly identical representations for all nodes in a connected component and loses the
distinctions it was meant to compute. Most graph neural networks therefore use two to four
layers, which is shallow relative to other domains and limits the range over which information
propagates.


graph_machine_learning

Over-squashing is a related and distinct problem. Information from an exponentially growing
neighbourhood must be compressed into a fixed-size vector, so long-range dependencies passing
through a bottleneck edge are compressed beyond recovery. Adding layers to reach further makes
over-squashing worse rather than better, which is why long-range tasks on graphs remain
difficult.


graph_machine_learning

Expressiveness and limits:
  WL bound             message passing distinguishes only what colour refinement
                       does
  indistinguishable graphs  structurally different graphs with identical local
                       structure; a six-cycle and two triangles
  cannot count substructures  triangles, cycles, cliques; provide them as features
  higher-order networks  operate on tuples of nodes; more expressive and far more
                       expensive
  positional encodings  break symmetry by giving nodes distinguishable positions;
                       random features or Laplacian eigenvectors
  over-smoothing       representations converge with depth; limits networks to a
                       few layers
  over-squashing       exponentially many nodes compressed through a bottleneck;
                       adding depth makes it worse
  bottleneck edges     where over-squashing concentrates; rewiring the graph to
                       add shortcuts is one remedy
  graph transformers   full attention between all nodes; removes the locality
                       constraint and requires positional encoding to reintroduce
                       structure
  heterophily          neighbours with different labels; standard message passing
                       assumes homophily and degrades badly when it does not hold
  the homophily assumption  frequently unstated; check it before assuming a graph
                       neural network is appropriate


graph_machine_learning

Heterophily deserves specific attention because it is an unstated assumption that determines
whether the whole approach applies. Message passing aggregates neighbours toward a shared
representation, which helps when connected nodes tend to share labels — homophily — and actively
hurts when they do not. In a fraud network, fraudsters connect to victims rather than to each
other; in a dating network, connections are frequently between rather than within categories.
Standard architectures underperform simple baselines that ignore the graph in these settings, and
architectures designed for heterophily separate the node's own representation from the aggregated
neighbourhood rather than mixing them.


graph_machine_learning

[user] Our graph neural network for fraud detection performs worse than a gradient-boosted model
on the same node features with no graph at all.
[assistant] tool_call gml.diagnose({"task": "node_classification_fraud",
"architecture": "GCN_3_layer", "homophily_ratio": 0.21, "avg_degree": 47,
"label_rate": 0.008, "baseline_gbm_auc": 0.91, "gnn_auc": 0.84,
"features_include_graph_stats": false})
[tool] {"primary_issue": "heterophilic_graph_with_homophily_assuming_architecture",
"homophily_ratio": 0.21, "secondary": ["high_degree_causes_oversmoothing",
"graph_features_not_given_to_baseline_comparison"],
"recommendations": ["heterophily_aware_architecture", "add_structural_features_to_gbm",
"separate_self_and_neighbour_representations"]}
[assistant] Your graph is heterophilic and your architecture assumes the opposite, so message
passing is actively destroying signal rather than adding it.


graph_machine_learning

The homophily ratio is 0.21, meaning only about a fifth of edges connect nodes with the same
label. Standard graph convolution averages a node's representation with its neighbours', which is
the right operation when neighbours share labels and the wrong one when they systematically do
not. In fraud networks this is expected: fraudsters connect to victims and to mule accounts
rather than to each other, so aggregating a fraudster's neighbourhood pulls its representation
toward the legitimate majority. With an average degree of 47 and three layers, each node is
averaging over a very large neighbourhood, which compounds it.


graph_machine_learning

Three things, and the second may be all you need.


graph_machine_learning

Use an architecture that does not assume homophily. The key change is separating the node's own
representation from the aggregated neighbourhood rather than mixing them — concatenate rather
than average, so the model can learn that a node's neighbourhood being unlike it is itself
informative. Architectures designed for heterophily also aggregate from higher-order
neighbourhoods separately, since in some heterophilic graphs the two-hop neighbourhood is
homophilic even when the one-hop is not.


graph_machine_learning

Before investing in that, run a fairer baseline comparison. Your gradient-boosted model uses node
features only and no graph information at all. Add structural features to it — degree,
neighbourhood label statistics where labels are available, clustering coefficient, counts of
triangles and other small subgraphs, and aggregated neighbour features — and see where it lands.
Graph-derived features in a strong tabular model frequently match or beat a graph neural network,
particularly on heterophilic graphs, and it is a much simpler system to operate.


graph_machine_learning

Your label rate of 0.8% is also worth attention. With very few labels, the semi-supervised
advantage that graph methods offer is largest in principle and the training signal is thin in
practice. Check whether the labelled nodes are representative or whether they were discovered
through some process that correlates with structure, since that would bias the model toward
finding fraud that resembles how fraud was previously found.


graph_machine_learning

## Scaling and Applications

Scaling graph neural networks to large graphs is constrained by the neighbourhood explosion: a
k-layer network's computation for one node depends on its k-hop neighbourhood, which in a graph
with high average degree can be a substantial fraction of the whole graph. Full-batch training
therefore requires the entire graph in memory and mini-batching is not straightforward, since
nodes are not independent.


graph_machine_learning

Three families of approach address this. Neighbour sampling selects a fixed number of neighbours
per layer, bounding the computation per node at the cost of variance and of an approximation.
Subgraph sampling trains on sampled subgraphs, which is efficient and biased by how the subgraphs
are drawn. Historical embedding methods cache representations from previous iterations to avoid
recursive expansion, trading staleness for bounded computation.


graph_machine_learning

The applications where graph methods clearly earn their complexity share a property: the
relationships carry information that is not recoverable from the entities alone, and the relevant
structure is local. Molecular property prediction is the clearest case, since a molecule is
literally a graph and its properties depend on substructure. Recommendation benefits from the
user-item interaction graph's higher-order connectivity. Fraud and abuse detection benefits
because coordinated activity has a distinctive structural signature. Knowledge graph completion is
inherently relational.


graph_machine_learning

Scaling and applications:
  neighbourhood explosion  k-hop neighbourhoods grow rapidly with degree
  full batch           requires the whole graph in memory
  neighbour sampling   fixed fan-out per layer; variance and approximation
  subgraph sampling    train on sampled subgraphs; efficient and sampling-biased
  historical embeddings  cache previous representations; staleness for bounded
                       cost
  distributed training  partition the graph; cross-partition edges require
                       communication, and partition quality dominates performance
  inference at scale   precompute embeddings where the graph is static; recompute
                       incrementally where it is not
  molecules            the clearest fit; the molecule is a graph and properties
                       depend on substructure
  drug discovery       property prediction, generation, and interaction prediction
  recommendation       higher-order user-item connectivity
  fraud and abuse      coordinated activity has structural signatures
  knowledge graphs     link prediction for completion; embedding methods and
                       message passing both apply
  traffic and routing  road networks with temporal dynamics
  program analysis     code as a graph of dependencies and control flow
  physics simulation   particles and meshes as graphs; learned simulators
                       generalize across configurations


graph_machine_learning

The honest summary of when to use these methods is worth stating because the field's enthusiasm
has outrun the evidence in several application areas. Graph neural networks are clearly the right
tool where the data is genuinely a graph whose structure carries the signal — molecules, physical
meshes, knowledge bases, and networks where local structure is diagnostic. They are frequently
outperformed by well-engineered features in a gradient-boosted model where the graph is
incidental to the prediction, where the graph is heterophilic, or where node features already
carry most of the signal. Running that baseline with graph-derived features included is the check
that establishes whether the additional complexity is buying anything, and it is skipped far more
often than it should be.


inference_optimization

## The Inference Workload

Serving a trained model is a different engineering problem from training it, and the difference
starts with what the hardware is limited by. Training is compute-bound: large batches keep the
arithmetic units busy and the memory traffic per operation is amortized. Autoregressive generation
is memory-bandwidth-bound: producing one token requires reading every weight in the model and
performing very little arithmetic per weight read, so the time is set by how fast weights can be
moved from memory rather than by how fast they can be multiplied.


inference_optimization

This single fact explains most of what works in inference optimization. Reducing the bytes per weight
directly reduces the time, which is why quantization gives near-proportional speedups on the decode
path. Increasing the arithmetic performed per weight read amortizes the transfer, which is why
batching helps enormously and why processing several tokens at once through speculative methods
helps too. Techniques that reduce arithmetic without reducing memory traffic help far less than their
operation counts suggest.


inference_optimization

The two phases of generation have opposite characteristics and should be measured and optimized
separately. Prefill processes the entire prompt at once, which is a large matrix operation that is
compute-bound and efficient. Decode produces one token at a time and is bandwidth-bound and
inefficient. Time to first token is dominated by prefill and scales with prompt length; time between
tokens is dominated by decode and is nearly independent of it. Reporting a single throughput number
for a system with both phases conceals which one is the problem.