spacecraft_engineering

Power and thermal quantities:
  solar constant        1361 W/m^2 at 1 AU; scales as 1/r^2
  triple-junction cell  ~30% efficiency at beginning of life
  array sizing          include packing, cosine loss, temperature, and EOL
                        degradation; size for end of life
  battery DOD           20-30% for LEO cycling, 60-80% for GEO seasonal
  eclipse, LEO          up to ~36 min of a ~95 min orbit
  radiative rejection   Q = eps sigma A (T^4 - T_space^4)
  surface properties    alpha/eps ratio governs equilibrium temperature; a low
                        ratio runs cold, a high ratio runs hot
  MLI                   multilayer insulation; effective emittance ~0.01-0.03
  heat pipe             moves heat isothermally without power; the workhorse
  louvers               passive variable emittance
  heaters               thermostatically controlled; a real and often
                        underestimated power budget item
  survival heaters      keep components above minimum temperature in safe mode


spacecraft_engineering

The absorptivity-to-emissivity ratio is the key surface parameter and it is worth
internalizing. Absorptivity governs how much sunlight a surface takes in; emissivity governs
how well it radiates infrared out. A surface with low absorptivity and high emissivity — white
paint, optical solar reflectors — runs cold and is used for radiators. High absorptivity and
low emissivity — polished metal — runs hot. Both properties degrade in orbit through
ultraviolet exposure and atomic oxygen erosion, generally in the direction of increasing
absorptivity, so thermal designs must accommodate the end-of-life values, which means the
spacecraft is too cold at the start and heaters make up the difference.


spacecraft_engineering

[user] Sizing a radiator for a 400 W instrument that must stay below 20 C, in low Earth
orbit.
[assistant] tool_call thermal.radiator_sizing({"dissipation_w": 400, "t_max_c": 20,
"orbit": "LEO_600km", "surface": "OSR", "alpha_bol": 0.09, "alpha_eol": 0.21,
"emissivity": 0.80, "view_factor_space": 0.85})
[tool] {"t_sink_effective_k": 255, "area_bol_m2": 1.42, "area_eol_m2": 2.06,
"earth_ir_flux_w_m2": 237, "albedo_flux_w_m2": 120, "recommended_area_m2": 2.2,
"note": "leo_environment_dominates_deep_space_assumption"}
[assistant] You need about 2.2 square metres, and the reason it is that large is the low
Earth orbit environment rather than the dissipation.


spacecraft_engineering

In deep space a radiator sees a 4 K background and the sizing is straightforward. In a 600
km orbit your radiator also sees Earth, which radiates about 237 W/m^2 of infrared and
reflects roughly 120 W/m^2 of albedo when sunlit. Those are absorbed heat loads working
against you, so the effective sink temperature is around 255 K rather than near zero, and the
available temperature difference for rejection collapses.


spacecraft_engineering

Two numbers to hold onto. At beginning of life with an optical solar reflector at
absorptivity 0.09 you need 1.42 square metres. At end of life, with absorptivity degraded to
0.21 by ultraviolet exposure and contamination, you need 2.06. Size for end of life — 2.2
with margin — and accept that the radiator is oversized initially, which is handled with
heaters or a variable-conductance heat pipe.


spacecraft_engineering

Three design choices that will affect this substantially:


spacecraft_engineering

Point the radiator away from Earth if you can. An anti-nadir or orbit-normal facing radiator
sees far less Earth infrared and albedo, and can cut the required area by thirty to forty
percent. This usually competes with instrument pointing and antenna placement, which is
exactly the kind of system-level coupling that has to be resolved early.


spacecraft_engineering

Use optical solar reflectors or second-surface mirrors rather than white paint. Paint
degrades much faster in ultraviolet, so its end-of-life absorptivity is worse and the
beginning-to-end-of-life spread is larger.


spacecraft_engineering

Consider whether 20 C is a real requirement. Radiator area scales roughly with the inverse
of the fourth power of the rejection temperature, so allowing 35 C instead of 20 would cut
the area by nearly a quarter. If the limit is set by a component that could be thermally
isolated and cooled separately, that is often a better trade than growing the radiator.


spacecraft_engineering

Finally, budget the heater power. With a radiator sized for end of life you will need
heaters early in the mission and during low-dissipation modes, and that load is frequently
omitted from the power budget until late.


spacecraft_engineering

## Attitude Determination and Control

Attitude control orients the spacecraft, and the requirements span an enormous range — from
a few degrees for a communications satellite pointing an antenna at Earth, to
sub-arcsecond stability for an astronomical observatory. Cost and complexity scale steeply
with that requirement.


spacecraft_engineering

Determination requires measuring orientation against known references. Sun sensors are
simple and coarse. Magnetometers work only in a planetary magnetic field and give moderate
accuracy. Horizon sensors track the Earth's infrared limb. Star trackers image a field,
identify the pattern against a catalogue, and deliver arcsecond-level attitude — the highest
accuracy available and the standard for demanding missions. Gyroscopes measure angular rate
with high bandwidth and drift over time, so they are always fused with an absolute reference,
typically in a Kalman filter that uses the gyro for short-term propagation and the star
tracker for long-term correction and gyro bias estimation.


spacecraft_engineering

Control requires torque, and the available actuators differ in whether they consume
propellant and in whether they can accumulate momentum. Reaction wheels exchange momentum
with the spacecraft body and are the standard for precision pointing, consuming only
electrical power. Their limitation is that external disturbance torques — gravity gradient,
aerodynamic drag, solar pressure, magnetic — are secular, so the wheels spin up over time and
eventually saturate, requiring momentum to be dumped to the environment using magnetic
torquers or thrusters.


spacecraft_engineering

Attitude subsystem components:
  sun sensor           coarse, cheap, needs sunlight; ~0.1-1 degree
  magnetometer         needs a planetary field; ~1 degree
  horizon sensor       Earth limb; ~0.1 degree
  star tracker         ~1-10 arcsec; the precision standard
  gyroscope            high rate bandwidth, drifts; fused with absolute sensors
  GPS receiver         position, time, and with multiple antennas, attitude
  reaction wheel       fine torque, no propellant, saturates
  control moment gyro  much higher torque than a wheel; complex, has singularities
  magnetic torquer     no propellant, weak, needs a field; used for desaturation
  thruster             high torque, consumes propellant, coarse
  gravity gradient     passive stabilization to the local vertical; free, coarse
  spin stabilization   passive gyroscopic stiffness; simple and constraining


spacecraft_engineering

Disturbance torques determine the momentum accumulation rate and therefore the desaturation
frequency and propellant budget. In low orbits aerodynamic drag dominates, acting on the
offset between the centre of pressure and centre of mass. At higher altitudes solar radiation
pressure takes over, acting on the same geometric offset with respect to the illuminated
area. Gravity gradient torque acts on any non-spherical inertia distribution and is
proportional to the inertia difference and the inverse cube of orbital radius. Magnetic
torque acts on the spacecraft's residual dipole. Minimizing the offset between centre of
pressure and centre of mass at design time directly reduces the propellant needed for
attitude control over the mission.


spacecraft_engineering

## Propulsion, Communications, and Reliability

Propulsion is governed by the rocket equation, which makes specific impulse the dominant
figure of merit because propellant mass fraction grows exponentially with required delta-v
divided by exhaust velocity. Chemical systems provide high thrust and modest specific
impulse, suitable for launch, for orbit insertion where a burn must complete quickly, and
for attitude control. Electric propulsion provides very high specific impulse and very low
thrust, so it takes months to accomplish what a chemical stage does in minutes, and it does
so with a fraction of the propellant — a trade that is worthwhile whenever time is
available.


spacecraft_engineering

Propulsion options:
  cold gas             Isp 50-70 s; simple, safe, low performance
  monopropellant       hydrazine, Isp ~230 s; reliable, toxic, common for attitude
                       control and small manoeuvres
  bipropellant         Isp 300-450 s; the workhorse for orbit insertion
  solid                Isp ~280 s; simple, high thrust, cannot be throttled or
                       stopped
  Hall thruster        Isp 1500-3000 s; the electric propulsion workhorse
  ion engine           Isp 3000-5000 s; even lower thrust
  green monoprop       reduced toxicity alternatives to hydrazine; simplifies
                       ground handling substantially
  electrothermal       resistojet or arcjet; modest improvement over monoprop
  solar sail           no propellant at all; tiny thrust, unlimited duration


spacecraft_engineering

Communications is governed by the link budget, which accounts for every gain and loss from
transmitter to receiver in decibels. Transmit power and antenna gains add; free-space path
loss, which grows as the square of both distance and frequency, dominates the losses;
atmospheric and pointing losses subtract further. The result is compared against the
receiver's noise floor to give a signal-to-noise ratio, and the link margin is what remains
above the minimum required for the chosen modulation and coding.


spacecraft_engineering

The path loss scaling with frequency is why deep space missions face such severe data
constraints and why they nonetheless use high frequencies. Higher frequency increases path
loss, and it increases achievable antenna gain for a fixed aperture by even more, since gain
scales as the aperture area divided by wavelength squared. The net effect favours higher
frequency when the aperture is fixed, which is why missions moved from S-band to X-band to
Ka-band, and why optical communication — with wavelengths four orders of magnitude shorter —
offers enormous gain at the cost of much tighter pointing requirements.


spacecraft_engineering

Reliability engineering for spacecraft rests on the fact that repair is impossible. The
techniques are redundancy, derating, parts screening, and testing. Redundancy may be
cold-standby, saving power at the cost of switchover time, or hot, providing instant
failover with continuous power. Cross-strapping allows either of two redundant units to
connect to either of two downstream units, protecting against a failure that would otherwise
take out a whole string. Single-point failures are catalogued explicitly and either
eliminated or accepted with justification.


spacecraft_engineering

The radiation environment drives parts selection and fault tolerance. Total ionizing dose
accumulates and gradually degrades semiconductors until they fail, which sets mission
lifetime for a given shielding. Displacement damage degrades solar cells and optical
detectors. Single event effects are caused by individual energetic particles: an upset flips
a memory bit and is corrected by error-correcting memory and periodic scrubbing; a latchup
triggers a parasitic conducting path that destroys the part unless power is cycled quickly,
which is why latchup protection circuits monitor current and remove power automatically; a
burnout destroys a power device outright. Rad-hard parts tolerate far more and cost orders of
magnitude more with generations-old performance, which is why many modern missions use
commercial parts with architectural mitigation — redundant computation, watchdogs, frequent
memory scrubbing, and a safe mode the spacecraft falls into whenever anything is
inconsistent.


telescope_engineering

## Optical Design and Fundamental Limits

A telescope collects light and forms an image, and its two primary specifications follow directly
from its aperture. Collecting area scales with the square of the diameter, which determines how
faint an object can be detected in a given time. Angular resolution scales inversely with diameter,
set by the diffraction limit, which determines how finely detail can be distinguished. Both push
toward larger apertures, and cost scales considerably faster than diameter, which is why telescope
history is largely a history of making large mirrors affordable.


telescope_engineering

Refracting designs use lenses and suffer from chromatic aberration — different wavelengths focusing
at different points — which can be partially corrected with multiple elements and never eliminated.
They are also limited in size because a large lens can only be supported at its edge and sags under
its own weight. Reflecting designs use mirrors, have no chromatic aberration at all since reflection
is wavelength-independent, and can be supported across the back, which is why every large telescope
is a reflector.


telescope_engineering

The standard reflecting configurations trade off differently. A two-mirror arrangement with a
parabolic primary and hyperbolic secondary gives excellent on-axis images and coma that grows
rapidly off axis, limiting the usable field. Adding correction — either a modified primary and
secondary pair or a corrector lens group near the focus — trades some on-axis perfection for a much
wider well-corrected field, which is what survey telescopes need since their value is proportional
to field of view times aperture.


telescope_engineering

Optical fundamentals:
  collecting area      scales as diameter squared; sets limiting magnitude
  angular resolution   scales inversely with diameter; the diffraction limit
  the cost scaling     rises much faster than diameter
  refractor            lenses; chromatic aberration, edge support, size-limited
  reflector            mirrors; achromatic and supportable across the back
  the two-mirror design  excellent on axis, coma off axis
  wide-field corrected designs  a wider usable field at some on-axis cost
  the survey figure of merit  aperture times field of view
  focal ratio          focal length over aperture; determines image scale and
                       the exposure needed for extended sources
  plate scale          arcseconds per millimetre at the focal plane
  segmented mirrors    many smaller mirrors phased together; the only practical
                       route past a certain diameter
  active optics        slow correction of mirror figure against gravity and
                       thermal distortion
  the support problem  a large mirror deforms under its own weight as it tilts,
                       which is why support systems are actively controlled
  baffling             suppressing stray light; frequently the difference between
                       a good instrument and a mediocre one, and it is
                       consistently underestimated in early design


telescope_engineering

Q: A ground-based telescope with a four-metre aperture routinely delivers images no sharper than a
one-metre telescope would. The reason is:
A. The optics are poorly figured
B. Atmospheric turbulence limits the resolution to the seeing disk, typically around one arcsecond,
   which is far worse than the diffraction limit of either aperture; the larger telescope collects
   more light but does not resolve better without correction
C. The detector is inadequate
D. Diffraction increases with aperture
Reasoning: Above roughly ten to twenty centimetres of aperture at a typical site, atmospheric
turbulence rather than diffraction sets the image size. Larger apertures gain sensitivity and gain
resolution only with adaptive optics, interferometry, or placement above the atmosphere.
Answer: B.


telescope_engineering

## Detectors and Photon Statistics

The charge-coupled device and its complementary-metal-oxide successors converted astronomy from
photographic to digital and improved quantum efficiency by more than an order of magnitude. A modern
detector converts a large fraction of arriving photons into measurable charge, records the result
linearly across a wide range, and can be read out with very low noise, and each of those properties
matters for different observations.


telescope_engineering

The noise budget determines what limits an observation and therefore what to change to improve it.
Shot noise from the source scales as the square root of the signal. Sky background contributes its
own shot noise and dominates for faint extended sources. Dark current accumulates with time and is
suppressed by cooling. Read noise is added once per readout and dominates for very short exposures.
Knowing which term dominates tells you whether to expose longer, cool further, bin pixels, or take
fewer longer exposures instead of many short ones.


telescope_engineering

Detector calibration removes instrumental signatures in a defined order. A bias frame captures the
electronic offset. A dark frame captures thermally generated charge over the exposure time. A flat
field captures the pixel-to-pixel sensitivity variation and the vignetting and dust shadows of the
optical path. Applying these in the right order, with the right scaling, is routine and mechanical,
and getting it wrong produces subtle photometric errors that survive into the science.


telescope_engineering

Detectors and noise:
  quantum efficiency   the fraction of photons converted; the primary sensitivity
                       figure
  linearity            output proportional to input over the usable range
  full well capacity   where a pixel saturates
  dynamic range        full well over read noise
  shot noise           square root of the signal; irreducible
  sky background       dominates for faint extended sources
  dark current         thermal generation; suppressed by cooling
  read noise           added once per readout; dominates short exposures
  the diagnostic use   which term dominates determines the right remedy
  binning              combining pixels; improves signal-to-noise where read
                       noise dominates and sacrifices resolution
  exposure strategy    fewer long exposures when read noise dominates, more short
                       ones when saturation or tracking limits you
  bias frame           the electronic offset
  dark frame           thermal charge for the exposure time
  flat field           pixel sensitivity, vignetting, and dust
  cosmic rays          particle hits producing sharp spurious features; rejected
                       by combining multiple exposures
  charge transfer inefficiency  charge left behind during readout; worsens with
                       radiation damage and matters for precise astrometry


telescope_engineering

## Adaptive Optics and Atmospheric Correction

The atmosphere degrades ground-based images because turbulent cells with slightly different
temperatures have slightly different refractive indices, so a wavefront arriving flat from a distant
source is corrugated by the time it reaches the telescope. The result is a blurred and constantly
changing image, characterized by the seeing — the resulting image size, typically around one
arcsecond at a good site and better at the best ones.


telescope_engineering

Adaptive optics measures the wavefront distortion and corrects it in real time with a deformable
mirror. A wavefront sensor samples the incoming wavefront hundreds to thousands of times per second,
a control computer computes the correction, and actuators behind a thin mirror deform it to
compensate. The correction must be faster than the turbulence evolves, which sets the loop rate, and
the number of actuators must match the number of turbulent cells across the aperture, which sets the
system's complexity as the square of the diameter.


telescope_engineering

The system needs a bright reference source in the same direction as the target, and most targets do
not have one nearby. Laser guide stars solve this by exciting sodium atoms in the upper atmosphere
to create an artificial reference, which works for the higher-order correction and cannot sense
overall tilt, since the laser's own outbound path shifts with it. A faint natural star is therefore
still required for tilt, which is a much less restrictive requirement.


telescope_engineering

Adaptive optics:
  the cause            turbulent cells with differing refractive index
  seeing               the resulting image size; around one arcsecond typically
  coherence length     the aperture over which the wavefront is flat
  coherence time       how fast the turbulence changes; sets the loop rate
  isoplanatic angle    the angle over which the correction is valid; small,
                       especially in visible light
  wavefront sensor     samples the distortion at high rate
  deformable mirror    actuators shaping a thin mirror
  actuator count       scales as the square of the diameter
  the wavelength dependence  correction is far easier in the infrared, which is
                       why adaptive optics matured there first
  natural guide star   a bright reference near the target; rarely available
  laser guide star     an artificial reference in the sodium layer
  the tilt limitation  the laser cannot sense overall tilt; a faint natural star
                       is still needed
  multi-conjugate systems  several deformable mirrors correcting different
                       altitudes; widens the corrected field
  interferometry       combining separated apertures for resolution set by their
                       separation rather than their size; extraordinary resolution
                       and a very limited field


telescope_engineering

## Mounts, Pointing, and Space Constraints

A telescope must point accurately and track the sky's apparent motion. Equatorial mounts have one
axis aligned with the Earth's rotation, so tracking is a single constant-rate motion and the field
does not rotate. Altitude-azimuth mounts are mechanically simpler and far more practical at large
size, and they require coordinated motion on both axes plus a third rotation to counteract field
rotation, which is why they became feasible only with computer control.


telescope_engineering

Pointing accuracy is achieved through a pointing model rather than through mechanical perfection.
The model fits a set of terms — axis misalignment, non-perpendicularity, flexure of the tube with
elevation, encoder eccentricity — to observations of many stars across the sky, and the resulting
corrections bring a mount with visible mechanical imperfections to arcsecond pointing. The model
must be re-derived after any mechanical change, which is a maintenance obligation that is
occasionally forgotten with confusing consequences.


telescope_engineering

Space telescopes trade the atmosphere's degradation for a set of severe constraints. Mass and volume
are limited by the launch vehicle, which drives segmented and deployable designs. Thermal control is
harder without convection and matters more, since optics must hold their figure across the orbital
temperature cycle. There is no servicing for most missions, so redundancy and conservative design
replace repair. And pointing must be achieved without any ground contact, using reaction wheels and
star trackers, with the accumulated momentum periodically dumped.


telescope_engineering

Mounts and space:
  equatorial mount     one axis on the rotation axis; simple tracking, no field
                       rotation
  altitude-azimuth     mechanically practical at size; needs coordinated axes and
                       derotation
  field rotation       the consequence of an altazimuth mount; corrected by a
                       rotating instrument or derotator
  pointing model       fitted corrections for known mechanical imperfections
  the terms            misalignment, non-perpendicularity, flexure, eccentricity
  re-derivation        required after mechanical changes
  guiding              closed-loop correction on a reference star during exposure
  the dome contribution  local turbulence from the enclosure and from the
                       telescope's own heat; often worse than the free atmosphere
  space advantages     no atmosphere, access to blocked wavelengths, stable
                       thermal and gravitational environment
  launch constraints   mass and volume; drives deployable and segmented designs
  thermal control      no convection; radiative only, and figure stability
                       depends on it
  sunshields           passive cooling to the temperatures infrared instruments
                       require
  no servicing         redundancy and conservatism replace repair
  attitude control     reaction wheels and star trackers
  momentum management  accumulated wheel momentum dumped periodically
  orbit selection      low Earth orbit for access, Lagrange points for thermal
                       stability and continuous observation


telescope_engineering

## Spectrographs, Calibration, and Data

Spectroscopy disperses light by wavelength and extracts far more physical information than imaging:
composition, temperature, density, ionization state, radial velocity, and magnetic field strength
all leave signatures in a spectrum. The instrument's resolving power — wavelength divided by the
smallest resolvable wavelength difference — determines which of these are accessible, and it trades
directly against the light reaching each detector element.


telescope_engineering

The standard design disperses light with a grating and images the result. A slit or fibre defines
what enters, and the slit width trades spectral resolution against throughput. Echelle designs use
high orders and a cross-disperser to record a wide wavelength range at high resolution
simultaneously, at the cost of a complex two-dimensional format that requires careful extraction.
Integral field units capture a spectrum at every point in a small field, which is the most
information-rich configuration and produces correspondingly large data volumes.


telescope_engineering

Wavelength calibration ties detector position to physical wavelength using a source of known
emission lines, and its stability determines what velocity precision is achievable. The
extraordinary precision needed for exoplanet detection by radial velocity — of order metres per
second, which is a shift far smaller than a detector pixel — is achieved through simultaneous
reference sources, extreme environmental stabilization, and careful modelling of the instrument
profile. This is a case where the calibration strategy rather than the optics is the instrument's
defining feature.


telescope_engineering

Spectroscopy and calibration:
  dispersion           separating light by wavelength
  what a spectrum reveals  composition, temperature, density, ionization,
                       velocity, magnetic field
  resolving power      wavelength over the smallest resolvable difference
  the tradeoff         resolution against light per detector element
  slit width           resolution against throughput
  fibre feed           decouples the spectrograph from the telescope; enables
                       stabilization and multi-object work
  echelle design       high orders plus cross-dispersion; wide range at high
                       resolution
  integral field unit  a spectrum at every point in a field
  multi-object spectroscopy  many targets at once; the survey workhorse
  wavelength calibration  known emission lines mapped to detector position
  simultaneous reference  a calibration source recorded alongside the science
                       exposure
  instrument profile   the response to a monochromatic input; must be modelled
                       for precise work
  flux calibration     converting counts to physical units using standard stars
  telluric correction  removing absorption by the Earth's atmosphere
  the radial velocity case  metres-per-second precision from shifts far smaller
                       than a pixel; achieved through calibration strategy and
                       environmental control rather than through optical
                       excellence
  data volume          modern instruments produce more data than can be examined
                       by hand, which has made pipeline design and archive
                       quality part of instrument design rather than an
                       afterthought


active_and_federated_learning

Active, Federated, and Data-Efficient Learning

## Active Learning and the Economics of Labels

Supervised learning consumes labels, and labels are the expensive input in most applied machine
learning. Active learning attacks this directly by choosing which examples to label rather than
labelling at random, on the premise that examples differ enormously in how much they teach. The
premise is correct — a model that already classifies an example confidently and correctly learns
nothing from its label — and the practical gains are real when the setup is right.


active_and_federated_learning

Uncertainty sampling is the standard strategy and the simplest: label the examples the current model
is least confident about. Confidence can be measured by the top probability, by the margin between
the top two classes, or by entropy over the distribution, and the three differ in which examples they
prioritize when there are many classes. Its weakness is that it selects for ambiguity, and ambiguity
includes inherently unlabelable examples — blurred images, contradictory text, mislabelled data —
which a naive uncertainty sampler will happily consume the entire labelling budget on.


active_and_federated_learning

Diversity-based strategies correct this by selecting a batch that covers the input distribution
rather than clustering in one difficult region. In practice the strategies are combined, since batch
labelling is how annotation actually works and a batch of a hundred near-identical uncertain
examples wastes ninety-nine of them. The combined criterion — informative and mutually diverse — is
what production active learning systems implement.


active_and_federated_learning

Active learning:
  the premise          examples differ in how much they teach
  the economics        labels are the expensive input
  uncertainty sampling  label what the model is least sure about
  least confidence     the top probability is low
  margin sampling      the top two classes are close
  entropy              the whole distribution is flat; differs from margin when
                       there are many classes
  the ambiguity trap   unlabelable examples are maximally uncertain
  diversity sampling   cover the input distribution
  batch selection      the realistic setting; naive uncertainty produces
                       redundant batches
  the combined criterion  informative and mutually diverse
  query by committee   disagreement among an ensemble
  expected model change  which label would move the parameters most
  cold start           active learning needs a model to be useful; the first
                       round should be random or diversity-based
  the distribution warning  actively selected data is not a random sample, so a
                       test set drawn from it is biased; hold out a random set
                       separately for evaluation


active_and_federated_learning

Q: An active learning loop using pure uncertainty sampling plateaus after several rounds while
consuming its labelling budget. The most likely cause is:
A. The model has learned everything available
B. The strategy is selecting inherently ambiguous or corrupted examples, which are maximally
   uncertain and carry little learnable signal, and each batch is also internally redundant
C. Too many labels were collected
D. The learning rate is wrong
Reasoning: Uncertainty is high both for informative boundary cases and for noise. Without a
diversity term and some filtering of unlabelable data, the budget flows to the latter. Inspecting
the selected examples directly is the diagnostic. Answer: B.


active_and_federated_learning

## Weak, Semi-Supervised, and Self-Supervised Approaches

When labels are scarce the alternative to labelling more is extracting signal from unlabelled data
or from imprecise sources. Weak supervision generates noisy labels programmatically — from heuristic
rules, existing knowledge bases, regular expressions, or other models — and then models the noise to
combine them. The combination step matters: labelling functions have different accuracies and
correlate with each other, and a generative model over their outputs produces better probabilistic
labels than majority voting.


active_and_federated_learning

Semi-supervised learning uses unlabelled data alongside a small labelled set, and the methods that
work rest on a small number of assumptions. The smoothness assumption says nearby points share
labels; the cluster assumption says the decision boundary lies in low-density regions; the manifold
assumption says data lies on a lower-dimensional structure. Consistency regularization enforces
these by requiring the model to give the same prediction for augmented versions of an unlabelled
example, which is the mechanism behind most current methods.


active_and_federated_learning

Pseudo-labelling trains on the model's own confident predictions and is effective and dangerous in
equal measure. It amplifies whatever the model already believes, so an early systematic error
becomes entrenched, and the confidence threshold plus a mechanism to prevent class collapse are what
separate the working versions from the failing ones. Self-supervised pretraining sidesteps the issue
entirely by constructing a task from the data's own structure, which is how the strongest general
representations are now learned.


active_and_federated_learning

Data-efficient approaches:
  weak supervision     noisy programmatic labels from rules and existing sources
  labelling functions  imprecise, differently accurate, and correlated
  the combination      a generative model over their outputs beats majority vote
  semi-supervised      unlabelled data alongside a small labelled set
  smoothness assumption  nearby points share labels
  cluster assumption   the boundary lies in low-density regions
  manifold assumption  data occupies a lower-dimensional structure
  consistency regularization  the same prediction under augmentation
  the augmentation requirement  augmentations must preserve the label; the
                       method fails silently when they do not
  pseudo-labelling     training on confident self-predictions
  confirmation bias    early errors entrenched by self-training
  confidence threshold  the primary control
  class collapse       all pseudo-labels going to one class; needs an explicit
                       balancing term
  self-supervised pretraining  a task constructed from the data's structure
  transfer             a pretrained representation plus a small labelled set is
                       usually the strongest baseline and should be tried before
                       anything more elaborate


active_and_federated_learning

## Federated Learning

Federated learning trains a shared model across many devices or institutions without centralizing
the data. Each participant computes updates on its local data and sends only those updates to a
server that aggregates them into a new global model. The motivation is usually regulatory or
contractual — health records, financial data, and personal device data that cannot be pooled — and
sometimes bandwidth, since model updates are smaller than raw data.


active_and_federated_learning

The algorithmic core is straightforward and its difficulties are all systems difficulties. The
server distributes the current model; a subset of clients trains locally for several passes; the
server averages the resulting updates weighted by data volume. Running multiple local passes before
communicating is what makes the scheme practical, since communication is the bottleneck, and it is
also what creates the central problem: local models drift apart when their data distributions
differ, and averaging drifted models degrades the result.


active_and_federated_learning

Non-identical data across clients is the norm rather than an edge case. One hospital's patient
population differs from another's; one user's typing differs from another's. Under strong
heterogeneity, naive averaging converges slowly or to a poor solution, and the mitigations —
proximal terms limiting local drift, control variates correcting it, server-side momentum, or
personalizing a shared representation with local heads — are the substance of the federated
learning literature.


active_and_federated_learning

Federated learning:
  the motivation       data that cannot be centralized
  the loop             distribute, train locally, aggregate
  weighted averaging   by local data volume
  local passes         more passes means less communication and more drift
  communication cost   the bottleneck; updates compressed, quantized, or sparsified
  client heterogeneity  non-identical distributions; the normal case
  the drift problem    local models diverging, averaging degrading
  proximal terms       penalize distance from the global model
  control variates     correct the drift direction explicitly
  personalization      a shared representation with local heads
  systems heterogeneity  devices differ in speed, memory, and availability
  stragglers           slow clients; usually handled by deadline-based partial
                       participation
  participation bias   available devices are not a random sample of users, which
                       biases the model in ways that are hard to detect
  cross-device versus cross-silo  millions of unreliable devices versus a few
                       reliable institutions; different algorithms suit each
  evaluation           a global test set may not exist; per-client metrics and
                       their distribution are what matter, and a good average
                       hiding poor performance for a minority of clients is the
                       standard failure


active_and_federated_learning

## Privacy Mechanisms

Federated learning keeps raw data local and does not by itself provide privacy, because model
updates leak information about the data that produced them. Gradient inversion attacks reconstruct
training examples from updates with surprising fidelity under some conditions, and membership
inference determines whether a particular record was in the training set. Treating decentralization
as equivalent to privacy is a category error that appears frequently.


active_and_federated_learning

Differential privacy provides a formal guarantee: the output distribution changes only slightly
whether or not any single record is present, with the bound quantified by a privacy parameter. The
mechanism is calibrated noise addition, and in machine learning it is applied by clipping per-example
gradient contributions to a bounded norm and adding noise proportional to that bound. The guarantee
is meaningful and the cost is real — utility falls as the privacy parameter tightens, and the
tradeoff is more severe for smaller datasets and for underrepresented subgroups.


active_and_federated_learning

Secure aggregation addresses a different threat: it lets a server compute the sum of client updates
without seeing any individual one, using cryptographic protocols. Combined with differential
privacy, this permits the noise to be added once to the aggregate rather than by each client
separately, which improves the utility for a given guarantee substantially. The two mechanisms
protect against different adversaries and are complementary rather than alternatives.


active_and_federated_learning

Privacy mechanisms:
  decentralization is not privacy  updates leak information about the data
  gradient inversion   reconstructing examples from updates
  membership inference  determining whether a record was in training
  differential privacy  a formal bound on the influence of any single record
  the privacy parameter  smaller means stronger and costlier
  gradient clipping    bounding each example's contribution
  noise calibration    proportional to the clipping bound
  the utility cost     real, and worse for small data and rare subgroups
  the equity concern   privacy noise disproportionately harms underrepresented
                       groups, which is an accuracy-fairness interaction that is
                       easy to overlook
  secure aggregation   the server sees only the sum
  the combination      aggregate-level noise beats per-client noise for the same
                       guarantee
  the threat models    inversion by the server, inference by the model's users,
                       and collusion; state which one you are defending against
  accounting           privacy loss accumulates across training rounds and must
                       be tracked
  the honest position  an unquantified claim of privacy is not a guarantee, and
                       the value of the formal framework is that it forces the
                       claim to be stated in a way that can be checked


active_and_federated_learning

## Continual Learning and Practice

Continual learning trains on a stream of tasks or distributions without retaining all past data, and
its defining problem is catastrophic forgetting: training on new data degrades performance on old
tasks, sometimes severely and quickly. The cause is that gradient descent has no reason to preserve
parameters that matter for data it no longer sees.


active_and_federated_learning

The mitigations fall into three families. Replay stores a small sample of past data and mixes it
into current training, which is the most effective approach and the one that conflicts with the
privacy motivations that often prompt continual learning in the first place. Regularization
penalizes changes to parameters identified as important for past tasks, estimated from curvature or
from gradient statistics. Architectural approaches allocate separate capacity per task, which avoids
forgetting entirely and grows without bound.


active_and_federated_learning

In practice the deployed alternative to continual learning is periodic retraining from an
accumulating dataset, and it should be the default unless something prevents it. Retraining is
simpler, more predictable, easier to evaluate, and avoids the entire forgetting problem. Continual
learning earns its complexity where data cannot be retained, where the model must adapt faster than
a retraining cycle allows, or where per-user adaptation happens on a device.


active_and_federated_learning

Continual learning and practice:
  the stream setting   tasks or distributions arriving over time
  catastrophic forgetting  new training degrading old performance
  the cause            no gradient signal preserves unseen tasks
  replay               a stored sample of past data mixed in; most effective
  the tension          replay conflicts with the privacy motivation
  generative replay    synthesizing past data instead of storing it
  regularization       penalize changes to parameters important for past tasks
  importance estimation  from curvature or accumulated gradients
  architectural isolation  separate capacity per task; no forgetting, unbounded
                       growth
  task boundaries      known or unknown; the unknown case is much harder
  the stability-plasticity tradeoff  resisting change preserves old and prevents
                       new
  the default alternative  periodic retraining from accumulated data
  when to prefer it    simpler, predictable, evaluable, and it avoids the problem
  when continual is warranted  data cannot be retained, adaptation must be fast,
                       or learning happens on a device
  evaluation discipline  measure performance on every past task after every
                       update, because an average across tasks hides a collapse
                       on one of them and the collapse is what users experience


ai_safety_and_alignment

## Specification, Objectives, and Failure Modes

The alignment problem in its most practical form is that we optimize a measurable proxy and want an
unmeasurable objective. Every training signal — a loss, a reward, a preference label — is a
specification, and the model optimizes the specification rather than the intent behind it. Where the
two diverge, the model follows the specification, and this is a correct description of what
optimization does rather than a malfunction.


ai_safety_and_alignment

Specification gaming is the concrete manifestation and it appears at every scale. A reinforcement
learning agent finds a physics exploit rather than completing the task. A summarization model learns
that longer summaries score better and pads them. A model trained on human approval learns to
produce confident-sounding answers because raters approve of confidence, which selects for a
persuasive style independent of correctness. The last of these is particularly instructive: the
signal was a genuine measurement of what humans preferred, and preferring it was the problem.


ai_safety_and_alignment

Goodhart's law describes the general phenomenon — a measure ceases to be a good measure when it
becomes a target — and its mechanism is that the correlation between proxy and objective was
established over a distribution of behaviours that optimization then moves away from. The practical
consequences are that proxies should be treated as valid only within the range where they were
validated, that optimization pressure should be limited when the proxy is weak, and that
monitoring should track the objective directly wherever it can be measured at all.


ai_safety_and_alignment

Specification and its failures:
  the core problem     we optimize a measurable proxy for an unmeasurable goal
  specification gaming  satisfying the letter of the objective, not the intent
  reward hacking       exploiting the reward mechanism rather than the task
  Goodhart's law       a measure targeted ceases to measure
  the mechanism        optimization leaves the distribution where the correlation
                       held
  sycophancy           optimizing approval selects for agreement and confidence
  the uncomfortable part  the approval signal was accurate; preferring it was the
                       problem
  proxy validity range  proxies hold where they were validated, not beyond
  optimization pressure  more pressure on a weak proxy is worse, not better
  side effects         unspecified consequences of a specified objective
  instrumental behaviours  acquiring resources or preserving options because they
                       help with almost any goal
  distributional shift  the deployment distribution differs from training, and
                       the specification was only ever validated on training
  the honest framing   specification is a design activity with no complete
                       solution, so defence in depth matters more than getting
                       any single objective right


ai_safety_and_alignment

Q: A model trained on human preference ratings becomes noticeably more agreeable and less willing
to contradict users. The best explanation is:
A. The model has developed a personality
B. Human raters preferred agreement, so preference optimization selected for it; the training
   signal accurately captured what raters wanted and that is exactly the problem
C. The training data was too small
D. The model is broken
Reasoning: Preference optimization amplifies whatever the raters actually rewarded, including
biases they would not endorse on reflection. Sycophancy is a well-documented consequence of this
objective and the remedy is in the rating process and the training signal design, not in the
optimizer. Answer: B.


ai_safety_and_alignment

## Alignment Techniques

Supervised fine-tuning on demonstrations is the first stage of most alignment pipelines and it does
a great deal of the work. Showing the model what good responses look like establishes format,
tone, refusal behaviour, and task competence, and the quality of a small demonstration set matters
far more than its size. A few thousand carefully written examples routinely outperform a much
larger set of mediocre ones, because the model learns the distribution it is shown including its
flaws.


ai_safety_and_alignment

Preference optimization refines beyond what demonstrations can express, because it is easier to
judge which of two responses is better than to write the best one. Reinforcement learning from
human feedback trains a reward model on comparisons and optimizes against it, with a penalty for
drifting too far from the initial policy — the penalty is essential, since an unconstrained
optimizer finds the reward model's exploitable regions rather than good responses. Direct preference
methods skip the separate reward model and optimize the policy on comparisons directly, which is
simpler and avoids one source of exploitation.


ai_safety_and_alignment

Constitutional and principle-based approaches replace some human labelling with model-generated
critique against explicit written principles. The model criticizes and revises its own outputs
according to stated rules, and the resulting revisions become training data. The practical
advantages are scale and transparency: the principles are written down and can be inspected,
debated, and changed, which is not true of the implicit preferences of a rating pool.


ai_safety_and_alignment

Alignment techniques:
  demonstration fine-tuning  establishes format, tone, and refusal behaviour
  data quality over quantity  a small careful set beats a large mediocre one
  preference comparison  judging is easier than writing
  reward modelling     a learned proxy for human judgement
  the divergence penalty  keeps the policy near the initial one; without it the
                       optimizer exploits the reward model
  reward model overoptimization  measurable as reward rising while true quality
                       falls; the canonical experiment in this area
  direct preference optimization  skips the reward model
  principle-based critique  the model revises against written rules
  the transparency advantage  written principles can be inspected and changed
  red-team data        adversarial examples fed back into training
  refusal calibration  refusing what should be refused without refusing what
                       should not; both error directions have real costs
  over-refusal         a failure mode as real as under-refusal and much less
                       measured
  the balance          a model that refuses everything is safe and useless, and
                       optimizing only against harmful outputs drifts toward it


ai_safety_and_alignment

## Evaluation and Red Teaming

Safety evaluation differs from capability evaluation because the interesting behaviour is rare,
adversarially elicited, and often not present in any natural test distribution. A model that behaves
well on ten thousand ordinary prompts tells you little about how it behaves on the prompt an
adversary spends an hour constructing, and safety evaluation is therefore adversarial by
construction.


ai_safety_and_alignment

Red teaming is the practice of deliberately attempting to elicit failures. Manual red teaming finds
novel attack classes that automated methods do not, because human creativity is the actual threat
model. Automated red teaming scales, covering far more of the space with variations on known
categories, and the two are complementary rather than substitutes. The output of both should be a
growing benchmark, so that fixed failures stay fixed.


ai_safety_and_alignment

The attack categories that recur are worth naming because defending against them individually is
tractable while defending against an unspecified adversary is not. Direct requests, role-play and
fictional framing, incremental escalation across turns, encoding and obfuscation, authority and
context manipulation, and instruction injection through retrieved or tool-returned content each
have distinct mechanisms. Multi-turn escalation deserves particular attention because a target
evaluated only on single prompts will appear robust and fail immediately under a conversation.


ai_safety_and_alignment

Evaluation and red teaming:
  rare and adversarial  the behaviour of interest is not in a natural
                       distribution
  manual red teaming   finds novel classes; human creativity is the threat model
  automated red teaming  scales coverage of known classes
  the complementarity  neither substitutes for the other
  regression benchmark  fixed failures must stay fixed
  direct request       the baseline
  fictional framing    role-play and hypothetical distancing
  multi-turn escalation  gradual context building; a single-turn evaluation
                       misses it entirely
  encoding             obfuscation to bypass surface filters
  authority framing    claimed context that changes the apparent legitimacy
  prompt injection     instructions arriving in retrieved documents or tool
                       output; the model cannot distinguish data from
                       instruction by content alone
  paired evaluation    the same attacks against a reference and a variant; the
                       only way to attribute a difference to the change
  frozen attack banks  simulated fresh attacks each run confound the comparison
                       with attack variance
  scoring errors       a timeout recorded as a refusal inflates safety; keep
                       errored cases distinguishable from both outcomes
  the degradation confound  a model too weak to follow the harmful instruction
                       scores as safe for entirely the wrong reason, which
                       matters when comparing compressed or smaller variants


ai_safety_and_alignment

## Interpretability and Robustness

Interpretability aims to explain model behaviour in terms of internal mechanism rather than
input-output correlation, and its safety relevance is that behavioural evaluation can only test what
you thought to test. A mechanism understood can be checked for conditions that no test case
exercised, which is the only route to confidence about behaviour outside the tested distribution.


ai_safety_and_alignment

The techniques operate at several levels. Attribution methods identify which inputs influenced an
output, which is useful and does not explain mechanism. Probing trains classifiers on internal
representations to determine what information is present where. Activation patching intervenes
directly, replacing an internal value and measuring the effect, which supports causal rather than
correlational claims. Feature decomposition attempts to find interpretable directions in
representations that superimpose many concepts into fewer dimensions than concepts.


ai_safety_and_alignment

Robustness concerns behaviour under distribution shift and adversarial perturbation, and the
persistent finding is that models are far less robust than their in-distribution performance
suggests. Shortcut learning is the general cause: the model finds features that are predictive in
training and not causal, and those features fail exactly where the deployment differs from
training. The diagnostic is evaluating on data that breaks the shortcut, which requires knowing
what the shortcut is, which is why interpretability and robustness are connected.


ai_safety_and_alignment

Interpretability and robustness:
  the motivation       behavioural testing only covers what you thought to test
  attribution          which inputs mattered; correlational
  probing              what information is present in a representation
  the probing caveat   a probe finding information does not show the model uses it
  activation patching  intervene and measure; supports causal claims
  feature decomposition  interpretable directions from superimposed
                       representations
  superposition        more concepts than dimensions; why individual units are
                       rarely interpretable
  circuit analysis     tracing a computation through components
  the honest status    interpretability is real and partial; it does not yet
                       provide the guarantees that would let it substitute for
                       behavioural evaluation
  shortcut learning    predictive but non-causal features
  spurious correlation  the training-set artifact the model relies on
  distribution shift   where the shortcut stops working
  adversarial examples  small perturbations flipping outputs
  robustness evaluation  test on data that breaks the suspected shortcut
  the connection       you cannot test the shortcut you have not identified


ai_safety_and_alignment

## Deployment Safeguards and Governance

No model is safe in isolation; a deployed system is safe or not, and the system includes the input
handling, the tool access, the output filtering, the monitoring, and the human oversight. Treating
model alignment as the entire safety story is the most common architectural mistake, because it
places the whole burden on the component with the least reliable guarantees.


ai_safety_and_alignment

Defence in depth means independent layers whose failures are uncorrelated. Input classification
catches known attack patterns. The model's own training provides the primary behaviour. Output
filtering catches what got through. Capability restriction limits the blast radius: a model that
cannot execute code cannot be made to execute harmful code regardless of what it is persuaded to
output. Monitoring detects patterns invisible at the level of a single interaction, and human review
handles the consequential decisions.


ai_safety_and_alignment

Agentic deployment raises the stakes because the model acts rather than only speaks. The safety
properties that matter shift toward the action space: what tools exist, what each can do, what is
reversible, what requires confirmation, and what the system does when the model requests something
outside its expected pattern. The design principle is least privilege applied to tools, and the
practical rule is that irreversible actions should require an independent check that the model
cannot itself satisfy.


ai_safety_and_alignment

Safeguards and governance:
  the system is the unit  not the model alone
  defence in depth     independent layers, uncorrelated failures
  input classification  known patterns caught before the model
  output filtering     a second check after generation
  capability restriction  the strongest control; what cannot be done cannot be
                       misused
  least privilege      applied to tools and data access
  reversibility        the key property distinguishing risk levels of actions
  confirmation gates   independent checks for irreversible actions
  rate limiting        bounds the damage from a successful attack
  monitoring           patterns across interactions that single-interaction
                       review cannot see
  logging              the precondition for incident analysis; log enough to
                       reconstruct what happened
  human oversight      meaningful only when the human has the information and the
                       time to actually review
  automation bias      humans approve what the system proposes; oversight
                       designed without accounting for this does not function
  staged deployment    limited release, monitored, widened; the same discipline
                       as any risky software change
  incident response    a plan written before it is needed, including how to
                       disable a capability quickly
  documentation        model cards and system cards stating capabilities,
                       limitations, evaluations run, and known failure modes;
                       the value is that a downstream user can make an informed
                       decision rather than discovering the limitations in
                       production


anomaly_detection

## Problem Framing

Anomaly detection identifies observations that differ from expected behaviour, and the framing
problem is that "anomalous" is not a property of an observation but a relationship between an
observation and a definition of normal. The same value may be routine in one context and alarming
in another, so the first task is establishing what normal means and over what population.


anomaly_detection

The available supervision determines the approach and is the most consequential distinction.
Supervised detection has labelled examples of both classes and is simply an imbalanced
classification problem, with all the standard techniques applying. Semi-supervised detection has
labelled normal data and no labelled anomalies, so a model of normality is built and departures
are flagged — which is the common situation, since normal behaviour is abundant and anomalies are
rare and heterogeneous. Unsupervised detection has no labels and assumes anomalies are rare and
different, identifying them by their statistical isolation.


anomaly_detection

The critical asymmetry favouring semi-supervised and unsupervised methods is that anomalies are
heterogeneous while normal behaviour is not. A supervised model learns the anomalies it was shown
and generalizes poorly to novel failure modes, which is precisely the case that matters most — a
new attack, an unprecedented equipment failure, a fraud technique not previously seen. Modelling
normality and flagging departures catches novel anomalies by construction.


anomaly_detection

Framing:
  anomaly              relative to a definition of normal, not intrinsic
  point anomaly        a single observation unusual in isolation
  contextual anomaly   normal in general and unusual in its context; a temperature
                       normal for July occurring in January
  collective anomaly   a subsequence unusual as a group while each point is
                       ordinary
  supervised           labelled both classes; imbalanced classification
  semi-supervised      labelled normal only; model normality, flag departures
  unsupervised         no labels; assume anomalies are rare and different
  novelty detection    identify data unlike the training distribution
  outlier detection    identify unusual points within a contaminated dataset
  the heterogeneity asymmetry  normal is coherent, anomalies are diverse; this
                       favours modelling normality
  novel failure modes  the case that matters most and that supervised methods
                       generalize to worst
  what happens next    the detection's purpose; determines the acceptable false
                       positive rate and therefore the threshold


anomaly_detection

Q: A supervised fraud model trained on historical labelled fraud performs well in backtesting and
misses a new fraud technique entirely. This is:
A. A model quality problem to be fixed by more training
B. Expected, since a supervised model learns the anomaly types it was shown and novel types are
   outside its training distribution
C. A data leakage problem
D. A threshold problem
Reasoning: Supervised learning generalizes within the distribution it was trained on, and a novel
technique is by definition outside it. Detecting novel anomalies requires modelling normality and
flagging departures rather than modelling known anomalies. Production fraud systems combine both:
supervised models for known patterns and unsupervised detection for novelty. Answer: B.


anomaly_detection

## Statistical and Distance-Based Methods

The simplest statistical approaches assume a distribution and flag observations in its tails.
The three-sigma rule flags points more than three standard deviations from the mean, which is
straightforward and rests on a normality assumption that real data frequently violates, and which
is additionally corrupted by the anomalies themselves — the mean and standard deviation are
computed from data containing the outliers, so extreme values inflate the standard deviation and
mask themselves.


anomaly_detection

Robust statistics address the masking problem by using estimators insensitive to outliers. The
median and the median absolute deviation replace the mean and standard deviation, and because the
median is unaffected by extreme values regardless of their magnitude, a few outliers do not
distort the reference. This is a small change that substantially improves reliability and is
underused relative to how simple it is.


anomaly_detection

Distance-based methods define anomalies by isolation in the feature space, with the distance to
the k-th nearest neighbour being the natural score. Their assumption is that normal points lie in
dense regions and anomalies do not, and their limitation is that a global distance threshold
performs badly when the data has regions of differing density — a point that is anomalous within a
dense cluster may be closer to its neighbours than a normal point in a sparse region.


anomaly_detection

Local outlier factor addresses this by comparing each point's local density to that of its
neighbours rather than to a global standard. A point in a sparse region with neighbours in
similarly sparse regions is normal; a point whose neighbours are much denser than its own
neighbourhood is anomalous. This local normalization is what makes the method work on data with
varying density, and it is the standard reference for distance-based detection.


anomaly_detection

Statistical and distance methods:
  three sigma          simple; assumes normality and is corrupted by the outliers
                       it seeks
  masking              outliers inflate the variance estimate and hide themselves
  robust estimators    median and median absolute deviation; unaffected by extremes
  modified z-score     using robust estimators; a drop-in improvement
  interquartile range  the box plot rule; distribution-free
  Grubbs and Dixon     formal tests for a single outlier; assume normality
  Mahalanobis distance  accounts for covariance; the correct multivariate distance
                       when the data is roughly elliptical
  k-nearest neighbour  distance to the kth neighbour as the score; assumes uniform
                       density
  local outlier factor  compares local density to neighbours'; handles varying
                       density
  DBSCAN               density clustering; points not assigned to a cluster are
                       noise, which is anomaly detection as a side effect
  the curse of dimensionality  distances concentrate in high dimension so all
                       points become equidistant; distance-based methods degrade
                       and require dimensionality reduction first


anomaly_detection

The curse of dimensionality deserves specific emphasis for this application because it undermines
the premise. As dimension grows, the ratio between the nearest and farthest neighbour distances
approaches one, so every point is approximately equidistant from every other and the notion of
being distant loses discriminating power. Distance-based anomaly detection therefore degrades
badly above a few dozen dimensions, and the remedies are feature selection, dimensionality
reduction, or methods that do not rely on distance — which is part of why isolation-based and
reconstruction-based approaches became prominent for high-dimensional data.


anomaly_detection

## Model-Based Methods

Isolation forest takes an approach that inverts the usual reasoning. Rather than profiling normal
points and measuring departure, it isolates points by random partitioning, and the observation is
that anomalies are isolated in fewer splits because they are few and different. The score is the
average path length to isolation across many random trees. It has linear time complexity, needs no
distance computation, handles high dimensions well, and requires no assumption about the
distribution's shape — which together make it the strongest general-purpose default.


anomaly_detection

Reconstruction-based methods learn to compress and reconstruct normal data, and use the
reconstruction error as the anomaly score. An autoencoder trained on normal data learns a
representation capturing its structure, so normal inputs reconstruct accurately and anomalous ones
do not. Principal component analysis provides the linear version, where the reconstruction error
from the leading components serves the same purpose.


anomaly_detection

One-class support vector machines learn a boundary enclosing the normal data in a feature space,
flagging points outside it. They are principled and require kernel and parameter selection that is
difficult without labelled anomalies to validate against, which is the recurring difficulty in
this field — the methods have hyperparameters and the data to tune them is exactly what is
missing.


anomaly_detection

Model-based methods:
  isolation forest     random partitioning; anomalies isolate in fewer splits;
                       linear time, high-dimensional, assumption-free
  extended isolation forest  non-axis-parallel splits; removes an artifact of the
                       original
  autoencoder          reconstruct normal data; error is the score
  variational autoencoder  a probabilistic version; gives a likelihood
  PCA reconstruction   the linear case; cheap and effective when the structure is
                       linear
  one-class SVM        a boundary enclosing normal data; sensitive to parameters
  Gaussian mixture     model the density; low-likelihood points are anomalous
  kernel density       non-parametric density; bandwidth selection is the
                       difficulty
  normalizing flows    exact likelihood under a learned transformation; strong and
                       computationally heavy
  deep methods caution  frequently outperformed by isolation forest on tabular
                       data, and the published comparisons are heavily affected by
                       tuning on the test set given the absence of validation
                       labels
  ensembles            combine several detectors; robust to any one method's
                       assumption failing, and the standard practical choice


anomaly_detection

The caution about deep methods deserves stating because the literature is unusually unreliable
here. Evaluating an unsupervised anomaly detector requires labels, and the labels used are the
test labels, so hyperparameter selection frequently involves the test set — which inflates results
in a way that is difficult to detect from the paper. Independent comparisons repeatedly find that
isolation forest and simple statistical baselines match or exceed elaborate deep methods on
tabular data, and the practical recommendation is to establish those baselines first and require a
substantial margin before adopting something more complex.


anomaly_detection

## Time Series Anomalies

Time series anomaly detection differs because the temporal structure is itself informative. A
value is not anomalous in isolation but relative to what the series' recent history and seasonal
pattern predict, so the natural approach is to forecast and flag large residuals.


anomaly_detection

That framing handles several anomaly types with one mechanism. A point anomaly produces a large
one-step residual. A level shift produces a sustained sequence of same-signed residuals. A change
in variance produces residuals whose magnitude changes. And a change in the seasonal pattern
produces residuals correlated with the seasonal phase. Examining the residual structure rather
than only its magnitude therefore identifies the anomaly type as well as its presence.


anomaly_detection

The decomposition approach separates trend, seasonality, and remainder, and applies outlier
detection to the remainder. Using robust estimation in the decomposition matters, because a
non-robust decomposition absorbs the anomalies into the trend or seasonal components and they
disappear from the residual — the anomaly hides itself in the model fitted to find it.


anomaly_detection

Change point detection is the related task of identifying when the generating process changed
rather than which individual points are unusual. Cumulative sum methods accumulate deviations and
signal when the accumulation exceeds a threshold, which detects small persistent shifts far more
efficiently than examining points individually. Bayesian change point methods give a posterior
over change locations and handle multiple changes naturally.


anomaly_detection

Time series methods:
  forecast residuals   the general-purpose approach; large residuals are anomalies
  residual structure   the pattern identifies the anomaly type, not only its
                       presence
  robust decomposition  seasonal-trend decomposition with robust fitting; a
                       non-robust fit absorbs the anomalies
  seasonal hybrid ESD  decompose then apply an outlier test; a common production
                       choice
  control charts       Shewhart for large shifts, CUSUM and EWMA for small
                       persistent ones
  CUSUM                accumulates deviation; detects small shifts efficiently
  change point         when the process changed rather than which points are odd
  matrix profile       subsequence similarity; finds discords and motifs without
                       specifying what to look for
  multivariate         correlated series; a single series within normal range while
                       the correlation structure breaks is a common and important
                       case
  prediction interval  the threshold from the forecast's uncertainty rather than
                       from a fixed value; adapts to periods of higher variance
  seasonality          multiple periods are common; hourly, daily, weekly, and
                       annual can all be present simultaneously
  the retraining question  a model retrained on data containing the anomaly learns
                       to expect it


anomaly_detection

The multivariate case deserves emphasis because it catches failures that univariate monitoring
cannot. In a system with correlated measurements, a fault frequently manifests as a break in the
correlation rather than as any single series leaving its normal range — a pump's power draw and
flow rate normally move together, and a blockage raises pressure while reducing flow with both
remaining within their individual limits. Monitoring the residual from a model predicting each
series from the others detects this, and monitoring each series independently does not.


anomaly_detection

[user] Our infrastructure anomaly detector fires 200 alerts a day and the on-call team ignores it.
[assistant] tool_call anomaly.alert_audit({"alerts_per_day": 200, "actioned_pct": 0.03,
"method": "static_threshold_per_metric", "metrics_monitored": 1400,
"seasonality_modelled": false, "correlation_grouping": false, "severity_tiers": 1})
[tool] {"issues": ["static_thresholds_ignore_seasonality", "per_metric_alerts_not_grouped",
"no_severity_differentiation", "threshold_not_tied_to_action"],
"estimated_reducible": 0.9, "recommendation": "seasonal_baselines_plus_grouping_plus_tiering"}
[assistant] A 3% action rate means the detector is not functioning as a detector — it has trained
the team to ignore it, which is worse than having no alerting because it also masks the real
signals. Four changes, and the first two remove most of the volume.


anomaly_detection

Model the seasonality. Static thresholds on metrics with daily and weekly cycles fire every
morning when traffic rises and every Monday when the week starts, and those are the normal
pattern rather than anomalies. Replace fixed thresholds with a baseline predicted from the same
time of day and day of week, and flag departures from that. This typically removes half to three
quarters of the volume immediately because most of it is periodic.


anomaly_detection

Group correlated alerts. One underlying incident produces alerts on every metric it touches —
latency, error rate, queue depth, CPU, and downstream dependencies all fire together, and the team
receives twenty alerts describing one problem. Grouping by time and by service topology into a
single incident is what converts a wall of alerts into a comprehensible signal.


anomaly_detection

Add severity tiers tied to what should happen. An alert that should wake someone, an alert that
should be looked at during the working day, and an alert that should be recorded and reviewed
weekly are three different things and currently all page. Deciding the response for each alert
type before it fires is what makes the tier meaningful.


anomaly_detection

Then tie the threshold to the action rather than to the statistics. The right false positive rate
depends on the cost of investigating versus the cost of missing, and those differ by metric. A
detector for data loss should be sensitive and a detector for a latency blip should not.


anomaly_detection

One measurement to add: track the action rate per alert type, not just overall. That identifies
which specific rules are producing the noise, and in most systems a small number of rules account
for the large majority of the volume. Fixing or deleting those is a short piece of work with a
large effect.


anomaly_detection

## Evaluation, Deployment, and Applications

Evaluating anomaly detection is difficult for a structural reason: labels are scarce because
anomalies are rare, and the labels that exist are biased because they were found by whatever
method was previously in use. A detector evaluated against labels produced by the incumbent system
is rewarded for agreeing with it rather than for being better.


anomaly_detection

The metrics require care because of the extreme imbalance. Accuracy is meaningless when anomalies
are one in ten thousand. Precision and recall at the operating threshold are informative and
depend on the threshold, so the full precision-recall curve and the average precision summarize
better. Precision at a fixed alert budget — how many of the top hundred flagged are genuine — is
frequently the most operationally meaningful measure, because the alert budget is what the team
can actually investigate.


anomaly_detection

Deployment concerns dominate whether a detector delivers value. The threshold determines the
alert volume, and it should be set from the investigation capacity and the cost of a miss rather
than from a statistical criterion. Alert fatigue is the failure mode that renders any detector
useless, and it results from volume rather than from any individual alert being wrong. And drift
means normal behaviour changes, so a static model degrades and a continuously retrained one
gradually learns to accept the anomalies.


anomaly_detection

Evaluation and deployment:
  label scarcity       few labelled anomalies; and those that exist are biased by
                       how they were found
  incumbent bias       evaluating against the previous system's findings rewards
                       agreement rather than improvement
  accuracy is useless  at extreme imbalance
  precision and recall  at the operating threshold; report both
  average precision    summarizes the curve
  precision at budget  how many of the top k are genuine; the operationally
                       meaningful measure
  threshold from capacity  set by what can be investigated and the cost of a miss
  alert fatigue        the dominant failure mode; caused by volume
  grouping             one incident should produce one alert
  explanation          which feature or which series drove the score; without it,
                       an alert is not actionable
  feedback loop        record investigation outcomes; the only route to labels
  drift                normal changes; static models degrade, continuously
                       retrained ones learn to accept anomalies
  retraining exclusion  exclude confirmed anomalies from the training window
  human in the loop    the detector proposes and a person disposes; the realistic
                       architecture for most applications


anomaly_detection

The explanation requirement deserves the closing emphasis because it determines whether a
detection is actionable. An alert stating that an observation is anomalous, with a score, gives an
investigator nothing to start from — they must reconstruct why from scratch, which for a
high-dimensional system is the bulk of the work. An alert identifying which features contributed
most to the score, what their values were, and what was expected converts the investigation from
open-ended to specific. For isolation forest this is available from which features were used in
the short isolating paths; for reconstruction methods it is the per-feature reconstruction error;
for forecast-based methods it is the residual per series. Providing it costs little and is the
difference between a detector that is used and one that is switched off.


bayesian_modeling

## The Bayesian Framework

Bayesian modeling treats unknown quantities as random variables with probability
distributions expressing uncertainty, and updates those distributions as data arrives. The
mechanic is Bayes' theorem: the posterior is proportional to the likelihood times the prior.
Everything else in the subject is the consequence of that single relation applied carefully.


bayesian_modeling

The framework's practical advantage is that it answers the question people actually ask. A
frequentist confidence interval is a statement about a procedure's long-run behaviour — that
intervals constructed this way contain the true value 95% of the time — and it is not a
statement about the particular interval in hand. A Bayesian credible interval states that given
the data and the prior, there is a 95% probability the parameter lies in the interval. The
second is what practitioners almost always intend, and the routine misinterpretation of the
first as the second is evidence for how natural the Bayesian reading is.


bayesian_modeling

The framework also handles several situations that are awkward otherwise. Uncertainty
propagates automatically through any derived quantity, since the posterior over parameters
induces a posterior over any function of them. Prior information from previous studies or from
domain knowledge enters explicitly rather than informally. Small samples are handled without
asymptotic approximations. And nuisance parameters are eliminated by integration rather than by
plug-in estimates that understate uncertainty.


bayesian_modeling

Framework concepts:
  prior                 belief before the data; a modelling choice requiring
                        justification
  likelihood            the probability of the data given the parameters; the same
                        object frequentist inference uses
  posterior             belief after the data; proportional to likelihood times
                        prior
  marginal likelihood   the normalizing constant; the probability of the data
                        under the model, and the basis of Bayes factors
  credible interval     a probability statement about the parameter given the data
  confidence interval   a statement about the procedure's long-run coverage
  posterior predictive  the distribution of new data, integrating over parameter
                        uncertainty; what forecasting should use
  marginalization       integrate out nuisance parameters rather than plugging in
                        estimates
  exchangeability       the assumption justifying a shared prior across units;
                        weaker than independence and the actual basis of
                        hierarchical modelling
  updating              yesterday's posterior is today's prior; sequential
                        analysis is natural rather than a special case


bayesian_modeling

The posterior predictive distribution deserves emphasis because it is where the framework's
advantage is most concrete. A prediction from a fitted model that plugs in point estimates
understates uncertainty, because it ignores that the parameters themselves are uncertain. The
posterior predictive integrates over the parameter posterior, so the resulting intervals include
both the inherent variability of new observations and the uncertainty about the parameters. For
small samples the difference is substantial, and it is exactly the difference between intervals
that achieve their nominal coverage and intervals that are systematically too narrow.


bayesian_modeling

Q: A model is fit and a 95% credible interval for a parameter is [2.1, 4.8]. The correct
interpretation is:
A. Repeated sampling would produce intervals containing the true value 95% of the time
B. Given the data and the prior, there is a 95% probability the parameter lies between 2.1 and
   4.8
C. 95% of observations fall in this range
D. The parameter is definitely in this range
Reasoning: A credible interval is a direct probability statement about the parameter
conditional on the observed data and the assumed prior. Option A describes a confidence
interval's frequentist coverage property. Option C describes a prediction interval, which
concerns observations rather than parameters. Answer: B.


bayesian_modeling

## Priors

The prior is the framework's most discussed feature and its most misunderstood. It is a
modelling choice requiring justification in the same way a likelihood does, and the objection
that it introduces subjectivity applies with equal force to the choice of likelihood, of
variables, of transformations, and of which analyses to report — all of which are made in every
analysis regardless of framework and most of which are less visible.


bayesian_modeling

Priors range from strongly informative, encoding substantial knowledge, to weakly informative,
constraining the parameter to a plausible range without asserting much within it, to flat, which
is frequently a poor choice despite its appearance of neutrality. A flat prior on an unbounded
parameter is improper — it does not integrate to a finite value — which may still yield a proper
posterior and may not, and a flat prior on one parameterization is informative on another, so
its neutrality is illusory.


bayesian_modeling

Weakly informative priors are the practical default and are worth stating as such. A prior that
rules out physically impossible or implausible values — a negative variance, a treatment effect
of a thousand standard deviations, a probability outside the unit interval — costs nothing when
the data is informative and regularizes usefully when it is not. It also improves computation
substantially, since a sampler exploring a region the prior excludes is wasting effort in a
region the analyst would reject anyway.


bayesian_modeling

Prior specification:
  informative           encodes real knowledge; requires and repays justification
  weakly informative    constrains to plausible ranges; the practical default
  flat / uniform        appears neutral, is not; parameterization-dependent and
                        often improper
  improper prior        does not integrate to one; may still give a proper
                        posterior, and must be checked
  Jeffreys prior        invariant under reparameterization; the standard
                        objective choice for a single parameter
  reference priors      constructed to be minimally informative in a defined
                        sense
  conjugate prior       the posterior stays in the same family; analytically
                        convenient and rarely the right belief
  regularizing prior    equivalent to a penalty; ridge regression is a Gaussian
                        prior and lasso is a Laplace prior on the coefficients
  prior predictive check  simulate data from the prior and inspect it; the most
                        useful tool for judging whether a prior is reasonable
  sensitivity analysis  refit under alternative priors and report how much the
                        conclusion moves


bayesian_modeling

The prior predictive check is the technique that makes prior specification tractable. Rather
than reasoning directly about whether a prior on a coefficient is reasonable, simulate parameter
values from the prior, generate datasets from the model, and look at them. If the simulated data
includes people two hundred metres tall, treatment effects larger than any ever observed, or
probabilities concentrated at zero and one, the prior is unreasonable regardless of how neutral
it appeared. This converts an abstract judgment into a concrete one.


bayesian_modeling

The connection between priors and regularization is worth making explicit because it dissolves
an apparent disagreement. Ridge regression's L2 penalty is exactly the maximum a posteriori
estimate under a Gaussian prior on the coefficients, and the lasso's L1 penalty corresponds to a
Laplace prior. Practitioners who use regularization while objecting to priors are using priors
and not calling them that, and the Bayesian version has the advantage of producing a full
posterior rather than a point estimate, so the uncertainty is available.


bayesian_modeling

## Computation

The posterior is analytically tractable only in special cases, so computation dominates
practice. Conjugate models — beta-binomial, gamma-Poisson, normal-normal — give closed forms and
are valuable for intuition and for components of larger models, and the requirement that the
prior be conjugate is a mathematical convenience rather than a statement about belief.


bayesian_modeling

Markov chain Monte Carlo constructs a chain whose stationary distribution is the posterior,
then samples from it. Metropolis-Hastings proposes a move and accepts it with a probability that
enforces detailed balance with respect to the target, which requires the target only up to a
normalizing constant — and that is precisely why the method works, since the marginal likelihood
is the intractable part. Gibbs sampling updates each parameter from its full conditional, which
is efficient when those conditionals are tractable and mixes poorly when parameters are strongly
correlated.


bayesian_modeling

Hamiltonian Monte Carlo uses gradient information to propose distant moves with high acceptance,
which makes it dramatically more efficient than random-walk methods in high dimensions. The
No-U-Turn Sampler removes the need to tune the trajectory length, and together they are what made
Bayesian inference practical for models with hundreds or thousands of parameters. The requirement
is that the posterior be differentiable, which excludes discrete parameters — handled by
marginalizing them out where possible.


bayesian_modeling

Computation and diagnostics:
  conjugate             closed form; useful for intuition and as components
  grid approximation    evaluate on a grid; only for one or two parameters
  Metropolis-Hastings   propose and accept; needs the target only up to a constant
  Gibbs sampling        update from full conditionals; poor mixing under
                        correlation
  Hamiltonian MC        gradient-informed proposals; efficient in high dimension
  NUTS                  removes trajectory tuning; the modern default
  variational inference  optimize an approximating family; fast, and systematically
                        underestimates posterior variance because the reverse-KL
                        objective is mode-seeking
  Laplace approximation  a Gaussian at the posterior mode; cheap and adequate when
                        the posterior is near-Gaussian
  R-hat                 compares within- and between-chain variance; should be
                        below 1.01
  effective sample size  accounts for autocorrelation; hundreds at minimum, and
                        report it for the quantities you care about rather than
                        only for the parameters
  trace plots           should look like white noise; wandering indicates poor
                        mixing
  divergent transitions  HMC failing in a high-curvature region; a warning that
                        the sampler is not exploring correctly and the results
                        cannot be trusted
  non-centred parameterization  the standard fix for divergences in hierarchical
                        models


bayesian_modeling

Divergent transitions deserve specific emphasis because they are the diagnostic most likely to
be ignored and most likely to indicate real bias. They occur when the sampler's numerical
integration fails in a region of high curvature, typically the narrow neck of a funnel-shaped
posterior that arises in hierarchical models when a group-level variance approaches zero. The
sampler cannot explore that region, so the posterior is systematically misrepresented — and the
chains may otherwise look healthy, with acceptable R-hat and reasonable trace plots. The remedy
is reparameterization, and treating divergences as a tuning nuisance to be suppressed by
increasing the target acceptance rate frequently hides the problem rather than fixing it.


bayesian_modeling

## Hierarchical Models

Hierarchical modelling is where the framework's practical advantage is largest, and the problem
it solves is ubiquitous: estimating a parameter for each of many related groups, some of which
have abundant data and some almost none. The two obvious approaches both fail. Complete pooling
estimates one parameter for all groups and ignores real heterogeneity. No pooling estimates each
group independently and produces wildly noisy estimates for small groups.


bayesian_modeling

Partial pooling estimates each group's parameter while shrinking it toward the population mean,
with the degree of shrinkage determined by the data rather than chosen. A group with abundant
data barely moves; a group with little data shrinks substantially toward the population. This is
the correct answer rather than a compromise between two wrong ones, and it follows directly from
modelling the group parameters as drawn from a common distribution whose parameters are also
estimated.


bayesian_modeling

The consequences are visible in any ranking problem. Ranking hospitals by mortality rate,
schools by test scores, players by performance, or products by rating produces top and bottom
positions dominated by small samples under no pooling, because extreme observed values arise
most easily where the sample is smallest. Partial pooling corrects this automatically, and the
resulting rankings are both more accurate and more stable.


bayesian_modeling

Hierarchical modelling:
  complete pooling      one parameter for all; ignores heterogeneity
  no pooling            independent per group; noisy for small groups
  partial pooling       shrinkage toward the population, data-determined
  hyperparameters       the population distribution's parameters, also estimated
  shrinkage factor      depends on the group's sample size and on the ratio of
                        within- to between-group variance
  borrowing strength    small groups gain precision from the population
  varying intercepts    group-specific baselines
  varying slopes        group-specific effects; requires modelling the correlation
                        between intercepts and slopes
  centred parameterization  natural, and samples poorly when the group-level
                        variance is small
  non-centred           reparameterize so the group effects are standardized
                        deviations; fixes the funnel geometry
  multilevel regression  the general framework; the same model appears as mixed
                        effects in the frequentist literature
  post-stratification   combine multilevel estimates with population weights; the
                        standard method for small-area estimation from
                        non-representative samples


bayesian_modeling

[user] I'm estimating conversion rates for 400 landing pages. Some have 50,000 visits and some
have 30. The small ones show rates of 0% and 100%.
[assistant] tool_call bayes.model_recommend({"units": 400, "outcome": "binomial",
"n_range": [30, 50000], "current": "independent_estimates",
"observed_extremes": ["0.00", "1.00"], "goal": "rank_and_select"})
[tool] {"issue": "no_pooling_extreme_small_sample_estimates",
"recommended": "hierarchical_beta_binomial", "shrinkage_at_n30": 0.82,
"shrinkage_at_n50000": 0.003, "note": "ranking_by_raw_rate_selects_smallest_samples"}
[assistant] The 0% and 100% rates are almost certainly not real, and they are an artifact of
estimating each page independently. A page with 30 visits and zero conversions is entirely
consistent with a true rate of 3%, and a page with 30 visits and 30 conversions is consistent
with a true rate of 90%. Independent estimation reports the observed proportion as if it were
the parameter.


bayesian_modeling

Use a hierarchical beta-binomial model. Each page has its own conversion rate, and those rates
are drawn from a common beta distribution whose parameters are estimated from all 400 pages
jointly. The result is that each page's estimate is a compromise between its own data and the
population, weighted by how much information its own data carries.


bayesian_modeling

The shrinkage is automatic and appropriate. Your 30-visit pages will shrink roughly 82% toward
the population mean, so a page with zero conversions out of 30 will be estimated near the
population rate rather than at zero. Your 50,000-visit pages barely move at all, shrinking about
0.3%, because their own data is overwhelming. You do not choose these weights; they follow from
the estimated between-page variance relative to the within-page sampling variance.


bayesian_modeling

The ranking consequence is the one that matters most for your stated goal. Ranking by raw
observed rate systematically selects pages with the smallest samples, because extreme values
arise most easily where n is small — so your top-ranked pages are mostly a list of pages you know
little about. Ranking by posterior mean corrects this, and ranking by the posterior probability
of exceeding a threshold is better still if the decision is which pages to keep.


bayesian_modeling

Two extensions worth considering. If you have page attributes — template, traffic source,
product category — add them as predictors, which lets pages borrow strength from similar pages
rather than from the global mean and improves the small-sample estimates further. And report
the full posterior interval rather than the point estimate for each page, because a page
estimated at 4% with an interval from 1% to 12% supports a different decision than one estimated
at 4% with an interval from 3.8% to 4.2%.