devops_and_infrastructure

## Reliability, Incidents, and Cost

Service level objectives make reliability a deliberate tradeoff rather than an aspiration.
An objective states a target for a measured indicator over a window — for instance that 99.9
percent of requests succeed over thirty days — and the complement is the error budget, the
amount of failure permitted. This reframes reliability usefully: a team within its error
budget can spend it on faster releases and riskier changes, while a team that has exhausted it
should stop feature work and address reliability. The budget converts an unbounded argument
about how reliable to be into a number both engineering and product can reason about.


devops_and_infrastructure

The right target is lower than instinct suggests. Each additional nine costs
disproportionately more, and beyond a point the user cannot perceive the difference because
their own network and device dominate. A service whose clients experience 99.9 percent
reliability from their connectivity gains nothing from being five nines itself. The objective
should be derived from what users need and notice, not from how many nines sound impressive.


devops_and_infrastructure

Incident response benefits from structure established before it is needed. A declared
incident should have a single coordinator whose job is coordination rather than debugging, a
communications channel, and a written timeline maintained as events unfold. The first
priority is mitigation — restore service — rather than diagnosis, which is why rollback should
be the reflex and root cause analysis should wait. Well-run incidents look procedural and
unhurried; badly run ones have several people independently changing production.


devops_and_infrastructure

Reliability and cost practices:
  SLI / SLO / budget    measured indicator, target, permitted shortfall
  choose targets from   user-perceptible impact, not aspiration
  incident roles        coordinator, communicator, subject matter experts
  mitigate first        rollback before diagnosis
  blameless postmortem  systems and conditions, not individuals; people describe
                        what they saw honestly only when it is safe to do so
  action items          owned and tracked; a postmortem without them is theatre
  chaos engineering     deliberately inject failure to verify assumptions; start
                        in staging and with a hypothesis
  game days             rehearse incidents; the response is a skill that decays
  runbooks              per-alert, kept current, written for someone woken at 3am
  right-sizing          the largest cloud saving; instances are usually
                        provisioned for a peak that never occurs
  autoscaling           on the metric that actually predicts load
  spot instances        large discounts for interruptible batch work
  lifecycle policies    tier and expire stored data automatically
  cost visibility       per-team attribution; unattributed cost is unmanaged


devops_and_infrastructure

Blameless postmortems are a technique with a specific mechanism rather than a cultural
nicety. The purpose of a postmortem is to learn what conditions allowed a failure, and that
requires accurate accounts from the people involved. If naming a cause risks consequences for
an individual, accounts become defensive and the actual sequence of events is not recovered.
Framing the analysis around systems, tooling, documentation, and conditions rather than
around individual decisions is what makes honest reconstruction possible — and the finding is
nearly always that a person acted reasonably given the information available, which locates
the fix in the information rather than in the person.


distributed_systems

## Failure Models and the Fallacies

A distributed system is one in which the failure of a computer you did not know
existed can render your own computer unusable. That definition, Leslie Lamport's, is
the whole subject in one sentence: the difficulty is not computation but partial
failure. A single machine either works or does not; a distributed system is
perpetually in a state where some fraction of it is broken, and the code must be
correct anyway.


distributed_systems

The eight fallacies of distributed computing enumerate the assumptions that seem
harmless and are not: the network is reliable, latency is zero, bandwidth is infinite,
the network is secure, topology does not change, there is one administrator, transport
cost is zero, and the network is homogeneous. Each has a corresponding class of
production incident. Assuming reliability produces code with no timeout and no retry
that hangs forever. Assuming zero latency produces the chatty interface that makes
sixty sequential calls to render one page and works fine in development where
everything is on localhost. Assuming infinite bandwidth produces the endpoint that
returns the entire table because pagination seemed like premature optimization.


distributed_systems

Failure models are ordered by severity, and the model you assume determines which
algorithms are available. Crash-stop failures mean a node halts and stays halted —
the simplest model and the one most practical systems target. Crash-recovery allows
the node to return, possibly having lost in-memory state, which is why durable write-
ahead logs exist. Omission failures drop messages. Byzantine failures allow arbitrary
behaviour including malicious lying, which requires 3f + 1 nodes to tolerate f faults
rather than the 2f + 1 sufficient for crashes, and is why Byzantine-tolerant protocols
are reserved for adversarial settings like blockchains rather than for a datacentre
where the machines are yours.


distributed_systems

The hardest failure is the one that is not a failure: the partial or gray failure. A
node that responds slowly rather than not at all, a disk that returns corrupted data
without an error, a network link that drops 3% of packets, a process that accepts
connections but has deadlocked its worker pool. These defeat naive health checks
because the node passes a liveness probe while doing no useful work. This is why
health checks should exercise the actual dependency path, why timeouts must be
treated as failures rather than as conditions to wait out, and why request hedging —
sending a duplicate request to a second replica after a delay — is worth its cost at
the tail.


distributed_systems

Timeout and retry discipline:
  every remote call has a timeout        no exceptions; a missing timeout is a bug
  timeout budgets are hierarchical       a caller's budget bounds all its callees
  retry only idempotent operations       or carry an idempotency key
  exponential backoff with jitter        without jitter, retries synchronize into
                                         a thundering herd that prevents recovery
  cap total retries and total time       infinite retry converts a blip into an
                                         outage by saturating the recovering node
  circuit breaker on sustained failure   stop calling a dead dependency; fail fast
  distinguish retryable from terminal    a 400 will fail identically forever


distributed_systems

Jitter deserves its own emphasis because its absence causes a specific and severe
failure mode. When a service becomes briefly unavailable, every client fails at
approximately the same moment. With deterministic backoff they all retry at the same
moment, producing a synchronized load spike that knocks the service down again just as
it recovers, repeating indefinitely. Adding uniform random jitter — full jitter, where
the delay is uniform on [0, backoff], is the standard recommendation — spreads the
retries and lets the service recover. This is a one-line change that repeatedly
determines whether an incident lasts two minutes or two hours.


distributed_systems

Q: A service adds retries with exponential backoff to improve reliability, and its
availability during dependency blips gets worse. The most likely cause is:
A. The backoff multiplier is too large
B. Retries without jitter synchronize clients into a thundering herd
C. The timeout is too long
D. Retries are not idempotent
Reasoning: Synchronized retries concentrate load at the exact moments the dependency
is attempting to recover, amplifying rather than absorbing the failure. Full jitter
spreads them. Option D is a real hazard but produces incorrect data rather than reduced
availability. Answer: B.


distributed_systems

## Consistency Models

The CAP theorem states that a system experiencing a network partition must choose
between consistency and availability. It is widely cited and widely misread. The
choice only applies *during a partition*: a system is not obliged to sacrifice
anything when the network is healthy, so "CP" and "AP" describe partition behaviour
rather than steady-state properties. Nor is availability binary — real systems
degrade specific operations rather than going wholly down. PACELC extends the
statement usefully: during a partition, choose availability or consistency; else, in
normal operation, choose latency or consistency. That second clause is the one that
governs day-to-day design, because the cost of strong consistency is paid on every
request, not only during rare partitions.


distributed_systems

Consistency models form a hierarchy from strongest to weakest. Linearizability, the
strongest, means every operation appears to take effect instantaneously at some point
between its invocation and its response, and all clients observe a single consistent
order consistent with real time. Sequential consistency drops the real-time constraint:
all clients see the same order, but it need not match wall-clock ordering. Causal
consistency guarantees only that causally related operations are seen in order, while
concurrent operations may be seen differently by different observers. Eventual
consistency promises only that replicas converge if updates stop, saying nothing about
what is observed meanwhile.


distributed_systems

The weaker models are not merely compromises; they buy availability and latency that
strong models cannot offer, and many applications genuinely do not need more. A social
feed, a view counter, a DNS record, a product catalogue — all tolerate stale reads.
Account balances, inventory decrements, and unique-username registration do not. The
design skill is identifying which operations in a system need which guarantee, since
applying the strictest requirement uniformly is expensive and applying the loosest
uniformly is incorrect.


distributed_systems

Consistency guarantees and typical mechanisms:
  linearizable            consensus (Raft/Paxos), single leader, quorum reads
  sequential              total order broadcast
  causal                  vector clocks, version vectors, dependency tracking
  read-your-writes        session stickiness, or read from the primary after write
  monotonic reads         pin a session to one replica
  bounded staleness       replica lag threshold with fallback to the primary
  eventual                asynchronous replication, anti-entropy, CRDTs


distributed_systems

Session guarantees are the pragmatic middle ground and are underused. Read-your-writes
means a user who posts a comment sees it immediately, even if other users do not for a
second — which is almost always what "consistency" means to a product manager.
Implementing it by routing a user's reads to the primary for a few seconds after their
write, or by passing a version token the replica waits for, delivers the perceptible
benefit of strong consistency at a small fraction of the cost.


distributed_systems

Conflict-free replicated data types sidestep coordination entirely by constraining the
data structure so that concurrent updates merge deterministically. A grow-only counter,
an add-only set, a last-writer-wins register, an observed-remove set: each defines a
merge operation that is commutative, associative, and idempotent, so replicas
converge regardless of message order or duplication. The cost is expressiveness — not
every operation can be phrased this way, and metadata can grow — but for collaborative
editing, presence, and distributed counters they eliminate coordination completely.


distributed_systems

## Consensus and Replication

Consensus is the problem of getting a set of nodes to agree on a single value despite
failures, and it is the foundation on which linearizable systems are built. The FLP
impossibility result proves that in a fully asynchronous system with even one faulty
process, no deterministic algorithm guarantees consensus — there is always a schedule
that prevents termination. Practical systems evade this with partial synchrony
assumptions and randomized timeouts: they guarantee safety always and liveness only
during periods when the network behaves, which is the right trade because an
occasional extra election is tolerable and disagreement is not.


distributed_systems

Raft is the consensus algorithm most commonly implemented because it was explicitly
designed for understandability. It elects a single leader by randomized timeout, and
the leader accepts all writes, appends them to a replicated log, and commits an entry
once a majority have persisted it. Terms provide logical time, so a stale leader
returning after a partition discovers a higher term and steps down. Safety rests on the
majority-overlap property: any two majorities of an odd-sized cluster share at least
one member, so a newly elected leader necessarily has every committed entry.


distributed_systems

Cluster sizing follows directly from majority quorums. A cluster of 2f + 1 nodes
tolerates f failures, so three nodes tolerate one and five tolerate two. Even-sized
clusters are strictly worse than the odd size below them: four nodes still need a
majority of three, tolerating only one failure while adding a machine that can fail.
Larger clusters improve fault tolerance and worsen write latency, since every commit
waits for a majority, which is why production clusters are almost always three or five
and scale is achieved by partitioning across many such groups rather than by growing
one.


distributed_systems

Replication topologies:
  single leader        all writes to one node, reads from any; simple, the default
                       for relational databases; failover is the hard part
  multi-leader         writes accepted at several nodes; needed for multi-region
                       write locality and offline clients; conflicts are inevitable
                       and must be resolved explicitly
  leaderless (Dynamo)  clients write to W and read from R of N replicas; strong
                       consistency when R + W > N; sloppy quorums and hinted
                       handoff trade consistency for availability
  chain replication    writes at the head, reads at the tail; strong consistency
                       with high throughput, poor tail latency during failures


distributed_systems

Replication lag is where correctness bugs concentrate in single-leader systems. A user
writes to the leader and immediately reads from a replica that has not yet applied the
change, and their own update appears to have vanished. The fixes are read-your-writes
routing, monotonic-read session pinning so a user never moves to a more stale replica,
and explicit lag monitoring with automatic removal of replicas beyond a threshold. The
failure is intermittent and load-dependent, which makes it hard to reproduce and easy
to dismiss as a client bug.


distributed_systems

Split-brain is the corresponding hazard in failover. A leader that is merely slow, not
dead, may be declared failed and replaced, leaving two nodes believing they are the
leader and accepting conflicting writes. Fencing tokens are the standard remedy: every
leader receives a monotonically increasing token, and the storage layer rejects any
write bearing a token lower than the highest it has seen, so the deposed leader's
writes are refused even if it never learns it was deposed. Systems that rely on a lease
timing out without a fencing mechanism are relying on clocks, which is exactly what
distributed systems cannot do.


distributed_systems

[user] Our Postgres read replicas are 4-8 seconds behind under load and users report
their profile edits disappearing on refresh.
[assistant] tool_call db.replication_status({"cluster": "prod-primary",
"replicas": 3, "metric_window": "1h"})
[tool] {"max_lag_seconds": 8.4, "mean_lag_seconds": 4.1, "wal_generation_mb_s": 42,
"replica_apply_mb_s": 38, "longest_query_on_replica_s": 61,
"conflicts": 0, "verdict": "apply_bound_plus_long_queries"}
[assistant] Two separate problems, and the user-visible one has a fast fix.


distributed_systems

The disappearing edits are a read-your-writes violation, not data loss. The user
writes to the primary, is redirected to a replica for the subsequent read, and the
replica has not applied the change yet. Nothing is lost — it appears a few seconds
later. Fix this at the routing layer rather than by chasing the lag: after a write,
pin that user's session to the primary for a window longer than your worst observed
lag, say 15 seconds. The more precise version is to capture the primary's WAL LSN at
write time, pass it back to the client, and have the replica wait until it has applied
that LSN before serving the read — Postgres exposes this well and it gives correctness
rather than a heuristic timeout.


distributed_systems

Separately, the lag itself is real and worth fixing. Your primary generates 42 MB/s of
WAL and replicas apply at 38 MB/s, so they cannot keep up and lag grows monotonically
under sustained load. Compounding it, you have a 61-second query running on a replica;
recovery conflicts with long read queries either pause apply or cancel the query
depending on your `max_standby_streaming_delay` setting, and pausing apply is what
turns a 4-second lag into an 8-second one.


distributed_systems

In priority order: move that long analytical query off the serving replicas onto a
dedicated one with a permissive delay setting, then reduce WAL volume at the source —
the usual culprits are unbatched small writes, an over-indexed hot table, and full-page
writes after checkpoints, so check `pg_stat_statements` for write amplification. If WAL
volume is genuinely required, you need to partition the write load rather than tune it.


distributed_systems

## Time, Ordering, and Idempotence

Physical clocks cannot be trusted for ordering. NTP-synchronized clocks in a datacentre
typically agree within a few milliseconds but can diverge by far more during network
disruption, and clocks jump backward when corrected — which is why comparing wall-clock
timestamps to determine which of two events happened first is unsound, and why
measuring elapsed time requires a monotonic clock rather than a wall clock. Last-writer-
wins conflict resolution based on wall-clock timestamps silently discards writes when
clocks are skewed, and the discarded write is the one from the node whose clock is
behind, which is not a property anyone intended.


distributed_systems

Logical clocks order events by causality instead. A Lamport clock is a counter
incremented on each local event and advanced to max(local, received) + 1 on message
receipt; it guarantees that if A causally precedes B then clock(A) < clock(B), but not
the converse, so it cannot distinguish causality from coincidence. Vector clocks carry
one counter per node and can distinguish causally related from concurrent events
exactly, at the cost of size linear in the number of nodes. Hybrid logical clocks
combine a physical timestamp with a logical counter, giving causally consistent
ordering that also stays close to wall-clock time, which makes them practical for
systems that need both.


distributed_systems

Google's Spanner takes the opposite approach with TrueTime: rather than avoiding
physical clocks, it bounds their uncertainty using GPS and atomic clocks, exposing an
interval rather than an instant. A transaction commits by waiting out the uncertainty
window, typically a few milliseconds, guaranteeing that its timestamp is genuinely in
the past for every observer. This buys externally consistent distributed transactions
at the cost of specialized hardware and a deliberate commit-wait latency — an explicit
purchase of consistency with latency, exactly as PACELC describes.


distributed_systems

Exactly-once delivery does not exist at the transport layer; exactly-once *processing*
is achievable and is what systems actually mean. The construction is at-least-once
delivery plus idempotent handling. Idempotence is obtained by carrying a unique key
with each request and having the receiver record processed keys, rejecting duplicates;
by making the operation naturally idempotent (set balance to X rather than add Y); or
by transactionally coupling the side effect with the record of having performed it.


distributed_systems

Messaging delivery semantics:
  at-most-once      fire and forget; fast, loses messages on failure
  at-least-once     retry until acknowledged; duplicates are guaranteed eventually
  exactly-once      at-least-once plus consumer-side idempotence or a transactional
                    write; not a transport property
  ordered           per-partition ordering is achievable; global ordering across
                    partitions is not, and demanding it eliminates parallelism
  dead-letter queue where poison messages go after N failures, so one bad message
                    cannot block a partition indefinitely


distributed_systems

The dual-write problem is the most common source of inconsistency in service
architectures: a handler writes to its database and then publishes an event, and the
process crashes between the two. The database is updated and no event exists, or the
event is published and the transaction rolls back. There is no ordering of the two
writes that fixes this. The correct solutions couple them atomically — the transactional
outbox, where the event is written to a table in the same transaction and a separate
relay publishes it, or change data capture, where the event stream is derived from the
database's replication log so it cannot diverge by construction.


distributed_systems

Q: A payment service uses at-least-once delivery and processes each message by charging
a card. To achieve exactly-once semantics it should:
A. Switch the broker to exactly-once mode
B. Deduplicate by an idempotency key recorded transactionally with the charge
C. Use at-most-once delivery instead
D. Add a longer visibility timeout
Reasoning: No broker configuration makes end-to-end exactly-once possible, because the
acknowledgement itself can be lost after the side effect. The receiver must make
processing idempotent, and the deduplication record must commit atomically with the
charge — recording it separately reintroduces the same crash window. Answer: B.


distributed_systems

Q: Two nodes' wall clocks differ by 300 ms. A last-writer-wins store resolves a
conflict between concurrent writes. The outcome is:
A. The genuinely later write is kept
B. The write from the node with the faster clock is kept, regardless of real order
C. Both writes are preserved
D. The write is rejected
Reasoning: LWW compares timestamps, so it selects whichever node reported a larger
number. With 300 ms of skew, a write that really happened later on the slow-clocked
node loses and is silently discarded. This is why LWW needs either bounded clock
uncertainty or logical clocks. Answer: B.


distributed_systems

## Partitioning, Caching, and Resilience

Partitioning splits data across nodes so that capacity scales horizontally. Range
partitioning keeps keys in sorted order, which makes range scans efficient and creates
hotspots when the access pattern is skewed toward recent keys — timestamps as the
leading component of a partition key is the classic mistake, funnelling all current
writes to one partition. Hash partitioning distributes uniformly and destroys range
locality. Consistent hashing places nodes and keys on a ring so that adding or
removing a node relocates only 1/n of keys rather than remapping everything, and
virtual nodes smooth the imbalance that arises from placing few nodes on the ring.


distributed_systems

Hot keys defeat any partitioning scheme, because a single key cannot be split by
definition. A celebrity account, a viral product, a global configuration row: all
concentrate load on one partition regardless of how well the rest is distributed. The
mitigations are read replicas for that key, key salting where the key is split into N
sub-keys that are aggregated on read (which trades write simplicity for read
complexity), or a dedicated cache layer in front of the hot subset.


distributed_systems

Caching is the highest-leverage and most bug-prone optimization in the field. The core
tension is that a cache is a second copy of data, and two copies can disagree. Cache
invalidation strategies each fail differently: TTL-only is simple and serves stale data
for up to the TTL; write-through updates the cache on write and can leave it
inconsistent if the two writes are not atomic; write-behind buffers writes and risks
loss; explicit invalidation is precise and misses cases that the code did not
anticipate. There is no strategy without a failure mode, so the design question is
which staleness is acceptable rather than how to eliminate it.


distributed_systems

Three cache failure patterns worth naming:
  cache stampede      an expired hot key causes every concurrent request to miss and
                      recompute simultaneously. Fix with a mutex or single-flight so
                      one request recomputes while others wait, or probabilistic early
                      expiry so refreshes spread out before the TTL.
  cache penetration   requests for keys that do not exist bypass the cache every time
                      and hit the database. Fix by caching negative results, or with a
                      Bloom filter over existing keys.
  cache avalanche     many keys expire at once, typically because they were populated
                      together at deploy or restart. Fix with randomized TTL jitter.


distributed_systems

Resilience patterns compose into a defensive posture. The circuit breaker tracks the
failure rate of a dependency and, above a threshold, stops issuing calls entirely for a
cooldown period, then admits a trial trickle before closing — converting a slow cascade
into fast local failures and giving the dependency room to recover. Bulkheads isolate
resources per dependency, so a thread pool exhausted by one slow downstream cannot
starve calls to healthy ones. Load shedding rejects excess requests at the edge with a
cheap error rather than accepting work that will time out anyway, which keeps latency
bounded for the requests that are served. Backpressure propagates capacity limits
upstream instead of allowing unbounded queues, since an unbounded queue converts an
overload into an unbounded latency increase and eventually an out-of-memory failure.


distributed_systems

Observability is what makes all of this operable. The three signals are complementary:
metrics are cheap aggregates good for alerting and trends but cannot explain a single
request; logs are detailed and expensive and answer questions about specific events;
distributed traces follow one request across service boundaries and are the only signal
that answers "where did the latency go" in a system of thirty services. Percentiles
matter far more than averages, because the mean is dominated by the common case while
users experience the tail — and in a request that fans out to a hundred backends, the
overall latency is governed by the slowest of the hundred, so a p99 backend latency
becomes a near-certainty at the fan-out level. Tail tolerance, not average
optimization, is what makes large systems feel fast.


frontend_engineering

## The Browser Rendering Model

A browser turns bytes into pixels through a pipeline whose stages explain nearly every
frontend performance problem. HTML is parsed into the Document Object Model, a tree of
nodes. CSS is parsed into the CSS Object Model and matched against those nodes to produce
computed styles. The two combine into a render tree of visible elements. Layout — also
called reflow — computes the geometry of every box. Paint fills in pixels. Composite
assembles the painted layers into the final image, often on the GPU.


frontend_engineering

The critical rendering path is what must complete before the user sees anything, and
optimizing it means removing work from that path. Stylesheets are render-blocking by
default, because rendering with unstyled content and then restyling would produce a flash
of unstyled text. Synchronous scripts are parser-blocking, because a script may call
document.write and alter the document structure, so the parser must stop and wait. The
async attribute lets a script download in parallel and execute whenever it arrives; defer
downloads in parallel and executes after parsing completes, in document order. For
anything that does not need to run during parsing, defer is almost always correct and is
underused.


frontend_engineering

Layout thrashing is the most common self-inflicted performance problem. The browser
batches style changes and performs layout once per frame when it can, but reading a
geometric property — offsetHeight, getBoundingClientRect, scrollTop — forces it to flush
pending changes and compute layout immediately in order to return an accurate value.
Alternating writes and reads in a loop therefore forces a synchronous layout on every
iteration, turning a linear operation into a quadratic one. The remedy is to batch all
reads first, then all writes.


frontend_engineering

Rendering pipeline stages and what triggers each:
  style        changing a class or inline style; recomputes matched rules
  layout       changing geometry — width, height, position, font-size, display;
               the most expensive stage, affects descendants and often siblings
  paint        changing appearance without geometry — color, background,
               box-shadow, border-radius
  composite    changing transform or opacity on a promoted layer; cheapest, can
               run off the main thread entirely
  key rule     animate transform and opacity, never width/height/top/left
  promotion    will-change: transform, or a 3D transform, promotes an element to
               its own layer; use sparingly, each layer costs memory


frontend_engineering

The reason transform and opacity are the animation primitives is that they can be applied
by the compositor without re-running layout or paint. Animating width forces layout on
every frame for the element and everything affected by it; animating transform: scaleX()
achieves a visually similar effect by manipulating an already-painted layer. The
difference is routinely the gap between a smooth sixty frames per second and visible
stutter.


frontend_engineering

Q: An animation changes an element's `left` property each frame and stutters. The most
effective fix is:
A. Reduce the animation duration
B. Use `transform: translateX()` instead
C. Add more keyframes
D. Increase the element's z-index
Reasoning: Changing `left` triggers layout on every frame, and layout is the most
expensive pipeline stage. `transform: translateX()` is handled by the compositor without
layout or paint, so the animation can run off the main thread. This is the single most
impactful frontend animation rule. Answer: B.


frontend_engineering

## HTML Semantics and Accessibility

Semantic HTML uses elements for their meaning rather than their default appearance, and
the payoff is that a great deal of behaviour comes free. A `button` element is focusable,
activates on both click and Enter and Space, announces itself as a button to assistive
technology, and participates in form submission. A `div` with a click handler has none of
these, and reproducing them requires a tabindex, a role, keyboard handlers, and
disabled-state management — all of which will be done incompletely.


frontend_engineering

Accessibility is not a separate concern layered on afterward; it is largely what semantic
markup delivers. Screen readers navigate by heading structure, so headings must be
hierarchical and must not skip levels for visual reasons — the size of a heading is a CSS
decision and its level is a structural one. Landmark elements let users jump directly to
navigation, main content, or search. Form inputs must have associated labels, since a
placeholder disappears on focus and is not announced reliably. Images need alternative
text that conveys their function, which for a decorative image is an empty alt attribute
rather than a description.


frontend_engineering

The first rule of ARIA is not to use ARIA: a native element with the right semantics is
better than any element with a role attribute, because native elements bring behaviour as
well as semantics. ARIA is for cases genuinely without a native equivalent — a tab
interface, a combobox with custom filtering, a live region announcing asynchronous
updates. When ARIA is used, the roles carry required states and keyboard interaction
patterns, and implementing the role without the expected keyboard behaviour produces
something that announces itself as a tablist and does not respond to arrow keys, which is
worse than no role at all.


frontend_engineering

Accessibility essentials:
  semantic elements      button, a, nav, main, header, footer, article, section
  heading hierarchy      one h1, no skipped levels; structure not size
  labels                 every input has a label, explicitly associated
  focus visibility       never remove the focus outline without replacing it
  focus management       move focus into a modal on open, restore it on close,
                         and trap it while open
  keyboard operation     everything reachable and operable without a mouse
  color contrast         4.5:1 for body text, 3:1 for large text and UI components
  motion                 respect prefers-reduced-motion
  live regions           aria-live for asynchronous updates the user must know about
  alt text               functional description, or empty for decoration
  testing                keyboard-only pass, screen reader pass, automated audit;
                         automated tools catch roughly a third of real issues


frontend_engineering

The focus outline deserves specific mention because removing it is one of the most
common and most damaging accessibility errors. `outline: none` without a replacement
makes an interface entirely unusable by keyboard, since the user cannot tell what is
focused. If the default outline is aesthetically unacceptable, replace it with a custom
visible indicator, and use `:focus-visible` so the indicator appears for keyboard
navigation without appearing on mouse clicks.


frontend_engineering

## CSS Layout and the Cascade

The cascade resolves which declaration wins when several apply. Specificity is computed
per selector as a tuple of inline styles, ID count, class and attribute and pseudo-class
count, and element and pseudo-element count, compared left to right. Origin and
importance sit above specificity, with author `!important` beating author normal, and
source order breaks ties among equally specific rules. The practical guidance follows:
keep specificity low and flat so that later rules can override earlier ones without
escalation, because specificity wars end with `!important` everywhere and a stylesheet
nobody can safely modify.


frontend_engineering

Flexbox lays out content along a single axis and is the right tool for components — a
navigation bar, a row of cards, a form field with a label and an input. The mental model
that resolves most confusion is that flexbox has a main axis, set by flex-direction, and
a cross axis perpendicular to it; `justify-content` distributes along the main axis and
`align-items` along the cross axis, and these swap meaning when the direction changes
from row to column.


frontend_engineering

Grid lays out in two dimensions simultaneously and is the right tool for page-level
structure and for any layout where rows and columns must align. Its distinguishing
capability is that the layout is defined on the container rather than on the children, so
items can be placed into named areas and rearranged without touching their markup — which
makes responsive rearrangement far cleaner than the nested-flexbox alternative.


frontend_engineering

Layout reference:
  flexbox                one-dimensional; components, alignment, distribution
  grid                   two-dimensional; page structure, aligned rows and columns
  flex: 1                shorthand for grow 1, shrink 1, basis 0%; equal division
  gap                    spacing between items; works in both flex and grid
  minmax()               grid track sizing with a floor and ceiling
  auto-fit / auto-fill   responsive column counts without media queries
  clamp(min, pref, max)  fluid typography and spacing without breakpoints
  aspect-ratio           reserves space, preventing layout shift on media
  container queries      style based on the container's size rather than the
                         viewport; what component-based design actually needed
  logical properties     inline-start/block-end rather than left/top; correct
                         under right-to-left and vertical writing modes
  cascade layers         @layer; explicit ordering that removes specificity wars


frontend_engineering

The box model is the source of persistent confusion resolved by one declaration. By
default, `width` sets the content box, so padding and border add to the rendered size,
making a 300-pixel-wide element with 20 pixels of padding actually 340 pixels wide.
Setting `box-sizing: border-box` makes `width` include padding and border, which is what
essentially everyone expects, and applying it universally in a reset is standard practice.


frontend_engineering

Margin collapsing is the second recurring surprise: adjacent vertical margins collapse to
the larger of the two rather than summing, and a parent's margin can collapse with its
first or last child's, causing the parent to appear to move. It does not occur in flex or
grid containers, or across a border, padding, or any content between them, which is why
the behaviour seems to appear and disappear unpredictably. Modern layouts using flex and
grid encounter it much less often.


frontend_engineering

[user] My page has a Cumulative Layout Shift of 0.34 and I can't find what's moving.
[assistant] tool_call web.cls_analysis({"url": "example", "cls": 0.34,
"viewport": [1280, 800]})
[tool] {"shifts": [{"element": "img.hero", "score": 0.19, "cause":
"no_dimensions"}, {"element": "div.ad-slot", "score": 0.09, "cause":
"late_injection_no_reserved_space"}, {"element": "body", "score": 0.06, "cause":
"font_swap_metric_mismatch"}], "largest_shift_time_ms": 1840}
[assistant] Three contributors, and the first two account for 82% of the score.


frontend_engineering

The hero image has no width and height attributes, so the browser cannot reserve space
before it loads and everything below jumps down when it arrives. Fix: set the intrinsic
`width` and `height` attributes on the img element — not CSS dimensions, the HTML
attributes — and the browser computes an aspect ratio and reserves the correct space
immediately. Keep your CSS `width: 100%; height: auto` for responsiveness; the attributes
and the CSS work together. This is the single highest-value fix and it is two attributes.


frontend_engineering

The ad slot is injected after load with no reserved space. Reserve it in CSS with a fixed
`min-height` matching the ad's dimensions, or `aspect-ratio` if the size is known. If the
ad size is variable, reserve the largest expected size — an empty region is far better for
the metric and for the reader than content that jumps.


frontend_engineering

The font swap is subtler. Your web font has different metrics from the fallback, so text
reflows when it loads. Two fixes: use `font-display: optional` so the browser skips the
swap if the font is not ready quickly, or — better — use the `size-adjust`,
`ascent-override`, and `descent-override` descriptors on an `@font-face` rule for the
fallback so its metrics match the web font and the swap causes no reflow at all.


frontend_engineering

Fixing the first two alone should bring you from 0.34 to about 0.06, comfortably under
the 0.1 threshold. Verify with field data rather than lab data — CLS in particular varies
a lot with real network conditions and with where users actually scroll.


frontend_engineering

## JavaScript Runtime and Asynchrony

JavaScript executes on a single thread with an event loop. Synchronous code runs to
completion; asynchronous callbacks wait in queues and run when the stack empties. There
are two queues with different priorities: the microtask queue, holding promise
continuations and queueMicrotask callbacks, is drained completely after each task and
between each callback; the macrotask queue, holding timers, I/O callbacks, and events,
yields one task per loop iteration. This is why a promise resolved immediately runs
before a setTimeout with zero delay, and why an infinite chain of microtasks starves the
page entirely while an infinite chain of timers does not.


frontend_engineering

The single-threaded model means any long synchronous computation blocks rendering, input
handling, and everything else. A task exceeding roughly 50 milliseconds is perceptible as
jank, and one exceeding a few hundred makes the page feel broken. The remedies are to
break the work into chunks yielding to the event loop between them, to move it to a Web
Worker which runs on a separate thread with message-passing communication, or to move it
off the client entirely.


frontend_engineering

Closures are the feature that most rewards precise understanding. A function retains
access to the variables of the scope in which it was defined, for as long as the function
exists. This is what makes callbacks, module patterns, and function factories work, and
it is also how memory leaks happen in long-lived applications: a closure holding a
reference to a large object keeps that object alive indefinitely, and an event listener
never removed keeps its closure and everything it captured. Detached DOM nodes still
referenced by a closure are the classic leak signature in a heap snapshot.


frontend_engineering

JavaScript concepts that matter in practice:
  event loop           call stack, microtasks drained fully, then one macrotask
  promise              a value available now, later, or never; always returns a
                       promise from .then, enabling chaining
  async/await          syntactic sugar over promises; await yields to the event
                       loop, it does not block the thread
  Promise.all          fails fast on first rejection
  Promise.allSettled   waits for all, reports each outcome; usually what is wanted
  this binding         determined by call site for regular functions, lexically
                       for arrow functions
  prototype chain      property lookup walks up until found or null
  equality             === compares without coercion; == has surprising rules
  const                binds the reference, not the value; objects remain mutable
  optional chaining    ?. and ?? for safe access and nullish defaults
  structured cloning   structuredClone() for deep copies including cycles


frontend_engineering

Error handling in asynchronous code is where correctness most often slips. An error
thrown inside a promise chain without a catch produces an unhandled rejection, which in
Node terminates the process by default and in browsers logs silently. An error thrown
inside a setTimeout callback cannot be caught by a try/catch surrounding the setTimeout
call, because the callback runs later on a different stack. And `await` inside a loop
serializes operations that could have run concurrently — the fix being to build an array
of promises and await Promise.all, though only when the operations are genuinely
independent and the target can tolerate the concurrency.


frontend_engineering

## State, Rendering, and Frameworks

Modern frameworks share a core idea: the interface is a function of state, and the
framework's job is to keep the rendered output synchronized with the state as it changes.
This is a substantial improvement over imperative DOM manipulation because it removes the
class of bug where the DOM and the application's model disagree — there is only one
source of truth and the view is derived.


frontend_engineering

The implementations differ in how they detect and apply changes. Virtual DOM approaches
re-run the component function, produce a lightweight tree, diff it against the previous
one, and apply the minimal set of DOM operations. Fine-grained reactivity tracks
dependencies at the level of individual values, so changing one value updates exactly the
DOM nodes that depend on it with no diffing at all. Compiled approaches analyze the
component at build time and generate direct update code. Each trades runtime overhead
against bundle size and mental model complexity, and for most applications the
differences are smaller than the differences in ecosystem and team familiarity.


frontend_engineering

State management is the perennial architectural question, and the useful discipline is to
categorize state before choosing a tool. Server state — data fetched from an API — has
caching, revalidation, and staleness concerns that general-purpose state containers
handle badly, and dedicated data-fetching libraries handle well. URL state — filters,
pagination, selected tab — belongs in the address bar so that pages are shareable and the
back button works. Form state has its own libraries. Ephemeral UI state — is this dropdown
open — belongs local to the component. What remains for a global store is usually far
less than teams expect, and the common failure is putting server data into a global store
and then hand-implementing caching and invalidation badly.


frontend_engineering

State categorization:
  server state     fetched, cached, revalidated; use a data-fetching library
  URL state        filters, pagination, tabs; shareable and back-button correct
  form state       inputs, validation, submission; use a form library
  local UI state   open/closed, hover, focus; component-local
  global UI state  theme, sidebar, auth session; small, a context or store
  derived state    computed from other state; never stored, always computed
  the rule         store the minimum; derive everything else


frontend_engineering

Derived state stored rather than computed is the single most common source of
inconsistency bugs. If a component stores both a list of items and its count, the two can
disagree; if it stores the list and computes the count on render, they cannot. The same
applies to filtered lists, totals, validity flags, and formatted values. The instinct to
memoize should follow a measurement, not precede it — most derived computations are far
cheaper than the bookkeeping required to cache them correctly.


frontend_engineering

## Performance and Tooling

Frontend performance is measured by what the user experiences, which is why the Core Web
Vitals are defined the way they are. Largest Contentful Paint measures loading — when the
biggest above-the-fold element renders — with a good threshold at 2.5 seconds.
Interaction to Next Paint measures responsiveness — the latency from an interaction to
the visual response — with a good threshold at 200 milliseconds. Cumulative Layout Shift
measures visual stability, with a good threshold at 0.1. Each corresponds to a distinct
user complaint: it is slow, it does not respond, it moves around while I read.


frontend_engineering

JavaScript is usually the dominant cost, and the cost is not only download but parse,
compile, and execute — the last of which is far more expensive per byte than images and
occurs on the main thread. The levers are code splitting so that a route loads only what
it needs, tree shaking so unused exports are eliminated, lazy loading below-the-fold
components, and honest auditing of dependencies. A date formatting library that adds 70
kilobytes to serve one format string is a real and common finding, and the platform's own
Intl API replaces most of that use.


frontend_engineering

Images typically dominate transferred bytes, and the fixes are well established: serve
modern formats with fallbacks, size images to the layout rather than shipping originals,
use responsive srcset so small screens receive small files, lazy-load anything below the
fold, and always set dimensions to prevent layout shift. A single unoptimized hero
photograph frequently outweighs the entire JavaScript bundle.


frontend_engineering

Performance levers by typical impact:
  reduce JavaScript      code split, tree shake, audit dependencies, lazy load
  optimize images        modern formats, correct sizing, responsive srcset,
                         lazy loading, explicit dimensions
  eliminate blocking     defer scripts, inline critical CSS, preload key resources
  cache aggressively     immutable hashed filenames with long max-age; a service
                         worker for offline and repeat visits
  reduce main-thread     break long tasks, use Web Workers, avoid layout thrash
  server-side render     content visible before JavaScript loads; streaming SSR
                         improves time to first byte further
  measure in the field   lab metrics on a fast machine systematically understate
                         real user experience; use real user monitoring


frontend_engineering

The tooling layer has consolidated around a few patterns worth knowing. A bundler
resolves the module graph and produces optimized output, with the current generation using
native-speed languages and native ES modules in development for near-instant startup. A
transpiler converts modern syntax for older targets, and the browserslist configuration
should reflect actual user data rather than a defensive guess, since targeting obsolete
browsers imposes a size penalty on everyone. Type checking catches a substantial class of
errors before runtime and pays for itself in any codebase with more than one contributor.
And the test pyramid applies as elsewhere: many fast unit tests over pure logic, fewer
integration tests over component behaviour, and a small number of end-to-end tests over
critical user journeys, with the middle layer testing what the user does rather than what
the implementation is — queries by accessible role and label rather than by CSS class,
which has the useful side effect of failing when the interface becomes inaccessible.


game_engines_and_realtime

## The Frame Loop and Time

A real-time engine is organized around a loop that must complete within a fixed budget, repeatedly,
without exception. At sixty frames per second the budget is under seventeen milliseconds for input,
simulation, animation, physics, audio, culling, rendering submission, and presentation. Exceeding
it once is a visible stutter; exceeding it regularly is an unplayable experience. This hard deadline
is the constraint from which the entire architecture of an engine follows.


game_engines_and_realtime

Time handling within the loop is more subtle than it appears. A variable timestep — advancing the
simulation by however long the last frame took — is simple and makes physics non-deterministic and
unstable at frame rate extremes. A fixed timestep advances simulation in constant increments,
running zero or several steps per frame as needed to catch up, which gives stable and reproducible
physics. The standard architecture uses a fixed timestep for simulation and interpolates rendered
positions between the two most recent states, which yields smooth visuals over deterministic
simulation.


game_engines_and_realtime

The accumulator pattern implements this: add the frame's elapsed time to an accumulator, run fixed
steps while the accumulator exceeds the step size, and render with the leftover fraction as an
interpolation factor. The failure mode to guard against is the spiral of death, where a slow frame
requires several simulation steps, which makes the next frame slower, requiring more steps. Clamping
the maximum steps per frame is the standard defence, and it means the simulation falls behind real
time rather than freezing entirely.


game_engines_and_realtime

The frame loop:
  the hard deadline    under seventeen milliseconds at sixty frames per second
  the architectural consequence  everything is designed around meeting it
  variable timestep    simple; non-deterministic physics, unstable at extremes
  fixed timestep       constant increments; stable and reproducible
  the standard hybrid  fixed simulation, interpolated rendering
  accumulator pattern  accumulate elapsed time, step while it exceeds the
                       increment
  interpolation factor  the leftover fraction, used to blend rendered state
  spiral of death      slow frames requiring more steps making frames slower
  the clamp            a maximum step count; the simulation falls behind rather
                       than freezing
  frame pacing         consistent frame times matter more than a high average;
                       irregular delivery is perceived as worse than a lower
                       steady rate
  vertical sync        presentation aligned with the display refresh; avoids
                       tearing and adds latency
  input latency        sampled as late as possible before simulation; the
                       difference is perceptible in responsive games
  the update ordering  input, simulation, animation, physics, audio, render;
                       ordering mistakes produce one-frame lags that are
                       surprisingly hard to diagnose


game_engines_and_realtime

Q: A game's physics behaves differently on a fast machine than on a slow one, with objects passing
through walls at low frame rates. The cause is:
A. A collision detection bug
B. A variable timestep — with long frames, objects move far enough in a single step to jump past
   thin geometry, so a fixed timestep with a bounded step size, plus swept collision tests, is the
   structural fix
C. The walls are too thin
D. Insufficient physics accuracy
Reasoning: Discrete collision detection samples positions at step boundaries. A large step moves an
object further than the collider's thickness, so no overlap is ever observed. Fixed small steps
bound the movement per test, and swept or continuous detection catches the remaining cases.
Answer: B.


game_engines_and_realtime

## Entity Architecture and Data Layout

The organization of game objects determines both what the code can express and how fast it runs.
Deep inheritance hierarchies were the historical approach and they fail predictably: a class tree
built around one taxonomy cannot express objects that combine features across branches, and the
tree accumulates a base class containing everything anyone ever needed.


game_engines_and_realtime

Composition replaces this with entities that own a set of components, each holding data for one
aspect — transform, mesh, collider, health, inventory. An entity is defined by which components it
has, so any combination is expressible without touching a hierarchy, and systems operate on all
entities possessing a required set of components. This is more flexible and it also happens to
enable a substantial performance improvement.


game_engines_and_realtime

That improvement comes from data layout. Storing components of the same type contiguously means a
system iterating over them reads memory sequentially, which is what caches are built for. The
alternative — objects scattered across the heap, each read pulling in a cache line mostly containing
data the loop does not use — is frequently an order of magnitude slower on the same algorithm. Since
memory latency rather than instruction count dominates most real-time workloads, arranging data for
the access pattern is usually the highest-leverage optimization available.


game_engines_and_realtime

Entity architecture:
  inheritance hierarchies  fail on combinations; accumulate a bloated base class
  composition          entities owning sets of components
  components as data   one aspect each, minimal behaviour
  systems              operate on entities having a required component set
  the flexibility gain  any combination expressible without hierarchy changes
  contiguous storage   components of a type stored together
  cache behaviour      sequential access is what hardware is optimized for
  the scattered alternative  each read pulling a mostly-unused cache line
  the magnitude        frequently an order of magnitude on identical algorithms
  memory latency dominance  the reason data layout beats algorithmic
                       micro-optimization in this domain
  structure of arrays  separate arrays per field; better when systems touch few
                       fields
  archetypes           entities grouped by their exact component set; makes
                       iteration dense
  handles over pointers  stable references that survive reallocation
  system ordering and dependencies  explicit rather than implicit; parallel
                       execution requires knowing which systems conflict
  the pragmatic caveat  a full data-oriented rewrite is rarely worth it for a
                       small project, and profiling first tells you whether the
                       bottleneck is where the architecture would help


game_engines_and_realtime

## Rendering Pipelines

The renderer turns scene state into an image, and its structure is dictated by the cost of the
operations involved. Culling removes what cannot be seen before any expensive work occurs: frustum
culling discards objects outside the view, occlusion culling discards objects hidden behind others,
and level-of-detail selection substitutes cheaper representations for distant objects. The best
optimization is always not drawing something.


game_engines_and_realtime

Draw call count is the classic bottleneck. Each submission of geometry to the graphics hardware
carries overhead, and thousands of small submissions cost more than the drawing itself. Batching
combines objects sharing material state; instancing draws many copies of one mesh with per-instance
data; indirect drawing lets the GPU determine what to draw without returning to the processor. Modern
graphics interfaces reduce per-call overhead substantially and do not eliminate the principle.


game_engines_and_realtime

Forward and deferred rendering differ in when lighting is computed. Forward rendering shades each
object as it is drawn, which handles transparency and varied materials naturally and costs
proportionally to objects times lights. Deferred rendering writes surface properties to intermediate
buffers and computes lighting once per pixel afterward, which decouples light count from object
count and complicates transparency and material variety. Tiled and clustered variants of both
partition the screen or the view volume to limit which lights affect which regions, which is the
approach most current engines take.


game_engines_and_realtime

Rendering:
  culling first        the best optimization is not drawing
  frustum culling      outside the view volume
  occlusion culling    hidden behind other geometry
  level of detail      cheaper representations at distance
  the transition problem  visible popping unless blended or geometrically smooth
  draw call overhead   per-submission cost dominating at high counts
  batching             combining objects with shared state
  instancing           many copies of one mesh with per-instance data
  indirect drawing     the GPU determining what to draw
  forward rendering    shade during drawing; natural transparency, cost scales
                       with objects times lights
  deferred rendering   surface properties then lighting; light count decoupled,
                       transparency awkward
  tiled and clustered  partitioning to limit light-region pairs; the common
                       modern approach
  shadow mapping       depth from the light's view; resolution and bias artifacts
                       are the perennial difficulties
  cascaded shadows     multiple resolutions by distance
  post-processing      tone mapping, anti-aliasing, and effects on the final image
  temporal techniques  reusing previous frames; large quality gains and
                       ghosting artifacts on fast motion that require careful
                       rejection heuristics


game_engines_and_realtime

## Physics, Animation, and Assets

Physics simulation runs in two phases. Broad phase finds potentially colliding pairs using spatial
partitioning, reducing the pair count from quadratic to near-linear. Narrow phase computes exact
contacts for the surviving pairs. Solving then resolves the contacts and constraints, typically by
iterating toward a solution rather than solving exactly, since exact solutions are too expensive and
iteration count becomes a direct accuracy-cost dial.


game_engines_and_realtime

Stability is the practical concern more than accuracy. Stacked objects jitter, constraints drift,
and fast objects tunnel through thin geometry. The mitigations are standard: a fixed timestep, a
sufficient iteration count, sleeping for objects at rest, continuous collision detection for fast
movers, and mass ratios kept within a bounded range because solvers behave badly when a very light
object is constrained against a very heavy one.


game_engines_and_realtime

Asset pipelines convert authored content into runtime formats, and their design determines the
team's iteration speed more than any runtime consideration. Source assets are large and editable;
runtime assets are compressed, platform-specific, and optimized for load. The conversion should be
incremental, cached, and parallel, because a pipeline requiring a full rebuild for a texture change
costs every artist several minutes many times a day, and that cost dwarfs almost any runtime
inefficiency it might be trading against.


game_engines_and_realtime

Physics and assets:
  broad phase          spatial partitioning to find candidate pairs
  narrow phase         exact contact computation
  the solver           iterative constraint resolution
  iteration count      a direct accuracy-cost dial
  stability over accuracy  the practical priority in games
  sleeping             resting objects removed from simulation
  continuous detection  swept tests for fast movers
  mass ratio limits    solvers degrade with extreme ratios
  skeletal animation   a hierarchy of transforms driving vertex positions
  blending and state machines  transitions between animations
  inverse kinematics   solving for joint angles from an end-effector target
  root motion          animation driving movement rather than the reverse
  source versus runtime assets  editable and large versus compressed and
                       optimized
  incremental builds   only what changed; cached and parallel
  the iteration cost   pipeline latency multiplied by team size and frequency
  hot reloading        assets and code updated without restarting; among the
                       highest-value tooling investments available
  texture compression  block formats decompressed by hardware; the format is
                       platform-specific and the quality tradeoff differs by
                       content type


game_engines_and_realtime

## Memory, Networking, and Tooling

Memory management in real-time systems avoids general-purpose allocation on the frame path, because
allocation has unpredictable cost and fragmentation accumulates over a long session. The standard
approaches are pools of fixed-size blocks for objects of known type, linear arena allocators reset
each frame for temporary data, and preallocation of everything needed at level load. The goal is a
frame that performs no allocation at all.


game_engines_and_realtime

Networking for real-time games is dominated by latency rather than by bandwidth. The
server-authoritative model prevents cheating and adds a round trip to every action, which is
unacceptable for responsive gameplay, so clients predict the outcome of their own actions
immediately and reconcile when the authoritative result arrives. Reconciliation must correct the
client without visible disruption, and entity interpolation displays other players slightly in the
past so their motion is smooth despite arriving in discrete updates.


game_engines_and_realtime

Tooling determines what the team can actually build. An in-engine editor, live parameter tweaking,
visual debugging overlays, a profiler integrated with the frame timeline, and deterministic replay
for reproducing bugs are each worth more than they appear on a schedule, because they multiply the
rate at which content and gameplay can be iterated. The engines that succeed commercially are
distinguished by their tools at least as much as by their runtime capabilities.


game_engines_and_realtime

Memory, networking, and tooling:
  no frame-path allocation  the goal for a real-time loop
  pool allocators      fixed-size blocks for known types
  arena allocators     linear, reset each frame for temporaries
  preallocation        everything reserved at load
  fragmentation        the long-session failure of general allocation
  budget tracking      per-subsystem memory budgets enforced in builds
  latency over bandwidth  the networking priority
  server authority     prevents cheating, adds a round trip
  client prediction    act immediately, reconcile on authoritative response
  reconciliation       correcting without visible disruption
  entity interpolation  others displayed slightly in the past for smoothness
  lag compensation     the server rewinding to validate a client's shot against
                       what that client saw
  delta compression    sending only changed state
  the tooling multiplier  editors, live tweaking, overlays, integrated profiling
  deterministic replay  the single most valuable debugging capability, and it
                       constrains the whole engine's design because achieving it
                       requires eliminating every source of nondeterminism
                       including floating-point variation across platforms


mobile_engineering

## Platform Models and Application Lifecycle

Mobile development differs from desktop and server development in a set of constraints that are
not negotiable: the operating system controls the application's lifetime, the resources are
constrained and shared, the network is unreliable, and the user can leave at any moment. Every
architectural decision that distinguishes good mobile code from bad follows from taking those
constraints seriously rather than treating them as edge cases.


mobile_engineering

The lifecycle is the first of these. An application is not a process that runs until it exits; it
is a set of components the system creates, suspends, and destroys according to its own resource
pressure. Anything held only in memory can vanish, and the correct mental model is that the process
may be terminated at any point when it is not in the foreground and must be able to restore its
state from persisted data on the next launch. Developers who test only by navigating within a
running app never encounter the state loss that users encounter routinely.


mobile_engineering

Configuration changes make this concrete on one platform and background termination on the other,
but the discipline is the same: identify what state must survive, persist it deliberately, and
restore it on entry. State that is expensive to recompute belongs in a layer that outlives the UI
component; state that is cheap should be recomputed rather than saved. The common failure is
holding everything in the view layer and discovering the loss only under memory pressure on a
device the developer does not own.


mobile_engineering

Platform and lifecycle:
  the system owns lifetime  the application does not decide when it stops
  foreground priority  the system reclaims background processes first
  process death        must be survivable; restore from persisted state
  saved state          small, serializable, restored on recreation
  the retained layer   state outliving UI components; a view model or equivalent
  configuration changes  rotation, theme, locale, window size; recreate cleanly
  cold, warm, hot start  three distinct startup paths with different budgets
  startup budget       the metric users feel most; measure all three separately
  permissions          requested at use, revocable at any time, and the revoked
                       path must work
  background execution  heavily restricted; scheduled work is deferred and
                       batched by the system
  the testing gap      developers test in the foreground on fast devices and
                       users are neither


mobile_engineering

Q: An app loses the user's in-progress form when they take a phone call and return ten minutes
later. The root cause is:
A. A bug in the operating system
B. The state lived only in memory in a UI component, and the system reclaimed the backgrounded
   process; state that must survive has to be persisted explicitly
C. The phone has insufficient memory
D. The form is too large
Reasoning: Background process termination under memory pressure is normal, documented behaviour on
every mobile platform, not a failure. The application is responsible for persisting state it needs
to survive and restoring it on recreation. Answer: B.


mobile_engineering

## UI Architecture and State Management

Mobile user interfaces are event-driven and asynchronous, and the architectural problem is keeping
the displayed state consistent with the underlying data while both change from multiple sources.
The pattern that has converged across platforms is unidirectional: a single source of truth holds
the state, the UI is a function of that state, and user events produce updates to it rather than
mutating the UI directly.


mobile_engineering

Declarative UI frameworks make this natural by construction, since the UI is described rather than
imperatively mutated, and the framework computes the difference. The cost is that the description
runs frequently, so it must be cheap and free of side effects. The recurring performance mistake in
declarative UI is doing work — allocation, formatting, sorting, database access — inside the
description rather than deriving it once and passing it in.


mobile_engineering

Separating layers is the other structural discipline. A view layer that only renders and emits
events, a presentation layer holding state and coordinating, and a data layer owning persistence and
network access is the standard decomposition. Its practical value is testability: the presentation
layer can be tested without a device, and the data layer can be tested without a UI, which converts
most of the logic from expensive instrumented tests into fast unit tests.


mobile_engineering

UI architecture:
  unidirectional flow  state down, events up
  single source of truth  one owner per piece of state
  UI as a function of state  render, do not mutate
  declarative frameworks  describe the result, the framework diffs
  the recomposition cost  the description runs often; keep it cheap and pure
  derived state        compute once, pass down; not inside the render path
  view layer           renders and emits; no logic
  presentation layer   holds state, coordinates; testable without a device
  data layer           persistence and network; testable without a UI
  the testability payoff  most logic moves from instrumented to unit tests
  navigation           a stack that must survive process death; encode it in
                       serializable arguments rather than object references
  deep links           entry at an arbitrary point; the back stack must be
                       synthesized sensibly
  accessibility        labels, focus order, dynamic type, contrast; retrofitting
                       is far more expensive than building it in


mobile_engineering

## Networking, Offline, and Synchronization

The mobile network is intermittent, slow, metered, and high-latency, and an application designed
for a reliable connection fails constantly in real use. The correct default assumption is that any
request may hang, fail, or succeed after the user has moved on, and that the application must
remain usable when none of them succeed.


mobile_engineering

Offline-first architecture inverts the usual data flow. The local database is the source of truth
for the UI, which reads from it and is never blocked on the network. Network operations update the
local store, and user actions are recorded locally and queued for transmission. The user experience
becomes immediate and connection-independent, and the cost is that synchronization and conflict
resolution become explicit problems that must be designed rather than avoided.


mobile_engineering

Conflict resolution requires a policy chosen deliberately. Last-write-wins is simple and silently
discards data. Per-field merging preserves more and requires field-level timestamps. Operation-based
synchronization transmits intents rather than states and merges naturally for many data types.
Server-authoritative resolution is simplest to reason about and requires the client to accept
rejection gracefully, which means the UI must be able to undo an optimistic update without
confusing the user.


mobile_engineering

Networking and sync:
  assume unreliable    hangs, failures, and late successes are the normal case
  timeouts             on every request; a request without one can hang forever
  retry with backoff   exponential, jittered, and bounded
  idempotency          retries must be safe; use a client-generated request key
  request cancellation  tied to the UI lifecycle; a completed request for a
                       discarded screen is wasted work
  offline-first        the local store is the source of truth for the UI
  optimistic update    apply locally, reconcile later, and be able to roll back
  outbox pattern       queue local mutations for transmission
  conflict policy      chosen deliberately; last-write-wins discards data silently
  field-level merge    preserves more, needs per-field metadata
  operation-based sync  transmit intent; merges naturally for many structures
  server authority     simplest to reason about; the client must handle rejection
  payload size         metered connections make this a user cost, not just a
                       latency cost
  image handling       size the request to the display; downloading full
                       resolution for a thumbnail is the single most common
                       bandwidth waste in mobile applications


mobile_engineering

## Performance, Memory, and Battery

Performance on mobile means frame rate first. A dropped frame is visible, and the budget for all
work on the UI thread between frames is a few milliseconds. Anything that can block — file access,
database queries, decoding, network, large allocations — must be off that thread, and the profiler
that shows the frame timeline is the tool that finds the violations rather than reasoning about
which operations are slow.


mobile_engineering

Memory pressure manifests as termination rather than as slowdown. The system kills the largest
background consumers first, so an application holding large caches is preferentially destroyed and
its users experience it as constant cold starts. Bitmaps dominate memory in most applications, and
sizing them to the display rather than to the source is usually the single largest available
reduction. Leaks in mobile code most often come from holding a reference to a UI context in
something that outlives it, which retains the entire view hierarchy.


mobile_engineering

Battery is the constraint users notice most and developers measure least. The dominant consumers
are the radio, the screen, and the GPS, and the radio's behaviour is the least intuitive: it stays
in a high-power state for seconds after a transmission, so many small requests spread over time
cost far more than the same bytes sent together. Batching network activity, aligning it with
system-scheduled wake windows, and never polling on a timer are the interventions that matter.


mobile_engineering

Performance and power:
  frame budget         a few milliseconds of UI-thread work per frame
  the main thread rule  no blocking work; profile rather than guess
  jank                 dropped frames; visible and measurable
  startup profiling    cold start is dominated by initialization ordering; defer
                       what is not needed for the first frame
  memory pressure      manifests as termination, not slowdown
  bitmap sizing        to the display, not the source; usually the largest win
  leaks                UI context references held past the component's life
  the radio tail       high power for seconds after transmission
  batching             many small requests cost far more than one combined
  wake alignment       use the system scheduler rather than timers
  never poll           push or scheduled sync; polling is a battery bug
  location cost        the highest-precision mode is dramatically more expensive;
                       request the accuracy actually needed
  wake locks           held longer than intended is a classic drain; every
                       acquisition needs a guaranteed release path
  measurement          on a mid-range device with a real battery, not a
                       flagship on a charger


mobile_engineering

## Release, Diagnostics, and Fragmentation

Mobile release differs from server release in that the deployment is not under your control. Users
update when they choose, review processes introduce delay, and a bad build cannot be rolled back —
it can only be superseded by a newer one that also takes time to propagate. The consequences are
architectural: feature flags with server-side control are how a bad feature is disabled without a
release, and staged rollouts are how a defect is caught before it reaches everyone.


mobile_engineering

Because many versions run simultaneously, backward compatibility with the server is a permanent
requirement rather than a migration concern. The server must support the oldest version still in
meaningful use, which means API changes are additive and removals happen long after the client
change. Forced upgrades exist as a last resort and are user-hostile enough that they should be
reserved for security issues.


mobile_engineering

Diagnostics carry the burden that reproduction cannot. The devices, operating system versions,
locales, screen sizes, and hardware capabilities in the field vastly exceed what can be tested, so
crash reporting with symbolication, structured logging, and metrics that segment by device class
and version are how problems are found. A crash rate that is acceptable in aggregate frequently
hides a specific device or version where the application is unusable, and only segmented reporting
surfaces that.


mobile_engineering

Release and diagnostics:
  you do not control deployment  users update when they choose
  no rollback          only a newer build, with its own propagation delay
  feature flags        server-controlled disable without a release
  staged rollout       a percentage first, monitored, then widened
  kill switch          for anything risky; the cheapest insurance in mobile
  version fragmentation  many versions in use simultaneously
  API compatibility    additive changes; removals long after
  forced upgrade       a last resort, for security
  crash reporting      symbolicated, with breadcrumbs leading to the crash
  segmented metrics    by device class, OS version, locale, and connection type
  the hidden failure   an acceptable aggregate rate hiding a device where the app
                       is unusable
  device fragmentation  screen sizes, memory, CPU, GPU, camera capability
  testing matrix       cannot be exhaustive; pick by installed base and by
                       the extremes of the capability range
  the low-end device   the most valuable single test device you can own, because
                       everything that is slow, memory-constrained, or
                       thermally throttled shows up there first and nowhere else


networking

## Layers and Encapsulation

Network protocols are organized in layers, each providing a service to the layer above and
using the one below. The value of the abstraction is that a change at one layer does not
require changes at others: an application works identically over Ethernet, Wi-Fi, or cellular
because the layers beneath present the same interface.


networking

The OSI model defines seven layers and is primarily a teaching and vocabulary tool; the
TCP/IP model with four or five layers describes what is actually deployed. The layers that
matter in practice are the link layer moving frames between directly connected devices, the
network layer routing packets across networks, the transport layer providing end-to-end
communication between processes, and the application layer implementing the specific
protocol.


networking

Encapsulation is the mechanism. Application data is wrapped in a transport header, that unit
in a network header, that in a link header and trailer, and each layer at the receiver strips
its own header and passes the payload up. The practical consequence is overhead: every layer
adds bytes, so small payloads are dominated by headers. A single-byte payload over TCP and IP
over Ethernet carries roughly 58 bytes of header, which is why chatty protocols sending many
tiny messages perform far worse than their data volume suggests.


networking

Layer responsibilities:
  physical        bits on the medium; voltages, modulation, connectors
  link            frames between directly connected nodes; MAC addressing,
                  error detection, media access
  network         packets across networks; IP addressing, routing, fragmentation
  transport       process-to-process; ports, reliability, ordering, flow control
  application     the specific protocol; HTTP, DNS, SMTP, SSH
  encapsulation   each layer prepends its header; ~58 bytes minimum for
                  TCP/IP/Ethernet
  MTU             maximum transmission unit; 1500 bytes on standard Ethernet
  MSS             maximum segment size; MTU minus IP and TCP headers
  path MTU        the smallest MTU along a route; discovery is broken when
                  intermediate devices block the relevant ICMP messages


networking

Path MTU discovery failure deserves specific mention because it produces one of the most
confusing failure modes in networking. A host sends packets sized for its local MTU with the
don't-fragment flag set; a router with a smaller MTU drops the packet and returns an ICMP
message indicating the correct size. If a firewall blocks that ICMP message — a common
misconfiguration justified as security hardening — the sender never learns and retransmits
the same oversized packet indefinitely. The symptom is a connection that establishes
successfully and then hangs as soon as a large transfer begins, because small packets pass
and large ones vanish silently.


networking

Q: A TCP connection completes its handshake and small requests succeed, but any response
larger than about 1400 bytes hangs. The most likely cause is:
A. The server is overloaded
B. Path MTU discovery is broken because ICMP is being filtered
C. DNS is misconfigured
D. The TCP window is too small
Reasoning: Handshake packets and small requests are below any plausible MTU and pass. Larger
packets exceed an intermediate link's MTU and are dropped, and the ICMP fragmentation-needed
message that would inform the sender is blocked, so the sender retransmits the same oversized
packets forever. This is the classic black-hole MTU problem. Answer: B.


networking

## Addressing and Routing

An IP address identifies an interface on a network. IPv4 uses 32 bits, giving about 4.3
billion addresses, which was exhausted years ago and is managed through network address
translation and increasingly fine-grained allocation. IPv6 uses 128 bits, giving a space large
enough that address scarcity is not a design consideration, and its adoption has been slow
because NAT worked well enough to defer the pain.


networking

Subnetting divides an address into a network portion and a host portion, with the split
specified by a prefix length in CIDR notation. A /24 gives 8 host bits, so 256 addresses of
which 254 are usable after the network and broadcast addresses. Understanding this arithmetic
is necessary for reading routing tables and firewall rules, and the common error is
forgetting that the network and broadcast addresses are reserved.


networking

Routing determines the path packets take. Within an organization, interior gateway protocols
distribute reachability information; between organizations, the Border Gateway Protocol
exchanges routes and is what makes the internet a network of networks. BGP's design reflects
its origins in a smaller, higher-trust internet: it accepts announcements largely on trust,
which is why route hijacks — an operator announcing address space they do not control,
whether by error or deliberately — can redirect substantial traffic, and why cryptographic
route origin validation has been an ongoing deployment effort for years.


networking

Addressing and routing:
  IPv4 / IPv6         32 bits / 128 bits
  CIDR                address plus prefix length; /24 has 254 usable hosts
  private ranges      10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16
  NAT                 many private addresses behind one public; breaks inbound
                      connections and end-to-end addressing
  ARP / NDP           resolve IP to link-layer address on the local segment
  default gateway     where packets go when the destination is off-network
  longest prefix match  the most specific matching route wins
  static vs dynamic   manual routes versus protocol-distributed
  BGP                 inter-domain routing; policy-driven, trust-based
  anycast             the same address announced from many locations; the network
                      routes to the nearest; used by DNS roots and CDNs
  VLAN                logical segmentation at the link layer
  MTU differences     tunnels and VPNs reduce the effective MTU


networking

Anycast deserves note because it does a great deal of work invisibly. The same IP address is
announced from many geographically distributed locations, and normal routing delivers each
client to whichever announcement is closest in routing terms. This provides low latency, load
distribution, and resilience without any application awareness, and it is how the DNS root
servers, public resolvers, and content delivery networks achieve global presence behind a
single address. Its limitation is that routing changes can move a client between instances
mid-connection, which is fine for stateless UDP request-response and problematic for long-lived
stateful connections.


networking

## Transport Protocols

TCP provides a reliable, ordered byte stream over an unreliable packet network, and nearly
everything about it follows from that goal. The three-way handshake establishes sequence
numbers and confirms bidirectional reachability before data flows, costing one round trip
before any application data can be sent. Sequence numbers and acknowledgements provide
ordering and detect loss; retransmission timers and fast retransmit on duplicate
acknowledgements recover it.


networking

Flow control prevents a fast sender from overwhelming a slow receiver, using a window the
receiver advertises. Congestion control prevents senders from overwhelming the network, and
it is inferred rather than signalled: TCP treats packet loss as evidence of congestion and
reduces its sending rate. This inference is what makes the internet stable under load and
what makes TCP perform poorly on links where loss is caused by radio interference rather than
by congestion — a lossy wireless link causes TCP to reduce its rate when the correct response
would be to retransmit and continue.


networking

Congestion control algorithms have evolved substantially. Classical loss-based algorithms
increase the window gradually and halve it on loss, producing the characteristic sawtooth and
requiring loss to discover capacity. This interacts badly with deep buffers in network
equipment: the buffers absorb the excess, so loss occurs only after latency has grown
enormously — bufferbloat, in which a saturated link produces seconds of delay. Rate-based
algorithms that model bandwidth and round-trip time directly avoid filling buffers and
maintain much lower latency at the same throughput.


networking

Transport characteristics:
  TCP                 reliable, ordered, connection-oriented, congestion-controlled
  UDP                 unreliable, unordered, connectionless, minimal overhead
  three-way handshake SYN, SYN-ACK, ACK; one round trip before data
  head-of-line blocking  a lost segment stalls delivery of everything behind it
                      even if it arrived
  Nagle's algorithm   coalesces small writes; interacts badly with delayed ACK,
                      producing latency spikes for request-response traffic
  TCP_NODELAY         disables Nagle; usually correct for interactive protocols
  slow start          exponential window growth from a small initial value
  congestion avoidance  linear growth after the first loss
  bufferbloat         excessive buffering converts congestion into latency rather
                      than loss
  QUIC                reliable streams over UDP; per-stream ordering avoids
                      head-of-line blocking, integrates encryption, and survives
                      network changes via connection IDs
  connection migration  QUIC connections survive an IP address change; TCP
                      connections do not


networking

Head-of-line blocking is the specific defect that motivated QUIC. TCP delivers a single
ordered byte stream, so if one segment is lost, everything received after it waits in the
buffer until the retransmission arrives — even data belonging to a completely independent
request multiplexed over the same connection. HTTP/2 multiplexed many streams over one TCP
connection and therefore made this worse rather than better under loss. QUIC implements
independent streams over UDP with per-stream ordering, so a loss affecting one stream does
not stall the others.


networking

## Names, TLS, and HTTP

DNS translates names to addresses through a hierarchical distributed database. A resolution
walks from the root through the top-level domain to the authoritative server for the zone,
with caching at every level governed by each record's time to live. That caching is what makes
the system scale and is also what makes changes propagate slowly: lowering the TTL well before
a planned change is the standard preparation, because the old value persists in caches for up
to its previous TTL after the change.


networking

DNS failures are disproportionately represented in outages for structural reasons. It is on
the critical path of nearly every request, its failures are often partial and
resolver-dependent, negative results are cached too, and the propagation delay makes
recovery slow. Adding a hostname that resolves for some users and not others is a very
common cause of confusing incident reports.


networking

TLS provides confidentiality, integrity, and authentication. The handshake negotiates
parameters, authenticates the server through a certificate chained to a trusted root, and
establishes session keys. TLS 1.3 removed the legacy options that caused most historical
vulnerabilities, reduced the handshake to one round trip, and made forward secrecy mandatory —
so that compromise of the server's long-term key does not permit decryption of previously
recorded sessions.


networking

Names, security, and HTTP:
  DNS record types     A, AAAA, CNAME, MX, TXT, NS, SRV, CAA
  TTL                  cache lifetime; lower it before planned changes
  negative caching     failures are cached too
  DNSSEC               cryptographic authentication of records; deployment is
                       partial
  DoH / DoT            encrypted DNS transport; privacy from the local network
  TLS handshake        1-RTT in 1.3; 0-RTT resumption with replay caveats
  certificate chain    leaf to intermediate to trusted root; missing intermediates
                       are a common misconfiguration that works in browsers with
                       cached intermediates and fails elsewhere
  SNI                  server name in the handshake; enables virtual hosting
  forward secrecy      ephemeral keys; past sessions stay safe if the key leaks
  HSTS                 forces HTTPS for a domain; prevents downgrade
  HTTP/1.1             one request at a time per connection; parallelism by
                       opening many connections
  HTTP/2               multiplexed streams, header compression, server push; one
                       TCP connection, subject to head-of-line blocking
  HTTP/3               HTTP over QUIC; independent streams, faster establishment,
                       connection migration


networking

The missing intermediate certificate is worth calling out as a recurring operational trap. A
server must present its leaf certificate and any intermediates; only the root is expected to
be in the client's trust store. Browsers often cache intermediates from previous sites, so a
server missing its intermediate works in a browser test and fails for command-line clients,
mobile applications, and other servers. Testing with a tool that does not cache intermediates
is the way to catch it before deployment.


networking

## Performance, Delivery, and Troubleshooting

Network performance is governed by three quantities that are frequently conflated. Bandwidth
is capacity, latency is delay, and the bandwidth-delay product is how much data must be in
flight to keep a link fully utilized. Increasing bandwidth does not reduce latency, which is
bounded below by the speed of light in fibre at roughly two-thirds of c — about 5 microseconds
per kilometre, giving a floor of roughly 60 milliseconds round trip for a transcontinental
path regardless of capacity.


networking

That floor is why round trips dominate perceived performance for small transfers and why
protocol design has focused so heavily on reducing them. A connection requiring a DNS lookup,
a TCP handshake, and a TLS handshake spends three round trips before the first byte of
application data, which on a 100 millisecond path is 300 milliseconds of pure protocol
overhead. Connection reuse, TLS session resumption, and QUIC's combined handshake each remove
round trips, and the cumulative effect on a page loading many resources is large.


networking

Content delivery networks address latency by moving content closer to users, which is the
only way to beat the speed-of-light floor. They also absorb traffic spikes, terminate TLS at
the edge to shorten handshake round trips, and provide denial-of-service mitigation by
distributing the attack surface. Their effectiveness depends on cache hit rate, which depends
on cache key design, on how aggressively cacheable responses are marked, and on whether
personalization forces per-user responses that cannot be shared.


networking

Performance and delivery:
  bandwidth-delay product  bytes in flight needed to saturate a link
  latency floor            ~5 us/km in fibre; unavoidable
  round trips              dominate small transfers; each protocol layer adds them
  connection reuse         keep-alive; avoids repeated handshakes
  TLS resumption           avoids the full handshake on reconnect
  CDN                      geographic proximity, TLS termination, spike absorption
  cache key                what distinguishes cacheable variants; too many
                           dimensions destroys the hit rate
  edge compute             run logic at the edge to keep responses cacheable
  load balancing L4        by IP and port; fast, protocol-agnostic
  load balancing L7        by path, header, or cookie; enables routing rules and
                           request-level retries
  health checks            must exercise the real dependency path
  connection draining      finish in-flight requests before removing a backend


networking

Troubleshooting benefits from working the layers in order rather than guessing. Confirm
physical and link connectivity, then address configuration, then reachability to the gateway,
then name resolution, then transport connectivity to the port, then the application protocol.
Each step has a standard tool: ping tests reachability and is frequently blocked by policy so
its failure is not conclusive; traceroute shows the path with the caveat that intermediate
hops may deprioritize the probe packets so their latency is not meaningful; a DNS query tool
separates resolution from connectivity; a port connection test separates transport from
application; and a packet capture answers what actually crossed the wire when everything else
disagrees.


networking

The single most useful troubleshooting discipline is to determine which side is failing
before investigating either. A capture at both ends of a failing connection immediately
distinguishes packets that were never sent, packets sent and not received, and packets
received and mishandled — three completely different problems that produce identical symptoms
in application logs. Without that, teams routinely spend days investigating an application
that is behaving correctly while the packets are being dropped by a firewall nobody
remembered was there.


observability_and_reliability

Observability and Reliability Engineering

## Signals and Instrumentation

Observability is the property that a system's internal state can be inferred from its outputs, and
it is distinguished from monitoring by the questions it supports. Monitoring answers questions
prepared in advance — is the error rate above threshold, is the queue growing. Observability answers
questions not anticipated, which requires that the emitted data carry enough dimensionality to slice
along attributes nobody thought to create a dashboard for.


observability_and_reliability

The three primary signal types have different costs and different uses. Metrics are numeric
aggregates over time, cheap to store and query at any retention, and limited by cardinality — every
distinct combination of label values is a separate series, so a label containing user identifiers
will destroy the metric system. Logs are discrete records with arbitrary content, expensive at
volume, and irreplaceable for the specific detail of a single event. Traces record the path of a
request across services with timing at each step, and they are what answers the question of where the
latency actually went.