Production systems flirt continuously with partial failure: packets drop, replicas stall in long GC pauses, routing blackholes subnets for seconds. Designs that assume the world is “up or nuked” give way to brittle retries and cascading timeouts.
Mature services model timeouts with jitter, bounded retries, hedging sparingly, and circuit breaking upstream so localized pain does not avalanche into fleet-wide outages.
Clocks and ordering myths
Wall clocks lie—skew, leap smearing, VMs freezing, flaky NTP bursts. Date.now() across hosts is not a global total order for correctness. Serious systems rely on version vectors, logical clocks, hybrid timestamps with uncertainty bounds, or single-writer shards—whatever matches the fidelity your invariants demand.
Interview tip: articulate why you need ordering before quoting buzzwords (“vector clocks”). Money and inventory disagree on casual approaches.
Consensus when it buys you coherence
Families like Raft underpin stores such as etcd, giving strongly consistent primitives (leader-backed writes, linearizable reads when configured)—at the cost of availability during partitions versus eventually consistent stores tuned for uptime.
Use consensus where metadata or coordination really needs it—not as a prestige install on every workload.
“Exactly-once” and idempotent money paths
Brokers rarely deliver truly exactly-once across producers, consumers, and side effects—you almost always compose idempotency keys, dedupe windows, inbox patterns, outbox relays, deduplicated settlements.
Double-submits happen from humans, mobiles, flaky intermediaries—not rare edge bugs.
Operational vocabulary that matters
- Bulkheads isolate blast radius across thread pools or connection pools.
- Admission control sheds load consciously instead of delaying everything equally.
- Tracing + correlation IDs traverse async hops so timelines join across logs.
Returning to fundamentals when pressure mounts is exactly what distinguishes engineers who tame distributed messes from those rewriting history after outages.