Observability is the property of a system that lets an engineer answer new questions about its behaviour from the outside, without shipping code to answer each one. Monitoring tells you whether the things somebody already thought to check are healthy. Observability is what you need when production fails in a way nobody predicted and the only evidence available is whatever the system was already emitting. The distinction is not marketing — it changes what you instrument, how much detail you keep, and how you query it.
The practice rests on three signals. Metrics are cheap aggregated numbers over time: good for trends, thresholds and capacity, weak at explaining one failed request. Logs are discrete events carrying detail, expensive at volume, and only useful under pressure when they are structured and queryable rather than free-text. Traces follow a single request across every service it touches and are the only signal that answers where the latency actually went in a distributed call graph. None of the three is sufficient alone. The value comes from correlation — a trace identifier carried into log lines, an exemplar linking a latency histogram bucket to the trace that produced it, and consistent resource attributes so that a service name means the same thing in all three.
Above the signals sits the layer that decides whether any of it pays for itself: service level indicators and objectives that define what working means numerically, error budgets that turn reliability into a spending decision rather than an argument, alerts fired on user-visible symptoms instead of on every internal cause, and a deliberate cost model — because metric cardinality, log retention and span sampling are the three levers that make a telemetry bill grow faster than the traffic it describes.
Why this skill matters now
Systems became harder to reason about faster than teams grew. A monolith on a known set of hosts could be understood from a load graph and a log file. A request that crosses a dozen services, a queue, a managed database and two third-party APIs cannot — and the failure modes that matter are increasingly partial: a degraded dependency, one bad pod, a single slow tenant, none of which trip a host-level threshold.
That is why observability is now a named engineering function rather than a side effect of owning monitoring tools. Organisations hire for it because the expensive part is not installing a collector. It is deciding what to instrument, agreeing objectives with the people who own the revenue, keeping instrumentation consistent across teams that deploy independently, and stopping telemetry spend from compounding without anyone noticing.
The skill is also portable in a way product knowledge is not. Vendors change, and most large estates run more than one backend anyway. Engineers who understand signals, cardinality, sampling and error budgets move between platforms without relearning the job; engineers who only know one query language start again every time procurement changes its mind.