Consul is a service networking platform: a distributed service registry with health checking, a replicated key/value store, and a service mesh built on top of both. Its architecture is agent-based. Every node runs a Consul agent in either client or server mode; servers form a Raft-replicated cluster of three or five per datacenter and hold the authoritative state, while clients hold no state, forward RPCs to the servers, and run the health checks for services registered locally. Membership and failure detection ride on a gossip protocol over separate LAN and WAN pools, which is how Consul notices a dead node without polling it.
Discovery is deliberately available two ways. The HTTP API suits applications written to use it, and the built-in DNS interface answers queries such as web.service.consul, which means services that cannot be modified at all can still resolve dynamically. Health checks come in several forms — HTTP, TCP, script, TTL, gRPC, alias and Docker — and they gate the registry directly, so a failing instance simply stops being returned. The key/value store, combined with sessions, provides distributed locking and leader election, and consul-template renders configuration files from registry and KV data and reloads the process that consumes them.
The mesh layer adds sidecar proxies, usually Envoy, and expresses authorization as intentions between service identities rather than as firewall rules between addresses. A built-in certificate authority — or Vault, or ACM Private CA — issues the certificates for mutual TLS. Configuration entries such as proxy-defaults, service-resolver, service-splitter and service-router provide L7 routing, retries and traffic shifting, and ingress, terminating and mesh gateways carry traffic into, out of and between meshes federated over WAN or connected by cluster peering.
Why this skill matters now
Service discovery stopped being optional the moment infrastructure became elastic. Once instances are created and destroyed automatically, static configuration files and hand-maintained load balancer pools describe a topology that no longer exists, and the resulting stale entries are a recurring source of outages.
Consul's particular value is that it does not assume a single platform. Most discovery and mesh tooling assumes Kubernetes; a large number of organisations run Kubernetes alongside virtual machines, bare metal and managed services that will never be containers. Consul registers all of them in one catalogue, resolves them over the same DNS interface, and can extend mesh identity and mutual TLS across that boundary. For organisations mid-migration — which is most of them — that is the deciding feature.
The skill demand reflects that breadth. Teams need the discovery and health-check layer working correctly first, then the KV and templating layer to remove configuration sprawl, and only then the mesh. Each layer has its own failure modes: gossip flapping on constrained networks, ACL bootstrap and token design, certificate rotation, and multi-datacenter topology decisions that are difficult to reverse.