Fluentd is an open-source data collector that sits between the things producing logs and the systems that store them. It is a CNCF graduated project, and it describes itself as a unified logging layer: rather than every application learning to write to every backend, applications and hosts emit to Fluentd, which parses, filters, transforms and routes to wherever the data needs to go. Adding a destination becomes a configuration change instead of a redeployment of every service.
The internal model is small and worth learning precisely. Every event has three parts — a tag, a timestamp and a record — and the tag is what drives routing. Configuration is a set of directives: source blocks define inputs, filter blocks transform matching events, match blocks send them to outputs, and label blocks group processing into named paths so a complex configuration stays readable instead of becoming a chain of re-tagging tricks. Everything beyond the core is a plugin, of which there are over a thousand, and events move internally in MessagePack rather than as text.
The part that decides whether a pipeline is dependable is buffering. Output plugins write into memory or file buffers organised as chunks, flushed on interval or size, retried with exponential backoff, and optionally diverted to a secondary output when retries are exhausted. Those settings — chunk limits, queue length, flush thread count, retry policy — determine what happens when a destination is slow or down, and they are the difference between a log pipeline that absorbs a backend outage and one that quietly drops a morning's data. Fluent Bit, the lighter sibling written in C, is commonly deployed as the per-node collector with Fluentd as the aggregation tier behind it, which is the shape most Kubernetes logging architectures take.
Why this skill matters now
Log destinations have multiplied. The same event stream now feeds a search platform for engineers, an object store for cheap long retention, a security information system for the compliance team, and sometimes a data warehouse for analytics. Writing that fan-out into every application is unworkable, and a dedicated collection and routing layer is now standard architecture rather than an optimisation.
Fluentd is the vendor-neutral option in that layer, and its plugin catalogue is the reason it stays chosen — whatever the source and whatever the destination, something already exists to connect them. In Kubernetes it is close to ubiquitous, usually paired with Fluent Bit as the node agent, and it is the piece that adds pod, namespace and label metadata to container logs that would otherwise be anonymous lines in a file.
The skill organisations need is reliability engineering, not configuration syntax. A minimal Fluentd configuration takes ten minutes. Sizing buffers so a two-hour backend outage does not lose data, understanding what at-least-once delivery means for duplicate records downstream, parsing multiline stack traces without merging two unrelated exceptions, keeping CPU in check when a single regex meets a hundred thousand events per second, and reducing volume before it reaches a per-gigabyte-priced destination — that is what a production log pipeline actually demands.