From Zero to Production: Architecture, Traces, Metrics, Logs, OTLP, Collector, Kubernetes, Sampling, Security, Scaling, Troubleshooting and Best Practices

Technology: OpenTelemetry / OTel
Level: Beginner → Intermediate → Advanced → Production
Last verified: September 15, 2026

1. What Is OpenTelemetry?

OpenTelemetry, usually abbreviated OTel, is an open-source, vendor-neutral observability framework used to:

instrument applications
generate telemetry
collect telemetry
process telemetry
correlate telemetry
export telemetry

The principal telemetry signals are:

Traces
Metrics
Logs
Baggage

Profiles are also being developed as an OpenTelemetry signal. In the current OTLP specification, traces, metrics, and logs are stable while profiles remain under development.

OpenTelemetry is a CNCF project created from the merger of OpenTracing and OpenCensus. It provides a common standard so applications do not need proprietary instrumentation for every monitoring vendor.

A useful one-line definition is:

OpenTelemetry is the standard instrumentation and telemetry pipeline layer between your applications/infrastructure and your observability backends.

2. What OpenTelemetry Is NOT

One of the most important things to understand is that OpenTelemetry is generally not your observability database or dashboard system.

OpenTelemetry does not replace tools such as:

Grafana
Prometheus
Jaeger
Tempo
Loki
Datadog
New Relic
Dynatrace
Elastic
Splunk

Instead, OpenTelemetry sends telemetry to systems like these.

The OpenTelemetry project intentionally leaves backend storage and visualization to other systems.

Conceptually:

Application
     |
     | Telemetry
     v
OpenTelemetry
     |
     | OTLP
     v
Observability Backend
     |
     v
Storage + Queries + Dashboards + Alerts

3. Why OpenTelemetry Exists

Before OpenTelemetry, a typical organization might install:

Datadog Agent
New Relic Agent
Jaeger Client
Prometheus Client
Zipkin Client
Fluent Bit
Vendor A SDK
Vendor B SDK
Code language: PHP (php)

Application code became tightly coupled to monitoring products.

Changing observability vendors could require:

replacing SDKs
changing instrumentation
modifying application code
rebuilding services
changing agents
maintaining several telemetry formats

OpenTelemetry changes the model to:

Applications
      |
      v
OpenTelemetry SDK / Auto Instrumentation
      |
      v
OpenTelemetry Collector
      |
      +--------> Backend A
      |
      +--------> Backend B
      |
      +--------> Backend C

The application speaks OpenTelemetry rather than a vendor-specific telemetry language.

OpenTelemetry’s two major principles are that you retain ownership of your telemetry and can use a common set of APIs and conventions rather than learning a new instrumentation model for every backend.

4. When Should You Use OpenTelemetry?

OpenTelemetry is particularly valuable when running:

Microservices

Example:

Browser
   |
Frontend
   |
API Gateway
   |
Orders Service
   |
Payment Service
   |
Kafka
   |
Notification Service

Distributed tracing lets you see one request across the entire system.

Kubernetes

OTel can observe:

Cluster
Nodes
Pods
Containers
Applications
HTTP traffic
gRPC traffic
Databases
Message queues
Kubernetes events
Infrastructure metrics

Multi-cloud environments

For example:

AWS
GCP
Azure
On-premises

All can emit a standardized telemetry format.

Vendor-neutral observability

You might begin with:

OTel → Jaeger

and later move to:

OTel → Grafana Tempo

or:

OTel → Datadog

without redesigning your application instrumentation.

Complex distributed systems

OTel is extremely useful when you need answers to questions such as:

Why is checkout slow?

Which service generated this error?

Which database call caused the latency?

Which Kafka consumer processed this request?

Which Kubernetes pod served the request?

Which version introduced the regression?
Code language: JavaScript (javascript)

5. Where OpenTelemetry Fits

A modern observability stack typically has four layers.

+-----------------------------------------------------+
|                 Applications                        |
| Java | Go | Python | Node | .NET | Rust | etc.     |
+----------------------------+------------------------+
                             |
                             v
+-----------------------------------------------------+
|             OpenTelemetry Instrumentation           |
| APIs | SDKs | Auto Instrumentation | Libraries      |
+----------------------------+------------------------+
                             |
                             | OTLP
                             v
+-----------------------------------------------------+
|              OpenTelemetry Collector                |
|                                                     |
| Receivers → Processors → Exporters                  |
+----------------------------+------------------------+
                             |
                             v
+-----------------------------------------------------+
|              Observability Backends                 |
| Tempo | Prometheus | Datadog | Jaeger | Elastic     |
+----------------------------+------------------------+
                             |
                             v
+-----------------------------------------------------+
|        Dashboards | Search | Alerts | SLOs          |
+-----------------------------------------------------+

6. OpenTelemetry Architecture

A more complete architecture looks like this:

                         APPLICATIONS
                              |
          +-------------------+-------------------+
          |                   |                   |
          v                   v                   v
       Service A           Service B           Service C
       OTel SDK            OTel SDK            OTel SDK
          |                   |                   |
          +-------------------+-------------------+
                              |
                              | OTLP
                              v
                    +--------------------+
                    | OTel Agent         |
                    | Collector          |
                    +--------------------+
                              |
                              v
                    +--------------------+
                    | OTel Gateway       |
                    | Collector          |
                    +--------------------+
                              |
            +-----------------+------------------+
            |                 |                  |
            v                 v                  v
          Traces            Metrics            Logs
            |                 |                  |
            v                 v                  v
          Tempo           Prometheus           Loki
          Jaeger           Datadog             Elastic
          Datadog          etc.                etc.

The Collector itself follows a pipeline model:

Receiver
   |
Processor
   |
Processor
   |
Exporter

The Collector can create separate pipelines for traces, metrics and logs.

7. Core OpenTelemetry Components

OpenTelemetry consists of several major pieces.

Component	Purpose
Specification	Defines expected OpenTelemetry behavior
API	Application-facing instrumentation interfaces
SDK	Implements telemetry generation and processing
Instrumentation Libraries	Instrument common frameworks
Auto Instrumentation	Adds telemetry with minimal/no code changes
Semantic Conventions	Standard attribute names
OTLP	OpenTelemetry wire protocol
Collector	Receives, processes and exports telemetry
Operator	Manages OTel on Kubernetes
Helm Charts	Deploy OTel components on Kubernetes

These are the primary components described by the OpenTelemetry project.

8. The Four Main Signals

8.1 Traces

A trace represents the lifecycle of a request across a distributed system.

Example:

Trace ID: abc123

Frontend
└── GET /checkout                 320 ms
    |
    ├── Cart Service              40 ms
    |
    ├── Inventory Service         70 ms
    |
    └── Payment Service          180 ms
        |
        └── PostgreSQL query      90 ms

You can immediately see:

Payment Service = slow
Database query = major contributor

9. What Is a Span?

A span represents a single operation.

A span may contain:

Span Name
Trace ID
Span ID
Parent Span ID
Start Time
End Time
Duration
Attributes
Events
Links
Status
Span Kind
Code language: PHP (php)

A trace is composed of multiple related spans.

Example:

Trace
│
├── Span: HTTP GET /checkout
│
├── Span: cart.get
│
├── Span: inventory.check
│
├── Span: payment.authorize
│
└── Span: SQL SELECT

10. Trace ID and Span ID

Each trace has a unique:

trace_id

Each span has its own:

span_id

Example:

trace_id = 4bf92f3577b34da6a3ce929d0e0e4736

frontend span:
span_id = 00f067aa0ba902b7

payment span:
span_id = 12ab67aa0ba90019

The common trace ID allows an observability backend to reconstruct the complete distributed request.

11. Context Propagation

Imagine:

Service A → Service B → Service C

How does Service C know it belongs to the same trace?

Through context propagation.

Service A sends tracing context to Service B.

Service B sends it to Service C.

OpenTelemetry normally uses the W3C Trace Context format by default.

For HTTP you may see a header such as:

traceparent:
00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

That essentially identifies:

trace
parent span
trace flags
Code language: PHP (php)

12. Baggage

Baggage allows contextual key/value data to travel with requests.

Example:

customer.tier = gold
region = japan
tenant.id = tenant-42

Architecture:

Frontend
   |
   | baggage
   v
Orders
   |
   | baggage
   v
Payments

All services can access that context.

However, baggage crosses process/service boundaries.

Therefore never put credentials, secrets or sensitive PII into baggage. OpenTelemetry explicitly warns against this.

13. Metrics

Metrics represent numerical measurements over time.

Examples:

requests_total
request_duration
cpu_usage
memory_usage
queue_depth
active_sessions
orders_total
payment_failures

Major OpenTelemetry metric instrument types include:

Instrument	Typical use
Counter	Number of requests
UpDownCounter	Active requests
Gauge	Current temperature/value
Histogram	Request latency distribution
Async Counter	Externally observed cumulative value
Async UpDownCounter	Externally observed variable total
Async Gauge	Externally observed current value

OpenTelemetry metrics also support aggregation and delta/cumulative temporality.

14. Counter Example

orders.created
Code language: CSS (css)

Values:

A counter should only increase.

Typical uses:

HTTP requests
Orders
Errors
Retries
Jobs completed
Messages processed

15. UpDownCounter

Unlike a Counter, an UpDownCounter can increase or decrease.

Example:

active_requests

Request starts:

+1

Request finishes:

-1

Useful for:

queue depth
active connections
active sessions
workers

16. Gauge

A Gauge records a current value.

Examples:

CPU temperature
memory usage
queue size
battery percentage

Conceptually:

The measurement does not need to be monotonic.

17. Histogram

Histograms are particularly important for latency.

Example observations:

10 ms
20 ms
30 ms
70 ms
400 ms
800 ms

Instead of storing every measurement independently, the SDK/backend can aggregate them.

You can then derive information such as:

P50
P90
P95
P99
request count
sum
distribution

18. Logs

Logs are timestamped event records.

Example:

2026-09-15T00:11:18Z ERROR Payment failed order_id=123

OpenTelemetry supports both structured and unstructured logs, although structured logs are preferable. OTel can also correlate logs with active traces by attaching trace and span identifiers.

Example:

{
  "timestamp": "2026-09-15T00:11:18Z",
  "severity": "ERROR",
  "message": "Payment failed",
  "order.id": "123",
  "trace_id": "abc123",
  "span_id": "def456"
}
Code language: JSON / JSON with Comments (json)

You can go from:

Error log
Code language: JavaScript (javascript)

directly to:

Distributed trace

which is enormously useful for troubleshooting.

19. Profiles

Profiles show code-level resource consumption.

For example:

Which function consumes CPU?

Where is memory being allocated?

Which code path is expensive?
Code language: JavaScript (javascript)

Conceptually:

Application
    |
    +--- traces
    +--- metrics
    +--- logs
    +--- profiles

Profiles are a newer part of the OpenTelemetry ecosystem and remain under development in OTLP rather than being as mature as traces, metrics and logs.

20. The OpenTelemetry API

The API is what instrumentation code interacts with.

For tracing:

Tracer
Span
Context

For metrics:

Meter
Counter
Histogram
Gauge

For application libraries, the usual recommendation is:

Library
   |
   v
OTel API

rather than forcing an SDK configuration onto applications consuming that library.

Standalone applications typically use both:

OTel API
+
OTel SDK

This distinction is explicitly part of OpenTelemetry’s recommended instrumentation model.

21. The OpenTelemetry SDK

The SDK implements the API and provides functionality such as:

Sampling
Span processing
Metric aggregation
Exporting
Resources
Configuration

Architecture:

Application code
      |
      v
OTel API
      |
      v
OTel SDK
      |
      v
OTLP Exporter

22. Automatic vs Manual Instrumentation

There are two main instrumentation approaches.

Automatic instrumentation

Automatically instruments frameworks such as:

HTTP servers
HTTP clients
gRPC
Database clients
Messaging libraries
Popular frameworks

Very little application code needs modification.

Manual instrumentation

Developers explicitly create spans, metrics and attributes.

Example:

checkout.validate_cart
payment.authorize
order.persist
Code language: CSS (css)

Manual instrumentation provides business-level visibility that automatic instrumentation cannot infer.

23. Best Instrumentation Strategy

Usually the best strategy is:

Automatic instrumentation
        +
Manual business instrumentation

Automatic instrumentation gives you:

HTTP
database
gRPC
framework
network
messaging

Manual instrumentation adds:

business operations
important workflows
domain attributes
custom metrics

OpenTelemetry explicitly supports using both approaches together.

24. Zero-Code Instrumentation

OpenTelemetry currently provides zero-code approaches for major languages including:

Java
.NET
Python
JavaScript / Node.js
Go
PHP

Mechanisms vary by language and can include:

Java agents
bytecode instrumentation
monkey patching
runtime hooks
eBPF

Zero-code instrumentation usually observes libraries around the edges of an application; important application-specific business operations may still benefit from manual instrumentation.

25. Semantic Conventions

Imagine one service reports:

http.method
Code language: CSS (css)

another:

HTTPMethod

another:

request_method

Querying the data becomes painful.

OpenTelemetry solves this with Semantic Conventions.

They standardize naming for telemetry across:

HTTP
RPC
databases
messaging
cloud
containers
Kubernetes
services
hosts
networks
etc.

OpenTelemetry semantic conventions apply across traces, metrics, logs, profiles and resources.

Examples include:

service.name
service.version
service.namespace

http.request.method

server.address
server.port

deployment.environment.name

k8s.cluster.name
k8s.namespace.name
k8s.pod.name
k8s.container.name
Code language: CSS (css)

26. Resources

A Resource describes the entity producing telemetry.

Example:

service.name = checkout
service.version = 2.7.1

deployment.environment.name = production

cloud.provider = aws
cloud.region = ap-northeast-1

k8s.cluster.name = production
k8s.namespace.name = commerce
Code language: PHP (php)

Resources answer:

“Where did this telemetry originate?”

OpenTelemetry recommends setting service.name explicitly; otherwise SDKs can fall back to an unknown_service value. OTEL_SERVICE_NAME is the standard environment variable for configuring it.

27. Recommended Resource Model

A good baseline is:

service.namespace
service.name
service.version
service.instance.id

deployment.environment.name

cloud.provider
cloud.region

k8s.cluster.name
k8s.namespace.name
k8s.deployment.name
k8s.pod.name
Code language: CSS (css)

For example:

service.namespace = ecommerce
service.name = checkout
service.version = 4.3.2

deployment.environment.name = production

cloud.provider = aws
cloud.region = ap-northeast-1

k8s.cluster.name = prod-eks

deployment.environment.name is the current semantic convention; the older deployment.environment attribute is deprecated.

28. OTLP — OpenTelemetry Protocol

OTLP is the native OpenTelemetry telemetry transport protocol.

It supports:

Traces
Metrics
Logs
Profiles

Two important transports are:

OTLP/gRPC
OTLP/HTTP

The default ports are:

Protocol	Port
OTLP/gRPC	4317
OTLP/HTTP	4318

For OTLP/HTTP the conventional endpoints include:

/v1/traces
/v1/metrics
/v1/logs

The current OTLP specification defines Protocol Buffers for the data and supports gRPC and HTTP transports.

29. OTLP/gRPC vs OTLP/HTTP

OTLP/gRPC

Common endpoint:

otel-collector:4317
Code language: CSS (css)

Advantages:

Efficient protobuf transport
HTTP/2
widely used service-to-collector protocol

OTLP/HTTP

Common endpoint:

http://otel-collector:4318
Code language: JavaScript (javascript)

Signal URLs generally become:

/v1/traces
/v1/metrics
/v1/logs

Advantages include simpler compatibility with environments where HTTP infrastructure is easier to operate.

The current generic exporter specification prefers http/protobuf as the default where backward-compatibility constraints do not require something else, although individual SDK defaults can vary.

30. Important OpenTelemetry Environment Variables

These are worth memorizing.

Service name

OTEL_SERVICE_NAME=checkout-service

Resource attributes

OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production,service.version=1.4.3"
Code language: JavaScript (javascript)

OTLP endpoint

OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
Code language: JavaScript (javascript)

Protocol

OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

or:

OTEL_EXPORTER_OTLP_PROTOCOL=grpc

Trace-specific endpoint

OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://otel-collector:4318/v1/traces
Code language: JavaScript (javascript)

Metrics endpoint

OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://otel-collector:4318/v1/metrics
Code language: JavaScript (javascript)

Logs endpoint

OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://otel-collector:4318/v1/logs
Code language: JavaScript (javascript)

Authentication headers

OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer TOKEN"
Code language: JavaScript (javascript)

Exporter endpoint, protocol, certificate and header variables are standardized in the OpenTelemetry exporter specification.

31. OpenTelemetry Collector

The Collector is one of the most important OTel components.

Its job is to:

Receive telemetry
Process telemetry
Transform telemetry
Filter telemetry
Sample telemetry
Batch telemetry
Route telemetry
Export telemetry

It is vendor-neutral and can receive and emit multiple telemetry formats.

32. Collector Architecture

The fundamental Collector model is:

RECEIVER
   |
   v
PROCESSOR
   |
   v
EXPORTER

More realistically:

             +----------+
Application -> Receiver |
             +----+-----+
                  |
                  v
            +-----------+
            | Processor |
            +-----+-----+
                  |
                  v
            +-----------+
            | Processor |
            +-----+-----+
                  |
                  v
            +-----------+
            | Exporter  |
            +-----+-----+
                  |
                  v
              Backend

33. Receivers

Receivers ingest telemetry.

Examples can include:

OTLP
Prometheus
Jaeger
Zipkin
Kafka
File logs
Host metrics
Kubernetes metrics
Cloud provider sources

Example:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
Code language: CSS (css)

34. Processors

Processors modify or manage telemetry.

Common functions include:

Batching
Filtering
Sampling
Resource detection
Attribute manipulation
Kubernetes metadata enrichment
Memory protection
Transformation
Redaction

A particularly important ordering pattern is:

memory_limiter
      |
drop / sampling processors
      |
context-dependent processors
      |
enrichment / transformation
      |
batch

Processor ordering matters because processing occurs sequentially. OpenTelemetry recommends the memory limiter early, dropping unnecessary telemetry before expensive processing, and batching after filtering/enrichment.

35. Exporters

Exporters send telemetry onward.

For example:

OTLP backend
Prometheus-compatible backend
Jaeger-compatible backend
Kafka
Vendor backend
Debug output

A single Collector can also fan out telemetry to several destinations.

             +--> Backend A
Collector ---+
             +--> Backend B

Each exporter receives a copy of the relevant data from the pipeline.

36. Extensions

Extensions provide additional Collector capabilities outside normal telemetry pipelines.

Examples include:

Authentication
Health checks
Service discovery
Storage
Debugging

The available extensions depend on the Collector distribution being used.

37. Connectors

Connectors connect one Collector pipeline to another and act as both an exporter and a receiver.

Example:

Trace pipeline
      |
      v
Span Metrics Connector
      |
      v
Metrics pipeline

This can generate metrics derived from spans.

Current Collector distributions include connectors for use cases such as service graphs, span metrics, routing and signal-to-metrics conversion.

38. Basic Collector Configuration

Create:

otel-config.yaml
Code language: CSS (css)

Example:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15

  batch:
    timeout: 5s

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:

    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]
Code language: CSS (css)

This creates:

OTLP
 |
 +--> traces  -> memory_limiter -> batch -> debug
 |
 +--> metrics -> memory_limiter -> batch -> debug
 |
 +--> logs    -> memory_limiter -> batch -> debug

The official Python exporter documentation uses essentially this OTLP receiver + debug exporter arrangement to validate an exporter locally.

39. Install OpenTelemetry Collector with Docker

The current official documentation examples use Collector 0.160.0.

docker pull otel/opentelemetry-collector:0.160.0

Run:

docker run \
  --rm \
  -p 4317:4317 \
  -p 4318:4318 \
  -v "$(pwd)/otel-config.yaml:/etc/otelcol/config.yaml" \
  otel/opentelemetry-collector:0.160.0
Code language: JavaScript (javascript)

The Collector now accepts:

OTLP/gRPC -> localhost:4317
OTLP/HTTP -> localhost:4318

Official Docker installation documentation supports mounting /etc/otelcol/config.yaml in this manner.

40. Linux Installation

OpenTelemetry publishes:

DEB
RPM
Linux binaries

for multiple architectures.

For example, the current documented DEB workflow uses:

wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.160.0/otelcol_0.160.0_linux_amd64.deb

sudo dpkg -i otelcol_0.160.0_linux_amd64.deb
Code language: JavaScript (javascript)

The default Linux package configuration is:

/etc/otelcol/config.yaml

and packaged installations can run via systemd.

41. Your First Instrumented Python Application

Install:

python -m venv .venv

source .venv/bin/activate

Then:

pip install flask requests
pip install opentelemetry-distro
pip install opentelemetry-exporter-otlp

Install compatible instrumentation:

opentelemetry-bootstrap -a install

OpenTelemetry’s Python zero-code flow uses opentelemetry-distro, the OTLP exporter and opentelemetry-bootstrap to detect installed libraries and install matching instrumentation packages.

Create:

app.py
Code language: CSS (css)

from flask import Flask
import requests

app = Flask(__name__)

@app.route("/")
def home():
    requests.get("https://example.com")
    return "Hello OpenTelemetry!"

if __name__ == "__main__":
    app.run(port=5000)
Code language: JavaScript (javascript)

Configure OpenTelemetry:

export OTEL_SERVICE_NAME=demo-python-service

export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=development"

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
Code language: JavaScript (javascript)

Run with auto instrumentation:

opentelemetry-instrument python app.py
Code language: CSS (css)

Request:

curl http://localhost:5000
Code language: JavaScript (javascript)

The flow becomes:

curl
 |
 v
Flask
 |
 | auto-generated spans
 v
OpenTelemetry Python
 |
 | OTLP/HTTP
 v
Collector
 |
 v
debug exporter

42. Manual Instrumentation

Automatic instrumentation is only the beginning.

Suppose checkout processing contains:

Validate Cart
Reserve Inventory
Authorize Payment
Create Order

Those operations are business concepts.

You can create spans manually.

Example Python:

from opentelemetry import trace

tracer = trace.get_tracer("checkout-service")

def checkout(order):
    with tracer.start_as_current_span("checkout.process") as span:

        span.set_attribute("order.type", order["type"])

        validate_cart(order)

        with tracer.start_as_current_span("inventory.reserve"):
            reserve_inventory(order)

        with tracer.start_as_current_span("payment.authorize"):
            authorize_payment(order)
Code language: JavaScript (javascript)

The resulting trace becomes:

HTTP POST /checkout
│
└── checkout.process
    │
    ├── inventory.reserve
    │
    └── payment.authorize

This is considerably more useful than seeing only generic HTTP calls.

43. Custom Metrics

Conceptually:

from opentelemetry import metrics

meter = metrics.get_meter("checkout-service")

orders = meter.create_counter(
    "orders.created",
    description="Number of successfully created orders"
)

orders.add(
    1,
    {
        "payment.method": "credit_card"
    }
)
Code language: JavaScript (javascript)

You could then analyze:

orders.created
Code language: CSS (css)

by safe, bounded dimensions such as payment method.

44. Metric Cardinality — Critical Production Topic

A metric such as:

http.requests
Code language: CSS (css)

with:

http.method = GET

has low cardinality.

But adding:

user.id = 183729101
session.id = abc987123
request.id = randomUUID

can create millions of unique time series.

This causes:

Huge memory consumption
Huge observability bills
Slow queries
Backend pressure
Collector pressure

Therefore avoid unbounded dimensions in metrics.

Bad:

order.id
user.id
session.id
request.id
full URL
random UUID
Code language: CSS (css)

Better:

http.request.method
http.response.status_code
service.name
deployment.environment.name
region
operation
Code language: CSS (css)

45. Span Cardinality

Span attributes tolerate more cardinality than metrics, but you still should not create dynamically unique span names.

Bad:

GET /users/12345
GET /users/89431
GET /users/82173

Better:

GET /users/{id}

Then store the relevant normalized information as appropriate attributes.

Stable span naming dramatically improves grouping and analysis.

46. Logs + Traces Correlation

Suppose your application logs:

Payment failed

Without correlation, you search manually.

With OTel:

{
  "message": "Payment failed",
  "trace_id": "8fabc...",
  "span_id": "2b73..."
}
Code language: JSON / JSON with Comments (json)

You can navigate:

Log
 |
 v
Trace
 |
 v
Payment Span
 |
 v
Database Span

This correlation is one of the strongest arguments for unifying telemetry through OpenTelemetry.

47. Collector Deployment Patterns

There are three especially important patterns.

Pattern 1 — Agent

App
 |
 v
Local Collector
 |
 v
Backend

The Collector runs close to the application, for example:

VM process
Kubernetes DaemonSet
Sidecar

The official documentation calls this the agent deployment pattern.

48. Pattern 2 — Gateway

App A --+
        |
App B --+--> Gateway Collector --> Backend
        |
App C --+

Advantages:

central configuration
central credentials
central routing
central filtering
central sampling
central policy

A gateway generally exposes a shared OTLP endpoint per cluster, data center or region.

49. Pattern 3 — Agent + Gateway

For medium and large environments, this is usually the strongest architecture:

                 Node 1
App A ---> Agent Collector
                    |
                 Node 2
App B ---> Agent Collector
                    |
                 Node 3
App C ---> Agent Collector
                    |
                    v
             Gateway Collectors
                    |
                    v
             Observability Backend

Agents perform lightweight/local operations.

Gateways perform:

sampling
filtering
batching
routing
authentication
credential management
backend fan-out

OpenTelemetry’s current deployment guidance highlights separation of concerns, scalable processing and stable egress/security as major advantages of the combined architecture.

50. Production Kubernetes Architecture

A mature Kubernetes deployment can look like this:

+---------------------------------------------------------+
| Kubernetes Cluster                                      |
|                                                         |
| Node A                       Node B                      |
| ----------------           ----------------             |
| Application Pods            Application Pods            |
|      |                           |                      |
|      v                           v                      |
| OTel Agent DS              OTel Agent DS                |
|      |                           |                      |
|      +------------+--------------+                      |
|                   |                                     |
|                   v                                     |
|          OTel Gateway Service                           |
|                   |                                     |
|          +--------+--------+                            |
|          |        |        |                            |
|          v        v        v                            |
|      Gateway  Gateway  Gateway                          |
+----------+--------+--------+----------------------------+
           |
           | TLS + OTLP
           v
+---------------------------------------------------------+
| Observability Platform                                  |
| Traces | Metrics | Logs                                 |
+---------------------------------------------------------+

51. Install Collector on Kubernetes with Helm

Add the official repository:

helm repo add open-telemetry \
  https://open-telemetry.github.io/opentelemetry-helm-charts

helm repo update
Code language: JavaScript (javascript)

Install as DaemonSet:

helm install otel-agent \
  open-telemetry/opentelemetry-collector \
  --set image.repository=otel/opentelemetry-collector-k8s \
  --set mode=daemonset
Code language: JavaScript (javascript)

Or gateway:

helm install otel-gateway \
  open-telemetry/opentelemetry-collector \
  --set image.repository=otel/opentelemetry-collector-k8s \
  --set mode=deployment
Code language: JavaScript (javascript)

The official chart supports:

daemonset
deployment
statefulset

modes.

52. Kubernetes Data Collection

Useful Collector components include:

k8sattributes
kubeletstats
hostmetrics
k8scluster
k8sobjects
filelog
prometheus

Conceptually:

                 Kubernetes
                     |
       +-------------+-------------+
       |             |             |
    Pod logs      Kubelet       API Server
       |             |             |
       v             v             v
    filelog     kubeletstats   k8scluster
       |             |             |
       +-------------+-------------+
                     |
                     v
                  Collector

53. Kubernetes Attributes Processor

One of the most valuable processors is:

k8sattributes

It can enrich telemetry with metadata such as:

k8s.cluster.name
k8s.namespace.name
k8s.pod.name
k8s.node.name
Code language: CSS (css)

The official Helm chart strongly recommends enabling the Kubernetes attributes preset or manually configuring this processor where appropriate.

Example Helm values:

mode: daemonset

presets:
  kubernetesAttributes:
    enabled: true
Code language: HTTP (http)

54. Kubernetes Node Metrics

For node/pod/container metrics:

presets:
  kubeletMetrics:
    enabled: true
Code language: JavaScript (javascript)

DaemonSet mode is generally preferred because the Kubelet Stats receiver observes the node where that Collector instance is running.

55. Host Metrics

Example:

presets:
  hostMetrics:
    enabled: true
Code language: JavaScript (javascript)

This can provide:

CPU
load
memory
disk
filesystem
network

The Helm chart documents this as another DaemonSet-oriented use case.

56. Cluster Metrics

Cluster-wide metrics are different.

Example:

mode: deployment

replicaCount: 1

presets:
  clusterMetrics:
    enabled: true
Code language: HTTP (http)

Why one replica?

Because several identical collectors gathering the same cluster-wide metrics can generate duplicate telemetry.

The official Helm documentation specifically warns about duplicate data for cluster-level receivers.

57. Kubernetes Events

You can also collect Kubernetes events:

presets:
  kubernetesEvents:
    enabled: true
Code language: JavaScript (javascript)

This allows operational events to join the rest of your telemetry pipeline.

58. OpenTelemetry Operator

The OpenTelemetry Operator manages:

OpenTelemetry Collectors
Automatic instrumentation

It can inject instrumentation into workloads rather than requiring every team to configure it independently.

Install the Operator, for example, using the official release manifest after satisfying its prerequisites:

kubectl apply -f \
https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
Code language: JavaScript (javascript)

For production, pin an explicitly tested Operator version rather than relying permanently on latest.

59. OpenTelemetryCollector Resource

Example:

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector

metadata:
  name: otel

spec:
  config:

    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318

    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15

      batch: {}

    exporters:
      debug: {}

    service:
      pipelines:

        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [debug]
Code language: HTTP (http)

The current Operator documentation uses the opentelemetry.io/v1beta1 Collector CRD in its getting-started example.

60. Kubernetes Automatic Instrumentation

An Instrumentation resource can define how applications export telemetry.

Example Python-oriented configuration:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation

metadata:
  name: default-instrumentation

spec:

  exporter:
    endpoint: http://otel-collector:4318

  propagators:
    - tracecontext
    - baggage

  sampler:
    type: parentbased_traceidratio
    argument: "1"
Code language: JavaScript (javascript)

Then annotate a Deployment:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-python: "true"
Code language: JavaScript (javascript)

Supported injection annotations include language-specific options for Java, Node.js, Python, .NET and Go.

61. Sampling

Tracing every request can become expensive.

Suppose:

50,000 requests / second

If every request creates:

20 spans

then:

1,000,000 spans / second

may be produced.

Sampling controls telemetry volume.

There are two important approaches:

Head Sampling
Tail Sampling

62. Head Sampling

The decision is made early.

Example:

Keep 10%
Drop 90%

Flow:

Request
   |
Sampling decision
   |
   +-- Keep → create/export trace
   |
   +-- Drop
Code language: JavaScript (javascript)

Advantages:

simple
fast
cheap
scalable

Disadvantage:

The system doesn’t yet know whether the request will eventually:

fail
become slow
contain an interesting anomaly

OpenTelemetry describes consistent probability sampling as a common head-sampling strategy.

63. Tail Sampling

Tail sampling decides after observing much or all of the trace.

For example:

IF error
    keep 100%

IF latency > 2 seconds
    keep 100%

IF VIP workflow
    keep 100%

ELSE
    keep 5%
Code language: PHP (php)

Advantages:

keep failures
keep slow requests
keep unusual transactions
reduce normal trace volume

This is considerably smarter than purely random sampling.

64. Tail Sampling Architecture

Tail sampling requires all spans belonging to a given trace to reach the same sampling Collector.

Bad:

Trace abc

Span 1 -> Gateway A
Span 2 -> Gateway B
Span 3 -> Gateway C

No gateway sees the whole trace.

Correct architecture:

Trace abc
   |
Trace-ID aware routing
   |
   v
Gateway B

Span 1 -> B
Span 2 -> B
Span 3 -> B

OpenTelemetry recommends trace-ID-aware load balancing for multi-gateway tail sampling and warns that this is an advanced architecture requiring careful testing.

65. Recommended Sampling Model

A mature system frequently uses:

Application
    |
    | Parent-based head sampling
    v
Agent
    |
    v
Trace-ID router
    |
    v
Tail Sampling Gateway
Code language: PHP (php)

Example policies:

100% errors
100% very slow requests
100% critical transactions
100% selected release/canary traffic
small percentage of healthy traffic

Avoid sampling so aggressively that normal baseline behavior becomes impossible to understand.

66. Filtering

Not all telemetry deserves to reach storage.

Possible noise:

/health
/readiness
/liveness
internal probes
low-value debug logs
known crawler traffic

Filtering these centrally can significantly reduce cost.

Conceptual pipeline:

Receiver
   |
Memory limiter
   |
Filter
   |
Transform
   |
Batch
   |
Exporter

67. Attribute Redaction

Telemetry can accidentally contain:

Authorization headers
API keys
tokens
passwords
email addresses
user IDs
credit-card information
database query parameters

Therefore implement controls such as:

Allow-list attributes
Delete sensitive attributes
Hash selected identifiers
Mask values
Drop sensitive records
Code language: PHP (php)

A good rule:

Treat observability pipelines as production data systems, not harmless debugging utilities.

68. Production Collector Configuration Model

A typical architecture looks more like:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:

  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15

  batch:
    timeout: 5s

exporters:

  otlp/backend:
    endpoint: ${env:OTEL_BACKEND_ENDPOINT}

service:

  pipelines:

    traces:
      receivers: [otlp]
      processors:
        - memory_limiter
        - batch
      exporters:
        - otlp/backend

    metrics:
      receivers: [otlp]
      processors:
        - memory_limiter
        - batch
      exporters:
        - otlp/backend

    logs:
      receivers: [otlp]
      processors:
        - memory_limiter
        - batch
      exporters:
        - otlp/backend

Credentials should come from your secrets mechanism rather than being hard-coded into Git.

69. Collector Resilience

Imagine:

Collector → Backend

and the backend becomes unavailable.

Without resilience:

telemetry → DROP

With a queue:

telemetry → queue → retry → backend

OpenTelemetry Collector exporters support retry and sending-queue mechanisms. Current Collector documentation describes exponential backoff and queue-based protection from temporary backend failures.

70. Persistent Queue / WAL

An in-memory queue disappears if the Collector crashes.

A more resilient design uses persistent storage:

Collector
   |
   v
Disk-backed queue / WAL
   |
   v
Backend

The file_storage extension can provide persistent queue storage.

Conceptual configuration:

extensions:
  file_storage:
    directory: /var/lib/otelcol/storage

exporters:
  otlp/backend:
    endpoint: backend:4317

    sending_queue:
      storage: file_storage

service:
  extensions:
    - file_storage
Code language: JavaScript (javascript)

This lets queued telemetry survive Collector restarts, subject to disk capacity and other failure modes.

71. Extreme Resilience — Kafka

For extremely critical telemetry:

Apps
 |
Collectors
 |
Kafka
 |
Gateway Collectors
 |
Observability Backend

Kafka introduces:

durability
decoupling
backpressure isolation
replay possibilities

But it also adds significant operational complexity.

Do not automatically add Kafka merely because it sounds more enterprise-grade.

Use it when telemetry durability requirements justify another distributed system.

72. Memory Limiter

The Collector must protect itself during spikes.

Without protection:

Traffic spike
   |
Huge telemetry volume
   |
Collector memory growth
   |
OOMKilled

With memory_limiter:

Traffic spike
   |
Memory threshold
   |
Backpressure / refused telemetry
   |
Collector remains alive

Monitor refusal metrics, because frequent refusal means capacity or downstream throughput is inadequate.

73. Batch Processor

Instead of:

Span -> network
Span -> network
Span -> network
Span -> network

batching creates:

Span
Span
Span
Span
  |
Batch
  |
Network

Benefits include:

fewer requests
better compression
lower networking overhead
better export efficiency
Code language: JavaScript (javascript)

OpenTelemetry recommends batching after processors that may drop telemetry.

74. Collector Scaling

Important Collector metrics include:

otelcol_exporter_queue_size
otelcol_exporter_queue_capacity

otelcol_exporter_send_failed_spans

otelcol_processor_refused_spans

and equivalent signal-specific metrics.

OpenTelemetry’s scaling guidance suggests paying close attention as exporter queues approach roughly 60–70% of capacity, while also determining whether the actual bottleneck is the Collector, network or backend.

75. Don’t Blindly Scale Collectors

Suppose:

Collector queue = full

It is tempting to add more Collectors.

But perhaps:

Backend is overloaded

Then:

More Collectors
    |
    v
More pressure
    |
    v
Backend gets worse

Always determine:

Collector bottleneck?
Network bottleneck?
Backend bottleneck?

before scaling.

This exact distinction is emphasized in OpenTelemetry’s Collector scaling guidance.

76. Collector Self-Observability

Your observability pipeline itself needs observability.

Monitor:

Collector CPU
Collector memory
received telemetry
accepted telemetry
dropped telemetry
export failures
queue utilization
export latency
processor refusals
restarts
Code language: JavaScript (javascript)

Alert on conditions such as:

export failures > threshold

queue utilization > threshold

memory near limit

Collector unavailable

telemetry refusal > 0 for sustained period
Code language: JavaScript (javascript)

Otherwise your monitoring system can silently fail while telling you everything is healthy.

77. Security Architecture

Production communication should ideally look like:

Application
   |
mTLS / trusted cluster network
   |
Agent
   |
TLS
   |
Gateway
   |
TLS + Authentication
   |
Backend

OpenTelemetry’s security guidance recommends encryption and authentication for Collector communications and TLS or mTLS in production as appropriate.

78. Collector Security Best Practices

Follow these principles:

Do not hard-code secrets

Avoid:

headers:
  api-key: super-secret-value
Code language: JavaScript (javascript)

Prefer:

Secret Manager
Kubernetes Secret
Vault
Environment variable
workload identity where supported

Use TLS

Avoid exposing unencrypted OTLP across untrusted networks.

Consider mTLS

Useful where client authentication at the transport layer is required.

Run least privilege

The Collector should normally not run as root.

Only grant privileged mounts or capabilities when a receiver genuinely requires them.

Restrict receivers

Do not accidentally expose:

0.0.0.0:4317
0.0.0.0:4318
Code language: CSS (css)

to the public internet without intentional authentication and network controls.

Minimize components

Only enable the Collector components you actually need.

OpenTelemetry specifically recommends reducing the component set to minimize attack surface and mentions the OpenTelemetry Collector Builder as an option for producing a constrained distribution.

79. Sensitive Data Governance

Create a telemetry data classification policy.

For example:

Category	Allowed?
Service name	Yes
Environment	Yes
Region	Yes
HTTP method	Yes
Status code	Yes
Order category	Yes
Password	Never
Authorization token	Never
API key	Never
Credit card number	Never
Sensitive PII	Usually no

Filtering should occur as close to ingestion as practical.

80. OpenTelemetry vs Prometheus

They are complementary.

Prometheus is primarily:

metrics collection/storage/querying

OpenTelemetry provides:

instrumentation
traces
metrics
logs
context propagation
telemetry processing
export
Code language: JavaScript (javascript)

Example architecture:

App
 |
OTel
 |
Collector
 |
Prometheus
 |
Grafana

Do not think:

OTel OR Prometheus

Think:

OTel + Prometheus

where appropriate.

81. OpenTelemetry vs Jaeger

Jaeger primarily focuses on distributed tracing storage/query/visualization.

OTel provides instrumentation and transport.

Architecture:

Applications
 |
OTel
 |
Collector
 |
Jaeger

82. OpenTelemetry vs Datadog/New Relic/Dynatrace

Commercial platforms provide:

backend storage
analytics
dashboards
alerts
APM UI
SLOs
incident tooling

OpenTelemetry can provide a vendor-neutral telemetry ingestion layer.

Apps
 |
OTel
 |
Collector
 |
Datadog / New Relic / Dynatrace
Code language: PHP (php)

This can reduce instrumentation coupling even when the observability backend remains commercial.

83. OpenTelemetry vs Fluent Bit / Fluentd

Fluent Bit and Fluentd historically focus heavily on:

logs

OpenTelemetry supports:

traces
metrics
logs

and increasingly unified signal processing.

Some organizations retain Fluent Bit for specialized log pipelines while adopting OpenTelemetry for traces/metrics; others consolidate parts of the stack into OTel Collector.

The right answer depends on required receivers, performance characteristics and operational maturity.

84. Typical End-to-End Request

Imagine:

Mobile Client
   |
API Gateway
   |
Order Service
   |
Kafka
   |
Payment Service
   |
PostgreSQL

A trace might look like:

TRACE: checkout

Mobile API request                 920 ms
│
├─ API Gateway                     20 ms
│
├─ Order Service                  120 ms
│  ├─ validate_cart               15 ms
│  └─ publish Kafka event         25 ms
│
├─ Kafka                           40 ms
│
└─ Payment Service               700 ms
   │
   ├─ payment.authorize          500 ms
   │
   └─ PostgreSQL                  80 ms
Code language: HTTP (http)

You instantly discover:

payment.authorize
Code language: CSS (css)

is consuming most latency.

85. RED Method with OpenTelemetry

For services, track:

Rate

requests / second

Errors

failed requests / second
error percentage

Duration

P50
P95
P99

OpenTelemetry traces + metrics make RED dashboards straightforward.

86. USE Method for Infrastructure

For infrastructure resources:

Utilization

CPU utilization
memory utilization

Saturation

queue depth
throttling

Errors

disk errors
network errors

Host and Kubernetes metrics can provide much of this information.

87. Golden Signals

A classic set of service signals is:

Latency
Traffic
Errors
Saturation

A mature observability implementation should expose all four.

88. Service Level Indicators

OTel metrics can feed SLIs such as:

Availability

successful_requests
-------------------
total_requests

Latency:

requests < 500ms
----------------
total_requests

Then build:

SLO
Error Budget
Alert
Code language: JavaScript (javascript)

in the monitoring backend.

89. Production Naming Strategy

Define conventions centrally.

For example:

service.namespace = ecommerce
service.name = checkout
service.version = 2.5.0
deployment.environment.name = production

Avoid teams inventing:

checkout
Checkout
checkout-service
checkout_prod
prod-checkout
checkout-v2

for the same logical service.

Stable identity is fundamental to useful observability.

90. Recommended Service Naming

Good:

checkout
payment
inventory
vehicle-api
telemetry-consumer

Poor:

checkout-prod-pod-6745678
checkout-v4-new
my-service
backend1
Code language: JavaScript (javascript)

Dynamic instance information belongs in:

service.instance.id
k8s.pod.name
Code language: CSS (css)

not the logical service.name.

The service semantic conventions explicitly define service.name as a logical service identity shared across horizontally scaled instances.

91. Recommended Environment Naming

Use the semantic convention:

deployment.environment.name
Code language: CSS (css)

For example:

development
test
staging
production

OpenTelemetry defines those as well-known values.

92. Cost Optimization

Observability cost is strongly influenced by:

telemetry volume
cardinality
retention
sampling
log verbosity
attribute size
number of services
number of environments

Control cost at multiple layers:

Application
  ↓
Head Sampling
  ↓
Agent Filtering
  ↓
Gateway Filtering
  ↓
Tail Sampling
  ↓
Backend Retention

Do not depend entirely on backend billing controls.

93. What Should Be Sampled?

Good candidates for reduction:

high-volume successful requests
health probes
repetitive background traffic
known internal noise
verbose debugging

Good candidates to preserve:

errors
timeouts
slow requests
critical business transactions
new release traffic
security-related events
rare operations
Code language: JavaScript (javascript)

94. What Not to Instrument Excessively

Do not create spans for every tiny function.

Bad:

parseInteger()
incrementCounter()
formatString()
getCurrentTime()

A trace containing thousands of trivial spans can become harder to understand than no trace at all.

Create spans around meaningful operations such as:

network requests
database calls
queue operations
business operations
expensive computation
critical workflow steps

95. Database Instrumentation

Capture useful information such as:

database system
operation
duration
server
query class
error status
Code language: JavaScript (javascript)

But be extremely careful with:

raw SQL
bind parameters
customer data
credentials

Database observability should not become a data-leak mechanism.

96. Messaging / Kafka Instrumentation

For messaging systems, useful telemetry can include:

publish
consume
queue/topic
latency
consumer processing duration
errors
retry count

Distributed tracing across asynchronous systems requires careful propagation of trace context through message metadata.

A typical flow is:

Producer
  |
trace context in message headers
  |
Kafka
  |
Consumer
  |
continued trace

97. Trace Links

Not every asynchronous relationship is naturally parent → child.

For batch processing or fan-in/fan-out systems, span Links can express relationships to other span contexts.

Example:

Message A ─┐
Message B ─┼─> Batch processor span
Message C ─┘

The batch processing span may link to the originating spans rather than pretending there is one simple parent.

Links are part of the OpenTelemetry Span model.

98. CI/CD Observability

Add deployment metadata such as:

service.version
git commit
release version
environment
Code language: CSS (css)

Then correlate:

Deployment v2.4.0
       |
       v
Error rate increases
       |
       v
Latency regression
Code language: JavaScript (javascript)

This makes release regression detection dramatically easier.

99. Recommended Adoption Strategy

Do not attempt to instrument the entire organization on day one.

A better journey is:

Phase 1
Collector + one service

Phase 2
Automatic traces

Phase 3
Resource conventions

Phase 4
Custom business spans

Phase 5
Application metrics

Phase 6
Log correlation

Phase 7
Kubernetes/infrastructure telemetry

Phase 8
Sampling + filtering

Phase 9
SLOs + alerts

Phase 10
Governance + scale

100. Phase 1 — Proof of Concept

Choose one representative service.

Deploy:

App
 |
OTel SDK
 |
Collector
 |
Backend

Validate:

trace received
service name correct
attributes correct
context propagation correct
latency correct

101. Phase 2 — Standardize

Define organization-wide conventions:

service naming
environment naming
resource attributes
sampling
PII policies
OTLP endpoints
backend routing
version metadata

This is where an OTel platform becomes maintainable rather than a collection of unrelated experiments.

102. Phase 3 — Centralize Collection

Move toward:

Applications
   |
Agent Collectors
   |
Gateway Collectors
   |
Backend

Centralize:

authentication
security
sampling
routing
filtering
redaction

103. Phase 4 — Add Metrics and Logs

Once tracing is stable:

Traces
+
Metrics
+
Logs

Then correlate them.

Example:

Alert:
P99 checkout latency high

        ↓

Metric identifies checkout service

        ↓

Trace identifies payment span

        ↓

Logs show payment timeout

That is practical observability.

104. Phase 5 — SLO-Based Observability

Build:

SLI
 |
v
SLO
 |
v
Error Budget
 |
v
Burn Rate Alert
 |
v
Trace investigation
Code language: JavaScript (javascript)

This moves observability from:

"lots of dashboards"
Code language: JSON / JSON with Comments (json)

to:

"actionable reliability engineering"
Code language: JSON / JSON with Comments (json)

105. Troubleshooting — No Telemetry Arriving

Use a layered approach.

Step 1 — Application

Verify:

OTel instrumentation loaded?
Correct service name?
Exporter enabled?

Step 2 — Endpoint

Verify:

OTEL_EXPORTER_OTLP_ENDPOINT

Check whether you meant:

or:

Step 3 — Protocol

Verify:

grpc

versus:

http/protobuf

A protocol mismatch is a very common failure.

Step 4 — Collector receiver

Verify:

receivers:
  otlp:
    protocols:
      grpc:
      http:

Step 5 — Collector pipeline

Defining a component is not enough.

It must be referenced in:

service:
  pipelines:

106. Use Debug Exporter

Temporarily:

exporters:
  debug:
    verbosity: detailed

Then:

service:
  pipelines:
    traces:
      exporters:
        - debug

If telemetry appears there:

Application → Collector = GOOD

Now investigate:

Collector → Backend

107. Collector Troubleshooting Flow

Application generating telemetry?
        |
       Yes
        |
Collector receiving telemetry?
        |
       Yes
        |
Processor dropping telemetry?
        |
       No
        |
Exporter succeeding?
        |
       Yes
        |
Backend ingesting telemetry?
        |
       Yes
        |
Backend query / dashboard correct?

Troubleshoot one boundary at a time.

108. Broken Distributed Trace

Symptoms:

Service A trace
Service B separate trace

Investigate:

W3C traceparent propagation
proxy/header stripping
HTTP instrumentation
gRPC instrumentation
message header propagation
custom middleware
async context handling
Code language: JavaScript (javascript)

OpenTelemetry context propagation is what allows downstream spans to join the same distributed trace.

109. Missing Kubernetes Metadata

Check:

k8sattributes processor enabled?
RBAC correct?
Processor actually in pipeline?
Pod association working?

Remember:

Declaring processor
!=
Using processor

It needs to appear in the pipeline.

110. Duplicate Metrics

Common Kubernetes cause:

Multiple gateway replicas
   |
   v
Same target scraped repeatedly

The OpenTelemetry Kubernetes documentation specifically warns that identical Collector replicas scraping the same targets can generate duplicate metrics.

Use:

Target Allocator
sharding
single-replica cluster receivers
appropriate DaemonSets

depending on the receiver.

111. OpenTelemetry Demo

The OpenTelemetry project maintains a complete demo application.

git clone https://github.com/open-telemetry/opentelemetry-demo.git

cd opentelemetry-demo

make start
Code language: PHP (php)

The full demo includes multiple services and observability backends and is one of the best practical environments for learning how OTel works end-to-end.

A very useful learning exercise is to:

1. Run the demo.
2. Generate traffic.
3. View traces.
4. Find a slow span.
5. Inspect related metrics.
6. Inspect related logs.
7. Modify Collector configuration.
8. Add filtering.
9. Experiment with sampling.
Code language: JavaScript (javascript)

112. Development vs Production Configuration

Development:

100% traces
debug exporter
simple Collector
no HA
minimal filtering

Production:

sampling
HA Collectors
TLS
authentication
queues
retries
resource limits
memory limiter
batching
redaction
PII controls
self-monitoring
autoscaling
version pinning
Code language: PHP (php)

Never promote a tutorial Collector configuration directly into production without applying those controls.

113. Recommended Production Architecture

For most sizeable cloud/Kubernetes environments:

                         APPLICATIONS
                              |
            Auto + Manual OTel Instrumentation
                              |
                              v
                    OTLP 4317 / 4318
                              |
                              v
             +--------------------------------+
             | Node-level OTel Agent          |
             | Collector DaemonSet            |
             |                                |
             | - logs                         |
             | - kubelet metrics              |
             | - host metrics                 |
             | - lightweight enrichment       |
             +---------------+----------------+
                             |
                             v
             +--------------------------------+
             | OTel Gateway Tier              |
             |                                |
             | memory protection              |
             | filtering                      |
             | redaction                      |
             | transformation                 |
             | tail sampling                  |
             | batching                       |
             | queue / retry                  |
             | backend credentials            |
             +---------------+----------------+
                             |
                             | TLS
                             v
             +--------------------------------+
             | Observability Backend          |
             |                                |
             | Traces                         |
             | Metrics                        |
             | Logs                           |
             +--------------------------------+

For tail sampling across horizontally scaled gateways:

Agents
  |
trace-ID aware load balancing
  |
Tail Sampling Gateways

not arbitrary round-robin routing.

114. Production HA

For gateways:

minimum multiple replicas where architecture permits
PodDisruptionBudget
anti-affinity / topology spread
resource requests
resource limits
health probes
HPA

But stateful processing such as tail sampling requires more careful routing than ordinary stateless gateway scaling.

115. Kubernetes Resource Management

Always define Collector resources.

Example conceptually:

resources:

  requests:
    cpu: 250m
    memory: 512Mi

  limits:
    cpu: "1"
    memory: 1Gi
Code language: JavaScript (javascript)

Then configure memory limiting appropriately below the container’s actual hard memory limit.

Leave headroom.

Do not configure:

memory_limiter threshold == container limit

because the process may be OOM-killed before protection becomes effective.

116. Network Architecture

Prefer:

Applications
   |
Cluster-private OTLP endpoint
   |
Gateway
   |
Controlled egress
   |
Backend
Code language: PHP (php)

over:

Every application
   |
Internet
   |
Observability vendor

Centralized egress provides:

simpler firewalling
credential isolation
network control
consistent telemetry policy

117. Credential Architecture

Prefer:

Application
   |
no vendor credential
   |
Collector
   |
backend credential
   |
Vendor

This keeps backend API keys out of hundreds of application workloads.

118. Vendor Migration Architecture

Because applications emit OTLP:

Apps
 |
OTel
 |
Collector
 |
Current Backend

Migration can become:

Apps
 |
OTel
 |
Collector
 |        |
 v        v
Old      New
Backend  Backend
Code language: PHP (php)

After validation:

Apps
 |
OTel
 |
Collector
 |
New Backend
Code language: PHP (php)

No major application instrumentation rewrite should be necessary.

That is one of OpenTelemetry’s most important strategic advantages.

119. Multi-Backend Export

During migration or for specialized purposes:

           +----> Backend A
           |
Collector--+
           |
           +----> Backend B

Be cautious, however.

Duplicating telemetry doubles downstream:

network volume
ingestion volume
storage cost

so use fan-out intentionally.

120. Common Anti-Patterns

Avoid these.

Anti-pattern 1

Every application exports directly to a vendor.

Problem:

credentials everywhere
vendor coupling
difficult migrations
inconsistent policy

Anti-pattern 2

100% tracing forever.

Problem:

massive telemetry volume
massive cost

Anti-pattern 3

High-cardinality metric labels.

Problem:

time-series explosion

Anti-pattern 4

Different service naming by every team.

Problem:

chaotic queries
broken dashboards

Anti-pattern 5

Sensitive customer data in spans/logs.

Problem:

security
privacy
compliance

Anti-pattern 6

No Collector self-monitoring.

Problem:

observability pipeline fails silently

Anti-pattern 7

Tail sampling behind random load balancing.

Problem:

fragmented traces
incorrect sampling decisions

Anti-pattern 8

Using latest everywhere.

Problem:

unexpected upgrades
configuration compatibility changes
hard-to-reproduce incidents

121. OpenTelemetry Production Best Practices

A strong production standard is:

Use OpenTelemetry as the application instrumentation standard.
Prefer OTLP between applications and Collectors.
Use automatic instrumentation for quick baseline coverage.
Add manual spans for important business operations.
Standardize service.name.
Standardize service.namespace.
Standardize service.version.
Use deployment.environment.name.
Follow semantic conventions.
Keep metric cardinality bounded.
Keep span names stable.
Propagate W3C Trace Context consistently.
Never put sensitive information into baggage.
Use Agent + Gateway architecture for substantial Kubernetes environments.
Keep agent processing lightweight.
Centralize expensive processing in gateways.
Use memory_limiter.
Filter/drop useless telemetry early.
Batch after filtering and enrichment.
Use queues and retries for remote exports.
Consider persistent queues for important telemetry.
Use tail sampling strategically.
Ensure trace-ID-aware routing when horizontally scaling tail-sampling gateways.
Enrich Kubernetes telemetry using k8sattributes.
Avoid duplicate Kubernetes scraping.
Protect OTLP endpoints.
Use TLS/mTLS where appropriate.
Keep vendor credentials in gateways rather than applications.
Apply least privilege.
Redact secrets and PII.
Monitor the Collector itself.
Alert on queue saturation and dropped telemetry.
Pin Collector/SDK/Operator versions.
Test upgrades in non-production first.
Treat the telemetry pipeline as production infrastructure.

122. Recommended Repository Structure

For an organization managing OTel through Infrastructure as Code:

observability/
│
├── collector/
│   │
│   ├── agent/
│   │   ├── values.yaml
│   │   └── config.yaml
│   │
│   └── gateway/
│       ├── values.yaml
│       └── config.yaml
│
├── operator/
│   ├── values.yaml
│   └── instrumentation.yaml
│
├── policies/
│   ├── attributes.md
│   ├── sampling.md
│   ├── pii.md
│   └── service-naming.md
│
├── dashboards/
│
├── alerts/
│
├── slo/
│
└── README.md

This separates:

collection
instrumentation
policy
visualization
alerting
SLOs

123. OpenTelemetry Governance

Large organizations should maintain an internal OTel specification.

Example:

OpenTelemetry Standard

1. Service Naming
2. Resource Attributes
3. Semantic Conventions
4. Instrumentation Rules
5. Metric Naming
6. Metric Cardinality
7. Span Naming
8. Sampling
9. PII Handling
10. Logging
11. Collector Architecture
12. Security
13. Backend Routing
14. Version Management
15. SLO Integration

Without governance, telemetry often turns into an expensive data lake nobody trusts.

124. Recommended Learning Path

Learn OpenTelemetry in this order:

Observability
   ↓
Traces / Spans
   ↓
Context Propagation
   ↓
Metrics
   ↓
Logs
   ↓
Resources
   ↓
Semantic Conventions
   ↓
OTLP
   ↓
SDK
   ↓
Auto Instrumentation
   ↓
Manual Instrumentation
   ↓
Collector
   ↓
Receivers / Processors / Exporters
   ↓
Kubernetes
   ↓
Agent + Gateway
   ↓
Sampling
   ↓
Security
   ↓
Scaling
   ↓
SLOs

Do not begin by memorizing hundreds of Collector configuration options.

Understand the telemetry model first.

125. OpenTelemetry Mental Model

If you remember only one architecture, remember:

                  SOURCE
                    |
                    v
            INSTRUMENTATION
                    |
                    v
                  OTLP
                    |
                    v
               COLLECTOR
                    |
            +-------+-------+
            |       |       |
         Receive  Process  Export
            |       |       |
            +-------+-------+
                    |
                    v
                  OTLP
                    |
                    v
                 BACKEND
                    |
                    v
       Dashboards / Alerts / SLOs

And for production Kubernetes:

Application
    |
OTel SDK / Auto Instrumentation
    |
    v
Agent Collector
    |
    v
Gateway Collector
    |
    v
Observability Backend

126. End-to-End Workflow

The complete telemetry lifecycle is:

1. Application receives request

2. Instrumentation creates root span

3. Trace context is propagated downstream

4. Child services create child spans

5. SDK records spans and measurements

6. Logs are correlated with trace/span context

7. SDK exports telemetry using OTLP

8. Agent Collector receives telemetry

9. Agent enriches telemetry

10. Gateway receives telemetry

11. Gateway applies:
       memory protection
       filtering
       redaction
       transformations
       sampling
       batching

12. Gateway exports using OTLP

13. Backend stores telemetry

14. Dashboard visualizes metrics

15. Alert detects SLO degradation

16. Engineer opens trace

17. Trace identifies slow component

18. Correlated logs expose error

19. Root cause is identified
Code language: JavaScript (javascript)

That is OpenTelemetry end to end.

127. Reference Architecture Summary

                                USERS
                                  |
                                  v
                        +-------------------+
                        | Load Balancer     |
                        +---------+---------+
                                  |
                                  v
+-------------------------------------------------------------+
|                     KUBERNETES                              |
|                                                             |
| +-------------+   +-------------+   +-------------+         |
| | Service A   |-->| Service B   |-->| Service C   |         |
| | OTel SDK    |   | OTel SDK    |   | OTel SDK    |         |
| +------+------+   +------+------+   +------+------+         |
|        |                 |                  |                |
|        +-----------------+------------------+                |
|                          |                                   |
|                          v                                   |
|                 +------------------+                         |
|                 | Agent Collector  |                         |
|                 | DaemonSet        |                         |
|                 +--------+---------+                         |
|                          |                                   |
|                          v                                   |
|               +----------------------+                       |
|               | Gateway Collectors   |                       |
|               |                      |                       |
|               | memory limiter       |                       |
|               | filtering            |                       |
|               | redaction            |                       |
|               | tail sampling        |                       |
|               | batching             |                       |
|               | queue/retry          |                       |
|               +----------+-----------+                       |
+--------------------------|-----------------------------------+
                           |
                           | OTLP + TLS
                           v
          +----------------+----------------+
          |                                 |
          v                                 v
 +----------------+                +----------------+
 | Trace Backend  |                | Metrics/Logs   |
 | Tempo/Jaeger   |                | Backend        |
 +-------+--------+                +-------+--------+
         |                                 |
         +----------------+----------------+
                          |
                          v
                +--------------------+
                | Grafana / Vendor UI|
                +---------+----------+
                          |
               +----------+-----------+
               |                      |
               v                      v
          Dashboards                Alerts
                                      |
                                      v
                                     SLO

128. Final Architecture Principles

OpenTelemetry works best when you think of it not as another monitoring agent but as an observability abstraction layer.

Your application should know:

"I generate telemetry."
Code language: JSON / JSON with Comments (json)

not:

"I generate Datadog telemetry."

"I generate New Relic telemetry."

"I generate Jaeger telemetry."
Code language: JSON / JSON with Comments (json)

The ideal architecture becomes:

Application
      |
      v
OpenTelemetry
      |
      v
Observability Platform

This separation provides:

Portability
Standardization
Consistent instrumentation
Vendor flexibility
Centralized governance
Security control
Cost control
Cross-signal correlation
Better troubleshooting

129. Final Production Checklist

Before calling an OpenTelemetry deployment production-ready, verify:

Instrumentation

☐ All critical services have instrumentation
☐ Automatic instrumentation validated
☐ Important business flows manually instrumented
☐ Distributed context propagation works

Resources

☐ service.name
☐ service.namespace
☐ service.version
☐ deployment.environment.name
☐ cloud metadata
☐ Kubernetes metadata
Code language: CSS (css)

Tracing

☐ Stable span names
☐ Appropriate attributes
☐ Errors represented correctly
☐ Sampling strategy documented
☐ Trace continuity tested

Metrics

☐ RED metrics available
☐ Infrastructure metrics available
☐ Cardinality reviewed
☐ Units defined
☐ Histograms used appropriately

Logs

☐ Structured logging
☐ Trace IDs correlated
☐ Sensitive fields removed
☐ Log volume controlled

Collector

☐ memory_limiter
☐ filtering
☐ enrichment
☐ batch
☐ queues
☐ retries
☐ persistent buffering where required

Kubernetes

☐ Agent Collector
☐ Gateway Collector
☐ Kubernetes attributes
☐ Kubelet metrics
☐ Host metrics
☐ Cluster receivers appropriately singleton/sharded
☐ No duplicate scraping

Security

☐ TLS
☐ authentication
☐ least privilege
☐ secret management
☐ OTLP endpoints not publicly exposed accidentally
☐ PII redaction

Reliability

☐ Multiple gateway replicas where appropriate
☐ resource requests/limits
☐ disruption protection
☐ queue monitoring
☐ exporter-failure monitoring
☐ Collector self-observability
Code language: PHP (php)

Operations

☐ Version pinning
☐ Upgrade procedure
☐ Rollback procedure
☐ Load testing
☐ Failure testing
☐ Capacity planning

Observability Outcomes

☐ Dashboards
☐ Alerts
☐ SLOs
☐ Error budgets
☐ Trace investigation workflow
☐ Log correlation
☐ Release/version correlation
Code language: JavaScript (javascript)

When those pieces are in place, OpenTelemetry stops being merely a tracing SDK and becomes the telemetry foundation of the platform.

130. One-Sentence Summary

OpenTelemetry standardizes how applications and infrastructure generate, correlate, collect, process and transport traces, metrics and logs, while the OpenTelemetry Collector provides the vendor-neutral control plane that routes this telemetry securely and reliably to the observability backend of your choice.

Rajesh Kumar

I’m Rajesh Kumar, a DevOps, SRE, DevSecOps, Cloud, and Platform Engineering expert passionate about sharing practical knowledge, real-world experiences, and industry best practices. I have worked at Cotocus and regularly write about technology, travel, investing, health, product reviews, and digital marketing through my various platforms.

I publish technical articles at DevOps School, travel stories at Holiday Landmark, stock market insights at Stocks Mantra, health and fitness guidance at My Medic Plus, product reviews at TrueReviewNow, and SEO and digital marketing strategies at Wizbrand.

Find Trusted Cardiac Hospitals

Compare heart hospitals by city and services — all in one place.

Explore Hospitals

Find the Best Cosmetic Hospitals

From Zero to Production: Architecture, Traces, Metrics, Logs, OTLP, Collector, Kubernetes, Sampling, Security, Scaling, Troubleshooting and Best Practices

1. What Is OpenTelemetry?

2. What OpenTelemetry Is NOT

3. Why OpenTelemetry Exists

4. When Should You Use OpenTelemetry?

Microservices

Kubernetes

Multi-cloud environments

Vendor-neutral observability

Complex distributed systems

5. Where OpenTelemetry Fits

6. OpenTelemetry Architecture

7. Core OpenTelemetry Components

8. The Four Main Signals

8.1 Traces

9. What Is a Span?

10. Trace ID and Span ID

11. Context Propagation

12. Baggage

13. Metrics

14. Counter Example

15. UpDownCounter

16. Gauge

17. Histogram

18. Logs

19. Profiles

20. The OpenTelemetry API

21. The OpenTelemetry SDK

22. Automatic vs Manual Instrumentation

Automatic instrumentation

Manual instrumentation

23. Best Instrumentation Strategy

24. Zero-Code Instrumentation

25. Semantic Conventions

26. Resources

27. Recommended Resource Model

28. OTLP — OpenTelemetry Protocol

29. OTLP/gRPC vs OTLP/HTTP

OTLP/gRPC

OTLP/HTTP

30. Important OpenTelemetry Environment Variables

Service name

Resource attributes

OTLP endpoint

Protocol

Trace-specific endpoint

Metrics endpoint

Logs endpoint

Authentication headers

31. OpenTelemetry Collector

32. Collector Architecture

33. Receivers

34. Processors

35. Exporters

36. Extensions

37. Connectors

38. Basic Collector Configuration

39. Install OpenTelemetry Collector with Docker

40. Linux Installation

41. Your First Instrumented Python Application

42. Manual Instrumentation

43. Custom Metrics

44. Metric Cardinality — Critical Production Topic

45. Span Cardinality

46. Logs + Traces Correlation

47. Collector Deployment Patterns

Pattern 1 — Agent

48. Pattern 2 — Gateway

49. Pattern 3 — Agent + Gateway

50. Production Kubernetes Architecture

51. Install Collector on Kubernetes with Helm

52. Kubernetes Data Collection

53. Kubernetes Attributes Processor

54. Kubernetes Node Metrics

55. Host Metrics

56. Cluster Metrics

57. Kubernetes Events

58. OpenTelemetry Operator

59. OpenTelemetryCollector Resource