{"id":78584,"date":"2026-09-14T16:17:34","date_gmt":"2026-09-14T16:17:34","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=78584"},"modified":"2026-09-14T16:17:42","modified_gmt":"2026-09-14T16:17:42","slug":"opentelemetry-complete-end-to-end-tutorial","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/opentelemetry-complete-end-to-end-tutorial\/","title":{"rendered":"OpenTelemetry \u2014 Complete End-to-End Tutorial"},"content":{"rendered":"\n

From Zero to Production: Architecture, Traces, Metrics, Logs, OTLP, Collector, Kubernetes, Sampling, Security, Scaling, Troubleshooting and Best Practices<\/h2>\n\n\n\n

Technology:<\/strong> OpenTelemetry \/ OTel
Level:<\/strong> Beginner \u2192 Intermediate \u2192 Advanced \u2192 Production
Last verified:<\/strong> September 15, 2026<\/p>\n\n\n\n

\n\n\n\n

1. What Is OpenTelemetry?<\/h1>\n\n\n\n

OpenTelemetry<\/strong>, usually abbreviated OTel<\/strong>, is an open-source, vendor-neutral observability framework used to:<\/p>\n\n\n\n

    \n
  • instrument applications<\/li>\n\n\n\n
  • generate telemetry<\/li>\n\n\n\n
  • collect telemetry<\/li>\n\n\n\n
  • process telemetry<\/li>\n\n\n\n
  • correlate telemetry<\/li>\n\n\n\n
  • export telemetry<\/li>\n<\/ul>\n\n\n\n

    The principal telemetry signals are:<\/p>\n\n\n\n

      \n
    • Traces<\/strong><\/li>\n\n\n\n
    • Metrics<\/strong><\/li>\n\n\n\n
    • Logs<\/strong><\/li>\n\n\n\n
    • Baggage<\/strong><\/li>\n<\/ul>\n\n\n\n

      Profiles are also being developed as an OpenTelemetry signal. In the current OTLP specification, traces, metrics, and logs are stable while profiles remain under development.<\/p>\n\n\n\n

      OpenTelemetry is a CNCF project<\/strong> created from the merger of OpenTracing and OpenCensus. It provides a common standard so applications do not need proprietary instrumentation for every monitoring vendor.<\/p>\n\n\n\n

      A useful one-line definition is:<\/p>\n\n\n\n

      \n

      OpenTelemetry is the standard instrumentation and telemetry pipeline layer between your applications\/infrastructure and your observability backends.<\/strong><\/p>\n<\/blockquote>\n\n\n\n

      \n\n\n\n

      2. What OpenTelemetry Is NOT<\/h1>\n\n\n\n

      One of the most important things to understand is that OpenTelemetry is generally not your observability database or dashboard system<\/strong>.<\/p>\n\n\n\n

      OpenTelemetry does not replace tools such as:<\/p>\n\n\n\n

        \n
      • Grafana<\/li>\n\n\n\n
      • Prometheus<\/li>\n\n\n\n
      • Jaeger<\/li>\n\n\n\n
      • Tempo<\/li>\n\n\n\n
      • Loki<\/li>\n\n\n\n
      • Datadog<\/li>\n\n\n\n
      • New Relic<\/li>\n\n\n\n
      • Dynatrace<\/li>\n\n\n\n
      • Elastic<\/li>\n\n\n\n
      • Splunk<\/li>\n<\/ul>\n\n\n\n

        Instead, OpenTelemetry sends telemetry to<\/strong> systems like these.<\/p>\n\n\n\n

        The OpenTelemetry project intentionally leaves backend storage and visualization to other systems.<\/p>\n\n\n\n

        Conceptually:<\/p>\n\n\n

        Application\n     |\n     | Telemetry\n     v\nOpenTelemetry\n     |\n     | OTLP\n     v\nObservability Backend\n     |\n     v\nStorage + Queries + Dashboards + Alerts\n<\/code><\/span><\/pre>\n\n\n
        \n\n\n\n
        3. Why OpenTelemetry Exists<\/h1>\n\n\n\n
        Before OpenTelemetry, a typical organization might install:<\/p>\n\n\n
        Datadog Agent\nNew<\/span> Relic Agent\nJaeger Client\nPrometheus Client\nZipkin Client\nFluent Bit\nVendor A SDK\nVendor B SDK\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
        Application code became tightly coupled to monitoring products.<\/p>\n\n\n\n
        Changing observability vendors could require:<\/p>\n\n\n\n
          \n
        • replacing SDKs<\/li>\n\n\n\n
        • changing instrumentation<\/li>\n\n\n\n
        • modifying application code<\/li>\n\n\n\n
        • rebuilding services<\/li>\n\n\n\n
        • changing agents<\/li>\n\n\n\n
        • maintaining several telemetry formats<\/li>\n<\/ul>\n\n\n\n
          OpenTelemetry changes the model to:<\/p>\n\n\n
          Applications\n      |\n      v\nOpenTelemetry SDK \/ Auto Instrumentation\n      |\n      v\nOpenTelemetry Collector\n      |\n      +--------> Backend A\n      |\n      +--------> Backend B\n      |\n      +--------> Backend C\n<\/code><\/span><\/pre>\n\n\n
          The application speaks OpenTelemetry<\/strong> rather than a vendor-specific telemetry language.<\/p>\n\n\n\n
          OpenTelemetry’s two major principles are that you retain ownership of your telemetry and can use a common set of APIs and conventions rather than learning a new instrumentation model for every backend.<\/p>\n\n\n\n
          \n\n\n\n
          4. When Should You Use OpenTelemetry?<\/h1>\n\n\n\n
          OpenTelemetry is particularly valuable when running:<\/p>\n\n\n\n
          Microservices<\/h3>\n\n\n\n
          Example:<\/p>\n\n\n
          Browser\n   |\nFrontend\n   |\nAPI Gateway\n   |\nOrders Service\n   |\nPayment Service\n   |\nKafka\n   |\nNotification Service\n<\/code><\/span><\/pre>\n\n\n
          Distributed tracing lets you see one request across the entire system.<\/p>\n\n\n\n
          \n\n\n\n
          Kubernetes<\/h3>\n\n\n\n
          OTel can observe:<\/p>\n\n\n
          Cluster\nNodes\nPods\nContainers\nApplications\nHTTP traffic\ngRPC traffic\nDatabases\nMessage queues\nKubernetes events\nInfrastructure metrics\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Multi-cloud environments<\/h3>\n\n\n\n
          For example:<\/p>\n\n\n
          AWS\nGCP\nAzure\nOn-premises\n<\/code><\/span><\/pre>\n\n\n
          All can emit a standardized telemetry format.<\/p>\n\n\n\n
          \n\n\n\n
          Vendor-neutral observability<\/h3>\n\n\n\n
          You might begin with:<\/p>\n\n\n
          OTel \u2192 Jaeger\n<\/code><\/span><\/pre>\n\n\n
          and later move to:<\/p>\n\n\n
          OTel \u2192 Grafana Tempo\n<\/code><\/span><\/pre>\n\n\n
          or:<\/p>\n\n\n
          OTel \u2192 Datadog\n<\/code><\/span><\/pre>\n\n\n
          without redesigning your application instrumentation.<\/p>\n\n\n\n
          \n\n\n\n
          Complex distributed systems<\/h3>\n\n\n\n
          OTel is extremely useful when you need answers to questions such as:<\/p>\n\n\n
          Why is checkout slow?\n\nWhich service generated this<\/span> error?\n\nWhich database call caused the latency?\n\nWhich Kafka consumer processed this<\/span> request?\n\nWhich Kubernetes pod served the request?\n\nWhich version introduced the regression?\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          \n\n\n\n
          5. Where OpenTelemetry Fits<\/h1>\n\n\n\n
          A modern observability stack typically has four layers.<\/p>\n\n\n
          +-----------------------------------------------------+\n|                 Applications                        |\n| Java | Go | Python | Node | .NET | Rust | etc.     |\n+----------------------------+------------------------+\n                             |\n                             v\n+-----------------------------------------------------+\n|             OpenTelemetry Instrumentation           |\n| APIs | SDKs | Auto Instrumentation | Libraries      |\n+----------------------------+------------------------+\n                             |\n                             | OTLP\n                             v\n+-----------------------------------------------------+\n|              OpenTelemetry Collector                |\n|                                                     |\n| Receivers \u2192 Processors \u2192 Exporters                  |\n+----------------------------+------------------------+\n                             |\n                             v\n+-----------------------------------------------------+\n|              Observability Backends                 |\n| Tempo | Prometheus | Datadog | Jaeger | Elastic     |\n+----------------------------+------------------------+\n                             |\n                             v\n+-----------------------------------------------------+\n|        Dashboards | Search | Alerts | SLOs          |\n+-----------------------------------------------------+\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          6. OpenTelemetry Architecture<\/h1>\n\n\n\n
          A more complete architecture looks like this:<\/p>\n\n\n\n
                                   APPLICATIONS\n                              |\n          +-------------------+-------------------+\n          |                   |                   |\n          v                   v                   v\n       Service A           Service B           Service C\n       OTel SDK            OTel SDK            OTel SDK\n          |                   |                   |\n          +-------------------+-------------------+\n                              |\n                              | OTLP\n                              v\n                    +--------------------+\n                    | OTel Agent         |\n                    | Collector          |\n                    +--------------------+\n                              |\n                              v\n                    +--------------------+\n                    | OTel Gateway       |\n                    | Collector          |\n                    +--------------------+\n                              |\n            +-----------------+------------------+\n            |                 |                  |\n            v                 v                  v\n          Traces            Metrics            Logs\n            |                 |                  |\n            v                 v                  v\n          Tempo           Prometheus           Loki\n          Jaeger           Datadog             Elastic\n          Datadog          etc.                etc.\n<\/code><\/pre>\n\n\n\n
          The Collector itself follows a pipeline model:<\/p>\n\n\n
          Receiver\n   |\nProcessor\n   |\nProcessor\n   |\nExporter\n<\/code><\/span><\/pre>\n\n\n
          The Collector can create separate pipelines for traces, metrics and logs.<\/p>\n\n\n\n
          \n\n\n\n
          7. Core OpenTelemetry Components<\/h1>\n\n\n\n
          OpenTelemetry consists of several major pieces.<\/p>\n\n\n\n
          Component<\/th>Purpose<\/th><\/tr><\/thead>
          Specification<\/td>Defines expected OpenTelemetry behavior<\/td><\/tr>
          API<\/td>Application-facing instrumentation interfaces<\/td><\/tr>
          SDK<\/td>Implements telemetry generation and processing<\/td><\/tr>
          Instrumentation Libraries<\/td>Instrument common frameworks<\/td><\/tr>
          Auto Instrumentation<\/td>Adds telemetry with minimal\/no code changes<\/td><\/tr>
          Semantic Conventions<\/td>Standard attribute names<\/td><\/tr>
          OTLP<\/td>OpenTelemetry wire protocol<\/td><\/tr>
          Collector<\/td>Receives, processes and exports telemetry<\/td><\/tr>
          Operator<\/td>Manages OTel on Kubernetes<\/td><\/tr>
          Helm Charts<\/td>Deploy OTel components on Kubernetes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
          These are the primary components described by the OpenTelemetry project.<\/p>\n\n\n\n
          \n\n\n\n
          8. The Four Main Signals<\/h1>\n\n\n\n
          8.1 Traces<\/h2>\n\n\n\n
          A trace represents the lifecycle of a request across a distributed system.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          Trace ID: abc123\n\nFrontend\n\u2514\u2500\u2500 GET \/checkout                 320 ms\n    |\n    \u251c\u2500\u2500 Cart Service              40 ms\n    |\n    \u251c\u2500\u2500 Inventory Service         70 ms\n    |\n    \u2514\u2500\u2500 Payment Service          180 ms\n        |\n        \u2514\u2500\u2500 PostgreSQL query      90 ms\n<\/code><\/span><\/pre>\n\n\n
          You can immediately see:<\/p>\n\n\n
          Payment Service = slow\nDatabase query = major contributor\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          9. What Is a Span?<\/h1>\n\n\n\n
          A span<\/strong> represents a single operation.<\/p>\n\n\n\n
          A span may contain:<\/p>\n\n\n
          Span Name\nTrace ID\nSpan ID\nParent<\/span> Span ID\nStart Time\nEnd Time\nDuration\nAttributes\nEvents\nLinks\nStatus\nSpan Kind\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          A trace is composed of multiple related spans.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          Trace\n\u2502\n\u251c\u2500\u2500 Span: HTTP GET \/checkout\n\u2502\n\u251c\u2500\u2500 Span: cart.get\n\u2502\n\u251c\u2500\u2500 Span: inventory.check\n\u2502\n\u251c\u2500\u2500 Span: payment.authorize\n\u2502\n\u2514\u2500\u2500 Span: SQL SELECT\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          10. Trace ID and Span ID<\/h1>\n\n\n\n
          Each trace has a unique:<\/p>\n\n\n
          trace_id\n<\/code><\/span><\/pre>\n\n\n
          Each span has its own:<\/p>\n\n\n
          span_id\n<\/code><\/span><\/pre>\n\n\n
          Example:<\/p>\n\n\n
          trace_id = 4bf92f3577b34da6a3ce929d0e0e4736\n\nfrontend span:\nspan_id = 00f067aa0ba902b7\n\npayment span:\nspan_id = 12ab67aa0ba90019\n<\/code><\/span><\/pre>\n\n\n
          The common trace ID allows an observability backend to reconstruct the complete distributed request.<\/p>\n\n\n\n
          \n\n\n\n
          11. Context Propagation<\/h1>\n\n\n\n
          Imagine:<\/p>\n\n\n
          Service A \u2192 Service B \u2192 Service C\n<\/code><\/span><\/pre>\n\n\n
          How does Service C know it belongs to the same trace?<\/p>\n\n\n\n
          Through context propagation<\/strong>.<\/p>\n\n\n\n
          Service A sends tracing context to Service B.<\/p>\n\n\n\n
          Service B sends it to Service C.<\/p>\n\n\n\n
          OpenTelemetry normally uses the W3C Trace Context format by default.<\/p>\n\n\n\n
          For HTTP you may see a header such as:<\/p>\n\n\n
          traceparent:\n00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01\n<\/code><\/span><\/pre>\n\n\n
          That essentially identifies:<\/p>\n\n\n
          trace\nparent<\/span> span\ntrace flags\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          \n\n\n\n
          12. Baggage<\/h1>\n\n\n\n
          Baggage allows contextual key\/value data to travel with requests.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          customer.tier = gold\nregion = japan\ntenant.id = tenant-42\n<\/code><\/span><\/pre>\n\n\n
          Architecture:<\/p>\n\n\n
          Frontend\n   |\n   | baggage\n   v\nOrders\n   |\n   | baggage\n   v\nPayments\n<\/code><\/span><\/pre>\n\n\n
          All services can access that context.<\/p>\n\n\n\n
          However, baggage crosses process\/service boundaries.<\/p>\n\n\n\n
          Therefore never put credentials, secrets or sensitive PII into baggage<\/strong>. OpenTelemetry explicitly warns against this.<\/p>\n\n\n\n
          \n\n\n\n
          13. Metrics<\/h1>\n\n\n\n
          Metrics represent numerical measurements over time.<\/p>\n\n\n\n
          Examples:<\/p>\n\n\n
          requests_total\nrequest_duration\ncpu_usage\nmemory_usage\nqueue_depth\nactive_sessions\norders_total\npayment_failures\n<\/code><\/span><\/pre>\n\n\n
          Major OpenTelemetry metric instrument types include:<\/p>\n\n\n\n
          Instrument<\/th>Typical use<\/th><\/tr><\/thead>
          Counter<\/td>Number of requests<\/td><\/tr>
          UpDownCounter<\/td>Active requests<\/td><\/tr>
          Gauge<\/td>Current temperature\/value<\/td><\/tr>
          Histogram<\/td>Request latency distribution<\/td><\/tr>
          Async Counter<\/td>Externally observed cumulative value<\/td><\/tr>
          Async UpDownCounter<\/td>Externally observed variable total<\/td><\/tr>
          Async Gauge<\/td>Externally observed current value<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
          OpenTelemetry metrics also support aggregation and delta\/cumulative temporality.<\/p>\n\n\n\n
          \n\n\n\n
          14. Counter Example<\/h1>\n\n\n
          orders<\/span>.created<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          Values:<\/p>\n\n\n
          1\n2\n3\n4\n5\n<\/code><\/span><\/pre>\n\n\n
          A counter should only increase.<\/p>\n\n\n\n
          Typical uses:<\/p>\n\n\n
          HTTP requests\nOrders\nErrors\nRetries\nJobs completed\nMessages processed\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          15. UpDownCounter<\/h1>\n\n\n\n
          Unlike a Counter, an UpDownCounter can increase or decrease.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          active_requests\n<\/code><\/span><\/pre>\n\n\n
          Request starts:<\/p>\n\n\n
          +1\n<\/code><\/span><\/pre>\n\n\n
          Request finishes:<\/p>\n\n\n
          -1\n<\/code><\/span><\/pre>\n\n\n
          Useful for:<\/p>\n\n\n
          queue depth\nactive connections\nactive sessions\nworkers\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          16. Gauge<\/h1>\n\n\n\n
          A Gauge records a current value.<\/p>\n\n\n\n
          Examples:<\/p>\n\n\n
          CPU temperature\nmemory usage\nqueue size\nbattery percentage\n<\/code><\/span><\/pre>\n\n\n
          Conceptually:<\/p>\n\n\n
          10\n18\n14\n22\n16\n<\/code><\/span><\/pre>\n\n\n
          The measurement does not need to be monotonic.<\/p>\n\n\n\n
          \n\n\n\n
          17. Histogram<\/h1>\n\n\n\n
          Histograms are particularly important for latency.<\/p>\n\n\n\n
          Example observations:<\/p>\n\n\n
          10 ms\n20 ms\n30 ms\n70 ms\n400 ms\n800 ms\n<\/code><\/span><\/pre>\n\n\n
          Instead of storing every measurement independently, the SDK\/backend can aggregate them.<\/p>\n\n\n\n
          You can then derive information such as:<\/p>\n\n\n
          P50\nP90\nP95\nP99\nrequest count\nsum\ndistribution\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          18. Logs<\/h1>\n\n\n\n
          Logs are timestamped event records.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          2026-09-15T00:11:18Z ERROR Payment failed order_id=123\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry supports both structured and unstructured logs, although structured logs are preferable. OTel can also correlate logs with active traces by attaching trace and span identifiers.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          {\n  \"timestamp\"<\/span>: \"2026-09-15T00:11:18Z\"<\/span>,\n  \"severity\"<\/span>: \"ERROR\"<\/span>,\n  \"message\"<\/span>: \"Payment failed\"<\/span>,\n  \"order.id\"<\/span>: \"123\"<\/span>,\n  \"trace_id\"<\/span>: \"abc123\"<\/span>,\n  \"span_id\"<\/span>: \"def456\"<\/span>\n}\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
          You can go from:<\/p>\n\n\n
          Error<\/span> log\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          directly to:<\/p>\n\n\n
          Distributed trace\n<\/code><\/span><\/pre>\n\n\n
          which is enormously useful for troubleshooting.<\/p>\n\n\n\n
          \n\n\n\n
          19. Profiles<\/h1>\n\n\n\n
          Profiles show code-level resource consumption.<\/p>\n\n\n\n
          For example:<\/p>\n\n\n
          Which function<\/span> consumes<\/span> CPU<\/span>?\n\nWhere<\/span> is<\/span> memory<\/span> being<\/span> allocated<\/span>?\n\nWhich<\/span> code<\/span> path<\/span> is<\/span> expensive<\/span>?\n<\/span><\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Conceptually:<\/p>\n\n\n
          Application\n    |\n    +--- traces\n    +--- metrics\n    +--- logs\n    +--- profiles\n<\/code><\/span><\/pre>\n\n\n
          Profiles are a newer part of the OpenTelemetry ecosystem and remain under development in OTLP rather than being as mature as traces, metrics and logs.<\/p>\n\n\n\n
          \n\n\n\n
          20. The OpenTelemetry API<\/h1>\n\n\n\n
          The API is what instrumentation code interacts with.<\/p>\n\n\n\n
          For tracing:<\/p>\n\n\n
          Tracer\nSpan\nContext\n<\/code><\/span><\/pre>\n\n\n
          For metrics:<\/p>\n\n\n
          Meter\nCounter\nHistogram\nGauge\n<\/code><\/span><\/pre>\n\n\n
          For application libraries, the usual recommendation is:<\/p>\n\n\n
          Library\n   |\n   v\nOTel API\n<\/code><\/span><\/pre>\n\n\n
          rather than forcing an SDK configuration onto applications consuming that library.<\/p>\n\n\n\n
          Standalone applications typically use both:<\/p>\n\n\n
          OTel API\n+\nOTel SDK\n<\/code><\/span><\/pre>\n\n\n
          This distinction is explicitly part of OpenTelemetry’s recommended instrumentation model.<\/p>\n\n\n\n
          \n\n\n\n
          21. The OpenTelemetry SDK<\/h1>\n\n\n\n
          The SDK implements the API and provides functionality such as:<\/p>\n\n\n
          Sampling\nSpan processing\nMetric aggregation\nExporting\nResources\nConfiguration\n<\/code><\/span><\/pre>\n\n\n
          Architecture:<\/p>\n\n\n
          Application code\n      |\n      v\nOTel API\n      |\n      v\nOTel SDK\n      |\n      v\nOTLP Exporter\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          22. Automatic vs Manual Instrumentation<\/h1>\n\n\n\n
          There are two main instrumentation approaches.<\/p>\n\n\n\n
          Automatic instrumentation<\/h2>\n\n\n\n
          Automatically instruments frameworks such as:<\/p>\n\n\n
          HTTP servers\nHTTP clients\ngRPC\nDatabase clients\nMessaging libraries\nPopular frameworks\n<\/code><\/span><\/pre>\n\n\n
          Very little application code needs modification.<\/p>\n\n\n\n
          \n\n\n\n
          Manual instrumentation<\/h2>\n\n\n\n
          Developers explicitly create spans, metrics and attributes.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          checkout<\/span>.validate_cart<\/span>\npayment<\/span>.authorize<\/span>\norder<\/span>.persist<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          Manual instrumentation provides business-level visibility that automatic instrumentation cannot infer.<\/p>\n\n\n\n
          \n\n\n\n
          23. Best Instrumentation Strategy<\/h1>\n\n\n\n
          Usually the best strategy is:<\/p>\n\n\n
          Automatic instrumentation\n        +\nManual business instrumentation\n<\/code><\/span><\/pre>\n\n\n
          Automatic instrumentation gives you:<\/p>\n\n\n
          HTTP\ndatabase\ngRPC\nframework\nnetwork\nmessaging\n<\/code><\/span><\/pre>\n\n\n
          Manual instrumentation adds:<\/p>\n\n\n
          business operations\nimportant workflows\ndomain attributes\ncustom metrics\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry explicitly supports using both approaches together.<\/p>\n\n\n\n
          \n\n\n\n
          24. Zero-Code Instrumentation<\/h1>\n\n\n\n
          OpenTelemetry currently provides zero-code approaches for major languages including:<\/p>\n\n\n
          Java\n.NET\nPython\nJavaScript \/ Node.js\nGo\nPHP\n<\/code><\/span><\/pre>\n\n\n
          Mechanisms vary by language and can include:<\/p>\n\n\n
          Java agents\nbytecode instrumentation\nmonkey patching\nruntime hooks\neBPF\n<\/code><\/span><\/pre>\n\n\n
          Zero-code instrumentation usually observes libraries around the edges of an application; important application-specific business operations may still benefit from manual instrumentation.<\/p>\n\n\n\n
          \n\n\n\n
          25. Semantic Conventions<\/h1>\n\n\n\n
          Imagine one service reports:<\/p>\n\n\n
          http<\/span>.method<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          another:<\/p>\n\n\n
          HTTPMethod\n<\/code><\/span><\/pre>\n\n\n
          another:<\/p>\n\n\n
          request_method\n<\/code><\/span><\/pre>\n\n\n
          Querying the data becomes painful.<\/p>\n\n\n\n
          OpenTelemetry solves this with Semantic Conventions<\/strong>.<\/p>\n\n\n\n
          They standardize naming for telemetry across:<\/p>\n\n\n
          HTTP\nRPC\ndatabases\nmessaging\ncloud\ncontainers\nKubernetes\nservices\nhosts\nnetworks\netc.\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry semantic conventions apply across traces, metrics, logs, profiles and resources.<\/p>\n\n\n\n
          Examples include:<\/p>\n\n\n
          service<\/span>.name<\/span>\nservice<\/span>.version<\/span>\nservice<\/span>.namespace<\/span>\n\nhttp<\/span>.request<\/span>.method<\/span>\n\nserver<\/span>.address<\/span>\nserver<\/span>.port<\/span>\n\ndeployment<\/span>.environment<\/span>.name<\/span>\n\nk8s<\/span>.cluster<\/span>.name<\/span>\nk8s<\/span>.namespace<\/span>.name<\/span>\nk8s<\/span>.pod<\/span>.name<\/span>\nk8s<\/span>.container<\/span>.name<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          \n\n\n\n
          26. Resources<\/h1>\n\n\n\n
          A Resource<\/strong> describes the entity producing telemetry.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          service.name = checkout\nservice.version = 2.7<\/span>.1<\/span>\n\ndeployment.environment.name = production\n\ncloud.provider = aws\ncloud.region = ap-northeast-1<\/span>\n\nk8s.cluster.name = production\nk8s.namespace<\/span>.name<\/span> = commerce<\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          Resources answer:<\/p>\n\n\n\n
          \n
          “Where did this telemetry originate?”<\/p>\n<\/blockquote>\n\n\n\n
          OpenTelemetry recommends setting service.name<\/code> explicitly; otherwise SDKs can fall back to an unknown_service<\/code> value. OTEL_SERVICE_NAME<\/code> is the standard environment variable for configuring it.<\/p>\n\n\n\n
          \n\n\n\n
          27. Recommended Resource Model<\/h1>\n\n\n\n
          A good baseline is:<\/p>\n\n\n
          service<\/span>.namespace<\/span>\nservice<\/span>.name<\/span>\nservice<\/span>.version<\/span>\nservice<\/span>.instance<\/span>.id<\/span>\n\ndeployment<\/span>.environment<\/span>.name<\/span>\n\ncloud<\/span>.provider<\/span>\ncloud<\/span>.region<\/span>\n\nk8s<\/span>.cluster<\/span>.name<\/span>\nk8s<\/span>.namespace<\/span>.name<\/span>\nk8s<\/span>.deployment<\/span>.name<\/span>\nk8s<\/span>.pod<\/span>.name<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          For example:<\/p>\n\n\n
          service.namespace = ecommerce\nservice.name = checkout\nservice.version = 4.3.2\n\ndeployment.environment.name = production\n\ncloud.provider = aws\ncloud.region = ap-northeast-1\n\nk8s.cluster.name = prod-eks\n<\/code><\/span><\/pre>\n\n\n
          deployment.environment.name<\/code> is the current semantic convention; the older deployment.environment<\/code> attribute is deprecated.<\/p>\n\n\n\n
          \n\n\n\n
          28. OTLP \u2014 OpenTelemetry Protocol<\/h1>\n\n\n\n
          OTLP is the native OpenTelemetry telemetry transport protocol.<\/p>\n\n\n\n
          It supports:<\/p>\n\n\n
          Traces\nMetrics\nLogs\nProfiles\n<\/code><\/span><\/pre>\n\n\n
          Two important transports are:<\/p>\n\n\n
          OTLP\/gRPC\nOTLP\/HTTP\n<\/code><\/span><\/pre>\n\n\n
          The default ports are:<\/p>\n\n\n\n
          Protocol<\/th>Port<\/th><\/tr><\/thead>
          OTLP\/gRPC<\/td>4317<\/td><\/tr>
          OTLP\/HTTP<\/td>4318<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
          For OTLP\/HTTP the conventional endpoints include:<\/p>\n\n\n
          \/v1\/traces\n\/v1\/metrics\n\/v1\/logs\n<\/code><\/span><\/pre>\n\n\n
          The current OTLP specification defines Protocol Buffers for the data and supports gRPC and HTTP transports.<\/p>\n\n\n\n
          \n\n\n\n
          29. OTLP\/gRPC vs OTLP\/HTTP<\/h1>\n\n\n\n
          OTLP\/gRPC<\/h2>\n\n\n\n
          Common endpoint:<\/p>\n\n\n
          otel-collector<\/span>:4317<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          Advantages:<\/p>\n\n\n
          Efficient protobuf transport\nHTTP\/2\nwidely used service-to-collector protocol\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          OTLP\/HTTP<\/h2>\n\n\n\n
          Common endpoint:<\/p>\n\n\n
          http:\/\/otel-collector:4318<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Signal URLs generally become:<\/p>\n\n\n
          \/v1\/traces\n\/v1\/metrics\n\/v1\/logs\n<\/code><\/span><\/pre>\n\n\n
          Advantages include simpler compatibility with environments where HTTP infrastructure is easier to operate.<\/p>\n\n\n\n
          The current generic exporter specification prefers http\/protobuf<\/code> as the default where backward-compatibility constraints do not require something else, although individual SDK defaults can vary.<\/p>\n\n\n\n
          \n\n\n\n
          30. Important OpenTelemetry Environment Variables<\/h1>\n\n\n\n
          These are worth memorizing.<\/p>\n\n\n\n
          Service name<\/h3>\n\n\n
          OTEL_SERVICE_NAME=checkout-service\n<\/code><\/span><\/pre>\n\n\n
          Resource attributes<\/h3>\n\n\n
          OTEL_RESOURCE_ATTRIBUTES=\"deployment.environment.name=production,service.version=1.4.3\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          OTLP endpoint<\/h3>\n\n\n
          OTEL_EXPORTER_OTLP_ENDPOINT=http:\/\/otel-collector:4318<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Protocol<\/h3>\n\n\n
          OTEL_EXPORTER_OTLP_PROTOCOL=http\/protobuf\n<\/code><\/span><\/pre>\n\n\n
          or:<\/p>\n\n\n
          OTEL_EXPORTER_OTLP_PROTOCOL=grpc\n<\/code><\/span><\/pre>\n\n\n
          Trace-specific endpoint<\/h3>\n\n\n
          OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http:\/\/otel-collector:4318\/v1\/traces<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Metrics endpoint<\/h3>\n\n\n
          OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http:\/\/otel-collector:4318\/v1\/metrics<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Logs endpoint<\/h3>\n\n\n
          OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http:\/\/otel-collector:4318\/v1\/logs<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Authentication headers<\/h3>\n\n\n
          OTEL_EXPORTER_OTLP_HEADERS=\"authorization=Bearer TOKEN\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Exporter endpoint, protocol, certificate and header variables are standardized in the OpenTelemetry exporter specification.<\/p>\n\n\n\n
          \n\n\n\n
          31. OpenTelemetry Collector<\/h1>\n\n\n\n
          The Collector<\/strong> is one of the most important OTel components.<\/p>\n\n\n\n
          Its job is to:<\/p>\n\n\n
          Receive telemetry\nProcess telemetry\nTransform telemetry\nFilter telemetry\nSample telemetry\nBatch telemetry\nRoute telemetry\nExport telemetry\n<\/code><\/span><\/pre>\n\n\n
          It is vendor-neutral and can receive and emit multiple telemetry formats.<\/p>\n\n\n\n
          \n\n\n\n
          32. Collector Architecture<\/h1>\n\n\n\n
          The fundamental Collector model is:<\/p>\n\n\n
          RECEIVER\n   |\n   v\nPROCESSOR\n   |\n   v\nEXPORTER\n<\/code><\/span><\/pre>\n\n\n
          More realistically:<\/p>\n\n\n\n
                       +----------+\nApplication -> Receiver |\n             +----+-----+\n                  |\n                  v\n            +-----------+\n            | Processor |\n            +-----+-----+\n                  |\n                  v\n            +-----------+\n            | Processor |\n            +-----+-----+\n                  |\n                  v\n            +-----------+\n            | Exporter  |\n            +-----+-----+\n                  |\n                  v\n              Backend\n<\/code><\/pre>\n\n\n\n
          \n\n\n\n
          33. Receivers<\/h1>\n\n\n\n
          Receivers ingest telemetry.<\/p>\n\n\n\n
          Examples can include:<\/p>\n\n\n
          OTLP\nPrometheus\nJaeger\nZipkin\nKafka\nFile logs\nHost metrics\nKubernetes metrics\nCloud provider sources\n<\/code><\/span><\/pre>\n\n\n
          Example:<\/p>\n\n\n
          receivers<\/span>:\n  otlp<\/span>:\n    protocols<\/span>:\n      grpc<\/span>:\n        endpoint<\/span>: 0.0<\/span>.0<\/span>.0<\/span>:4317<\/span>\n      http<\/span>:\n        endpoint<\/span>: 0.0<\/span>.0<\/span>.0<\/span>:4318<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          \n\n\n\n
          34. Processors<\/h1>\n\n\n\n
          Processors modify or manage telemetry.<\/p>\n\n\n\n
          Common functions include:<\/p>\n\n\n
          Batching\nFiltering\nSampling\nResource detection\nAttribute manipulation\nKubernetes metadata enrichment\nMemory protection\nTransformation\nRedaction\n<\/code><\/span><\/pre>\n\n\n
          A particularly important ordering pattern is:<\/p>\n\n\n
          memory_limiter\n      |\ndrop \/ sampling processors\n      |\ncontext-dependent processors\n      |\nenrichment \/ transformation\n      |\nbatch\n<\/code><\/span><\/pre>\n\n\n
          Processor ordering matters because processing occurs sequentially. OpenTelemetry recommends the memory limiter early, dropping unnecessary telemetry before expensive processing, and batching after filtering\/enrichment.<\/p>\n\n\n\n
          \n\n\n\n
          35. Exporters<\/h1>\n\n\n\n
          Exporters send telemetry onward.<\/p>\n\n\n\n
          For example:<\/p>\n\n\n
          OTLP backend\nPrometheus-compatible backend\nJaeger-compatible backend\nKafka\nVendor backend\nDebug output\n<\/code><\/span><\/pre>\n\n\n
          A single Collector can also fan out telemetry to several destinations.<\/p>\n\n\n\n
                       +--> Backend A\nCollector ---+\n             +--> Backend B\n<\/code><\/pre>\n\n\n\n
          Each exporter receives a copy of the relevant data from the pipeline.<\/p>\n\n\n\n
          \n\n\n\n
          36. Extensions<\/h1>\n\n\n\n
          Extensions provide additional Collector capabilities outside normal telemetry pipelines.<\/p>\n\n\n\n
          Examples include:<\/p>\n\n\n
          Authentication\nHealth checks\nService discovery\nStorage\nDebugging\n<\/code><\/span><\/pre>\n\n\n
          The available extensions depend on the Collector distribution being used.<\/p>\n\n\n\n
          \n\n\n\n
          37. Connectors<\/h1>\n\n\n\n
          Connectors connect one Collector pipeline to another and act as both an exporter and a receiver.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          Trace pipeline\n      |\n      v\nSpan Metrics Connector\n      |\n      v\nMetrics pipeline\n<\/code><\/span><\/pre>\n\n\n
          This can generate metrics derived from spans.<\/p>\n\n\n\n
          Current Collector distributions include connectors for use cases such as service graphs, span metrics, routing and signal-to-metrics conversion.<\/p>\n\n\n\n
          \n\n\n\n
          38. Basic Collector Configuration<\/h1>\n\n\n\n
          Create:<\/p>\n\n\n
          otel-config<\/span>.yaml<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          Example:<\/p>\n\n\n
          receivers<\/span>:\n  otlp<\/span>:\n    protocols<\/span>:\n      grpc<\/span>:\n        endpoint<\/span>: 0.0<\/span>.0<\/span>.0<\/span>:4317<\/span>\n      http<\/span>:\n        endpoint<\/span>: 0.0<\/span>.0<\/span>.0<\/span>:4318<\/span>\n\nprocessors<\/span>:\n  memory_limiter<\/span>:\n    check_interval<\/span>: 1s<\/span>\n    limit_percentage<\/span>: 75\n    spike_limit_percentage<\/span>: 15\n\n  batch<\/span>:\n    timeout<\/span>: 5s<\/span>\n\nexporters<\/span>:\n  debug<\/span>:\n    verbosity<\/span>: detailed<\/span>\n\nservice<\/span>:\n  pipelines<\/span>:\n\n    traces<\/span>:\n      receivers<\/span>: [otlp]<\/span>\n      processors<\/span>: [memory_limiter, batch]<\/span>\n      exporters<\/span>: [debug]<\/span>\n\n    metrics<\/span>:\n      receivers<\/span>: [otlp]<\/span>\n      processors<\/span>: [memory_limiter, batch]<\/span>\n      exporters<\/span>: [debug]<\/span>\n\n    logs<\/span>:\n      receivers<\/span>: [otlp]<\/span>\n      processors<\/span>: [memory_limiter, batch]<\/span>\n      exporters<\/span>: [debug]<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          This creates:<\/p>\n\n\n
          OTLP\n |\n +--> traces  -> memory_limiter -> batch -> debug\n |\n +--> metrics -> memory_limiter -> batch -> debug\n |\n +--> logs    -> memory_limiter -> batch -> debug\n<\/code><\/span><\/pre>\n\n\n
          The official Python exporter documentation uses essentially this OTLP receiver + debug exporter arrangement to validate an exporter locally.<\/p>\n\n\n\n
          \n\n\n\n
          39. Install OpenTelemetry Collector with Docker<\/h1>\n\n\n\n
          The current official documentation examples use Collector 0.160.0<\/code>.<\/p>\n\n\n
          docker pull otel\/opentelemetry-collector:0.160.0\n<\/code><\/span><\/pre>\n\n\n
          Run:<\/p>\n\n\n
          docker run \\\n  --rm \\\n  -p 4317<\/span>:4317<\/span> \\\n  -p 4318<\/span>:4318<\/span> \\\n  -v \"$(pwd)\/otel-config.yaml:\/etc\/otelcol\/config.yaml\"<\/span> \\\n  otel\/opentelemetry-collector:0.160<\/span>.0<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          The Collector now accepts:<\/p>\n\n\n
          OTLP\/gRPC -> localhost:4317\nOTLP\/HTTP -> localhost:4318\n<\/code><\/span><\/pre>\n\n\n
          Official Docker installation documentation supports mounting \/etc\/otelcol\/config.yaml<\/code> in this manner.<\/p>\n\n\n\n
          \n\n\n\n
          40. Linux Installation<\/h1>\n\n\n\n
          OpenTelemetry publishes:<\/p>\n\n\n
          DEB\nRPM\nLinux binaries\n<\/code><\/span><\/pre>\n\n\n
          for multiple architectures.<\/p>\n\n\n\n
          For example, the current documented DEB workflow uses:<\/p>\n\n\n
          wget https:\/\/github.com\/open-telemetry\/opentelemetry-collector-releases\/releases\/download\/v0.160.0\/otelcol_0.160.0_linux_amd64.deb<\/span>\n\nsudo dpkg -i otelcol_0.160<\/span>.0<\/span>_linux_amd64.deb\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          The default Linux package configuration is:<\/p>\n\n\n
          \/etc\/otelcol\/config.yaml\n<\/code><\/span><\/pre>\n\n\n
          and packaged installations can run via systemd<\/code>.<\/p>\n\n\n\n
          \n\n\n\n
          41. Your First Instrumented Python Application<\/h1>\n\n\n\n
          Install:<\/p>\n\n\n
          python -m venv .venv\n\nsource .venv\/bin\/activate\n<\/code><\/span><\/pre>\n\n\n
          Then:<\/p>\n\n\n
          pip install flask requests\npip install opentelemetry-distro\npip install opentelemetry-exporter-otlp\n<\/code><\/span><\/pre>\n\n\n
          Install compatible instrumentation:<\/p>\n\n\n
          opentelemetry-bootstrap -a install\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry’s Python zero-code flow uses opentelemetry-distro<\/code>, the OTLP exporter and opentelemetry-bootstrap<\/code> to detect installed libraries and install matching instrumentation packages.<\/p>\n\n\n\n
          Create:<\/p>\n\n\n
          app<\/span>.py<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n
          from<\/span> flask import<\/span> Flask\nimport<\/span> requests\n\napp = Flask(__name__)\n\n@app.route(\"\/\"<\/span>)\ndef home():\n    requests.get(\"https:\/\/example.com\"<\/span>)\n    return<\/span> \"Hello OpenTelemetry!\"<\/span>\n\nif<\/span> __name__ == \"__main__\"<\/span>:\n    app.run(port=5000<\/span>)\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Configure OpenTelemetry:<\/p>\n\n\n
          export<\/span> OTEL_SERVICE_NAME=demo-python-service\n\nexport<\/span> OTEL_RESOURCE_ATTRIBUTES=\"deployment.environment.name=development\"<\/span>\n\nexport<\/span> OTEL_EXPORTER_OTLP_ENDPOINT=http:\/\/localhost:4318<\/span>\n\nexport<\/span> OTEL_EXPORTER_OTLP_PROTOCOL=http\/protobuf\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Run with auto instrumentation:<\/p>\n\n\n
          opentelemetry-instrument<\/span> python<\/span> app<\/span>.py<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          Request:<\/p>\n\n\n
          curl http:\/\/localhost:5000<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          The flow becomes:<\/p>\n\n\n
          curl\n |\n v\nFlask\n |\n | auto-generated spans\n v\nOpenTelemetry Python\n |\n | OTLP\/HTTP\n v\nCollector\n |\n v\ndebug exporter\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          42. Manual Instrumentation<\/h1>\n\n\n\n
          Automatic instrumentation is only the beginning.<\/p>\n\n\n\n
          Suppose checkout processing contains:<\/p>\n\n\n
          Validate Cart\nReserve Inventory\nAuthorize Payment\nCreate Order\n<\/code><\/span><\/pre>\n\n\n
          Those operations are business concepts.<\/p>\n\n\n\n
          You can create spans manually.<\/p>\n\n\n\n
          Example Python:<\/p>\n\n\n
          from<\/span> opentelemetry import<\/span> trace\n\ntracer = trace.get_tracer(\"checkout-service\"<\/span>)\n\ndef checkout(order):\n    with<\/span> tracer.start_as_current_span(\"checkout.process\"<\/span>) as<\/span> span:\n\n        span.set_attribute(\"order.type\"<\/span>, order[\"type\"<\/span>])\n\n        validate_cart(order)\n\n        with<\/span> tracer.start_as_current_span(\"inventory.reserve\"<\/span>):\n            reserve_inventory(order)\n\n        with<\/span> tracer.start_as_current_span(\"payment.authorize\"<\/span>):\n            authorize_payment(order)\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          The resulting trace becomes:<\/p>\n\n\n
          HTTP POST \/checkout\n\u2502\n\u2514\u2500\u2500 checkout.process\n    \u2502\n    \u251c\u2500\u2500 inventory.reserve\n    \u2502\n    \u2514\u2500\u2500 payment.authorize\n<\/code><\/span><\/pre>\n\n\n
          This is considerably more useful than seeing only generic HTTP calls.<\/p>\n\n\n\n
          \n\n\n\n
          43. Custom Metrics<\/h1>\n\n\n\n
          Conceptually:<\/p>\n\n\n
          from<\/span> opentelemetry import<\/span> metrics\n\nmeter = metrics.get_meter(\"checkout-service\"<\/span>)\n\norders = meter.create_counter(\n    \"orders.created\"<\/span>,\n    description=\"Number of successfully created orders\"<\/span>\n)\n\norders.add(\n    1<\/span>,\n    {\n        \"payment.method\"<\/span>: \"credit_card\"<\/span>\n    }\n)\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          You could then analyze:<\/p>\n\n\n
          orders<\/span>.created<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          by safe, bounded dimensions such as payment method.<\/p>\n\n\n\n
          \n\n\n\n
          44. Metric Cardinality \u2014 Critical Production Topic<\/h1>\n\n\n\n
          A metric such as:<\/p>\n\n\n
          http<\/span>.requests<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          with:<\/p>\n\n\n
          http.method = GET\n<\/code><\/span><\/pre>\n\n\n
          has low cardinality.<\/p>\n\n\n\n
          But adding:<\/p>\n\n\n
          user.id = 183729101\nsession.id = abc987123\nrequest.id = randomUUID\n<\/code><\/span><\/pre>\n\n\n
          can create millions of unique time series.<\/p>\n\n\n\n
          This causes:<\/p>\n\n\n
          Huge memory consumption\nHuge observability bills\nSlow queries\nBackend pressure\nCollector pressure\n<\/code><\/span><\/pre>\n\n\n
          Therefore avoid unbounded dimensions in metrics.<\/p>\n\n\n\n
          Bad:<\/p>\n\n\n
          order<\/span>.id<\/span>\nuser<\/span>.id<\/span>\nsession<\/span>.id<\/span>\nrequest<\/span>.id<\/span>\nfull<\/span> URL<\/span>\nrandom<\/span> UUID<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          Better:<\/p>\n\n\n
          http<\/span>.request<\/span>.method<\/span>\nhttp<\/span>.response<\/span>.status_code<\/span>\nservice<\/span>.name<\/span>\ndeployment<\/span>.environment<\/span>.name<\/span>\nregion<\/span>\noperation<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          \n\n\n\n
          45. Span Cardinality<\/h1>\n\n\n\n
          Span attributes tolerate more cardinality than metrics, but you still should not create dynamically unique span names<\/strong>.<\/p>\n\n\n\n
          Bad:<\/p>\n\n\n
          GET \/users\/12345\nGET \/users\/89431\nGET \/users\/82173\n<\/code><\/span><\/pre>\n\n\n
          Better:<\/p>\n\n\n
          GET \/users\/{id}\n<\/code><\/span><\/pre>\n\n\n
          Then store the relevant normalized information as appropriate attributes.<\/p>\n\n\n\n
          Stable span naming dramatically improves grouping and analysis.<\/p>\n\n\n\n
          \n\n\n\n
          46. Logs + Traces Correlation<\/h1>\n\n\n\n
          Suppose your application logs:<\/p>\n\n\n
          Payment failed\n<\/code><\/span><\/pre>\n\n\n
          Without correlation, you search manually.<\/p>\n\n\n\n
          With OTel:<\/p>\n\n\n
          {\n  \"message\"<\/span>: \"Payment failed\"<\/span>,\n  \"trace_id\"<\/span>: \"8fabc...\"<\/span>,\n  \"span_id\"<\/span>: \"2b73...\"<\/span>\n}\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
          You can navigate:<\/p>\n\n\n
          Log\n |\n v\nTrace\n |\n v\nPayment Span\n |\n v\nDatabase Span\n<\/code><\/span><\/pre>\n\n\n
          This correlation is one of the strongest arguments for unifying telemetry through OpenTelemetry.<\/p>\n\n\n\n
          \n\n\n\n
          47. Collector Deployment Patterns<\/h1>\n\n\n\n
          There are three especially important patterns.<\/p>\n\n\n\n
          Pattern 1 \u2014 Agent<\/h2>\n\n\n
          App\n |\n v\nLocal Collector\n |\n v\nBackend\n<\/code><\/span><\/pre>\n\n\n
          The Collector runs close to the application, for example:<\/p>\n\n\n
          VM process\nKubernetes DaemonSet\nSidecar\n<\/code><\/span><\/pre>\n\n\n
          The official documentation calls this the agent deployment pattern<\/strong>.<\/p>\n\n\n\n
          \n\n\n\n
          48. Pattern 2 \u2014 Gateway<\/h1>\n\n\n
          App A --+\n        |\nApp B --+--> Gateway Collector --> Backend\n        |\nApp C --+\n<\/code><\/span><\/pre>\n\n\n
          Advantages:<\/p>\n\n\n
          central configuration\ncentral credentials\ncentral routing\ncentral filtering\ncentral sampling\ncentral policy\n<\/code><\/span><\/pre>\n\n\n
          A gateway generally exposes a shared OTLP endpoint per cluster, data center or region.<\/p>\n\n\n\n
          \n\n\n\n
          49. Pattern 3 \u2014 Agent + Gateway<\/h1>\n\n\n\n
          For medium and large environments, this is usually the strongest architecture:<\/p>\n\n\n\n
                           Node 1\nApp A ---> Agent Collector\n                    |\n                 Node 2\nApp B ---> Agent Collector\n                    |\n                 Node 3\nApp C ---> Agent Collector\n                    |\n                    v\n             Gateway Collectors\n                    |\n                    v\n             Observability Backend\n<\/code><\/pre>\n\n\n\n
          Agents perform lightweight\/local operations.<\/p>\n\n\n\n
          Gateways perform:<\/p>\n\n\n
          sampling\nfiltering\nbatching\nrouting\nauthentication\ncredential management\nbackend fan-out\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry’s current deployment guidance highlights separation of concerns, scalable processing and stable egress\/security as major advantages of the combined architecture.<\/p>\n\n\n\n
          \n\n\n\n
          50. Production Kubernetes Architecture<\/h1>\n\n\n\n
          A mature Kubernetes deployment can look like this:<\/p>\n\n\n
          +---------------------------------------------------------+\n| Kubernetes Cluster                                      |\n|                                                         |\n| Node A                       Node B                      |\n| ----------------           ----------------             |\n| Application Pods            Application Pods            |\n|      |                           |                      |\n|      v                           v                      |\n| OTel Agent DS              OTel Agent DS                |\n|      |                           |                      |\n|      +------------+--------------+                      |\n|                   |                                     |\n|                   v                                     |\n|          OTel Gateway Service                           |\n|                   |                                     |\n|          +--------+--------+                            |\n|          |        |        |                            |\n|          v        v        v                            |\n|      Gateway  Gateway  Gateway                          |\n+----------+--------+--------+----------------------------+\n           |\n           | TLS + OTLP\n           v\n+---------------------------------------------------------+\n| Observability Platform                                  |\n| Traces | Metrics | Logs                                 |\n+---------------------------------------------------------+\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          51. Install Collector on Kubernetes with Helm<\/h1>\n\n\n\n
          Add the official repository:<\/p>\n\n\n
          helm repo add open-telemetry \\\n  https<\/span>:\/\/open-telemetry.github.io\/opentelemetry-helm-charts<\/span>\n\nhelm repo update\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Install as DaemonSet:<\/p>\n\n\n
          helm install otel-agent \\\n  open-telemetry\/opentelemetry-collector \\\n  --set<\/span> image.repository=otel\/opentelemetry-collector-k8s \\\n  --set<\/span> mode=daemonset\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Or gateway:<\/p>\n\n\n
          helm install otel-gateway \\\n  open-telemetry\/opentelemetry-collector \\\n  --set<\/span> image.repository=otel\/opentelemetry-collector-k8s \\\n  --set<\/span> mode=deployment\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          The official chart supports:<\/p>\n\n\n
          daemonset\ndeployment\nstatefulset\n<\/code><\/span><\/pre>\n\n\n
          modes.<\/p>\n\n\n\n
          \n\n\n\n
          52. Kubernetes Data Collection<\/h1>\n\n\n\n
          Useful Collector components include:<\/p>\n\n\n
          k8sattributes\nkubeletstats\nhostmetrics\nk8scluster\nk8sobjects\nfilelog\nprometheus\n<\/code><\/span><\/pre>\n\n\n
          Conceptually:<\/p>\n\n\n\n
                           Kubernetes\n                     |\n       +-------------+-------------+\n       |             |             |\n    Pod logs      Kubelet       API Server\n       |             |             |\n       v             v             v\n    filelog     kubeletstats   k8scluster\n       |             |             |\n       +-------------+-------------+\n                     |\n                     v\n                  Collector\n<\/code><\/pre>\n\n\n\n
          \n\n\n\n
          53. Kubernetes Attributes Processor<\/h1>\n\n\n\n
          One of the most valuable processors is:<\/p>\n\n\n
          k8sattributes\n<\/code><\/span><\/pre>\n\n\n
          It can enrich telemetry with metadata such as:<\/p>\n\n\n
          k8s<\/span>.cluster<\/span>.name<\/span>\nk8s<\/span>.namespace<\/span>.name<\/span>\nk8s<\/span>.pod<\/span>.name<\/span>\nk8s<\/span>.node<\/span>.name<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          The official Helm chart strongly recommends enabling the Kubernetes attributes preset or manually configuring this processor where appropriate.<\/p>\n\n\n\n
          Example Helm values:<\/p>\n\n\n
          mode<\/span>: daemonset\n\npresets:\n  kubernetesAttributes:\n    enabled: true\n<\/code><\/span>Code language:<\/span> HTTP<\/span> (<\/span>http<\/span>)<\/span><\/small><\/pre>\n\n\n
          \n\n\n\n
          54. Kubernetes Node Metrics<\/h1>\n\n\n\n
          For node\/pod\/container metrics:<\/p>\n\n\n
          presets:\n  kubeletMetrics:\n    enabled: true<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          DaemonSet mode is generally preferred because the Kubelet Stats receiver observes the node where that Collector instance is running.<\/p>\n\n\n\n
          \n\n\n\n
          55. Host Metrics<\/h1>\n\n\n\n
          Example:<\/p>\n\n\n
          presets:\n  hostMetrics:\n    enabled: true<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          This can provide:<\/p>\n\n\n
          CPU\nload\nmemory\ndisk\nfilesystem\nnetwork\n<\/code><\/span><\/pre>\n\n\n
          The Helm chart documents this as another DaemonSet-oriented use case.<\/p>\n\n\n\n
          \n\n\n\n
          56. Cluster Metrics<\/h1>\n\n\n\n
          Cluster-wide metrics are different.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          mode<\/span>: deployment\n\nreplicaCount: 1\n\npresets:\n  clusterMetrics:\n    enabled: true\n<\/code><\/span>Code language:<\/span> HTTP<\/span> (<\/span>http<\/span>)<\/span><\/small><\/pre>\n\n\n
          Why one replica?<\/p>\n\n\n\n
          Because several identical collectors gathering the same cluster-wide metrics can generate duplicate telemetry.<\/p>\n\n\n\n
          The official Helm documentation specifically warns about duplicate data for cluster-level receivers.<\/p>\n\n\n\n
          \n\n\n\n
          57. Kubernetes Events<\/h1>\n\n\n\n
          You can also collect Kubernetes events:<\/p>\n\n\n
          presets:\n  kubernetesEvents:\n    enabled: true<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          This allows operational events to join the rest of your telemetry pipeline.<\/p>\n\n\n\n
          \n\n\n\n
          58. OpenTelemetry Operator<\/h1>\n\n\n\n
          The OpenTelemetry Operator manages:<\/p>\n\n\n
          OpenTelemetry Collectors\nAutomatic instrumentation\n<\/code><\/span><\/pre>\n\n\n
          It can inject instrumentation into workloads rather than requiring every team to configure it independently.<\/p>\n\n\n\n
          Install the Operator, for example, using the official release manifest after satisfying its prerequisites:<\/p>\n\n\n
          kubectl apply -f \\\nhttps<\/span>:\/\/github.com\/open-telemetry\/opentelemetry-operator\/releases\/latest\/download\/opentelemetry-operator.yaml<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          For production, pin an explicitly tested Operator version rather than relying permanently on latest<\/code>.<\/p>\n\n\n\n
          \n\n\n\n
          59. OpenTelemetryCollector Resource<\/h1>\n\n\n\n
          Example:<\/p>\n\n\n
          apiVersion<\/span>: opentelemetry.io\/v1beta1\nkind<\/span>: OpenTelemetryCollector\n\nmetadata:\n  name: otel\n\nspec:\n  config:\n\n    receivers:\n      otlp:\n        protocols:\n          grpc:\n            endpoint: 0.0.0.0:4317\n          http:\n            endpoint: 0.0.0.0:4318\n\n    processors:\n      memory_limiter:\n        check_interval: 1s\n        limit_percentage: 75\n        spike_limit_percentage: 15\n\n      batch: {}\n\n    exporters:\n      debug: {}\n\n    service:\n      pipelines:\n\n        traces:\n          receivers: [otlp]\n          processors: [memory_limiter, batch]\n          exporters: [debug]\n<\/code><\/span>Code language:<\/span> HTTP<\/span> (<\/span>http<\/span>)<\/span><\/small><\/pre>\n\n\n
          The current Operator documentation uses the opentelemetry.io\/v1beta1<\/code> Collector CRD in its getting-started example.<\/p>\n\n\n\n
          \n\n\n\n
          60. Kubernetes Automatic Instrumentation<\/h1>\n\n\n\n
          An Instrumentation resource can define how applications export telemetry.<\/p>\n\n\n\n
          Example Python-oriented configuration:<\/p>\n\n\n
          apiVersion: opentelemetry.io\/v1alpha1\nkind<\/span>: Instrumentation\n\nmetadata<\/span>:\n  name: default<\/span>-instrumentation\n\nspec<\/span>:\n\n  exporter:\n    endpoint: http:\/\/otel-collector:4318<\/span>\n\n  propagators:\n    - tracecontext\n    - baggage\n\n  sampler<\/span>:\n    type: parentbased_traceidratio\n    argument<\/span>: \"1\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Then annotate a Deployment:<\/p>\n\n\n
          metadata:\n  annotations:\n    instrumentation.opentelemetry.io\/inject-python: \"true\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Supported injection annotations include language-specific options for Java, Node.js, Python, .NET and Go.<\/p>\n\n\n\n
          \n\n\n\n
          61. Sampling<\/h1>\n\n\n\n
          Tracing every request can become expensive.<\/p>\n\n\n\n
          Suppose:<\/p>\n\n\n
          50,000 requests \/ second\n<\/code><\/span><\/pre>\n\n\n
          If every request creates:<\/p>\n\n\n
          20 spans\n<\/code><\/span><\/pre>\n\n\n
          then:<\/p>\n\n\n
          1,000,000 spans \/ second\n<\/code><\/span><\/pre>\n\n\n
          may be produced.<\/p>\n\n\n\n
          Sampling controls telemetry volume.<\/p>\n\n\n\n
          There are two important approaches:<\/p>\n\n\n
          Head Sampling\nTail Sampling\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          62. Head Sampling<\/h1>\n\n\n\n
          The decision is made early.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          Keep 10%\nDrop 90%\n<\/code><\/span><\/pre>\n\n\n
          Flow:<\/p>\n\n\n
          Request\n   |\nSampling decision\n   |\n   +-- Keep \u2192 create\/export<\/span> trace\n   |\n   +-- Drop\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Advantages:<\/p>\n\n\n
          simple\nfast\ncheap\nscalable\n<\/code><\/span><\/pre>\n\n\n
          Disadvantage:<\/p>\n\n\n\n
          The system doesn’t yet know whether the request will eventually:<\/p>\n\n\n
          fail\nbecome slow\ncontain an interesting anomaly\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry describes consistent probability sampling as a common head-sampling strategy.<\/p>\n\n\n\n
          \n\n\n\n
          63. Tail Sampling<\/h1>\n\n\n\n
          Tail sampling decides after observing much or all of the trace<\/strong>.<\/p>\n\n\n\n
          For example:<\/p>\n\n\n
          IF<\/span> error\n    keep 100<\/span>%\n\nIF<\/span> latency > 2<\/span> seconds\n    keep 100<\/span>%\n\nIF<\/span> VIP workflow\n    keep 100<\/span>%\n\nELSE<\/span>\n    keep 5<\/span>%\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          Advantages:<\/p>\n\n\n
          keep failures\nkeep slow requests\nkeep unusual transactions\nreduce normal trace volume\n<\/code><\/span><\/pre>\n\n\n
          This is considerably smarter than purely random sampling.<\/p>\n\n\n\n
          \n\n\n\n
          64. Tail Sampling Architecture<\/h1>\n\n\n\n
          Tail sampling requires all spans belonging to a given trace to reach the same sampling Collector.<\/p>\n\n\n\n
          Bad:<\/p>\n\n\n
          Trace abc\n\nSpan 1 -> Gateway A\nSpan 2 -> Gateway B\nSpan 3 -> Gateway C\n<\/code><\/span><\/pre>\n\n\n
          No gateway sees the whole trace.<\/p>\n\n\n\n
          Correct architecture:<\/p>\n\n\n
          Trace abc\n   |\nTrace-ID aware routing\n   |\n   v\nGateway B\n\nSpan 1 -> B\nSpan 2 -> B\nSpan 3 -> B\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry recommends trace-ID-aware load balancing for multi-gateway tail sampling and warns that this is an advanced architecture requiring careful testing.<\/p>\n\n\n\n
          \n\n\n\n
          65. Recommended Sampling Model<\/h1>\n\n\n\n
          A mature system frequently uses:<\/p>\n\n\n
          Application\n    |\n    | Parent<\/span>-based head sampling\n    v\nAgent\n    |\n    v\nTrace-ID router\n    |\n    v\nTail Sampling Gateway\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          Example policies:<\/p>\n\n\n
          100% errors\n100% very slow requests\n100% critical transactions\n100% selected release\/canary traffic\nsmall percentage of healthy traffic\n<\/code><\/span><\/pre>\n\n\n
          Avoid sampling so aggressively that normal baseline behavior becomes impossible to understand.<\/p>\n\n\n\n
          \n\n\n\n
          66. Filtering<\/h1>\n\n\n\n
          Not all telemetry deserves to reach storage.<\/p>\n\n\n\n
          Possible noise:<\/p>\n\n\n
          \/health\n\/readiness\n\/liveness\ninternal probes\nlow-value debug logs\nknown crawler traffic\n<\/code><\/span><\/pre>\n\n\n
          Filtering these centrally can significantly reduce cost.<\/p>\n\n\n\n
          Conceptual pipeline:<\/p>\n\n\n
          Receiver\n   |\nMemory limiter\n   |\nFilter\n   |\nTransform\n   |\nBatch\n   |\nExporter\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          67. Attribute Redaction<\/h1>\n\n\n\n
          Telemetry can accidentally contain:<\/p>\n\n\n
          Authorization headers\nAPI keys\ntokens\npasswords\nemail addresses\nuser IDs\ncredit-card information\ndatabase query parameters\n<\/code><\/span><\/pre>\n\n\n
          Therefore implement controls such as:<\/p>\n\n\n
          Allow-list<\/span> attributes\nDelete sensitive attributes\nHash selected identifiers\nMask values\nDrop sensitive records\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          A good rule:<\/p>\n\n\n\n
          \n
          Treat observability pipelines as production data systems, not harmless debugging utilities.<\/p>\n<\/blockquote>\n\n\n\n
          \n\n\n\n
          68. Production Collector Configuration Model<\/h1>\n\n\n\n
          A typical architecture looks more like:<\/p>\n\n\n
          receivers:\n  otlp:\n    protocols:\n      grpc:\n        endpoint: 0.0.0.0:4317\n      http:\n        endpoint: 0.0.0.0:4318\n\nprocessors:\n\n  memory_limiter:\n    check_interval: 1s\n    limit_percentage: 75\n    spike_limit_percentage: 15\n\n  batch:\n    timeout: 5s\n\nexporters:\n\n  otlp\/backend:\n    endpoint: ${env:OTEL_BACKEND_ENDPOINT}\n\nservice:\n\n  pipelines:\n\n    traces:\n      receivers: [otlp]\n      processors:\n        - memory_limiter\n        - batch\n      exporters:\n        - otlp\/backend\n\n    metrics:\n      receivers: [otlp]\n      processors:\n        - memory_limiter\n        - batch\n      exporters:\n        - otlp\/backend\n\n    logs:\n      receivers: [otlp]\n      processors:\n        - memory_limiter\n        - batch\n      exporters:\n        - otlp\/backend\n<\/code><\/span><\/pre>\n\n\n
          Credentials should come from your secrets mechanism rather than being hard-coded into Git.<\/p>\n\n\n\n
          \n\n\n\n
          69. Collector Resilience<\/h1>\n\n\n\n
          Imagine:<\/p>\n\n\n
          Collector \u2192 Backend\n<\/code><\/span><\/pre>\n\n\n
          and the backend becomes unavailable.<\/p>\n\n\n\n
          Without resilience:<\/p>\n\n\n
          telemetry \u2192 DROP\n<\/code><\/span><\/pre>\n\n\n
          With a queue:<\/p>\n\n\n
          telemetry \u2192 queue \u2192 retry \u2192 backend\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry Collector exporters support retry and sending-queue mechanisms. Current Collector documentation describes exponential backoff and queue-based protection from temporary backend failures.<\/p>\n\n\n\n
          \n\n\n\n
          70. Persistent Queue \/ WAL<\/h1>\n\n\n\n
          An in-memory queue disappears if the Collector crashes.<\/p>\n\n\n\n
          A more resilient design uses persistent storage:<\/p>\n\n\n
          Collector\n   |\n   v\nDisk-backed queue \/ WAL\n   |\n   v\nBackend\n<\/code><\/span><\/pre>\n\n\n
          The file_storage<\/code> extension can provide persistent queue storage.<\/p>\n\n\n\n
          Conceptual configuration:<\/p>\n\n\n
          extensions:\n  file_storage:\n    directory: \/var\/<\/span>lib\/otelcol\/storage\n\nexporters<\/span>:\n  otlp\/backend:\n    endpoint: backend:4317<\/span>\n\n    sending_queue<\/span>:\n      storage: file_storage\n\nservice<\/span>:\n  extensions:\n    - file_storage\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          This lets queued telemetry survive Collector restarts, subject to disk capacity and other failure modes.<\/p>\n\n\n\n
          \n\n\n\n
          71. Extreme Resilience \u2014 Kafka<\/h1>\n\n\n\n
          For extremely critical telemetry:<\/p>\n\n\n
          Apps\n |\nCollectors\n |\nKafka\n |\nGateway Collectors\n |\nObservability Backend\n<\/code><\/span><\/pre>\n\n\n
          Kafka introduces:<\/p>\n\n\n
          durability\ndecoupling\nbackpressure isolation\nreplay possibilities\n<\/code><\/span><\/pre>\n\n\n
          But it also adds significant operational complexity.<\/p>\n\n\n\n
          Do not automatically add Kafka merely because it sounds more enterprise-grade.<\/p>\n\n\n\n
          Use it when telemetry durability requirements justify another distributed system.<\/p>\n\n\n\n
          \n\n\n\n
          72. Memory Limiter<\/h1>\n\n\n\n
          The Collector must protect itself during spikes.<\/p>\n\n\n\n
          Without protection:<\/p>\n\n\n
          Traffic spike\n   |\nHuge telemetry volume\n   |\nCollector memory growth\n   |\nOOMKilled\n<\/code><\/span><\/pre>\n\n\n
          With memory_limiter<\/code>:<\/p>\n\n\n
          Traffic spike\n   |\nMemory threshold\n   |\nBackpressure \/ refused telemetry\n   |\nCollector remains alive\n<\/code><\/span><\/pre>\n\n\n
          Monitor refusal metrics, because frequent refusal means capacity or downstream throughput is inadequate.<\/p>\n\n\n\n
          \n\n\n\n
          73. Batch Processor<\/h1>\n\n\n\n
          Instead of:<\/p>\n\n\n
          Span -> network\nSpan -> network\nSpan -> network\nSpan -> network\n<\/code><\/span><\/pre>\n\n\n
          batching creates:<\/p>\n\n\n
          Span\nSpan\nSpan\nSpan\n  |\nBatch\n  |\nNetwork\n<\/code><\/span><\/pre>\n\n\n
          Benefits include:<\/p>\n\n\n
          fewer requests\nbetter compression\nlower networking overhead\nbetter export<\/span> efficiency\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          OpenTelemetry recommends batching after processors that may drop telemetry.<\/p>\n\n\n\n
          \n\n\n\n
          74. Collector Scaling<\/h1>\n\n\n\n
          Important Collector metrics include:<\/p>\n\n\n
          otelcol_exporter_queue_size\notelcol_exporter_queue_capacity\n\notelcol_exporter_send_failed_spans\n\notelcol_processor_refused_spans\n<\/code><\/span><\/pre>\n\n\n
          and equivalent signal-specific metrics.<\/p>\n\n\n\n
          OpenTelemetry’s scaling guidance suggests paying close attention as exporter queues approach roughly 60\u201370% of capacity<\/strong>, while also determining whether the actual bottleneck is the Collector, network or backend.<\/p>\n\n\n\n
          \n\n\n\n
          75. Don’t Blindly Scale Collectors<\/h1>\n\n\n\n
          Suppose:<\/p>\n\n\n
          Collector queue = full\n<\/code><\/span><\/pre>\n\n\n
          It is tempting to add more Collectors.<\/p>\n\n\n\n
          But perhaps:<\/p>\n\n\n
          Backend is overloaded\n<\/code><\/span><\/pre>\n\n\n
          Then:<\/p>\n\n\n
          More Collectors\n    |\n    v\nMore pressure\n    |\n    v\nBackend gets worse\n<\/code><\/span><\/pre>\n\n\n
          Always determine:<\/p>\n\n\n
          Collector bottleneck?\nNetwork bottleneck?\nBackend bottleneck?\n<\/code><\/span><\/pre>\n\n\n
          before scaling.<\/p>\n\n\n\n
          This exact distinction is emphasized in OpenTelemetry’s Collector scaling guidance.<\/p>\n\n\n\n
          \n\n\n\n
          76. Collector Self-Observability<\/h1>\n\n\n\n
          Your observability pipeline itself needs observability.<\/p>\n\n\n\n
          Monitor:<\/p>\n\n\n
          Collector CPU\nCollector memory\nreceived telemetry\naccepted telemetry\ndropped telemetry\nexport<\/span> failures\nqueue utilization\nexport<\/span> latency\nprocessor refusals\nrestarts\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Alert on conditions such as:<\/p>\n\n\n
          export<\/span> failures > threshold\n\nqueue utilization > threshold\n\nmemory near limit\n\nCollector unavailable\n\ntelemetry refusal > 0<\/span> for<\/span> sustained period\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Otherwise your monitoring system can silently fail while telling you everything is healthy.<\/p>\n\n\n\n
          \n\n\n\n
          77. Security Architecture<\/h1>\n\n\n\n
          Production communication should ideally look like:<\/p>\n\n\n
          Application\n   |\nmTLS \/ trusted cluster network\n   |\nAgent\n   |\nTLS\n   |\nGateway\n   |\nTLS + Authentication\n   |\nBackend\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry’s security guidance recommends encryption and authentication for Collector communications and TLS or mTLS in production as appropriate.<\/p>\n\n\n\n
          \n\n\n\n
          78. Collector Security Best Practices<\/h1>\n\n\n\n
          Follow these principles:<\/p>\n\n\n\n
          Do not hard-code secrets<\/h3>\n\n\n\n
          Avoid:<\/p>\n\n\n
          headers:\n  api-key: super<\/span>-secret-value\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Prefer:<\/p>\n\n\n
          Secret Manager\nKubernetes Secret\nVault\nEnvironment variable\nworkload identity where supported\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Use TLS<\/h3>\n\n\n\n
          Avoid exposing unencrypted OTLP across untrusted networks.<\/p>\n\n\n\n
          \n\n\n\n
          Consider mTLS<\/h3>\n\n\n\n
          Useful where client authentication at the transport layer is required.<\/p>\n\n\n\n
          \n\n\n\n
          Run least privilege<\/h3>\n\n\n\n
          The Collector should normally not run as root.<\/p>\n\n\n\n
          Only grant privileged mounts or capabilities when a receiver genuinely requires them.<\/p>\n\n\n\n
          \n\n\n\n
          Restrict receivers<\/h3>\n\n\n\n
          Do not accidentally expose:<\/p>\n\n\n
          0.0<\/span>.0<\/span>.0<\/span>:4317<\/span>\n0.0<\/span>.0<\/span>.0<\/span>:4318<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          to the public internet without intentional authentication and network controls.<\/p>\n\n\n\n
          \n\n\n\n
          Minimize components<\/h3>\n\n\n\n
          Only enable the Collector components you actually need.<\/p>\n\n\n\n
          OpenTelemetry specifically recommends reducing the component set to minimize attack surface and mentions the OpenTelemetry Collector Builder as an option for producing a constrained distribution.<\/p>\n\n\n\n
          \n\n\n\n
          79. Sensitive Data Governance<\/h1>\n\n\n\n
          Create a telemetry data classification policy.<\/p>\n\n\n\n
          For example:<\/p>\n\n\n\n
          Category<\/th>Allowed?<\/th><\/tr><\/thead>
          Service name<\/td>Yes<\/td><\/tr>
          Environment<\/td>Yes<\/td><\/tr>
          Region<\/td>Yes<\/td><\/tr>
          HTTP method<\/td>Yes<\/td><\/tr>
          Status code<\/td>Yes<\/td><\/tr>
          Order category<\/td>Yes<\/td><\/tr>
          Password<\/td>Never<\/td><\/tr>
          Authorization token<\/td>Never<\/td><\/tr>
          API key<\/td>Never<\/td><\/tr>
          Credit card number<\/td>Never<\/td><\/tr>
          Sensitive PII<\/td>Usually no<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
          Filtering should occur as close to ingestion as practical.<\/p>\n\n\n\n
          \n\n\n\n
          80. OpenTelemetry vs Prometheus<\/h1>\n\n\n\n
          They are complementary.<\/p>\n\n\n\n
          Prometheus is primarily:<\/p>\n\n\n
          metrics collection\/storage\/querying\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry provides:<\/p>\n\n\n
          instrumentation\ntraces\nmetrics\nlogs\ncontext propagation\ntelemetry processing\nexport<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Example architecture:<\/p>\n\n\n
          App\n |\nOTel\n |\nCollector\n |\nPrometheus\n |\nGrafana\n<\/code><\/span><\/pre>\n\n\n
          Do not think:<\/p>\n\n\n
          OTel OR Prometheus\n<\/code><\/span><\/pre>\n\n\n
          Think:<\/p>\n\n\n
          OTel + Prometheus\n<\/code><\/span><\/pre>\n\n\n
          where appropriate.<\/p>\n\n\n\n
          \n\n\n\n
          81. OpenTelemetry vs Jaeger<\/h1>\n\n\n\n
          Jaeger primarily focuses on distributed tracing storage\/query\/visualization.<\/p>\n\n\n\n
          OTel provides instrumentation and transport.<\/p>\n\n\n\n
          Architecture:<\/p>\n\n\n
          Applications\n |\nOTel\n |\nCollector\n |\nJaeger\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          82. OpenTelemetry vs Datadog\/New Relic\/Dynatrace<\/h1>\n\n\n\n
          Commercial platforms provide:<\/p>\n\n\n
          backend storage\nanalytics\ndashboards\nalerts\nAPM UI\nSLOs\nincident tooling\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry can provide a vendor-neutral telemetry ingestion layer.<\/p>\n\n\n
          Apps\n |\nOTel\n |\nCollector\n |\nDatadog \/ New<\/span> Relic \/ Dynatrace\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          This can reduce instrumentation coupling even when the observability backend remains commercial.<\/p>\n\n\n\n
          \n\n\n\n
          83. OpenTelemetry vs Fluent Bit \/ Fluentd<\/h1>\n\n\n\n
          Fluent Bit and Fluentd historically focus heavily on:<\/p>\n\n\n
          logs\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry supports:<\/p>\n\n\n
          traces\nmetrics\nlogs\n<\/code><\/span><\/pre>\n\n\n
          and increasingly unified signal processing.<\/p>\n\n\n\n
          Some organizations retain Fluent Bit for specialized log pipelines while adopting OpenTelemetry for traces\/metrics; others consolidate parts of the stack into OTel Collector.<\/p>\n\n\n\n
          The right answer depends on required receivers, performance characteristics and operational maturity.<\/p>\n\n\n\n
          \n\n\n\n
          84. Typical End-to-End Request<\/h1>\n\n\n\n
          Imagine:<\/p>\n\n\n
          Mobile Client\n   |\nAPI Gateway\n   |\nOrder Service\n   |\nKafka\n   |\nPayment Service\n   |\nPostgreSQL\n<\/code><\/span><\/pre>\n\n\n
          A trace might look like:<\/p>\n\n\n
          TRACE<\/span>: checkout\n\nMobile API request                 920 ms\n\u2502\n\u251c\u2500 API Gateway                     20 ms\n\u2502\n\u251c\u2500 Order Service                  120 ms\n\u2502  \u251c\u2500 validate_cart               15 ms\n\u2502  \u2514\u2500 publish Kafka event         25 ms\n\u2502\n\u251c\u2500 Kafka                           40 ms\n\u2502\n\u2514\u2500 Payment Service               700 ms\n   \u2502\n   \u251c\u2500 payment.authorize          500 ms\n   \u2502\n   \u2514\u2500 PostgreSQL                  80 ms\n<\/code><\/span>Code language:<\/span> HTTP<\/span> (<\/span>http<\/span>)<\/span><\/small><\/pre>\n\n\n
          You instantly discover:<\/p>\n\n\n
          payment<\/span>.authorize<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          is consuming most latency.<\/p>\n\n\n\n
          \n\n\n\n
          85. RED Method with OpenTelemetry<\/h1>\n\n\n\n
          For services, track:<\/p>\n\n\n\n
          Rate<\/h2>\n\n\n
          requests \/ second\n<\/code><\/span><\/pre>\n\n\n
          Errors<\/h2>\n\n\n
          failed requests \/ second\nerror percentage\n<\/code><\/span><\/pre>\n\n\n
          Duration<\/h2>\n\n\n
          P50\nP95\nP99\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry traces + metrics make RED dashboards straightforward.<\/p>\n\n\n\n
          \n\n\n\n
          86. USE Method for Infrastructure<\/h1>\n\n\n\n
          For infrastructure resources:<\/p>\n\n\n\n
          Utilization<\/h2>\n\n\n
          CPU utilization\nmemory utilization\n<\/code><\/span><\/pre>\n\n\n
          Saturation<\/h2>\n\n\n
          queue depth\nthrottling\n<\/code><\/span><\/pre>\n\n\n
          Errors<\/h2>\n\n\n
          disk errors\nnetwork errors\n<\/code><\/span><\/pre>\n\n\n
          Host and Kubernetes metrics can provide much of this information.<\/p>\n\n\n\n
          \n\n\n\n
          87. Golden Signals<\/h1>\n\n\n\n
          A classic set of service signals is:<\/p>\n\n\n
          Latency\nTraffic\nErrors\nSaturation\n<\/code><\/span><\/pre>\n\n\n
          A mature observability implementation should expose all four.<\/p>\n\n\n\n
          \n\n\n\n
          88. Service Level Indicators<\/h1>\n\n\n\n
          OTel metrics can feed SLIs such as:<\/p>\n\n\n
          Availability\n\nsuccessful_requests\n-------------------\ntotal_requests\n<\/code><\/span><\/pre>\n\n\n
          Latency:<\/p>\n\n\n
          requests < 500ms\n----------------\ntotal_requests\n<\/code><\/span><\/pre>\n\n\n
          Then build:<\/p>\n\n\n
          SLO\nError<\/span> Budget\nAlert\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          in the monitoring backend.<\/p>\n\n\n\n
          \n\n\n\n
          89. Production Naming Strategy<\/h1>\n\n\n\n
          Define conventions centrally.<\/p>\n\n\n\n
          For example:<\/p>\n\n\n
          service.namespace = ecommerce\nservice.name = checkout\nservice.version = 2.5.0\ndeployment.environment.name = production\n<\/code><\/span><\/pre>\n\n\n
          Avoid teams inventing:<\/p>\n\n\n
          checkout\nCheckout\ncheckout-service\ncheckout_prod\nprod-checkout\ncheckout-v2\n<\/code><\/span><\/pre>\n\n\n
          for the same logical service.<\/p>\n\n\n\n
          Stable identity is fundamental to useful observability.<\/p>\n\n\n\n
          \n\n\n\n
          90. Recommended Service Naming<\/h1>\n\n\n\n
          Good:<\/p>\n\n\n
          checkout\npayment\ninventory\nvehicle-api\ntelemetry-consumer\n<\/code><\/span><\/pre>\n\n\n
          Poor:<\/p>\n\n\n
          checkout-prod-pod-6745678<\/span>\ncheckout-v4-new<\/span>\nmy-service\nbackend1\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Dynamic instance information belongs in:<\/p>\n\n\n
          service<\/span>.instance<\/span>.id<\/span>\nk8s<\/span>.pod<\/span>.name<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          not the logical service.name<\/code>.<\/p>\n\n\n\n
          The service semantic conventions explicitly define service.name<\/code> as a logical service identity shared across horizontally scaled instances.<\/p>\n\n\n\n
          \n\n\n\n
          91. Recommended Environment Naming<\/h1>\n\n\n\n
          Use the semantic convention:<\/p>\n\n\n
          deployment<\/span>.environment<\/span>.name<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          For example:<\/p>\n\n\n
          development\ntest\nstaging\nproduction\n<\/code><\/span><\/pre>\n\n\n
          OpenTelemetry defines those as well-known values.<\/p>\n\n\n\n
          \n\n\n\n
          92. Cost Optimization<\/h1>\n\n\n\n
          Observability cost is strongly influenced by:<\/p>\n\n\n
          telemetry volume\ncardinality\nretention\nsampling\nlog verbosity\nattribute size\nnumber of services\nnumber of environments\n<\/code><\/span><\/pre>\n\n\n
          Control cost at multiple layers:<\/p>\n\n\n
          Application\n  \u2193\nHead Sampling\n  \u2193\nAgent Filtering\n  \u2193\nGateway Filtering\n  \u2193\nTail Sampling\n  \u2193\nBackend Retention\n<\/code><\/span><\/pre>\n\n\n
          Do not depend entirely on backend billing controls.<\/p>\n\n\n\n
          \n\n\n\n
          93. What Should Be Sampled?<\/h1>\n\n\n\n
          Good candidates for reduction:<\/p>\n\n\n
          high-volume successful requests\nhealth probes\nrepetitive background traffic\nknown internal noise\nverbose debugging\n<\/code><\/span><\/pre>\n\n\n
          Good candidates to preserve:<\/p>\n\n\n
          errors\ntimeouts\nslow requests\ncritical business transactions\nnew<\/span> release traffic\nsecurity-related events\nrare operations\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          \n\n\n\n
          94. What Not to Instrument Excessively<\/h1>\n\n\n\n
          Do not create spans for every tiny function.<\/p>\n\n\n\n
          Bad:<\/p>\n\n\n
          parseInteger()\nincrementCounter()\nformatString()\ngetCurrentTime()\n<\/code><\/span><\/pre>\n\n\n
          A trace containing thousands of trivial spans can become harder to understand than no trace at all.<\/p>\n\n\n\n
          Create spans around meaningful operations such as:<\/p>\n\n\n
          network requests\ndatabase calls\nqueue operations\nbusiness operations\nexpensive computation\ncritical workflow steps\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          95. Database Instrumentation<\/h1>\n\n\n\n
          Capture useful information such as:<\/p>\n\n\n
          database system\noperation\nduration\nserver\nquery class<\/span>\nerror<\/span> status<\/span>\n<\/span><\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          But be extremely careful with:<\/p>\n\n\n
          raw SQL\nbind parameters\ncustomer data\ncredentials\n<\/code><\/span><\/pre>\n\n\n
          Database observability should not become a data-leak mechanism.<\/p>\n\n\n\n
          \n\n\n\n
          96. Messaging \/ Kafka Instrumentation<\/h1>\n\n\n\n
          For messaging systems, useful telemetry can include:<\/p>\n\n\n
          publish\nconsume\nqueue\/topic\nlatency\nconsumer processing duration\nerrors\nretry count\n<\/code><\/span><\/pre>\n\n\n
          Distributed tracing across asynchronous systems requires careful propagation of trace context through message metadata.<\/p>\n\n\n\n
          A typical flow is:<\/p>\n\n\n
          Producer\n  |\ntrace context in message headers\n  |\nKafka\n  |\nConsumer\n  |\ncontinued trace\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          97. Trace Links<\/h1>\n\n\n\n
          Not every asynchronous relationship is naturally parent \u2192 child.<\/p>\n\n\n\n
          For batch processing or fan-in\/fan-out systems, span Links<\/strong> can express relationships to other span contexts.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          Message A \u2500\u2510\nMessage B \u2500\u253c\u2500> Batch processor span\nMessage C \u2500\u2518\n<\/code><\/span><\/pre>\n\n\n
          The batch processing span may link to the originating spans rather than pretending there is one simple parent.<\/p>\n\n\n\n
          Links are part of the OpenTelemetry Span model.<\/p>\n\n\n\n
          \n\n\n\n
          98. CI\/CD Observability<\/h1>\n\n\n\n
          Add deployment metadata such as:<\/p>\n\n\n
          service<\/span>.version<\/span>\ngit<\/span> commit<\/span>\nrelease<\/span> version<\/span>\nenvironment<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          Then correlate:<\/p>\n\n\n
          Deployment v2.4<\/span>.0<\/span>\n       |\n       v\nError<\/span> rate increases\n       |\n       v\nLatency regression\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          This makes release regression detection dramatically easier.<\/p>\n\n\n\n
          \n\n\n\n
          99. Recommended Adoption Strategy<\/h1>\n\n\n\n
          Do not attempt to instrument the entire organization on day one.<\/p>\n\n\n\n
          A better journey is:<\/p>\n\n\n
          Phase 1\nCollector + one service\n\nPhase 2\nAutomatic traces\n\nPhase 3\nResource conventions\n\nPhase 4\nCustom business spans\n\nPhase 5\nApplication metrics\n\nPhase 6\nLog correlation\n\nPhase 7\nKubernetes\/infrastructure telemetry\n\nPhase 8\nSampling + filtering\n\nPhase 9\nSLOs + alerts\n\nPhase 10\nGovernance + scale\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          100. Phase 1 \u2014 Proof of Concept<\/h1>\n\n\n\n
          Choose one representative service.<\/p>\n\n\n\n
          Deploy:<\/p>\n\n\n
          App\n |\nOTel SDK\n |\nCollector\n |\nBackend\n<\/code><\/span><\/pre>\n\n\n
          Validate:<\/p>\n\n\n
          trace received\nservice name correct\nattributes correct\ncontext propagation correct\nlatency correct\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          101. Phase 2 \u2014 Standardize<\/h1>\n\n\n\n
          Define organization-wide conventions:<\/p>\n\n\n
          service naming\nenvironment naming\nresource attributes\nsampling\nPII policies\nOTLP endpoints\nbackend routing\nversion metadata\n<\/code><\/span><\/pre>\n\n\n
          This is where an OTel platform becomes maintainable rather than a collection of unrelated experiments.<\/p>\n\n\n\n
          \n\n\n\n
          102. Phase 3 \u2014 Centralize Collection<\/h1>\n\n\n\n
          Move toward:<\/p>\n\n\n
          Applications\n   |\nAgent Collectors\n   |\nGateway Collectors\n   |\nBackend\n<\/code><\/span><\/pre>\n\n\n
          Centralize:<\/p>\n\n\n
          authentication\nsecurity\nsampling\nrouting\nfiltering\nredaction\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          103. Phase 4 \u2014 Add Metrics and Logs<\/h1>\n\n\n\n
          Once tracing is stable:<\/p>\n\n\n
          Traces\n+\nMetrics\n+\nLogs\n<\/code><\/span><\/pre>\n\n\n
          Then correlate them.<\/p>\n\n\n\n
          Example:<\/p>\n\n\n
          Alert:\nP99 checkout latency high\n\n        \u2193\n\nMetric identifies checkout service\n\n        \u2193\n\nTrace identifies payment span\n\n        \u2193\n\nLogs show payment timeout\n<\/code><\/span><\/pre>\n\n\n
          That is practical observability.<\/p>\n\n\n\n
          \n\n\n\n
          104. Phase 5 \u2014 SLO-Based Observability<\/h1>\n\n\n\n
          Build:<\/p>\n\n\n
          SLI\n |\nv\nSLO\n |\nv\nError<\/span> Budget\n |\nv\nBurn Rate Alert\n |\nv\nTrace investigation\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          This moves observability from:<\/p>\n\n\n
          \"lots of dashboards\"<\/span>\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
          to:<\/p>\n\n\n
          \"actionable reliability engineering\"<\/span>\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
          \n\n\n\n
          105. Troubleshooting \u2014 No Telemetry Arriving<\/h1>\n\n\n\n
          Use a layered approach.<\/p>\n\n\n\n
          Step 1 \u2014 Application<\/h2>\n\n\n\n
          Verify:<\/p>\n\n\n
          OTel instrumentation loaded?\nCorrect service name?\nExporter enabled?\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Step 2 \u2014 Endpoint<\/h2>\n\n\n\n
          Verify:<\/p>\n\n\n
          OTEL_EXPORTER_OTLP_ENDPOINT\n<\/code><\/span><\/pre>\n\n\n
          Check whether you meant:<\/p>\n\n\n
          4317\n<\/code><\/span><\/pre>\n\n\n
          or:<\/p>\n\n\n
          4318\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Step 3 \u2014 Protocol<\/h2>\n\n\n\n
          Verify:<\/p>\n\n\n
          grpc\n<\/code><\/span><\/pre>\n\n\n
          versus:<\/p>\n\n\n
          http\/protobuf\n<\/code><\/span><\/pre>\n\n\n
          A protocol mismatch is a very common failure.<\/p>\n\n\n\n
          \n\n\n\n
          Step 4 \u2014 Collector receiver<\/h2>\n\n\n\n
          Verify:<\/p>\n\n\n
          receivers:\n  otlp:\n    protocols:\n      grpc:\n      http:\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Step 5 \u2014 Collector pipeline<\/h2>\n\n\n\n
          Defining a component is not enough.<\/p>\n\n\n\n
          It must be referenced in:<\/p>\n\n\n
          service:\n  pipelines:\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          106. Use Debug Exporter<\/h1>\n\n\n\n
          Temporarily:<\/p>\n\n\n
          exporters:\n  debug:\n    verbosity: detailed\n<\/code><\/span><\/pre>\n\n\n
          Then:<\/p>\n\n\n
          service:\n  pipelines:\n    traces:\n      exporters:\n        - debug\n<\/code><\/span><\/pre>\n\n\n
          If telemetry appears there:<\/p>\n\n\n
          Application \u2192 Collector = GOOD\n<\/code><\/span><\/pre>\n\n\n
          Now investigate:<\/p>\n\n\n
          Collector \u2192 Backend\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          107. Collector Troubleshooting Flow<\/h1>\n\n\n
          Application generating telemetry?\n        |\n       Yes\n        |\nCollector receiving telemetry?\n        |\n       Yes\n        |\nProcessor dropping telemetry?\n        |\n       No\n        |\nExporter succeeding?\n        |\n       Yes\n        |\nBackend ingesting telemetry?\n        |\n       Yes\n        |\nBackend query \/ dashboard correct?\n<\/code><\/span><\/pre>\n\n\n
          Troubleshoot one boundary at a time.<\/p>\n\n\n\n
          \n\n\n\n
          108. Broken Distributed Trace<\/h1>\n\n\n\n
          Symptoms:<\/p>\n\n\n
          Service A trace\nService B separate trace\n<\/code><\/span><\/pre>\n\n\n
          Investigate:<\/p>\n\n\n
          W3C traceparent propagation\nproxy\/header stripping\nHTTP instrumentation\ngRPC instrumentation\nmessage header propagation\ncustom middleware\nasync<\/span> context handling\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          OpenTelemetry context propagation is what allows downstream spans to join the same distributed trace.<\/p>\n\n\n\n
          \n\n\n\n
          109. Missing Kubernetes Metadata<\/h1>\n\n\n\n
          Check:<\/p>\n\n\n
          k8sattributes processor enabled?\nRBAC correct?\nProcessor actually in pipeline?\nPod association working?\n<\/code><\/span><\/pre>\n\n\n
          Remember:<\/p>\n\n\n
          Declaring processor\n!=\nUsing processor\n<\/code><\/span><\/pre>\n\n\n
          It needs to appear in the pipeline.<\/p>\n\n\n\n
          \n\n\n\n
          110. Duplicate Metrics<\/h1>\n\n\n\n
          Common Kubernetes cause:<\/p>\n\n\n
          Multiple gateway replicas\n   |\n   v\nSame target scraped repeatedly\n<\/code><\/span><\/pre>\n\n\n
          The OpenTelemetry Kubernetes documentation specifically warns that identical Collector replicas scraping the same targets can generate duplicate metrics.<\/p>\n\n\n\n
          Use:<\/p>\n\n\n
          Target Allocator\nsharding\nsingle-replica cluster receivers\nappropriate DaemonSets\n<\/code><\/span><\/pre>\n\n\n
          depending on the receiver.<\/p>\n\n\n\n
          \n\n\n\n
          111. OpenTelemetry Demo<\/h1>\n\n\n\n
          The OpenTelemetry project maintains a complete demo application.<\/p>\n\n\n
          git clone<\/span> https:\/\/github.com\/open-telemetry\/opentelemetry-demo.git<\/span>\n\ncd opentelemetry-demo\n\nmake start\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          The full demo includes multiple services and observability backends and is one of the best practical environments for learning how OTel works end-to-end.<\/p>\n\n\n\n
          A very useful learning exercise is to:<\/p>\n\n\n
          1.<\/span> Run the demo.\n2.<\/span> Generate traffic.\n3.<\/span> View traces.\n4.<\/span> Find a slow span.\n5.<\/span> Inspect related metrics.\n6.<\/span> Inspect related logs.\n7.<\/span> Modify Collector configuration.\n8.<\/span> Add filtering.\n9.<\/span> Experiment with<\/span> sampling.\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          \n\n\n\n
          112. Development vs Production Configuration<\/h1>\n\n\n\n
          Development:<\/p>\n\n\n
          100% traces\ndebug exporter\nsimple Collector\nno HA\nminimal filtering\n<\/code><\/span><\/pre>\n\n\n
          Production:<\/p>\n\n\n
          sampling\nHA Collectors\nTLS\nauthentication\nqueues\nretries\nresource limits\nmemory limiter\nbatching\nredaction\nPII controls\nself<\/span>-monitoring\nautoscaling\nversion pinning\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          Never promote a tutorial Collector configuration directly into production without applying those controls.<\/p>\n\n\n\n
          \n\n\n\n
          113. Recommended Production Architecture<\/h1>\n\n\n\n
          For most sizeable cloud\/Kubernetes environments:<\/p>\n\n\n\n
                                   APPLICATIONS\n                              |\n            Auto + Manual OTel Instrumentation\n                              |\n                              v\n                    OTLP 4317 \/ 4318\n                              |\n                              v\n             +--------------------------------+\n             | Node-level OTel Agent          |\n             | Collector DaemonSet            |\n             |                                |\n             | - logs                         |\n             | - kubelet metrics              |\n             | - host metrics                 |\n             | - lightweight enrichment       |\n             +---------------+----------------+\n                             |\n                             v\n             +--------------------------------+\n             | OTel Gateway Tier              |\n             |                                |\n             | memory protection              |\n             | filtering                      |\n             | redaction                      |\n             | transformation                 |\n             | tail sampling                  |\n             | batching                       |\n             | queue \/ retry                  |\n             | backend credentials            |\n             +---------------+----------------+\n                             |\n                             | TLS\n                             v\n             +--------------------------------+\n             | Observability Backend          |\n             |                                |\n             | Traces                         |\n             | Metrics                        |\n             | Logs                           |\n             +--------------------------------+\n<\/code><\/pre>\n\n\n\n
          For tail sampling across horizontally scaled gateways:<\/p>\n\n\n
          Agents\n  |\ntrace-ID aware load balancing\n  |\nTail Sampling Gateways\n<\/code><\/span><\/pre>\n\n\n
          not arbitrary round-robin routing.<\/p>\n\n\n\n
          \n\n\n\n
          114. Production HA<\/h1>\n\n\n\n
          For gateways:<\/p>\n\n\n
          minimum multiple replicas where architecture permits\nPodDisruptionBudget\nanti-affinity \/ topology spread\nresource requests\nresource limits\nhealth probes\nHPA\n<\/code><\/span><\/pre>\n\n\n
          But stateful processing such as tail sampling requires more careful routing than ordinary stateless gateway scaling.<\/p>\n\n\n\n
          \n\n\n\n
          115. Kubernetes Resource Management<\/h1>\n\n\n\n
          Always define Collector resources.<\/p>\n\n\n\n
          Example conceptually:<\/p>\n\n\n
          resources:\n\n  requests:\n    cpu: 250<\/span>m\n    memory<\/span>: 512<\/span>Mi\n\n  limits<\/span>:\n    cpu: \"1\"<\/span>\n    memory<\/span>: 1<\/span>Gi\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Then configure memory limiting appropriately below the container’s actual hard memory limit.<\/p>\n\n\n\n
          Leave headroom.<\/p>\n\n\n\n
          Do not configure:<\/p>\n\n\n
          memory_limiter threshold == container limit\n<\/code><\/span><\/pre>\n\n\n
          because the process may be OOM-killed before protection becomes effective.<\/p>\n\n\n\n
          \n\n\n\n
          116. Network Architecture<\/h1>\n\n\n\n
          Prefer:<\/p>\n\n\n
          Applications\n   |\nCluster-private<\/span> OTLP endpoint\n   |\nGateway\n   |\nControlled egress\n   |\nBackend\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          over:<\/p>\n\n\n
          Every application\n   |\nInternet\n   |\nObservability vendor\n<\/code><\/span><\/pre>\n\n\n
          Centralized egress provides:<\/p>\n\n\n
          simpler firewalling\ncredential isolation\nnetwork control\nconsistent telemetry policy\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          117. Credential Architecture<\/h1>\n\n\n\n
          Prefer:<\/p>\n\n\n
          Application\n   |\nno vendor credential\n   |\nCollector\n   |\nbackend credential\n   |\nVendor\n<\/code><\/span><\/pre>\n\n\n
          This keeps backend API keys out of hundreds of application workloads.<\/p>\n\n\n\n
          \n\n\n\n
          118. Vendor Migration Architecture<\/h1>\n\n\n\n
          Because applications emit OTLP:<\/p>\n\n\n
          Apps\n |\nOTel\n |\nCollector\n |\nCurrent Backend\n<\/code><\/span><\/pre>\n\n\n
          Migration can become:<\/p>\n\n\n
          Apps\n |\nOTel\n |\nCollector\n |        |\n v        v\nOld      New<\/span>\nBackend  Backend\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          After validation:<\/p>\n\n\n
          Apps\n |\nOTel\n |\nCollector\n |\nNew<\/span> Backend\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
          No major application instrumentation rewrite should be necessary.<\/p>\n\n\n\n
          That is one of OpenTelemetry’s most important strategic advantages.<\/p>\n\n\n\n
          \n\n\n\n
          119. Multi-Backend Export<\/h1>\n\n\n\n
          During migration or for specialized purposes:<\/p>\n\n\n\n
                     +----> Backend A\n           |\nCollector--+\n           |\n           +----> Backend B\n<\/code><\/pre>\n\n\n\n
          Be cautious, however.<\/p>\n\n\n\n
          Duplicating telemetry doubles downstream:<\/p>\n\n\n
          network volume\ningestion volume\nstorage cost\n<\/code><\/span><\/pre>\n\n\n
          so use fan-out intentionally.<\/p>\n\n\n\n
          \n\n\n\n
          120. Common Anti-Patterns<\/h1>\n\n\n\n
          Avoid these.<\/p>\n\n\n\n
          Anti-pattern 1<\/h3>\n\n\n\n
          Every application exports directly to a vendor.<\/p>\n\n\n\n
          Problem:<\/p>\n\n\n
          credentials everywhere\nvendor coupling\ndifficult migrations\ninconsistent policy\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Anti-pattern 2<\/h3>\n\n\n\n
          100% tracing forever.<\/p>\n\n\n\n
          Problem:<\/p>\n\n\n
          massive telemetry volume\nmassive cost\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Anti-pattern 3<\/h3>\n\n\n\n
          High-cardinality metric labels.<\/p>\n\n\n\n
          Problem:<\/p>\n\n\n
          time-series explosion\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Anti-pattern 4<\/h3>\n\n\n\n
          Different service naming by every team.<\/p>\n\n\n\n
          Problem:<\/p>\n\n\n
          chaotic queries\nbroken dashboards\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Anti-pattern 5<\/h3>\n\n\n\n
          Sensitive customer data in spans\/logs.<\/p>\n\n\n\n
          Problem:<\/p>\n\n\n
          security\nprivacy\ncompliance\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Anti-pattern 6<\/h3>\n\n\n\n
          No Collector self-monitoring.<\/p>\n\n\n\n
          Problem:<\/p>\n\n\n
          observability pipeline fails silently\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Anti-pattern 7<\/h3>\n\n\n\n
          Tail sampling behind random load balancing.<\/p>\n\n\n\n
          Problem:<\/p>\n\n\n
          fragmented traces\nincorrect sampling decisions\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          Anti-pattern 8<\/h3>\n\n\n\n
          Using latest<\/code> everywhere.<\/p>\n\n\n\n
          Problem:<\/p>\n\n\n
          unexpected upgrades\nconfiguration compatibility changes\nhard-to-reproduce incidents\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          121. OpenTelemetry Production Best Practices<\/h1>\n\n\n\n
          A strong production standard is:<\/p>\n\n\n\n
            \n
          1. Use OpenTelemetry as the application instrumentation standard.<\/li>\n\n\n\n
          2. Prefer OTLP between applications and Collectors.<\/li>\n\n\n\n
          3. Use automatic instrumentation for quick baseline coverage.<\/li>\n\n\n\n
          4. Add manual spans for important business operations.<\/li>\n\n\n\n
          5. Standardize service.name<\/code>.<\/li>\n\n\n\n
          6. Standardize service.namespace<\/code>.<\/li>\n\n\n\n
          7. Standardize service.version<\/code>.<\/li>\n\n\n\n
          8. Use deployment.environment.name<\/code>.<\/li>\n\n\n\n
          9. Follow semantic conventions.<\/li>\n\n\n\n
          10. Keep metric cardinality bounded.<\/li>\n\n\n\n
          11. Keep span names stable.<\/li>\n\n\n\n
          12. Propagate W3C Trace Context consistently.<\/li>\n\n\n\n
          13. Never put sensitive information into baggage.<\/li>\n\n\n\n
          14. Use Agent + Gateway architecture for substantial Kubernetes environments.<\/li>\n\n\n\n
          15. Keep agent processing lightweight.<\/li>\n\n\n\n
          16. Centralize expensive processing in gateways.<\/li>\n\n\n\n
          17. Use memory_limiter<\/code>.<\/li>\n\n\n\n
          18. Filter\/drop useless telemetry early.<\/li>\n\n\n\n
          19. Batch after filtering and enrichment.<\/li>\n\n\n\n
          20. Use queues and retries for remote exports.<\/li>\n\n\n\n
          21. Consider persistent queues for important telemetry.<\/li>\n\n\n\n
          22. Use tail sampling strategically.<\/li>\n\n\n\n
          23. Ensure trace-ID-aware routing when horizontally scaling tail-sampling gateways.<\/li>\n\n\n\n
          24. Enrich Kubernetes telemetry using k8sattributes<\/code>.<\/li>\n\n\n\n
          25. Avoid duplicate Kubernetes scraping.<\/li>\n\n\n\n
          26. Protect OTLP endpoints.<\/li>\n\n\n\n
          27. Use TLS\/mTLS where appropriate.<\/li>\n\n\n\n
          28. Keep vendor credentials in gateways rather than applications.<\/li>\n\n\n\n
          29. Apply least privilege.<\/li>\n\n\n\n
          30. Redact secrets and PII.<\/li>\n\n\n\n
          31. Monitor the Collector itself.<\/li>\n\n\n\n
          32. Alert on queue saturation and dropped telemetry.<\/li>\n\n\n\n
          33. Pin Collector\/SDK\/Operator versions.<\/li>\n\n\n\n
          34. Test upgrades in non-production first.<\/li>\n\n\n\n
          35. Treat the telemetry pipeline as production infrastructure.<\/li>\n<\/ol>\n\n\n\n
            \n\n\n\n
            122. Recommended Repository Structure<\/h1>\n\n\n\n
            For an organization managing OTel through Infrastructure as Code:<\/p>\n\n\n
            observability\/\n\u2502\n\u251c\u2500\u2500 collector\/\n\u2502   \u2502\n\u2502   \u251c\u2500\u2500 agent\/\n\u2502   \u2502   \u251c\u2500\u2500 values.yaml\n\u2502   \u2502   \u2514\u2500\u2500 config.yaml\n\u2502   \u2502\n\u2502   \u2514\u2500\u2500 gateway\/\n\u2502       \u251c\u2500\u2500 values.yaml\n\u2502       \u2514\u2500\u2500 config.yaml\n\u2502\n\u251c\u2500\u2500 operator\/\n\u2502   \u251c\u2500\u2500 values.yaml\n\u2502   \u2514\u2500\u2500 instrumentation.yaml\n\u2502\n\u251c\u2500\u2500 policies\/\n\u2502   \u251c\u2500\u2500 attributes.md\n\u2502   \u251c\u2500\u2500 sampling.md\n\u2502   \u251c\u2500\u2500 pii.md\n\u2502   \u2514\u2500\u2500 service-naming.md\n\u2502\n\u251c\u2500\u2500 dashboards\/\n\u2502\n\u251c\u2500\u2500 alerts\/\n\u2502\n\u251c\u2500\u2500 slo\/\n\u2502\n\u2514\u2500\u2500 README.md\n<\/code><\/span><\/pre>\n\n\n
            This separates:<\/p>\n\n\n
            collection\ninstrumentation\npolicy\nvisualization\nalerting\nSLOs\n<\/code><\/span><\/pre>\n\n\n
            \n\n\n\n
            123. OpenTelemetry Governance<\/h1>\n\n\n\n
            Large organizations should maintain an internal OTel specification.<\/p>\n\n\n\n
            Example:<\/p>\n\n\n
            OpenTelemetry Standard\n\n1. Service Naming\n2. Resource Attributes\n3. Semantic Conventions\n4. Instrumentation Rules\n5. Metric Naming\n6. Metric Cardinality\n7. Span Naming\n8. Sampling\n9. PII Handling\n10. Logging\n11. Collector Architecture\n12. Security\n13. Backend Routing\n14. Version Management\n15. SLO Integration\n<\/code><\/span><\/pre>\n\n\n
            Without governance, telemetry often turns into an expensive data lake nobody trusts.<\/p>\n\n\n\n
            \n\n\n\n
            124. Recommended Learning Path<\/h1>\n\n\n\n
            Learn OpenTelemetry in this order:<\/p>\n\n\n
            Observability\n   \u2193\nTraces \/ Spans\n   \u2193\nContext Propagation\n   \u2193\nMetrics\n   \u2193\nLogs\n   \u2193\nResources\n   \u2193\nSemantic Conventions\n   \u2193\nOTLP\n   \u2193\nSDK\n   \u2193\nAuto Instrumentation\n   \u2193\nManual Instrumentation\n   \u2193\nCollector\n   \u2193\nReceivers \/ Processors \/ Exporters\n   \u2193\nKubernetes\n   \u2193\nAgent + Gateway\n   \u2193\nSampling\n   \u2193\nSecurity\n   \u2193\nScaling\n   \u2193\nSLOs\n<\/code><\/span><\/pre>\n\n\n
            Do not begin by memorizing hundreds of Collector configuration options.<\/p>\n\n\n\n
            Understand the telemetry model first.<\/p>\n\n\n\n
            \n\n\n\n
            125. OpenTelemetry Mental Model<\/h1>\n\n\n\n
            If you remember only one architecture, remember:<\/p>\n\n\n\n
                              SOURCE\n                    |\n                    v\n            INSTRUMENTATION\n                    |\n                    v\n                  OTLP\n                    |\n                    v\n               COLLECTOR\n                    |\n            +-------+-------+\n            |       |       |\n         Receive  Process  Export\n            |       |       |\n            +-------+-------+\n                    |\n                    v\n                  OTLP\n                    |\n                    v\n                 BACKEND\n                    |\n                    v\n       Dashboards \/ Alerts \/ SLOs\n<\/code><\/pre>\n\n\n\n
            And for production Kubernetes:<\/p>\n\n\n
            Application\n    |\nOTel SDK \/ Auto Instrumentation\n    |\n    v\nAgent Collector\n    |\n    v\nGateway Collector\n    |\n    v\nObservability Backend\n<\/code><\/span><\/pre>\n\n\n
            \n\n\n\n
            126. End-to-End Workflow<\/h1>\n\n\n\n
            The complete telemetry lifecycle is:<\/p>\n\n\n
            1.<\/span> Application receives request\n\n2.<\/span> Instrumentation creates root span\n\n3.<\/span> Trace context is propagated downstream\n\n4.<\/span> Child services create child spans\n\n5.<\/span> SDK records spans and measurements\n\n6.<\/span> Logs are correlated with<\/span> trace\/span context\n\n7.<\/span> SDK exports telemetry using OTLP\n\n8.<\/span> Agent Collector receives telemetry\n\n9.<\/span> Agent enriches telemetry\n\n10.<\/span> Gateway receives telemetry\n\n11.<\/span> Gateway applies:\n       memory protection\n       filtering\n       redaction\n       transformations\n       sampling\n       batching\n\n12.<\/span> Gateway exports using OTLP\n\n13.<\/span> Backend stores telemetry\n\n14.<\/span> Dashboard visualizes metrics\n\n15.<\/span> Alert detects SLO degradation\n\n16.<\/span> Engineer opens trace\n\n17.<\/span> Trace identifies slow component\n\n18.<\/span> Correlated logs expose error\n\n19.<\/span> Root cause is identified\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
            That is OpenTelemetry end to end.<\/p>\n\n\n\n
            \n\n\n\n
            127. Reference Architecture Summary<\/h1>\n\n\n\n
                                            USERS\n                                  |\n                                  v\n                        +-------------------+\n                        | Load Balancer     |\n                        +---------+---------+\n                                  |\n                                  v\n+-------------------------------------------------------------+\n|                     KUBERNETES                              |\n|                                                             |\n| +-------------+   +-------------+   +-------------+         |\n| | Service A   |-->| Service B   |-->| Service C   |         |\n| | OTel SDK    |   | OTel SDK    |   | OTel SDK    |         |\n| +------+------+   +------+------+   +------+------+         |\n|        |                 |                  |                |\n|        +-----------------+------------------+                |\n|                          |                                   |\n|                          v                                   |\n|                 +------------------+                         |\n|                 | Agent Collector  |                         |\n|                 | DaemonSet        |                         |\n|                 +--------+---------+                         |\n|                          |                                   |\n|                          v                                   |\n|               +----------------------+                       |\n|               | Gateway Collectors   |                       |\n|               |                      |                       |\n|               | memory limiter       |                       |\n|               | filtering            |                       |\n|               | redaction            |                       |\n|               | tail sampling        |                       |\n|               | batching             |                       |\n|               | queue\/retry          |                       |\n|               +----------+-----------+                       |\n+--------------------------|-----------------------------------+\n                           |\n                           | OTLP + TLS\n                           v\n          +----------------+----------------+\n          |                                 |\n          v                                 v\n +----------------+                +----------------+\n | Trace Backend  |                | Metrics\/Logs   |\n | Tempo\/Jaeger   |                | Backend        |\n +-------+--------+                +-------+--------+\n         |                                 |\n         +----------------+----------------+\n                          |\n                          v\n                +--------------------+\n                | Grafana \/ Vendor UI|\n                +---------+----------+\n                          |\n               +----------+-----------+\n               |                      |\n               v                      v\n          Dashboards                Alerts\n                                      |\n                                      v\n                                     SLO\n<\/code><\/pre>\n\n\n\n
            \n\n\n\n
            128. Final Architecture Principles<\/h1>\n\n\n\n
            OpenTelemetry works best when you think of it not as another monitoring agent but as an observability abstraction layer<\/strong>.<\/p>\n\n\n\n
            Your application should know:<\/p>\n\n\n
            \"I generate telemetry.\"<\/span>\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
            not:<\/p>\n\n\n
            \"I generate Datadog telemetry.\"<\/span>\n\n\"I generate New Relic telemetry.\"<\/span>\n\n\"I generate Jaeger telemetry.\"<\/span>\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
            The ideal architecture becomes:<\/p>\n\n\n
            Application\n      |\n      v\nOpenTelemetry\n      |\n      v\nObservability Platform\n<\/code><\/span><\/pre>\n\n\n
            This separation provides:<\/p>\n\n\n
            Portability\nStandardization\nConsistent instrumentation\nVendor flexibility\nCentralized governance\nSecurity control\nCost control\nCross-signal correlation\nBetter troubleshooting\n<\/code><\/span><\/pre>\n\n\n
            \n\n\n\n
            129. Final Production Checklist<\/h1>\n\n\n\n
            Before calling an OpenTelemetry deployment production-ready, verify:<\/p>\n\n\n\n
            Instrumentation<\/h3>\n\n\n
            \u2610 All critical services have instrumentation\n\u2610 Automatic instrumentation validated\n\u2610 Important business flows manually instrumented\n\u2610 Distributed context propagation works\n<\/code><\/span><\/pre>\n\n\n
            Resources<\/h3>\n\n\n
            \u2610 service<\/span>.name<\/span>\n\u2610 service<\/span>.namespace<\/span>\n\u2610 service<\/span>.version<\/span>\n\u2610 deployment<\/span>.environment<\/span>.name<\/span>\n\u2610 cloud<\/span> metadata<\/span>\n\u2610 Kubernetes<\/span> metadata<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
            Tracing<\/h3>\n\n\n
            \u2610 Stable span names\n\u2610 Appropriate attributes\n\u2610 Errors represented correctly\n\u2610 Sampling strategy documented\n\u2610 Trace continuity tested\n<\/code><\/span><\/pre>\n\n\n
            Metrics<\/h3>\n\n\n
            \u2610 RED metrics available\n\u2610 Infrastructure metrics available\n\u2610 Cardinality reviewed\n\u2610 Units defined\n\u2610 Histograms used appropriately\n<\/code><\/span><\/pre>\n\n\n
            Logs<\/h3>\n\n\n
            \u2610 Structured logging\n\u2610 Trace IDs correlated\n\u2610 Sensitive fields removed\n\u2610 Log volume controlled\n<\/code><\/span><\/pre>\n\n\n
            Collector<\/h3>\n\n\n
            \u2610 memory_limiter\n\u2610 filtering\n\u2610 enrichment\n\u2610 batch\n\u2610 queues\n\u2610 retries\n\u2610 persistent buffering where required\n<\/code><\/span><\/pre>\n\n\n
            Kubernetes<\/h3>\n\n\n
            \u2610 Agent Collector\n\u2610 Gateway Collector\n\u2610 Kubernetes attributes\n\u2610 Kubelet metrics\n\u2610 Host metrics\n\u2610 Cluster receivers appropriately singleton\/sharded\n\u2610 No duplicate scraping\n<\/code><\/span><\/pre>\n\n\n
            Security<\/h3>\n\n\n
            \u2610 TLS\n\u2610 authentication\n\u2610 least privilege\n\u2610 secret management\n\u2610 OTLP endpoints not publicly exposed accidentally\n\u2610 PII redaction\n<\/code><\/span><\/pre>\n\n\n
            Reliability<\/h3>\n\n\n
            \u2610 Multiple gateway replicas where appropriate\n\u2610 resource requests\/limits\n\u2610 disruption protection\n\u2610 queue monitoring\n\u2610 exporter-failure monitoring\n\u2610 Collector self<\/span>-observability\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
            Operations<\/h3>\n\n\n
            \u2610 Version pinning\n\u2610 Upgrade procedure\n\u2610 Rollback procedure\n\u2610 Load testing\n\u2610 Failure testing\n\u2610 Capacity planning\n<\/code><\/span><\/pre>\n\n\n
            Observability Outcomes<\/h3>\n\n\n
            \u2610 Dashboards\n\u2610 Alerts\n\u2610 SLOs\n\u2610 Error<\/span> budgets\n\u2610 Trace investigation workflow\n\u2610 Log correlation\n\u2610 Release\/version correlation\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
            When those pieces are in place, OpenTelemetry stops being merely a tracing SDK and becomes the telemetry foundation of the platform<\/strong>.<\/p>\n\n\n\n
            \n\n\n\n
            130. One-Sentence Summary<\/h1>\n\n\n\n
            OpenTelemetry standardizes how applications and infrastructure generate, correlate, collect, process and transport traces, metrics and logs, while the OpenTelemetry Collector provides the vendor-neutral control plane that routes this telemetry securely and reliably to the observability backend of your choice.<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"
            From Zero to Production: Architecture, Traces, Metrics, Logs, OTLP, Collector, Kubernetes, Sampling, Security, Scaling, Troubleshooting and Best Practices Technology: OpenTelemetry \/ OTelLevel: Beginner \u2192 Intermediate \u2192 Advanced… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[11138],"tags":[],"class_list":["post-78584","post","type-post","status-publish","format-standard","hentry","category-best-tools"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/78584","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=78584"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/78584\/revisions"}],"predecessor-version":[{"id":78585,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/78584\/revisions\/78585"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=78584"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=78584"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=78584"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}