{"id":26859,"date":"2026-06-23T03:57:47","date_gmt":"2026-06-23T03:57:47","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=26859"},"modified":"2026-06-23T03:57:50","modified_gmt":"2026-06-23T03:57:50","slug":"datadog-agent-troubleshooting-guide","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/datadog-agent-troubleshooting-guide\/","title":{"rendered":"Datadog Troubleshooting Master Guide"},"content":{"rendered":"\n

It covers:<\/p>\n\n\n\n

Datadog Agent, Kubernetes Agent, Cluster Agent, integrations, logs, APM\/traces, custom metrics, DogStatsD, OpenTelemetry, API keys, Terraform, monitors, SLOs, RUM, Synthetics, cloud integrations, cost, permissions, and common \u201cdata missing in Datadog\u201d scenarios.<\/p>\n\n\n\n

1. First Principle: Understand the Datadog Data Flow<\/h2>\n\n\n\n

Before troubleshooting Datadog, always identify where the data path is broken<\/strong>.<\/p>\n\n\n\n

A typical telemetry path looks like this:<\/p>\n\n\n

flowchart<\/span> LR<\/span>\n    A<\/span>[Application \/ Host \/ Container \/ Cloud Service]<\/span> --<\/span>> B<\/span>[Datadog Agent \/ SDK \/ Collector \/ Integration]<\/span>\n    B<\/span> --<\/span>> C<\/span>[Network \/ Proxy \/ TLS \/ Firewall]<\/span>\n    C<\/span> --<\/span>> D<\/span>[Datadog Intake Endpoint]<\/span>\n    D<\/span> --<\/span>> E<\/span>[Datadog Processing Pipeline]<\/span>\n    E<\/span> --<\/span>> F<\/span>[Indexes \/ Metrics Store \/ Trace Store \/ RUM \/ Synthetics]<\/span>\n    F<\/span> --<\/span>> G<\/span>[Dashboard \/ Monitor \/ SLO \/ Explorer]<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Most Datadog problems fall into one of these buckets:<\/p>\n\n\n\n
Problem Area<\/th>Typical Issue<\/th><\/tr><\/thead>
Source<\/td>App is not emitting logs, metrics, or traces<\/td><\/tr>
Agent \/ SDK<\/td>Agent not running, wrong config, tracer not loaded<\/td><\/tr>
Network<\/td>Proxy, firewall, DNS, TLS, site mismatch<\/td><\/tr>
Authentication<\/td>Wrong API key, wrong app key, wrong Datadog site<\/td><\/tr>
Processing<\/td>Log pipeline, index filter, sampling, retention, exclusion<\/td><\/tr>
Query\/UI<\/td>Wrong tags, wrong environment, wrong timeframe, wrong aggregation<\/td><\/tr>
Cost controls<\/td>Metrics\/logs\/traces intentionally dropped or not indexed<\/td><\/tr>
Permissions<\/td>User cannot see data because of RBAC, restriction queries, or scoped app keys<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Datadog\u2019s own Agent troubleshooting checklist starts with exactly these basics: internet\/proxy access, correct API key, correct Datadog site, only one Agent per host, restarting after YAML changes, then checking Agent status and logs. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
2. Universal Troubleshooting Checklist<\/h1>\n\n\n\n
Use this before debugging any specific Datadog product.<\/p>\n\n\n\n
2.1 Check the Datadog site<\/h2>\n\n\n\n
Datadog has multiple sites, such as US1, US3, US5, EU, AP1, AP2, and US1-FED. If your Agent, SDK, Terraform provider, API request, RUM SDK, or Synthetic Private Location is configured for the wrong site, data may be sent successfully but to the wrong Datadog intake.<\/p>\n\n\n\n
Common values:<\/p>\n\n\n
datadoghq<\/span>.com<\/span>\nus3<\/span>.datadoghq<\/span>.com<\/span>\nus5<\/span>.datadoghq<\/span>.com<\/span>\ndatadoghq<\/span>.eu<\/span>\nap1<\/span>.datadoghq<\/span>.com<\/span>\nap2<\/span>.datadoghq<\/span>.com<\/span>\nddog-gov<\/span>.com<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Check:<\/p>\n\n\n
sudo datadog-agent status\nsudo datadog-agent configcheck\ngrep -i \"site\"<\/span> \/etc\/datadog-agent\/datadog.yaml\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
For Kubernetes, check Helm\/Operator values:<\/p>\n\n\n
helm get<\/span> values datadog -n datadog\nkubectl get<\/span> datadogagent -A -o yaml\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
2.2 Check API key and application key<\/h2>\n\n\n\n
There are two key types:<\/p>\n\n\n\n
Key Type<\/th>Used For<\/th><\/tr><\/thead>
API key<\/td>Agent and intake submission: metrics, logs, events<\/td><\/tr>
Application key<\/td>Datadog API operations: Terraform, dashboards, monitors, user-level actions<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Datadog API keys are unique to the organization and are required by the Agent to submit metrics and events. Application keys are used with the API key for programmatic API access and inherit permissions from the user or service account unless scoped differently. (Datadog<\/a>)<\/p>\n\n\n\n
Check:<\/p>\n\n\n
grep -i \"api_key\"<\/span> \/etc\/datog-agent\/datadog.yaml\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
But do not<\/strong> print secrets in CI logs or Slack.<\/p>\n\n\n\n
For API validation:<\/p>\n\n\n
curl -X GET \"https:\/\/api.<DATADOG_SITE>\/api\/v1\/validate\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
For API + app key validation:<\/p>\n\n\n
curl -X GET \"https:\/\/api.<DATADOG_SITE>\/api\/v2\/validate_keys\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
2.3 Check network, proxy, DNS, TLS<\/h2>\n\n\n\n
Many Datadog issues are not Datadog issues. They are egress issues.<\/p>\n\n\n\n
Check:<\/p>\n\n\n
curl -v https:\/\/api.datadoghq.com<\/span>\ncurl -v https:\/\/app.datadoghq.com<\/span>\ncurl -v https:\/\/trace.agent.datadoghq.com<\/span>\ncurl -v https:\/\/http-intake.logs.datadoghq.com<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
For your actual site, replace with the correct regional endpoint.<\/p>\n\n\n\n
From Kubernetes:<\/p>\n\n\n
kubectl run curl-test --rm -it --image=curlimages\/curl -- sh\ncurl -v https:\/\/api.datadoghq.com<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Common causes:<\/p>\n\n\n\n
Symptom<\/th>Likely Cause<\/th><\/tr><\/thead>
Agent runs but no data<\/td>Firewall or proxy blocks Datadog intake<\/td><\/tr>
TLS handshake error<\/td>Corporate proxy, custom CA, SSL inspection<\/td><\/tr>
Works on host, fails in pod<\/td>Kubernetes NetworkPolicy, egress policy, NAT issue<\/td><\/tr>
Works in dev, fails in prod<\/td>Different outbound firewall\/proxy<\/td><\/tr>
Random timeout<\/td>Proxy saturation or DNS issue<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
2.4 Check Agent health and status<\/h2>\n\n\n\n
Core commands:<\/p>\n\n\n
sudo datadog-agent status\nsudo datadog-agent health\nsudo datadog-agent diagnose\nsudo datadog-agent configcheck\nsudo datadog-agent flare\n<\/code><\/span><\/pre>\n\n\n
For Docker:<\/p>\n\n\n
docker exec -it <datadog-agent-container<\/span>><\/span> agent status\ndocker exec -it <datadog-agent-container<\/span>><\/span> agent health\ndocker exec -it <datadog-agent-container<\/span>><\/span> agent diagnose\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
For Kubernetes:<\/p>\n\n\n
kubectl exec -it <datadog-agent-pod<\/span>><\/span> -n datadog -- agent status\nkubectl exec -it <datadog-agent-pod<\/span>><\/span> -n datadog -- agent health\nkubectl exec -it <datadog-agent-pod<\/span>><\/span> -n datadog -- agent configcheck\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Datadog documents datadog-agent status<\/code>, configcheck<\/code>, diagnose<\/code>, flare<\/code>, health<\/code>, hostname<\/code>, stream-logs<\/code>, and check-specific commands as core troubleshooting commands. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
2.5 Check Agent logs<\/h2>\n\n\n\n
Linux:<\/p>\n\n\n
sudo journalctl -u datadog-agent.service -r\nsudo tail -f \/var<\/span>\/log\/datadog\/agent.log\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Kubernetes:<\/p>\n\n\n
kubectl logs -n datadog <datadog-agent-pod<\/span>><\/span>\nkubectl logs -n datadog <datadog-cluster-agent-pod<\/span>><\/span>\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Check for:<\/p>\n\n\n
API key invalid\nconnection refused\ntimeout\nproxy error\nTLS error\ninvalid YAML\npermission denied\nunable to resolve hostname\ncheck failed\nlogs-agent stopped\ntrace-agent stopped\nprocess-agent stopped\n<\/code><\/span><\/pre>\n\n\n
For systemd systems, Datadog recommends systemctl status datadog-agent<\/code> and journalctl -u datadog-agent.service<\/code> when the Agent fails to start. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
3. Datadog Agent Troubleshooting<\/h1>\n\n\n\n
3.1 Agent not starting<\/h2>\n\n\n\n
Symptoms<\/h3>\n\n\n
datadog-agent.service failed\nAgent container CrashLoopBackOff\nNo host appears in Infrastructure List<\/span>\nNo system.cpu\/system.mem metrics\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Check<\/h3>\n\n\n\n
Linux:<\/p>\n\n\n
sudo<\/span> systemctl<\/span> status<\/span> datadog-agent<\/span>\nsudo<\/span> journalctl<\/span> -u<\/span> datadog-agent<\/span>.service<\/span> -r<\/span>\nsudo<\/span> datadog-agent<\/span> status<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Kubernetes:<\/p>\n\n\n
kubectl get pods -n datadog\nkubectl describe pod <agent-pod<\/span>><\/span> -n datadog\nkubectl logs <agent-pod<\/span>><\/span> -n datadog\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Common causes<\/h3>\n\n\n\n
Cause<\/th>Fix<\/th><\/tr><\/thead>
Invalid YAML<\/td>Run datadog-agent configcheck<\/code><\/td><\/tr>
Missing API key<\/td>Verify Secret or datadog.yaml<\/code><\/td><\/tr>
Wrong site<\/td>Set correct DD_SITE<\/code><\/td><\/tr>
Permission denied<\/td>Check file permissions, container securityContext<\/td><\/tr>
Hostname detection failure<\/td>Set hostname or fix cloud metadata access<\/td><\/tr>
Proxy not configured<\/td>Add proxy config<\/td><\/tr>
Duplicate Agent<\/td>Ensure only one Agent per host\/node<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
The official Agent checklist explicitly calls out hostname detection, proxy access, API key, site, duplicate Agents, restart after YAML changes, status command, logs, debug mode, and Agent flare. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
3.2 Agent installed but host not visible<\/h2>\n\n\n\n
Check the basic metrics<\/h3>\n\n\n\n
In Datadog Metrics Explorer, query:<\/p>\n\n\n
system<\/span>.cpu<\/span>.user<\/span>\nsystem<\/span>.mem<\/span>.used<\/span>\ndatadog<\/span>.agent<\/span>.running<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
On host:<\/p>\n\n\n
sudo datadog-agent status\nsudo datadog-agent diagnose\nsudo datadog-agent hostname\n<\/code><\/span><\/pre>\n\n\n
Likely causes<\/h3>\n\n\n\n
Cause<\/th>Explanation<\/th><\/tr><\/thead>
Wrong API key<\/td>Agent sends to invalid org<\/td><\/tr>
Wrong Datadog site<\/td>Agent sends to wrong region<\/td><\/tr>
Network blocked<\/td>Intake unreachable<\/td><\/tr>
Clock skew<\/td>Timestamps too far in past\/future<\/td><\/tr>
Hostname conflict<\/td>Multiple hosts collapse into one<\/td><\/tr>
Duplicate Agent<\/td>Data conflict or host confusion<\/td><\/tr>
Host filter<\/td>You are filtering wrong tags in UI<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Fix pattern<\/h3>\n\n\n\n
    \n
  1. Confirm Agent service is running.<\/li>\n\n\n\n
  2. Confirm API key and site.<\/li>\n\n\n\n
  3. Confirm outbound HTTPS access.<\/li>\n\n\n\n
  4. Confirm hostname.<\/li>\n\n\n\n
  5. Confirm correct UI timeframe.<\/li>\n\n\n\n
  6. Search by hostname, cloud instance ID, Kubernetes node name, and tags.<\/li>\n<\/ol>\n\n\n\n
    \n\n\n\n
    3.3 Agent running but integration metrics missing<\/h2>\n\n\n\n
    Example: Agent metrics exist, but PostgreSQL, Redis, NGINX, Kafka, MySQL, or JVM metrics are missing.<\/p>\n\n\n\n
    Check<\/h3>\n\n\n
    sudo datadog-agent status\nsudo datadog-agent configcheck\nsudo datadog-agent check <integration_name<\/span>><\/span>\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
    Examples:<\/p>\n\n\n
    sudo datadog-agent check postgres\nsudo datadog-agent check redisdb\nsudo datadog-agent check nginx\nsudo datadog-agent check kafka\n<\/code><\/span><\/pre>\n\n\n
    What to inspect in status output<\/h3>\n\n\n\n
    Look under:<\/p>\n\n\n
    Running Checks\n===========\npostgres\nredisdb\nnginx\nkafka\n<\/code><\/span><\/pre>\n\n\n
    A healthy check should show metric samples, no critical errors, and recent successful runs. Datadog\u2019s command reference notes that properly configured integrations appear under Running Checks<\/code> with no warnings or errors. (Datadog<\/a>)<\/p>\n\n\n\n
    Common causes<\/h3>\n\n\n\n
    Cause<\/th>Fix<\/th><\/tr><\/thead>
    Wrong integration config path<\/td>Use correct conf.d\/<check>.d\/conf.yaml<\/code><\/td><\/tr>
    Bad YAML indentation<\/td>Validate YAML<\/td><\/tr>
    Wrong host\/port<\/td>Test with curl<\/code>, nc<\/code>, psql<\/code>, redis-cli<\/code><\/td><\/tr>
    Missing credentials<\/td>Verify secret injection<\/td><\/tr>
    DB user lacks permissions<\/td>Grant monitoring permissions<\/td><\/tr>
    TLS required<\/td>Add TLS parameters<\/td><\/tr>
    Check disabled<\/td>Enable integration<\/td><\/tr>
    Autodiscovery labels wrong<\/td>Fix pod annotations\/labels<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    \n\n\n\n
    3.4 Agent high CPU or memory<\/h2>\n\n\n\n
    Check<\/h3>\n\n\n
    top\nps aux | grep datadog\nsudo datadog-agent status\nsudo datadog-agent health\n<\/code><\/span><\/pre>\n\n\n
    In Kubernetes:<\/p>\n\n\n
    kubectl top pod -n datadog\nkubectl describe pod <agent-pod<\/span>><\/span> -n datadog\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
    Common causes<\/h3>\n\n\n\n
    Cause<\/th>Why It Happens<\/th><\/tr><\/thead>
    Too many logs<\/td>Agent tails too many files or multiline logs are expensive<\/td><\/tr>
    High-cardinality metrics<\/td>Too many tag combinations<\/td><\/tr>
    Heavy integrations<\/td>JMX, database, Kubernetes, or custom checks too frequent<\/td><\/tr>
    Process Agent enabled broadly<\/td>Process collection can be noisy<\/td><\/tr>
    Network Performance Monitoring<\/td>System-probe needs resources<\/td><\/tr>
    Security modules<\/td>Workload\/security monitoring adds overhead<\/td><\/tr>
    Debug logging enabled<\/td>Huge log volume<\/td><\/tr>
    Large Kubernetes cluster<\/td>Too much metadata, events, pods, containers<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    Fix<\/h3>\n\n\n\n
    Reduce collection scope, increase Agent resources, tune log exclusions, reduce check frequency, avoid unnecessary debug logs, control tags, and use Cluster Agent for Kubernetes metadata and cluster-level functions.<\/p>\n\n\n\n
    \n\n\n\n
    3.5 Agent debug mode<\/h2>\n\n\n\n
    Use debug mode only temporarily.<\/p>\n\n\n\n
    Why?<\/p>\n\n\n\n
    Because debug mode increases log volume and may expose configuration details.<\/p>\n\n\n\n
    Typical approach:<\/p>\n\n\n
    sudo datadog-agent config set<\/span> log_level debug\nsudo systemctl restart datadog-agent\nsudo tail -f \/var\/log\/datadog\/agent.log\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
    After troubleshooting:<\/p>\n\n\n
    sudo datadog-agent config set<\/span> log_level info\nsudo systemctl restart datadog-agent\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
    \n\n\n\n
    3.6 Agent flare<\/h2>\n\n\n\n
    Use a flare when you need to send diagnostic data to Datadog Support.<\/p>\n\n\n
    sudo datadog-agent flare\n<\/code><\/span><\/pre>\n\n\n
    In Kubernetes:<\/p>\n\n\n
    kubectl exec -it <agent-pod<\/span>><\/span> -n datadog -- agent flare\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
    A flare gathers Agent configuration files and logs into an archive and removes sensitive information such as API keys, passwords, proxy credentials, and SNMP community strings. (Datadog<\/a>)<\/p>\n\n\n\n
    \n\n\n\n
    4. Kubernetes Datadog Troubleshooting<\/h1>\n\n\n\n
    4.1 Recommended Kubernetes installation approach<\/h2>\n\n\n\n
    For Kubernetes, prefer:<\/p>\n\n\n\n
      \n
    1. Datadog Operator, or<\/li>\n\n\n\n
    2. Helm chart.<\/li>\n<\/ol>\n\n\n\n
      Avoid manually maintaining large raw DaemonSet YAML unless you have a strict reason.<\/p>\n\n\n\n
      Datadog describes the Operator as the recommended Kubernetes installation method because it reports deployment status, health, and errors in the custom resource status and reduces misconfiguration risk through higher-level configuration options. (Datadog<\/a>)<\/p>\n\n\n\n
      \n\n\n\n
      4.2 Kubernetes architecture<\/h2>\n\n\n
      flowchart<\/span> TB<\/span>\n    subgraph<\/span> K8S<\/span>[Kubernetes Cluster]<\/span>\n        A<\/span>[Datadog Agent DaemonSet<br>one per node]<\/span>\n        B<\/span>[Datadog Cluster Agent<br>cluster-level metadata]<\/span>\n        C<\/span>[Kubelet \/ Container Runtime]<\/span>\n        D<\/span>[Kubernetes API Server]<\/span>\n        E<\/span>[Application Pods]<\/span>\n    end<\/span>\n\n    E<\/span> --<\/span>> A<\/span>\n    C<\/span> --<\/span>> A<\/span>\n    D<\/span> --<\/span>> B<\/span>\n    B<\/span> --<\/span>> A<\/span>\n    A<\/span> --<\/span>> F<\/span>[Datadog Intake]<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
      \n\n\n\n
      4.3 Agent pod not running<\/h2>\n\n\n\n
      Check<\/h3>\n\n\n
      kubectl get pods -n datadog -o wide\nkubectl describe pod <agent-pod<\/span>><\/span> -n datadog\nkubectl logs <agent-pod<\/span>><\/span> -n datadog\nkubectl get daemonset -n datadog\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
      Common causes<\/h3>\n\n\n\n
      Symptom<\/th>Cause<\/th>Fix<\/th><\/tr><\/thead>
      ImagePullBackOff<\/td>Registry blocked or wrong image<\/td>Check image registry, pull secret, egress<\/td><\/tr>
      CrashLoopBackOff<\/td>Bad config, missing key, hostname issue<\/td>Read logs and config<\/td><\/tr>
      Pending<\/td>No node resources or taints<\/td>Add tolerations\/resources<\/td><\/tr>
      Permission denied<\/td>PSP\/PSA\/securityContext<\/td>Fix RBAC\/securityContext<\/td><\/tr>
      Cannot mount volumes<\/td>Host path restricted<\/td>Check node filesystem and policies<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
      \n\n\n\n
      4.4 Cluster Agent not working<\/h2>\n\n\n\n
      Check<\/h3>\n\n\n
      kubectl get<\/span> deploy -n datadog\nkubectl get<\/span> pods -n datadog | grep cluster\nkubectl logs -n datadog deploy\/datadog-cluster-agent\nkubectl exec -n datadog deploy\/datadog-cluster-agent -- datadog-cluster-agent status\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
      Common causes<\/h3>\n\n\n\n
      Cause<\/th>Fix<\/th><\/tr><\/thead>
      Missing RBAC<\/td>Reinstall\/fix Helm or Operator manifests<\/td><\/tr>
      Token mismatch between Agent and Cluster Agent<\/td>Check shared token secret<\/td><\/tr>
      API server blocked<\/td>Check NetworkPolicy<\/td><\/tr>
      Wrong namespace<\/td>Check install namespace<\/td><\/tr>
      Version mismatch<\/td>Upgrade Agent\/Cluster Agent together<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
      \n\n\n\n
      4.5 Kubernetes metrics missing<\/h2>\n\n\n\n
      Check:<\/p>\n\n\n
      kubectl<\/span> get<\/span> pods<\/span> -n<\/span> datadog<\/span>\nkubectl<\/span> logs<\/span> -n<\/span> datadog<\/span> <agent-pod<\/span>>\nkubectl<\/span> logs<\/span> -n<\/span> datadog<\/span> <cluster-agent-pod<\/span>>\nkubectl<\/span> auth<\/span> can-i<\/span> get<\/span> pods<\/span> --as<\/span> system<\/span>:serviceaccount<\/span>:datadog<\/span>:datadog-agent<\/span>\nkubectl<\/span> auth<\/span> can-i<\/span> list<\/span> nodes<\/span> --as<\/span> system<\/span>:serviceaccount<\/span>:datadog<\/span>:datadog-agent<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
      Query in Datadog:<\/p>\n\n\n
      kubernetes<\/span>.cpu<\/span>.usage<\/span>.total<\/span>\nkubernetes<\/span>.memory<\/span>.usage<\/span>\nkubernetes<\/span>.pods<\/span>.running<\/span>\nkubernetes_state<\/span>.pod<\/span>.status_phase<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
      Possible issues:<\/p>\n\n\n\n
      Issue<\/th>Explanation<\/th><\/tr><\/thead>
      Kubelet auth problem<\/td>Agent cannot scrape kubelet<\/td><\/tr>
      RBAC missing<\/td>Agent cannot list Kubernetes resources<\/td><\/tr>
      Kube-state-metrics disabled\/missing<\/td>kubernetes_state.*<\/code> missing<\/td><\/tr>
      Cluster Agent unhealthy<\/td>Metadata and external metrics affected<\/td><\/tr>
      Tags missing<\/td>Metrics exist but not under expected tags<\/td><\/tr>
      Wrong cluster name<\/td>Searching wrong kube_cluster_name<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
      \n\n\n\n
      4.6 Kubernetes log collection missing<\/h2>\n\n\n\n
      Datadog recommends file-based log collection in Kubernetes because it scales better for ephemeral pods and is more resource-efficient than Docker socket polling. (Datadog<\/a>)<\/p>\n\n\n\n
      Check Agent settings<\/h3>\n\n\n\n
      For Helm\/Operator, confirm logs are enabled:<\/p>\n\n\n
      logs:\n  enabled: true<\/span>\n  containerCollectAll<\/span>: true<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
      Or equivalent Operator config.<\/p>\n\n\n\n
      Commands<\/h3>\n\n\n
      kubectl exec -it <agent-pod<\/span>><\/span> -n datadog -- agent status\nkubectl exec -it <agent-pod<\/span>><\/span> -n datadog -- agent stream-logs\nkubectl logs <application-pod<\/span>><\/span> -n <namespace<\/span>><\/span>\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
      Common causes<\/h3>\n\n\n\n
      Cause<\/th>Fix<\/th><\/tr><\/thead>
      Logs not enabled<\/td>Enable logs in Helm\/Operator<\/td><\/tr>
      App logs to file, not stdout<\/td>Configure file log collection<\/td><\/tr>
      Autodiscovery annotation wrong<\/td>Fix pod annotation<\/td><\/tr>
      Exclusion rule dropping logs<\/td>Check DD_CONTAINER_EXCLUDE_LOGS<\/code><\/td><\/tr>
      Log pipeline drops or remaps badly<\/td>Check pipelines<\/td><\/tr>
      Index excludes logs<\/td>Check log indexes<\/td><\/tr>
      Timeframe wrong<\/td>Search wider time window<\/td><\/tr>
      Permissions issue on log files<\/td>Fix hostPath\/securityContext<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
      Datadog\u2019s container log troubleshooting explains that container logs are collected either file-based or socket-based, with file-based collection being the recommended Kubernetes default. (Datadog<\/a>)<\/p>\n\n\n\n
      \n\n\n\n
      4.7 APM traces missing from Kubernetes apps<\/h2>\n\n\n\n
      This is one of the most common production problems.<\/p>\n\n\n\n
      Data path<\/h3>\n\n\n
      flowchart<\/span> LR<\/span>\n    A<\/span>[Application Pod with Datadog tracer]<\/span> --<\/span>> B<\/span>[Datadog Agent on node]<\/span>\n    B<\/span> --<\/span>> C<\/span>[Trace Agent :8126]<\/span>\n    C<\/span> --<\/span>> D<\/span>[Datadog APM Intake]<\/span>\n    D<\/span> --<\/span>> E<\/span>[Trace Explorer \/ Service Catalog]<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
      Check in app pod<\/h3>\n\n\n
      env | grep DD_\n<\/code><\/span><\/pre>\n\n\n
      Important variables:<\/p>\n\n\n
      DD_ENV\nDD_SERVICE\nDD_VERSION\nDD_AGENT_HOST\nDD_TRACE_AGENT_PORT\nDD_TRACE_ENABLED\nDD_LOGS_INJECTION\n<\/code><\/span><\/pre>\n\n\n
      Common Kubernetes issue<\/h3>\n\n\n\n
      Many apps default to sending traces to:<\/p>\n\n\n
      localhost<\/span>:8126<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
      But inside a Kubernetes application container, localhost<\/code> means the application pod itself, not the Datadog Agent pod.<\/p>\n\n\n\n
      Typical fix:<\/p>\n\n\n
      env:\n  - name: DD_AGENT_HOST\n    valueFrom<\/span>:\n      fieldRef:\n        fieldPath: status.hostIP\n  - name: DD_TRACE_AGENT_PORT\n    value<\/span>: \"8126\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
      Also make sure APM is enabled in the Agent.<\/p>\n\n\n\n
      \n\n\n\n
      5. Log Management Troubleshooting<\/h1>\n\n\n\n
      5.1 Logs not appearing in Datadog<\/h2>\n\n\n\n
      First question<\/h3>\n\n\n\n
      Are logs missing from:<\/p>\n\n\n\n
        \n
      1. The application?<\/li>\n\n\n\n
      2. The Agent?<\/li>\n\n\n\n
      3. Datadog intake?<\/li>\n\n\n\n
      4. Log Explorer?<\/li>\n\n\n\n
      5. A specific index?<\/li>\n\n\n\n
      6. A specific dashboard\/monitor?<\/li>\n<\/ol>\n\n\n\n
        This distinction matters.<\/p>\n\n\n\n
        Check source<\/h3>\n\n\n
        kubectl logs <pod<\/span>><\/span> -n <namespace<\/span>><\/span>\ndocker logs <container<\/span>><\/span>\ntail -f \/path\/to\/app.log\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
        If logs are not visible here, Datadog cannot collect them.<\/p>\n\n\n\n
        Check Agent<\/h3>\n\n\n
        sudo datadog-agent status\nsudo datadog-agent stream-logs\nsudo tail -f \/var<\/span>\/log\/datadog\/agent.log\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
        Kubernetes:<\/p>\n\n\n
        kubectl exec -it <agent-pod<\/span>><\/span> -n datadog -- agent status\nkubectl exec -it <agent-pod<\/span>><\/span> -n datadog -- agent stream-logs\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
        Datadog\u2019s log troubleshooting documentation focuses on missing logs, data access, timestamp misalignment, ingestion\/processing issues, and index\/exclusion behavior. (Datadog<\/a>)<\/p>\n\n\n\n
        \n\n\n\n
        5.2 Logs are collected but not searchable<\/h2>\n\n\n\n
        Likely cause: logs are ingested but not indexed.<\/p>\n\n\n\n
        Check:<\/p>\n\n\n\n
        Area<\/th>What to Check<\/th><\/tr><\/thead>
        Log indexes<\/td>Does index filter match the log?<\/td><\/tr>
        Index order<\/td>Is another index\/exclusion catching it first?<\/td><\/tr>
        Exclusion filters<\/td>Are logs sampled or dropped?<\/td><\/tr>
        Retention<\/td>Is the log outside retention?<\/td><\/tr>
        Permissions<\/td>Does user have access to index?<\/td><\/tr>
        Time field<\/td>Is timestamp wrong?<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
        Datadog usage metrics such as datadog.estimated_usage.logs.ingested_events<\/code> can be filtered by tags like datadog_index<\/code>, datadog_is_excluded<\/code>, service<\/code>, and status<\/code>; if datadog_index:N\/A<\/code>, the logs may not match any index. (Datadog<\/a>)<\/p>\n\n\n\n
        \n\n\n\n
        5.3 Logs have wrong timestamp<\/h2>\n\n\n\n
        Symptoms:<\/p>\n\n\n
        Logs appear in<\/span> future\nLogs appear hours late\nLogs disappear from<\/span> current view\nLog monitor does not trigger\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
        Check:<\/p>\n\n\n\n
        Cause<\/th>Fix<\/th><\/tr><\/thead>
        App emits local timezone without timezone offset<\/td>Emit ISO 8601 UTC<\/td><\/tr>
        Custom timestamp parser wrong<\/td>Fix pipeline date parser<\/td><\/tr>
        Host time skew<\/td>Fix NTP<\/td><\/tr>
        Container image timezone mismatch<\/td>Use UTC<\/td><\/tr>
        Logs delayed by batch pipeline<\/td>Widen search window<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
        Best practice: emit logs in structured JSON with an explicit UTC timestamp.<\/p>\n\n\n\n
        \n\n\n\n
        5.4 Logs missing service<\/code>, env<\/code>, or version<\/code><\/h2>\n\n\n\n
        This breaks correlation.<\/p>\n\n\n\n
        Check:<\/p>\n\n\n
        service<\/span>\nenv<\/span>\nversion<\/span>\ndd<\/span>.service<\/span>\ndd<\/span>.env<\/span>\ndd<\/span>.version<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
        Kubernetes fix:<\/p>\n\n\n
        labels:\n  tags.datadoghq.com\/env: prod\n  tags.datadoghq.com\/service: checkout\n  tags.datadoghq.com\/version: \"1.2.3\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
        Or environment variables:<\/p>\n\n\n
        env<\/span>:\n  -<\/span> name<\/span>: DD_ENV<\/span>\n    value<\/span>: prod<\/span>\n  -<\/span> name<\/span>: DD_SERVICE<\/span>\n    value<\/span>: checkout<\/span>\n  -<\/span> name<\/span>: DD_VERSION<\/span>\n    value<\/span>: \"1.2<\/span>.3<\/span>\"\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
        Unified Service Tagging is one of the most important Datadog design requirements because it correlates metrics, logs, traces, deployments, services, and versions.<\/p>\n\n\n\n
        \n\n\n\n
        5.5 Multiline logs are broken<\/h2>\n\n\n\n
        Symptoms:<\/p>\n\n\n
        Stack traces split into many log events\nJava\/Python exceptions appear as<\/span> many separate logs\nError<\/span> Tracking does not group correctly\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
        Fix by configuring multiline processing.<\/p>\n\n\n\n
        Example concept:<\/p>\n\n\n
        logs:\n  - type: file\n    path<\/span>: \/var\/<\/span>log\/app.log\n    service<\/span>: my-service\n    source<\/span>: java\n    log_processing_rules<\/span>:\n      - type: multi_line\n        name<\/span>: new_log_start_with_date\n        pattern<\/span>: \\d{4<\/span>}-\\d{2<\/span>}-\\d{2<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
        Do not overuse multiline rules; they can increase Agent CPU usage.<\/p>\n\n\n\n
        \n\n\n\n
        5.6 Log pipeline parsing issue<\/h2>\n\n\n\n
        Symptoms:<\/p>\n\n\n
        status is wrong\nservice is wrong\nJSON<\/span> not parsed\nfields are missing\nseverity not recognized\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
        Check:<\/p>\n\n\n\n
          \n
        1. Raw log event.<\/li>\n\n\n\n
        2. Pipeline order.<\/li>\n\n\n\n
        3. Grok\/parser rule.<\/li>\n\n\n\n
        4. Attribute remapper.<\/li>\n\n\n\n
        5. Status remapper.<\/li>\n\n\n\n
        6. Service remapper.<\/li>\n\n\n\n
        7. Date remapper.<\/li>\n<\/ol>\n\n\n\n
          Best practice:<\/p>\n\n\n\n
          Emit structured JSON:<\/p>\n\n\n
          {\n  \"timestamp\"<\/span>: \"2026-06-23T10:15:00Z\"<\/span>,\n  \"level\"<\/span>: \"ERROR\"<\/span>,\n  \"service\"<\/span>: \"checkout\"<\/span>,\n  \"env\"<\/span>: \"prod\"<\/span>,\n  \"message\"<\/span>: \"Payment failed\"<\/span>,\n  \"order_id\"<\/span>: \"12345\"<\/span>\n}\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
          \n\n\n\n
          6. APM \/ Trace Troubleshooting<\/h1>\n\n\n\n
          6.1 APM components<\/h2>\n\n\n\n
          APM depends on three things:<\/p>\n\n\n
          flowchart<\/span> LR<\/span>\n    A<\/span>[Application Code]<\/span> --<\/span>> B<\/span>[Datadog Tracer \/ SDK]<\/span>\n    B<\/span> --<\/span>> C<\/span>[Datadog Agent Trace Agent]<\/span>\n    C<\/span> --<\/span>> D<\/span>[Datadog APM Intake]<\/span>\n    D<\/span> --<\/span>> E<\/span>[Trace Explorer \/ Services \/ APM Metrics]<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          Datadog\u2019s APM troubleshooting docs cover trace retention, trace metrics, services, data volume, connection errors, resource usage, security, tracer startup logs, tracer debug logs, and SDK configuration. (Datadog<\/a>)<\/p>\n\n\n\n
          \n\n\n\n
          6.2 No traces in Datadog<\/h2>\n\n\n\n
          Check 1: Is application instrumented?<\/h3>\n\n\n\n
          Java:<\/p>\n\n\n
          java -javaagent:\/path\/<\/span>dd-java-agent.jar -jar app.jar\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Python:<\/p>\n\n\n
          ddtrace-run<\/span> python<\/span> app<\/span>.py<\/span>\nddtrace-run<\/span> --info<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          Node.js:<\/p>\n\n\n
          require<\/span>('dd-trace'<\/span>).init()\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Go:<\/p>\n\n\n\n
          Use Datadog tracing libraries and wrap supported frameworks.<\/p>\n\n\n\n
          .NET:<\/p>\n\n\n\n
          Check tracer installation and environment variables.<\/p>\n\n\n\n
          \n\n\n\n
          Check 2: Is the tracer sending to the Agent?<\/h3>\n\n\n\n
          From app container:<\/p>\n\n\n
          curl -v http:\/\/${DD_AGENT_HOST}:8126\/info<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          If this fails, fix networking first.<\/p>\n\n\n\n
          Check 3: Is Agent APM enabled?<\/h3>\n\n\n\n
          Agent status should show APM \/ trace-agent.<\/p>\n\n\n
          sudo datadog-agent status\n<\/code><\/span><\/pre>\n\n\n
          Kubernetes:<\/p>\n\n\n
          kubectl exec -it <agent-pod<\/span>><\/span> -n datadog -- agent status\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
          Look for:<\/p>\n\n\n
          APM Agent\n=========\nStatus: Running\n<\/code><\/span><\/pre>\n\n\n
          \n\n\n\n
          6.3 Traces reach Agent but not Datadog<\/h2>\n\n\n\n
          Check Agent logs:<\/p>\n\n\n
          sudo tail -f \/var<\/span>\/log\/datadog\/trace-agent.log\nsudo tail -f \/var<\/span>\/log\/datadog\/agent.log\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
          Kubernetes:<\/p>\n\n\n
          kubectl logs <agent-pod<\/span>><\/span> -n datadog | grep -i trace\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
          Common causes:<\/p>\n\n\n\n
          Cause<\/th>Fix<\/th><\/tr><\/thead>
          Wrong API key\/site<\/td>Correct Agent config<\/td><\/tr>
          Datadog intake blocked<\/td>Fix egress<\/td><\/tr>
          Trace-agent disabled<\/td>Enable APM<\/td><\/tr>
          Payload too large<\/td>Reduce tags\/spans<\/td><\/tr>
          Sampling\/retention confusion<\/td>Check ingestion and retention<\/td><\/tr>
          Malformed spans<\/td>Upgrade tracer<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
          \n\n\n\n
          6.4 Service missing from APM Service Catalog<\/h2>\n\n\n\n
          Possible causes:<\/p>\n\n\n\n
          Cause<\/th>Explanation<\/th><\/tr><\/thead>
          No traffic<\/td>Tracer only sends spans when instrumented requests happen<\/td><\/tr>
          Wrong service name<\/td>DD_SERVICE<\/code> missing or auto-detected unexpectedly<\/td><\/tr>
          Wrong env<\/td>Looking at env:prod<\/code>, data sent to env:staging<\/code><\/td><\/tr>
          Sampling<\/td>Few\/no retained traces<\/td><\/tr>
          Unsupported framework<\/td>Auto-instrumentation may not cover it<\/td><\/tr>
          Agent not reachable<\/td>Tracer cannot connect to Agent<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
          Check:<\/p>\n\n\n
          env | grep DD_\n<\/code><\/span><\/pre>\n\n\n
          Important:<\/p>\n\n\n
          DD_SERVICE\nDD_ENV\nDD_VERSION\nDD_TAGS\nDD_SERVICE_MAPPING\n<\/code><\/span><\/pre>\n\n\n
          Datadog tracer configuration uses DD_SERVICE<\/code>, DD_ENV<\/code>, DD_VERSION<\/code>, DD_TAGS<\/code>, and service mapping variables across language libraries, with language-specific fallback behavior if service is not explicitly configured. (Datadog<\/a>)<\/p>\n\n\n\n
          \n\n\n\n
          6.5 Too many services in APM<\/h2>\n\n\n\n
          Symptoms:<\/p>\n\n\n
          checkout-prod\ncheckout-stage\ncheckout-v1\ncheckout-worker-123\npostgres-prod\nredis-checkout-prod\n<\/code><\/span><\/pre>\n\n\n
          Root problem: environment, version, host, region, or shard has been included in the service name.<\/p>\n\n\n\n
          Bad:<\/p>\n\n\n
          checkout-prod\ncheckout-uat\ncheckout-v1-2-3\n<\/code><\/span><\/pre>\n\n\n
          Good:<\/p>\n\n\n
          service<\/span>:checkout<\/span>\nenv<\/span>:prod<\/span>\nversion<\/span>:1.2.3<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
          Use env<\/code>, version<\/code>, team<\/code>, region<\/code>, and other tags instead of encoding them in the service name.<\/p>\n\n\n\n
          \n\n\n\n
          6.6 Trace exists but related logs are missing<\/h2>\n\n\n\n
          To correlate logs and traces:<\/p>\n\n\n\n
            \n
          1. Logs must be collected.<\/li>\n\n\n\n
          2. Tracer must inject trace\/span IDs into logs.<\/li>\n\n\n\n
          3. Logs must include service<\/code>, env<\/code>, and version<\/code>.<\/li>\n\n\n\n
          4. Log parser must preserve trace fields.<\/li>\n\n\n\n
          5. Logs must be indexed\/searchable.<\/li>\n<\/ol>\n\n\n\n
            Check fields:<\/p>\n\n\n
            dd<\/span>.trace_id<\/span>\ndd<\/span>.span_id<\/span>\ntrace_id<\/span>\nspan_id<\/span>\nservice<\/span>\nenv<\/span>\nversion<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
            Enable:<\/p>\n\n\n
            DD_LOGS_INJECTION=true<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
            Datadog supports correlating traces with logs, RUM, synthetics, profiles, and DBM when the telemetry carries the right metadata. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            6.7 Trace metrics missing or confusing<\/h2>\n\n\n\n
            APM generates trace metrics such as request count, error count, latency, and Apdex.<\/p>\n\n\n\n
            Common confusion:<\/p>\n\n\n\n
            Situation<\/th>Explanation<\/th><\/tr><\/thead>
            Trace Explorer shows spans but monitor metric does not<\/td>Trace Explorer and trace metrics can have different retention\/query behavior<\/td><\/tr>
            p95 latency differs between views<\/td>Different aggregation, filters, or sampling<\/td><\/tr>
            No trace.*<\/code> metric<\/td>Service may not be instrumented or no traffic<\/td><\/tr>
            Service appears but no endpoint<\/td>Resource naming issue<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Datadog\u2019s APM troubleshooting separates trace retention, trace metrics, services, ingestion\/sampling, and retention filters because these are related but not identical views of tracing data. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            6.8 APM high cost or high trace volume<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Problem<\/th>Fix<\/th><\/tr><\/thead>
            Too many spans<\/td>Reduce unnecessary custom spans<\/td><\/tr>
            High-cardinality span tags<\/td>Remove user ID, request ID, session ID from indexed tags<\/td><\/tr>
            Excessive error traces<\/td>Fix app or tune sampling<\/td><\/tr>
            Background jobs too noisy<\/td>Sample lower<\/td><\/tr>
            Too many services\/resources<\/td>Normalize service\/resource names<\/td><\/tr>
            Health checks traced<\/td>Drop or filter health check traces<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Use ingestion controls, sampling, retention filters, and service\/resource normalization.<\/p>\n\n\n\n
            \n\n\n\n
            6.9 Single Step Instrumentation troubleshooting<\/h2>\n\n\n\n
            Single Step Instrumentation can inject tracers automatically for supported environments and languages, but troubleshooting should confirm whether injection actually happened.<\/p>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Area<\/th>What to verify<\/th><\/tr><\/thead>
            Agent version<\/td>SSI requires supported Agent version<\/td><\/tr>
            Language support<\/td>Java, Python, Node.js, PHP, .NET etc. depending on setup<\/td><\/tr>
            Process instrumentation<\/td>Was tracer injected into the process?<\/td><\/tr>
            Environment<\/td>Linux hosts, containers, Kubernetes supported scenarios<\/td><\/tr>
            Fleet Automation<\/td>Use instrumentation insights if available<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Datadog\u2019s SSI troubleshooting page says injection issues can be checked through Fleet Automation or manually at container level; process-level insights show instrumentation status and SDK installation details, and current docs mention Agent v7.68.2+ for that SSI troubleshooting flow. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            7. DogStatsD and Custom Metrics Troubleshooting<\/h1>\n\n\n\n
            7.1 Custom metric not visible<\/h2>\n\n\n\n
            Data path<\/h3>\n\n\n
            flowchart<\/span> LR<\/span>\n    A<\/span>[Application]<\/span> --<\/span>> B<\/span>[DogStatsD client]<\/span>\n    B<\/span> --<\/span>> C<\/span>[Datadog Agent DogStatsD :8125]<\/span>\n    C<\/span> --<\/span>> D<\/span>[Datadog Metrics Intake]<\/span>\n    D<\/span> --<\/span>> E<\/span>[Metrics Explorer]<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
            Check<\/h3>\n\n\n
            sudo datadog-agent status\nsudo datadog-agent configcheck\n<\/code><\/span><\/pre>\n\n\n
            Look for DogStatsD section.<\/p>\n\n\n\n
            Test UDP locally:<\/p>\n\n\n
            echo<\/span> -n \"custom.test.metric:1|c|#env:dev,service:test\"<\/span> | nc -4<\/span>u -w1 127.0<\/span>.0<\/span>.1<\/span> 8125<\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
            Then search:<\/p>\n\n\n
            custom<\/span>.test<\/span>.metric<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
            \n\n\n\n
            7.2 DogStatsD issue in Kubernetes<\/h2>\n\n\n\n
            Common problem: app sends metrics to localhost:8125<\/code>, but DogStatsD is in the Datadog Agent pod on the node.<\/p>\n\n\n\n
            Fix patterns:<\/p>\n\n\n\n
            Option 1: Use host IP<\/h3>\n\n\n
            env<\/span>:\n  -<\/span> name<\/span>: DD_AGENT_HOST<\/span>\n    valueFrom<\/span>:\n      fieldRef<\/span>:\n        fieldPath<\/span>: status<\/span>.hostIP<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
            Option 2: Use Unix Domain Socket<\/h3>\n\n\n\n
            Better for local node-level communication if configured.<\/p>\n\n\n\n
            Option 3: Use service-based routing<\/h3>\n\n\n\n
            Possible but be careful with load balancing because metrics may be sent to an Agent not on the same node.<\/p>\n\n\n\n
            \n\n\n\n
            7.3 Custom metrics cost\/cardinality issue<\/h2>\n\n\n\n
            Symptoms:<\/p>\n\n\n
            Custom metrics count exploded\nBill increased\nMetrics Summary shows many tag combinations\nDashboards slow\n<\/code><\/span><\/pre>\n\n\n
            Common bad tags:<\/p>\n\n\n
            user_id\nrequest_id\nsession_id\npod_uid\ncontainer_id\ntimestamp\norder_id\ntransaction_id\n<\/code><\/span><\/pre>\n\n\n
            Better tags:<\/p>\n\n\n
            env\nservice\nversion\nregion\nteam\ncluster\nnamespace<\/span>\nendpoint<\/span>\nstatus<\/span>\nmethod<\/span>\nprovider<\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
            Datadog\u2019s Metrics without Limits decouples custom metric ingestion and indexing, allowing allowlists or blocklists of tags to control which tags remain queryable and reduce billable custom metric volume. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            8. Integration Troubleshooting<\/h1>\n\n\n\n
            8.1 Integration check fails<\/h2>\n\n\n\n
            Example status:<\/p>\n\n\n
            Instance<\/span> ID<\/span>: postgres<\/span>:...<\/span>\nError<\/span>: connection<\/span> refused<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
            Use:<\/p>\n\n\n
            sudo datadog-agent check <integration<\/span>><\/span>\nsudo datadog-agent configcheck\nsudo datadog-agent status\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
            Common integration failure categories:<\/p>\n\n\n\n
            Category<\/th>Example<\/th><\/tr><\/thead>
            Network<\/td>Cannot connect to DB\/cache\/app<\/td><\/tr>
            Auth<\/td>Wrong username\/password\/token<\/td><\/tr>
            Permission<\/td>DB monitoring user lacks grants<\/td><\/tr>
            TLS<\/td>Certificate or CA mismatch<\/td><\/tr>
            YAML<\/td>Bad indentation or wrong key<\/td><\/tr>
            Autodiscovery<\/td>Wrong pod annotation<\/td><\/tr>
            Version<\/td>Integration requires newer Agent<\/td><\/tr>
            Timeout<\/td>Check too heavy or endpoint slow<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Datadog\u2019s integration troubleshooting guide focuses on YAML files, check status, missing metrics, and resolving integration configuration issues through Agent status\/config\/check commands. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            8.2 JMX integration issues<\/h2>\n\n\n\n
            Common for Kafka, Cassandra, JVM apps, Tomcat, Solr, etc.<\/p>\n\n\n\n
            Check:<\/p>\n\n\n
            sudo datadog-agent status\nsudo datadog-agent jmx list<\/span> collected\nsudo datadog-agent jmx list<\/span> matching\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
            Common causes:<\/p>\n\n\n\n
            Cause<\/th>Fix<\/th><\/tr><\/thead>
            JMX port not exposed<\/td>Expose JMX<\/td><\/tr>
            RMI hostname wrong<\/td>Set java.rmi.server.hostname<\/code><\/td><\/tr>
            SSL required<\/td>Configure SSL<\/td><\/tr>
            Auth required<\/td>Add user\/password<\/td><\/tr>
            Too many beans<\/td>Limit collection<\/td><\/tr>
            Wrong bean pattern<\/td>Fix include\/exclude config<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            \n\n\n\n
            8.3 Database integration issue<\/h2>\n\n\n\n
            For PostgreSQL\/MySQL\/etc., check from the Agent host\/pod:<\/p>\n\n\n
            nc -vz <db-host<\/span>><\/span> 5432\npsql -h <db-host<\/span>><\/span> -U <monitor_user<\/span>><\/span> -d <db<\/span>><\/span>\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
            For MySQL:<\/p>\n\n\n
            nc -vz <db-host<\/span>><\/span> 3306\nmysql -h <db-host<\/span>><\/span> -u <monitor_user<\/span>><\/span> -p\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
            Common causes:<\/p>\n\n\n\n
            Cause<\/th>Fix<\/th><\/tr><\/thead>
            DB SG\/firewall blocks Agent<\/td>Allow Agent source<\/td><\/tr>
            Wrong DB endpoint<\/td>Use correct writer\/reader endpoint<\/td><\/tr>
            TLS required<\/td>Configure TLS<\/td><\/tr>
            Monitoring user lacks permissions<\/td>Grant required permissions<\/td><\/tr>
            Password rotation broke secret<\/td>Update secret<\/td><\/tr>
            Agent cannot resolve DNS<\/td>Fix DNS\/VPC\/resolver<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            \n\n\n\n
            9. AWS \/ Cloud Integration Troubleshooting<\/h1>\n\n\n\n
            9.1 AWS integration not working<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Area<\/th>What to Verify<\/th><\/tr><\/thead>
            IAM role<\/td>Trust policy allows Datadog to assume role<\/td><\/tr>
            External ID<\/td>Correct external ID<\/td><\/tr>
            Permissions<\/td>Required AWS permissions attached<\/td><\/tr>
            Regions<\/td>Correct regions enabled<\/td><\/tr>
            Services<\/td>Specific AWS service integration enabled<\/td><\/tr>
            SCP<\/td>Organization SCP is not blocking<\/td><\/tr>
            CloudWatch<\/td>Metrics exist in CloudWatch<\/td><\/tr>
            Delay<\/td>CloudWatch metrics are not always instant<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            A Datadog AWS sts:AssumeRole<\/code> error means the trust policy associated with the Datadog integration role is wrong, and Datadog notes that the error may persist in the UI for a few hours while changes propagate. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            9.2 CloudWatch and Datadog metric values differ<\/h2>\n\n\n\n
            This is common and not always an error.<\/p>\n\n\n\n
            Causes:<\/p>\n\n\n\n
            Cause<\/th>Explanation<\/th><\/tr><\/thead>
            Time aggregation<\/td>Datadog may display per-second values<\/td><\/tr>
            Space aggregation<\/td>AWS and Datadog aggregate dimensions differently<\/td><\/tr>
            Different period<\/td>CloudWatch period may be 5 min vs Datadog 1 min<\/td><\/tr>
            Different filters<\/td>AWS dimensions must match Datadog tags<\/td><\/tr>
            Metric delay<\/td>CloudWatch metrics can be delayed<\/td><\/tr>
            Detailed monitoring<\/td>Some AWS metrics are 1-min only if detailed monitoring is enabled<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Datadog\u2019s AWS troubleshooting guide specifically calls out time aggregation, space aggregation, matching dimensions\/tags, one-minute precision, region, namespace, metric name, and CloudWatch API delays. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            10. API and Terraform Troubleshooting<\/h1>\n\n\n\n
            10.1 API returns 403 Forbidden<\/h2>\n\n\n\n
            Likely causes:<\/p>\n\n\n\n
            Cause<\/th>Fix<\/th><\/tr><\/thead>
            Wrong app key<\/td>Use valid app key<\/td><\/tr>
            App key lacks scope<\/td>Add required scope<\/td><\/tr>
            Key belongs to wrong org<\/td>Use key from same org<\/td><\/tr>
            Service account lacks permission<\/td>Grant role\/scope<\/td><\/tr>
            Wrong site endpoint<\/td>Use correct regional API endpoint<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Remember: API key validates organization intake access; app key controls programmatic actions and permissions.<\/p>\n\n\n\n
            \n\n\n\n
            10.2 API returns 401 Unauthorized<\/h2>\n\n\n\n
            Likely causes:<\/p>\n\n\n\n
            Cause<\/th>Fix<\/th><\/tr><\/thead>
            Missing API key<\/td>Add DD-API-KEY<\/code><\/td><\/tr>
            Missing app key<\/td>Add DD-APPLICATION-KEY<\/code><\/td><\/tr>
            Invalid key<\/td>Rotate\/regenerate<\/td><\/tr>
            Key not propagated yet<\/td>Retry with backoff<\/td><\/tr>
            Wrong site<\/td>Use correct site<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Datadog recommends allowing a brief propagation period for new API\/application keys and using retry with short exponential backoff for transient errors. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            10.3 Terraform provider cannot connect<\/h2>\n\n\n\n
            Check provider config:<\/p>\n\n\n
            provider \"datadog\"<\/span> {\n  api_key = var<\/span>.datadog_api_key\n  app_key = var<\/span>.datadog_app_key\n  api_url = \"https:\/\/api.datadoghq.com\/\"<\/span>\n}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
            For EU\/AP\/etc., use the correct site API URL.<\/p>\n\n\n\n
            Common errors:<\/p>\n\n\n\n
            Error<\/th>Likely Cause<\/th><\/tr><\/thead>
            403<\/td>App key missing scope<\/td><\/tr>
            401<\/td>Bad key<\/td><\/tr>
            404<\/td>Wrong site\/org\/resource<\/td><\/tr>
            429<\/td>API rate limit<\/td><\/tr>
            Provider diff always changing<\/td>Field computed by Datadog or ordering issue<\/td><\/tr>
            Monitor query invalid<\/td>Datadog query syntax problem<\/td><\/tr>
            State drift<\/td>UI changes outside Terraform<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            The Datadog Terraform provider is used to manage Datadog resources such as monitors, dashboards, logs configuration, and integrations through Terraform. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            10.4 Terraform monitor fails validation<\/h2>\n\n\n\n
            Common causes:<\/p>\n\n\n\n
            Cause<\/th>Fix<\/th><\/tr><\/thead>
            Query syntax invalid<\/td>Test query in Datadog UI first<\/td><\/tr>
            Wrong metric name<\/td>Check Metrics Explorer<\/td><\/tr>
            Wrong rollup<\/td>Remove or correct rollup<\/td><\/tr>
            Missing quotes around tags<\/td>Fix syntax<\/td><\/tr>
            No data during validation<\/td>Use require_full_window=false<\/code> if appropriate<\/td><\/tr>
            Invalid notification target<\/td>Check Slack\/PagerDuty integration<\/td><\/tr>
            Restricted role missing<\/td>Check RBAC<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Best practice: manually test query in Datadog Metrics Explorer before putting it into Terraform.<\/p>\n\n\n\n
            \n\n\n\n
            11. Monitor and Alert Troubleshooting<\/h1>\n\n\n\n
            11.1 Monitor not alerting<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Area<\/th>What to Verify<\/th><\/tr><\/thead>
            Query<\/td>Does it return data?<\/td><\/tr>
            Time window<\/td>Is evaluation window too short\/long?<\/td><\/tr>
            Threshold<\/td>Is threshold reachable?<\/td><\/tr>
            Grouping<\/td>Is it grouped by wrong tag?<\/td><\/tr>
            No data behavior<\/td>Is no-data alert disabled?<\/td><\/tr>
            Require full window<\/td>Is window fully populated?<\/td><\/tr>
            Evaluation delay<\/td>Needed for delayed cloud metrics<\/td><\/tr>
            Muting\/downtime<\/td>Monitor muted?<\/td><\/tr>
            Permissions<\/td>Can user see monitor\/data?<\/td><\/tr>
            Notification<\/td>Is destination configured?<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            \n\n\n\n
            11.2 Monitor stuck in No Data<\/h2>\n\n\n\n
            Causes:<\/p>\n\n\n\n
            Cause<\/th>Fix<\/th><\/tr><\/thead>
            Metric stopped reporting<\/td>Check Agent\/source<\/td><\/tr>
            Rollup too sparse<\/td>Adjust rollup\/window<\/td><\/tr>
            Require full window enabled<\/td>Disable if inappropriate<\/td><\/tr>
            Cloud metric delayed<\/td>Add evaluation delay<\/td><\/tr>
            Tag changed<\/td>Fix query tags<\/td><\/tr>
            New group delay<\/td>Tune group delay<\/td><\/tr>
            Logs\/traces not indexed<\/td>Fix indexing\/retention<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Datadog\u2019s No Data troubleshooting warns that rollups and evaluation windows can cause unexpected No Data behavior, especially when Require Full Window is enabled and the rollup interval does not fully populate the evaluation window. (Datadog<\/a>)<\/p>\n\n\n\n
            For AWS\/crawler-based metrics, Datadog recommends using an evaluation delay, commonly at least 900 seconds for AWS CloudWatch delays. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            11.3 Monitor alerts too noisy<\/h2>\n\n\n\n
            Fix strategy:<\/p>\n\n\n\n
            Noise Source<\/th>Fix<\/th><\/tr><\/thead>
            Short spike<\/td>Increase evaluation window<\/td><\/tr>
            New pods alerting immediately<\/td>Add new group delay<\/td><\/tr>
            One bad pod pages whole team<\/td>Use multi-alert with routing<\/td><\/tr>
            Low-priority symptom<\/td>Use composite monitor<\/td><\/tr>
            Batch job silence<\/td>Use no-data only when expected cadence is known<\/td><\/tr>
            Delayed cloud metrics<\/td>Add evaluation delay<\/td><\/tr>
            Too many warning states<\/td>Remove or tune warning threshold<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Good alert rule:<\/p>\n\n\n
            Page only when customer impact is likely and action is required.\n<\/code><\/span><\/pre>\n\n\n
            Bad alert:<\/p>\n\n\n
            CPU > 80% for 1 minute\n<\/code><\/span><\/pre>\n\n\n
            Better alert:<\/p>\n\n\n
            Checkout p95 latency > 1s and error rate > 5% for 10 minutes in prod\n<\/code><\/span><\/pre>\n\n\n
            \n\n\n\n
            12. SLO Troubleshooting<\/h1>\n\n\n\n
            12.1 SLO shows wrong reliability<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Area<\/th>What to Verify<\/th><\/tr><\/thead>
            SLI query<\/td>Good events \/ total events correct?<\/td><\/tr>
            Time window<\/td>7, 30, 90 days?<\/td><\/tr>
            Tags<\/td>Correct service\/env\/region?<\/td><\/tr>
            Metric type<\/td>Count\/rate\/distribution usage correct?<\/td><\/tr>
            Missing data<\/td>Are missing periods counted correctly?<\/td><\/tr>
            Monitor-based SLO<\/td>Are underlying monitors correct?<\/td><\/tr>
            Grouped SLO<\/td>Are groups expected?<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            SLOs are used to define and track reliability targets and help teams communicate service performance; Datadog also exposes burn rate status such as healthy, elevated, and critical. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            12.2 Burn-rate alert too noisy or too late<\/h2>\n\n\n\n
            Use multiple windows:<\/p>\n\n\n\n
            Alert Type<\/th>Purpose<\/th><\/tr><\/thead>
            Fast burn<\/td>Detect serious incident quickly<\/td><\/tr>
            Slow burn<\/td>Detect gradual reliability erosion<\/td><\/tr>
            Warning<\/td>Notify before hard breach<\/td><\/tr>
            Critical<\/td>Page responders<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            For monitor-based SLOs, Datadog notes that burn-rate alerts can only be set on the overall SLO, not per group. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            13. Synthetic Monitoring Troubleshooting<\/h1>\n\n\n\n
            13.1 API test fails but endpoint works manually<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Area<\/th>What to Verify<\/th><\/tr><\/thead>
            Location<\/td>Test from same region\/location<\/td><\/tr>
            Private\/public<\/td>Internal endpoint needs Private Location<\/td><\/tr>
            DNS<\/td>Datadog location resolves same DNS?<\/td><\/tr>
            TLS<\/td>Cert chain valid?<\/td><\/tr>
            Headers<\/td>Required auth headers present?<\/td><\/tr>
            Proxy\/WAF<\/td>Is Datadog blocked?<\/td><\/tr>
            Response assertion<\/td>Assertion too strict?<\/td><\/tr>
            Timeout<\/td>Endpoint slow<\/td><\/tr>
            IP allowlist<\/td>Datadog locations allowed?<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            \n\n\n\n
            13.2 Browser test is flaky<\/h2>\n\n\n\n
            Common causes:<\/p>\n\n\n\n
            Cause<\/th>Fix<\/th><\/tr><\/thead>
            Dynamic elements<\/td>Use stable selectors<\/td><\/tr>
            Slow page load<\/td>Add wait\/assertion<\/td><\/tr>
            Animation\/loading state<\/td>Wait for element<\/td><\/tr>
            Third-party widget<\/td>Avoid depending on it<\/td><\/tr>
            Captcha\/MFA<\/td>Use test bypass or API auth<\/td><\/tr>
            Session expiry<\/td>Refresh login token<\/td><\/tr>
            Test data collision<\/td>Generate unique data<\/td><\/tr>
            Region-specific latency<\/td>Run from relevant locations<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            \n\n\n\n
            13.3 Browser recorder does not work<\/h2>\n\n\n\n
            Datadog\u2019s Synthetics troubleshooting explains that CSP or X-Frame-Options<\/code> can prevent the recorder iframe from loading the site, and the workaround is to record in a popup. It also notes HTTP pages do not load in the iframe recorder, extension permissions can block recording, Chrome policies can block the extension, and incognito mode can help record login flows from a fresh session. (Datadog<\/a>)<\/p>\n\n\n\n
            Practical checks:<\/p>\n\n\n
            CSP\nX-Frame-Options\nCSRF token behavior\nChrome extension permissions\nCorporate browser policy\nLogin\/session state\nHTTP vs HTTPS\n<\/code><\/span><\/pre>\n\n\n
            \n\n\n\n
            13.4 Private Location troubleshooting<\/h2>\n\n\n\n
            Private Location issues are usually network or runtime issues.<\/p>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Area<\/th>What to Verify<\/th><\/tr><\/thead>
            Worker running<\/td>Container\/service is healthy<\/td><\/tr>
            API key<\/td>Correct Datadog org\/site<\/td><\/tr>
            Outbound access<\/td>Worker can reach Datadog<\/td><\/tr>
            Internal DNS<\/td>Worker can resolve private app<\/td><\/tr>
            Firewall<\/td>Worker can reach target<\/td><\/tr>
            Proxy<\/td>Proxy config correct<\/td><\/tr>
            TLS<\/td>Internal CA trusted<\/td><\/tr>
            Resources<\/td>Worker has enough CPU\/memory<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            \n\n\n\n
            14. RUM and Session Replay Troubleshooting<\/h1>\n\n\n\n
            14.1 RUM data missing<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Area<\/th>What to Verify<\/th><\/tr><\/thead>
            RUM SDK loaded<\/td>Browser\/mobile SDK initialized<\/td><\/tr>
            Client token<\/td>Correct token<\/td><\/tr>
            Application ID<\/td>Correct app ID<\/td><\/tr>
            Site<\/td>Correct regional site<\/td><\/tr>
            Sampling<\/td>Session sample rate not zero<\/td><\/tr>
            Environment<\/td>env<\/code> correct<\/td><\/tr>
            CSP<\/td>Browser allowed to send intake requests<\/td><\/tr>
            Ad blockers<\/td>Some users may block telemetry<\/td><\/tr>
            Consent<\/td>Tracking consent enabled if required<\/td><\/tr>
            Timeframe<\/td>Search correct time window<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Datadog RUM captures real user performance, errors, analytics\/usage, and support data; sessions include views, actions, resources, errors, and related signals. A RUM session can last up to 4 hours and expires after 15 minutes of inactivity. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            14.2 Session Replay missing<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Cause<\/th>Fix<\/th><\/tr><\/thead>
            Replay not enabled<\/td>Enable Session Replay<\/td><\/tr>
            Replay sample rate low<\/td>Adjust sample rate<\/td><\/tr>
            Privacy masking<\/td>Expected masking behavior<\/td><\/tr>
            User not sampled<\/td>Test with controlled sampling<\/td><\/tr>
            CSP blocks intake<\/td>Update CSP<\/td><\/tr>
            Permissions<\/td>User lacks replay access<\/td><\/tr>
            Data deleted<\/td>Check retention\/deletion policy<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            \n\n\n\n
            14.3 RUM errors do not connect to APM traces<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n
            Allowed tracing URLs\nCORS headers\nBackend tracer enabled\nTrace context propagation\nservice\/env\/version alignment\n<\/code><\/span><\/pre>\n\n\n
            Frontend-to-backend trace correlation requires both RUM and backend APM to propagate compatible trace context.<\/p>\n\n\n\n
            \n\n\n\n
            15. OpenTelemetry Troubleshooting<\/h1>\n\n\n\n
            15.1 OTel data not reaching Datadog<\/h2>\n\n\n\n
            Possible data paths:<\/p>\n\n\n
            flowchart<\/span> TB<\/span>\n    A<\/span>[OTel SDK]<\/span> --<\/span>> B<\/span>[OTel Collector]<\/span>\n    B<\/span> --<\/span>> C<\/span>[Datadog Exporter]<\/span>\n    C<\/span> --<\/span>> D<\/span>[Datadog Intake]<\/span>\n\n    A2<\/span>[OTel SDK]<\/span> --<\/span>> B2<\/span>[Datadog Agent OTLP Receiver]<\/span>\n    B2<\/span> --<\/span>> D<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
            Datadog supports OpenTelemetry data through the OpenTelemetry Collector with Datadog exporter and through OTLP ingest in the Datadog Agent. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            15.2 OTLP ingest into Datadog Agent not working<\/h2>\n\n\n\n
            Check Agent OTLP receiver config.<\/p>\n\n\n\n
            Important:<\/p>\n\n\n
            OTLP gRPC default<\/span>: 4317<\/span>\nOTLP HTTP default<\/span>: 4318<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
            Application:<\/p>\n\n\n
            env | grep OTEL_\n<\/code><\/span><\/pre>\n\n\n
            Common variables:<\/p>\n\n\n
            OTEL_EXPORTER_OTLP_ENDPOINT\nOTEL_SERVICE_NAME\nOTEL_RESOURCE_ATTRIBUTES\nOTEL_TRACES_EXPORTER\nOTEL_METRICS_EXPORTER\nOTEL_LOGS_EXPORTER\n<\/code><\/span><\/pre>\n\n\n
            Datadog Agent supports OTLP traces and metrics through gRPC or HTTP since Agent 6.32\/7.32, and OTLP logs since Agent 6.48\/7.48. Datadog also warns that OTLP ingest is intended for an Agent local to each telemetry-generating host, not a remote Agent on another host. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            15.3 Duplicate hosts with OpenTelemetry<\/h2>\n\n\n\n
            Cause:<\/p>\n\n\n\n
            You are sending host telemetry through both the Datadog Agent and OpenTelemetry with inconsistent hostname\/resource attributes.<\/p>\n\n\n\n
            Fix:<\/p>\n\n\n\n
            Standardize:<\/p>\n\n\n
            host<\/span>.name<\/span>\nservice<\/span>.name<\/span>\ndeployment<\/span>.environment<\/span>\nservice<\/span>.version<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
            \n\n\n\n
            15.4 Some Datadog features missing with OTel<\/h2>\n\n\n\n
            OpenTelemetry is great, but not every Datadog-native capability works the same way with only OTel SDK instrumentation.<\/p>\n\n\n\n
            Datadog notes that data from applications instrumented with OpenTelemetry SDKs cannot be used in some proprietary products, including App and API Protection, Continuous Profiler, and Ingestion Rules. (Datadog<\/a>)<\/p>\n\n\n\n
            So when debugging OTel setups, ask:<\/p>\n\n\n
            Are we using OTel SDK only?\nAre we using Datadog tracer?\nAre we using Datadog Agent OTLP ingest?\nAre we using Datadog Distribution of OpenTelemetry Collector?\nWhich Datadog features are required?\n<\/code><\/span><\/pre>\n\n\n
            \n\n\n\n
            16. Security \/ Cloud SIEM \/ Sensitive Data Troubleshooting<\/h1>\n\n\n\n
            16.1 Security signals missing<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Area<\/th>What to Verify<\/th><\/tr><\/thead>
            Log source<\/td>Security logs are ingested<\/td><\/tr>
            Indexing<\/td>Logs are indexed or available to SIEM<\/td><\/tr>
            Detection rule<\/td>Rule enabled<\/td><\/tr>
            Rule query<\/td>Query matches events<\/td><\/tr>
            Severity<\/td>Filter not hiding it<\/td><\/tr>
            Suppression<\/td>Signal not suppressed<\/td><\/tr>
            Permissions<\/td>User can view security data<\/td><\/tr>
            Timestamp<\/td>Event time valid<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            \n\n\n\n
            16.2 Sensitive data still visible<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Cause<\/th>Fix<\/th><\/tr><\/thead>
            Scanner rule not enabled<\/td>Enable rule<\/td><\/tr>
            Pattern does not match<\/td>Adjust regex\/pattern<\/td><\/tr>
            Pipeline order wrong<\/td>Redact before indexing<\/td><\/tr>
            Data in traces\/RUM too<\/td>Enable scanner across needed telemetry<\/td><\/tr>
            Already indexed old data<\/td>Consider deletion\/remediation process<\/td><\/tr>
            Custom field missed<\/td>Add custom scanner rule<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Best practice: prevent secrets at source. Scanning is a safety net, not the main control.<\/p>\n\n\n\n
            \n\n\n\n
            17. Cloud Cost \/ Datadog Cost Troubleshooting<\/h1>\n\n\n\n
            17.1 Cloud cost data missing<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Cloud<\/th>What to Verify<\/th><\/tr><\/thead>
            AWS<\/td>CUR configured, S3 access, IAM permissions<\/td><\/tr>
            GCP<\/td>BigQuery billing export, bucket\/dataset setup<\/td><\/tr>
            Azure<\/td>Export permissions and billing scope<\/td><\/tr>
            Tags<\/td>Cost allocation tags enabled<\/td><\/tr>
            Delay<\/td>Cost data is not real-time<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Datadog Cloud Cost Management ingests cloud cost data and transforms it into metrics that can be queried in Cost Explorer; it is meant to correlate cost increases with usage and infrastructure changes. (Datadog<\/a>)<\/p>\n\n\n\n
            \n\n\n\n
            17.2 Datadog bill increased<\/h2>\n\n\n\n
            Investigate:<\/p>\n\n\n\n
            Cost Driver<\/th>Where to Look<\/th><\/tr><\/thead>
            Custom metrics<\/td>Metrics Summary, Metrics without Limits<\/td><\/tr>
            Indexed logs<\/td>Log indexes, exclusion filters<\/td><\/tr>
            Ingested logs<\/td>Usage metrics<\/td><\/tr>
            APM spans<\/td>APM ingestion\/retention<\/td><\/tr>
            RUM sessions<\/td>RUM usage<\/td><\/tr>
            Synthetics<\/td>Test frequency, locations, browser tests<\/td><\/tr>
            Containers\/hosts<\/td>Infrastructure list<\/td><\/tr>
            Cloud cost<\/td>Cost Explorer<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            Common root causes:<\/p>\n\n\n
            containerCollectAll enabled without exclusions\ndebug logs enabled\nhigh-cardinality custom metric tags\nnew<\/span> Kubernetes cluster added\nlogs indexed for<\/span> all environments\nsynthetics running every minute from<\/span> many locations\ntrace sampling too high\nRUM sample rate too high\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
            \n\n\n\n
            18. Dashboard and Query Troubleshooting<\/h1>\n\n\n\n
            18.1 Dashboard shows no data<\/h2>\n\n\n\n
            Check:<\/p>\n\n\n\n
            Cause<\/th>Fix<\/th><\/tr><\/thead>
            Wrong timeframe<\/td>Expand time range<\/td><\/tr>
            Wrong template variable<\/td>Clear variables<\/td><\/tr>
            Wrong env\/service tag<\/td>Verify tag values<\/td><\/tr>
            Metric missing<\/td>Check Metrics Explorer<\/td><\/tr>
            Query aggregation wrong<\/td>Adjust rollup\/grouping<\/td><\/tr>
            User lacks data permission<\/td>Check RBAC<\/td><\/tr>
            Data delayed<\/td>Wait\/add delay for cloud metrics<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            \n\n\n\n
            18.2 Dashboard values differ from monitor values<\/h2>\n\n\n\n
            Common reasons:<\/p>\n\n\n\n
            Reason<\/th>Explanation<\/th><\/tr><\/thead>
            Different time window<\/td>Dashboard last 1h vs monitor last 5m<\/td><\/tr>
            Different rollup<\/td>avg vs sum vs max<\/td><\/tr>
            Different grouping<\/td>grouped vs aggregated<\/td><\/tr>
            Different filters<\/td>env\/service mismatch<\/td><\/tr>
            Evaluation delay<\/td>Monitor evaluates older window<\/td><\/tr>
            Require full window<\/td>Monitor behavior differs<\/td><\/tr>
            Timezone\/display<\/td>UI interpretation<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            \n\n\n\n
            19. Incident Troubleshooting Method<\/h1>\n\n\n\n
            When there is a production incident, do not randomly click dashboards. Use a fixed flow.<\/p>\n\n\n
            flowchart<\/span> TD<\/span>\n    A<\/span>[Alert Fires]<\/span> --<\/span>> B<\/span>[Confirm Customer Impact]<\/span>\n    B<\/span> --<\/span>> C<\/span>[Check SLO \/ Error Budget]<\/span>\n    C<\/span> --<\/span>> D<\/span>[Identify Service + Env + Version]<\/span>\n    D<\/span> --<\/span>> E<\/span>[Check Recent Changes]<\/span>\n    E<\/span> --<\/span>> F<\/span>[Metrics: traffic, errors, latency, saturation]<\/span>\n    F<\/span> --<\/span>> G<\/span>[Traces: slow\/failing requests]<\/span>\n    G<\/span> --<\/span>> H<\/span>[Logs: errors around trace IDs]<\/span>\n    H<\/span> --<\/span>> I<\/span>[Infra\/K8s: pods, nodes, deploys]<\/span>\n    I<\/span> --<\/span>> J<\/span>[Cloud\/DB\/Queue Dependencies]<\/span>\n    J<\/span> --<\/span>> K<\/span>[Mitigate]<\/span>\n    K<\/span> --<\/span>> L<\/span>[Postmortem + Monitor Fixes]<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
            Golden questions:<\/p>\n\n\n
            What changed?\nWho is affected?\nWhich service owns the issue?\nIs it regional?\nIs it version-specific?\nIs it dependency-specific?\nIs it traffic-related?\nIs the alert actionable?\nIs this<\/span> an observability gap?\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
            \n\n\n\n
            20. Troubleshooting Cheat Sheet<\/h1>\n\n\n\n
            Agent<\/h2>\n\n\n
            sudo datadog-agent status\nsudo datadog-agent health\nsudo datadog-agent diagnose\nsudo datadog-agent configcheck\nsudo datadog-agent check <integration<\/span>><\/span>\nsudo datadog-agent flare\nsudo journalctl -u datadog-agent.service -r\nsudo tail -f \/var\/log\/datadog\/agent.log\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
            Docker Agent<\/h2>\n\n\n
            docker ps | grep datadog\ndocker logs <agent-container<\/span>><\/span>\ndocker exec -it <agent-container<\/span>><\/span> agent status\ndocker exec -it <agent-container<\/span>><\/span> agent configcheck\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
            Kubernetes Agent<\/h2>\n\n\n
            kubectl get pods -n datadog -o wide\nkubectl logs -n datadog <agent-pod<\/span>><\/span>\nkubectl describe pod -n datadog <agent-pod<\/span>><\/span>\nkubectl exec -it -n datadog <agent-pod<\/span>><\/span> -- agent status\nkubectl exec -it -n datadog <agent-pod<\/span>><\/span> -- agent configcheck\nkubectl exec -it -n datadog <agent-pod<\/span>><\/span> -- agent stream-logs\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
            Cluster Agent<\/h2>\n\n\n
            kubectl get<\/span> deploy -n datadog\nkubectl logs -n datadog deploy\/datadog-cluster-agent\nkubectl exec -n datadog deploy\/datadog-cluster-agent -- datadog-cluster-agent status\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
            APM<\/h2>\n\n\n
            env | grep DD_\ncurl -v http:\/\/${DD_AGENT_HOST}:8126\/info<\/span>\nsudo datadog-agent status\nsudo tail -f \/var<\/span>\/log\/datadog\/trace-agent.log\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
            Logs<\/h2>\n\n\n
            kubectl logs <pod<\/span>><\/span> -n <namespace<\/span>><\/span>\ndocker logs <container<\/span>><\/span>\nsudo datadog-agent stream-logs\nsudo datadog-agent status\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
            API<\/h2>\n\n\n
            curl -X GET \"https:\/\/api.<DATADOG_SITE>\/api\/v1\/validate\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span>\n\ncurl -X GET \"https:\/\/api.<DATADOG_SITE>\/api\/v2\/validate_keys\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
            \n\n\n\n
            21. Most Common Datadog Problems and Fast Fixes<\/h1>\n\n\n\n
            Problem<\/th>Most Likely Cause<\/th>First Fix<\/th><\/tr><\/thead>
            Host not visible<\/td>Wrong API key\/site\/network<\/td>Run Agent status\/diagnose<\/td><\/tr>
            Logs missing<\/td>Logs not enabled or not indexed<\/td>Check Agent status + indexes<\/td><\/tr>
            Traces missing<\/td>App cannot reach Agent on 8126<\/td>Check DD_AGENT_HOST<\/code><\/td><\/tr>
            Metrics missing<\/td>Integration check failing<\/td>Run agent check <name><\/code><\/td><\/tr>
            Monitor No Data<\/td>Rollup\/window\/evaluation issue<\/td>Check Require Full Window<\/td><\/tr>
            API 403<\/td>App key lacks permission<\/td>Check scopes\/role<\/td><\/tr>
            Terraform 403<\/td>Wrong\/scoped app key<\/td>Use service account key<\/td><\/tr>
            AWS metrics delayed<\/td>CloudWatch crawl delay<\/td>Add evaluation delay<\/td><\/tr>
            Too many services<\/td>Bad service naming<\/td>Use env<\/code>\/version<\/code> tags<\/td><\/tr>
            Cost spike<\/td>High-cardinality\/log indexing<\/td>Check Metrics Summary\/log indexes<\/td><\/tr>
            K8s metadata missing<\/td>Cluster Agent\/RBAC issue<\/td>Check Cluster Agent logs<\/td><\/tr>
            Browser test flaky<\/td>Dynamic selector\/timing<\/td>Add stable assertions<\/td><\/tr>
            RUM missing<\/td>Wrong app\/client token\/site<\/td>Check SDK config<\/td><\/tr>
            OTel duplicate hosts<\/td>Hostname\/resource mismatch<\/td>Align resource attributes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
            \n\n\n\n
            22. Best-Practice Troubleshooting Mindset<\/h1>\n\n\n\n
            The best Datadog engineers do not start with dashboards. They start with the telemetry path.<\/p>\n\n\n\n
            Ask in this order:<\/p>\n\n\n\n
              \n
            1. Is the source producing data?<\/strong><\/li>\n\n\n\n
            2. Is the Agent\/SDK\/Collector receiving it?<\/strong><\/li>\n\n\n\n
            3. Can it reach Datadog intake?<\/strong><\/li>\n\n\n\n
            4. Is Datadog accepting it?<\/strong><\/li>\n\n\n\n
            5. Is it being processed, indexed, retained, or sampled?<\/strong><\/li>\n\n\n\n
            6. Are we querying it correctly?<\/strong><\/li>\n\n\n\n
            7. Do tags match our standard?<\/strong><\/li>\n\n\n\n
            8. Is the user allowed to see it?<\/strong><\/li>\n\n\n\n
            9. Is cost control intentionally dropping or reducing it?<\/strong><\/li>\n\n\n\n
            10. Can we prevent the same issue with better tagging, monitors, dashboards, or IaC?<\/strong><\/li>\n<\/ol>\n\n\n\n
              The biggest real-world Datadog issues are usually not \u201cDatadog is broken.\u201d They are usually:<\/p>\n\n\n
              wrong site\nwrong key\nwrong tag\nwrong environment\nwrong Agent config\nwrong Kubernetes networking\nwrong log index\nwrong trace endpoint\nwrong monitor window<\/span>\nwrong permissions\ntoo much cardinality\ntoo much noisy data\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
              That\u2019s the Datadog troubleshooting game in one sentence: follow the telemetry, then follow the tags.<\/strong><\/p>\n\n\n\n
              <\/p>\n","protected":false},"excerpt":{"rendered":"
              It covers: Datadog Agent, Kubernetes Agent, Cluster Agent, integrations, logs, APM\/traces, custom metrics, DogStatsD, OpenTelemetry, API keys, Terraform, monitors, SLOs, RUM, Synthetics, cloud integrations, cost, permissions, and… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[25768],"tags":[],"class_list":["post-26859","post","type-post","status-publish","format-standard","hentry","category-datadog"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/26859","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=26859"}],"version-history":[{"count":4,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/26859\/revisions"}],"predecessor-version":[{"id":77128,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/26859\/revisions\/77128"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=26859"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=26859"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=26859"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
            Datadog\u2019s OTel troubleshooting notes that duplicate host entries can happen when a host is monitored through multiple ingestion methods, such as OTLP plus Datadog Agent or DogStatsD plus OTLP, without aligning on a single hostname resource attribute. (Datadog<\/a>)<\/p>\n\n\n\n
          Datadog\u2019s Python documentation specifically notes that ddtrace-run --info<\/code> can be used to check configuration after setup. (Datadog<\/a>)<\/p>\n\n\n\n
      The Agent DaemonSet collects node, container, logs, traces, and process data. The Cluster Agent centralizes cluster-level data collection and helps reduce direct API server pressure from node Agents. The Cluster Agent is enabled by default for modern Helm and Operator installations. (Datadog<\/a>)<\/p>\n\n\n\n
Datadog recommends scoped application keys with least privilege, and newer organizations created after August 20, 2025 have One-Time Read mode enabled by default for application keys, meaning the app key secret is visible only at creation time. (Datadog<\/a>)<\/p>\n\n\n\n
The site configured in datadog.yaml<\/code> must match the organization\u2019s Datadog platform site. (Datadog<\/a>)<\/p>\n\n\n\n