It covers:

Datadog Agent, Kubernetes Agent, Cluster Agent, integrations, logs, APM/traces, custom metrics, DogStatsD, OpenTelemetry, API keys, Terraform, monitors, SLOs, RUM, Synthetics, cloud integrations, cost, permissions, and common “data missing in Datadog” scenarios.

1. First Principle: Understand the Datadog Data Flow

Before troubleshooting Datadog, always identify where the data path is broken.

A typical telemetry path looks like this:

flowchart LR
    A[Application / Host / Container / Cloud Service] --> B[Datadog Agent / SDK / Collector / Integration]
    B --> C[Network / Proxy / TLS / Firewall]
    C --> D[Datadog Intake Endpoint]
    D --> E[Datadog Processing Pipeline]
    E --> F[Indexes / Metrics Store / Trace Store / RUM / Synthetics]
    F --> G[Dashboard / Monitor / SLO / Explorer]
Code language: CSS (css)

Most Datadog problems fall into one of these buckets:

Datadog’s own Agent troubleshooting checklist starts with exactly these basics: internet/proxy access, correct API key, correct Datadog site, only one Agent per host, restarting after YAML changes, then checking Agent status and logs. (Datadog)

2. Universal Troubleshooting Checklist

Use this before debugging any specific Datadog product.

2.1 Check the Datadog site

Datadog has multiple sites, such as US1, US3, US5, EU, AP1, AP2, and US1-FED. If your Agent, SDK, Terraform provider, API request, RUM SDK, or Synthetic Private Location is configured for the wrong site, data may be sent successfully but to the wrong Datadog intake.

Common values:

datadoghq.com
us3.datadoghq.com
us5.datadoghq.com
datadoghq.eu
ap1.datadoghq.com
ap2.datadoghq.com
ddog-gov.com
Code language: CSS (css)

Check:

sudo datadog-agent status
sudo datadog-agent configcheck
grep -i "site" /etc/datadog-agent/datadog.yaml
Code language: JavaScript (javascript)

For Kubernetes, check Helm/Operator values:

helm get values datadog -n datadog
kubectl get datadogagent -A -o yaml
Code language: JavaScript (javascript)

The site configured in datadog.yaml must match the organization’s Datadog platform site. (Datadog)

2.2 Check API key and application key

There are two key types:

Datadog API keys are unique to the organization and are required by the Agent to submit metrics and events. Application keys are used with the API key for programmatic API access and inherit permissions from the user or service account unless scoped differently. (Datadog)

Check:

grep -i "api_key" /etc/datog-agent/datadog.yaml
Code language: JavaScript (javascript)

But do not print secrets in CI logs or Slack.

For API validation:

curl -X GET "https://api.<DATADOG_SITE>/api/v1/validate" \
  -H "DD-API-KEY: ${DD_API_KEY}"
Code language: JavaScript (javascript)

For API + app key validation:

curl -X GET "https://api.<DATADOG_SITE>/api/v2/validate_keys" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}"
Code language: JavaScript (javascript)

Datadog recommends scoped application keys with least privilege, and newer organizations created after August 20, 2025 have One-Time Read mode enabled by default for application keys, meaning the app key secret is visible only at creation time. (Datadog)

2.3 Check network, proxy, DNS, TLS

Many Datadog issues are not Datadog issues. They are egress issues.

Check:

curl -v https://api.datadoghq.com
curl -v https://app.datadoghq.com
curl -v https://trace.agent.datadoghq.com
curl -v https://http-intake.logs.datadoghq.com
Code language: JavaScript (javascript)

For your actual site, replace with the correct regional endpoint.

From Kubernetes:

kubectl run curl-test --rm -it --image=curlimages/curl -- sh
curl -v https://api.datadoghq.com
Code language: JavaScript (javascript)

Common causes:

2.4 Check Agent health and status

Core commands:

sudo datadog-agent status
sudo datadog-agent health
sudo datadog-agent diagnose
sudo datadog-agent configcheck
sudo datadog-agent flare

For Docker:

docker exec -it <datadog-agent-container> agent status
docker exec -it <datadog-agent-container> agent health
docker exec -it <datadog-agent-container> agent diagnose
Code language: HTML, XML (xml)

For Kubernetes:

kubectl exec -it <datadog-agent-pod> -n datadog -- agent status
kubectl exec -it <datadog-agent-pod> -n datadog -- agent health
kubectl exec -it <datadog-agent-pod> -n datadog -- agent configcheck
Code language: HTML, XML (xml)

Datadog documents datadog-agent status, configcheck, diagnose, flare, health, hostname, stream-logs, and check-specific commands as core troubleshooting commands. (Datadog)

2.5 Check Agent logs

Linux:

sudo journalctl -u datadog-agent.service -r
sudo tail -f /var/log/datadog/agent.log
Code language: JavaScript (javascript)

Kubernetes:

kubectl logs -n datadog <datadog-agent-pod>
kubectl logs -n datadog <datadog-cluster-agent-pod>
Code language: HTML, XML (xml)

Check for:

API key invalid
connection refused
timeout
proxy error
TLS error
invalid YAML
permission denied
unable to resolve hostname
check failed
logs-agent stopped
trace-agent stopped
process-agent stopped

For systemd systems, Datadog recommends systemctl status datadog-agent and journalctl -u datadog-agent.service when the Agent fails to start. (Datadog)

3. Datadog Agent Troubleshooting

3.1 Agent not starting

Symptoms

datadog-agent.service failed
Agent container CrashLoopBackOff
No host appears in Infrastructure List
No system.cpu/system.mem metrics
Code language: PHP (php)

Check

Linux:

sudo systemctl status datadog-agent
sudo journalctl -u datadog-agent.service -r
sudo datadog-agent status
Code language: CSS (css)

Kubernetes:

kubectl get pods -n datadog
kubectl describe pod <agent-pod> -n datadog
kubectl logs <agent-pod> -n datadog
Code language: HTML, XML (xml)

Common causes

The official Agent checklist explicitly calls out hostname detection, proxy access, API key, site, duplicate Agents, restart after YAML changes, status command, logs, debug mode, and Agent flare. (Datadog)

3.2 Agent installed but host not visible

Check the basic metrics

In Datadog Metrics Explorer, query:

system.cpu.user
system.mem.used
datadog.agent.running
Code language: CSS (css)

On host:

sudo datadog-agent status
sudo datadog-agent diagnose
sudo datadog-agent hostname

Likely causes

Fix pattern

1. Confirm Agent service is running.
2. Confirm API key and site.
3. Confirm outbound HTTPS access.
4. Confirm hostname.
5. Confirm correct UI timeframe.
6. Search by hostname, cloud instance ID, Kubernetes node name, and tags.

3.3 Agent running but integration metrics missing

Example: Agent metrics exist, but PostgreSQL, Redis, NGINX, Kafka, MySQL, or JVM metrics are missing.

Check

sudo datadog-agent status
sudo datadog-agent configcheck
sudo datadog-agent check <integration_name>
Code language: HTML, XML (xml)

Examples:

sudo datadog-agent check postgres
sudo datadog-agent check redisdb
sudo datadog-agent check nginx
sudo datadog-agent check kafka

What to inspect in status output

Look under:

Running Checks
===========
postgres
redisdb
nginx
kafka

A healthy check should show metric samples, no critical errors, and recent successful runs. Datadog’s command reference notes that properly configured integrations appear under Running Checks with no warnings or errors. (Datadog)

Common causes

3.4 Agent high CPU or memory

Check

top
ps aux | grep datadog
sudo datadog-agent status
sudo datadog-agent health

In Kubernetes:

kubectl top pod -n datadog
kubectl describe pod <agent-pod> -n datadog
Code language: HTML, XML (xml)

Common causes

Fix

Reduce collection scope, increase Agent resources, tune log exclusions, reduce check frequency, avoid unnecessary debug logs, control tags, and use Cluster Agent for Kubernetes metadata and cluster-level functions.

3.5 Agent debug mode

Use debug mode only temporarily.

Why?

Because debug mode increases log volume and may expose configuration details.

Typical approach:

sudo datadog-agent config set log_level debug
sudo systemctl restart datadog-agent
sudo tail -f /var/log/datadog/agent.log
Code language: JavaScript (javascript)

After troubleshooting:

sudo datadog-agent config set log_level info
sudo systemctl restart datadog-agent
Code language: JavaScript (javascript)

3.6 Agent flare

Use a flare when you need to send diagnostic data to Datadog Support.

sudo datadog-agent flare

In Kubernetes:

kubectl exec -it <agent-pod> -n datadog -- agent flare
Code language: HTML, XML (xml)

A flare gathers Agent configuration files and logs into an archive and removes sensitive information such as API keys, passwords, proxy credentials, and SNMP community strings. (Datadog)

4. Kubernetes Datadog Troubleshooting

4.1 Recommended Kubernetes installation approach

For Kubernetes, prefer:

1. Datadog Operator, or
2. Helm chart.

Avoid manually maintaining large raw DaemonSet YAML unless you have a strict reason.

Datadog describes the Operator as the recommended Kubernetes installation method because it reports deployment status, health, and errors in the custom resource status and reduces misconfiguration risk through higher-level configuration options. (Datadog)

4.2 Kubernetes architecture

flowchart TB
    subgraph K8S[Kubernetes Cluster]
        A[Datadog Agent DaemonSet<br>one per node]
        B[Datadog Cluster Agent<br>cluster-level metadata]
        C[Kubelet / Container Runtime]
        D[Kubernetes API Server]
        E[Application Pods]
    end

    E --> A
    C --> A
    D --> B
    B --> A
    A --> F[Datadog Intake]
Code language: CSS (css)

The Agent DaemonSet collects node, container, logs, traces, and process data. The Cluster Agent centralizes cluster-level data collection and helps reduce direct API server pressure from node Agents. The Cluster Agent is enabled by default for modern Helm and Operator installations. (Datadog)

4.3 Agent pod not running

Check

kubectl get pods -n datadog -o wide
kubectl describe pod <agent-pod> -n datadog
kubectl logs <agent-pod> -n datadog
kubectl get daemonset -n datadog
Code language: HTML, XML (xml)

Common causes

4.4 Cluster Agent not working

Check

kubectl get deploy -n datadog
kubectl get pods -n datadog | grep cluster
kubectl logs -n datadog deploy/datadog-cluster-agent
kubectl exec -n datadog deploy/datadog-cluster-agent -- datadog-cluster-agent status
Code language: JavaScript (javascript)

Common causes

4.5 Kubernetes metrics missing

Check:

kubectl get pods -n datadog
kubectl logs -n datadog <agent-pod>
kubectl logs -n datadog <cluster-agent-pod>
kubectl auth can-i get pods --as system:serviceaccount:datadog:datadog-agent
kubectl auth can-i list nodes --as system:serviceaccount:datadog:datadog-agent
Code language: CSS (css)

Query in Datadog:

kubernetes.cpu.usage.total
kubernetes.memory.usage
kubernetes.pods.running
kubernetes_state.pod.status_phase
Code language: CSS (css)

Possible issues:

4.6 Kubernetes log collection missing

Datadog recommends file-based log collection in Kubernetes because it scales better for ephemeral pods and is more resource-efficient than Docker socket polling. (Datadog)

Check Agent settings

For Helm/Operator, confirm logs are enabled:

logs:
  enabled: true
  containerCollectAll: true
Code language: JavaScript (javascript)

Or equivalent Operator config.

Commands

kubectl exec -it <agent-pod> -n datadog -- agent status
kubectl exec -it <agent-pod> -n datadog -- agent stream-logs
kubectl logs <application-pod> -n <namespace>
Code language: HTML, XML (xml)

Common causes

Datadog’s container log troubleshooting explains that container logs are collected either file-based or socket-based, with file-based collection being the recommended Kubernetes default. (Datadog)

4.7 APM traces missing from Kubernetes apps

This is one of the most common production problems.

Data path

flowchart LR
    A[Application Pod with Datadog tracer] --> B[Datadog Agent on node]
    B --> C[Trace Agent :8126]
    C --> D[Datadog APM Intake]
    D --> E[Trace Explorer / Service Catalog]
Code language: CSS (css)

Check in app pod

env | grep DD_

Important variables:

DD_ENV
DD_SERVICE
DD_VERSION
DD_AGENT_HOST
DD_TRACE_AGENT_PORT
DD_TRACE_ENABLED
DD_LOGS_INJECTION

Common Kubernetes issue

Many apps default to sending traces to:

localhost:8126
Code language: CSS (css)

But inside a Kubernetes application container, localhost means the application pod itself, not the Datadog Agent pod.

Typical fix:

env:
  - name: DD_AGENT_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: DD_TRACE_AGENT_PORT
    value: "8126"
Code language: JavaScript (javascript)

Also make sure APM is enabled in the Agent.

5. Log Management Troubleshooting

5.1 Logs not appearing in Datadog

First question

Are logs missing from:

1. The application?
2. The Agent?
3. Datadog intake?
4. Log Explorer?
5. A specific index?
6. A specific dashboard/monitor?

This distinction matters.

Check source

kubectl logs <pod> -n <namespace>
docker logs <container>
tail -f /path/to/app.log
Code language: HTML, XML (xml)

If logs are not visible here, Datadog cannot collect them.

Check Agent

sudo datadog-agent status
sudo datadog-agent stream-logs
sudo tail -f /var/log/datadog/agent.log
Code language: JavaScript (javascript)

Kubernetes:

kubectl exec -it <agent-pod> -n datadog -- agent status
kubectl exec -it <agent-pod> -n datadog -- agent stream-logs
Code language: HTML, XML (xml)

Datadog’s log troubleshooting documentation focuses on missing logs, data access, timestamp misalignment, ingestion/processing issues, and index/exclusion behavior. (Datadog)

5.2 Logs are collected but not searchable

Likely cause: logs are ingested but not indexed.

Check:

Datadog usage metrics such as datadog.estimated_usage.logs.ingested_events can be filtered by tags like datadog_index, datadog_is_excluded, service, and status; if datadog_index:N/A, the logs may not match any index. (Datadog)

5.3 Logs have wrong timestamp

Symptoms:

Logs appear in future
Logs appear hours late
Logs disappear from current view
Log monitor does not trigger
Code language: JavaScript (javascript)

Check:

Best practice: emit logs in structured JSON with an explicit UTC timestamp.

5.4 Logs missing service, env, or version

This breaks correlation.

Check:

service
env
version
dd.service
dd.env
dd.version
Code language: CSS (css)

Kubernetes fix:

labels:
  tags.datadoghq.com/env: prod
  tags.datadoghq.com/service: checkout
  tags.datadoghq.com/version: "1.2.3"
Code language: JavaScript (javascript)

Or environment variables:

env:
  - name: DD_ENV
    value: prod
  - name: DD_SERVICE
    value: checkout
  - name: DD_VERSION
    value: "1.2.3"
Code language: CSS (css)

Unified Service Tagging is one of the most important Datadog design requirements because it correlates metrics, logs, traces, deployments, services, and versions.

5.5 Multiline logs are broken

Symptoms:

Stack traces split into many log events
Java/Python exceptions appear as many separate logs
Error Tracking does not group correctly
Code language: JavaScript (javascript)

Fix by configuring multiline processing.

Example concept:

logs:
  - type: file
    path: /var/log/app.log
    service: my-service
    source: java
    log_processing_rules:
      - type: multi_line
        name: new_log_start_with_date
        pattern: \d{4}-\d{2}-\d{2}
Code language: JavaScript (javascript)

Do not overuse multiline rules; they can increase Agent CPU usage.

5.6 Log pipeline parsing issue

Symptoms:

status is wrong
service is wrong
JSON not parsed
fields are missing
severity not recognized
Code language: JavaScript (javascript)

Check:

1. Raw log event.
2. Pipeline order.
3. Grok/parser rule.
4. Attribute remapper.
5. Status remapper.
6. Service remapper.
7. Date remapper.

Best practice:

Emit structured JSON:

{
  "timestamp": "2026-06-23T10:15:00Z",
  "level": "ERROR",
  "service": "checkout",
  "env": "prod",
  "message": "Payment failed",
  "order_id": "12345"
}
Code language: JSON / JSON with Comments (json)

6. APM / Trace Troubleshooting

6.1 APM components

APM depends on three things:

flowchart LR
    A[Application Code] --> B[Datadog Tracer / SDK]
    B --> C[Datadog Agent Trace Agent]
    C --> D[Datadog APM Intake]
    D --> E[Trace Explorer / Services / APM Metrics]
Code language: CSS (css)

Datadog’s APM troubleshooting docs cover trace retention, trace metrics, services, data volume, connection errors, resource usage, security, tracer startup logs, tracer debug logs, and SDK configuration. (Datadog)

6.2 No traces in Datadog

Check 1: Is application instrumented?

Java:

java -javaagent:/path/dd-java-agent.jar -jar app.jar
Code language: JavaScript (javascript)

Python:

ddtrace-run python app.py
ddtrace-run --info
Code language: CSS (css)

Node.js:

require('dd-trace').init()
Code language: JavaScript (javascript)

Go:

Use Datadog tracing libraries and wrap supported frameworks.

.NET:

Check tracer installation and environment variables.

Datadog’s Python documentation specifically notes that ddtrace-run --info can be used to check configuration after setup. (Datadog)

Check 2: Is the tracer sending to the Agent?

From app container:

curl -v http://${DD_AGENT_HOST}:8126/info
Code language: JavaScript (javascript)

If this fails, fix networking first.

Check 3: Is Agent APM enabled?

Agent status should show APM / trace-agent.

sudo datadog-agent status

Kubernetes:

kubectl exec -it <agent-pod> -n datadog -- agent status
Code language: HTML, XML (xml)

Look for:

APM Agent
=========
Status: Running

6.3 Traces reach Agent but not Datadog

Check Agent logs:

sudo tail -f /var/log/datadog/trace-agent.log
sudo tail -f /var/log/datadog/agent.log
Code language: JavaScript (javascript)

Kubernetes:

kubectl logs <agent-pod> -n datadog | grep -i trace
Code language: HTML, XML (xml)

Common causes:

6.4 Service missing from APM Service Catalog

Possible causes:

Check:

env | grep DD_

Important:

DD_SERVICE
DD_ENV
DD_VERSION
DD_TAGS
DD_SERVICE_MAPPING

Datadog tracer configuration uses DD_SERVICE, DD_ENV, DD_VERSION, DD_TAGS, and service mapping variables across language libraries, with language-specific fallback behavior if service is not explicitly configured. (Datadog)

6.5 Too many services in APM

Symptoms:

checkout-prod
checkout-stage
checkout-v1
checkout-worker-123
postgres-prod
redis-checkout-prod

Root problem: environment, version, host, region, or shard has been included in the service name.

Bad:

checkout-prod
checkout-uat
checkout-v1-2-3

Good:

service:checkout
env:prod
version:1.2.3
Code language: CSS (css)

Use env, version, team, region, and other tags instead of encoding them in the service name.

6.6 Trace exists but related logs are missing

To correlate logs and traces:

1. Logs must be collected.
2. Tracer must inject trace/span IDs into logs.
3. Logs must include service, env, and version.
4. Log parser must preserve trace fields.
5. Logs must be indexed/searchable.

Check fields:

dd.trace_id
dd.span_id
trace_id
span_id
service
env
version
Code language: CSS (css)

Enable:

DD_LOGS_INJECTION=true
Code language: JavaScript (javascript)

Datadog supports correlating traces with logs, RUM, synthetics, profiles, and DBM when the telemetry carries the right metadata. (Datadog)

6.7 Trace metrics missing or confusing

APM generates trace metrics such as request count, error count, latency, and Apdex.

Common confusion:

Datadog’s APM troubleshooting separates trace retention, trace metrics, services, ingestion/sampling, and retention filters because these are related but not identical views of tracing data. (Datadog)

6.8 APM high cost or high trace volume

Check:

Use ingestion controls, sampling, retention filters, and service/resource normalization.

6.9 Single Step Instrumentation troubleshooting

Single Step Instrumentation can inject tracers automatically for supported environments and languages, but troubleshooting should confirm whether injection actually happened.

Check:

Datadog’s SSI troubleshooting page says injection issues can be checked through Fleet Automation or manually at container level; process-level insights show instrumentation status and SDK installation details, and current docs mention Agent v7.68.2+ for that SSI troubleshooting flow. (Datadog)

7. DogStatsD and Custom Metrics Troubleshooting

7.1 Custom metric not visible

Data path

flowchart LR
    A[Application] --> B[DogStatsD client]
    B --> C[Datadog Agent DogStatsD :8125]
    C --> D[Datadog Metrics Intake]
    D --> E[Metrics Explorer]
Code language: CSS (css)

Check

sudo datadog-agent status
sudo datadog-agent configcheck

Look for DogStatsD section.

Test UDP locally:

echo -n "custom.test.metric:1|c|#env:dev,service:test" | nc -4u -w1 127.0.0.1 8125
Code language: PHP (php)

Then search:

custom.test.metric
Code language: CSS (css)

7.2 DogStatsD issue in Kubernetes

Common problem: app sends metrics to localhost:8125, but DogStatsD is in the Datadog Agent pod on the node.

Fix patterns:

Option 1: Use host IP

env:
  - name: DD_AGENT_HOST
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
Code language: CSS (css)

Option 2: Use Unix Domain Socket

Better for local node-level communication if configured.

Option 3: Use service-based routing

Possible but be careful with load balancing because metrics may be sent to an Agent not on the same node.

7.3 Custom metrics cost/cardinality issue

Symptoms:

Custom metrics count exploded
Bill increased
Metrics Summary shows many tag combinations
Dashboards slow

Common bad tags:

user_id
request_id
session_id
pod_uid
container_id
timestamp
order_id
transaction_id

Better tags:

env
service
version
region
team
cluster
namespace
endpoint
status
method
provider
Code language: PHP (php)

Datadog’s Metrics without Limits decouples custom metric ingestion and indexing, allowing allowlists or blocklists of tags to control which tags remain queryable and reduce billable custom metric volume. (Datadog)

8. Integration Troubleshooting

8.1 Integration check fails

Example status:

Instance ID: postgres:...
Error: connection refused
Code language: CSS (css)

Use:

sudo datadog-agent check <integration>
sudo datadog-agent configcheck
sudo datadog-agent status
Code language: HTML, XML (xml)

Common integration failure categories:

Datadog’s integration troubleshooting guide focuses on YAML files, check status, missing metrics, and resolving integration configuration issues through Agent status/config/check commands. (Datadog)

8.2 JMX integration issues

Common for Kafka, Cassandra, JVM apps, Tomcat, Solr, etc.

Check:

sudo datadog-agent status
sudo datadog-agent jmx list collected
sudo datadog-agent jmx list matching
Code language: PHP (php)

Common causes:

8.3 Database integration issue

For PostgreSQL/MySQL/etc., check from the Agent host/pod:

nc -vz <db-host> 5432
psql -h <db-host> -U <monitor_user> -d <db>
Code language: HTML, XML (xml)

For MySQL:

nc -vz <db-host> 3306
mysql -h <db-host> -u <monitor_user> -p
Code language: HTML, XML (xml)

Common causes:

9. AWS / Cloud Integration Troubleshooting

9.1 AWS integration not working

Check:

A Datadog AWS sts:AssumeRole error means the trust policy associated with the Datadog integration role is wrong, and Datadog notes that the error may persist in the UI for a few hours while changes propagate. (Datadog)

9.2 CloudWatch and Datadog metric values differ

This is common and not always an error.

Causes:

Datadog’s AWS troubleshooting guide specifically calls out time aggregation, space aggregation, matching dimensions/tags, one-minute precision, region, namespace, metric name, and CloudWatch API delays. (Datadog)

10. API and Terraform Troubleshooting

10.1 API returns 403 Forbidden

Likely causes:

Remember: API key validates organization intake access; app key controls programmatic actions and permissions.

10.2 API returns 401 Unauthorized

Likely causes:

Datadog recommends allowing a brief propagation period for new API/application keys and using retry with short exponential backoff for transient errors. (Datadog)

10.3 Terraform provider cannot connect

Check provider config:

provider "datadog" {
  api_key = var.datadog_api_key
  app_key = var.datadog_app_key
  api_url = "https://api.datadoghq.com/"
}
Code language: JavaScript (javascript)

For EU/AP/etc., use the correct site API URL.

Common errors:

The Datadog Terraform provider is used to manage Datadog resources such as monitors, dashboards, logs configuration, and integrations through Terraform. (Datadog)

10.4 Terraform monitor fails validation

Common causes:

Best practice: manually test query in Datadog Metrics Explorer before putting it into Terraform.

11. Monitor and Alert Troubleshooting

11.1 Monitor not alerting

Check:

11.2 Monitor stuck in No Data

Causes:

Datadog’s No Data troubleshooting warns that rollups and evaluation windows can cause unexpected No Data behavior, especially when Require Full Window is enabled and the rollup interval does not fully populate the evaluation window. (Datadog)

For AWS/crawler-based metrics, Datadog recommends using an evaluation delay, commonly at least 900 seconds for AWS CloudWatch delays. (Datadog)

11.3 Monitor alerts too noisy

Fix strategy:

Good alert rule:

Page only when customer impact is likely and action is required.

Bad alert:

CPU > 80% for 1 minute

Better alert:

Checkout p95 latency > 1s and error rate > 5% for 10 minutes in prod

12. SLO Troubleshooting

12.1 SLO shows wrong reliability

Check:

SLOs are used to define and track reliability targets and help teams communicate service performance; Datadog also exposes burn rate status such as healthy, elevated, and critical. (Datadog)

12.2 Burn-rate alert too noisy or too late

Use multiple windows:

For monitor-based SLOs, Datadog notes that burn-rate alerts can only be set on the overall SLO, not per group. (Datadog)

13. Synthetic Monitoring Troubleshooting

13.1 API test fails but endpoint works manually

Check:

13.2 Browser test is flaky

Common causes:

13.3 Browser recorder does not work

Datadog’s Synthetics troubleshooting explains that CSP or X-Frame-Options can prevent the recorder iframe from loading the site, and the workaround is to record in a popup. It also notes HTTP pages do not load in the iframe recorder, extension permissions can block recording, Chrome policies can block the extension, and incognito mode can help record login flows from a fresh session. (Datadog)

Practical checks:

CSP
X-Frame-Options
CSRF token behavior
Chrome extension permissions
Corporate browser policy
Login/session state
HTTP vs HTTPS

13.4 Private Location troubleshooting

Private Location issues are usually network or runtime issues.

Check:

14. RUM and Session Replay Troubleshooting

14.1 RUM data missing

Check:

Datadog RUM captures real user performance, errors, analytics/usage, and support data; sessions include views, actions, resources, errors, and related signals. A RUM session can last up to 4 hours and expires after 15 minutes of inactivity. (Datadog)

14.2 Session Replay missing

Check:

14.3 RUM errors do not connect to APM traces

Check:

Allowed tracing URLs
CORS headers
Backend tracer enabled
Trace context propagation
service/env/version alignment

Frontend-to-backend trace correlation requires both RUM and backend APM to propagate compatible trace context.

15. OpenTelemetry Troubleshooting

15.1 OTel data not reaching Datadog

Possible data paths:

flowchart TB
    A[OTel SDK] --> B[OTel Collector]
    B --> C[Datadog Exporter]
    C --> D[Datadog Intake]

    A2[OTel SDK] --> B2[Datadog Agent OTLP Receiver]
    B2 --> D
Code language: CSS (css)

Datadog supports OpenTelemetry data through the OpenTelemetry Collector with Datadog exporter and through OTLP ingest in the Datadog Agent. (Datadog)

15.2 OTLP ingest into Datadog Agent not working

Check Agent OTLP receiver config.

Important:

OTLP gRPC default: 4317
OTLP HTTP default: 4318
Code language: JavaScript (javascript)

Application:

env | grep OTEL_

Common variables:

OTEL_EXPORTER_OTLP_ENDPOINT
OTEL_SERVICE_NAME
OTEL_RESOURCE_ATTRIBUTES
OTEL_TRACES_EXPORTER
OTEL_METRICS_EXPORTER
OTEL_LOGS_EXPORTER

Datadog Agent supports OTLP traces and metrics through gRPC or HTTP since Agent 6.32/7.32, and OTLP logs since Agent 6.48/7.48. Datadog also warns that OTLP ingest is intended for an Agent local to each telemetry-generating host, not a remote Agent on another host. (Datadog)

15.3 Duplicate hosts with OpenTelemetry

Cause:

You are sending host telemetry through both the Datadog Agent and OpenTelemetry with inconsistent hostname/resource attributes.

Fix:

Standardize:

host.name
service.name
deployment.environment
service.version
Code language: CSS (css)

Datadog’s OTel troubleshooting notes that duplicate host entries can happen when a host is monitored through multiple ingestion methods, such as OTLP plus Datadog Agent or DogStatsD plus OTLP, without aligning on a single hostname resource attribute. (Datadog)

15.4 Some Datadog features missing with OTel

OpenTelemetry is great, but not every Datadog-native capability works the same way with only OTel SDK instrumentation.

Datadog notes that data from applications instrumented with OpenTelemetry SDKs cannot be used in some proprietary products, including App and API Protection, Continuous Profiler, and Ingestion Rules. (Datadog)

So when debugging OTel setups, ask:

Are we using OTel SDK only?
Are we using Datadog tracer?
Are we using Datadog Agent OTLP ingest?
Are we using Datadog Distribution of OpenTelemetry Collector?
Which Datadog features are required?

16. Security / Cloud SIEM / Sensitive Data Troubleshooting

16.1 Security signals missing

Check:

16.2 Sensitive data still visible

Check:

Best practice: prevent secrets at source. Scanning is a safety net, not the main control.

17. Cloud Cost / Datadog Cost Troubleshooting

17.1 Cloud cost data missing

Check:

Datadog Cloud Cost Management ingests cloud cost data and transforms it into metrics that can be queried in Cost Explorer; it is meant to correlate cost increases with usage and infrastructure changes. (Datadog)

17.2 Datadog bill increased

Investigate:

Common root causes:

containerCollectAll enabled without exclusions
debug logs enabled
high-cardinality custom metric tags
new Kubernetes cluster added
logs indexed for all environments
synthetics running every minute from many locations
trace sampling too high
RUM sample rate too high
Code language: JavaScript (javascript)

18. Dashboard and Query Troubleshooting

18.1 Dashboard shows no data

Check:

18.2 Dashboard values differ from monitor values

Common reasons:

19. Incident Troubleshooting Method

When there is a production incident, do not randomly click dashboards. Use a fixed flow.

flowchart TD
    A[Alert Fires] --> B[Confirm Customer Impact]
    B --> C[Check SLO / Error Budget]
    C --> D[Identify Service + Env + Version]
    D --> E[Check Recent Changes]
    E --> F[Metrics: traffic, errors, latency, saturation]
    F --> G[Traces: slow/failing requests]
    G --> H[Logs: errors around trace IDs]
    H --> I[Infra/K8s: pods, nodes, deploys]
    I --> J[Cloud/DB/Queue Dependencies]
    J --> K[Mitigate]
    K --> L[Postmortem + Monitor Fixes]
Code language: CSS (css)

Golden questions:

What changed?
Who is affected?
Which service owns the issue?
Is it regional?
Is it version-specific?
Is it dependency-specific?
Is it traffic-related?
Is the alert actionable?
Is this an observability gap?
Code language: JavaScript (javascript)

20. Troubleshooting Cheat Sheet

Agent

sudo datadog-agent status
sudo datadog-agent health
sudo datadog-agent diagnose
sudo datadog-agent configcheck
sudo datadog-agent check <integration>
sudo datadog-agent flare
sudo journalctl -u datadog-agent.service -r
sudo tail -f /var/log/datadog/agent.log
Code language: HTML, XML (xml)

Docker Agent

docker ps | grep datadog
docker logs <agent-container>
docker exec -it <agent-container> agent status
docker exec -it <agent-container> agent configcheck
Code language: HTML, XML (xml)

Kubernetes Agent

kubectl get pods -n datadog -o wide
kubectl logs -n datadog <agent-pod>
kubectl describe pod -n datadog <agent-pod>
kubectl exec -it -n datadog <agent-pod> -- agent status
kubectl exec -it -n datadog <agent-pod> -- agent configcheck
kubectl exec -it -n datadog <agent-pod> -- agent stream-logs
Code language: HTML, XML (xml)

Cluster Agent

kubectl get deploy -n datadog
kubectl logs -n datadog deploy/datadog-cluster-agent
kubectl exec -n datadog deploy/datadog-cluster-agent -- datadog-cluster-agent status
Code language: JavaScript (javascript)

APM

env | grep DD_
curl -v http://${DD_AGENT_HOST}:8126/info
sudo datadog-agent status
sudo tail -f /var/log/datadog/trace-agent.log
Code language: JavaScript (javascript)

Logs

kubectl logs <pod> -n <namespace>
docker logs <container>
sudo datadog-agent stream-logs
sudo datadog-agent status
Code language: HTML, XML (xml)

API

curl -X GET "https://api.<DATADOG_SITE>/api/v1/validate" \
  -H "DD-API-KEY: ${DD_API_KEY}"

curl -X GET "https://api.<DATADOG_SITE>/api/v2/validate_keys" \
  -H "DD-API-KEY: ${DD_API_KEY}" \
  -H "DD-APPLICATION-KEY: ${DD_APP_KEY}"
Code language: JavaScript (javascript)

21. Most Common Datadog Problems and Fast Fixes

22. Best-Practice Troubleshooting Mindset

The best Datadog engineers do not start with dashboards. They start with the telemetry path.

Ask in this order:

1. Is the source producing data?
2. Is the Agent/SDK/Collector receiving it?
3. Can it reach Datadog intake?
4. Is Datadog accepting it?
5. Is it being processed, indexed, retained, or sampled?
6. Are we querying it correctly?
7. Do tags match our standard?
8. Is the user allowed to see it?
9. Is cost control intentionally dropping or reducing it?
10. Can we prevent the same issue with better tagging, monitors, dashboards, or IaC?

The biggest real-world Datadog issues are usually not “Datadog is broken.” They are usually:

wrong site
wrong key
wrong tag
wrong environment
wrong Agent config
wrong Kubernetes networking
wrong log index
wrong trace endpoint
wrong monitor window
wrong permissions
too much cardinality
too much noisy data
Code language: JavaScript (javascript)

That’s the Datadog troubleshooting game in one sentence: follow the telemetry, then follow the tags.

Rajesh Kumar

I’m a DevOps/SRE/DevSecOps/Cloud Expert passionate about sharing knowledge and experiences. I have worked at Cotocus. I share tech blog at DevOps School, travel stories at Holiday Landmark, stock market tips at Stocks Mantra, health and fitness guidance at My Medic Plus, product reviews at TrueReviewNow , and SEO strategies at Wizbrand.

Do you want to learn Quantum Computing?

Please find my social handles as below;

Rajesh Kumar Personal Website
Rajesh Kumar at YOUTUBE
Rajesh Kumar at INSTAGRAM
Rajesh Kumar at X
Rajesh Kumar at FACEBOOK
Rajesh Kumar at LINKEDIN
Rajesh Kumar at WIZBRAND

Rajesh Kumar DailyLogs