{"id":33511,"date":"2026-06-23T02:02:21","date_gmt":"2026-06-23T02:02:21","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=33511"},"modified":"2026-06-23T02:02:24","modified_gmt":"2026-06-23T02:02:24","slug":"datadog-tutorials-create-monitor-alerts-using-datadog-api","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/datadog-tutorials-create-monitor-alerts-using-datadog-api\/","title":{"rendered":"Datadog Tutorial: Create Monitors \/ Alerts using Datadog API \u2014 Step by Step"},"content":{"rendered":"\n

<\/p>\n\n\n\n

This guide uses the current Datadog Monitor API v1<\/strong>, which is still the main API for creating metric, log, APM, and many other monitor types. Datadog\u2019s API docs list \/api\/v1\/monitor<\/code> for create\/get\/update\/delete monitor operations, and \/api\/v1\/monitor\/validate<\/code> for validating a monitor definition before creating it. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n

We will create monitors using:<\/p>\n\n\n

curl\nPython\nJSON<\/span> payload files\nDatadog API keys from<\/span> environment variables\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
No outdated dogapi<\/code> or old legacy-only approach. We will use direct REST API and the current datadog-api-client<\/code> Python library where needed.<\/p>\n\n\n\n
\n\n\n\n
1. What is a Datadog Monitor?<\/h1>\n\n\n\n
A Datadog Monitor<\/strong> is an alerting rule.<\/p>\n\n\n\n
It watches something like:<\/p>\n\n\n
CPU\nMemory\nDisk\nLogs\nAPM traces\nError<\/span> rate\nLatency\nSynthetic test result\nKubernetes pod status\nCustom metric\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Then it triggers alert states such as:<\/p>\n\n\n
OK\nWarn\nAlert\nNo Data\n<\/code><\/span><\/pre>\n\n\n
A monitor normally contains:<\/p>\n\n\n
{\n  \"name\"<\/span>: \"Monitor name\"<\/span>,\n  \"type\"<\/span>: \"query alert\"<\/span>,\n  \"query\"<\/span>: \"avg(last_5m):avg:system.cpu.user{*} > 80\"<\/span>,\n  \"message\"<\/span>: \"Alert message\"<\/span>,\n  \"tags\"<\/span>: [\"env:prod\"<\/span>, \"team:devops\"<\/span>],\n  \"options\"<\/span>: {\n    \"thresholds\"<\/span>: {\n      \"critical\"<\/span>: 80<\/span>,\n      \"warning\"<\/span>: 70<\/span>\n    }\n  }\n}\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
2. Monitor API endpoint<\/h1>\n\n\n\n
The main API endpoint is:<\/p>\n\n\n
POST \/api\/v1\/monitor\n<\/code><\/span><\/pre>\n\n\n
Full pattern:<\/p>\n\n\n
https:\/\/api.<DATADOG_SITE<\/span>><\/span>\/api\/v1\/monitor\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Examples:<\/p>\n\n\n
US1:     https:\/\/api.datadoghq.com\/api\/v1\/monitor<\/span>\nUS3:     https:\/\/api.us3.datadoghq.com\/api\/v1\/monitor<\/span>\nUS5:     https:\/\/api.us5.datadoghq.com\/api\/v1\/monitor<\/span>\nEU:      https:\/\/api.datadoghq.eu\/api\/v1\/monitor<\/span>\nAP1:     https:\/\/api.ap1.datadoghq.com\/api\/v1\/monitor<\/span>\nAP2:     https:\/\/api.ap2.datadoghq.com\/api\/v1\/monitor<\/span>\nGov:     https:\/\/api.ddog-gov.com\/api\/v1\/monitor<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
3. Required API keys<\/h1>\n\n\n\n
You need:<\/p>\n\n\n
DD_API_KEY\nDD_APP_KEY\nDD_SITE\n<\/code><\/span><\/pre>\n\n\n
Datadog states that API requests must be authenticated. API keys are used for write\/reporting access, and application keys are also required for read operations and many API actions. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
3.1 Export environment variables<\/h2>\n\n\n\n
Use this on your terminal:<\/p>\n\n\n
export<\/span> DD_SITE=\"datadoghq.com\"<\/span>\nexport<\/span> DD_API_KEY=\"replace_with_your_datadog_api_key\"<\/span>\nexport<\/span> DD_APP_KEY=\"replace_with_your_datadog_application_key\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
For Japan\/AP1 site, use:<\/p>\n\n\n
export<\/span> DD_SITE=\"ap1.datadoghq.com\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
For EU:<\/p>\n\n\n
export<\/span> DD_SITE=\"datadoghq.eu\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Do not<\/strong> hardcode real keys inside scripts.<\/p>\n\n\n\n
\n\n\n\n
4. Validate API access<\/h1>\n\n\n\n
First validate your API key.<\/p>\n\n\n
curl -sS -X GET \"https:\/\/api.${DD_SITE}\/api\/v1\/validate\"<\/span> \\\n  -H \"Accept: application\/json\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> | jq\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Expected output:<\/p>\n\n\n
{\n  \"valid\"<\/span>: true<\/span>\n}\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
5. Recommended folder structure<\/h1>\n\n\n\n
Create a small working directory:<\/p>\n\n\n
mkdir -p datadog-monitor-api-lab\/monitors\ncd datadog-monitor-api-lab\n<\/code><\/span><\/pre>\n\n\n
Recommended structure:<\/p>\n\n\n
datadog-monitor-api-lab\/\n\u251c\u2500\u2500 monitors\/\n\u2502   \u251c\u2500\u2500 cpu-idle-low.json\n\u2502   \u251c\u2500\u2500 apache-5<\/span>xx-log-monitor.json\n\u2502   \u251c\u2500\u2500 apm-error-rate.json\n\u2502   \u2514\u2500\u2500 disk-space-low.json\n\u251c\u2500\u2500 create-monitor.sh\n\u251c\u2500\u2500 validate-monitor.sh\n\u251c\u2500\u2500 get<\/span>-monitor.sh\n\u251c\u2500\u2500 update-monitor.sh\n\u2514\u2500\u2500 delete-monitor.sh\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
6. Validate monitor JSON before creating<\/h1>\n\n\n\n
This is very important.<\/p>\n\n\n\n
Before creating a monitor, use:<\/p>\n\n\n
POST \/api\/v1\/monitor\/validate\n<\/code><\/span><\/pre>\n\n\n
Datadog provides a monitor validation endpoint and examples for validating monitor definitions before creating them. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
Create this script:<\/p>\n\n\n
cat > validate-monitor.sh <<'EOF'<\/span>\n#!\/usr\/bin\/env bash<\/span>\nset -euo pipefail\n\nFILE=\"${1:?Usage: .\/validate-monitor.sh <monitor-json-file>}\"<\/span>\n\ncurl -sS -X POST \"https:\/\/api.${DD_SITE}\/api\/v1\/monitor\/validate\"<\/span> \\\n  -H \"Accept: application\/json\"<\/span> \\\n  -H \"Content-Type: application\/json\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\"<\/span> \\\n  -d @\"${FILE}\"<\/span> | jq\nEOF\n\nchmod +x validate-monitor.sh\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Usage:<\/p>\n\n\n
.\/validate-monitor.sh monitors\/cpu-idle-low.json\n<\/code><\/span><\/pre>\n\n\n
If valid, Datadog usually returns an empty object or success-style response.<\/p>\n\n\n\n
\n\n\n\n
7. Create monitor script<\/h1>\n\n\n\n
Create:<\/p>\n\n\n
cat > create-monitor.sh <<'EOF'<\/span>\n#!\/usr\/bin\/env bash<\/span>\nset -euo pipefail\n\nFILE=\"${1:?Usage: .\/create-monitor.sh <monitor-json-file>}\"<\/span>\n\ncurl -sS -X POST \"https:\/\/api.${DD_SITE}\/api\/v1\/monitor\"<\/span> \\\n  -H \"Accept: application\/json\"<\/span> \\\n  -H \"Content-Type: application\/json\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\"<\/span> \\\n  -d @\"${FILE}\"<\/span> | jq\nEOF\n\nchmod +x create-monitor.sh\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Usage:<\/p>\n\n\n
.\/create-monitor.sh monitors\/cpu-idle-low.json\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
8. Example 1: Create CPU monitor using Datadog API<\/h1>\n\n\n\n
This monitor alerts when CPU idle is too low. Low idle means CPU is heavily used.<\/p>\n\n\n\n
Create JSON file:<\/p>\n\n\n
cat > monitors\/cpu-idle-low.json <<'EOF'<\/span>\n{\n  \"name\"<\/span>: \"[LAB] High CPU Usage - CPU idle below 10% by host\"<\/span>,\n  \"type\"<\/span>: \"query alert\"<\/span>,\n  \"query\"<\/span>: \"avg(last_5m):avg:system.cpu.idle{*} by {host} < 10\"<\/span>,\n  \"message\"<\/span>: \"CPU idle is below 10% for host {{host.name}}.\\n\\nCurrent value: {{value}}\\n\\nPlease check:\\n- Top processes\\n- Recent deployments\\n- Load average\\n- Container CPU usage\\n\\nNotify: @slack-devops-alerts\"<\/span>,\n  \"tags\"<\/span>: [\n    \"lab:datadog-api\"<\/span>,\n    \"monitor_type:metric\"<\/span>,\n    \"team:devops\"<\/span>,\n    \"env:training\"<\/span>\n  ],\n  \"priority\"<\/span>: 3<\/span>,\n  \"options\"<\/span>: {\n    \"thresholds\"<\/span>: {\n      \"critical\"<\/span>: 10<\/span>,\n      \"warning\"<\/span>: 20<\/span>\n    },\n    \"notify_no_data\"<\/span>: false<\/span>,\n    \"include_tags\"<\/span>: true<\/span>,\n    \"new_group_delay\"<\/span>: 300<\/span>,\n    \"renotify_interval\"<\/span>: 60<\/span>,\n    \"require_full_window\"<\/span>: false<\/span>,\n    \"notification_preset_name\"<\/span>: \"show_all\"<\/span>\n  }\n}\nEOF\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Validate:<\/p>\n\n\n
.\/validate-monitor.sh monitors\/cpu-idle-low.json\n<\/code><\/span><\/pre>\n\n\n
Create:<\/p>\n\n\n
.\/create-monitor.sh monitors\/cpu-idle-low.json\n<\/code><\/span><\/pre>\n\n\n
Expected response includes an id<\/code>:<\/p>\n\n\n
{\n  \"id\"<\/span>: 123456789<\/span>,\n  \"name\"<\/span>: \"[LAB] High CPU Usage - CPU idle below 10% by host\"<\/span>,\n  \"type\"<\/span>: \"query alert\"<\/span>\n}\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
Save the monitor ID:<\/p>\n\n\n
export<\/span> MONITOR_ID=\"123456789\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
9. Example 2: Create disk space monitor<\/h1>\n\n\n\n
This alerts when root filesystem usage is above 85%.<\/p>\n\n\n
cat > monitors\/disk-space-low.json <<'EOF'<\/span>\n{\n  \"name\"<\/span>: \"[LAB] Disk Usage High - Root filesystem above 85%\"<\/span>,\n  \"type\"<\/span>: \"query alert\"<\/span>,\n  \"query\"<\/span>: \"avg(last_5m):avg:system.disk.in_use{device:\/} by {host} > 0.85\"<\/span>,\n  \"message\"<\/span>: \"Root disk usage is above 85% on {{host.name}}.\\n\\nCurrent value: {{value}}\\n\\nAction:\\n- Check large files\\n- Check logs\\n- Check Docker\/container disk usage\\n- Clean old temporary files\\n\\nNotify: @slack-devops-alerts\"<\/span>,\n  \"tags\"<\/span>: [\n    \"lab:datadog-api\"<\/span>,\n    \"monitor_type:metric\"<\/span>,\n    \"team:devops\"<\/span>,\n    \"env:training\"<\/span>\n  ],\n  \"priority\"<\/span>: 3<\/span>,\n  \"options\"<\/span>: {\n    \"thresholds\"<\/span>: {\n      \"critical\"<\/span>: 0.85<\/span>,\n      \"warning\"<\/span>: 0.75<\/span>\n    },\n    \"notify_no_data\"<\/span>: false<\/span>,\n    \"include_tags\"<\/span>: true<\/span>,\n    \"new_group_delay\"<\/span>: 300<\/span>,\n    \"renotify_interval\"<\/span>: 120<\/span>,\n    \"require_full_window\"<\/span>: false<\/span>\n  }\n}\nEOF\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Validate and create:<\/p>\n\n\n
.\/validate-monitor.sh monitors\/disk-space-low.json\n.\/create-monitor.sh monitors\/disk-space-low.json\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
10. Example 3: Create Apache 5xx log monitor<\/h1>\n\n\n\n
This monitor watches Apache logs and alerts if 5xx errors exceed 5 logs in the last 5 minutes.<\/p>\n\n\n\n
Datadog log monitors evaluate indexed logs only, and log monitor queries use the same search logic as Log Explorer. Datadog also notes that log monitors have a maximum rolling window of two days. (Datadog Monitoring<\/a>)<\/p>\n\n\n
cat > monitors\/apache-5<\/span>xx-log-monitor.json <<'EOF'<\/span>\n{\n  \"name\"<\/span>: \"[LAB] Apache 5xx Errors High by Host\"<\/span>,\n  \"type\"<\/span>: \"log alert\"<\/span>,\n  \"query\"<\/span>: \"logs(\\\"source:apache @http.status_code:[500 TO 599]\\\").index(\\\"*\\\").rollup(\\\"count\\\").by(\\\"host\\\").last(\\\"5m\\\") > 5\"<\/span>,\n  \"message\"<\/span>: \"Apache 5xx errors are high on host {{host.name}}.\\n\\nCurrent value: {{value}}\\n\\nInvestigation steps:\\n- Open Log Explorer\\n- Search: source:apache @http.status_code:[500 TO 599]\\n- Group by host and URL path\\n- Check Apache error logs\\n- Check backend application health\\n\\nNotify: @slack-devops-alerts\"<\/span>,\n  \"tags\"<\/span>: [\n    \"lab:datadog-api\"<\/span>,\n    \"monitor_type:log\"<\/span>,\n    \"team:devops\"<\/span>,\n    \"service:apache\"<\/span>,\n    \"env:training\"<\/span>\n  ],\n  \"priority\"<\/span>: 2<\/span>,\n  \"options\"<\/span>: {\n    \"thresholds\"<\/span>: {\n      \"critical\"<\/span>: 5<\/span>,\n      \"warning\"<\/span>: 2<\/span>\n    },\n    \"enable_logs_sample\"<\/span>: true<\/span>,\n    \"notify_no_data\"<\/span>: false<\/span>,\n    \"include_tags\"<\/span>: true<\/span>,\n    \"renotify_interval\"<\/span>: 60<\/span>,\n    \"require_full_window\"<\/span>: false<\/span>,\n    \"on_missing_data\"<\/span>: \"resolve\"<\/span>\n  }\n}\nEOF\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Validate and create:<\/p>\n\n\n
.\/validate-monitor.sh monitors\/apache-5xx-log-monitor.json\n.\/create-monitor.sh monitors\/apache-5xx-log-monitor.json\n<\/code><\/span><\/pre>\n\n\n
If your Apache logs do not have @http.status_code<\/code>, use a fallback query:<\/p>\n\n\n
logs(\"source:apache 500 OR 502 OR 503 OR 504\"<\/span>).index(\"*\"<\/span>).rollup(\"count\"<\/span>).by(\"host\"<\/span>).last(\"5m\"<\/span>) > 5<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
11. Example 4: Create Ubuntu failed SSH login log monitor<\/h1>\n\n\n\n
This alerts when failed SSH attempts exceed 10 in 5 minutes.<\/p>\n\n\n
cat > monitors\/ubuntu-failed-ssh-log-monitor.json <<'EOF'<\/span>\n{\n  \"name\"<\/span>: \"[LAB] Ubuntu Failed SSH Login Attempts High\"<\/span>,\n  \"type\"<\/span>: \"log alert\"<\/span>,\n  \"query\"<\/span>: \"logs(\\\"\\\\\\\"Failed password\\\\\\\" OR \\\\\\\"Invalid user\\\\\\\"\\\").index(\\\"*\\\").rollup(\\\"count\\\").by(\\\"host\\\").last(\\\"5m\\\") > 10\"<\/span>,\n  \"message\"<\/span>: \"High failed SSH login activity detected on {{host.name}}.\\n\\nCurrent value: {{value}}\\n\\nInvestigation:\\n- Search logs for: \\\"Failed password\\\" OR \\\"Invalid user\\\"\\n- Check source IPs\\n- Check usernames attempted\\n- Consider firewall\/security group review\\n\\nNotify: @slack-security-alerts\"<\/span>,\n  \"tags\"<\/span>: [\n    \"lab:datadog-api\"<\/span>,\n    \"monitor_type:log\"<\/span>,\n    \"team:security\"<\/span>,\n    \"source:ubuntu\"<\/span>,\n    \"env:training\"<\/span>\n  ],\n  \"priority\"<\/span>: 2<\/span>,\n  \"options\"<\/span>: {\n    \"thresholds\"<\/span>: {\n      \"critical\"<\/span>: 10<\/span>,\n      \"warning\"<\/span>: 5<\/span>\n    },\n    \"enable_logs_sample\"<\/span>: true<\/span>,\n    \"notify_no_data\"<\/span>: false<\/span>,\n    \"include_tags\"<\/span>: true<\/span>,\n    \"renotify_interval\"<\/span>: 60<\/span>,\n    \"require_full_window\"<\/span>: false<\/span>,\n    \"on_missing_data\"<\/span>: \"resolve\"<\/span>\n  }\n}\nEOF\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Validate and create:<\/p>\n\n\n
.\/validate-monitor.sh monitors\/ubuntu-failed-ssh-log-monitor.json\n.\/create-monitor.sh monitors\/ubuntu-failed-ssh-log-monitor.json\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
12. Example 5: Create APM service error rate monitor<\/h1>\n\n\n\n
APM monitors can alert at service level on hits, errors, and latency measures. Datadog states APM metric monitors work like regular metric monitors but are tailored for APM, and Analytics monitors can alert on Indexed Spans. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
This example uses Datadog APM trace metrics.<\/p>\n\n\n
cat > monitors\/apm-error-rate.json <<'EOF'<\/span>\n{\n  \"name\"<\/span>: \"[LAB] APM Error Rate High - checkout-api\"<\/span>,\n  \"type\"<\/span>: \"query alert\"<\/span>,\n  \"query\"<\/span>: \"sum(last_5m):sum:trace.web.request.errors{service:checkout-api,env:prod}.as_count() \/ sum:trace.web.request.hits{service:checkout-api,env:prod}.as_count() * 100 > 5\"<\/span>,\n  \"message\"<\/span>: \"APM error rate is above 5% for checkout-api in prod.\\n\\nCurrent value: {{value}}%\\n\\nCheck:\\n- APM Service Page\\n- Error traces\\n- Recent deployments\\n- Related logs\\n- Downstream dependency errors\\n\\nNotify: @slack-devops-alerts\"<\/span>,\n  \"tags\"<\/span>: [\n    \"lab:datadog-api\"<\/span>,\n    \"monitor_type:apm\"<\/span>,\n    \"service:checkout-api\"<\/span>,\n    \"env:prod\"<\/span>,\n    \"team:backend\"<\/span>\n  ],\n  \"priority\"<\/span>: 1<\/span>,\n  \"options\"<\/span>: {\n    \"thresholds\"<\/span>: {\n      \"critical\"<\/span>: 5<\/span>,\n      \"warning\"<\/span>: 2<\/span>\n    },\n    \"notify_no_data\"<\/span>: false<\/span>,\n    \"include_tags\"<\/span>: true<\/span>,\n    \"renotify_interval\"<\/span>: 30<\/span>,\n    \"require_full_window\"<\/span>: false<\/span>,\n    \"new_group_delay\"<\/span>: 300<\/span>\n  }\n}\nEOF\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Validate and create:<\/p>\n\n\n
.\/validate-monitor.sh monitors\/apm-error-rate.json\n.\/create-monitor.sh monitors\/apm-error-rate.json\n<\/code><\/span><\/pre>\n\n\n
Important: Replace trace.web.request.errors<\/code> and trace.web.request.hits<\/code> with the actual operation-specific APM metrics used by your service. Common examples are based on operation names such as:<\/p>\n\n\n
trace<\/span>.web<\/span>.request<\/span>.hits<\/span>\ntrace<\/span>.web<\/span>.request<\/span>.errors<\/span>\ntrace<\/span>.servlet<\/span>.request<\/span>.hits<\/span>\ntrace<\/span>.servlet<\/span>.request<\/span>.errors<\/span>\ntrace<\/span>.fastapi<\/span>.request<\/span>.hits<\/span>\ntrace<\/span>.fastapi<\/span>.request<\/span>.errors<\/span>\ntrace<\/span>.express<\/span>.request<\/span>.hits<\/span>\ntrace<\/span>.express<\/span>.request<\/span>.errors<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Check Datadog Metrics Explorer for your actual trace metric names before finalizing.<\/p>\n\n\n\n
\n\n\n\n
13. Example 6: Create Kubernetes pod restart monitor<\/h1>\n\n\n\n
This alerts when Kubernetes container restarts increase.<\/p>\n\n\n
cat > monitors\/kubernetes-container-restarts.json <<'EOF'<\/span>\n{\n  \"name\"<\/span>: \"[LAB] Kubernetes Container Restarts High\"<\/span>,\n  \"type\"<\/span>: \"query alert\"<\/span>,\n  \"query\"<\/span>: \"sum(last_10m):sum:kubernetes.containers.restarts{*} by {kube_namespace,kube_deployment} > 3\"<\/span>,\n  \"message\"<\/span>: \"Kubernetes container restarts are high.\\n\\nNamespace: {{kube_namespace.name}}\\nDeployment: {{kube_deployment.name}}\\nCurrent value: {{value}}\\n\\nCheck:\\n- kubectl describe pod\\n- CrashLoopBackOff\\n- OOMKilled\\n- Recent deployment\\n- Application logs\\n\\nNotify: @slack-devops-alerts\"<\/span>,\n  \"tags\"<\/span>: [\n    \"lab:datadog-api\"<\/span>,\n    \"monitor_type:kubernetes\"<\/span>,\n    \"team:devops\"<\/span>,\n    \"env:training\"<\/span>\n  ],\n  \"priority\"<\/span>: 2<\/span>,\n  \"options\"<\/span>: {\n    \"thresholds\"<\/span>: {\n      \"critical\"<\/span>: 3<\/span>,\n      \"warning\"<\/span>: 1<\/span>\n    },\n    \"notify_no_data\"<\/span>: false<\/span>,\n    \"include_tags\"<\/span>: true<\/span>,\n    \"renotify_interval\"<\/span>: 60<\/span>,\n    \"require_full_window\"<\/span>: false<\/span>,\n    \"new_group_delay\"<\/span>: 300<\/span>\n  }\n}\nEOF\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Validate and create:<\/p>\n\n\n
.\/validate-monitor.sh monitors\/kubernetes-container-restarts.json\n.\/create-monitor.sh monitors\/kubernetes-container-restarts.json\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
14. Get monitor details by ID<\/h1>\n\n\n\n
Create script:<\/p>\n\n\n
cat > get-monitor.sh <<'EOF'<\/span>\n#!\/usr\/bin\/env bash<\/span>\nset -euo pipefail\n\nMONITOR_ID=\"${1:?Usage: .\/get-monitor.sh <monitor-id>}\"<\/span>\n\ncurl -sS -X GET \"https:\/\/api.${DD_SITE}\/api\/v1\/monitor\/${MONITOR_ID}\"<\/span> \\\n  -H \"Accept: application\/json\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\"<\/span> | jq\nEOF\n\nchmod +x get-monitor.sh\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Usage:<\/p>\n\n\n
.\/get<\/span>-monitor.sh 123456789\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Datadog lists GET \/api\/v1\/monitor\/{monitor_id}<\/code> for retrieving a monitor\u2019s details. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
\n\n\n\n
15. Search monitors<\/h1>\n\n\n\n
Search monitors by query:<\/p>\n\n\n
curl -sS -X GET \"https:\/\/api.${DD_SITE}\/api\/v1\/monitor\/search?query=lab:datadog-api\"<\/span> \\\n  -H \"Accept: application\/json\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\"<\/span> | jq\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Search by name:<\/p>\n\n\n
curl -sS -X GET \"https:\/\/api.${DD_SITE}\/api\/v1\/monitor\/search?query=%5BLAB%5D\"<\/span> \\\n  -H \"Accept: application\/json\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\"<\/span> | jq\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
16. Get all monitors<\/h1>\n\n\n
curl -sS -X GET \"https:\/\/api.${DD_SITE}\/api\/v1\/monitor\"<\/span> \\\n  -H \"Accept: application\/json\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\"<\/span> | jq\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
17. Update an existing monitor<\/h1>\n\n\n\n
To update a monitor, use:<\/p>\n\n\n
PUT \/api\/v1\/monitor\/{monitor_id}\n<\/code><\/span><\/pre>\n\n\n
Datadog lists PUT \/api\/v1\/monitor\/{monitor_id}<\/code> as the edit monitor endpoint. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
Create script:<\/p>\n\n\n
cat > update-monitor.sh <<'EOF'<\/span>\n#!\/usr\/bin\/env bash<\/span>\nset -euo pipefail\n\nMONITOR_ID=\"${1:?Usage: .\/update-monitor.sh <monitor-id> <monitor-json-file>}\"<\/span>\nFILE=\"${2:?Usage: .\/update-monitor.sh <monitor-id> <monitor-json-file>}\"<\/span>\n\ncurl -sS -X PUT \"https:\/\/api.${DD_SITE}\/api\/v1\/monitor\/${MONITOR_ID}\"<\/span> \\\n  -H \"Accept: application\/json\"<\/span> \\\n  -H \"Content-Type: application\/json\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\"<\/span> \\\n  -d @\"${FILE}\"<\/span> | jq\nEOF\n\nchmod +x update-monitor.sh\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Example: change CPU critical threshold from 10<\/code> to 5<\/code>.<\/p>\n\n\n
cp monitors\/cpu-idle-low.json monitors\/cpu-idle-low-updated.json\n<\/code><\/span><\/pre>\n\n\n
Edit:<\/p>\n\n\n
\"critical\"<\/span>: 5<\/span>,\n\"warning\"<\/span>: 15<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Also update the query:<\/p>\n\n\n
\"query\"<\/span>: \"avg(last_5m):avg:system.cpu.idle{*} by {host} < 5\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Then update:<\/p>\n\n\n
.\/update-monitor.sh 123456789 monitors\/cpu-idle-low-updated.json\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
18. Delete a monitor<\/h1>\n\n\n\n
Create script:<\/p>\n\n\n
cat > delete-monitor.sh <<'EOF'<\/span>\n#!\/usr\/bin\/env bash<\/span>\nset -euo pipefail\n\nMONITOR_ID=\"${1:?Usage: .\/delete-monitor.sh <monitor-id>}\"<\/span>\n\ncurl -sS -X DELETE \"https:\/\/api.${DD_SITE}\/api\/v1\/monitor\/${MONITOR_ID}\"<\/span> \\\n  -H \"Accept: application\/json\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\"<\/span> | jq\nEOF\n\nchmod +x delete-monitor.sh\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Usage:<\/p>\n\n\n
.\/delete<\/span>-monitor.sh 123456789<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
19. Python version using current Datadog API client<\/h1>\n\n\n\n
Install:<\/p>\n\n\n
python3 -m venv .venv\nsource .venv\/bin\/activate\npip install datadog-api-client\n<\/code><\/span><\/pre>\n\n\n
Create:<\/p>\n\n\n
cat > create_metric_monitor.py <<'EOF'<\/span>\nimport<\/span> os\nimport<\/span> sys\nfrom<\/span> datadog_api_client import<\/span> ApiClient, Configuration\nfrom<\/span> datadog_api_client.v1.api.monitors_api import<\/span> MonitorsApi\nfrom<\/span> datadog_api_client.v1.model.monitor import<\/span> Monitor\nfrom<\/span> datadog_api_client.v1.model.monitor_options import<\/span> MonitorOptions\nfrom<\/span> datadog_api_client.v1.model.monitor_thresholds import<\/span> MonitorThresholds\nfrom<\/span> datadog_api_client.v1.model.monitor_type import<\/span> MonitorType\n\n\ndef require_env(name: str) -> str:\n    value = os.getenv(name)\n    if<\/span> not value:\n        print(f\"Missing required environment variable: {name}\"<\/span>, file=sys.stderr)\n        sys.exit(1<\/span>)\n    return<\/span> value\n\n\ndef main() -> None:\n    require_env(\"DD_API_KEY\"<\/span>)\n    require_env(\"DD_APP_KEY\"<\/span>)\n    require_env(\"DD_SITE\"<\/span>)\n\n    configuration = Configuration()\n\n    body = Monitor(\n        name=\"[LAB] Python API - CPU idle below 10% by host\"<\/span>,\n        type=MonitorType.QUERY_ALERT,\n        query=\"avg(last_5m):avg:system.cpu.idle{*} by {host} < 10\"<\/span>,\n        message=(\n            \"CPU idle is below 10% on {{host.name}}.\\n\\n\"<\/span>\n            \"Current value: {{value}}\\n\\n\"<\/span>\n            \"Check top processes and recent deployments.\\n\\n\"<\/span>\n            \"Notify: @slack-devops-alerts\"<\/span>\n        ),\n        tags=[\n            \"lab:datadog-api\"<\/span>,\n            \"created_by:python\"<\/span>,\n            \"team:devops\"<\/span>,\n            \"env:training\"<\/span>,\n        ],\n        priority=3<\/span>,\n        options=MonitorOptions(\n            thresholds=MonitorThresholds(\n                critical=10<\/span>,\n                warning=20<\/span>,\n            ),\n            notify_no_data=False,\n            include_tags=True,\n            new_group_delay=300<\/span>,\n            renotify_interval=60<\/span>,\n            require_full_window=False,\n        ),\n    )\n\n    with<\/span> ApiClient(configuration) as<\/span> api_client:\n        api = MonitorsApi(api_client)\n\n        print(\"Validating monitor...\"<\/span>)\n        api.validate_monitor(body=body)\n\n        print(\"Creating monitor...\"<\/span>)\n        response = api.create_monitor(body=body)\n\n        print(\"Created monitor:\"<\/span>)\n        print(response)\n\n\nif<\/span> __name__ == \"__main__\"<\/span>:\n    main()\nEOF\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Run:<\/p>\n\n\n
python3<\/span> create_metric_monitor<\/span>.py<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
20. Python script to create monitor from JSON file<\/h1>\n\n\n\n
This is more reusable for students.<\/p>\n\n\n
cat > create_monitor_from_json.py <<'EOF'<\/span>\nimport json\nimport os\nimport sys\nimport requests\n\n\ndef require_env(name: str) -> str:\n    value = os.getenv(name)\n    if<\/span> not value:\n        print<\/span>(f\"Missing required environment variable: {name}\"<\/span>, file=sys.stderr)\n        sys.exit<\/span>(1<\/span>)\n    return<\/span> value\n\n\ndef main() -> None:\n    if<\/span> len(sys.argv) != 2<\/span>:\n        print<\/span>(\"Usage: python3 create_monitor_from_json.py <monitor-json-file>\"<\/span>, file=sys.stderr)\n        sys.exit<\/span>(1<\/span>)\n\n    file_path = sys.argv[1<\/span>]\n\n    dd_site = require_env(\"DD_SITE\"<\/span>)\n    dd_api_key = require_env(\"DD_API_KEY\"<\/span>)\n    dd_app_key = require_env(\"DD_APP_KEY\"<\/span>)\n\n    with open(file_path, \"r\"<\/span>, encoding=\"utf-8\"<\/span>) as<\/span> f:\n        payload = json.load(f)\n\n    headers = {\n        \"Accept\"<\/span>: \"application\/json\"<\/span>,\n        \"Content-Type\"<\/span>: \"application\/json\"<\/span>,\n        \"DD-API-KEY\"<\/span>: dd_api_key,\n        \"DD-APPLICATION-KEY\"<\/span>: dd_app_key,\n    }\n\n    validate_url = f\"https:\/\/api.{dd_site}\/api\/v1\/monitor\/validate\"<\/span>\n    create_url = f\"https:\/\/api.{dd_site}\/api\/v1\/monitor\"<\/span>\n\n    print<\/span>(f\"Validating {file_path} ...\"<\/span>)\n    validate_response = requests.post(validate_url, headers=headers, json=payload, timeout=30<\/span>)\n\n    if<\/span> validate_response.status_code >= 400<\/span>:\n        print<\/span>(\"Validation failed:\"<\/span>)\n        print<\/span>(validate_response.status_code)\n        print<\/span>(validate_response.text)\n        sys.exit<\/span>(1<\/span>)\n\n    print<\/span>(\"Validation OK. Creating monitor ...\"<\/span>)\n    create_response = requests.post(create_url, headers=headers, json=payload, timeout=30<\/span>)\n\n    if<\/span> create_response.status_code >= 400<\/span>:\n        print<\/span>(\"Create failed:\"<\/span>)\n        print<\/span>(create_response.status_code)\n        print<\/span>(create_response.text)\n        sys.exit<\/span>(1<\/span>)\n\n    print<\/span>(json.dumps(create_response.json(), indent=2<\/span>))\n\n\nif<\/span> __name__ == \"__main__\"<\/span>:\n    main()\nEOF\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Install dependency:<\/p>\n\n\n
pip install requests\n<\/code><\/span><\/pre>\n\n\n
Run:<\/p>\n\n\n
python3 create_monitor_from_json.py monitors\/apache-5xx-log-monitor.json\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
21. Monitor type cheat sheet<\/h1>\n\n\n\n
Common monitor API type<\/code> values:<\/p>\n\n\n\n
Monitor<\/th>API type<\/th><\/tr><\/thead>
Metric threshold monitor<\/td>query alert<\/code><\/td><\/tr>
Log monitor<\/td>log alert<\/code><\/td><\/tr>
Service check monitor<\/td>service check<\/code><\/td><\/tr>
Event monitor<\/td>event alert<\/code><\/td><\/tr>
Process monitor<\/td>process alert<\/code><\/td><\/tr>
Composite monitor<\/td>composite<\/code><\/td><\/tr>
Synthetics alert monitor<\/td>Usually created automatically from Synthetic tests<\/td><\/tr>
Cost monitor<\/td>cost alert<\/code><\/td><\/tr>
RUM monitor<\/td>rum alert<\/code><\/td><\/tr>
CI pipelines monitor<\/td>ci-pipelines alert<\/code><\/td><\/tr>
Error Tracking monitor<\/td>Depends on Error Tracking monitor type\/query model<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
For most infrastructure labs, you will use:<\/p>\n\n\n
query alert\nlog alert\nservice check\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
22. Query examples for common monitor types<\/h1>\n\n\n\n
CPU high<\/h2>\n\n\n
avg(last_5m):avg:system.cpu.idle{*} by {host} < 10\n<\/code><\/span><\/pre>\n\n\n
Memory high<\/h2>\n\n\n
avg(last_5m):avg:system.mem.pct_usable{*} by {host} < 0.10\n<\/code><\/span><\/pre>\n\n\n
Disk high<\/h2>\n\n\n
avg(last_5m):avg:system.disk.in_use{device:\/} by {host} > 0.85\n<\/code><\/span><\/pre>\n\n\n
Apache 5xx logs<\/h2>\n\n\n
logs(\"source:apache @http.status_code:[500 TO 599]\"<\/span>).index(\"*\"<\/span>).rollup(\"count\"<\/span>).by(\"host\"<\/span>).last(\"5m\"<\/span>) > 5<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Ubuntu failed SSH<\/h2>\n\n\n
logs(\"\\\"Failed password\\\" OR \\\"Invalid user\\\"\"<\/span>).index(\"*\"<\/span>).rollup(\"count\"<\/span>).by(\"host\"<\/span>).last(\"5m\"<\/span>) > 10<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Kubernetes restarts<\/h2>\n\n\n
sum(last_10m):sum:kubernetes.containers.restarts{*} by {kube_namespace,kube_deployment} > 3\n<\/code><\/span><\/pre>\n\n\n
APM error rate<\/h2>\n\n\n
sum(last_5m):sum:trace.web.request.errors{service:checkout-api,env:prod}.as_count() \/ sum:trace.web.request.hits{service:checkout-api,env:prod}.as_count() * 100 > 5\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
23. Important monitor options<\/h1>\n\n\n\n
Datadog\u2019s monitor API options include useful controls like include_tags<\/code>, new_group_delay<\/code>, no_data_timeframe<\/code>, notify_no_data<\/code>, notify_by<\/code>, and on_missing_data<\/code>. For example, new_group_delay<\/code> skips evaluations for new groups while they initialize, notify_no_data<\/code> controls notifications when data stops reporting, and notify_by<\/code> changes alert granularity for grouped monitors. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
Common options:<\/p>\n\n\n
{\n  \"thresholds\"<\/span>: {\n    \"critical\"<\/span>: 80<\/span>,\n    \"warning\"<\/span>: 70<\/span>\n  },\n  \"notify_no_data\"<\/span>: false<\/span>,\n  \"include_tags\"<\/span>: true<\/span>,\n  \"new_group_delay\"<\/span>: 300<\/span>,\n  \"renotify_interval\"<\/span>: 60<\/span>,\n  \"require_full_window\"<\/span>: false<\/span>\n}\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
My practical recommendation<\/h2>\n\n\n\n
For lab and beginner monitors:<\/p>\n\n\n
\"require_full_window\"<\/span>: false<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Why? It makes monitors evaluate more quickly and avoids confusion during short lab sessions.<\/p>\n\n\n\n
For production monitors, tune this based on the metric and business need.<\/p>\n\n\n\n
\n\n\n\n
24. Notification examples<\/h1>\n\n\n\n
Datadog notifications are usually written in the message<\/code> field.<\/p>\n\n\n\n
Examples:<\/p>\n\n\n
Notify<\/span>: @slack-devops-alerts<\/span>\nNotify:<\/span> @pagerduty-platform\nNotify:<\/span> @webhook-my-webhook\nNotify:<\/span> @team-platform\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Example message:<\/p>\n\n\n
\"message\"<\/span>: \"High CPU detected on {{host.name}}.\\n\\nCurrent value: {{value}}\\n\\nNotify: @slack-devops-alerts\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Useful template variables:<\/p>\n\n\n
{{value}}\n{{host.name}}\n{{service.name}}\n{{env.name}}\n{{kube_namespace.name}}\n{{kube_deployment.name}}\n<\/code><\/span><\/pre>\n\n\n
The exact variable availability depends on your monitor grouping.<\/p>\n\n\n\n
\n\n\n\n
25. Best practice: naming convention<\/h1>\n\n\n\n
Use a standard format:<\/p>\n\n\n
[ENV]<\/span> [SERVICE]<\/span> Symptom<\/span> -<\/span> Scope<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Examples:<\/p>\n\n\n
[PROD]<\/span> checkout-api<\/span> Error<\/span> Rate<\/span> High<\/span>\n[PROD]<\/span> Apache<\/span> 5xx<\/span> Errors<\/span> High<\/span> by<\/span> Host<\/span>\n[STAGE]<\/span> Kubernetes<\/span> Container<\/span> Restarts<\/span> High<\/span>\n[TRAINING]<\/span> Ubuntu<\/span> Failed<\/span> SSH<\/span> Login<\/span> Attempts<\/span> High<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
For labs:<\/p>\n\n\n
[LAB]<\/span> Apache<\/span> 5xx<\/span> Errors<\/span> High<\/span> by<\/span> Host<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
26. Best practice: tagging monitors<\/h1>\n\n\n\n
Always tag monitors.<\/p>\n\n\n\n
Recommended tags:<\/p>\n\n\n
env<\/span>:prod<\/span>\nteam<\/span>:devops<\/span>\nservice<\/span>:apache<\/span>\nmonitor_type<\/span>:log<\/span>\nmanaged_by<\/span>:api<\/span>\nseverity<\/span>:p2<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Example:<\/p>\n\n\n
\"tags\"<\/span>: [\n  \"env:prod\"<\/span>,\n  \"team:devops\"<\/span>,\n  \"service:apache\"<\/span>,\n  \"monitor_type:log\"<\/span>,\n  \"managed_by:api\"<\/span>\n]\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
This helps with monitor search, ownership, dashboards, reporting, and cleanup.<\/p>\n\n\n\n
\n\n\n\n
27. Student lab flow<\/h1>\n\n\n\n
Use this sequence for students:<\/p>\n\n\n
1.<\/span> Export DD_SITE, DD_API_KEY, DD_APP_KEY.\n2.<\/span> Validate API key.\n3.<\/span> Create JSON<\/span> monitor definition.\n4.<\/span> Validate monitor JSON<\/span> using \/api\/v1\/monitor\/validate.\n5.<\/span> Create monitor using \/api\/v1\/monitor.\n6.<\/span> Capture monitor ID.\n7.<\/span> View monitor in<\/span> Datadog UI.\n8.<\/span> Update monitor threshold.\n9.<\/span> Search monitor using API.\n10.<\/span> Delete lab monitor.\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
28. Full lab command sequence<\/h1>\n\n\n
export DD_SITE=\"datadoghq.com\"<\/span>\nexport DD_API_KEY=\"replace_me\"<\/span>\nexport DD_APP_KEY=\"replace_me\"<\/span>\n\ncurl -sS -X GET \"https:\/\/api.${DD_SITE}\/api\/v1\/validate\"<\/span> \\\n  -H \"Accept: application\/json\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> | jq\n\n.\/validate-monitor.sh monitors\/cpu-idle-low.json\n\n.\/create-monitor.sh monitors\/cpu-idle-low.json\n\nexport MONITOR_ID=\"replace_with_created_monitor_id\"<\/span>\n\n.\/get-monitor.sh \"${MONITOR_ID}\"<\/span>\n\ncurl -sS -X GET \"https:\/\/api.${DD_SITE}\/api\/v1\/monitor\/search?query=lab:datadog-api\"<\/span> \\\n  -H \"Accept: application\/json\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> \\\n  -H \"DD-APPLICATION-KEY: ${DD_APP_KEY}\"<\/span> | jq\n\n.\/delete-monitor.sh \"${MONITOR_ID}\"<\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
29. Troubleshooting<\/h1>\n\n\n\n
Problem: 403 Forbidden<\/code><\/h2>\n\n\n\n
Possible causes:<\/p>\n\n\n
Wrong API key\nWrong APP key\nWrong DD_SITE\nApplication key does not have permission\nAccount\/org mismatch\n<\/code><\/span><\/pre>\n\n\n
Check:<\/p>\n\n\n
echo<\/span> \"$DD_SITE\"<\/span>\ncurl -sS -X GET \"https:\/\/api.${DD_SITE}\/api\/v1\/validate\"<\/span> \\\n  -H \"DD-API-KEY: ${DD_API_KEY}\"<\/span> | jq\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Problem: monitor validation fails<\/h2>\n\n\n\n
Common causes:<\/p>\n\n\n
Wrong monitor type\nWrong query syntax\nThreshold mismatch\nMetric does not exist\nInvalid log query escaping\nInvalid JSON<\/span>\nUnsupported option for<\/span> that monitor type\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Validate JSON locally:<\/p>\n\n\n
jq . monitors\/apache-5xx-log-monitor.json\n<\/code><\/span><\/pre>\n\n\n
Problem: log monitor returns no data<\/h2>\n\n\n\n
Check:<\/p>\n\n\n
Are<\/span> logs<\/span> indexed<\/span>?\nIs<\/span> the<\/span> source<\/span> correct<\/span>?\nIs<\/span> @http<\/span>.status_code parsed?\nIs time range correct?\nIs the index correct?\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Datadog log monitors evaluate indexed logs only. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
Problem: APM monitor does not work<\/h2>\n\n\n\n
Check actual metric names in Metrics Explorer. Your application may use:<\/p>\n\n\n
trace<\/span>.web<\/span>.request<\/span>.*\ntrace<\/span>.servlet<\/span>.request<\/span>.*\ntrace<\/span>.express<\/span>.request<\/span>.*\ntrace<\/span>.fastapi<\/span>.request<\/span>.*\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
The service and env tags must match real APM data.<\/p>\n\n\n\n
\n\n\n\n
30. Final reusable template<\/h1>\n\n\n\n
Use this as a base for any monitor:<\/p>\n\n\n
{\n  \"name\"<\/span>: \"[ENV] Service Symptom - Scope\"<\/span>,\n  \"type\"<\/span>: \"query alert\"<\/span>,\n  \"query\"<\/span>: \"avg(last_5m):avg:some.metric{env:prod,service:my-service} by {host} > 80\"<\/span>,\n  \"message\"<\/span>: \"Alert message here.\\n\\nCurrent value: {{value}}\\n\\nNotify: @slack-devops-alerts\"<\/span>,\n  \"tags\"<\/span>: [\n    \"env:prod\"<\/span>,\n    \"service:my-service\"<\/span>,\n    \"team:devops\"<\/span>,\n    \"managed_by:api\"<\/span>\n  ],\n  \"priority\"<\/span>: 3<\/span>,\n  \"options\"<\/span>: {\n    \"thresholds\"<\/span>: {\n      \"critical\"<\/span>: 80<\/span>,\n      \"warning\"<\/span>: 70<\/span>\n    },\n    \"notify_no_data\"<\/span>: false<\/span>,\n    \"include_tags\"<\/span>: true<\/span>,\n    \"new_group_delay\"<\/span>: 300<\/span>,\n    \"renotify_interval\"<\/span>: 60<\/span>,\n    \"require_full_window\"<\/span>: false<\/span>\n  }\n}\n<\/code><\/span>Code language:<\/span> JSON \/ JSON with Comments<\/span> (<\/span>json<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
31. Final recommendation<\/h1>\n\n\n\n
For production, do not create monitors manually one by one forever. Use the API workflow like this:<\/p>\n\n\n
Monitor JSON<\/span> in<\/span> Git\n        \u2193\nPull request review\n        \u2193\nCI validates monitor via \/api\/v1\/monitor\/validate\n        \u2193\nCI creates\/updates monitor through API\n        \u2193\nMonitor tagged with<\/span> managed_by:api\n        \u2193\nDashboard\/SLO\/incident workflow uses same tags\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
For students, the best learning sequence is:<\/p>\n\n\n
Create metric monitor\nCreate log monitor\nCreate APM monitor\nValidate JSON<\/span>\nUpdate threshold\nSearch monitor\nDelete monitor\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
That gives them the full Datadog alerting lifecycle using the API.<\/p>\n\n\n\n
\n\n\n\n
Old Content<\/h3>\n\n\n\n
\n\n\n
curl -X POST -H \"Content-type: application\/json\"<\/span> -H \"DD-API-KEY: XXXXXXXXXXXXXXXXX\"<\/span> -H \"DD-APPLICATION-KEY: ddapp_XXXXXXXXXXX\"<\/span> -d @devops.json \"https:\/\/api.datadoghq.com\/api\/v1\/monitor\"<\/span><\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Datadog\u2019s current Python examples use datadog_api_client<\/code>, Configuration<\/code>, ApiClient<\/code>, and MonitorsApi<\/code> for monitor operations. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
Datadog lists DELETE \/api\/v1\/monitor\/{monitor_id}<\/code> for deleting a monitor. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
Datadog lists GET \/api\/v1\/monitor<\/code> for getting all monitors. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
Datadog lists GET \/api\/v1\/monitor\/search<\/code> for monitor search. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
Datadog\u2019s authentication API includes GET \/api\/v1\/validate<\/code> to check whether an API key is valid; invalid keys return 403<\/code>. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
Datadog\u2019s monitor API examples use DD-API-KEY<\/code> and DD-APPLICATION-KEY<\/code> headers when creating monitors. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n
For metric monitors, Datadog says any metric reporting to Datadog can be alerted on, and threshold alerts compare metric values to a static threshold over a selected time period. (Datadog Monitoring<\/a>)<\/p>\n\n\n\n