{"id":26986,"date":"2022-02-17T07:00:30","date_gmt":"2022-02-17T07:00:30","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=26986"},"modified":"2022-03-04T12:51:38","modified_gmt":"2022-03-04T12:51:38","slug":"datadog-data-collection-resolution-and-retention-interval","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/datadog-data-collection-resolution-and-retention-interval\/","title":{"rendered":"Datadog Data Collection, Resolution, and Retention Interval"},"content":{"rendered":"\n

Collection interval for Datadog Agent<\/h1>\n\n\n\n

To change the collection interval of your check, use min_collection_interval in the configuration file. The default value is 15 which means the check method from your class is invoked with the same interval as the rest of the integrations on the Agent.<\/p>\n\n\n\n

The min_collection_interval parameter is available for both standard and custom integrations<\/p>\n\n\n\n

\n\n\n
init_config:\n\ninstances:\n  - min_collection_interval: 30<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
If the min_collection_interval is set to 30, it does not mean that the metric is collected every 30 seconds, but rather that it could be collected as often as every 30 seconds.<\/p>\n\n\n\n
The collector tries to run the check every 30 seconds but the check might need to wait in line, depending on how many integrations are enabled on the same Agent. Also if the check method takes more than 30 seconds to finish, the Agent skips execution until the next interval.<\/p>\n\n\n\n
Collection interval for Datadog standard integrations<\/h1>\n\n\n\n
The default collection interval for all Datadog standard integrations is 15 seconds. To change the collection interval, use the parameter min_collection_interval.<\/p>\n\n\n\n
DogStatsD Flush Interval<\/h1>\n\n\n\n
DogStatsD uses a flush interval of 10 seconds. Every 10 seconds, DogStatsD checks all data received since the last flush. All values that correspond to the same metric name and the same tags are aggregated together into a single value.<\/p>\n\n\n\n
About DogStatsD
In v6, DogStatsD is a Golang implementation of Etsy\u2019s StatsD metric aggregation daemon. It is used to receive and roll up arbitrary metrics over UDP or Unix socket, thus allowing custom code to be instrumented without adding latency.<\/p>\n\n\n\n
Datadog Forwarder Flush Interval<\/h1>\n\n\n\n
The Agent forwarder send metrics over HTTPS to Datadog. Buffering prevents network splits from affecting metric reporting. Metrics are buffered in memory until a limit in size or number of outstanding send requests are reached. Afterwards, the oldest metrics are discarded to keep the forwarder\u2019s memory footprint manageable. Logs are sent over an SSL-encrypted TCP connection to Datadog.<\/p>\n\n\n\n
Datadog Data Collection, Resolution, and Retention Interval.<\/h2>\n\n\n\n
Find below a summary of Datadog data collection, resolution, and retention:<\/p>\n\n\n\n
PRODUCT CATEGORY<\/th>SOURCE<\/th>COLLECTION METHODS<\/th>COLLECTION INTERVAL<\/th>MINIMUM RESOLUTION<\/th>DEFAULT RETENTION<\/th><\/tr><\/thead>
Alerting<\/td>Service Level Objectives<\/td>Datadog monitors, Datadog Synthetic monitoring, or metrics (infrastructure, APM trace, custom)<\/td>Data source-dependent<\/td>Data source-dependent<\/td>7, 30, or 90 days (chosen by user in SLO configuration)<\/td><\/tr>
APM<\/td>Error Tracking<\/td>APM spans (unsampled)<\/td>2 minutes<\/td>60 seconds<\/td>15 days<\/td><\/tr>
APM<\/td>Indexed spans (unsampled)<\/td>Datadog Agent + tracing library<\/td>10 seconds<\/td>1 millisecond<\/td>Plan<\/td><\/tr>
APM<\/td>Profiles<\/td>Datadog Agent + tracing library<\/td>60 seconds<\/td>60 seconds<\/td>7 days<\/td><\/tr>
APM<\/td>Profile metrics<\/td>Datadog Agent + tracing library<\/td>60 seconds<\/td>60 seconds<\/td>1 month<\/td><\/tr>
APM<\/td>Services\/resources statistics and span summaries<\/td>Datadog Agent + tracing library<\/td>10 seconds<\/td>10 seconds<\/td>30 days<\/td><\/tr>
APM<\/td>Traces (sampled)<\/td>Datadog Agent + tracing library<\/td>10 seconds<\/td>1 millisecond<\/td>15 days. Viewed traces are retained long-term. See Trace Storage<\/a> for details.<\/td><\/tr>
APM<\/td>Trace metrics (unsampled)<\/td>Datadog Agent + tracing library<\/td>10 seconds<\/td>1 second<\/td>15 months<\/td><\/tr>
Cloud<\/td>Alibaba<\/td>API crawler<\/td>10 min (default<\/a>)<\/td>1 min<\/td>15 months<\/td><\/tr>
Cloud<\/td>AWS<\/td>API crawler<\/td>10 min (default<\/a>)<\/td>1 min<\/td>15 months<\/td><\/tr>
Cloud<\/td>Azure<\/td>API crawler<\/td>2 min (default<\/a>)<\/td>1 min<\/td>15 months<\/td><\/tr>
Cloud<\/td>Google Cloud<\/td>API crawler<\/td>5 min (default<\/a>)<\/td>1 min<\/td>15 months<\/td><\/tr>
CSPM<\/td>Findings<\/td>Datadog Cloud Security Posture Management<\/td>15 minutes to 4 hours depending on resource type<\/td>1 minute<\/td>15 months<\/td><\/tr>
Database Monitoring<\/td>Query Metrics<\/td>Datadog Agent + enabled integrations<\/td>10 seconds<\/td>1 second<\/td>3 months<\/td><\/tr>
Database Monitoring<\/td>Query Samples<\/td>Datadog Agent + enabled integrations<\/td>1 minute<\/td>n\/a<\/td>15 days<\/td><\/tr>
Incident Management<\/td>Incident Management<\/td>Incident data<\/td>n\/a<\/td>n\/a<\/td>15 months<\/td><\/tr>
Infrastructure<\/td>Agent integrations<\/td>Datadog Agent + enabled integrations<\/td>15 seconds<\/td>1 second<\/td>15 months<\/td><\/tr>
Infrastructure<\/td>Custom metrics (Agent check)<\/td>Datadog Agent + custom Agent check<\/td>15 seconds<\/td>1 second<\/td>15 months<\/td><\/tr>
Infrastructure<\/td>Custom metrics (API)<\/td>POST directly to Datadog\u2019s API<\/td>Real time<\/td>1 second<\/td>15 months<\/td><\/tr>
Infrastructure<\/td>Custom metrics (StatsD)<\/td>Datadog Agent (built-in statsD collector)<\/td>15 seconds<\/td>1 second<\/td>15 months<\/td><\/tr>
Infrastructure<\/td>Events<\/td>Datadog Agent, integrations, or API<\/td>Real time<\/td>1 second<\/td>13 months<\/td><\/tr>
Infrastructure<\/td>Live containers<\/td>Datadog Agent + enabled Docker integration or Datadog container Agent<\/td>2 seconds<\/td>1 second<\/td>36 hours<\/td><\/tr>
Infrastructure<\/td>Live processes<\/td>Datadog Agent + Process Agent<\/td>2 seconds<\/td>1 second<\/td>36 hours<\/td><\/tr>
Infrastructure<\/td>Network Performance Monitoring<\/td>System Probe<\/td>5 minutes (default<\/a>)<\/td>1 min<\/td>7 days<\/td><\/tr>
Infrastructure<\/td>System metrics<\/td>Datadog Agent<\/td>15 seconds<\/td>1 second<\/td>15 months<\/td><\/tr>
Logs<\/td>Logs<\/td>Datadog Agent + Logs, 3rd party log collectors, or API<\/td>Real time<\/td>1 millisecond<\/td>Plan<\/td><\/tr>
Real User Monitoring<\/td>Error Tracking<\/td>RUM SDK<\/td>2 minutes<\/td>60 seconds<\/td>15 days<\/td><\/tr>
Real User Monitoring<\/td>Real User Monitoring<\/td>RUM SDK<\/td>Real time<\/td>1 millisecond<\/td>30 days for session, view, action and error events
15 days for resource and long task events<\/td><\/tr>
Real User Monitoring<\/td>Session Replays<\/td>RUM SDK<\/td>Real time<\/td>1 millisecond<\/td>30 days<\/td><\/tr>
Cloud SIEM<\/td>Security Signals<\/td>Datadog Cloud SIEM<\/td>Real time<\/td>1 millisecond<\/td>15 months<\/td><\/tr>
Synthetic Monitoring<\/td>API Test metrics<\/td>Datadog Synthetic Monitoring application<\/td>User-defined<\/td>1 min<\/td>15 months<\/td><\/tr>
Synthetic Monitoring<\/td>API Test results<\/td>Datadog Synthetic Monitoring application<\/td>User-defined<\/td>1 min<\/td>Result seen in the UI by user: 15 months
Result not seen in the UI by user: 2 months<\/td><\/tr>
Synthetic Monitoring<\/td>Browser Test metrics<\/td>Datadog Synthetic Monitoring application<\/td>User-defined<\/td>5 min<\/td>15 months<\/td><\/tr>
Synthetic Monitoring<\/td>Browser Test results<\/td>Datadog Synthetic Monitoring application<\/td>User-defined<\/td>5 min<\/td>Result seen in the UI by user: 15 months
Result not seen in the UI by user: 2 months<\/td><\/tr>
Synthetic Monitoring<\/td>Batches<\/td>Datadog Synthetic Monitoring application (through calls to the Synthetics trigger API endpoint<\/a> or to the Synthetics CI CLI<\/a>)<\/td>Depending on calls to the Synthetics trigger API endpoint<\/a> or to the Synthetic CI CLI<\/a><\/td>n\/a<\/td>15 days<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Reference<\/h2>\n\n\n\n
https:\/\/docs.datadoghq.com\/developers\/guide\/data-collection-resolution-retention\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Collection interval for Datadog Agent To change the collection interval of your check, use min_collection_interval in the configuration file. The default value is 15 which means the check method from… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[2],"tags":[],"class_list":["post-26986","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/26986","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=26986"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/26986\/revisions"}],"predecessor-version":[{"id":26988,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/26986\/revisions\/26988"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=26986"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=26986"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=26986"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}