Uncategorised

Datadog Data Collection, Resolution, and Retention Interval

February 17, 2022 Leave a Comment

Collection interval for Datadog Agent

To change the collection interval of your check, use min_collection_interval in the configuration file. The default value is 15 which means the check method from your class is invoked with the same interval as the rest of the integrations on the Agent.

The min_collection_interval parameter is available for both standard and custom integrations

init_config:

instances:
  - min_collection_interval: 30

If the min_collection_interval is set to 30, it does not mean that the metric is collected every 30 seconds, but rather that it could be collected as often as every 30 seconds.

The collector tries to run the check every 30 seconds but the check might need to wait in line, depending on how many integrations are enabled on the same Agent. Also if the check method takes more than 30 seconds to finish, the Agent skips execution until the next interval.

Collection interval for Datadog standard integrations

The default collection interval for all Datadog standard integrations is 15 seconds. To change the collection interval, use the parameter min_collection_interval.

DogStatsD Flush Interval

DogStatsD uses a flush interval of 10 seconds. Every 10 seconds, DogStatsD checks all data received since the last flush. All values that correspond to the same metric name and the same tags are aggregated together into a single value.

About DogStatsD
In v6, DogStatsD is a Golang implementation of Etsy’s StatsD metric aggregation daemon. It is used to receive and roll up arbitrary metrics over UDP or Unix socket, thus allowing custom code to be instrumented without adding latency.

Datadog Forwarder Flush Interval

The Agent forwarder send metrics over HTTPS to Datadog. Buffering prevents network splits from affecting metric reporting. Metrics are buffered in memory until a limit in size or number of outstanding send requests are reached. Afterwards, the oldest metrics are discarded to keep the forwarder’s memory footprint manageable. Logs are sent over an SSL-encrypted TCP connection to Datadog.

Datadog Data Collection, Resolution, and Retention Interval.

Find below a summary of Datadog data collection, resolution, and retention:

PRODUCT CATEGORYSOURCECOLLECTION METHODSCOLLECTION INTERVALMINIMUM RESOLUTIONDEFAULT RETENTION
AlertingService Level ObjectivesDatadog monitors, Datadog Synthetic monitoring, or metrics (infrastructure, APM trace, custom)Data source-dependentData source-dependent7, 30, or 90 days (chosen by user in SLO configuration)
APMError TrackingAPM spans (unsampled)2 minutes60 seconds15 days
APMIndexed spans (unsampled)Datadog Agent + tracing library10 seconds1 millisecondPlan
APMProfilesDatadog Agent + tracing library60 seconds60 seconds7 days
APMProfile metricsDatadog Agent + tracing library60 seconds60 seconds1 month
APMServices/resources statistics and span summariesDatadog Agent + tracing library10 seconds10 seconds30 days
APMTraces (sampled)Datadog Agent + tracing library10 seconds1 millisecond15 days. Viewed traces are retained long-term. See Trace Storage for details.
APMTrace metrics (unsampled)Datadog Agent + tracing library10 seconds1 second15 months
CloudAlibabaAPI crawler10 min (default)1 min15 months
CloudAWSAPI crawler10 min (default)1 min15 months
CloudAzureAPI crawler2 min (default)1 min15 months
CloudGoogle CloudAPI crawler5 min (default)1 min15 months
CSPMFindingsDatadog Cloud Security Posture Management15 minutes to 4 hours depending on resource type1 minute15 months
Database MonitoringQuery MetricsDatadog Agent + enabled integrations10 seconds1 second3 months
Database MonitoringQuery SamplesDatadog Agent + enabled integrations1 minuten/a15 days
Incident ManagementIncident ManagementIncident datan/an/a15 months
InfrastructureAgent integrationsDatadog Agent + enabled integrations15 seconds1 second15 months
InfrastructureCustom metrics (Agent check)Datadog Agent + custom Agent check15 seconds1 second15 months
InfrastructureCustom metrics (API)POST directly to Datadog’s APIReal time1 second15 months
InfrastructureCustom metrics (StatsD)Datadog Agent (built-in statsD collector)15 seconds1 second15 months
InfrastructureEventsDatadog Agent, integrations, or APIReal time1 second13 months
InfrastructureLive containersDatadog Agent + enabled Docker integration or Datadog container Agent2 seconds1 second36 hours
InfrastructureLive processesDatadog Agent + Process Agent2 seconds1 second36 hours
InfrastructureNetwork Performance MonitoringSystem Probe5 minutes (default)1 min7 days
InfrastructureSystem metricsDatadog Agent15 seconds1 second15 months
LogsLogsDatadog Agent + Logs, 3rd party log collectors, or APIReal time1 millisecondPlan
Real User MonitoringError TrackingRUM SDK2 minutes60 seconds15 days
Real User MonitoringReal User MonitoringRUM SDKReal time1 millisecond30 days for session, view, action and error events
15 days for resource and long task events
Real User MonitoringSession ReplaysRUM SDKReal time1 millisecond30 days
Cloud SIEMSecurity SignalsDatadog Cloud SIEMReal time1 millisecond15 months
Synthetic MonitoringAPI Test metricsDatadog Synthetic Monitoring applicationUser-defined1 min15 months
Synthetic MonitoringAPI Test resultsDatadog Synthetic Monitoring applicationUser-defined1 minResult seen in the UI by user: 15 months
Result not seen in the UI by user: 2 months
Synthetic MonitoringBrowser Test metricsDatadog Synthetic Monitoring applicationUser-defined5 min15 months
Synthetic MonitoringBrowser Test resultsDatadog Synthetic Monitoring applicationUser-defined5 minResult seen in the UI by user: 15 months
Result not seen in the UI by user: 2 months
Synthetic MonitoringBatchesDatadog Synthetic Monitoring application (through calls to the Synthetics trigger API endpoint or to the Synthetics CI CLI)Depending on calls to the Synthetics trigger API endpoint or to the Synthetic CI CLIn/a15 days

Reference

https://docs.datadoghq.com/developers/guide/data-collection-resolution-retention/

Rajesh Kumar
Follow me
Latest posts by Rajesh Kumar (see all)
Subscribe
Notify of
guest
0 Comments
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x