1. Simple Meaning
In Datadog, Synthetic Tests are the actual checks that run against your application.
A Synthetic Test Suite is a logical group of multiple Synthetic Tests.
Think of it like this:
| Concept | Simple Meaning |
|---|---|
| Synthetic Test | One actual test/check |
| Synthetic Test Suite | A folder/group containing many tests |
| SLO | Reliability target based on uptime/availability |
| Dashboard SLO Widget | Visual display of an existing SLO |
Example:
You may have these individual Synthetic Tests:
| Synthetic Test | Purpose |
|---|---|
| Login API uptime test | Checks login endpoint |
| Homepage browser test | Checks website loads |
| Checkout browser test | Checks checkout flow |
| Payment API test | Checks payment endpoint |
| gRPC health check | Checks gRPC service health |
Then you can create a Test Suite called:
Production Critical User Journey
And add all those tests inside it.
The suite does not replace the tests. It only groups them and gives you a combined operational view.
2. What is a Datadog Synthetic Test?
A Synthetic Test is an automated test run by Datadog to check whether your application, endpoint, website, API, or user journey is working.
It can run from:
| Location Type | Meaning |
|---|---|
| Datadog managed location | Public Datadog location around the world |
| Private location | Your internal network, VPC, Kubernetes, or private environment |
Synthetic Tests are useful because they test the system from the outside, like a real user or client.
Example:
Every 5 minutes:
Datadog calls https://api.example.com/health
Expected status: 200
Expected response time: less than 1 second
If it fails for 2 runs, alert the team.
That is a Synthetic Test.
3. Types of Synthetic Tests
Datadog supports multiple kinds of Synthetic Tests.
3.1 API Tests
API tests check backend endpoints and network services.
Common API test types:
| API Test Type | Purpose |
|---|---|
| HTTP | Check REST/API endpoint |
| SSL | Check certificate validity |
| DNS | Check DNS resolution |
| TCP | Check TCP port connectivity |
| UDP | Check UDP service |
| ICMP | Ping-style network check |
| WebSocket | Check WebSocket service |
| gRPC | Check gRPC endpoint |
| Multistep API | Chain multiple API calls together |
Example use cases:
| Use Case | Test Type |
|---|---|
Check /health endpoint | HTTP API test |
| Check certificate expiry | SSL test |
| Check DNS record | DNS test |
| Check gRPC health endpoint | gRPC test |
| Login → get token → call protected API | Multistep API test |
3.2 Browser Tests
Browser tests simulate real user actions in a browser.
Example:
Open homepage
Click Login
Enter username and password
Click Submit
Verify dashboard is visible
Browser tests are useful for validating frontend flows.
Common use cases:
| Use Case | Browser Test? |
|---|---|
| Login flow | Yes |
| Checkout flow | Yes |
| Search flow | Yes |
| Form submission | Yes |
| Page rendering validation | Yes |
3.3 Mobile Application Tests
Mobile app tests validate mobile application flows on real devices.
Example:
Open mobile app
Tap login
Enter credentials
Verify home screen appears
Use this when your customer journey is inside an Android or iOS app.
4. What is a Synthetic Test Suite?
A Synthetic Test Suite is a collection of existing Synthetic Tests.
It helps you organize tests by:
| Grouping Style | Example |
|---|---|
| User journey | Login suite, checkout suite, booking suite |
| Environment | Production suite, staging suite |
| Service | Payment service suite, auth service suite |
| Team | Platform team suite, frontend team suite |
| Criticality | Tier-1 critical tests, non-critical tests |
| Location | Japan user journey, US user journey |
| Application | Web app suite, mobile app suite |
Example:
Suite Name: Production Checkout Journey
Tests inside:
1. Homepage browser test
2. Login browser test
3. Product search browser test
4. Add to cart browser test
5. Payment API test
6. Order confirmation API test
The Test Suite helps you see whether the overall checkout journey is healthy.
5. Main Difference Between Synthetic Test and Synthetic Test Suite
| Area | Synthetic Test | Synthetic Test Suite |
|---|---|---|
| Main purpose | Runs an actual check | Groups multiple tests |
| Does it execute a request/browser/mobile flow? | Yes | No, it triggers or organizes existing tests |
| Unit of monitoring | One endpoint, one browser flow, one mobile flow, one API sequence | A collection of tests |
| Alerting | Alert on individual test failure | Alert on suite-level condition |
| Dashboard use | Can be shown directly or used for SLO creation | Used for grouped visibility and suite health |
| SLO creation | Usually manual Monitor-based SLO | Current docs do not clearly show automatic SLO creation |
| Ownership | Usually owned by service/app team | Usually owned by team, product, journey, or environment |
| Best for | Precise technical monitoring | Business journey or operational grouping |
| Example | GET /health should return 200 | Production Checkout Critical Tests |
6. Features Available in Synthetic Tests
These are mainly features of individual Synthetic Tests.
| Feature | Available in Synthetic Test? | Notes |
|---|---|---|
| API endpoint check | Yes | HTTP, DNS, SSL, TCP, UDP, ICMP, WebSocket, gRPC |
| Browser journey recording | Yes | Simulates real user flow |
| Mobile app journey testing | Yes | Tests Android/iOS app flows |
| Assertions | Yes | Validate status code, text, response time, UI element, etc. |
| Request headers/body | Yes | Useful for API tests |
| Authentication handling | Yes | API tokens, cookies, login steps, variables |
| Multistep API chaining | Yes | Pass data from one request to another |
| Test frequency | Yes | Example: every 1 min, 5 min, 15 min |
| Managed locations | Yes | Run from Datadog locations |
| Private locations | Yes | Run from inside private network |
| Alert conditions | Yes | Configure when test should alert |
| Retry behavior | Yes | Reduce false positives |
| Per-test uptime | Yes | Available from test results |
| Monitor-based SLO source | Yes | Common SLO source for uptime |
| CI/CD execution | Yes | Can run tests in pipeline |
| Terraform/API management | Yes | Can manage tests as code |
7. Features Available in Synthetic Test Suites
These are mainly features of Test Suites.
| Feature | Available in Test Suite? | Notes |
|---|---|---|
| Group multiple tests | Yes | Main purpose of Test Suite |
| Add existing tests | Yes | You add already-created Synthetic Tests |
| Group by journey/environment/team | Yes | Very useful for organization |
| Suite-level view | Yes | See related tests in one place |
| Suite-level test runs tab | Yes | Useful for grouped execution history |
| Global uptime metrics | Yes | Suite view can show aggregated uptime-style metrics |
| Suite-level alerting | Yes, depending on site/support | Suite alerts are separate from individual alerts |
| Mark tests critical/ignored | Yes | Used for suite-level alert decisions |
| Alert when percentage of critical tests fail | Yes | Useful for reducing noise |
| Run multiple tests manually | Yes | Current docs mention max 50 tests can be run at a time |
| Maximum tests per suite | Yes | Current docs mention up to 300 tests per suite |
| Replace individual tests | No | Suite only groups tests; it does not become the real test |
| Per-endpoint assertion | No | Assertions live inside individual tests |
| Per-step browser recording | No | Browser steps live inside browser tests |
| Per-test request configuration | No | Request config lives inside the individual test |
8. Feature Comparison: What One Has and the Other Does Not
8.1 Features Synthetic Tests Have, But Test Suites Do Not
| Feature | Synthetic Test | Test Suite |
|---|---|---|
| Actual HTTP/API request execution | Yes | No |
| Actual browser recording steps | Yes | No |
| Actual mobile app test steps | Yes | No |
| Assertions | Yes | No |
| Request headers/body | Yes | No |
| gRPC proto/method behavior config | Yes | No |
| DNS/SSL/TCP/UDP/ICMP configuration | Yes | No |
| Authentication details | Yes | No |
| Response validation | Yes | No |
| Direct monitor for one endpoint/flow | Yes | No |
| Direct source for Monitor-based SLO | Yes | Not usually the direct source |
8.2 Features Test Suites Have, But Synthetic Tests Do Not
| Feature | Synthetic Test | Test Suite |
|---|---|---|
| Group many tests together | No | Yes |
| View journey-level health | Limited | Yes |
| Suite-level alerting | No | Yes |
| Mark tests as Critical or Ignored for suite alert | No | Yes |
| Alert based on percentage of critical tests failing | No | Yes |
| Run selected tests together | No | Yes |
| Manage tests by business journey | Limited | Yes |
| See all related tests in one place | No | Yes |
| Troubleshoot grouped failures | Limited | Yes |
9. SLO Behavior: Very Important
This is where confusion usually happens.
A Synthetic Test and an SLO are not the same thing.
A Synthetic Test checks whether something is working.
An SLO defines a reliability target, such as:
The login API should be available 99.9% of the time over 30 days.
9.1 For Individual Synthetic Test SLO
The usual flow is:
Synthetic Test
↓
Datadog Monitor
↓
Monitor-based SLO
↓
Dashboard SLO Widget
Example:
| Item | Example |
|---|---|
| Synthetic Test | prod-login-api-health |
| Monitor | Auto/linked Synthetic monitor |
| SLO | Login API availability 99.9% |
| Dashboard widget | SLO widget showing error budget |
9.2 For Test Suite SLO
Be careful here.
A Test Suite gives you grouped visibility and suite-level alerting, but you should not assume that creating a Test Suite will always create a visible SLO automatically.
If you want a Dashboard SLO widget, the safest approach is:
| Requirement | Recommended Action |
|---|---|
| Show SLO for one endpoint | Create Monitor-based SLO from that Synthetic Test |
| Show SLO for one browser journey | Create Monitor-based SLO from that Browser Test |
| Show SLO for critical business journey | Create SLO from one or more important monitors, or use suite-level alerting separately |
| Show grouped health | Use Test Suite view and dashboard widgets |
| Avoid missing SLO confusion | Create SLOs manually and name/tag them clearly |
10. Alerting Difference
10.1 Synthetic Test Alerting
Synthetic Test alerting is precise.
Example:
Alert if login API fails from 2 of 3 locations for 5 minutes.
Good for:
| Use Case | Why |
|---|---|
| Endpoint down | Direct failure |
| SSL expiry | Direct certificate issue |
| DNS problem | Direct DNS validation |
| One browser flow broken | Direct user journey issue |
| gRPC method broken | Direct RPC validation |
10.2 Test Suite Alerting
Test Suite alerting is broader.
Example:
Alert if 30% of critical tests in the checkout suite fail.
Good for:
| Use Case | Why |
|---|---|
| Avoid alert storm | One suite alert instead of many test alerts |
| Business journey monitoring | Checkout/login/payment grouped together |
| Operations dashboard | Easier for NOC/SRE/team dashboard |
| Prioritization | Critical tests can trigger alert; ignored tests do not |
11. Recommended Design Pattern
For a production application, use both.
Layer 1: Individual Synthetic Tests
Create individual tests for each important endpoint or journey.
Example:
| Test Name | Type | Purpose |
|---|---|---|
prod-homepage-browser | Browser | Homepage availability |
prod-login-api | HTTP API | Login endpoint |
prod-checkout-browser | Browser | Checkout user journey |
prod-payment-api | HTTP API | Payment endpoint |
prod-grpc-health | gRPC | gRPC service health |
Layer 2: Test Suites
Group those tests into suites.
| Suite Name | Included Tests |
|---|---|
prod-critical-uptime-suite | Login, homepage, payment, checkout |
prod-checkout-journey-suite | Search, cart, payment, order confirmation |
staging-release-validation-suite | Smoke tests before release |
auth-service-suite | Login, token, refresh, logout |
Layer 3: SLOs
Create SLOs for what leadership and SRE care about.
| SLO Name | Source |
|---|---|
Login API Availability 99.9% | Login API Synthetic monitor |
Checkout Journey Availability 99.5% | Checkout browser monitor or selected monitors |
Payment API Availability 99.9% | Payment API Synthetic monitor |
Production Critical Uptime | Carefully selected critical monitors |
Layer 4: Dashboard
Create a dashboard with:
| Widget | Purpose |
|---|---|
| SLO widget | Reliability target and error budget |
| Synthetic test summary | Current test health |
| Timeseries | Latency trend |
| Top list/table | Failing locations/tests |
| Test Suite status | Business journey health |
| Logs/APM links | Troubleshooting |
12. Naming Convention Best Practice
Use consistent names. This makes SLOs, dashboards, and suites much easier to manage.
Recommended naming:
<env>-<service-or-journey>-<test-type>-<purpose>
Examples:
| Name | Meaning |
|---|---|
prod-auth-api-login-health | Production login API health check |
prod-web-browser-checkout-flow | Production checkout browser flow |
stg-payment-api-smoke | Staging payment smoke test |
prod-mobile-login-flow | Production mobile login test |
For suites:
<env>-<business-area>-suite
Examples:
| Suite Name | Meaning |
|---|---|
prod-critical-uptime-suite | Critical production checks |
prod-checkout-journey-suite | Checkout journey tests |
stg-release-validation-suite | Pre-release staging checks |
prod-auth-service-suite | Auth service tests |
For SLOs:
<env> <service/journey> Availability <target>
Examples:
| SLO Name | Meaning |
|---|---|
prod login API availability 99.9% | SLO for login API |
prod checkout journey availability 99.5% | SLO for checkout flow |
prod payment API availability 99.9% | SLO for payment endpoint |
13. Tagging Best Practice
Tags are very important in Datadog.
Recommended tags:
| Tag | Example |
|---|---|
env | env:prod |
service | service:auth |
team | team:platform |
journey | journey:checkout |
criticality | criticality:tier1 |
region | region:ap-northeast-1 |
owner | owner:sre |
Example:
env:prod
service:payment
team:platform
journey:checkout
criticality:tier1
This helps with filtering:
| Area | Benefit |
|---|---|
| Synthetic test list | Quickly find tests |
| Test Suites | Add tests by tag |
| Dashboards | Filter by env/team/service |
| SLOs | Organize ownership |
| Cost attribution | Understand Synthetic usage |
14. Practical Tutorial: How to Build This in Datadog
Step 1: Create Individual Synthetic Tests
Go to:
Digital Experience → Synthetic Monitoring → New Test
Choose test type:
| Requirement | Choose |
|---|---|
| Check REST API | API Test → HTTP |
| Check gRPC service | API Test → gRPC |
| Check website flow | Browser Test |
| Check mobile app flow | Mobile Application Test |
| Check SSL expiry | API Test → SSL |
| Check DNS | API Test → DNS |
Configure:
Name
URL/endpoint
Locations
Frequency
Assertions
Retries
Alerting condition
Tags
Notification message
Save the test.
Step 2: Create a Test Suite
Go to:
Digital Experience → Test Suites → New Test Suite
Configure:
Suite name
Add existing Synthetic Tests
Mark tests as Critical or Ignored
Configure suite alerting if available
Save Suite
Example:
Suite Name: prod-checkout-journey-suite
Tests:
- prod-homepage-browser
- prod-login-api
- prod-product-search-browser
- prod-add-to-cart-browser
- prod-payment-api
- prod-order-confirmation-api
Step 3: Create SLOs Manually
Go to:
Service Management → Service Level Objectives → New SLO
Choose:
Monitor-based SLO
Then select the Synthetic monitor.
Example:
SLO Name: prod login API availability 99.9%
Source Monitor: prod-auth-api-login-health
Target: 99.9%
Window: 30 days
Tags:
env:prod
service:auth
team:platform
Save.
Step 4: Add SLOs to Dashboard
Go to:
Dashboard → Edit → Add Widget → SLO Widget
Select the SLO you created.
Important:
A dashboard SLO widget does not create SLOs. It only displays SLOs that already exist.
So if a new Synthetic Test is missing from the dashboard, check:
Was an SLO created for it?
Was the SLO added to the dashboard widget?
Are dashboard filters hiding it?
Are tags correct?
15. When to Use Synthetic Test vs Test Suite
| Scenario | Use Synthetic Test | Use Test Suite |
|---|---|---|
| Monitor one API endpoint | Yes | Optional |
| Monitor one SSL certificate | Yes | Optional |
| Monitor one gRPC service | Yes | Optional |
| Monitor login browser journey | Yes | Optional |
| Group all login-related checks | No | Yes |
| Group all checkout tests | No | Yes |
| Create business journey view | No | Yes |
| Create per-endpoint SLO | Yes | No |
| Reduce alert noise from many related tests | Maybe | Yes |
| Run smoke tests together | Maybe | Yes |
| Build executive dashboard | Yes, via SLOs | Yes, for grouped context |
16. Common Confusions
Confusion 1: “I created a Synthetic Test, but no SLO appeared.”
That is expected.
Create a Monitor-based SLO manually.
Confusion 2: “I created a Test Suite, but no new SLO appeared.”
Do not assume Test Suite creation automatically creates a visible SLO in your SLO list.
Check:
Service Level Objectives page
Dashboard widget filters
Suite page
Permissions
Datadog site support
If you need an SLO, create it manually.
Confusion 3: “I added tests to a suite, but they are missing old results.”
Suite results normally apply from the time the test was added to the suite. For older history, check the individual test page.
Confusion 4: “Should I mute individual test alerts after adding suite alerting?”
Maybe yes.
If you enable suite-level alerting and individual test alerting, you may get duplicate notifications.
Common approach:
| Test Type | Alerting Strategy |
|---|---|
| Tier-1 critical individual test | Keep individual alert |
| Non-critical test in suite | Mute or reduce individual alert |
| Business journey suite | Enable suite alert |
| SLO-backed test | Keep alert carefully aligned with SLO |
17. Recommended Alerting Strategy
Use this model:
| Level | Purpose | Example |
|---|---|---|
| Individual Synthetic Alert | Precise technical failure | Login API down |
| Suite Alert | Business journey degraded | Checkout suite failing |
| SLO Burn/Error Budget Alert | Reliability risk | Error budget burning too fast |
| Dashboard | Visibility and trend | SLO, uptime, latency, failures |
Best practice:
Do not alert everyone for everything.
Recommended pattern:
| Test Criticality | Alert Behavior |
|---|---|
| Tier-1 critical | Pager/urgent alert |
| Tier-2 important | Team channel alert |
| Tier-3 informational | Dashboard only |
| Noisy/non-critical | Suite-only or muted |
18. Example Final Design
Synthetic Tests
| Test | Type | Frequency | Alert |
|---|---|---|---|
prod-auth-api-login-health | HTTP API | 5 min | Yes |
prod-web-browser-login-flow | Browser | 10 min | Yes |
prod-payment-api-health | HTTP API | 5 min | Yes |
prod-checkout-browser-flow | Browser | 15 min | Yes |
prod-grpc-order-health | gRPC | 5 min | Yes |
Test Suites
| Suite | Tests Included | Purpose |
|---|---|---|
prod-critical-uptime-suite | Auth, payment, checkout, order | Production critical health |
prod-checkout-journey-suite | Homepage, search, cart, payment, confirmation | Business flow |
stg-release-validation-suite | Smoke tests | Pre-release validation |
SLOs
| SLO | Source | Target |
|---|---|---|
prod auth API availability | Auth API monitor | 99.9% |
prod payment API availability | Payment API monitor | 99.9% |
prod checkout journey availability | Checkout browser monitor | 99.5% |
Dashboard
| Dashboard Section | Widgets |
|---|---|
| Executive summary | SLO widgets |
| Synthetic health | Test status widgets |
| Business journey | Test Suite status |
| Troubleshooting | Failed locations, latency, APM/log links |
| Reliability | Error budget and burn rate |
19. Final Recommendation
Use Synthetic Tests for actual monitoring.
Use Test Suites for organization, grouped execution, suite-level visibility, and journey-level alerting.
Use SLOs separately for reliability targets and dashboards.
The clean design is:
Synthetic Tests = actual checks
Test Suites = grouped operational view
SLOs = reliability target
Dashboard = visibility layer
Alerts = action layer
For your case, where you created uptime tests and display them on a dashboard using SLO widgets:
For each important Synthetic Test:
1. Confirm the Synthetic Test exists.
2. Confirm its monitor exists.
3. Manually create a Monitor-based SLO.
4. Add that SLO to the dashboard SLO widget.
5. Optionally add the test into a Test Suite for grouped visibility.
Do not depend on Test Suite creation to automatically create every SLO you need.
That manual SLO step is the safest and cleanest production approach.
