Target Audience

This lab is suitable for:

DevOps engineers, SREs, cloud engineers, platform engineers, application engineers, monitoring engineers, and students learning Datadog from practical implementation.

Final Outcome

By the end, students will build this:

flowchart LR
    A[Ubuntu Linux Server] --> B[Datadog Agent]
    C[Python Flask App] --> D[Application Logs]
    C --> E[APM Traces]
    C --> F[Custom Metrics]
    G[Browser User] --> H[RUM Events]
    I[Datadog Synthetics] --> C
    B --> J[Datadog Platform]
    D --> J
    E --> J
    F --> J
    H --> J
    I --> J
    J --> K[Monitors / Alerts]
    J --> L[Dashboards]
Code language: CSS (css)

The Datadog Agent runs on the Ubuntu host and collects infrastructure metrics, logs, traces, and service checks. Datadog documentation confirms the Agent collects host events and metrics and can also collect logs and traces when configured. (Datadog)

Overall Lab Structure

Lab Environment

Recommended Environment

Important Datadog Concepts Used in This Lab

Use the following standard tags throughout the lab:

env:lab
service:datadog-lab-shop
version:1.0.0
team:training
owner:student
Code language: CSS (css)

Datadog recommends Unified Service Tagging with env, service, and version so telemetry can be correlated across metrics, logs, traces, containers, and services. (Datadog)

Part 1: Create the Sample Python Project

Students should create one GitHub repository named:

datadog-lab-shop

This repo will contain the Python Flask application used across all assignments.

Project Structure

datadog-lab-shop/
├── app.py
├── requirements.txt
├── README.md
├── templates/
│   └── index.html
├── systemd/
│   └── datadog-lab-shop.service
├── datadog/
│   └── lab-shop-logs.yaml
└── load-test/
    └── generate-traffic.sh

requirements.txt

flask==3.0.3
gunicorn==22.0.0
ddtrace
datadog

Datadog’s Python tracing documentation says Python applications can be instrumented by installing the tracing library and prefixing the application command with ddtrace-run. (Datadog)

app.py

import json
import logging
import os
import random
import time
from datetime import datetime, timezone

from flask import Flask, jsonify, render_template, request
from datadog import initialize, statsd


APP_NAME = os.getenv("DD_SERVICE", "datadog-lab-shop")
ENV = os.getenv("DD_ENV", "lab")
VERSION = os.getenv("DD_VERSION", "1.0.0")
LOG_FILE = os.getenv("APP_LOG_FILE", "/var/log/datadog-lab-shop/app.log")

os.makedirs(os.path.dirname(LOG_FILE), exist_ok=True)

app = Flask(__name__)

initialize(
    statsd_host=os.getenv("DD_AGENT_HOST", "127.0.0.1"),
    statsd_port=int(os.getenv("DD_DOGSTATSD_PORT", "8125")),
)

logger = logging.getLogger(APP_NAME)
logger.setLevel(logging.INFO)

file_handler = logging.FileHandler(LOG_FILE)
file_handler.setLevel(logging.INFO)

console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)

logger.addHandler(file_handler)
logger.addHandler(console_handler)


def log_event(level, message, **kwargs):
    event = {
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "level": level,
        "service": APP_NAME,
        "env": ENV,
        "version": VERSION,
        "message": message,
        **kwargs,
    }
    line = json.dumps(event)
    if level == "ERROR":
        logger.error(line)
    elif level == "WARNING":
        logger.warning(line)
    else:
        logger.info(line)


@app.route("/")
def home():
    statsd.increment("datadog_lab_shop.page.views", tags=[f"env:{ENV}", f"service:{APP_NAME}"])
    log_event("INFO", "Home page viewed", path="/", client_ip=request.remote_addr)
    return render_template(
        "index.html",
        service=APP_NAME,
        env=ENV,
        version=VERSION,
        rum_app_id=os.getenv("DD_RUM_APPLICATION_ID", ""),
        rum_client_token=os.getenv("DD_RUM_CLIENT_TOKEN", ""),
        rum_site=os.getenv("DD_SITE", "datadoghq.com"),
    )


@app.route("/health")
def health():
    statsd.increment("datadog_lab_shop.health.check", tags=[f"env:{ENV}", f"service:{APP_NAME}"])
    return jsonify({
        "status": "ok",
        "service": APP_NAME,
        "env": ENV,
        "version": VERSION,
        "timestamp": datetime.now(timezone.utc).isoformat()
    })


@app.route("/api/products")
def products():
    statsd.increment("datadog_lab_shop.products.requests", tags=[f"env:{ENV}", f"service:{APP_NAME}"])
    log_event("INFO", "Products API called", path="/api/products", client_ip=request.remote_addr)

    return jsonify({
        "products": [
            {"id": 1, "name": "Datadog T-Shirt", "price": 20},
            {"id": 2, "name": "Observability Mug", "price": 12},
            {"id": 3, "name": "SRE Sticker Pack", "price": 5}
        ]
    })


@app.route("/api/cart", methods=["POST"])
def cart():
    statsd.increment("datadog_lab_shop.cart.add", tags=[f"env:{ENV}", f"service:{APP_NAME}"])
    log_event("INFO", "Item added to cart", path="/api/cart", client_ip=request.remote_addr)
    return jsonify({"message": "Item added to cart"}), 201


@app.route("/api/checkout", methods=["POST"])
def checkout():
    payment_status = random.choice(["success", "success", "success", "failed"])

    statsd.increment(
        "datadog_lab_shop.checkout.attempt",
        tags=[f"env:{ENV}", f"service:{APP_NAME}", f"status:{payment_status}"]
    )

    if payment_status == "failed":
        log_event(
            "ERROR",
            "Checkout payment failed",
            path="/api/checkout",
            payment_provider="demo-pay",
            client_ip=request.remote_addr
        )
        return jsonify({"status": "failed", "reason": "payment declined"}), 500

    log_event("INFO", "Checkout successful", path="/api/checkout", client_ip=request.remote_addr)
    return jsonify({"status": "success", "order_id": random.randint(1000, 9999)})


@app.route("/api/slow")
def slow():
    delay = random.uniform(1.5, 4.0)
    time.sleep(delay)

    statsd.histogram(
        "datadog_lab_shop.slow.request.duration",
        delay,
        tags=[f"env:{ENV}", f"service:{APP_NAME}"]
    )

    log_event("WARNING", "Slow endpoint called", path="/api/slow", delay_seconds=delay)
    return jsonify({"message": "slow response", "delay_seconds": delay})


@app.route("/api/error")
def error():
    statsd.increment("datadog_lab_shop.error.generated", tags=[f"env:{ENV}", f"service:{APP_NAME}"])
    log_event("ERROR", "Manual test error generated", path="/api/error", client_ip=request.remote_addr)
    return jsonify({"error": "manual test error"}), 500


@app.route("/api/random")
def random_behavior():
    choice = random.choice(["ok", "slow", "error"])

    if choice == "slow":
        time.sleep(2)
        log_event("WARNING", "Random slow behavior", path="/api/random")
        return jsonify({"status": "slow"})

    if choice == "error":
        log_event("ERROR", "Random error behavior", path="/api/random")
        return jsonify({"status": "error"}), 500

    log_event("INFO", "Random normal behavior", path="/api/random")
    return jsonify({"status": "ok"})


if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)
Code language: JavaScript (javascript)

templates/index.html

<!DOCTYPE html>
<html>
<head>
    <title>Datadog Lab Shop</title>

    {% if rum_app_id and rum_client_token %}
    <script
      src="https://www.datadoghq-browser-agent.com/us1/v6/datadog-rum.js"
      type="text/javascript">
    </script>
    <script>
      window.DD_RUM && window.DD_RUM.init({
        applicationId: "{{ rum_app_id }}",
        clientToken: "{{ rum_client_token }}",
        site: "{{ rum_site }}",
        service: "{{ service }}",
        env: "{{ env }}",
        version: "{{ version }}",
        sessionSampleRate: 100,
        sessionReplaySampleRate: 20,
        trackUserInteractions: true,
        trackResources: true,
        trackLongTasks: true,
        defaultPrivacyLevel: "mask-user-input"
      });

      window.DD_RUM && window.DD_RUM.setUser({
        id: "student-user-001",
        name: "Datadog Student",
        email: "student@example.com"
      });
    </script>
    {% endif %}

    <style>
        body {
            font-family: Arial, sans-serif;
            margin: 40px;
        }
        button {
            padding: 10px;
            margin: 5px;
        }
        pre {
            background: #f3f3f3;
            padding: 15px;
        }
    </style>
</head>
<body>
    <h1>Datadog Lab Shop</h1>
    <p>Service: {{ service }}</p>
    <p>Environment: {{ env }}</p>
    <p>Version: {{ version }}</p>

    <button onclick="callApi('/api/products')">Products</button>
    <button onclick="postApi('/api/cart')">Add to Cart</button>
    <button onclick="postApi('/api/checkout')">Checkout</button>
    <button onclick="callApi('/api/slow')">Slow API</button>
    <button onclick="callApi('/api/error')">Error API</button>
    <button onclick="callApi('/api/random')">Random API</button>

    <h2>Response</h2>
    <pre id="output">Click a button...</pre>

    <script>
        async function callApi(path) {
            try {
                const response = await fetch(path);
                const data = await response.json();
                document.getElementById("output").innerText =
                    JSON.stringify(data, null, 2);
            } catch (error) {
                document.getElementById("output").innerText = error;
            }
        }

        async function postApi(path) {
            try {
                const response = await fetch(path, { method: "POST" });
                const data = await response.json();
                document.getElementById("output").innerText =
                    JSON.stringify(data, null, 2);
            } catch (error) {
                document.getElementById("output").innerText = error;
            }
        }
    </script>
</body>
</html>
Code language: HTML, XML (xml)

Datadog’s RUM Browser SDK is used to monitor real user page performance, interactions, resources, and frontend errors. (Datadog)

systemd/datadog-lab-shop.service

[Unit]
Description=Datadog Lab Shop Python Flask Application
After=network.target datadog-agent.service

[Service]
User=ubuntu
Group=ubuntu
WorkingDirectory=/opt/datadog-lab-shop
Environment=DD_ENV=lab
Environment=DD_SERVICE=datadog-lab-shop
Environment=DD_VERSION=1.0.0
Environment=DD_AGENT_HOST=127.0.0.1
Environment=DD_LOGS_INJECTION=true
Environment=APP_LOG_FILE=/var/log/datadog-lab-shop/app.log
ExecStart=/opt/datadog-lab-shop/venv/bin/ddtrace-run /opt/datadog-lab-shop/venv/bin/gunicorn -b 0.0.0.0:8000 app:app
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
Code language: JavaScript (javascript)

datadog/lab-shop-logs.yaml

logs:
  - type: file
    path: /var/log/datadog-lab-shop/app.log
    service: datadog-lab-shop
    source: python
    tags:
      - env:lab
      - team:training
      - project:datadog-assignment
Code language: JavaScript (javascript)

Datadog log collection on hosts requires enabling log collection in the Agent and defining a log configuration under conf.d. (Datadog)

load-test/generate-traffic.sh

#!/usr/bin/env bash

APP_URL="${APP_URL:-http://localhost:8000}"

for i in {1..100}; do
  curl -s "${APP_URL}/health" >/dev/null
  curl -s "${APP_URL}/api/products" >/dev/null
  curl -s -X POST "${APP_URL}/api/cart" >/dev/null
  curl -s -X POST "${APP_URL}/api/checkout" >/dev/null
  curl -s "${APP_URL}/api/random" >/dev/null

  if (( i % 10 == 0 )); then
    curl -s "${APP_URL}/api/error" >/dev/null
  fi

  if (( i % 15 == 0 )); then
    curl -s "${APP_URL}/api/slow" >/dev/null
  fi

  sleep 1
done
Code language: JavaScript (javascript)

Part 2: Base Server Setup

Step 1: Prepare Ubuntu

sudo apt update
sudo apt install -y python3 python3-venv python3-pip git curl jq

Step 2: Clone the Student Project

sudo mkdir -p /opt/datadog-lab-shop
sudo chown ubuntu:ubuntu /opt/datadog-lab-shop

cd /opt/datadog-lab-shop
git clone <your-github-repo-url> .
Code language: HTML, XML (xml)

Step 3: Create Python Virtual Environment

cd /opt/datadog-lab-shop

python3 -m venv venv
source venv/bin/activate

pip install --upgrade pip
pip install -r requirements.txt

Step 4: Create Log Directory

sudo mkdir -p /var/log/datadog-lab-shop
sudo chown ubuntu:ubuntu /var/log/datadog-lab-shop
Code language: JavaScript (javascript)

Step 5: Run App Manually First

cd /opt/datadog-lab-shop
source venv/bin/activate

DD_ENV=lab \
DD_SERVICE=datadog-lab-shop \
DD_VERSION=1.0.0 \
DD_AGENT_HOST=127.0.0.1 \
DD_LOGS_INJECTION=true \
APP_LOG_FILE=/var/log/datadog-lab-shop/app.log \
ddtrace-run gunicorn -b 0.0.0.0:8000 app:app
Code language: JavaScript (javascript)

Test:

curl http://localhost:8000/health
curl http://localhost:8000/api/products
curl http://localhost:8000/api/error
Code language: JavaScript (javascript)

Part 3: Assignment 1 — Infrastructure Monitoring

Objective

Install the Datadog Agent on Ubuntu and collect host-level metrics.

The Datadog Agent collects system-level checks such as CPU, disk, IO, memory, network, NTP, uptime, file handles, and load, depending on the platform. (Datadog)

Student Tasks

Task 1: Install Datadog Agent

Set these variables first:

export DD_API_KEY="<your_datadog_api_key>"
export DD_SITE="<your_datadog_site>"
Code language: JavaScript (javascript)

Install the Agent using the Datadog UI-generated command or the official Agent installation method from Datadog.

After installation:

sudo systemctl status datadog-agent
sudo datadog-agent status

Task 2: Add Host Tags

Edit:

sudo vi /etc/datadog-agent/datadog.yaml

Add:

tags:
  - env:lab
  - team:training
  - project:datadog-assignment
  - service:datadog-lab-shop
Code language: CSS (css)

Restart:

sudo systemctl restart datadog-agent

Validate:

sudo datadog-agent status
sudo datadog-agent health

Task 3: Generate CPU and Memory Activity

Install stress tool:

sudo apt install -y stress

Generate CPU load:

stress --cpu 2 --timeout 120

Generate memory load:

stress --vm 1 --vm-bytes 512M --timeout 120

Task 4: Validate in Datadog

Search these metrics:

system.cpu.user
system.cpu.idle
system.mem.used
system.disk.used
system.net.bytes_sent
datadog.agent.running
Code language: CSS (css)

Deliverables

Grading Rubric

Part 4: Assignment 2 — Log Monitoring

Objective

Enable Datadog log collection and collect Python application logs.

Datadog logs can be collected through the Agent from files, TCP/UDP, journald, and Windows channels. For custom log files, a config is placed under the Agent conf.d directory. (Datadog)

Student Tasks

Task 1: Enable Logs in Agent

Edit:

sudo vi /etc/datadog-agent/datadog.yaml

Set:

logs_enabled: true
Code language: JavaScript (javascript)

Restart:

sudo systemctl restart datadog-agent

Task 2: Add App Log Collection

Create config directory:

sudo mkdir -p /etc/datadog-agent/conf.d/datadog-lab-shop.d

Copy config:

sudo cp /opt/datadog-lab-shop/datadog/lab-shop-logs.yaml \
  /etc/datadog-agent/conf.d/datadog-lab-shop.d/conf.yaml

Restart:

sudo systemctl restart datadog-agent

Validate:

sudo datadog-agent status
sudo datadog-agent configcheck

Task 3: Generate App Logs

curl http://localhost:8000/
curl http://localhost:8000/api/products
curl http://localhost:8000/api/slow
curl http://localhost:8000/api/error
curl -X POST http://localhost:8000/api/checkout
Code language: JavaScript (javascript)

Check local logs:

tail -f /var/log/datadog-lab-shop/app.log
Code language: JavaScript (javascript)

Check what Agent is reading:

sudo datadog-agent stream-logs

Task 4: Query Logs in Datadog

Use Log Explorer queries:

service:datadog-lab-shop
Code language: CSS (css)

service:datadog-lab-shop status:error
Code language: CSS (css)

service:datadog-lab-shop @path:/api/error
Code language: JavaScript (javascript)

service:datadog-lab-shop @level:ERROR

Task 5: Create Log Facets

Create facets for:

level
path
env
version
payment_provider
delay_seconds

Deliverables

Grading Rubric

Part 5: Assignment 3 — APM

Objective

Instrument the Python Flask application with Datadog APM and generate traces.

Datadog APM provides visibility into application performance, request traces, errors, latency, services, and dependencies. (Datadog)

Student Tasks

Task 1: Enable APM in Agent

Edit:

sudo vi /etc/datadog-agent/datadog.yaml

Ensure:

apm_config:
  enabled: true
Code language: JavaScript (javascript)

Restart:

sudo systemctl restart datadog-agent

Validate:

sudo datadog-agent status

Look for APM / Trace Agent section.

Task 2: Run App with ddtrace-run

Manual run:

cd /opt/datadog-lab-shop
source venv/bin/activate

DD_ENV=lab \
DD_SERVICE=datadog-lab-shop \
DD_VERSION=1.0.0 \
DD_AGENT_HOST=127.0.0.1 \
DD_LOGS_INJECTION=true \
ddtrace-run gunicorn -b 0.0.0.0:8000 app:app
Code language: JavaScript (javascript)

Datadog documents ddtrace-run as the standard command prefix for Python automatic instrumentation. (Datadog)

Task 3: Check Python Tracer Config

source /opt/datadog-lab-shop/venv/bin/activate
ddtrace-run --info

Task 4: Generate Traces

for i in {1..50}; do
  curl -s http://localhost:8000/health >/dev/null
  curl -s http://localhost:8000/api/products >/dev/null
  curl -s http://localhost:8000/api/slow >/dev/null
  curl -s http://localhost:8000/api/error >/dev/null
done
Code language: JavaScript (javascript)

Task 5: Validate in Datadog

Go to:

APM → Services → datadog-lab-shop

Check:

Requests
Errors
Latency
p95 latency
Resources
Traces
Service Map
Code language: JavaScript (javascript)

APM trace metrics commonly include hits, errors, latency, and Apdex. (Datadog)

Deliverables

Grading Rubric

Part 6: Assignment 4 — RUM

Objective

Enable Real User Monitoring for the frontend page of the Flask app.

RUM monitors real user activity and experience in web and mobile applications, including performance, frontend errors, user actions, network requests, and resources. (Datadog)

Important Note

RUM is not installed on the Linux server like the Agent. RUM is added to the browser frontend using the Datadog Browser SDK.

Student Tasks

Task 1: Create RUM Application

In Datadog:

Digital Experience → RUM Applications → New Application
Code language: PHP (php)

Choose:

Application type: Browser
Name: datadog-lab-shop

Copy:

Application ID
Client Token
Datadog Site

Task 2: Add RUM Values to systemd Service

Edit:

sudo vi /etc/systemd/system/datadog-lab-shop.service

Add:

Environment=DD_RUM_APPLICATION_ID=<your_rum_application_id>
Environment=DD_RUM_CLIENT_TOKEN=<your_rum_client_token>
Environment=DD_SITE=<your_datadog_site>
Code language: HTML, XML (xml)

Reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart datadog-lab-shop

Task 3: Open Browser and Generate RUM Events

Open:

http://<server-public-ip>:8000/
Code language: HTML, XML (xml)

Click all buttons:

Products
Add to Cart
Checkout
Slow API
Error API
Random API
Code language: JavaScript (javascript)

Task 4: Validate in Datadog

Go to:

Digital Experience → RUM Explorer

Check:

Views
Actions
Resources
Errors
Sessions
User Journey

Deliverables

Grading Rubric

Part 7: Assignment 5 — Synthetic Monitoring

Objective

Create synthetic tests to monitor application uptime and user behavior.

Datadog Synthetic Monitoring can create API, browser, mobile, and network tests to simulate user flows and check application availability and performance. (Datadog)

Student Tasks

Task 1: Create API Test for Health Endpoint

Go to:

Digital Experience → Tests → New Test → API Test
Code language: PHP (php)

Configure:

Datadog API tests monitor endpoints and can alert on latency, status code, headers, body content, and other conditions. (Datadog)

Task 2: Create API Test for Error Endpoint

Create another API test:

http://<server-public-ip>:8000/api/error
Code language: HTML, XML (xml)

Expected assertion:

Status code is 500

Purpose: teach expected failure vs unexpected failure.

Task 3: Create Browser Test

Go to:

Digital Experience → Tests → New Test → Browser Test
Code language: PHP (php)

Flow:

Open homepage
Click Products
Validate response contains Datadog T-Shirt
Click Add to Cart
Validate response contains Item added
Click Checkout
Validate response changes

Task 4: Trigger Failure

Stop the app:

sudo systemctl stop datadog-lab-shop

Wait for synthetic test failure.

Restart:

sudo systemctl start datadog-lab-shop

Deliverables

Grading Rubric

Part 8: Assignment 6 — Alerts / Monitors

Objective

Create meaningful Datadog monitors across infrastructure, logs, APM, RUM, and Synthetics.

Datadog supports many monitor types, including metric, host, log, APM, RUM, synthetic, anomaly, forecast, outlier, composite, SLO, and more. (Datadog)

Student Tasks

Create at least six monitors.

Monitor 1: Host CPU Monitor

Type:

Metric Monitor

Query concept:

avg:system.cpu.user{env:lab,service:datadog-lab-shop} by {host}

Alert condition:

Above 80 for 5 minutes

Message:

High CPU detected on {{host.name}} for datadog-lab-shop lab.
Check running processes and recent load test.

Monitor 2: Host No Data Monitor

Type:

Host Monitor or Metric Monitor

Purpose:

Alert if Agent stops reporting.

Use metric:

datadog.agent.running
Code language: CSS (css)

No-data alerts are useful when data is expected continuously, such as Agent host metrics. Datadog monitor configuration supports missing-data behavior such as No Data, evaluate as zero, or last known status depending on monitor type. (Datadog)

Monitor 3: Log Error Monitor

Type:

Log Monitor

Query:

service:datadog-lab-shop @level:ERROR

Condition:

More than 5 errors in 5 minutes

Message:

Application error spike detected for datadog-lab-shop.
Check Log Explorer and APM traces for /api/error or checkout failures.

Monitor 4: APM Error Rate Monitor

Type:

APM Monitor

Target:

service:datadog-lab-shop
Code language: CSS (css)

Condition:

Error rate above 5% for 5 minutes
Code language: JavaScript (javascript)

Monitor 5: APM Latency Monitor

Type:

APM Monitor

Condition:

p95 latency above 1 second for 5 minutes

Use /api/slow to trigger.

Monitor 6: Synthetic Test Monitor

Type:

Synthetic Monitor

Target:

LAB - Health API Test

Condition:

Alert when test fails from selected locations
Code language: JavaScript (javascript)

Monitor 7: RUM Error Monitor

Type:

RUM Monitor

Condition:

Frontend error count > 0 or resource error rate above threshold

Datadog has a dedicated RUM monitor type for monitoring real user performance and errors. (Datadog)

Deliverables

Grading Rubric

Part 9: Assignment 7 — Dashboard

Objective

Create one unified Datadog dashboard for the Python application.

Datadog dashboards provide visibility across products and can include widgets, template variables, permissions, and troubleshooting context. (Datadog)

Dashboard Name

Datadog Lab Shop - Full Stack Observability Dashboard

Required Template Variables

Create dashboard template variables:

Template variables dynamically filter widgets by tags, attributes, and facets. (Datadog)

Required Dashboard Widgets

Datadog supports custom widgets and cloning out-of-the-box dashboards as a starting point. (Datadog)

Deliverables

Grading Rubric

Part 10: Final Project — Complete Datadog Observability Implementation

Project Title

End-to-End Datadog Observability for Python Flask Application on Ubuntu

Final Project Scenario

You are hired as a DevOps/SRE engineer. Your company has a Python Flask application running on Ubuntu. The engineering team wants complete observability using Datadog.

You must implement:

Infrastructure monitoring
Log monitoring
APM tracing
RUM browser monitoring
Synthetic monitoring
Alerting
Dashboarding
Troubleshooting documentation
GitHub repository
Final report
Code language: PHP (php)

Final Project Step-by-Step

Step 1: Prepare GitHub Repository

Repo name:

datadog-lab-shop

Required branches:

main
feature/datadog-observability

Required folders:

app
datadog
systemd
load-test
screenshots
docs

Step 2: Deploy App on Ubuntu

sudo mkdir -p /opt/datadog-lab-shop
sudo chown ubuntu:ubuntu /opt/datadog-lab-shop

cd /opt/datadog-lab-shop
git clone <your-github-repo-url> .

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
Code language: HTML, XML (xml)

Install systemd service:

sudo cp systemd/datadog-lab-shop.service /etc/systemd/system/datadog-lab-shop.service
sudo systemctl daemon-reload
sudo systemctl enable datadog-lab-shop
sudo systemctl start datadog-lab-shop

Validate:

sudo systemctl status datadog-lab-shop
curl http://localhost:8000/health
Code language: JavaScript (javascript)

Step 3: Install and Configure Datadog Agent

sudo datadog-agent status
sudo datadog-agent health

Configure tags:

tags:
  - env:lab
  - service:datadog-lab-shop
  - version:1.0.0
  - team:training
  - project:datadog-final-project
Code language: CSS (css)

Restart:

sudo systemctl restart datadog-agent

Step 4: Enable Logs

logs_enabled: true
Code language: JavaScript (javascript)

Copy log config:

sudo mkdir -p /etc/datadog-agent/conf.d/datadog-lab-shop.d

sudo cp /opt/datadog-lab-shop/datadog/lab-shop-logs.yaml \
  /etc/datadog-agent/conf.d/datadog-lab-shop.d/conf.yaml

sudo systemctl restart datadog-agent

Validate:

sudo datadog-agent status
sudo datadog-agent stream-logs

Step 5: Enable APM

Confirm in datadog.yaml:

apm_config:
  enabled: true
Code language: JavaScript (javascript)

Restart:

sudo systemctl restart datadog-agent
sudo systemctl restart datadog-lab-shop

Validate:

curl http://localhost:8000/api/products
curl http://localhost:8000/api/slow
curl http://localhost:8000/api/error
Code language: JavaScript (javascript)

Step 6: Enable RUM

Create RUM app in Datadog and add environment variables to the systemd service:

Environment=DD_RUM_APPLICATION_ID=<rum_application_id>
Environment=DD_RUM_CLIENT_TOKEN=<rum_client_token>
Environment=DD_SITE=<datadog_site>
Code language: HTML, XML (xml)

Restart:

sudo systemctl daemon-reload
sudo systemctl restart datadog-lab-shop

Open in browser:

http://<server-public-ip>:8000/
Code language: HTML, XML (xml)

Step 7: Create Synthetic Tests

Create:

Synthetic tests can run manually, on schedule, or from CI/CD pipelines, and can use public or private locations. (Datadog)

Step 8: Create Monitors

Required final project monitors:

Step 9: Create Final Dashboard

Dashboard must include:

Host health
Agent health
Application logs
Error logs
APM latency
APM error rate
APM request rate
RUM sessions
RUM actions
Synthetic uptime
Monitor status
Custom business metrics
Code language: JavaScript (javascript)

Step 10: Create Final Report

Students must submit a report with this structure:

1. Project Overview
2. Architecture Diagram
3. Ubuntu Server Setup
4. Datadog Agent Setup
5. Infrastructure Monitoring
6. Log Monitoring
7. APM Setup
8. RUM Setup
9. Synthetic Monitoring
10. Alerts / Monitors
11. Dashboard
12. Test Scenarios
13. Troubleshooting
14. Screenshots
15. Lessons Learned
16. GitHub Repository

Final Project Test Scenarios

Students must demonstrate these scenarios.

Scenario 1: CPU Spike

stress --cpu 2 --timeout 120

Expected result:

CPU metric increases
CPU monitor may trigger
Dashboard reflects spike

Scenario 2: Application Error

curl http://localhost:8000/api/error
Code language: JavaScript (javascript)

Expected result:

Error log appears
APM error trace appears
Log/APM monitor may trigger
Code language: JavaScript (javascript)

Scenario 3: Slow API

curl http://localhost:8000/api/slow
Code language: JavaScript (javascript)

Expected result:

APM latency increases
Slow trace visible
Latency monitor may trigger

Scenario 4: Checkout Failure

for i in {1..20}; do
  curl -s -X POST http://localhost:8000/api/checkout
done
Code language: JavaScript (javascript)

Expected result:

Some checkout failures generated
Custom metric appears
Error logs created
APM error traces created
Code language: JavaScript (javascript)

Scenario 5: Synthetic Failure

sudo systemctl stop datadog-lab-shop

Expected result:

Synthetic API test fails
Synthetic monitor alerts
Dashboard shows test failure

Recover:

sudo systemctl start datadog-lab-shop

Scenario 6: RUM User Journey

Open browser and click:

Products
Add to Cart
Checkout
Slow API
Error API
Code language: JavaScript (javascript)

Expected result:

RUM session visible
Actions visible
Resource requests visible
Frontend/backend relationship visible

Final Project Deliverables

Final Project Grading Rubric

Student README Template

Students should include this in their GitHub repo.

# Datadog Lab Shop

## Project Overview

This project demonstrates end-to-end Datadog observability for a Python Flask application running on Ubuntu Linux.

## Features Implemented

- Infrastructure Monitoring
- Log Monitoring
- APM Tracing
- Real User Monitoring
- Synthetic Monitoring
- Alerts / Monitors
- Dashboard
- Custom Metrics

## Application Endpoints

| Endpoint | Purpose |
|---|---|
| `/` | Frontend page |
| `/health` | Health check |
| `/api/products` | Product API |
| `/api/cart` | Cart API |
| `/api/checkout` | Checkout API |
| `/api/slow` | Slow endpoint |
| `/api/error` | Error endpoint |
| `/api/random` | Random behavior endpoint |

## Setup Steps

1. Install Python dependencies.
2. Install Datadog Agent.
3. Enable logs.
4. Enable APM.
5. Configure RUM.
6. Create Synthetic tests.
7. Create monitors.
8. Create dashboard.

## Validation

Run:

```bash
curl http://localhost:8000/health
curl http://localhost:8000/api/products
curl http://localhost:8000/api/error
Code language: PHP (php)

Screenshots

Add screenshots under the screenshots/ folder.

Lessons Learned

Write what you learned about Datadog infrastructure monitoring, logs, APM, RUM, synthetics, alerts, and dashboards.

---

# Common Troubleshooting Section for Students

| Problem | Likely Cause | Fix |
|---|---|---|
| Host not visible | Agent not running, wrong API key, wrong site | `sudo datadog-agent status` |
| Logs not visible | `logs_enabled` false or log config wrong | Check `datadog.yaml` and `conf.d` |
| APM not visible | App not started with `ddtrace-run` | Restart app with `ddtrace-run` |
| RUM not visible | Wrong RUM app ID/client token/site | Check browser console and env vars |
| Synthetic test fails | App port blocked or server private | Open firewall or use private location |
| Monitor not triggering | Wrong query or threshold | Test query in Metrics/Logs/APM explorer |
| Dashboard empty | Wrong template variable | Clear variables and check tags |
| No custom metrics | DogStatsD not reachable | Check UDP 8125 and Agent status |

---

# Instructor Notes

## Recommended Lab Duration

| Module | Duration |
|---|---:|
| Intro + project setup | 1 hour |
| Infra monitoring | 1 hour |
| Logs | 1.5 hours |
| APM | 1.5 hours |
| RUM | 1 hour |
| Synthetics | 1 hour |
| Alerts | 1.5 hours |
| Dashboard | 1 hour |
| Final project | 3–5 hours |

Total recommended duration:

```text
12 to 15 hours
Code language: PHP (php)

Recommended Team Size

1 to 3 students per team

Recommended Final Demo

Each student/team should present:

1. Application running
2. Datadog host view
3. Logs Explorer
4. APM service page
5. RUM session
6. Synthetic test
7. Alert trigger
8. Dashboard
9. Lessons learned

Final Assignment Summary

This version turns the old “Datadog assignment” into a real, production-style, hands-on project. It gives students a working Python app, real failure scenarios, actual Datadog telemetry, screenshots to submit, and a final project they can show in interviews or internal assessments.

Rajesh Kumar

I’m a DevOps/SRE/DevSecOps/Cloud Expert passionate about sharing knowledge and experiences. I have worked at Cotocus. I share tech blog at DevOps School, travel stories at Holiday Landmark, stock market tips at Stocks Mantra, health and fitness guidance at My Medic Plus, product reviews at TrueReviewNow , and SEO strategies at Wizbrand.

Do you want to learn Quantum Computing?

Please find my social handles as below;

Rajesh Kumar Personal Website
Rajesh Kumar at YOUTUBE
Rajesh Kumar at INSTAGRAM
Rajesh Kumar at X
Rajesh Kumar at FACEBOOK
Rajesh Kumar at LINKEDIN
Rajesh Kumar at WIZBRAND

Rajesh Kumar DailyLogs