{"id":31255,"date":"2026-06-23T04:17:34","date_gmt":"2026-06-23T04:17:34","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=31255"},"modified":"2026-06-23T04:17:36","modified_gmt":"2026-06-23T04:17:36","slug":"datadog-assignment-project","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/datadog-assignment-project\/","title":{"rendered":"Datadog Assignment & Project Master Plan"},"content":{"rendered":"\n

Target Audience<\/h2>\n\n\n\n

This lab is suitable for:<\/p>\n\n\n\n

DevOps engineers, SREs, cloud engineers, platform engineers, application engineers, monitoring engineers, and students learning Datadog from practical implementation.<\/p>\n\n\n\n

Final Outcome<\/h2>\n\n\n\n

By the end, students will build this:<\/p>\n\n\n

flowchart<\/span> LR<\/span>\n    A<\/span>[Ubuntu Linux Server]<\/span> --<\/span>> B<\/span>[Datadog Agent]<\/span>\n    C<\/span>[Python Flask App]<\/span> --<\/span>> D<\/span>[Application Logs]<\/span>\n    C<\/span> --<\/span>> E<\/span>[APM Traces]<\/span>\n    C<\/span> --<\/span>> F<\/span>[Custom Metrics]<\/span>\n    G<\/span>[Browser User]<\/span> --<\/span>> H<\/span>[RUM Events]<\/span>\n    I<\/span>[Datadog Synthetics]<\/span> --<\/span>> C<\/span>\n    B<\/span> --<\/span>> J<\/span>[Datadog Platform]<\/span>\n    D<\/span> --<\/span>> J<\/span>\n    E<\/span> --<\/span>> J<\/span>\n    F<\/span> --<\/span>> J<\/span>\n    H<\/span> --<\/span>> J<\/span>\n    I<\/span> --<\/span>> J<\/span>\n    J<\/span> --<\/span>> K<\/span>[Monitors \/ Alerts]<\/span>\n    J<\/span> --<\/span>> L<\/span>[Dashboards]<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
The Datadog Agent runs on the Ubuntu host and collects infrastructure metrics, logs, traces, and service checks. Datadog documentation confirms the Agent collects host events and metrics and can also collect logs and traces when configured. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
Overall Lab Structure<\/h1>\n\n\n\n
Assignment<\/th>Topic<\/th>Main Skill<\/th><\/tr><\/thead>
Assignment 1<\/td>Infrastructure Monitoring<\/td>Install Agent, collect Linux metrics, tags, host health<\/td><\/tr>
Assignment 2<\/td>Log Monitoring<\/td>Enable log collection, collect app logs, parse\/filter logs<\/td><\/tr>
Assignment 3<\/td>APM<\/td>Instrument Python Flask app with traces<\/td><\/tr>
Assignment 4<\/td>RUM<\/td>Add browser monitoring to frontend page<\/td><\/tr>
Assignment 5<\/td>Synthetic Monitoring<\/td>Create API and browser tests<\/td><\/tr>
Assignment 6<\/td>Alerts \/ Monitors<\/td>Create infrastructure, log, APM, RUM, and synthetic alerts<\/td><\/tr>
Assignment 7<\/td>Dashboard<\/td>Build one unified service dashboard<\/td><\/tr>
Final Project<\/td>Complete Observability Project<\/td>End-to-end Datadog implementation using Python app<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Lab Environment<\/h1>\n\n\n\n
Recommended Environment<\/h2>\n\n\n\n
Component<\/th>Recommendation<\/th><\/tr><\/thead>
OS<\/td>Ubuntu 22.04 or 24.04<\/td><\/tr>
CPU<\/td>2 vCPU minimum<\/td><\/tr>
RAM<\/td>2 GB minimum, 4 GB preferred<\/td><\/tr>
Python<\/td>Python 3.10+<\/td><\/tr>
Web App<\/td>Flask<\/td><\/tr>
App Port<\/td>8000<\/td><\/tr>
Datadog Agent<\/td>Agent 7.x<\/td><\/tr>
Datadog Site<\/td>Your Datadog site, for example US1, EU, AP1, AP2<\/td><\/tr>
GitHub<\/td>One student repo per student or team<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Important Datadog Concepts Used in This Lab<\/h1>\n\n\n\n
Concept<\/th>Why It Matters<\/th><\/tr><\/thead>
Agent<\/td>Collects infrastructure metrics, logs, traces, and service data<\/td><\/tr>
Tags<\/td>Used for filtering, grouping, dashboards, alerts, and correlation<\/td><\/tr>
env<\/code><\/td>Environment tag, for example lab<\/code>, dev<\/code>, prod<\/code><\/td><\/tr>
service<\/code><\/td>Application\/service name<\/td><\/tr>
version<\/code><\/td>App release version<\/td><\/tr>
Logs<\/td>Application and system events<\/td><\/tr>
APM<\/td>Backend request tracing<\/td><\/tr>
RUM<\/td>Real browser user experience monitoring<\/td><\/tr>
Synthetics<\/td>Proactive uptime and behavior testing<\/td><\/tr>
Monitors<\/td>Alerting rules<\/td><\/tr>
Dashboards<\/td>Visual observability views<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Use the following standard tags throughout the lab:<\/p>\n\n\n
env<\/span>:lab<\/span>\nservice<\/span>:datadog-lab-shop<\/span>\nversion<\/span>:1.0.0<\/span>\nteam<\/span>:training<\/span>\nowner<\/span>:student<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Datadog recommends Unified Service Tagging with env<\/code>, service<\/code>, and version<\/code> so telemetry can be correlated across metrics, logs, traces, containers, and services. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
Part 1: Create the Sample Python Project<\/h1>\n\n\n\n
Students should create one GitHub repository named:<\/p>\n\n\n
datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
This repo will contain the Python Flask application used across all assignments.<\/p>\n\n\n\n
Project Structure<\/h2>\n\n\n
datadog-lab-shop\/\n\u251c\u2500\u2500 app.py\n\u251c\u2500\u2500 requirements.txt\n\u251c\u2500\u2500 README.md\n\u251c\u2500\u2500 templates\/\n\u2502   \u2514\u2500\u2500 index.html\n\u251c\u2500\u2500 systemd\/\n\u2502   \u2514\u2500\u2500 datadog-lab-shop.service\n\u251c\u2500\u2500 datadog\/\n\u2502   \u2514\u2500\u2500 lab-shop-logs.yaml\n\u2514\u2500\u2500 load-test\/\n    \u2514\u2500\u2500 generate-traffic.sh\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
requirements.txt<\/code><\/h2>\n\n\n
flask==3.0.3\ngunicorn==22.0.0\nddtrace\ndatadog\n<\/code><\/span><\/pre>\n\n\n
Datadog\u2019s Python tracing documentation says Python applications can be instrumented by installing the tracing library and prefixing the application command with ddtrace-run<\/code>. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
app.py<\/code><\/h2>\n\n\n
import<\/span> json\nimport<\/span> logging\nimport<\/span> os\nimport<\/span> random\nimport<\/span> time\nfrom<\/span> datetime import<\/span> datetime, timezone\n\nfrom<\/span> flask import<\/span> Flask, jsonify, render_template, request\nfrom<\/span> datadog import<\/span> initialize, statsd\n\n\nAPP_NAME = os.getenv(\"DD_SERVICE\"<\/span>, \"datadog-lab-shop\"<\/span>)\nENV = os.getenv(\"DD_ENV\"<\/span>, \"lab\"<\/span>)\nVERSION = os.getenv(\"DD_VERSION\"<\/span>, \"1.0.0\"<\/span>)\nLOG_FILE = os.getenv(\"APP_LOG_FILE\"<\/span>, \"\/var\/log\/datadog-lab-shop\/app.log\"<\/span>)\n\nos.makedirs(os.path.dirname(LOG_FILE), exist_ok=True)\n\napp = Flask(__name__)\n\ninitialize(\n    statsd_host=os.getenv(\"DD_AGENT_HOST\"<\/span>, \"127.0.0.1\"<\/span>),\n    statsd_port=int(os.getenv(\"DD_DOGSTATSD_PORT\"<\/span>, \"8125\"<\/span>)),\n)\n\nlogger = logging.getLogger(APP_NAME)\nlogger.setLevel(logging.INFO)\n\nfile_handler = logging.FileHandler(LOG_FILE)\nfile_handler.setLevel(logging.INFO)\n\nconsole_handler = logging.StreamHandler()\nconsole_handler.setLevel(logging.INFO)\n\nlogger.addHandler(file_handler)\nlogger.addHandler(console_handler)\n\n\ndef log_event(level, message, **kwargs):\n    event = {\n        \"timestamp\"<\/span>: datetime.now(timezone.utc).isoformat(),\n        \"level\"<\/span>: level,\n        \"service\"<\/span>: APP_NAME,\n        \"env\"<\/span>: ENV,\n        \"version\"<\/span>: VERSION,\n        \"message\"<\/span>: message,\n        **kwargs,\n    }\n    line = json.dumps(event)\n    if<\/span> level == \"ERROR\"<\/span>:\n        logger.error(line)\n    elif level == \"WARNING\"<\/span>:\n        logger.warning(line)\n    else<\/span>:\n        logger.info(line)\n\n\n@app.route(\"\/\"<\/span>)\ndef home():\n    statsd.increment(\"datadog_lab_shop.page.views\"<\/span>, tags=[f\"env:{ENV}\"<\/span>, f\"service:{APP_NAME}\"<\/span>])\n    log_event(\"INFO\"<\/span>, \"Home page viewed\"<\/span>, path=\"\/\"<\/span>, client_ip=request.remote_addr)\n    return<\/span> render_template(\n        \"index.html\"<\/span>,\n        service=APP_NAME,\n        env=ENV,\n        version=VERSION,\n        rum_app_id=os.getenv(\"DD_RUM_APPLICATION_ID\"<\/span>, \"\"<\/span>),\n        rum_client_token=os.getenv(\"DD_RUM_CLIENT_TOKEN\"<\/span>, \"\"<\/span>),\n        rum_site=os.getenv(\"DD_SITE\"<\/span>, \"datadoghq.com\"<\/span>),\n    )\n\n\n@app.route(\"\/health\"<\/span>)\ndef health():\n    statsd.increment(\"datadog_lab_shop.health.check\"<\/span>, tags=[f\"env:{ENV}\"<\/span>, f\"service:{APP_NAME}\"<\/span>])\n    return<\/span> jsonify({\n        \"status\"<\/span>: \"ok\"<\/span>,\n        \"service\"<\/span>: APP_NAME,\n        \"env\"<\/span>: ENV,\n        \"version\"<\/span>: VERSION,\n        \"timestamp\"<\/span>: datetime.now(timezone.utc).isoformat()\n    })\n\n\n@app.route(\"\/api\/products\"<\/span>)\ndef products():\n    statsd.increment(\"datadog_lab_shop.products.requests\"<\/span>, tags=[f\"env:{ENV}\"<\/span>, f\"service:{APP_NAME}\"<\/span>])\n    log_event(\"INFO\"<\/span>, \"Products API called\"<\/span>, path=\"\/api\/products\"<\/span>, client_ip=request.remote_addr)\n\n    return<\/span> jsonify({\n        \"products\"<\/span>: [\n            {\"id\"<\/span>: 1<\/span>, \"name\"<\/span>: \"Datadog T-Shirt\"<\/span>, \"price\"<\/span>: 20<\/span>},\n            {\"id\"<\/span>: 2<\/span>, \"name\"<\/span>: \"Observability Mug\"<\/span>, \"price\"<\/span>: 12<\/span>},\n            {\"id\"<\/span>: 3<\/span>, \"name\"<\/span>: \"SRE Sticker Pack\"<\/span>, \"price\"<\/span>: 5<\/span>}\n        ]\n    })\n\n\n@app.route(\"\/api\/cart\"<\/span>, methods=[\"POST\"<\/span>])\ndef cart():\n    statsd.increment(\"datadog_lab_shop.cart.add\"<\/span>, tags=[f\"env:{ENV}\"<\/span>, f\"service:{APP_NAME}\"<\/span>])\n    log_event(\"INFO\"<\/span>, \"Item added to cart\"<\/span>, path=\"\/api\/cart\"<\/span>, client_ip=request.remote_addr)\n    return<\/span> jsonify({\"message\"<\/span>: \"Item added to cart\"<\/span>}), 201<\/span>\n\n\n@app.route(\"\/api\/checkout\"<\/span>, methods=[\"POST\"<\/span>])\ndef checkout():\n    payment_status = random.choice([\"success\"<\/span>, \"success\"<\/span>, \"success\"<\/span>, \"failed\"<\/span>])\n\n    statsd.increment(\n        \"datadog_lab_shop.checkout.attempt\"<\/span>,\n        tags=[f\"env:{ENV}\"<\/span>, f\"service:{APP_NAME}\"<\/span>, f\"status:{payment_status}\"<\/span>]\n    )\n\n    if<\/span> payment_status == \"failed\"<\/span>:\n        log_event(\n            \"ERROR\"<\/span>,\n            \"Checkout payment failed\"<\/span>,\n            path=\"\/api\/checkout\"<\/span>,\n            payment_provider=\"demo-pay\"<\/span>,\n            client_ip=request.remote_addr\n        )\n        return<\/span> jsonify({\"status\"<\/span>: \"failed\"<\/span>, \"reason\"<\/span>: \"payment declined\"<\/span>}), 500<\/span>\n\n    log_event(\"INFO\"<\/span>, \"Checkout successful\"<\/span>, path=\"\/api\/checkout\"<\/span>, client_ip=request.remote_addr)\n    return<\/span> jsonify({\"status\"<\/span>: \"success\"<\/span>, \"order_id\"<\/span>: random.randint(1000<\/span>, 9999<\/span>)})\n\n\n@app.route(\"\/api\/slow\"<\/span>)\ndef slow():\n    delay = random.uniform(1.5<\/span>, 4.0<\/span>)\n    time.sleep(delay)\n\n    statsd.histogram(\n        \"datadog_lab_shop.slow.request.duration\"<\/span>,\n        delay,\n        tags=[f\"env:{ENV}\"<\/span>, f\"service:{APP_NAME}\"<\/span>]\n    )\n\n    log_event(\"WARNING\"<\/span>, \"Slow endpoint called\"<\/span>, path=\"\/api\/slow\"<\/span>, delay_seconds=delay)\n    return<\/span> jsonify({\"message\"<\/span>: \"slow response\"<\/span>, \"delay_seconds\"<\/span>: delay})\n\n\n@app.route(\"\/api\/error\"<\/span>)\ndef error():\n    statsd.increment(\"datadog_lab_shop.error.generated\"<\/span>, tags=[f\"env:{ENV}\"<\/span>, f\"service:{APP_NAME}\"<\/span>])\n    log_event(\"ERROR\"<\/span>, \"Manual test error generated\"<\/span>, path=\"\/api\/error\"<\/span>, client_ip=request.remote_addr)\n    return<\/span> jsonify({\"error\"<\/span>: \"manual test error\"<\/span>}), 500<\/span>\n\n\n@app.route(\"\/api\/random\"<\/span>)\ndef random_behavior():\n    choice = random.choice([\"ok\"<\/span>, \"slow\"<\/span>, \"error\"<\/span>])\n\n    if<\/span> choice == \"slow\"<\/span>:\n        time.sleep(2<\/span>)\n        log_event(\"WARNING\"<\/span>, \"Random slow behavior\"<\/span>, path=\"\/api\/random\"<\/span>)\n        return<\/span> jsonify({\"status\"<\/span>: \"slow\"<\/span>})\n\n    if<\/span> choice == \"error\"<\/span>:\n        log_event(\"ERROR\"<\/span>, \"Random error behavior\"<\/span>, path=\"\/api\/random\"<\/span>)\n        return<\/span> jsonify({\"status\"<\/span>: \"error\"<\/span>}), 500<\/span>\n\n    log_event(\"INFO\"<\/span>, \"Random normal behavior\"<\/span>, path=\"\/api\/random\"<\/span>)\n    return<\/span> jsonify({\"status\"<\/span>: \"ok\"<\/span>})\n\n\nif<\/span> __name__ == \"__main__\"<\/span>:\n    app.run(host=\"0.0.0.0\"<\/span>, port=8000<\/span>)\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
templates\/index.html<\/code><\/h2>\n\n\n
<!DOCTYPE html<\/span>><\/span>\n<html<\/span>><\/span>\n<head<\/span>><\/span>\n    <title<\/span>><\/span>Datadog Lab Shop<\/title<\/span>><\/span>\n\n    {% if rum_app_id and rum_client_token %}\n    <script<\/span>\n      src<\/span>=\"https:\/\/www.datadoghq-browser-agent.com\/us1\/v6\/datadog-rum.js\"<\/span>\n      type<\/span>=\"text\/javascript\"<\/span>><\/span>\n    <\/script<\/span>><\/span>\n    <script<\/span>><\/span>\n      window<\/span>.DD_RUM && window<\/span>.DD_RUM.init({\n        applicationId<\/span>: \"{{ rum_app_id }}\"<\/span>,\n        clientToken<\/span>: \"{{ rum_client_token }}\"<\/span>,\n        site<\/span>: \"{{ rum_site }}\"<\/span>,\n        service<\/span>: \"{{ service }}\"<\/span>,\n        env<\/span>: \"{{ env }}\"<\/span>,\n        version<\/span>: \"{{ version }}\"<\/span>,\n        sessionSampleRate<\/span>: 100<\/span>,\n        sessionReplaySampleRate<\/span>: 20<\/span>,\n        trackUserInteractions<\/span>: true<\/span>,\n        trackResources<\/span>: true<\/span>,\n        trackLongTasks<\/span>: true<\/span>,\n        defaultPrivacyLevel<\/span>: \"mask-user-input\"<\/span>\n      });\n\n      window<\/span>.DD_RUM && window<\/span>.DD_RUM.setUser({\n        id<\/span>: \"student-user-001\"<\/span>,\n        name<\/span>: \"Datadog Student\"<\/span>,\n        email<\/span>: \"student@example.com\"<\/span>\n      });\n    <\/span><\/script<\/span>><\/span>\n    {% endif %}\n\n    <style<\/span>><\/span>\n        body<\/span> {\n            font-family<\/span>: Arial, sans-serif;\n            margin<\/span>: 40px<\/span>;\n        }\n        button<\/span> {\n            padding<\/span>: 10px<\/span>;\n            margin<\/span>: 5px<\/span>;\n        }\n        pre<\/span> {\n            background<\/span>: #f3f3f3<\/span>;\n            padding<\/span>: 15px<\/span>;\n        }\n    <\/span><\/style<\/span>><\/span>\n<\/head<\/span>><\/span>\n<body<\/span>><\/span>\n    <h1<\/span>><\/span>Datadog Lab Shop<\/h1<\/span>><\/span>\n    <p<\/span>><\/span>Service: {{ service }}<\/p<\/span>><\/span>\n    <p<\/span>><\/span>Environment: {{ env }}<\/p<\/span>><\/span>\n    <p<\/span>><\/span>Version: {{ version }}<\/p<\/span>><\/span>\n\n    <button<\/span> onclick<\/span>=\"callApi('\/api\/products')\"<\/span>><\/span>Products<\/button<\/span>><\/span>\n    <button<\/span> onclick<\/span>=\"postApi('\/api\/cart')\"<\/span>><\/span>Add to Cart<\/button<\/span>><\/span>\n    <button<\/span> onclick<\/span>=\"postApi('\/api\/checkout')\"<\/span>><\/span>Checkout<\/button<\/span>><\/span>\n    <button<\/span> onclick<\/span>=\"callApi('\/api\/slow')\"<\/span>><\/span>Slow API<\/button<\/span>><\/span>\n    <button<\/span> onclick<\/span>=\"callApi('\/api\/error')\"<\/span>><\/span>Error API<\/button<\/span>><\/span>\n    <button<\/span> onclick<\/span>=\"callApi('\/api\/random')\"<\/span>><\/span>Random API<\/button<\/span>><\/span>\n\n    <h2<\/span>><\/span>Response<\/h2<\/span>><\/span>\n    <pre<\/span> id<\/span>=\"output\"<\/span>><\/span>Click a button...<\/pre<\/span>><\/span>\n\n    <script<\/span>><\/span>\n        async<\/span> function<\/span> callApi<\/span>(path<\/span>) <\/span>{\n            try<\/span> {\n                const<\/span> response = await<\/span> fetch(path);\n                const<\/span> data = await<\/span> response.json();\n                document<\/span>.getElementById(\"output\"<\/span>).innerText =\n                    JSON<\/span>.stringify(data, null<\/span>, 2<\/span>);\n            } catch<\/span> (error) {\n                document<\/span>.getElementById(\"output\"<\/span>).innerText = error;\n            }\n        }\n\n        async<\/span> function<\/span> postApi<\/span>(path<\/span>) <\/span>{\n            try<\/span> {\n                const<\/span> response = await<\/span> fetch(path, { method<\/span>: \"POST\"<\/span> });\n                const<\/span> data = await<\/span> response.json();\n                document<\/span>.getElementById(\"output\"<\/span>).innerText =\n                    JSON<\/span>.stringify(data, null<\/span>, 2<\/span>);\n            } catch<\/span> (error) {\n                document<\/span>.getElementById(\"output\"<\/span>).innerText = error;\n            }\n        }\n    <\/span><\/script<\/span>><\/span>\n<\/body<\/span>><\/span>\n<\/html<\/span>><\/span>\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
systemd\/datadog-lab-shop.service<\/code><\/h2>\n\n\n
[Unit]\nDescription=Datadog Lab Shop Python Flask Application\nAfter=network.target datadog-agent.service\n\n[Service]\nUser=ubuntu\nGroup=ubuntu\nWorkingDirectory=\/opt\/<\/span>datadog-lab-shop\nEnvironment=DD_ENV=lab\nEnvironment=DD_SERVICE=datadog-lab-shop\nEnvironment=DD_VERSION=1.0<\/span>.0<\/span>\nEnvironment=DD_AGENT_HOST=127.0<\/span>.0<\/span>.1<\/span>\nEnvironment=DD_LOGS_INJECTION=true<\/span>\nEnvironment=APP_LOG_FILE=\/var\/<\/span>log\/datadog-lab-shop\/app.log\nExecStart=\/opt\/<\/span>datadog-lab-shop\/venv\/bin\/ddtrace-run \/opt\/datadog-lab-shop\/venv\/bin\/gunicorn -b 0.0<\/span>.0<\/span>.0<\/span>:8000<\/span> app:app\nRestart=always\nRestartSec=5<\/span>\n\n[Install]\nWantedBy=multi-user.target\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
datadog\/lab-shop-logs.yaml<\/code><\/h2>\n\n\n
logs:\n  - type: file\n    path<\/span>: \/var\/<\/span>log\/datadog-lab-shop\/app.log\n    service<\/span>: datadog-lab-shop\n    source<\/span>: python\n    tags<\/span>:\n      - env:lab\n      - team:training\n      - project:datadog-assignment\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
load-test\/generate-traffic.sh<\/code><\/h2>\n\n\n
#!\/usr\/bin\/env bash<\/span>\n\nAPP_URL=\"${APP_URL:-http:\/\/localhost:8000}\"<\/span>\n\nfor<\/span> i in<\/span> {1.<\/span>.100<\/span>}; do<\/span>\n  curl -s \"${APP_URL}\/health\"<\/span> >\/dev\/<\/span>null<\/span>\n  curl -s \"${APP_URL}\/api\/products\"<\/span> >\/dev\/<\/span>null<\/span>\n  curl -s -X POST \"${APP_URL}\/api\/cart\"<\/span> >\/dev\/<\/span>null<\/span>\n  curl -s -X POST \"${APP_URL}\/api\/checkout\"<\/span> >\/dev\/<\/span>null<\/span>\n  curl -s \"${APP_URL}\/api\/random\"<\/span> >\/dev\/<\/span>null<\/span>\n\n  if<\/span> (( i % 10<\/span> == 0<\/span> )); then\n    curl -s \"${APP_URL}\/api\/error\"<\/span> >\/dev\/<\/span>null<\/span>\n  fi\n\n  if<\/span> (( i % 15<\/span> == 0<\/span> )); then\n    curl -s \"${APP_URL}\/api\/slow\"<\/span> >\/dev\/<\/span>null<\/span>\n  fi\n\n  sleep 1<\/span>\ndone\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Part 2: Base Server Setup<\/h1>\n\n\n\n
Step 1: Prepare Ubuntu<\/h2>\n\n\n
sudo apt update\nsudo apt install -y python3 python3-venv python3-pip git curl jq\n<\/code><\/span><\/pre>\n\n\n
Step 2: Clone the Student Project<\/h2>\n\n\n
sudo mkdir -p \/opt\/datadog-lab-shop\nsudo chown ubuntu:ubuntu \/opt\/datadog-lab-shop\n\ncd \/opt\/datadog-lab-shop\ngit clone <your-github-repo-url<\/span>><\/span> .\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Step 3: Create Python Virtual Environment<\/h2>\n\n\n
cd \/opt\/datadog-lab-shop\n\npython3 -m venv venv\nsource venv\/bin\/activate\n\npip install --upgrade pip\npip install -r requirements.txt\n<\/code><\/span><\/pre>\n\n\n
Step 4: Create Log Directory<\/h2>\n\n\n
sudo mkdir -p \/var<\/span>\/log\/datadog-lab-shop\nsudo chown ubuntu:ubuntu \/var<\/span>\/log\/datadog-lab-shop\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Step 5: Run App Manually First<\/h2>\n\n\n
cd \/opt\/datadog-lab-shop\nsource venv\/bin\/activate\n\nDD_ENV=lab \\\nDD_SERVICE=datadog-lab-shop \\\nDD_VERSION=1.0<\/span>.0<\/span> \\\nDD_AGENT_HOST=127.0<\/span>.0<\/span>.1<\/span> \\\nDD_LOGS_INJECTION=true<\/span> \\\nAPP_LOG_FILE=\/var\/<\/span>log\/datadog-lab-shop\/app.log \\\nddtrace-run gunicorn -b 0.0<\/span>.0<\/span>.0<\/span>:8000<\/span> app:app\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Test:<\/p>\n\n\n
curl http:\/\/localhost:8000\/health<\/span>\ncurl http:\/\/localhost:8000\/api\/products<\/span>\ncurl http:\/\/localhost:8000\/api\/error<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Part 3: Assignment 1 \u2014 Infrastructure Monitoring<\/h1>\n\n\n\n
Objective<\/h2>\n\n\n\n
Install the Datadog Agent on Ubuntu and collect host-level metrics.<\/p>\n\n\n\n
The Datadog Agent collects system-level checks such as CPU, disk, IO, memory, network, NTP, uptime, file handles, and load, depending on the platform. (Datadog<\/a>)<\/p>\n\n\n\n
Student Tasks<\/h2>\n\n\n\n
Task 1: Install Datadog Agent<\/h3>\n\n\n\n
Set these variables first:<\/p>\n\n\n
export<\/span> DD_API_KEY=\"<your_datadog_api_key>\"<\/span>\nexport<\/span> DD_SITE=\"<your_datadog_site>\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Install the Agent using the Datadog UI-generated command or the official Agent installation method from Datadog.<\/p>\n\n\n\n
After installation:<\/p>\n\n\n
sudo systemctl status datadog-agent\nsudo datadog-agent status\n<\/code><\/span><\/pre>\n\n\n
Task 2: Add Host Tags<\/h3>\n\n\n\n
Edit:<\/p>\n\n\n
sudo vi \/etc\/datadog-agent\/datadog.yaml\n<\/code><\/span><\/pre>\n\n\n
Add:<\/p>\n\n\n
tags<\/span>:\n  -<\/span> env<\/span>:lab<\/span>\n  -<\/span> team<\/span>:training<\/span>\n  -<\/span> project<\/span>:datadog-assignment<\/span>\n  -<\/span> service<\/span>:datadog-lab-shop<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Restart:<\/p>\n\n\n
sudo systemctl restart datadog-agent\n<\/code><\/span><\/pre>\n\n\n
Validate:<\/p>\n\n\n
sudo datadog-agent status\nsudo datadog-agent health\n<\/code><\/span><\/pre>\n\n\n
Task 3: Generate CPU and Memory Activity<\/h3>\n\n\n\n
Install stress tool:<\/p>\n\n\n
sudo apt install -y stress\n<\/code><\/span><\/pre>\n\n\n
Generate CPU load:<\/p>\n\n\n
stress --cpu 2 --timeout 120\n<\/code><\/span><\/pre>\n\n\n
Generate memory load:<\/p>\n\n\n
stress --vm 1 --vm-bytes 512M --timeout 120\n<\/code><\/span><\/pre>\n\n\n
Task 4: Validate in Datadog<\/h3>\n\n\n\n
Search these metrics:<\/p>\n\n\n
system<\/span>.cpu<\/span>.user<\/span>\nsystem<\/span>.cpu<\/span>.idle<\/span>\nsystem<\/span>.mem<\/span>.used<\/span>\nsystem<\/span>.disk<\/span>.used<\/span>\nsystem<\/span>.net<\/span>.bytes_sent<\/span>\ndatadog<\/span>.agent<\/span>.running<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Deliverables<\/h2>\n\n\n\n
Deliverable<\/th>Required Evidence<\/th><\/tr><\/thead>
Agent installed<\/td>Screenshot of host in Infrastructure List<\/td><\/tr>
Tags applied<\/td>Screenshot showing env:lab<\/code> and service:datadog-lab-shop<\/code><\/td><\/tr>
CPU test<\/td>Screenshot of CPU spike<\/td><\/tr>
Memory test<\/td>Screenshot of memory usage<\/td><\/tr>
Agent status<\/td>Paste output of sudo datadog-agent status<\/code> summary<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Grading Rubric<\/h2>\n\n\n\n
Criteria<\/th>Marks<\/th><\/tr><\/thead>
Agent installed correctly<\/td>20<\/td><\/tr>
Host tags added correctly<\/td>20<\/td><\/tr>
CPU\/memory metrics visible<\/td>25<\/td><\/tr>
Student explains top 5 Linux metrics<\/td>20<\/td><\/tr>
Screenshots and command evidence<\/td>15<\/td><\/tr>
Total<\/td>100<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Part 4: Assignment 2 \u2014 Log Monitoring<\/h1>\n\n\n\n
Objective<\/h2>\n\n\n\n
Enable Datadog log collection and collect Python application logs.<\/p>\n\n\n\n
Datadog logs can be collected through the Agent from files, TCP\/UDP, journald, and Windows channels. For custom log files, a config is placed under the Agent conf.d<\/code> directory. (Datadog<\/a>)<\/p>\n\n\n\n
Student Tasks<\/h2>\n\n\n\n
Task 1: Enable Logs in Agent<\/h3>\n\n\n\n
Edit:<\/p>\n\n\n
sudo vi \/etc\/datadog-agent\/datadog.yaml\n<\/code><\/span><\/pre>\n\n\n
Set:<\/p>\n\n\n
logs_enabled: true<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Restart:<\/p>\n\n\n
sudo systemctl restart datadog-agent\n<\/code><\/span><\/pre>\n\n\n
Task 2: Add App Log Collection<\/h3>\n\n\n\n
Create config directory:<\/p>\n\n\n
sudo mkdir -p \/etc\/datadog-agent\/conf.d\/datadog-lab-shop.d\n<\/code><\/span><\/pre>\n\n\n
Copy config:<\/p>\n\n\n
sudo cp \/opt\/datadog-lab-shop\/datadog\/lab-shop-logs.yaml \\\n  \/etc\/datadog-agent\/conf.d\/datadog-lab-shop.d\/conf.yaml\n<\/code><\/span><\/pre>\n\n\n
Restart:<\/p>\n\n\n
sudo systemctl restart datadog-agent\n<\/code><\/span><\/pre>\n\n\n
Validate:<\/p>\n\n\n
sudo datadog-agent status\nsudo datadog-agent configcheck\n<\/code><\/span><\/pre>\n\n\n
Task 3: Generate App Logs<\/h3>\n\n\n
curl http:\/\/localhost:8000\/<\/span>\ncurl http:\/\/localhost:8000\/api\/products<\/span>\ncurl http:\/\/localhost:8000\/api\/slow<\/span>\ncurl http:\/\/localhost:8000\/api\/error<\/span>\ncurl -X POST http:\/\/localhost:8000\/api\/checkout<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Check local logs:<\/p>\n\n\n
tail -f \/var<\/span>\/log\/datadog-lab-shop\/app.log\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Check what Agent is reading:<\/p>\n\n\n
sudo datadog-agent stream-logs\n<\/code><\/span><\/pre>\n\n\n
Task 4: Query Logs in Datadog<\/h3>\n\n\n\n
Use Log Explorer queries:<\/p>\n\n\n
service<\/span>:datadog-lab-shop<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n
service<\/span>:datadog-lab-shop<\/span> status<\/span>:error<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n
service:datadog-lab-shop @path:\/api\/<\/span>error\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n
service:datadog-lab-shop @level:ERROR\n<\/code><\/span><\/pre>\n\n\n
Task 5: Create Log Facets<\/h3>\n\n\n\n
Create facets for:<\/p>\n\n\n
level\npath\nenv\nversion\npayment_provider\ndelay_seconds\n<\/code><\/span><\/pre>\n\n\n
Deliverables<\/h2>\n\n\n\n
Deliverable<\/th>Required Evidence<\/th><\/tr><\/thead>
Log collection enabled<\/td>Screenshot of Logs Agent in datadog-agent status<\/code><\/td><\/tr>
App logs visible<\/td>Screenshot from Log Explorer<\/td><\/tr>
Error logs visible<\/td>Query screenshot for @level:ERROR<\/code><\/td><\/tr>
Facets created<\/td>Screenshot of facets<\/td><\/tr>
Explanation<\/td>Explain difference between ingested and indexed logs<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Grading Rubric<\/h2>\n\n\n\n
Criteria<\/th>Marks<\/th><\/tr><\/thead>
Log collection enabled<\/td>20<\/td><\/tr>
App log file collected<\/td>25<\/td><\/tr>
Log queries demonstrated<\/td>20<\/td><\/tr>
Facets created correctly<\/td>20<\/td><\/tr>
Troubleshooting explanation<\/td>15<\/td><\/tr>
Total<\/td>100<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Part 5: Assignment 3 \u2014 APM<\/h1>\n\n\n\n
Objective<\/h2>\n\n\n\n
Instrument the Python Flask application with Datadog APM and generate traces.<\/p>\n\n\n\n
Datadog APM provides visibility into application performance, request traces, errors, latency, services, and dependencies. (Datadog<\/a>)<\/p>\n\n\n\n
Student Tasks<\/h2>\n\n\n\n
Task 1: Enable APM in Agent<\/h3>\n\n\n\n
Edit:<\/p>\n\n\n
sudo vi \/etc\/datadog-agent\/datadog.yaml\n<\/code><\/span><\/pre>\n\n\n
Ensure:<\/p>\n\n\n
apm_config:\n  enabled: true<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Restart:<\/p>\n\n\n
sudo systemctl restart datadog-agent\n<\/code><\/span><\/pre>\n\n\n
Validate:<\/p>\n\n\n
sudo datadog-agent status\n<\/code><\/span><\/pre>\n\n\n
Look for APM \/ Trace Agent section.<\/p>\n\n\n\n
Task 2: Run App with ddtrace-run<\/code><\/h3>\n\n\n\n
Manual run:<\/p>\n\n\n
cd \/opt\/datadog-lab-shop\nsource venv\/bin\/activate\n\nDD_ENV=lab \\\nDD_SERVICE=datadog-lab-shop \\\nDD_VERSION=1.0<\/span>.0<\/span> \\\nDD_AGENT_HOST=127.0<\/span>.0<\/span>.1<\/span> \\\nDD_LOGS_INJECTION=true<\/span> \\\nddtrace-run gunicorn -b 0.0<\/span>.0<\/span>.0<\/span>:8000<\/span> app:app\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Task 3: Check Python Tracer Config<\/h3>\n\n\n
source \/opt\/datadog-lab-shop\/venv\/bin\/activate\nddtrace-run --info\n<\/code><\/span><\/pre>\n\n\n
Task 4: Generate Traces<\/h3>\n\n\n
for<\/span> i in<\/span> {1.<\/span>.50<\/span>}; do<\/span>\n  curl -s http:\/\/localhost:8000\/health >\/dev\/null<\/span>\n  curl -s http:\/\/localhost:8000\/api\/products >\/dev\/null<\/span>\n  curl -s http:\/\/localhost:8000\/api\/slow >\/dev\/null<\/span>\n  curl -s http:\/\/localhost:8000\/api\/error >\/dev\/null<\/span>\ndone\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Task 5: Validate in Datadog<\/h3>\n\n\n\n
Go to:<\/p>\n\n\n
APM \u2192 Services \u2192 datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
Check:<\/p>\n\n\n
Requests\nErrors\nLatency\np95 latency\nResources\nTraces\nService Map<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Deliverables<\/h2>\n\n\n\n
Deliverable<\/th>Required Evidence<\/th><\/tr><\/thead>
APM enabled<\/td>Screenshot of APM Agent status<\/td><\/tr>
Service visible<\/td>Screenshot of APM service page<\/td><\/tr>
Traces visible<\/td>Screenshot of a trace<\/td><\/tr>
Error trace visible<\/td>Screenshot of \/api\/error<\/code> trace<\/td><\/tr>
Slow trace visible<\/td>Screenshot of \/api\/slow<\/code> trace<\/td><\/tr>
Explanation<\/td>Explain trace, span, service, resource<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Grading Rubric<\/h2>\n\n\n\n
Criteria<\/th>Marks<\/th><\/tr><\/thead>
APM Agent enabled<\/td>20<\/td><\/tr>
Python app instrumented<\/td>25<\/td><\/tr>
Traces visible<\/td>25<\/td><\/tr>
Errors and latency demonstrated<\/td>20<\/td><\/tr>
Concept explanation<\/td>10<\/td><\/tr>
Total<\/td>100<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Part 6: Assignment 4 \u2014 RUM<\/h1>\n\n\n\n
Objective<\/h2>\n\n\n\n
Enable Real User Monitoring for the frontend page of the Flask app.<\/p>\n\n\n\n
RUM monitors real user activity and experience in web and mobile applications, including performance, frontend errors, user actions, network requests, and resources. (Datadog<\/a>)<\/p>\n\n\n\n
Important Note<\/h2>\n\n\n\n
RUM is not installed on the Linux server like the Agent. RUM is added to the browser frontend<\/strong> using the Datadog Browser SDK.<\/p>\n\n\n\n
Student Tasks<\/h2>\n\n\n\n
Task 1: Create RUM Application<\/h3>\n\n\n\n
In Datadog:<\/p>\n\n\n
Digital Experience \u2192 RUM Applications \u2192 New<\/span> Application\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Choose:<\/p>\n\n\n
Application type: Browser\nName: datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
Copy:<\/p>\n\n\n
Application ID\nClient Token\nDatadog Site\n<\/code><\/span><\/pre>\n\n\n
Task 2: Add RUM Values to systemd Service<\/h3>\n\n\n\n
Edit:<\/p>\n\n\n
sudo vi \/etc\/systemd\/system\/datadog-lab-shop.service\n<\/code><\/span><\/pre>\n\n\n
Add:<\/p>\n\n\n
Environment=DD_RUM_APPLICATION_ID=<your_rum_application_id<\/span>><\/span>\nEnvironment=DD_RUM_CLIENT_TOKEN=<your_rum_client_token<\/span>><\/span>\nEnvironment=DD_SITE=<your_datadog_site<\/span>><\/span>\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Reload and restart:<\/p>\n\n\n
sudo systemctl daemon-reload\nsudo systemctl restart datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
Task 3: Open Browser and Generate RUM Events<\/h3>\n\n\n\n
Open:<\/p>\n\n\n
http:\/\/<server-public-ip<\/span>><\/span>:8000\/\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Click all buttons:<\/p>\n\n\n
Products\nAdd to Cart\nCheckout\nSlow API\nError<\/span> API\nRandom API\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Task 4: Validate in Datadog<\/h3>\n\n\n\n
Go to:<\/p>\n\n\n
Digital Experience \u2192 RUM Explorer\n<\/code><\/span><\/pre>\n\n\n
Check:<\/p>\n\n\n
Views\nActions\nResources\nErrors\nSessions\nUser Journey\n<\/code><\/span><\/pre>\n\n\n
Deliverables<\/h2>\n\n\n\n
Deliverable<\/th>Required Evidence<\/th><\/tr><\/thead>
RUM app created<\/td>Screenshot of RUM application<\/td><\/tr>
Browser page monitored<\/td>Screenshot of RUM session<\/td><\/tr>
User actions visible<\/td>Screenshot showing button clicks<\/td><\/tr>
Resource timing visible<\/td>Screenshot of API resource calls<\/td><\/tr>
Explanation<\/td>Difference between RUM and APM<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Grading Rubric<\/h2>\n\n\n\n
Criteria<\/th>Marks<\/th><\/tr><\/thead>
RUM application created<\/td>20<\/td><\/tr>
Browser SDK configured<\/td>25<\/td><\/tr>
Sessions\/actions\/resources visible<\/td>30<\/td><\/tr>
RUM\/APM difference explained<\/td>15<\/td><\/tr>
Screenshot quality<\/td>10<\/td><\/tr>
Total<\/td>100<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Part 7: Assignment 5 \u2014 Synthetic Monitoring<\/h1>\n\n\n\n
Objective<\/h2>\n\n\n\n
Create synthetic tests to monitor application uptime and user behavior.<\/p>\n\n\n\n
Datadog Synthetic Monitoring can create API, browser, mobile, and network tests to simulate user flows and check application availability and performance. (Datadog<\/a>)<\/p>\n\n\n\n
Student Tasks<\/h2>\n\n\n\n
Task 1: Create API Test for Health Endpoint<\/h3>\n\n\n\n
Go to:<\/p>\n\n\n
Digital Experience \u2192 Tests \u2192 New<\/span> Test \u2192 API Test\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Configure:<\/p>\n\n\n\n
Field<\/th>Value<\/th><\/tr><\/thead>
Name<\/td>LAB – Health API Test<\/td><\/tr>
URL<\/td>http:\/\/<server-public-ip>:8000\/health<\/code><\/td><\/tr>
Method<\/td>GET<\/td><\/tr>
Locations<\/td>Choose 1 or 2 public locations<\/td><\/tr>
Frequency<\/td>Every 5 or 10 minutes for lab<\/td><\/tr>
Assertion 1<\/td>Status code is 200<\/td><\/tr>
Assertion 2<\/td>Response body contains ok<\/code><\/td><\/tr>
Assertion 3<\/td>Response time less than 1000 ms<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Datadog API tests monitor endpoints and can alert on latency, status code, headers, body content, and other conditions. (Datadog<\/a>)<\/p>\n\n\n\n
Task 2: Create API Test for Error Endpoint<\/h3>\n\n\n\n
Create another API test:<\/p>\n\n\n
http:\/\/<server-public-ip<\/span>><\/span>:8000\/api\/error\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Expected assertion:<\/p>\n\n\n
Status code is 500\n<\/code><\/span><\/pre>\n\n\n
Purpose: teach expected failure vs unexpected failure.<\/p>\n\n\n\n
Task 3: Create Browser Test<\/h3>\n\n\n\n
Go to:<\/p>\n\n\n
Digital Experience \u2192 Tests \u2192 New<\/span> Test \u2192 Browser Test\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Flow:<\/p>\n\n\n
Open homepage\nClick Products\nValidate response contains Datadog T-Shirt\nClick Add to Cart\nValidate response contains Item added\nClick Checkout\nValidate response changes\n<\/code><\/span><\/pre>\n\n\n
Task 4: Trigger Failure<\/h3>\n\n\n\n
Stop the app:<\/p>\n\n\n
sudo systemctl stop datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
Wait for synthetic test failure.<\/p>\n\n\n\n
Restart:<\/p>\n\n\n
sudo systemctl start datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
Deliverables<\/h2>\n\n\n\n
Deliverable<\/th>Required Evidence<\/th><\/tr><\/thead>
API test created<\/td>Screenshot of test config<\/td><\/tr>
Browser test created<\/td>Screenshot of browser steps<\/td><\/tr>
Failure captured<\/td>Screenshot of failed synthetic result<\/td><\/tr>
Recovery captured<\/td>Screenshot of recovered result<\/td><\/tr>
Explanation<\/td>API test vs browser test<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Grading Rubric<\/h2>\n\n\n\n
Criteria<\/th>Marks<\/th><\/tr><\/thead>
API test configured correctly<\/td>25<\/td><\/tr>
Browser test configured correctly<\/td>30<\/td><\/tr>
Failure\/recovery demonstrated<\/td>25<\/td><\/tr>
Explanation of use cases<\/td>10<\/td><\/tr>
Clean test naming\/tags<\/td>10<\/td><\/tr>
Total<\/td>100<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Part 8: Assignment 6 \u2014 Alerts \/ Monitors<\/h1>\n\n\n\n
Objective<\/h2>\n\n\n\n
Create meaningful Datadog monitors across infrastructure, logs, APM, RUM, and Synthetics.<\/p>\n\n\n\n
Datadog supports many monitor types, including metric, host, log, APM, RUM, synthetic, anomaly, forecast, outlier, composite, SLO, and more. (Datadog<\/a>)<\/p>\n\n\n\n
Student Tasks<\/h2>\n\n\n\n
Create at least six monitors.<\/p>\n\n\n\n
\n\n\n\n
Monitor 1: Host CPU Monitor<\/h2>\n\n\n\n
Type:<\/p>\n\n\n
Metric Monitor\n<\/code><\/span><\/pre>\n\n\n
Query concept:<\/p>\n\n\n
avg:system.cpu.user{env:lab,service:datadog-lab-shop} by {host}\n<\/code><\/span><\/pre>\n\n\n
Alert condition:<\/p>\n\n\n
Above 80 for 5 minutes\n<\/code><\/span><\/pre>\n\n\n
Message:<\/p>\n\n\n
High CPU detected on {{host.name}} for datadog-lab-shop lab.\nCheck running processes and recent load test.\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Monitor 2: Host No Data Monitor<\/h2>\n\n\n\n
Type:<\/p>\n\n\n
Host Monitor or Metric Monitor\n<\/code><\/span><\/pre>\n\n\n
Purpose:<\/p>\n\n\n
Alert if Agent stops reporting.\n<\/code><\/span><\/pre>\n\n\n
Use metric:<\/p>\n\n\n
datadog<\/span>.agent<\/span>.running<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
No-data alerts are useful when data is expected continuously, such as Agent host metrics. Datadog monitor configuration supports missing-data behavior such as No Data, evaluate as zero, or last known status depending on monitor type. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
Monitor 3: Log Error Monitor<\/h2>\n\n\n\n
Type:<\/p>\n\n\n
Log Monitor\n<\/code><\/span><\/pre>\n\n\n
Query:<\/p>\n\n\n
service:datadog-lab-shop @level:ERROR\n<\/code><\/span><\/pre>\n\n\n
Condition:<\/p>\n\n\n
More than 5 errors in 5 minutes\n<\/code><\/span><\/pre>\n\n\n
Message:<\/p>\n\n\n
Application error spike detected for datadog-lab-shop.\nCheck Log Explorer and APM traces for \/api\/error or checkout failures.\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Monitor 4: APM Error Rate Monitor<\/h2>\n\n\n\n
Type:<\/p>\n\n\n
APM Monitor\n<\/code><\/span><\/pre>\n\n\n
Target:<\/p>\n\n\n
service<\/span>:datadog-lab-shop<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Condition:<\/p>\n\n\n
Error<\/span> rate above 5<\/span>% for<\/span> 5<\/span> minutes\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Monitor 5: APM Latency Monitor<\/h2>\n\n\n\n
Type:<\/p>\n\n\n
APM Monitor\n<\/code><\/span><\/pre>\n\n\n
Condition:<\/p>\n\n\n
p95 latency above 1 second for 5 minutes\n<\/code><\/span><\/pre>\n\n\n
Use \/api\/slow<\/code> to trigger.<\/p>\n\n\n\n
\n\n\n\n
Monitor 6: Synthetic Test Monitor<\/h2>\n\n\n\n
Type:<\/p>\n\n\n
Synthetic Monitor\n<\/code><\/span><\/pre>\n\n\n
Target:<\/p>\n\n\n
LAB - Health API Test\n<\/code><\/span><\/pre>\n\n\n
Condition:<\/p>\n\n\n
Alert when test fails from<\/span> selected locations\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Monitor 7: RUM Error Monitor<\/h2>\n\n\n\n
Type:<\/p>\n\n\n
RUM Monitor\n<\/code><\/span><\/pre>\n\n\n
Condition:<\/p>\n\n\n
Frontend error count > 0 or resource error rate above threshold\n<\/code><\/span><\/pre>\n\n\n
Datadog has a dedicated RUM monitor type for monitoring real user performance and errors. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
Deliverables<\/h2>\n\n\n\n
Deliverable<\/th>Required Evidence<\/th><\/tr><\/thead>
CPU monitor<\/td>Screenshot<\/td><\/tr>
No-data monitor<\/td>Screenshot<\/td><\/tr>
Log error monitor<\/td>Screenshot<\/td><\/tr>
APM error monitor<\/td>Screenshot<\/td><\/tr>
APM latency monitor<\/td>Screenshot<\/td><\/tr>
Synthetic monitor<\/td>Screenshot<\/td><\/tr>
RUM monitor<\/td>Screenshot<\/td><\/tr>
Alert test<\/td>Screenshot of at least one triggered alert<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Grading Rubric<\/h2>\n\n\n\n
Criteria<\/th>Marks<\/th><\/tr><\/thead>
Correct monitor types<\/td>25<\/td><\/tr>
Query correctness<\/td>25<\/td><\/tr>
Alert thresholds reasonable<\/td>15<\/td><\/tr>
Notification messages useful<\/td>15<\/td><\/tr>
Trigger\/recovery demonstrated<\/td>20<\/td><\/tr>
Total<\/td>100<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Part 9: Assignment 7 \u2014 Dashboard<\/h1>\n\n\n\n
Objective<\/h2>\n\n\n\n
Create one unified Datadog dashboard for the Python application.<\/p>\n\n\n\n
Datadog dashboards provide visibility across products and can include widgets, template variables, permissions, and troubleshooting context. (Datadog<\/a>)<\/p>\n\n\n\n
Dashboard Name<\/h2>\n\n\n
Datadog Lab Shop - Full Stack Observability Dashboard\n<\/code><\/span><\/pre>\n\n\n
Required Template Variables<\/h2>\n\n\n\n
Create dashboard template variables:<\/p>\n\n\n\n
Variable<\/th>Source<\/th><\/tr><\/thead>
env<\/code><\/td>env<\/code> tag<\/td><\/tr>
service<\/code><\/td>service<\/code> tag<\/td><\/tr>
host<\/code><\/td>host<\/code> tag<\/td><\/tr>
version<\/code><\/td>version<\/code> tag<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Template variables dynamically filter widgets by tags, attributes, and facets. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
Required Dashboard Widgets<\/h2>\n\n\n\n
Section<\/th>Widget<\/th>Query \/ Source<\/th><\/tr><\/thead>
Infrastructure<\/td>CPU Usage<\/td>system.cpu.user{env:$env}<\/code><\/td><\/tr>
Infrastructure<\/td>Memory Used<\/td>system.mem.used{env:$env}<\/code><\/td><\/tr>
Infrastructure<\/td>Disk Used<\/td>system.disk.used{env:$env}<\/code><\/td><\/tr>
Infrastructure<\/td>Network Traffic<\/td>system.net.bytes_sent<\/code>, system.net.bytes_rcvd<\/code><\/td><\/tr>
Agent<\/td>Agent Running<\/td>datadog.agent.running{env:$env}<\/code><\/td><\/tr>
Logs<\/td>Error Logs Count<\/td>service:$service @level:ERROR<\/code><\/td><\/tr>
Logs<\/td>Recent Error Logs<\/td>Log stream widget<\/td><\/tr>
APM<\/td>Request Rate<\/td>APM service metrics<\/td><\/tr>
APM<\/td>Error Rate<\/td>APM service metrics<\/td><\/tr>
APM<\/td>p95 Latency<\/td>APM service metrics<\/td><\/tr>
APM<\/td>Top Resources<\/td>\/health<\/code>, \/api\/products<\/code>, \/api\/slow<\/code>, \/api\/error<\/code><\/td><\/tr>
RUM<\/td>Sessions<\/td>RUM data<\/td><\/tr>
RUM<\/td>User Actions<\/td>Button clicks<\/td><\/tr>
RUM<\/td>Resource Timing<\/td>API resource calls<\/td><\/tr>
Synthetics<\/td>Health Test Status<\/td>Synthetic test widget<\/td><\/tr>
Alerts<\/td>Active Monitors<\/td>Monitor summary<\/td><\/tr>
Business<\/td>Checkout Attempts<\/td>datadog_lab_shop.checkout.attempt<\/code><\/td><\/tr>
Business<\/td>Checkout Failures<\/td>datadog_lab_shop.checkout.attempt{status:failed}<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Datadog supports custom widgets and cloning out-of-the-box dashboards as a starting point. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
Deliverables<\/h2>\n\n\n\n
Deliverable<\/th>Required Evidence<\/th><\/tr><\/thead>
Dashboard created<\/td>Screenshot<\/td><\/tr>
Template variables<\/td>Screenshot<\/td><\/tr>
Infra widgets<\/td>Screenshot<\/td><\/tr>
Log widgets<\/td>Screenshot<\/td><\/tr>
APM widgets<\/td>Screenshot<\/td><\/tr>
RUM widgets<\/td>Screenshot<\/td><\/tr>
Synthetic widgets<\/td>Screenshot<\/td><\/tr>
Alert widgets<\/td>Screenshot<\/td><\/tr>
Explanation<\/td>5-minute dashboard walkthrough<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Grading Rubric<\/h2>\n\n\n\n
Criteria<\/th>Marks<\/th><\/tr><\/thead>
Dashboard structure<\/td>20<\/td><\/tr>
Template variables<\/td>15<\/td><\/tr>
Infra\/log\/APM\/RUM\/synthetic coverage<\/td>35<\/td><\/tr>
Useful visual design<\/td>15<\/td><\/tr>
Explanation and troubleshooting flow<\/td>15<\/td><\/tr>
Total<\/td>100<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Part 10: Final Project \u2014 Complete Datadog Observability Implementation<\/h1>\n\n\n\n
Project Title<\/h2>\n\n\n
End-to-End Datadog Observability for Python Flask Application on Ubuntu\n<\/code><\/span><\/pre>\n\n\n
Final Project Scenario<\/h2>\n\n\n\n
You are hired as a DevOps\/SRE engineer. Your company has a Python Flask application running on Ubuntu. The engineering team wants complete observability using Datadog.<\/p>\n\n\n\n
You must implement:<\/p>\n\n\n
Infrastructure monitoring\nLog monitoring\nAPM tracing\nRUM browser monitoring\nSynthetic monitoring\nAlerting\nDashboarding\nTroubleshooting documentation\nGitHub repository\nFinal<\/span> report\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Final Project Step-by-Step<\/h1>\n\n\n\n
Step 1: Prepare GitHub Repository<\/h2>\n\n\n\n
Repo name:<\/p>\n\n\n
datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
Required branches:<\/p>\n\n\n
main\nfeature\/datadog-observability\n<\/code><\/span><\/pre>\n\n\n
Required folders:<\/p>\n\n\n
app\ndatadog\nsystemd\nload-test\nscreenshots\ndocs\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Step 2: Deploy App on Ubuntu<\/h2>\n\n\n
sudo mkdir -p \/opt\/datadog-lab-shop\nsudo chown ubuntu:ubuntu \/opt\/datadog-lab-shop\n\ncd \/opt\/datadog-lab-shop\ngit clone <your-github-repo-url<\/span>><\/span> .\n\npython3 -m venv venv\nsource venv\/bin\/activate\npip install -r requirements.txt\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Install systemd service:<\/p>\n\n\n
sudo cp systemd\/datadog-lab-shop.service \/etc\/systemd\/system\/datadog-lab-shop.service\nsudo systemctl daemon-reload\nsudo systemctl enable datadog-lab-shop\nsudo systemctl start datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
Validate:<\/p>\n\n\n
sudo systemctl status datadog-lab-shop\ncurl http:\/\/localhost:8000\/health<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Step 3: Install and Configure Datadog Agent<\/h2>\n\n\n
sudo datadog-agent status\nsudo datadog-agent health\n<\/code><\/span><\/pre>\n\n\n
Configure tags:<\/p>\n\n\n
tags<\/span>:\n  -<\/span> env<\/span>:lab<\/span>\n  -<\/span> service<\/span>:datadog-lab-shop<\/span>\n  -<\/span> version<\/span>:1.0.0<\/span>\n  -<\/span> team<\/span>:training<\/span>\n  -<\/span> project<\/span>:datadog-final-project<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Restart:<\/p>\n\n\n
sudo systemctl restart datadog-agent\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Step 4: Enable Logs<\/h2>\n\n\n
logs_enabled: true<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Copy log config:<\/p>\n\n\n
sudo mkdir -p \/etc\/datadog-agent\/conf.d\/datadog-lab-shop.d\n\nsudo cp \/opt\/datadog-lab-shop\/datadog\/lab-shop-logs.yaml \\\n  \/etc\/datadog-agent\/conf.d\/datadog-lab-shop.d\/conf.yaml\n\nsudo systemctl restart datadog-agent\n<\/code><\/span><\/pre>\n\n\n
Validate:<\/p>\n\n\n
sudo datadog-agent status\nsudo datadog-agent stream-logs\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Step 5: Enable APM<\/h2>\n\n\n\n
Confirm in datadog.yaml<\/code>:<\/p>\n\n\n
apm_config:\n  enabled: true<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Restart:<\/p>\n\n\n
sudo systemctl restart datadog-agent\nsudo systemctl restart datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
Validate:<\/p>\n\n\n
curl http:\/\/localhost:8000\/api\/products<\/span>\ncurl http:\/\/localhost:8000\/api\/slow<\/span>\ncurl http:\/\/localhost:8000\/api\/error<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Step 6: Enable RUM<\/h2>\n\n\n\n
Create RUM app in Datadog and add environment variables to the systemd service:<\/p>\n\n\n
Environment=DD_RUM_APPLICATION_ID=<rum_application_id<\/span>><\/span>\nEnvironment=DD_RUM_CLIENT_TOKEN=<rum_client_token<\/span>><\/span>\nEnvironment=DD_SITE=<datadog_site<\/span>><\/span>\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
Restart:<\/p>\n\n\n
sudo systemctl daemon-reload\nsudo systemctl restart datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
Open in browser:<\/p>\n\n\n
http:\/\/<server-public-ip<\/span>><\/span>:8000\/\n<\/code><\/span>Code language:<\/span> HTML, XML<\/span> (<\/span>xml<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Step 7: Create Synthetic Tests<\/h2>\n\n\n\n
Create:<\/p>\n\n\n\n
Test<\/th>Type<\/th>Endpoint<\/th><\/tr><\/thead>
Health API Test<\/td>API<\/td>\/health<\/code><\/td><\/tr>
Products API Test<\/td>API<\/td>\/api\/products<\/code><\/td><\/tr>
Homepage Browser Test<\/td>Browser<\/td>\/<\/code><\/td><\/tr>
Checkout Journey Test<\/td>Browser<\/td>\/ \u2192 cart \u2192 checkout<\/code><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Synthetic tests can run manually, on schedule, or from CI\/CD pipelines, and can use public or private locations. (Datadog<\/a>)<\/p>\n\n\n\n
\n\n\n\n
Step 8: Create Monitors<\/h2>\n\n\n\n
Required final project monitors:<\/p>\n\n\n\n
Monitor<\/th>Type<\/th>Purpose<\/th><\/tr><\/thead>
Host CPU High<\/td>Metric<\/td>Infra health<\/td><\/tr>
Host Memory High<\/td>Metric<\/td>Infra health<\/td><\/tr>
Agent No Data<\/td>Metric\/Host<\/td>Agent failure<\/td><\/tr>
Application Error Logs<\/td>Log<\/td>App error spike<\/td><\/tr>
APM Error Rate<\/td>APM<\/td>Backend error rate<\/td><\/tr>
APM p95 Latency<\/td>APM<\/td>Slow API detection<\/td><\/tr>
Synthetic Health Failure<\/td>Synthetic<\/td>Uptime failure<\/td><\/tr>
RUM Error<\/td>RUM<\/td>Frontend error detection<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Step 9: Create Final Dashboard<\/h2>\n\n\n\n
Dashboard must include:<\/p>\n\n\n
Host health\nAgent health\nApplication logs\nError<\/span> logs\nAPM latency\nAPM error rate\nAPM request rate\nRUM sessions\nRUM actions\nSynthetic uptime\nMonitor status\nCustom business metrics\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Step 10: Create Final Report<\/h2>\n\n\n\n
Students must submit a report with this structure:<\/p>\n\n\n
1. Project Overview\n2. Architecture Diagram\n3. Ubuntu Server Setup\n4. Datadog Agent Setup\n5. Infrastructure Monitoring\n6. Log Monitoring\n7. APM Setup\n8. RUM Setup\n9. Synthetic Monitoring\n10. Alerts \/ Monitors\n11. Dashboard\n12. Test Scenarios\n13. Troubleshooting\n14. Screenshots\n15. Lessons Learned\n16. GitHub Repository\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Final Project Test Scenarios<\/h1>\n\n\n\n
Students must demonstrate these scenarios.<\/p>\n\n\n\n
Scenario 1: CPU Spike<\/h2>\n\n\n
stress --cpu 2 --timeout 120\n<\/code><\/span><\/pre>\n\n\n
Expected result:<\/p>\n\n\n
CPU metric increases\nCPU monitor may trigger\nDashboard reflects spike\n<\/code><\/span><\/pre>\n\n\n
Scenario 2: Application Error<\/h2>\n\n\n
curl http:\/\/localhost:8000\/api\/error<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Expected result:<\/p>\n\n\n
Error<\/span> log appears\nAPM error trace appears\nLog\/APM monitor may trigger\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Scenario 3: Slow API<\/h2>\n\n\n
curl http:\/\/localhost:8000\/api\/slow<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Expected result:<\/p>\n\n\n
APM latency increases\nSlow trace visible\nLatency monitor may trigger\n<\/code><\/span><\/pre>\n\n\n
Scenario 4: Checkout Failure<\/h2>\n\n\n
for<\/span> i in<\/span> {1.<\/span>.20<\/span>}; do<\/span>\n  curl -s -X POST http:\/\/localhost:8000\/api\/checkout<\/span>\ndone\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Expected result:<\/p>\n\n\n
Some checkout failures generated\nCustom metric appears\nError<\/span> logs created\nAPM error traces created\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Scenario 5: Synthetic Failure<\/h2>\n\n\n
sudo systemctl stop datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
Expected result:<\/p>\n\n\n
Synthetic API test fails\nSynthetic monitor alerts\nDashboard shows test failure\n<\/code><\/span><\/pre>\n\n\n
Recover:<\/p>\n\n\n
sudo systemctl start datadog-lab-shop\n<\/code><\/span><\/pre>\n\n\n
Scenario 6: RUM User Journey<\/h2>\n\n\n\n
Open browser and click:<\/p>\n\n\n
Products\nAdd to Cart\nCheckout\nSlow API\nError<\/span> API\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Expected result:<\/p>\n\n\n
RUM session visible\nActions visible\nResource requests visible\nFrontend\/backend relationship visible\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Final Project Deliverables<\/h1>\n\n\n\n
Deliverable<\/th>Description<\/th><\/tr><\/thead>
GitHub repository<\/td>Full source code and configs<\/td><\/tr>
README<\/td>Setup instructions<\/td><\/tr>
Screenshots<\/td>Infra, logs, APM, RUM, synthetics, monitors, dashboard<\/td><\/tr>
Final report<\/td>Complete project explanation<\/td><\/tr>
Architecture diagram<\/td>Mermaid or image<\/td><\/tr>
Troubleshooting guide<\/td>Common issues and fixes<\/td><\/tr>
Demo video optional<\/td>5\u201310 minute walkthrough<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Final Project Grading Rubric<\/h1>\n\n\n\n
Category<\/th>Marks<\/th><\/tr><\/thead>
GitHub project quality<\/td>10<\/td><\/tr>
Ubuntu app deployment<\/td>10<\/td><\/tr>
Datadog Agent and infra monitoring<\/td>10<\/td><\/tr>
Log collection and queries<\/td>10<\/td><\/tr>
APM traces and service visibility<\/td>15<\/td><\/tr>
RUM implementation<\/td>10<\/td><\/tr>
Synthetic tests<\/td>10<\/td><\/tr>
Monitors and alert quality<\/td>10<\/td><\/tr>
Dashboard quality<\/td>10<\/td><\/tr>
Final report and explanation<\/td>5<\/td><\/tr>
Total<\/td>100<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\n\n\n\n
Student README Template<\/h1>\n\n\n\n
Students should include this in their GitHub repo.<\/p>\n\n\n
# Datadog Lab Shop<\/span>\n\n## Project Overview<\/span>\n\nThis project demonstrates end-to-end Datadog observability for<\/span> a Python Flask application running on Ubuntu Linux.\n\n## Features Implemented<\/span>\n\n- Infrastructure Monitoring\n- Log Monitoring\n- APM Tracing\n- Real User Monitoring\n- Synthetic Monitoring\n- Alerts \/ Monitors\n- Dashboard\n- Custom Metrics\n\n## Application Endpoints<\/span>\n\n| Endpoint | Purpose |\n|---|---|\n| `\/` | Frontend page |\n| `\/health` | Health check |\n| `\/api\/products` | Product API |\n| `\/api\/cart` | Cart API |\n| `\/api\/checkout` | Checkout API |\n| `\/api\/slow` | Slow endpoint |\n| `\/api\/error` | Error endpoint |\n| `\/api\/random` | Random behavior endpoint |\n\n## Setup Steps<\/span>\n\n1.<\/span> Install Python dependencies.\n2.<\/span> Install Datadog Agent.\n3.<\/span> Enable logs.\n4.<\/span> Enable APM.\n5.<\/span> Configure RUM.\n6.<\/span> Create Synthetic tests.\n7.<\/span> Create monitors.\n8.<\/span> Create dashboard.\n\n## Validation<\/span>\n\nRun:\n\n```bash\ncurl http:\/\/localhost:8000\/health<\/span>\ncurl http:\/\/localhost:8000\/api\/products<\/span>\ncurl http:\/\/localhost:8000\/api\/error<\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Screenshots<\/h2>\n\n\n\n
Add screenshots under the screenshots\/<\/code> folder.<\/p>\n\n\n\n
Lessons Learned<\/h2>\n\n\n\n
Write what you learned about Datadog infrastructure monitoring, logs, APM, RUM, synthetics, alerts, and dashboards.<\/p>\n\n\n
\n---\n\n# Common Troubleshooting Section for Students<\/span>\n\n| Problem | Likely Cause | Fix |\n|---|---|---|\n| Host not visible | Agent not running, wrong API key, wrong site | `sudo datadog-agent status` |\n| Logs not visible | `logs_enabled` false<\/span> or<\/span> log config wrong | Check `datadog.yaml` and<\/span> `conf.d` |\n| APM not visible | App not started with `ddtrace-run` | Restart app with `ddtrace-run` |\n| RUM not visible | Wrong RUM app ID\/client token\/site | Check browser console and<\/span> env vars |\n| Synthetic test fails | App port blocked or<\/span> server private<\/span> | Open firewall or<\/span> use<\/span> private<\/span> location<\/span> |\n| Monitor<\/span> not<\/span> triggering<\/span> | Wrong<\/span> query<\/span> or<\/span> threshold<\/span> | Test<\/span> query<\/span> in<\/span> Metrics<\/span>\/Logs<\/span>\/APM<\/span> explorer<\/span> |\n| Dashboard<\/span> empty<\/span> | Wrong<\/span> template<\/span> variable<\/span> | Clear<\/span> variables<\/span> and<\/span> check<\/span> tags<\/span> |\n| No<\/span> custom<\/span> metrics<\/span> | DogStatsD<\/span> not<\/span> reachable<\/span> | Check<\/span> UDP<\/span> 8125 and<\/span> Agent<\/span> status<\/span> |\n\n---\n\n# Instructor<\/span> Notes<\/span>\n\n## Recommended<\/span> Lab<\/span> Duration<\/span>\n\n| Module<\/span> | Duration<\/span> |\n|---|---:|\n| Intro<\/span> + project<\/span> setup<\/span> | 1 hour<\/span> |\n| Infra<\/span> monitoring<\/span> | 1 hour<\/span> |\n| Logs<\/span> | 1.5 hours<\/span> |\n| APM<\/span> | 1.5 hours<\/span> |\n| RUM<\/span> | 1 hour<\/span> |\n| Synthetics<\/span> | 1 hour<\/span> |\n| Alerts<\/span> | 1.5 hours<\/span> |\n| Dashboard<\/span> | 1 hour<\/span> |\n| Final<\/span> project<\/span> | 3\u20135 hours<\/span> |\n\nTotal<\/span> recommended<\/span> duration<\/span>:\n\n```text<\/span>\n12 to<\/span> 15 hours<\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Recommended Team Size<\/h2>\n\n\n
1 to 3 students per team\n<\/code><\/span><\/pre>\n\n\n
Recommended Final Demo<\/h2>\n\n\n\n
Each student\/team should present:<\/p>\n\n\n
1. Application running\n2. Datadog host view\n3. Logs Explorer\n4. APM service page\n5. RUM session\n6. Synthetic test\n7. Alert trigger\n8. Dashboard\n9. Lessons learned\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Final Assignment Summary<\/h1>\n\n\n\n
Assignment<\/th>Final Skill<\/th><\/tr><\/thead>
Infra<\/td>Student can install Agent and monitor Linux<\/td><\/tr>
Logs<\/td>Student can collect and query app logs<\/td><\/tr>
APM<\/td>Student can trace Python app requests<\/td><\/tr>
RUM<\/td>Student can monitor browser user experience<\/td><\/tr>
Synthetics<\/td>Student can test uptime and user journeys<\/td><\/tr>
Alerts<\/td>Student can create actionable monitors<\/td><\/tr>
Dashboard<\/td>Student can build a full observability view<\/td><\/tr>
Final Project<\/td>Student can implement real Datadog observability end to end<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
This version turns the old \u201cDatadog assignment\u201d into a real, production-style, hands-on project. It gives students a working Python app, real failure scenarios, actual Datadog telemetry, screenshots to submit, and a final project they can show in interviews or internal assessments.<\/p>\n\n\n\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
Target Audience This lab is suitable for: DevOps engineers, SREs, cloud engineers, platform engineers, application engineers, monitoring engineers, and students learning Datadog from practical implementation. Final Outcome… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[25768],"tags":[],"class_list":["post-31255","post","type-post","status-publish","format-standard","hentry","category-datadog"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/31255","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=31255"}],"version-history":[{"count":7,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/31255\/revisions"}],"predecessor-version":[{"id":77132,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/31255\/revisions\/77132"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=31255"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=31255"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=31255"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
APM trace metrics commonly include hits, errors, latency, and Apdex. (Datadog<\/a>)<\/p>\n\n\n\n
Datadog documents ddtrace-run<\/code> as the standard command prefix for Python automatic instrumentation. (Datadog<\/a>)<\/p>\n\n\n\n
Datadog log collection on hosts requires enabling log collection in the Agent and defining a log configuration under conf.d<\/code>. (Datadog<\/a>)<\/p>\n\n\n\n
Datadog\u2019s RUM Browser SDK is used to monitor real user page performance, interactions, resources, and frontend errors. (Datadog<\/a>)<\/p>\n\n\n\n