Lab Manual: Datadog Logs Search, Filter, Sorting, Display, and Analysis

Hands-on Datadog Logs Explorer Lab using Ubuntu Linux Logs and Apache Logs

Lab Objective

By the end of this lab, students will be able to:

1. Open Datadog Log Explorer.
2. Search logs using free-text search.
3. Filter logs using source, service, host, status, and attributes.
4. Use Boolean operators such as AND, OR, and exclusion with -.
5. Search exact phrases using double quotes.
6. Use wildcard search.
7. Search Apache logs by HTTP status code, path, method, and client IP.
8. Search Ubuntu logs for SSH, sudo, cron, systemd, authentication, and errors.
9. Use facets to narrow down logs.
10. Open individual logs and inspect log attributes.
11. Add/remove columns in the log list.
12. Sort and display logs for troubleshooting.
13. Use Log Analytics to group logs by fields.
14. Use visualizations such as Top List, Timeseries, Table, Pie Chart, and Tree Map.
15. Use Log Patterns to identify repeated log messages.
16. Use Live Tail for near-real-time log checking.
17. Create Saved Views for repeatable troubleshooting.
18. Export/share log views for team collaboration.

1. Lab Environment

1.1 Assumptions

This lab assumes the following logs are already available in Datadog:

Ubuntu Linux logs
Apache access logs
Apache error logs

Common Ubuntu log examples:

/var/log/syslog
/var/log/auth.log
/var/log/kern.log
/var/log/dpkg.log
Code language: JavaScript (javascript)

Common Apache log examples:

/var/log/apache2/access.log
/var/log/apache2/error.log
Code language: JavaScript (javascript)

The exact Datadog fields may vary depending on your Datadog Agent configuration, log pipeline, parser, and integration setup.

1.2 Required Access

Students need access to:

Datadog account
Logs product
Log Explorer
Permission to search logs
Permission to create Saved Views, if allowed by your organization

Optional permissions:

Permission to create facets
Permission to create log-based metrics
Permission to create dashboards
Permission to create monitors

For this lab, students mainly need read/search access.

1.3 Expected Log Sources

Your logs may appear with sources like:

source:syslog
source:ubuntu
source:linux
source:apache
source:apache.access
source:apache.error
Code language: CSS (css)

Services may appear as:

service:system
service:ubuntu
service:apache
service:web
service:httpd
Code language: CSS (css)

Hosts may appear as:

host:ubuntu-demo
host:linux-demo
host:web-server-01
Code language: CSS (css)

Because Datadog setups differ, students should first discover the actual values in their own Log Explorer.

2. Datadog Logs Explorer Mental Model

Datadog Log Explorer is used for:

Search
Filter
Inspect
Display
Group
Visualize
Export
Save
Troubleshoot

Simple mental model:

Raw logs arrive in Datadog
        ↓
Datadog parses/enriches logs
        ↓
Students search and filter logs
        ↓
Students inspect individual log attributes
        ↓
Students group and visualize logs
        ↓
Students save views for repeatable troubleshooting

3. Important Datadog Log Terms

3.1 Log Event

A single log line or structured log entry.

Example Apache log event:

192.168.1.10 - - [23/Jun/2026:10:10:00 +0000] "GET /index.html HTTP/1.1" 200 1024
Code language: JavaScript (javascript)

Example Ubuntu auth log event:

Jun 23 10:11:12 ubuntu sshd[1234]: Failed password for invalid user admin from 10.0.0.5 port 53321 ssh2
Code language: CSS (css)

3.2 Source

The technology or integration that generated the log.

Examples:

source:apache
source:syslog
source:linux
Code language: CSS (css)

3.3 Service

The logical application or service name.

Examples:

service:apache
service:ubuntu
service:web
Code language: CSS (css)

3.4 Host

The server or VM that generated the log.

Example:

host:ubuntu-demo
Code language: CSS (css)

3.5 Status

The severity level of a log.

Common values:

status:info
status:warn
status:error
status:critical
Code language: CSS (css)

3.6 Attribute

A parsed field from the log.

Examples:

@http.status_code
@http.method
@http.url
@network.client.ip
@user.name
Code language: CSS (css)

Attributes usually use the @ prefix when searched.

3.7 Facet

A facet is a searchable/filterable field shown in the left-side panel.

Examples:

Source
Service
Host
Status
HTTP Status Code
HTTP Method
Client IP
URL Path

Facets make searching easier because students can click values instead of typing full queries.

4. Lab Flow Overview

flowchart TD
    A[Open Datadog Log Explorer] --> B[Confirm Logs Are Available]
    B --> C[Filter by Time Range]
    C --> D[Filter Ubuntu Logs]
    C --> E[Filter Apache Logs]
    D --> F[Search Linux Events]
    E --> G[Search Apache Requests]
    F --> H[Use Facets]
    G --> H
    H --> I[Open Log Side Panel]
    I --> J[Add Columns]
    J --> K[Group and Visualize]
    K --> L[Patterns and Live Tail]
    L --> M[Create Saved Views]
    M --> N[Final Troubleshooting Exercises]
Code language: CSS (css)

5. Lab 1: Open Datadog Log Explorer

Goal

Open the Datadog Logs Console and confirm that logs are visible.

Steps

1. Log in to Datadog.
2. In the left menu, go to:

Logs → Explorer

3. Set the time range to:

Past 15 minutes

4. If no logs appear, change the time range to:

Past 1 hour

5. If still no logs appear, change the time range to:

Past 24 hours

Expected Result

Students should see a list of log events.

Each row usually shows information such as:

Timestamp
Status
Host
Service
Source
Message/content

Student Check

Write down:

Total logs visible: __________
Time range used: __________

6. Lab 2: Understand the Log Explorer Screen

Goal

Identify the main areas of Log Explorer.

Key UI Areas

Students should locate:

Search bar
Time range selector
Facet panel
Log list/table
Visualization selector
Log side panel
Saved Views
Export/share options
Code language: PHP (php)

Exercise

Click one log entry.

Observe the side panel.

Identify:

Timestamp
Host
Source
Service
Status
Message
Tags
Attributes

Student Check

Write down any five fields visible in the selected log:

1. __________
2. __________
3. __________
4. __________
5. __________

7. Lab 3: Discover Available Sources

Goal

Find out how Ubuntu and Apache logs are tagged in your Datadog environment.

Steps

1. Clear the search bar.
2. Set time range:

Past 24 hours

3. In the left facet panel, find:

Source

4. Expand the Source facet.
5. Look for values related to Apache or Linux.

Possible values:

apache
syslog
ubuntu
linux
agent

Search Query Option

Students can also type:

source:apache
Code language: CSS (css)

Then try:

source:syslog
Code language: CSS (css)

Then try:

source:linux
Code language: CSS (css)

Expected Result

Datadog should show logs matching that source.

Student Check

Write down the source values found:

Apache source value: __________
Ubuntu/Linux source value: __________

8. Lab 4: Discover Services

Goal

Find the service names attached to Ubuntu and Apache logs.

Steps

1. Clear the search bar.
2. Set time range:

Past 24 hours

3. In the left facet panel, expand:

Service

4. Look for Apache/Linux-related services.

Possible service values:

apache
web
ubuntu
system
syslog

Search Query Option

Try:

service:apache
Code language: CSS (css)

Try:

service:web
Code language: CSS (css)

Try:

service:ubuntu
Code language: CSS (css)

Student Check

Write down:

Apache service value: __________
Ubuntu service value: __________

9. Lab 5: Basic Free-Text Search

Goal

Search logs by words in the message/content.

Steps

In the search bar, try these one by one.

Search for Apache

apache

Search for SSH

ssh

Search for failed logins

failed

Search for sudo

sudo

Search for cron

cron

Expected Result

Datadog displays logs where the message contains the searched term.

Student Check

For each query, record whether logs were found:

10. Lab 6: Exact Phrase Search

Goal

Search for exact phrases.

Steps

Try these queries:

"Failed password"
Code language: JSON / JSON with Comments (json)

"Accepted password"
Code language: JSON / JSON with Comments (json)

"Invalid user"
Code language: JSON / JSON with Comments (json)

"GET /"
Code language: JSON / JSON with Comments (json)

"POST /"
Code language: JSON / JSON with Comments (json)

Explanation

Double quotes search for a phrase.

Use this when you want logs containing a specific sentence or phrase.

Student Check

Which exact phrase returned the most logs?

Answer: ____________________
Code language: HTTP (http)

11. Lab 7: Boolean Search with AND

Goal

Search for logs that match multiple conditions.

Syntax

term1 AND term2

If you enter two search terms without an operator, Datadog generally treats it like an AND-style search.

Exercises

Search for SSH failed events:

ssh AND failed

Search for Apache GET requests:

apache AND GET

Search for sudo authentication events:

sudo AND authentication

Search for failed password messages:

failed AND password

Student Check

Which query is most useful for investigating failed SSH login attempts?

Answer: ____________________
Code language: HTTP (http)

12. Lab 8: Boolean Search with OR

Goal

Search for logs matching either one condition or another.

Syntax

term1 OR term2

Exercises

Search for SSH or sudo logs:

ssh OR sudo

Search for failed or invalid login logs:

failed OR invalid

Search for GET or POST Apache requests:

GET OR POST

Search for warning or error logs:

status:warn OR status:error
Code language: CSS (css)

Student Check

How many logs are returned for:

ssh OR sudo

Record count:

Count: __________
Code language: HTTP (http)

13. Lab 9: Exclusion Search

Goal

Exclude unwanted logs from search results.

Syntax

query -term

or:

query AND -term

Exercises

Search for Apache logs but exclude successful 200 responses:

source:apache AND -@http.status_code:200

If @http.status_code does not exist in your logs, try:

apache AND -200

Search for SSH logs but exclude accepted logins:

ssh AND -accepted

Search for failed logs but exclude cron:

failed AND -cron

Search for error logs but exclude Apache:

status:error AND -source:apache
Code language: CSS (css)

Student Check

Why is exclusion useful?

Answer: ____________________
Code language: HTTP (http)

14. Lab 10: Filter by Time Range

Goal

Understand how time range changes the logs displayed.

Steps

Run the same query with different time ranges.

Query:

source:apache
Code language: CSS (css)

Time ranges:

Past 15 minutes
Past 1 hour
Past 4 hours
Past 24 hours

Student Check

Record the result count:

Discussion

The larger the time range, the more logs you usually see.

For real troubleshooting, start with a narrow time range around the incident.

15. Lab 11: Filter by Host

Goal

Show logs from a specific Ubuntu host.

Steps

1. Clear the search bar.
2. Open the Host facet.
3. Select one host.
4. Observe the query automatically added.

Example query:

host:ubuntu-demo
Code language: CSS (css)

Combine Host and Source

Search Apache logs from one host:

host:ubuntu-demo source:apache
Code language: CSS (css)

Search syslog/Linux logs from one host:

host:ubuntu-demo source:syslog
Code language: CSS (css)

Replace ubuntu-demo with your actual host name.

Student Check

Write the host value used:

Host: ____________________
Code language: HTTP (http)

16. Lab 12: Filter by Status

Goal

Search logs by severity.

Common Status Queries

status:info
Code language: CSS (css)

status:warn
Code language: CSS (css)

status:error
Code language: CSS (css)

status:critical
Code language: CSS (css)

Exercises

Search all error logs:

status:error
Code language: CSS (css)

Search Apache error logs:

source:apache status:error
Code language: CSS (css)

Search Ubuntu error logs:

source:syslog status:error
Code language: CSS (css)

If your source is different, replace source:syslog.

Student Check

Which source has more error logs?

Apache / Ubuntu / Other: ____________________

17. Lab 13: Apache Access Log Search

Goal

Search Apache access logs using HTTP-specific fields.

Common Apache Fields

Depending on parsing, students may see fields such as:

@http.method
@http.status_code
@http.url
@http.url_details.path
@network.client.ip
@http.useragent
Code language: CSS (css)

Step 1: Find Apache Logs

Try:

source:apache
Code language: CSS (css)

or:

service:apache
Code language: CSS (css)

Step 2: Open One Apache Log

Click an Apache access log.

In the side panel, look for attributes such as:

http
network
url
status_code
method
client_ip

Step 3: Search GET Requests

Try:

@http.method:GET

If that does not work, try free text:

GET

Step 4: Search POST Requests

@http.method:POST

Fallback:

POST

Step 5: Search Status Code 200

@http.status_code:200

Fallback:

200

Step 6: Search 404 Errors

@http.status_code:404

Fallback:

404

Step 7: Search 5xx Errors

@http.status_code:[500 TO 599]

Fallback:

500 OR 502 OR 503 OR 504

Student Check

Record counts:

18. Lab 14: Apache Troubleshooting Scenario — Find Broken URLs

Scenario

Users are reporting that some Apache pages are returning errors.

Goal

Find which URLs are returning 404 or 5xx responses.

Steps

Search 404 logs:

source:apache @http.status_code:404

If attributes are not available:

source:apache 404
Code language: CSS (css)

Open one 404 log.

Look for URL/path fields:

@http.url
@http.url_details.path
@url
@request
Code language: CSS (css)

Add the URL/path field as a column.

Group by URL/path using Log Analytics.

Analytics Steps

1. Search:

source:apache @http.status_code:404

2. Switch from log list to analytics/visualization mode.
3. Group by:

@http.url_details.path
Code language: CSS (css)

or:

@http.url
Code language: CSS (css)

4. Visualization:

Top List
Code language: PHP (php)

5. Sort by count descending.

Expected Result

Students should see top broken URLs.

Student Check

Top broken URL/path:

URL/path: ____________________
Count: ____________________
Code language: HTTP (http)

19. Lab 15: Apache Troubleshooting Scenario — Find Top Client IPs

Goal

Find which client IPs are generating the most Apache requests.

Query

source:apache
Code language: CSS (css)

Analytics Steps

1. Open Log Analytics.
2. Group by one of these fields:

@network.client.ip
@http.client_ip
@client_ip
Code language: CSS (css)

3. Visualization:

Top List
Code language: PHP (php)

4. Sort by count.

Expected Result

Students should see top client IP addresses.

Student Check

Top client IP:

IP: ____________________
Request count: ____________________

Bonus

Find client IPs causing 404:

source:apache @http.status_code:404

Group by:

@network.client.ip
Code language: CSS (css)

20. Lab 16: Apache Troubleshooting Scenario — Identify Error Spikes

Goal

Use timeseries view to find when Apache errors increased.

Query

source:apache status:error
Code language: CSS (css)

or:

source:apache @http.status_code:[500 TO 599]

Steps

1. Set time range:

Past 24 hours

2. Switch visualization to:

Timeseries

3. Group by:

service

or:

host

4. Observe spikes.

Student Check

Approximate time of highest spike:

Time: ____________________
Code language: HTTP (http)

Possible cause:

Answer: ____________________
Code language: HTTP (http)

21. Lab 17: Ubuntu Auth Logs — Failed SSH Login

Goal

Search Ubuntu authentication logs for failed SSH login attempts.

Queries

Try these:

"Failed password"
Code language: JSON / JSON with Comments (json)

ssh AND failed

sshd AND "Failed password"
Code language: JavaScript (javascript)

source:syslog sshd failed
Code language: CSS (css)

If your source is different, replace source:syslog.

Open a Failed Login Log

Look for fields or message content showing:

Username
Source IP
Port
sshd process
Host
Timestamp

Student Check

Record one failed login example:

Username attempted: ____________________
Source IP: ____________________
Host: ____________________
Timestamp: ____________________

22. Lab 18: Ubuntu Auth Logs — Accepted SSH Login

Goal

Find successful SSH login events.

Queries

"Accepted password"
Code language: JSON / JSON with Comments (json)

sshd AND accepted

ssh AND accepted

Compare Failed vs Accepted

Failed:

"Failed password"
Code language: JSON / JSON with Comments (json)

Accepted:

"Accepted password"
Code language: JSON / JSON with Comments (json)

Student Check

Which is higher?

Failed / Accepted: ____________________

23. Lab 19: Ubuntu Sudo Activity

Goal

Search for sudo command usage.

Queries

sudo

sudo AND COMMAND

"sudo:" AND COMMAND
Code language: JavaScript (javascript)

source:syslog sudo
Code language: CSS (css)

Open One Sudo Log

Look for:

User
Command
Working directory
Target user
Timestamp
Host

Student Check

Record one sudo command:

User: ____________________
Command: ____________________
Host: ____________________
Time: ____________________
Code language: HTTP (http)

24. Lab 20: Ubuntu Cron Logs

Goal

Search for scheduled job logs.

Queries

cron

CRON

source:syslog cron
Code language: CSS (css)

source:syslog CRON
Code language: CSS (css)

Discussion

Linux logs may have uppercase CRON, depending on syslog format.

Student Check

Did you find cron logs?

Yes / No: ____________________

25. Lab 21: Ubuntu Systemd Logs

Goal

Find systemd service events.

Queries

systemd

source:syslog systemd
Code language: CSS (css)

systemd AND started

systemd AND stopped

systemd AND failed

Student Check

Record one service-related event:

Service/process: ____________________
Message: ____________________
Timestamp: ____________________
Code language: HTTP (http)

26. Lab 22: Wildcard Search

Goal

Use wildcard search to match partial words or patterns.

Apache Examples

Search anything containing GET:

*GET*

Search services starting with apach:

service:apach*
Code language: CSS (css)

Search sources starting with sys:

source:sys*
Code language: CSS (css)

Search messages containing NETWORK:

*NETWORK*

Ubuntu Examples

*fail*

*auth*

*sudo*

Important Note

Wildcard search can be useful, but avoid using very broad wildcards over large time ranges because it can return too many logs and slow down investigation.

Student Check

Try:

*fail*

Record the count:

Count: ____________________
Code language: HTTP (http)

27. Lab 23: Attribute Search

Goal

Search structured fields with the @ prefix.

Examples

Search a URL path:

@http.url_details.path:"/index.html"
Code language: JavaScript (javascript)

Search HTTP status code:

@http.status_code:404

Search HTTP method:

@http.method:GET

Search client IP:

@network.client.ip:192.168.1.10

Steps

1. Open an Apache log.
2. Find available attributes in the side panel.
3. Pick one attribute.
4. Click the attribute value.
5. Choose an option such as:

Filter for this value
Code language: JavaScript (javascript)

or manually type the attribute query.

Student Check

Write one attribute query you successfully used:

Query: ____________________
Code language: HTTP (http)

28. Lab 24: Numeric Range Search

Goal

Use range search for numeric fields.

HTTP Status Ranges

Search successful responses:

@http.status_code:[200 TO 299]

Search redirects:

@http.status_code:[300 TO 399]

Search client errors:

@http.status_code:[400 TO 499]

Search server errors:

@http.status_code:[500 TO 599]

Response Time Example

If your Apache logs contain response time:

@http.response_time:>100

or:

@duration:>1000000

The exact field name depends on your parser.

Student Check

Which HTTP status range has the highest count?

2xx / 3xx / 4xx / 5xx: ____________________

29. Lab 25: Facet Filtering

Goal

Use the left-side facet panel instead of typing all queries manually.

Steps

1. Clear the search bar.
2. Set time range:

Past 24 hours

3. Use facets to filter:

Source = apache
Status = error
Host = your Ubuntu host

4. Observe the query generated by Datadog.

Example:

source:apache status:error host:ubuntu-demo
Code language: CSS (css)

Exercise

Use facets to build this investigation:

Apache errors from one host in the past 24 hours
Code language: JavaScript (javascript)

Student Check

Final query generated:

Query: ____________________
Code language: HTTP (http)

30. Lab 26: Log Side Panel Deep Dive

Goal

Inspect one log fully.

Steps

1. Search:

source:apache
Code language: CSS (css)

2. Click one log row.
3. In the side panel, inspect:

General context
Tags
Attributes
Message/content
Host
Service
Source
Status

4. Click an attribute value.
5. Try these actions if available:

Add as column
Filter for this value
Exclude this value
Create facet
Copy value
Code language: JavaScript (javascript)

Student Check

Write down:

Message: ____________________
Source: ____________________
Service: ____________________
Host: ____________________
Status: ____________________
Code language: HTTP (http)

31. Lab 27: Add and Remove Columns

Goal

Customize the log list display.

Steps

1. Open one Apache log.
2. Find the HTTP status field, for example:

@http.status_code
Code language: CSS (css)

3. Add it as a column.
4. Add HTTP method as a column:

@http.method
Code language: CSS (css)

5. Add URL path as a column:

@http.url_details.path
Code language: CSS (css)

6. Add client IP as a column:

@network.client.ip
Code language: CSS (css)

7. Return to the log list and observe the table.

Recommended Apache Columns

Time
Host
Service
Source
Status
@http.method
@http.status_code
@http.url_details.path
@network.client.ip
Code language: CSS (css)

Recommended Ubuntu Columns

Time
Host
Service
Source
Status
process
message
user

Actual field names may differ.

Student Check

List the columns you added:

1. __________
2. __________
3. __________
4. __________

32. Lab 28: Display Logs by Newest and Oldest

Goal

Understand timestamp ordering.

Steps

1. Search:

source:apache
Code language: CSS (css)

2. Look at the timestamp column.
3. Change the display/order if the UI allows:

Newest first
Oldest first

4. Compare the first and last visible logs.

Student Check

Newest visible log timestamp:

Timestamp: ____________________
Code language: HTTP (http)

Oldest visible log timestamp:

Timestamp: ____________________
Code language: HTTP (http)

33. Lab 29: Group Apache Logs by Status Code

Goal

Use Log Analytics to count logs by HTTP status code.

Steps

1. Search:

source:apache
Code language: CSS (css)

2. Switch to analytics/grouping mode.
3. Group by:

@http.status_code
Code language: CSS (css)

4. Visualization:

Top List
Code language: PHP (php)

5. Sort by count descending.

Expected Result

Students should see values like:

200
301
302
403
404
500

Student Check

Top HTTP status code:

Status code: ____________________
Count: ____________________

34. Lab 30: Group Apache Logs by HTTP Method

Goal

Find request volume by HTTP method.

Query

source:apache
Code language: CSS (css)

Group By

@http.method
Code language: CSS (css)

Expected Values

GET
POST
PUT
DELETE
HEAD
OPTIONS

Student Check

Most common HTTP method:

Method: ____________________
Count: ____________________
Code language: HTTP (http)

35. Lab 31: Group Apache Errors by URL Path

Goal

Identify which URL paths produce the most errors.

Query

source:apache @http.status_code:[400 TO 599]

Fallback:

source:apache 404 OR 500 OR 502 OR 503
Code language: CSS (css)

Group By

@http.url_details.path
Code language: CSS (css)

or:

@http.url
Code language: CSS (css)

Visualization

Top List
Code language: PHP (php)

Student Check

Top error URL:

URL/path: ____________________
Count: ____________________
Code language: HTTP (http)

36. Lab 32: Group Ubuntu Logs by Host

Goal

Find which host generates the most Linux logs.

Query

source:syslog
Code language: CSS (css)

or your actual Linux source.

Group By

host

Visualization

Top List
Code language: PHP (php)

Student Check

Top host:

Host: ____________________
Count: ____________________
Code language: HTTP (http)

37. Lab 33: Group Ubuntu Authentication Failures

Goal

Analyze failed login logs.

Query

"Failed password"
Code language: JSON / JSON with Comments (json)

or:

sshd AND failed

Group By Options

Depending on parsed fields, try:

host
@user.name
@network.client.ip
Code language: CSS (css)

If no parsed fields exist, use Patterns instead.

Student Check

Most common source IP or host:

Value: ____________________
Count: ____________________
Code language: HTTP (http)

38. Lab 34: Use Timeseries View

Goal

Display logs over time.

Steps

1. Search:

source:apache
Code language: CSS (css)

2. Switch visualization to:

Timeseries

3. Set time range:

Past 24 hours

4. Group by:

@http.status_code
Code language: CSS (css)

or:

status

Expected Result

Students should see log volume over time.

Student Check

When was the highest Apache traffic?

Time: ____________________
Code language: HTTP (http)

39. Lab 35: Use Table View

Goal

Create a table-style log analysis.

Steps

1. Search:

source:apache
Code language: CSS (css)

2. Switch to analytics.
3. Choose visualization:

Table

4. Group by:

@http.status_code
@http.method
Code language: CSS (css)

5. Measure:

count

Expected Result

A table showing count by status code and method.

Student Check

Which method/status combination is most common?

Method: ____________________
Status: ____________________
Count: ____________________
Code language: HTTP (http)

40. Lab 36: Use Pie Chart

Goal

Display log distribution by category.

Query

source:apache
Code language: CSS (css)

Visualization

Pie Chart

Group By

@http.status_code
Code language: CSS (css)

or:

status

Student Check

Which category has the largest slice?

Answer: ____________________
Code language: HTTP (http)

41. Lab 37: Use Tree Map

Goal

Visualize large contributors.

Query

source:apache
Code language: CSS (css)

Visualization

Tree Map
Code language: JavaScript (javascript)

Group By

@http.url_details.path
Code language: CSS (css)

or:

@network.client.ip
Code language: CSS (css)

Student Check

Largest block:

Field value: ____________________
Count: ____________________

42. Lab 38: Log Patterns

Goal

Use patterns to identify repeated types of log messages.

Why Patterns Matter

Raw logs can be noisy. Patterns group similar logs together so students can quickly identify repeated behavior.

Useful for:

Repeated Apache errors
Repeated SSH failures
Repeated sudo activity
Repeated cron messages
Repeated systemd messages

Steps

1. Search:

source:syslog
Code language: CSS (css)

or:

source:apache
Code language: CSS (css)

2. Switch view to:

Patterns

3. Observe grouped messages.
4. Click a pattern to inspect matching logs.

Example Pattern Results

Ubuntu:

Failed password for invalid user * from *
Code language: JavaScript (javascript)

Apache:

GET * HTTP/1.1 404

Student Check

Write one pattern found:

Pattern: ____________________
Count: ____________________
Code language: HTTP (http)

43. Lab 39: Live Tail

Goal

Use Live Tail to see logs as they arrive.

Steps

1. Open Datadog Log Explorer.
2. Change the time selector to:

Live Tail

3. Search:

source:apache
Code language: CSS (css)

4. Generate Apache traffic from a browser or terminal if allowed.
5. Observe logs arriving.

Example command from a machine that can reach Apache:

curl http://<apache-server-ip>/
Code language: HTML, XML (xml)

Generate 404 traffic:

curl http://<apache-server-ip>/does-not-exist
Code language: HTML, XML (xml)

Important Note

Live Tail may sample logs when very high log volume is flowing. If students need to see a specific event, they should narrow the query using source, host, service, or status.

Student Check

Did new logs appear in Live Tail?

Yes / No: ____________________

44. Lab 40: Build a Saved View for Apache Errors

Goal

Create a reusable troubleshooting view.

Query

source:apache status:error
Code language: CSS (css)

or:

source:apache @http.status_code:[400 TO 599]

Recommended Time Range

Past 24 hours

Recommended Columns

Time
Host
@http.status_code
@http.method
@http.url_details.path
@network.client.ip
Message
Code language: CSS (css)

Steps

1. Apply query.
2. Add useful columns.
3. Choose a useful visualization, such as:

Log Stream

or:

Top List grouped by @http.status_code
Code language: CSS (css)

4. Save the view.
5. Name it:

Apache Error Troubleshooting
Code language: JavaScript (javascript)

Student Check

Saved View name:

Name: ____________________
Code language: HTTP (http)

45. Lab 41: Build a Saved View for Ubuntu Authentication

Goal

Create a reusable view for Linux authentication events.

Query

sshd OR sudo OR "Failed password" OR "Accepted password"
Code language: JavaScript (javascript)

Optional source-scoped query:

source:syslog (sshd OR sudo OR "Failed password" OR "Accepted password")
Code language: JavaScript (javascript)

Recommended Time Range

Past 24 hours

Recommended Columns

Time
Host
Source
Status
Message

Save View Name

Ubuntu Authentication Activity

Student Check

Saved View created?

Yes / No: ____________________

46. Lab 42: Build a Saved View for Apache 404 Investigation

Goal

Create a saved view for broken URL troubleshooting.

Query

source:apache @http.status_code:404

Fallback:

source:apache 404
Code language: CSS (css)

Group By

@http.url_details.path
Code language: CSS (css)

Visualization

Top List
Code language: PHP (php)

Save View Name

Apache 404 Broken URL Analysis

Student Check

Saved View created?

Yes / No: ____________________

47. Lab 43: Export or Share Logs

Goal

Share investigation results with a teammate.

Steps

1. Open one useful search, for example:

source:apache @http.status_code:[500 TO 599]

2. Use Datadog share/export options.
3. Copy the view link or export the selected result if your permissions allow.
4. Open one individual log.
5. Use the side panel share/copy option if available.

Student Check

What did you share/export?

View link / Log JSON / CSV / Screenshot / Other: ____________________
Code language: JavaScript (javascript)

48. Lab 44: Practical Investigation 1 — Apache 404 Spike

Scenario

A website owner reports that users are seeing “page not found” errors.

Student Task

Find:

How many 404 logs exist?
Which URL has the most 404s?
Which client IP generated the most 404s?
Which host served those errors?
When did the 404s happen?

Suggested Queries

source:apache @http.status_code:404

Fallback:

source:apache 404
Code language: CSS (css)

Suggested Groupings

@http.url_details.path
@network.client.ip
host
Code language: CSS (css)

Student Answer

Total 404 logs: ____________________
Top broken URL: ____________________
Top client IP: ____________________
Top host: ____________________
Peak time: ____________________

49. Lab 45: Practical Investigation 2 — Apache Server Error

Scenario

Users report HTTP 500 errors.

Student Task

Find:

How many 5xx errors occurred?
Which endpoint had the most errors?
Which host generated them?
Were errors concentrated at one time?

Query

source:apache @http.status_code:[500 TO 599]

Fallback:

source:apache 500 OR 502 OR 503 OR 504
Code language: CSS (css)

Visualizations

Timeseries grouped by host
Top List grouped by @http.url_details.path
Table grouped by @http.status_code and host
Code language: CSS (css)

Student Answer

Total 5xx logs: ____________________
Top endpoint: ____________________
Top host: ____________________
Peak time: ____________________

50. Lab 46: Practical Investigation 3 — Failed SSH Login Attempts

Scenario

Security team suspects repeated SSH login failures.

Student Task

Find:

How many failed SSH attempts happened?
Which host was targeted?
Which username was attempted?
Which source IP appeared most often?

Query

"Failed password"
Code language: JSON / JSON with Comments (json)

or:

sshd AND failed

Grouping Options

Try:

host
@user.name
@network.client.ip
Code language: CSS (css)

If parsed fields are missing, use Patterns and manual inspection.

Student Answer

Total failed attempts: ____________________
Top host: ____________________
Top username: ____________________
Top source IP: ____________________

51. Lab 47: Practical Investigation 4 — Sudo Activity

Scenario

Admin wants to review sudo usage.

Query

sudo AND COMMAND

or:

source:syslog sudo
Code language: CSS (css)

Student Task

Find:

Which users ran sudo?
Which commands were run?
Which host generated the logs?
When did sudo activity happen?

Student Answer

User: ____________________
Command: ____________________
Host: ____________________
Timestamp: ____________________
Code language: HTTP (http)

52. Lab 48: Practical Investigation 5 — Linux Service Problems

Scenario

A Linux service may have failed or restarted.

Queries

systemd AND failed

systemd AND stopped

systemd AND started

status:error source:syslog
Code language: CSS (css)

Student Task

Find:

Which service had an issue?
Which host?
What time?
Was it started/stopped/failed?

Student Answer

Service/process: ____________________
Host: ____________________
Event type: ____________________
Timestamp: ____________________

53. Lab 49: Advanced Query Cheat Sheet

General Search

apache

ssh

sudo

cron

systemd

Exact Phrase

"Failed password"
Code language: JSON / JSON with Comments (json)

"Accepted password"
Code language: JSON / JSON with Comments (json)

"GET /index.html"
Code language: JSON / JSON with Comments (json)

Boolean Operators

ssh AND failed

ssh OR sudo

apache AND -200

status:error AND source:apache
Code language: CSS (css)

Tags and Reserved Attributes

source:apache
Code language: CSS (css)

service:apache
Code language: CSS (css)

host:ubuntu-demo
Code language: CSS (css)

status:error
Code language: CSS (css)

Attribute Search

@http.method:GET

@http.status_code:404

@http.url_details.path:"/index.html"
Code language: JavaScript (javascript)

@network.client.ip:192.168.1.10

Range Search

@http.status_code:[200 TO 299]

@http.status_code:[400 TO 499]

@http.status_code:[500 TO 599]

@http.response_time:>100

Wildcard

service:apach*
Code language: CSS (css)

source:sys*
Code language: CSS (css)

*fail*

*auth*

54. Lab 50: Student Final Challenge

Final Scenario

You are the on-call engineer. A manager says:

The website had issues today, and we also saw suspicious login activity on the server.

Student Mission

Use Datadog Logs Explorer to answer:

Apache/Web Questions

1. How many Apache logs were generated in the past 24 hours?
2. How many HTTP 4xx responses occurred?
3. How many HTTP 5xx responses occurred?
4. Which URL/path had the most errors?
5. Which client IP generated the most requests?
6. Which host served the most errors?
7. What time did the highest error spike happen?

Ubuntu/Linux Questions

1. How many failed SSH login attempts occurred?
2. Were there successful SSH logins?
3. Which host had the most authentication activity?
4. Were any sudo commands run?
5. Were there any systemd failed/stopped service messages?

Deliverable

Students must submit:

1. Queries used
2. Screenshots of Log Explorer results
3. Top findings
4. Saved View names created
5. One short incident summary

55. Sample Student Incident Summary

Use this template:

Incident Summary:

Time range investigated:
Past 24 hours

Apache findings:
- Total Apache logs:
- Total 4xx errors:
- Total 5xx errors:
- Top error URL:
- Top client IP:
- Peak error time:

Ubuntu/Linux findings:
- Failed SSH attempts:
- Successful SSH attempts:
- Sudo activity:
- Systemd/service errors:

Conclusion:
The main web issue appears to be related to __________.
The main Linux/security observation is __________.

Next recommended action:
__________.

56. Instructor Validation Checklist

Before running this lab, the instructor should verify:

Datadog Agent is sending logs
Ubuntu/Linux logs are visible
Apache access logs are visible
Apache error logs are visible
Logs have source tags
Logs have service tags
Host field is visible
Apache fields are parsed, if possible
Students have Log Explorer access
Students can create Saved Views, if required

57. Troubleshooting During Lab

Problem: No logs visible

Try:

Increase time range to Past 24 hours
Clear search query
Check source facet
Check host facet
Confirm logs are ingested

Problem: Apache source does not work

Try:

service:apache
Code language: CSS (css)

apache

Look at the Source facet and find the actual source value.

Problem: @http.status_code does not work

Apache logs may not be parsed into HTTP attributes.

Fallback:

source:apache 404
Code language: CSS (css)

source:apache 500
Code language: CSS (css)

Ask instructor to check Apache log pipeline/parser.

Problem: @network.client.ip does not exist

Try:

@client.ip
Code language: CSS (css)

@client_ip
Code language: CSS (css)

@http.client_ip
Code language: CSS (css)

Or inspect one Apache log in the side panel and find the actual field.

Problem: Ubuntu source is not syslog

Check the Source facet.

Possible alternatives:

source:ubuntu
source:linux
source:system
source:agent
Code language: CSS (css)

Problem: Too many logs

Narrow query using:

source
service
host
status
time range
specific phrase

Example:

source:apache host:ubuntu-demo @http.status_code:404

58. Best Practices Students Should Remember

Use narrow time ranges first

Start with:

Past 15 minutes
Past 1 hour

Then expand if needed.

Use facets before complex queries

Facets help avoid spelling mistakes and reveal actual available values.

Prefer attributes for accurate search

Better:

@http.status_code:404

Less accurate:

404

Use exact phrase search for known messages

Example:

"Failed password"
Code language: JSON / JSON with Comments (json)

Use exclusion to remove noise

Example:

source:apache AND -@http.status_code:200

Add useful columns

Good columns make troubleshooting faster.

Save repeatable views

Create Saved Views for common troubleshooting scenarios.

Use Patterns when logs are noisy

Patterns quickly reveal repeated messages.

Use Analytics for counts and grouping

Analytics helps answer “how many,” “which one,” and “when.”

59. Recommended Lab Timing

Total suggested duration:

3 to 4 hours

For a shorter workshop, run Labs 1–18, 25–31, and the Final Challenge.

60. Final Learning Outcome

After completing this lab, students should be comfortable using Datadog Logs Explorer to:

Search logs
Filter logs
Sort and display logs
Inspect log attributes
Use facets
Analyze Apache access/error logs
Analyze Ubuntu Linux logs
Find failed SSH attempts
Find sudo activity
Find Apache 404/5xx errors
Create useful columns
Group logs by fields
Use visualizations
Use patterns
Use Live Tail
Create Saved Views
Summarize troubleshooting findings
Code language: PHP (php)

This lab gives students the foundation needed for real-world log troubleshooting in Datadog.

Older Content As Below

List of example of all search techniques in datadog for apache

Datadog is a monitoring and analytics platform that provides various search techniques to help you analyze Apache web server data effectively. Here are some example search techniques you can use in Datadog for Apache monitoring:

1. Basic Text Search:
   • Search for specific log messages or keywords in Apache logs.
   • Example: apache.error OR "500 Internal Server Error"
2. Filtering by Hostname:
   • Narrow down your search by specifying the hostname or server where Apache is running.
   • Example: hostname:webserver-1
3. Time-based Search:
   • Search for Apache logs within a specific time frame.
   • Example: @timestamp:[NOW-1h TO NOW]
4. Status Code Analysis:
   • Analyze Apache access logs by filtering on HTTP status codes.
   • Example: http.status:404
5. URL and Path-based Search:
   • Analyze requests for specific URLs or paths.
   • Example: http.url:/product/details
6. User-Agent Filtering:
   • Filter logs based on the user-agent header from client requests.
   • Example: http.user_agent:"Mozilla/5.0"
7. IP Address Filtering:
   • Search for logs related to specific IP addresses or IP ranges.
   • Example: http.remote_ip:192.168.1.1
8. Request Method Analysis:
   • Analyze logs by HTTP request methods (GET, POST, etc.).
   • Example: http.method:POST
9. Response Time Analysis:
   • Search for Apache logs with specific response times.
   • Example: http.response_time_ms:>100
10. Combining Filters:
    • Combine multiple filters to narrow down your search results.
    • Example: http.status:500 AND http.url:/admin
11. Grouping and Aggregation:
    • Use aggregation functions like avg, sum, or count to analyze log data.
    • Example: avg(http.response_time_ms) by http.url
12. Alerting Queries:
    • Create alerting queries to monitor specific conditions and trigger alerts.
    • Example: http.status:500 > 10 TIMESERIES
13. Correlation Searches:
    • Identify patterns and correlations between different log fields.
    • Example: http.status:500 GROUPBY http.url
14. Histograms and Percentiles:
    • Analyze response time distributions using histograms and percentiles.
    • Example: histogram(http.response_time_ms)
15. Pattern Matching and Regular Expressions:
    • Use regular expressions to match complex patterns in log messages.
    • Example: http.url:/api/(v\d+)/endpoint

List of example of all search techniques in datadog for syslog

Datadog provides powerful search and analysis capabilities for syslog data. Here are some example search techniques you can use in Datadog for syslog monitoring:

1. Basic Text Search:
   • Search for specific log messages or keywords in syslog messages.
   • Example: syslog.message:"error"
2. Filtering by Hostname:
   • Narrow down your search by specifying the hostname or server generating the syslog messages.
   • Example: hostname:webserver-1
3. Time-based Search:
   • Search for syslog messages within a specific time frame.
   • Example: @timestamp:[NOW-1h TO NOW]
4. Log Severity Levels:
   • Filter logs based on severity levels (e.g., INFO, WARNING, ERROR).
   • Example: syslog.severity:ERROR
5. Log Facility:
   • Filter logs by the facility they belong to (e.g., auth, authpriv, cron).
   • Example: syslog.facility:auth
6. IP Address Filtering:
   • Search for logs related to specific IP addresses or IP ranges.
   • Example: syslog.source_ip:192.168.1.1
7. User Identification:
   • Filter syslog logs by user-related information.
   • Example: syslog.user:"john_doe"
8. Process Identification:
   • Analyze syslog logs by identifying the process or application generating the logs.
   • Example: syslog.process:"sshd"
9. Message Type:
   • Filter syslog logs based on message type or format.
   • Example: syslog.message_type:"authentication failure"
10. Log Pattern Matching:
    • Use regular expressions to match complex patterns in syslog messages.
    • Example: syslog.message:/^ERROR:.*$/
11. Alerting Queries:
    • Create alerting queries to monitor specific conditions and trigger alerts.
    • Example: syslog.severity:ERROR > 5 TIMESERIES
12. Grouping and Aggregation:
    • Use aggregation functions like count or avg to analyze log data.
    • Example: count(syslog.message) by syslog.process
13. Correlation Searches:
    • Identify patterns and correlations between different log fields.
    • Example: syslog.severity:ERROR GROUPBY syslog.process
14. Histograms and Percentiles:
    • Analyze log message distributions using histograms and percentiles.
    • Example: histogram(syslog.message_length)
15. Custom Tags and Attributes:
    • Leverage custom tags or attributes to further categorize and filter syslog logs.
    • Example: @mytag:myvalue

Rajesh Kumar

I’m a DevOps/SRE/DevSecOps/Cloud Expert passionate about sharing knowledge and experiences. I have worked at Cotocus. I share tech blog at DevOps School, travel stories at Holiday Landmark, stock market tips at Stocks Mantra, health and fitness guidance at My Medic Plus, product reviews at TrueReviewNow , and SEO strategies at Wizbrand.

Do you want to learn Quantum Computing?

Please find my social handles as below;

Rajesh Kumar Personal Website
Rajesh Kumar at YOUTUBE
Rajesh Kumar at INSTAGRAM
Rajesh Kumar at X
Rajesh Kumar at FACEBOOK
Rajesh Kumar at LINKEDIN
Rajesh Kumar at WIZBRAND

Rajesh Kumar DailyLogs