Last documentation verification: 11 October 2026
Reference Falco release: 0.45.0 (released 21 September 2026)
Reference host lab: Ubuntu Server 24.04 LTS, Linux kernel with modern eBPF support
Reference companion versions in Falco 0.45.0: falcoctl 0.14.2, falco-rules 5.2.0, Helm chart 9.2.0
Audience: Beginners, students, software developers, DevOps engineers, SREs, security engineers
Format: 180-minute core training, plus optional in-depth reference and stretch exercises
Level: Beginner to practical intermediate, with clearly marked production considerations

Important lab boundaries: Follow the exercise commands only on machines, Docker engines, and Kubernetes clusters you own or are authorized to test. Falco requires privileged Linux kernel access for syscall capture. This guide was checked against published documentation, not executed against a live Falco kernel and Kubernetes cluster in this writing environment. Example outputs are illustrative unless directly described by the cited official quickstarts. Distribution packaging, ruleset versions, Helm values and plugin defaults can evolve. Inspect the installed versions before using the steps in production.

How to use this book

The core path is exactly three hours. In each chapter, do the section labeled Core first; use the longer explanations, variations and exercises for later study. A new learner can read this in order and keep one Ubuntu virtual machine for Chapters 4-8, 10 and 11. Chapter 9 needs a separate, authorized Kubernetes cluster with Linux nodes.

Ch.	Topic	Core time	Suggested focus
1	Introduction to Falco	10 min	Theory 10
2	Linux and Security Fundamentals	10 min	Theory 10
3	Falco Architecture and Working	15 min	Theory 15
4	Falco Installation and Setup	20 min	Theory 5 / lab 15
5	Falco Runtime Security Detection	20 min	Theory 5 / lab 15
6	Falco Rules – Fundamentals to Intermediate	25 min	Theory 5 / lab 20
7	Writing and Testing Custom Falco Rules	25 min	Lab 25
8	Falco Alerts and Integrations	10 min	Lab 10
9	Falco with Kubernetes	20 min	Theory 5 / lab 15
10	Rule Tuning and False Positives	15 min	Lab 15
11	Operations, Troubleshooting and Next Steps	10 min	Lab 10
Total	11 chapters	180 min	Theory 55 / lab 125

Realistic timing: The full reference contains more depth than can be read or exercised in 180 minutes. The three-hour plan assumes a prepared Ubuntu VM, downloadable packages, and an existing Kubernetes test cluster. Creating a cluster from scratch, troubleshooting special kernels, or production integration can require substantial additional time.

Required equipment and privileges

• A Linux x86_64 or ARM64 VM with at least 2 vCPUs, 4 GiB RAM and 20 GiB disk for the host portion (recommended lab allocation, not an official Falco minimum).
• Ubuntu Server 24.04 LTS with sudo privileges and outbound access to the official Falco apt repository. A kernel roughly 5.8 or newer is a useful starting point; the actual requirements are the BPF ring buffer and BTF features, not a fixed version number.
• A Docker-capable Linux VM for container exercises; permissions to install a container runtime.
• A test Kubernetes cluster with Linux nodes, Helm, kubectl, and permission to deploy privileged node-level security software. Kubernetes 1.29+ is required for the documented Falco Operator path.
• A plain-text editor (nano or vim) and ability to copy terminal commands into a Bash shell.
• No AWS account, public IP, Slack webhook, or paid service is required for the host labs.

Never run the Falco host monitoring exercise inside an ordinary macOS or Windows shell. Falco’s syscall sensor observes the Linux kernel. Windows and macOS learners should first create an Ubuntu Linux VM as described in Chapter 4.

---

Chapter 1. Introduction to Falco – 10 minutes

Overview and learning outcomes

Core goal: Explain why runtime threat detection matters, where Falco fits, and when it does not help. At the end you should be able to describe Falco to another engineer in 30 seconds and contrast it with container-image scanning.

What is Falco?

Falco is an open-source runtime security detection engine in the cloud-native ecosystem. It observes events from Linux hosts and workloads, evaluates them against security rules, and emits alerts when activity matches a suspicious pattern. Falco commonly runs on a Linux host or as a node-level DaemonSet in Kubernetes. It can also ingest supported external event sources through plugins.

A simple analogy: an image scanner checks a building blueprint before construction; Falco watches the people entering rooms once the building is in use. Both are helpful, but they answer different questions.

Suppose an application was reviewed, its container image scanned, and its deployment approved. Twenty minutes after it starts, a vulnerability is exploited and the application unexpectedly launches /bin/bash. Image scanning cannot tell you that it happened at runtime. A Falco rule can watch process execution and alert on a shell appearing in the container.

Why it is needed

Modern services are ephemeral, frequently deployed, and composed of many third-party dependencies. Security checks made before deployment do not tell you what a process actually did on a running machine. Falco provides the runtime part of a defense-in-depth program:

1. Development and CI: dependency and image scanners find known issues before release.
2. Deployment: admission controls and Kubernetes policy engines guard against unsafe workload configuration.
3. Runtime: Falco observes process, file and network-related events and warns of suspicious behavior.
4. Response: an analyst, SIEM, or separately configured response system investigates and takes action.

Falco vs neighboring tools

Tool or technique	Main question answered	Example	Falco’s role
SCA / dependency scanner	Are vulnerable libraries installed?	A CVE in a package	Not a replacement
Image scanner (e.g., Trivy)	Is the built image vulnerable or misconfigured?	Outdated OpenSSL	Complementary
Kubernetes admission policy	Should a workload be allowed to start?	Privileged pod rejected	Complementary
Linux auditd / general logs	What audit/system records were produced?	User executed a command	Falco adds detection rules and container context
SIEM	How do we correlate events across systems?	Alerts from many hosts	Falco is an event/detection source
Falco	What suspicious activity is happening now?	Unexpected shell or file access	Primary runtime detector

Common use cases

• A shell starts inside an application container.
• An untrusted process reads /etc/shadow.
• A workload modifies a sensitive directory or executable.
• Unexpected commands or network connections appear in a host/container.
• Security teams need Kubernetes-oriented runtime observations and audit enrichment.
• A cloud audit plugin ingests supported external sources, if separately configured.

What Falco cannot guarantee

Falco is primarily detection and alerting, not an automatic prevention control. Without a matching rule, supported event source and healthy collection path, activity may go undetected. It is not a vulnerability scanner, antivirus guarantee, network packet-capture replacement, complete forensics suite, or universal container-escape prevention tool. Rules have false positives and false negatives. Attacks that do not generate monitored events or are deliberately tuned around rule conditions might escape detection. The default plugin configuration also does not mean every cloud or Kubernetes audit source is automatically enabled.

Core exercise: identify the tool

Classify these situations:

1. A container image contains an outdated vulnerable package -> image/SCA scanning.
2. Someone executes bash inside a running container -> Falco runtime rule.
3. A developer requests a privileged pod that policy forbids -> admission/policy control.
4. Events from dozens of clusters need centralized correlation -> SIEM/observability, with Falco as an input.

Knowledge check (answers included)

• Q: Does Falco replace image scanning? A: No; Falco detects runtime behavior, scanners inspect artifacts and dependencies.
• Q: Does Falco block a suspicious action by default? A: No; its primary product behavior is to alert.
• Q: Why install on every Kubernetes node? A: Workloads can execute on any node, so kernel events must be collected where processes run.

You are ready for Chapter 2 when: you can explain the difference between prevention, scanning, detection and response.

Official references: https://falco.org/docs/ ; https://falco.org/docs/concepts/

---

Chapter 2. Linux and Security Fundamentals – 10 minutes

Overview and learning outcomes

Core goal: Understand the Linux events and process metadata that Falco rules evaluate. You do not need prior kernel programming knowledge.

Processes, IDs, users and permissions

A process is a running program. The kernel assigns a PID (process ID). A process usually has a parent process (PPID), an executable, command-line arguments, and a security identity (UID/GID). Linux file permissions and capabilities control what an identity may do. A container adds isolation mechanisms but still uses its host’s Linux kernel.

Run in your Ubuntu terminal (no sudo needed):

# Show your effective user and numeric IDs
id

# Show a sample of running processes and their parent IDs
ps -eo pid,ppid,user,comm,args | head -n 12

# Show the Linux kernel release
uname -r

# Show permissions on important configuration files
ls -l /etc/passwd /etc/shadow

Verify: id shows UID/GID, ps shows PID/PPID, and /etc/shadow is normally not readable by an ordinary user. The exact output depends on your Linux image.

System calls: the interface between programs and the kernel

A user-space process cannot directly read arbitrary protected files or create arbitrary network sockets. It asks the kernel through a system call (syscall). Falco’s Linux event source captures selected syscall activity and other supported events, then enriches it with useful context.

Event family	Meaning	Security question
execve / execveat	Execute a program	Did a shell or unexpected executable start?
open / openat / openat2	Open a file	Did an unexpected process access a sensitive file?
connect	Initiate a socket connection	Did a process establish an unusual connection?
chmod / permission changes	Change file permissions	Was a protected file modified?
Process-related events	Lifecycle and process metadata	Who launched this program and under which identity?

An application call such as cat /etc/passwd may involve a new process and file-opening system calls. An attempted syscall and a successful syscall are different events/results. Good rules examine event types and, when applicable, the result (evt.failed or evt.rawres).

Container isolation concepts

• A namespace gives processes a restricted view of PIDs, mounts, networks, and other resources.
• A cgroup manages and identifies groups of processes and constrains resource usage.
• A container runtime creates and manages container processes.
• A Kubernetes Pod groups containers and adds orchestration metadata.
• Containers generally share the host kernel; this is why Falco observes their runtime activity at the node.

A Falco rule can use proc.name to select the executing process, user.name to report its identity, fd.name to name a file or socket, and container.id to distinguish containerized activity from host activity.

Core exercise: reason about events

Execute:

# Creates a new process and opens /etc/passwd for reading
cat /etc/passwd > /dev/null

# Opens an ordinary lab file for writing
printf 'hello Falco\n' > /tmp/falco-intro.txt

# View the file you just created
cat /tmp/falco-intro.txt

Predict the events that could be relevant to runtime rules: an executable launch (when a new program is started), a file opened for reading, and a file opened for writing. Exact captured event types can vary by process implementation and the Falco driver’s instrumentation.

Cleanup: rm -f /tmp/falco-intro.txt.

Knowledge check

• Q: Is a PID a process name? A: No; it is a numeric process identifier.
• Q: Is execve itself malicious? A: No; it is an ordinary syscall. Security comes from the circumstances, process and rule.
• Q: Are containers separate kernels? A: Normally not; standard Linux containers share the host kernel.

You are ready for Chapter 3 when: you can explain why a file read or process spawn has meaningful syscall evidence.

Official references: https://falco.org/docs/concepts/event-sources/kernel/ ; https://falco.org/docs/rules/supported-fields/

---

Chapter 3. Falco Architecture and Working – 15 minutes

Overview and learning outcomes

Core goal: Trace a kernel event from an application to a Falco rule and an output destination, and know what modern eBPF, the kernel module, plugins, libscap and libsinsp do.

Architecture diagram

flowchart TD
    A[Linux apps and container processes] --> B[Linux kernel syscalls and tracepoints]
    B --> C[Modern eBPF probe or kernel module]
    C --> D[libscap event collection]
    D --> E[libsinsp inspection and enrichment]
    F[External events through source plugins] --> E
    G[Container metadata plugin] --> E
    E --> H[Falco rules engine]
    I[Rules, macros and lists] --> H
    H --> J{Rule matched?}
    J -->|Yes| K[Alert format and priority]
    J -->|No| L[Continue monitoring]
    K --> M[stdout, file, syslog or HTTP]
    M --> N[Falcosidekick, SIEM or alert workflows]

Reading the diagram: Linux processes make syscalls; a kernel instrumentation driver captures relevant activity; Falco libraries translate and enrich it; rules evaluate events; alerts go to configured outputs. Plugin-sourced events enter through the event-source plugin mechanism rather than through Linux syscall capture. Enrichment plugin fields require the relevant plugin to be loaded/configured.

Modern eBPF vs kernel module

eBPF (extended Berkeley Packet Filter) enables bounded, verified programs to run in the Linux kernel to observe events. Falco’s modern eBPF probe is embedded in the Falco userspace binary. It uses BPF Type Format (BTF) information and a BPF ring buffer; the driver documentation indicates that kernels >=5.8 are usually sufficient, but capability detection matters more than a simple version check. Since Falco 0.38.0, modern eBPF is the default driver; it remains the standard in 0.45.0.

A kernel module is a separately installed kernel component. It can support some older/incompatible kernel environments but requires suitable privileges, a matching driver build and careful version compatibility. A kernel module is not inherently the safer option for a beginner.

Do not use old tutorials that recommend legacy eBPF, gVisor engine, or Falco gRPC output for modern releases. They were removed in Falco 0.44.0. The modern eBPF sensor is different from the removed legacy eBPF implementation.

Topic	Modern eBPF	Kernel module
Beginner recommendation	Default if supported	Fallback if necessary
Kernel-side artifact	Embedded CO-RE probe loaded by Falco	Separate .ko module
Requirements	BPF ring buffer and BTF features	Compatible module / kernel build
Operational issue to investigate	Privileges, BTF, feature support, lockdown	Build/signing, Secure Boot, module mismatch
Monitoring scope	Linux kernel events	Linux kernel events

libscap and libsinsp

libscap handles low-level event capture and driver interaction. libsinsp builds inspectable event and process context used by Falco. The container plugin can enrich events with names/images by communicating with supported container runtimes. On Kubernetes, additional metadata enrichment can associate container IDs with pods and namespaces.

What plugins do

Two important distinctions:

• Event-source plugins supply events such as supported cloud/audit streams rather than syscall events. They must be configured with the appropriate inputs and permissions.
• Field-extraction/enrichment plugins add context to events, such as container or Kubernetes metadata. They do not automatically mean that external audit logs are collected.

For example, a Kubernetes process-execution alert is a syscall event about a pod’s process. A Kubernetes API audit event, if configured through a relevant source plugin, is a control-plane event about an API request. These are not interchangeable.

Working example: concept-to-alert walk-through

1. A process in an authorized test container starts bash.
2. The kernel observes an execve/execveat execution event.
3. Falco’s driver delivers a selected event to the inspection pipeline.
4. The rule engine tests evt.type, proc.name, and container.id.
5. If the condition matches, an alert includes the rule name, priority, command, user, and (if available) container metadata.
6. A configured output sends that alert to journald/stdout or an HTTP receiver such as Falcosidekick.

Core exercise: match architecture to commands

When Falco is installed later, run:

# Inspect the installed binary's Falco, library and driver requirements
sudo falco --version

# List supported field names for syscall-based rules
sudo falco --list=syscall -N | head -n 30

# List the available event types (read only)
sudo falco --list-events | head -n 30

Do not be surprised if some fields show container or plugin-specific metadata on one deployment but not another; what is available depends on the loaded plugins and configuration.

Knowledge check

• Q: Which layer translates kernel driver events for user space? A: libscap.
• Q: Is a Kubernetes audit event the same as a syscall event? A: No; they have distinct sources and meanings.
• Q: Is modern eBPF the same as the removed legacy eBPF probe? A: No.
• Q: Can an eBPF rule be evaluated before the Falco engine sees it? A: Event collection and user-space rule evaluation are separate stages; the kernel probe is not itself a replacement for the Falco rules engine.

You are ready for Chapter 4 when: you can explain the flow syscall -> driver -> Falco inspection -> rule -> alert.

Official references: https://falco.org/docs/concepts/event-sources/kernel/architecture/ ; https://falco.org/docs/concepts/event-sources/kernel/ ; https://falco.org/docs/concepts/plugins/

---

Chapter 4. Falco Installation and Setup – 20 minutes

Overview and learning outcomes

Core goal: Build one functional Falco Linux host, confirm event capture, find its configuration, and safely prepare a custom-rules directory. You will learn installation and service management rather than merely copying commands.

Quick installation and configuration (Windows, macOS and Ubuntu)

A recurring question is, “Can I just run Falco on my MacBook or Windows laptop?” Not for Linux syscall inspection of the native macOS or Windows kernel. Use a real Linux virtual machine; execute all Falco commands inside it. A macOS/Windows terminal is used only to create and enter the VM. Docker Desktop on Windows or macOS is also not the recommended shortcut for the Kubernetes Falco sensor lab; use a cluster with accessible Linux nodes instead.

Path A: Windows 10/11 with Ubuntu using Canonical Multipass

Prerequisites: administrator access; a working hardware virtualization backend (Hyper-V or another provider supported by the installed Multipass version); sufficient available memory and disk space. On corporate-managed computers, check virtualization/VM policy first.

1. Download the current Multipass Windows installer from https://canonical.com/multipass/install . Run it as administrator and complete the official installer. An exact unattended installer command is intentionally not supplied because the installer version and URL are not fixed; do not paste an unverified external URL into an elevated PowerShell session.
2. Open a new Windows PowerShell session (ordinary user after installation) and run:

# Confirm Windows can locate the installed command
multipass version

# Create an Ubuntu 24.04 VM; allocation assumes enough free host resources
multipass launch 24.04 --name falco-lab --cpus 2 --memory 4G --disk 20G

# Verify the instance is Running
multipass list

# Enter the Ubuntu shell. All further apt/Falco commands are run inside Ubuntu.
multipass shell falco-lab

3. Expected: the prompt changes to a Linux shell. In Ubuntu, run uname -s; it should say Linux. Continue with the Ubuntu installation below.
4. If launch fails: verify virtualization is enabled and that the Multipass service and configured virtualization backend are working; inspect multipass info falco-lab. Do not assume WSL2’s kernel offers the required modern eBPF environment; a full VM is more predictable for this exercise.
5. Cleanup later (PowerShell, not Ubuntu): multipass delete falco-lab --purge permanently removes the VM and its files. Back up custom rules first.

Path B: macOS (Intel or Apple Silicon) with Lima

Prerequisites: supported macOS, Homebrew already installed from https://brew.sh/ , virtualization permitted on your Mac. Lima creates an Ubuntu Linux VM; Falco monitors that Ubuntu VM, not all processes running directly on macOS.

Run in macOS Terminal:

# Install Lima using Homebrew
brew install lima

# Confirm Lima CLI was installed
limactl --version

# Create Ubuntu VM using the current Ubuntu LTS Lima template
limactl start --name=falco-lab template://ubuntu-lts

# Check VM state
limactl list

# Enter the Ubuntu VM
limactl shell falco-lab

Expected: you enter a Linux shell. Verify with uname -s, uname -m, and cat /etc/os-release. The ubuntu-lts template selects the configured current LTS image; for a strict 24.04 exercise, verify the image release before continuing or use an explicit 24.04 Lima template appropriate for your Lima installation.

If VM startup fails: check virtualization permission, free disk/RAM and limactl list; check the Lima documentation for platform-specific VM drivers. brew install lima installs the VM manager only, not Falco itself.

Cleanup later (macOS Terminal): limactl stop falco-lab && limactl delete falco-lab removes the test VM once you confirm no needed files remain.

Path C: Ubuntu Linux 24.04 (native host or VM)

Prerequisites: a disposable or explicitly authorized Ubuntu environment with sudo, outbound HTTPS, and a Linux kernel supporting modern eBPF features. Do not install this lab onto a production machine.

Run inside an Ubuntu Bash shell (also applies after entering Windows/macOS Ubuntu VMs):

# 1. Identify the machine and verify it is Linux
uname -s
uname -r
cat /etc/os-release
id

# 2. Prepare standard apt/repository tools
sudo apt-get update
sudo apt-get install -y curl ca-certificates gnupg dialog

# 3. Install Falco's official APT signing key (keyring, not apt-key)
sudo install -d -m 0755 /usr/share/keyrings
curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | \
  sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg

# 4. Add Falco's signed official package repository
printf '%s\n' \
  'deb [signed-by=/usr/share/keyrings/falco-archive-keyring.gpg] https://download.falco.org/packages/deb stable main' | \
  sudo tee /etc/apt/sources.list.d/falcosecurity.list > /dev/null

# 5. Refresh package indexes and inspect the repository candidate
sudo apt-get update
apt-cache policy falco

# 6. Install Falco and accompanying package components
sudo apt-get install -y falco

Interactive installer: select Modern eBPF when prompted for a driver and accept official automatic rules updates for the training VM if offered. Package prompts differ by version and by whether apt is interactive. Falco 0.45.0 is the verified reference release; apt-cache policy falco and falco --version determine the actual package you received. If the repository has published a newer version by the time you run this, consult its release notes before changing example configurations.

Why these commands matter: We install dependencies, import the repository signing key into its own keyring, register a signed package source, refresh apt indexes, then install Falco with systemd integration. sudo is needed for system-wide package operations and runtime inspection permissions.

Start, enable and verify the service

# Report the installed Falco version and companion API information
sudo falco --version

# Start the modern eBPF Falco unit and enable it at boot
sudo systemctl enable --now falco-modern-bpf.service

# Verify it is active
sudo systemctl is-active falco-modern-bpf.service
sudo systemctl status falco-modern-bpf.service --no-pager

# Show recent daemon logs and verify the syscall source opened
sudo journalctl -u falco-modern-bpf.service -n 60 --no-pager

Expected: is-active says active, and logs report rules loading and the syscall source using modern eBPF. If you receive Unit ... could not be found, check the installed unit names with systemctl list-unit-files | grep falco, and confirm which driver/service you selected during package setup. Do not start several Falco instances on a production node just to find a working one.

Inspect files and make a backup

# List the installed configuration/rule paths
sudo ls -la /etc/falco
sudo ls -la /etc/falco/rules.d

# Preserve original configuration before experimenting
sudo cp -a /etc/falco/falco.yaml /etc/falco/falco.yaml.lab-backup

# Search the actual configuration for selected settings
sudo grep -nE '^(engine:|rules_files:|json_output:|stdout_output:|syslog_output:|http_output:)' /etc/falco/falco.yaml

# List available runtime fields and defined rule names
sudo falco --list=syscall -N | head -n 25
sudo falco -L | head -n 45

# Parse configuration/rules without consuming runtime events
sudo falco --dry-run -c /etc/falco/falco.yaml

Important paths (host packages):

/etc/falco/
|-- falco.yaml                 # Main daemon settings
|-- falco_rules.yaml           # Distribution/default rules (do not hand-edit)
|-- falco_rules.local.yaml     # Local override file (if shipped)
`-- rules.d/                   # Recommended home for new rules

Modern distributions can also install rules/plug-ins as managed artifacts via falcoctl. Check rules_files: on your host rather than assuming every installation has the same list. The typical default loads the official file followed by local rules and the rules.d directory; the order matters when overriding/tuning.

Configuration essentials

Setting/feature	Purpose	Learning guidance
engine.kind: modern_ebpf	Choose the modern eBPF capture engine	Default/current recommendation if supported
rules_files	Ordered rule file paths	Put custom rules in rules.d when included
json_output	Structured one-JSON-object-per-alert output	Useful for machines/SIEM
stdout_output.enabled	Print alerts to stdout	Convenient in containers
syslog_output.enabled	Forward to syslog	Convenient on traditional hosts
http_output.enabled / url	Send alerts to HTTP receiver	Falcosidekick or other receiver
metrics.enabled	Emit Falco operational metrics	Turn on for production observation

Do not blindly overwrite the entire production falco.yaml. Make small, reviewed, reversible changes and preserve defaults required by your installed version.

Installation diagnostics

Symptom	First diagnostic	Likely corrective action	Success criterion
apt cannot locate package	apt-cache policy falco; inspect repo file	Check apt update, official URL, signing key, proxy/DNS	Candidate version listed
NO_PUBKEY or signature error	sudo apt-get update output	Reimport official key; verify URL and keyring file permissions	Apt index refresh succeeds
Falco service not found	`systemctl list-unit-files	grep falco`	Confirm package installed and selected driver unit
Falco fails opening eBPF probe	sudo journalctl -u falco-modern-bpf.service -n 100	Check kernel BTF/ring-buffer support, capabilities, kernel lockdown, VM type	Logs show syscall source opened
Permission denied	id; service logs	Run under documented privileges or supported service unit	Falco runs without permission error
VM has no systemd	ps -p 1 -o comm=	Use full Ubuntu VM/native host	PID 1 is systemd
No rule alerts	Chapter 5 checklist	Verify the service, rules loaded, alert output, and event trigger	Controlled test generates alert

Knowledge check

• Q: Why does a Mac or Windows learner need an Ubuntu VM? A: Falco’s syscall driver observes a Linux kernel, not the native macOS/Windows kernel.
• Q: What is the simplest post-install service check? A: sudo systemctl is-active falco-modern-bpf.service, followed by the service journal.
• Q: Where should environment-specific rule files usually live? A: Under /etc/falco/rules.d when that directory is included in rules_files, rather than modifying the vendor-managed rules file.
• Q: What proves syntax/configuration correctness before runtime monitoring? A: falco --dry-run, supplemented by falco -V to validate specific rule files.

Completion checkpoint: sudo falco --version, systemctl is-active and journalctl all work. You know where configuration and custom rules belong.

Official references: https://falco.org/docs/getting-started/falco-linux-quickstart/ ; https://falco.org/docs/installation/ ; https://falco.org/docs/reference/daemon/config-options/ ; https://canonical.com/multipass/install

---

Chapter 5. Falco Runtime Security Detection – 20 minutes

Overview and learning outcomes

Core goal: Trigger a real, controlled runtime detection on a Linux VM and read the alert context. Extend that detection to a container, then examine file modification behavior.

What does a detection actually mean?

A runtime alert is evidence that an observed event matched a rule. It is not automatically proof of compromise. The analyst must determine who initiated the activity, whether it was expected, what the process accessed, and whether its behavior deserves a response. Rules should be evaluated for the workload’s normal baseline.

Lab 5A: read a sensitive file without leaking it

Prerequisite: the Chapter 4 Ubuntu host with Falco active. Terminal A watches Falco logs; Terminal B generates one safe test event.

Terminal A (Ubuntu):

# Stream the journal for Falco's modern eBPF service
sudo journalctl -fu falco-modern-bpf.service

Terminal B (same Ubuntu VM):

# Access sensitive system metadata as root, discarding every output byte
sudo cat /etc/shadow > /dev/null

Expected: on the documented default ruleset, an alert similar to Sensitive file opened for reading by non-trusted program appears, often with priority Warning. It may include file /etc/shadow, process cat, user root, parent process sudo, and container_id=host. Exact wording and activation depends on your installed ruleset and configured outputs.

Why it works: the rule assesses file access by the process, not whether the contents were printed. Redirecting to /dev/null prevents displaying or recording password hashes in training notes.

Verification if the streaming terminal shows no alert:

# Did Falco start and load rules?
sudo systemctl is-active falco-modern-bpf.service
sudo journalctl -u falco-modern-bpf.service -n 100 --no-pager

# Are rules loaded and is the relevant behavior covered?
sudo falco -L | grep -i -E 'sensitive|shadow|shell' | head -n 20

# Search recent journal entries; output path may differ by configuration
sudo journalctl _COMM=falco --since '10 minutes ago' --no-pager | grep -i -E 'Sensitive|shadow'

If the default rule is not shipped/enabled in your installed ruleset, do not assume that Falco itself failed. Chapter 7 includes a deterministic custom rule exercise.

Lab 5B: detect shell execution in a container

Prerequisite: Docker Engine installed inside the Ubuntu lab VM, with the Falco service on that same Linux host. Do not execute Docker containers on macOS/Windows Docker Desktop while monitoring a separate VM; the events will be on a different Linux kernel.

Install a lab Docker Engine if not already present (Ubuntu):

sudo apt-get update
sudo apt-get install -y docker.io
sudo systemctl enable --now docker
sudo docker version
sudo docker info --format '{{.ServerVersion}}'

This uses Ubuntu’s distribution-packaged docker.io for convenience in a disposable lab. For a managed production environment, evaluate the supported vendor runtime/packages, update policy and Falco container-plugin integration separately.

Terminal A:

sudo journalctl -fu falco-modern-bpf.service

Terminal B:

# Download a small published image if not already present
sudo docker pull ubuntu:24.04

# Run a short shell inside a new named Linux container
sudo docker run --rm --name falco-training ubuntu:24.04 \
  bash -c 'id; echo falco-container-lab; sleep 2'

Expected: if a shell-in-container default rule is enabled and matches the exact shell behavior, you should see an execution-related alert. Some default rules require an interactive terminal; in that case try a terminal-attached session in a real terminal:

sudo docker run --rm -it --name falco-interactive ubuntu:24.04 bash
# Inside the container type: id
# Then type: exit

Important: A container shell is not automatically a security incident. It can be an authorized diagnostic action. You will create an explicit custom detection for this action in Chapter 7, without relying on changing names/default behavior of distribution rules.

Lab 5C: understand sensitive file modifications safely

We want to detect writes, but we should not change /etc/shadow or alter system configuration. Chapter 7 creates a rule that treats /tmp/falco-lab-protected.txt as a mock sensitive path. For now, use:

printf 'initial contents\n' > /tmp/falco-lab-protected.txt
printf 'authorized lab modification\n' >> /tmp/falco-lab-protected.txt
cat /tmp/falco-lab-protected.txt

You should not expect a default Falco rule to alert on an ordinary /tmp file. This is intentional: writing benign files is expected Linux behavior. A custom rule gives meaning to your controlled path and proves the policy can be tailored.

How to read an alert

A representative, illustrative text alert might look like:

12:34:56.000000000: Warning Sensitive file opened for reading by non-trusted program
(file=/etc/shadow user=root process=cat parent=sudo container_id=host)

Analyze these fields in order:

1. Time: when the matching event occurred (verify timezone and collection latency).
2. Severity: informational classification configured on the rule; not an automated incident verdict.
3. Rule name: which rule matched; use this to find its source YAML.
4. Process / command: the process and its arguments; beware that command lines may contain secrets.
5. User / parent: identity and process ancestry, useful for differentiating a scheduled job from an injected shell.
6. File / connection target: the resource accessed.
7. Container / Kubernetes context: if available, which workload triggered it.

Common mistakes and fixes

• Nothing appears: check that the test runs on the same Linux node as Falco; confirm the correct service and alert output are in use.
• A container ID appears as host: the event was host-side, or container context was not captured as expected; check namespace/cgroup/container metadata and plugin setup.
• The rule does not fire for bash -c: some default rules are terminal-specific; use an interactive shell or Chapter 7’s explicit custom rule.
• Too many alerts: do not globally disable detections; first identify the offending rule and what normal activity produces the noise.
• Unexpected sensitive content in logs: stop logging unnecessary command arguments and redact in your forwarding/retention layer.

Knowledge check

• Q: Why direct /etc/shadow to /dev/null? A: To trigger a read without displaying the content.
• Q: Does a rule trigger simply because a path looks sensitive? A: No; it needs an event and a matching loaded rule.
• Q: Is a shell inside a container always malicious? A: No; it can be legitimate, though often worth investigating.

Completion checkpoint: You can trigger and find a Falco runtime alert, identify its rule/priority/process and explain why it fired.

Official references: https://falco.org/docs/getting-started/falco-linux-quickstart/ ; https://falco.org/docs/getting-started/falco-docker-quickstart/ ; https://falco.org/docs/reference/rules/default-rules/

---

Chapter 6. Falco Rules – Fundamentals to Intermediate – 25 minutes

Overview and learning outcomes

Core goal: Understand every essential field in a Falco YAML rule, use evt.*, proc.*, fd.* and container.*, and distinguish a rule from a macro and a list. The hands-on exercise explores your installed ruleset before you author your own.

The three YAML building blocks

A rule evaluates an event and emits an alert if the condition is true. A macro stores a reusable condition fragment. A list stores reusable literal values (such as executable names). A rules file is a YAML list of entries, each starting with - rule:, - macro: or - list:.

Example (for explanation; Chapter 7 will install a complete two-rule variant):

- list: lab_shell_programs
  items: [bash, sh, dash]

- macro: lab_process_execution_in_container
  condition: evt.type in (execve, execveat) and container.id != host

- rule: LAB Shell Started in Container
  desc: Detect a shell program started in a test container.
  condition: >
    lab_process_execution_in_container
    and proc.name in (lab_shell_programs)
  output: >
    LAB container shell started
    process=%proc.name command=%proc.cmdline
    user=%user.name container=%container.id
  priority: NOTICE
  tags: [lab, container, process, mitre_execution, T1059]

YAML field	Meaning	Required?
rule	Unique human-readable name	Yes (rule entry)
desc	Human-readable detection description	Yes
condition	Boolean event-matching expression	Yes
output	What the alert should display, including %field placeholders	Yes
priority	Alert severity, aligned to syslog naming	Yes
tags	Searchable/filterable classifications and mapping hints	No
enabled	Whether the rule is active	No; normally true
source	Event source, usually syscall by default	No
exceptions	Structured known-good behaviors excluded from alerting	No
append	Extend an existing rule/macro/list when supported	No

YAML syntax rules: Two-space indentation is easier to read. A dash introduces each list entry. > folds multiple lines into a single logical string, useful for long conditions and outputs. Strings may be quoted to avoid YAML punctuation ambiguity. Never insert tabs for indentation. condition is a Falco expression, not Python, JavaScript, or a Linux shell command.

Understand the field families

Family	Example filter field	Meaning
Event	evt.type	Kernel event type, e.g., execve or openat
Event	evt.is_open_write	Whether an open operation was for writing
Event	evt.failed	Whether an event failed
Process	proc.name	Executable/process name
Process	proc.cmdline	Command-line arguments (avoid exposing secrets unnecessarily)
Process	proc.pid	Process ID
Process	proc.pname	Parent process name
User	user.name	Username
File/socket	fd.name	Full file path or socket descriptor representation
File/socket	fd.directory	Directory of a file descriptor
Container	container.id	Container identifier, or host for host context
Container	container.name	Container name when metadata is available

The field expression in a condition uses the field name, such as proc.name = bash. In output, prefix with %, such as process=%proc.name.

Check what your version supports:

# Every supported syscall event field, with names only
sudo falco --list=syscall -N | less

# Broad list of installed rules and their descriptions
sudo falco -L | less

# A specific available rule name can be inspected with -l
sudo falco -l 'Read sensitive file untrusted'

The last name is used by some official rulesets; if your installed ruleset does not contain it, use a name returned by falco -L. There is no requirement that a particular upstream rule retain the same name forever.

Conditions and operators

The commonly used operators include:

=                Equal
!=               Not equal
in (...)         Value is in a set or Falco list
contains         Substring match
startswith       String prefix match
and              Both expressions must match
or               One expression must match
not              Logical negation
(...)            Explicit evaluation precedence

Example conditions:

# A process called bash or sh started in a container
evt.type in (execve, execveat) and
container.id != host and
proc.name in (bash, sh)

# A file under /etc was opened for writing
evt.type in (open, openat, openat2) and
evt.is_open_write=true and
fd.name startswith /etc/

# A process called python3 ran as root
evt.type in (execve, execveat) and
proc.name = python3 and
user.uid = 0

Avoid unbounded rules, such as proc.name != systemd, that force evaluation over large numbers of unrelated events. The upstream style guide recommends positive evt.type clauses because event selection is an important efficiency technique. Use parentheses for mixed and/or, e.g., evt.type in (execve, execveat) and (proc.name=bash or proc.name=sh). A rule can be syntactically valid yet still be operationally noisy or blind.

Rule priorities and tags

Falco recognizes EMERGENCY, ALERT, CRITICAL, ERROR, WARNING, NOTICE, INFORMATIONAL, and DEBUG, ordered from most severe to least severe. Think of severity as triage guidance, not evidence of proven compromise. ERROR is often appropriate for unauthorized state-changing behavior, while NOTICE may be appropriate for unexpected but ambiguous actions.

tags are strings used to classify/filter a rule. Common tags refer to container, filesystem, process, host, maturity or MITRE ATT&CK tactics and techniques. Mapping a rule to mitre_execution and T1059 helps analysts group detections. A tag is documentation/classification; it does not magically change the event being captured.

Default rules vs local rules

• Default rules are published by the Falco community and may be updated independently via falcoctl artifacts.
• Local rules belong to your environment and address workloads/paths that upstream rules cannot know about.
• Overrides and exceptions refine existing rules. Changes should be reviewable and survive upgrades.
• Rule ordering and overlapping matches matter; more specific matching rules can be shadowed by broader earlier rules in some circumstances. Test the actual emitted alerts, not only whether YAML loads.

Never modify the vendor-supplied falco_rules.yaml directly as your standard workflow. Package and rule artifact updates can overwrite it. Store custom files in version control and deploy through a supported rules path, Helm values, or Operator Rulesfile resources.

Guided exercise: browse the ruleset

# Write the rule names/descriptions to an analysis file in your home directory
sudo falco -L > "$HOME/falco-default-rules.txt"

# Find likely shell, sensitive-file or filesystem protections
grep -niE 'shell|sensitive|write below|container' "$HOME/falco-default-rules.txt" | head -n 30

# Explore the live rules file without editing it
sudo grep -nE '^[-] (rule|macro|list):' /etc/falco/falco_rules.yaml | head -n 35

Expected: you can locate several rule names, read a condition in the installed YAML, and identify at least one referenced macro/list. If falco -L displays fewer names than expected, check which artifacts/rule paths were loaded in the daemon’s configuration.

Knowledge check

• Q: What is the distinction between a list and a macro? A: A list holds values; a macro holds a condition expression.
• Q: Is proc.name=bash enough to detect suspicious shell use? A: Usually not; include event type and context.
• Q: Why specify evt.type positively? A: It helps select relevant events efficiently and avoids needlessly expensive matching.
• Q: What do tags change? A: Classification/filtering, not the underlying syscall or rule result.

Completion checkpoint: You can explain and read a rule, macro, list, output field, priority and MITRE tag before writing any custom code.

Official references: https://falco.org/docs/concepts/rules/ ; https://falco.org/docs/concepts/rules/basic-elements/ ; https://falco.org/docs/concepts/rules/style-guide/ ; https://falco.org/docs/rules/supported-fields/

---

Chapter 7. Writing and Testing Custom Falco Rules – 25 minutes

Overview and learning outcomes

Core goal: Write a complete custom rules file, validate its syntax, deploy it, trigger two different behaviors, and debug a rule that does not fire. This is the most valuable beginner-to-intermediate lab.

File layout for our project

/etc/falco/
`-- rules.d/
    `-- 90-falco-training.yaml   # Two complete custom detections

/tmp/
`-- falco-lab-protected.txt     # Safe mock sensitive file

Step 1: create a standalone custom-rules file

Ubuntu terminal; sudo required to write into /etc/falco/rules.d:

# Ensure rules.d exists (some installations create it automatically)
sudo install -d -m 0755 /etc/falco/rules.d

# Copy-ready rules: creates both a shell detection and a controlled file-write detection
sudo tee /etc/falco/rules.d/90-falco-training.yaml > /dev/null <<'YAML'
- list: lab_shell_binaries
  items: [bash, sh, dash]

- macro: lab_shell_exec_in_container
  condition: >
    evt.type in (execve, execveat)
    and container.id != host

- rule: LAB Shell Started in Container
  desc: Detect the execution of a shell program inside a Linux test container.
  condition: >
    lab_shell_exec_in_container
    and proc.name in (lab_shell_binaries)
  output: >
    LAB shell started in container
    process=%proc.name command=%proc.cmdline
    user=%user.name container_id=%container.id container_name=%container.name
  priority: NOTICE
  tags: [lab, container, process, mitre_execution, T1059]

- rule: LAB Protected File Opened for Writing
  desc: Detect a write-open operation on a harmless mock sensitive file in /tmp.
  condition: >
    evt.type in (open, openat, openat2)
    and evt.is_open_write=true
    and fd.name = /tmp/falco-lab-protected.txt
  output: >
    LAB protected file opened for writing
    file=%fd.name process=%proc.name
    user=%user.name pid=%proc.pid command=%proc.cmdline
  priority: WARNING
  tags: [lab, filesystem, file_access]
YAML

# Display the saved file to confirm every line was written
sudo sed -n '1,100p' /etc/falco/rules.d/90-falco-training.yaml

Why these specific conditions? The process rule selects execution syscalls and only container contexts, then requires a shell executable. The file rule selects open/openat/openat2 events, requires a write-open flag, and targets one harmless absolute filename. Neither rule searches every event or operates on actual sensitive system data.

Step 2: validate before applying

# Validate all required default rules plus our standalone lab rule file.
# If your installed config has different rule paths, use those paths instead.
sudo falco -V /etc/falco/falco_rules.yaml \
  -V /etc/falco/rules.d/90-falco-training.yaml

# Validate the live configuration and its selected rules files
sudo falco --dry-run -c /etc/falco/falco.yaml

Expected: Falco returns an exit code of zero and no rule-loading validation errors. -V validates the specified rules files and exits; --dry-run verifies configuration/rule setup without consuming kernel events. If a macro or list referenced by your rule is in another file, include that dependency during validation. Our lab macro/list are self-contained.

If validation reports unexpected key, unknown filtercheck, invalid syntax, or a YAML mapping error, compare indentation, rule names and Falco’s version-specific supported fields. A parser accepting YAML is not enough: Falco must also accept the Falco expression language.

Step 3: check rule search path, then load changes

# Confirm your live config includes rules.d (either directly or through another file)
sudo grep -n -A8 '^rules_files:' /etc/falco/falco.yaml

# Restart the appropriate test host daemon to ensure rules reload
sudo systemctl restart falco-modern-bpf.service

# Verify startup and identify whether the custom rules file was loaded
sudo systemctl is-active falco-modern-bpf.service
sudo journalctl -u falco-modern-bpf.service -n 100 --no-pager | \
  grep -iE 'rules|90-falco-training|error|warning'

If rules.d is not in the selected rule files, add the directory to rules_files in a safely reviewed copy of the config, then validate again. Do not assume merely creating a file makes Falco consume it. Restarting the daemon can briefly interrupt event collection; this is acceptable only in a disposable lab. Production uses a controlled rollout/reload strategy.

Step 4: trigger the file-write rule (most deterministic first)

Terminal A:

sudo journalctl -fu falco-modern-bpf.service

Terminal B:

# Create/open precisely the protected mock file for writing
printf 'LAB write 1\n' > /tmp/falco-lab-protected.txt

# Append a second line (another write-open operation)
printf 'LAB write 2\n' >> /tmp/falco-lab-protected.txt

# Verify contents; reading the file should NOT satisfy the write-open rule
cat /tmp/falco-lab-protected.txt

Expected: one or more LAB Protected File Opened for Writing alerts. The exact process name can be the shell itself because redirection is performed by the shell. If a different path is resolved due to mount namespaces or redirection behavior, check fd.name in diagnostic logs and confirm that the rule path exactly matches the actual path.

Negative test (important): cat /tmp/falco-lab-protected.txt must not trigger the write-open rule (it could independently trigger other enabled rules). A useful rule is validated by both a positive trigger and a negative control.

Step 5: trigger shell execution in a container

Terminal B:

# Docker must run on the Linux host observed by Falco
sudo docker run --rm --name falco-shell-test ubuntu:24.04 \
  bash -c 'echo custom-shell-rule-test; id'

Expected: a LAB Shell Started in Container notice if the rule is loaded, the container metadata event is recognized, and the shell execution is not preempted by an earlier more general rule. If you see a different default shell rule, that is evidence that detection is operational, but you still need to validate your own rule separately. Refer to the next diagnostic procedure.

Step 6: troubleshoot a rule that fails to trigger

Debug in this order:

1. Source: Is the process actually running on this Linux kernel? sudo docker info, uname -r.
2. Daemon: Is Falco active? sudo systemctl is-active falco-modern-bpf.service.
3. Loaded rules: Did startup mention 90-falco-training.yaml? Check rules_files and journal logs.
4. Syntax: Does falco -V succeed with the required dependencies?
5. Event filters: Is the event type correct (execve/execveat, open/openat/openat2)? Are you looking at successful vs failed events?
6. Context: Does container.id equal host unexpectedly? Is the container plugin providing metadata? Does fd.name match the path?
7. Ordering: Is a broader earlier rule matching the same events? Review matching rules and load order.
8. Output: Are alerts going to stdout/syslog/HTTP while you are watching another destination?

Optional advanced isolation test (only on the disposable Ubuntu VM): Stop the host service temporarily and run Falco with only your custom rules in a foreground terminal. This separates your rule logic from overlapping default rules, but is an operational change; do not do it on a shared monitored host.

# Terminal A: temporarily stop the test service, then run a dedicated test ruleset
sudo systemctl stop falco-modern-bpf.service
sudo falco -c /etc/falco/falco.yaml \
  -r /etc/falco/rules.d/90-falco-training.yaml \
  -o engine.kind=modern_ebpf \
  -o stdout_output.enabled=true

# Terminal B: repeat the Docker and mock-file lab triggers above.
# Terminal A: press Ctrl-C to stop foreground Falco when finished.

# Restore normal monitored service afterward (Ubuntu terminal)
sudo systemctl start falco-modern-bpf.service
sudo systemctl is-active falco-modern-bpf.service

If the foreground test fails to start, review its error text rather than repeatedly restarting. This method still relies on your installed configuration’s plugin and engine settings. It is not guaranteed to reconstruct container metadata if plugins are absent.

Step 7: how an intermediate engineer improves the rule

For a real workload, ask these questions before shipping:

• Which container image or Kubernetes namespace is allowed to run this shell, if any?
• Should only interactive shells be suspicious, or also programmatic shells?
• Should the rule ignore a known maintenance job but still alert for all other workloads?
• Is proc.name stable enough, or should you also inspect executable path, parent process, command or container image?
• Can a malicious user avoid this rule by launching a different interpreter, direct syscalls or a renamed executable?
• How many legitimate alerts per day appear during a staging observation window?

Production sample: A Go or Java application container unexpectedly launching bash is often more noteworthy than a CI runner container designed to execute shell scripts. The intermediate skill is writing the same syscall rule but applying workload-specific policy and meaningful exceptions.

Step 8: cleanup and rollback

# Remove the mock data file (the rule remains installed for further chapters)
rm -f /tmp/falco-lab-protected.txt

# Optional: remove lab rules and return to a default-rules host
# WARNING: this removes the two training detections created in this chapter.
# sudo rm -f /etc/falco/rules.d/90-falco-training.yaml
# sudo systemctl restart falco-modern-bpf.service

For most learners, keep 90-falco-training.yaml until completing Chapter 10, because it will be used to practice exceptions. Source-control the custom file and keep your human-written rule changes separate from the automatically updated upstream rules.

Knowledge check

• Q: Why test a negative case? A: To confirm a rule excludes non-targeted behavior, not merely that some alert exists.
• Q: Why validate configuration and rules before restarting? A: To catch parser/configuration errors without interrupting monitoring.
• Q: Why might a custom rule not fire despite valid YAML? A: The event/field/context does not match, the file is not loaded, another rule takes precedence, or the output is not observed.
• Q: Why not build the lab on /etc/shadow writes? A: Real system files are sensitive and modification risks harming the host; a mock path is enough to prove detection logic.

Completion checkpoint: You have authored a rules file, validated it, loaded it, tested a true and false case, and can explain a debugging sequence.

Official references: https://falco.org/docs/concepts/rules/custom-ruleset/ ; https://falco.org/docs/reference/daemon/cli-arguments/ ; https://falco.org/docs/concepts/rules/conditions/ ; https://falco.org/docs/concepts/rules/style-guide/

---

Chapter 8. Falco Alerts and Integrations – 10 minutes

Overview and learning outcomes

Core goal: Understand alert transports, switch between text/JSON formats, learn Falcosidekick’s function, and identify safe integration patterns for Slack, SIEM and monitoring tools.

From event to alert to integration

flowchart LR
    A[Falco rule match] --> B[Priority and structured output]
    B --> C[stdout or file]
    B --> D[Syslog / journald]
    B --> E[HTTP/HTTPS]
    E --> F[Falcosidekick]
    F --> G[Slack / Teams / Chat]
    F --> H[SIEM and log systems]
    F --> I[Prometheus / Datadog / alerting]

Falco supports standard output, file output, syslog, spawned program output, and HTTP/HTTPS output. Falco gRPC output is no longer available in 0.45.0 (removed in 0.44.0), so avoid older examples centered on it.

Plain text vs JSON output

Text is human-friendly, while JSON is machine-friendly. A simplified illustrative JSON alert (keys from the official JSON output format) might look like:

{
  "time": "2026-10-11T02:30:00.000000000Z",
  "rule": "LAB Protected File Opened for Writing",
  "priority": "Warning",
  "output": "LAB protected file opened for writing file=/tmp/falco-lab-protected.txt",
  "hostname": "ubuntu-lab",
  "tags": ["lab", "filesystem"],
  "output_fields": {
    "fd.name": "/tmp/falco-lab-protected.txt",
    "proc.name": "bash",
    "user.name": "ubuntu"
  }
}

The illustration is not the exact JSON produced by every release/rule; the real values and fields depend on the rule’s output placeholders and available plugin fields. The official documented keys include time, rule, priority, output, hostname, tags and output_fields.

Lab: enable JSON in an isolated foreground run

Do not start a second competing Falco process against a shared/production host. On the disposable Ubuntu VM, use the isolation procedure from Chapter 7:

# Terminal A, training VM only
sudo systemctl stop falco-modern-bpf.service
sudo falco -c /etc/falco/falco.yaml \
  -r /etc/falco/rules.d/90-falco-training.yaml \
  -o engine.kind=modern_ebpf \
  -o json_output=true \
  -o stdout_output.enabled=true

# Terminal B: safely trigger one custom rule
printf 'JSON test\n' >> /tmp/falco-lab-protected.txt

# Read a JSON object in Terminal A; Ctrl-C to exit foreground Falco.
# Restore the service when the test is finished.
sudo systemctl start falco-modern-bpf.service

Expected: the same rule name and event context now appear as JSON, generally one JSON object per alert line. You may optionally pipe captured JSON lines into jq for analysis once the format is configured, but do not assume all journald messages are pure JSON (daemon startup notices often are not).

Falcosidekick: a forwarder, not an additional kernel monitor

Falcosidekick accepts Falco alerts via HTTP and forwards them to configured destinations. It can add fields, filter by priorities and expose useful metrics. It is a delivery layer: Falco detects; Falcosidekick routes; the receiving system stores, visualizes or triggers action.

A typical Kubernetes-side configuration concept is:

# Falco configuration fragment only; not a complete replacement falco.yaml
json_output: true
http_output:
  enabled: true
  url: http://falco-falcosidekick.falco.svc.cluster.local:2801

The exact service name depends on your Helm/Operator release, namespace and chart configuration. Check kubectl get svc -n falco before committing a URL. Use only authorized, TLS-protected endpoints across trust boundaries; HTTP may be acceptable inside a narrowly scoped local lab but is not a general recommendation for untrusted networks.

Quick chart-based lab integration (only if you used Helm deployment in Chapter 9):

# Deploy the chart's optional Falcosidekick component with Falco
helm upgrade falco falcosecurity/falco \
  --namespace falco \
  --set falcosidekick.enabled=true \
  --set tty=true

# Look for sidekick service and pod names before checking logs
kubectl get pods,svc -n falco

Warning: a helm upgrade with only these flags can drop previous CLI-only customizations; for real deployments use a checked-in values file and helm get values -n falco falco -a to preserve desired state. Validate the currently installed chart version and supported keys before use. Do not create Slack/Datadog credentials merely for this study exercise.

Integration patterns and guardrails

Destination	Why use it	Careful about
Local journal/stdout	First lab and direct troubleshooting	Journal retention and access
Falcosidekick -> Slack/Teams	Operational awareness	Rate limits, alert noise, webhook secrets
Falcosidekick -> SIEM	Correlation, long-term investigation	Field normalization, TLS, identity, retention
Falcosidekick -> Datadog/Prometheus	Operational views and monitoring	Distinguish Falco alerts from sensor health counters
Custom incident response automation	Enrichment, tickets, containment	Explicit approvals, safeguards and rollback

Security note: Falco outputs can contain filenames, usernames, commands and arguments; command arguments sometimes include tokens or credentials. Treat alert streams as security-sensitive telemetry. Use authorization, secret stores, transport security, sensible retention and redaction where practical. Never paste actual webhook tokens into a tutorial or into a Kubernetes ConfigMap in plaintext.

Knowledge check

• Q: Does JSON change the rule condition? A: No; it changes how the alert is represented.
• Q: Does Falcosidekick read kernel syscalls directly? A: No; it forwards alerts produced by Falco.
• Q: Can the 2026 Falco engine use gRPC output examples from old tutorials? A: Not on the 0.45.0 release.

Completion checkpoint: You can find alerts in logs, recognize JSON fields and describe how to safely forward alerts.

Official references: https://falco.org/docs/concepts/outputs/ ; https://falco.org/docs/concepts/outputs/channels/ ; https://falco.org/docs/concepts/outputs/forwarding/ ; https://falco.org/blog/falco-0-44-0/

---

Chapter 9. Falco with Kubernetes – 20 minutes

Overview and learning outcomes

Core goal: Understand the node-level DaemonSet model; deploy Falco using the fast Helm lab method; inspect a pod-generated alert; and learn the current operator-based management model. Also understand how Kubernetes metadata joins a syscall event.

Prerequisites and operational boundaries

• An authorized Kubernetes test cluster with Linux worker nodes, not a production cluster.
• The cluster nodes must support the chosen Falco sensor’s kernel features/privileges.
• kubectl configured to the intended context; helm installed locally.
• Permissions to create namespaces, DaemonSets, ServiceAccounts, roles and potentially privileged pods. For the operator, Kubernetes 1.29+ is required by the official guide.
• A network path for pulling chart images and rule artifacts.

Host platform distinction: Helm and kubectl can run on Windows, macOS or Ubuntu because they are remote cluster administration clients. Falco itself runs on the cluster’s Linux workers. Docker Desktop Kubernetes on macOS/Windows is not the official Linux-node Falco lab target; use a suitable test Linux-node cluster instead.

Preflight in the terminal where kubectl is configured:

# Confirm the intended cluster and current context before installing anything
kubectl config current-context
kubectl cluster-info
kubectl get nodes -o wide
kubectl auth can-i create daemonsets -n kube-system

# Verify the CLI dependencies are installed
kubectl version --client
helm version

Do not proceed if the context unexpectedly points at production. The DaemonSet experiment grants deep node visibility and must pass your organization’s security review when performed outside an isolated lab.

Kubernetes architecture

flowchart TD
    K[Kubernetes API server] --> M[Workload / Pod metadata]
    S[Node 1: application pods] --> F1[Node 1: Falco sensor pod]
    T[Node 2: application pods] --> F2[Node 2: Falco sensor pod]
    F1 --> R[Falco rule matches]
    F2 --> R
    M --> P[k8s metadata plugin + collector if configured]
    P --> R
    R --> O[Logs / Falcosidekick / SIEM]

Each node-level Falco instance observes Linux events on its node. Workload metadata may be attached by the container plugin and, when configured, the Kubernetes metadata plugin/collector. Kubernetes API audit events require a different plugin/source integration; simply running Falco as a DaemonSet does not automatically ingest all API audit records.

Deployment choice in October 2026

Falco Operator is the officially recommended management model as of 2026, with Falco, Rulesfile, Plugin, Config and Component custom resources (CRDs). The classic Falco Helm chart remains supported and is simpler for a short training lab. Both should be understood; do one primary installation on a test cluster, not two overlapping copies.

Choice	Advantages	Use in this guide
Falco Helm chart	Short install, easy start, broad familiarity	Core 20-minute lab
Falco Operator	Declarative management of instances, rules, plugins and configurations	Intermediate extension and preferred long-term design

Path A: Core 20-minute lab using the supported Falco Helm chart

Step A1: install Falco

# Register the official Falco Helm charts repository
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

# Inspect the chart version available in your current repository
helm show chart falcosecurity/falco | head -n 30

# Install into a separate namespace
helm install falco falcosecurity/falco \
  --namespace falco --create-namespace --set tty=true

# Inspect release and DaemonSet placement
helm list -n falco
kubectl get daemonsets,pods -n falco -o wide

# Wait for the Falco pods to be ready
kubectl wait pods --for=condition=Ready --all -n falco --timeout=180s

Expected: the number of ready Falco DaemonSet pods reflects schedulable, supported nodes, subject to chart scheduling/tolerations and DaemonSet rules. One falco pod per compatible node is the usual design, not a general guarantee. If pods remain Pending, inspect scheduling, privileges, kernel compatibility, image pull access and tolerations.

Troubleshooting commands:

kubectl get events -n falco --sort-by=.lastTimestamp | tail -n 25
kubectl describe daemonset -n falco falco
kubectl get pods -n falco -o wide
kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --since=10m --prefix=true

Depending on the chart and deployment flags, the daemonset/pod/label names may differ. Read helm list, kubectl get ds -n falco and actual pod labels rather than blindly using the name falco for everything.

Step A2: deploy one harmless workload

# Isolate the demonstration application from existing namespaces
kubectl create namespace falco-lab

# Create a single small web server for triggering a controlled event
kubectl -n falco-lab create deployment nginx --image=nginx:stable

# Wait until its pod is ready
kubectl -n falco-lab wait --for=condition=Ready pod \
  -l app=nginx --timeout=180s

# Inspect the assigned node so you can correlate it with one Falco pod
kubectl -n falco-lab get pods -l app=nginx -o wide

Step A3: generate a sensitive-file read in the test pod

# Execute this command ONLY against the demo workload in the test namespace.
# It reads a sensitive file without printing it to your terminal.
kubectl -n falco-lab exec deployment/nginx -- \
  sh -c 'cat /etc/shadow > /dev/null'

# Search recent Falco pod logs for any matching sensitivity alert
kubectl -n falco logs -l app.kubernetes.io/name=falco \
  -c falco --since=10m --prefix=true | \
  grep -iE 'sensitive|shadow|warning'

Expected: an alert from the Falco pod running on the same node as the nginx pod, if the corresponding default rule is loaded. The output may contain container_id, container_image, and potentially k8s_ns/k8s_pod_name if the enrichment integration is configured. Some image variants or security policies may block reading /etc/shadow; that failure itself is not proof of a Falco problem. The base nginx image might also vary over time. If cat is absent, use an image with the necessary utility and keep the exercise on an isolated test namespace.

If you receive no alert: check Falco startup/rule logs, target node scheduling, ruleset maturity/enablement, container runtime socket and metadata plugin, and whether your query includes the right namespace/pod. The default ruleset may change; a dedicated custom detection removes uncertainty.

Step A4: add a custom rule with Helm values

The official chart supports the customRules value. Save this complete values file on the machine running Helm:

cat > falco-custom-values.yaml <<'YAML'
customRules:
  custom-lab.yaml: |-
    - rule: LAB Kubernetes Shell Exec
      desc: Detect shell execution inside a test Kubernetes container.
      condition: >
        evt.type in (execve, execveat)
        and container.id != host
        and proc.name in (bash, sh)
      output: >
        LAB Kubernetes container shell started
        process=%proc.name user=%user.name
        container=%container.id command=%proc.cmdline
      priority: NOTICE
      tags: [lab, kubernetes, container, mitre_execution, T1059]
YAML

# Apply the custom-rule settings to the existing Helm release
helm upgrade falco falcosecurity/falco \
  --namespace falco -f falco-custom-values.yaml --set tty=true

# Wait for the chart components to settle
kubectl -n falco get pods
kubectl -n falco wait pods --for=condition=Ready --all --timeout=180s

# Trigger shell execution from the lab workload and review alerts
kubectl -n falco-lab exec deployment/nginx -- sh -c 'echo shell-lab'
kubectl -n falco logs -l app.kubernetes.io/name=falco \
  -c falco --since=10m --prefix=true | grep 'LAB Kubernetes'

Why this matters: the Helm chart packages the custom YAML into a Kubernetes ConfigMap and makes it available to Falco. In production, preserve your complete approved values file instead of repeatedly adding independent CLI --set options. The exact upgrade behavior of existing values should be tested and reviewed for your chart version.

Step A5: clean up only resources from this lab

# Deletes the demonstration nginx deployment and namespace
kubectl delete namespace falco-lab

# Removes the Falco Helm release (node monitoring stops)
helm uninstall falco -n falco

# Optional: remove its now-empty namespace after verifying contents
kubectl delete namespace falco

# Remove the local chart override file if not needed
rm -f falco-custom-values.yaml

Impact: uninstalling Falco stops monitoring the test cluster. Never apply cleanup to a shared cluster that relies on this Falco release for security coverage.

Path B: Intermediate extension – recommended Falco Operator

This is an alternative deployment; use on a clean test cluster or after cleaning up Path A. The Operator manages Falco as Kubernetes custom resources. For current operator releases, inspect helm show chart ... and the operator version matrix: the default deployed Falco binary can lag the newest Falco engine unless you explicitly pin or update it.

# Install the operator controller and its CRDs
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm show chart falcosecurity/falco-operator | head -n 30
helm install falco-operator falcosecurity/falco-operator \
  --namespace falco-operator --create-namespace

# Check controller health
kubectl wait pods -n falco-operator \
  --for=condition=Ready --all --timeout=180s

# Create a dedicated namespace for Falco instances and artifacts
kubectl create namespace falco

Create the Falco instance (run from a Bash-compatible shell with kubectl context set):

cat <<'YAML' | kubectl apply -n falco -f -
apiVersion: instance.falcosecurity.dev/v1alpha1
kind: Falco
metadata:
  name: falco
spec: {}
YAML

Install the container enrichment plugin (required for several default container.* fields):

cat <<'YAML' | kubectl apply -n falco -f -
apiVersion: artifact.falcosecurity.dev/v1alpha1
kind: Plugin
metadata:
  name: container
spec:
  ociArtifact:
    image:
      repository: falcosecurity/plugins/plugin/container
      tag: latest
    registry:
      name: ghcr.io
YAML

Install the official rules artifact:

cat <<'YAML' | kubectl apply -n falco -f -
apiVersion: artifact.falcosecurity.dev/v1alpha1
kind: Rulesfile
metadata:
  name: falco-rules
spec:
  ociArtifact:
    image:
      repository: falcosecurity/rules/falco-rules
      tag: latest
    registry:
      name: ghcr.io
  priority: 50
YAML

# Verify reconciliation and workload status
kubectl get falco,plugins,rulesfiles,configs,components -n falco
kubectl get pods -n falco -o wide
kubectl get daemonsets -n falco -o wide

Expected: Falco instance, plugin and rules artifact resources become reconciled and available. Do not omit plugin/rules setup in a manual Operator install: the Falco instance alone can run without the rule content needed to generate useful alerts. The official alternative full-stack quickstart includes the k8smeta plugin, metadata collector, Falcosidekick and UI; see the official operator guide for exact deployment and cleanup procedures.

Inspect actual Falco image version:

# Inspect the running image. This is crucial because Operator defaults can lag Falco releases.
kubectl get daemonsets -n falco \
  -o jsonpath='{range .items[*]}{.metadata.name}{" -> "}{.spec.template.spec.containers[*].image}{"\n"}{end}'

For real environments, verify and pin both operator/chart and Falco engine versions in tested configuration, and update the deployment per supported CRD API rather than assuming latest always means Falco 0.45.0.

Operator removal, lab only: delete Rulesfile, Plugin, and Config objects before deleting the Falco instance. Then delete remaining Component resources and the operator Helm release, following the official ordered removal guide. Be especially cautious with operator CRDs: deleting a CRD removes every custom resource of that type, so do not do it in a shared cluster.

Kubernetes metadata: what is and is not automatic

A process-level event starts with the node, and a container ID can be associated with a container runtime. Kubernetes namespace, pod and workload details may require k8smeta and its collector (or another supported metadata setup). If k8s.pod.* or Kubernetes output fields display <NA>, verify enrichment rather than claiming that Falco cannot see any syscalls. Also distinguish Kubernetes API audit-log ingestion from Kubernetes workload metadata enrichment; these are separate functions.

Production guardrails (intermediate)

• Validate compatibility with your exact node OS, kernel, runtime, managed-service constraints and privilege policy.
• Confirm one sensor is scheduled on each node type you intend to monitor; use tolerations/node selectors carefully.
• Check that privileged DaemonSets are allowed and can load modern eBPF, or plan a supported alternative architecture.
• Do not assume Amazon EKS Auto Mode, Fargate or another managed/abstracted compute mode supports every privileged host-level DaemonSet. Validate with vendor documentation and a staging proof-of-concept; these environments have different node control restrictions.
• Plan immutable-version upgrades, metadata plugin availability, secret handling, node reboots, cost and event/drop metrics.
• Distinguish kube-system infrastructure noise from application workload detections; tune only after observing the workload baseline.

Knowledge check

• Q: Why does Falco normally run as a DaemonSet? A: It captures per-node Linux events; a node-local sensor is needed for workloads on that node.
• Q: What is the currently preferred Kubernetes management path? A: The Falco Operator, with Helm still supported.
• Q: Does loading the container plugin automatically mean Kubernetes API audit events are ingested? A: No.
• Q: Why inspect the actual image version after installing the operator? A: The operator’s default Falco engine version may differ from the newest upstream release.

Completion checkpoint: You can install and verify a Falco DaemonSet on a Linux-node test cluster, trigger a pod-related event, find the correct sensor logs, and explain the Operator’s custom-resource model.

Official references: https://falco.org/docs/getting-started/falco-kubernetes-quickstart/ ; https://falco.org/docs/setup/operator/ ; https://falco.org/docs/getting-started/deployment/ ; https://github.com/falcosecurity/falco-operator/blob/main/docs/version-matrix.md

---

Chapter 10. Rule Tuning and False Positives – 15 minutes

Overview and learning outcomes

Core goal: Explain and reduce false positives without creating dangerous blind spots; add a targeted exception; understand rule enablement, alert noise, severity and dropped event counters.

The four outcomes of a detection rule

Reality	Falco alerts	Falco stays silent
Suspicious behavior happened	True positive	False negative
Legitimate behavior happened	False positive	True negative

A false positive could be a known maintenance script intentionally changing a protected configuration file. A false negative could be a malicious interpreter that an over-specific bash-only rule failed to catch. Neither outcome is fixed by indiscriminately disabling all noisy rules.

Recommended tuning sequence

1. Confirm the detection and its precise output: rule, process, parent process, user, file, image and node.
2. Investigate the activity: was it an approved operation, a routine job, or an unexplained anomaly?
3. Measure frequency and blast radius: how many hosts/containers and events per minute?
4. Write a narrow exception identifying an approved actor and its permitted target/action.
5. Retest positive and negative cases: an approved actor should be excluded; a different actor must still alert.
6. Review and version the change: record owner, reason, expiration/revalidation and expected tests.
7. Monitor detection quality over time: new application versions and changes in normal workloads can make old exceptions inaccurate.

Lab 10A: add an exception to our custom file-write rule

Prerequisite: keep Chapter 7’s 90-falco-training.yaml installed and active on the Ubuntu lab VM. This chapter creates a second rules file that overrides the rule defined earlier. The 95- prefix sorts after 90-; confirm your rules directory loads in filename order and appears after default rules.

Goal: exclude writes made to the mock file by the known bash training process, while still alerting when an alternative process writes to that same file.

# Create a separate override file; do not edit default Falco rules
sudo tee /etc/falco/rules.d/95-falco-training-exceptions.yaml > /dev/null <<'YAML'
- rule: LAB Protected File Opened for Writing
  exceptions:
    - name: allowed_training_bash_writer
      fields: [proc.name, fd.name]
      comps: [=, =]
      values:
        - [bash, /tmp/falco-lab-protected.txt]
  override:
    exceptions: append
YAML

# Validate the base rule AND its later exception override in one pass
sudo falco -V /etc/falco/rules.d/90-falco-training.yaml \
  -V /etc/falco/rules.d/95-falco-training-exceptions.yaml

# Validate the configured full ruleset
sudo falco --dry-run -c /etc/falco/falco.yaml

# Reload via a controlled service restart on the lab host
sudo systemctl restart falco-modern-bpf.service
sudo systemctl is-active falco-modern-bpf.service

Why override.exceptions: append? The modern rule override format lets a later file add exception definitions or values without replacing the original entire rule. The legacy append: true syntax is deprecated; new policy should use override.

Watch alerts in Terminal A:

sudo journalctl -fu falco-modern-bpf.service

Run two test writers from Terminal B:

# Known-good actor: explicitly execute Bash to perform the redirection.
# The specified (bash, mock-file) tuple should now be excluded.
bash -c 'printf "trusted bash update\n" >> /tmp/falco-lab-protected.txt'

# Different actor: Python opens the exact same mock file for append.
# An alert SHOULD still be produced if python3 is installed.
python3 -c "open('/tmp/falco-lab-protected.txt', 'a').write('python change\\n')"

# Inspect the mock file itself
cat /tmp/falco-lab-protected.txt

Expected: bash + this exact mock file no longer triggers this LAB rule; python3 + the same file still triggers LAB Protected File Opened for Writing. Other enabled Falco rules may independently alert, and that does not mean the exception failed. If both cases alert under the LAB rule, ensure the later file was loaded and the observed process names actually match. If both are silent, confirm the original rule is still loaded and producing detections.

Important distinction: This lab is intended to demonstrate a narrow two-field exception, not a general production recommendation to trust every bash process. A real exception should typically also consider workload identity, executable path, container image, namespace, parent identity or a restricted deployment/maintenance window when available.

How to enable and disable rules selectively

A rule may include enabled: false in its definition or be disabled by a later rule override. Prefer documented Falco rule controls/overrides over deleting upstream YAML. Example illustration only for a later override file:

- rule: LAB Shell Started in Container
  enabled: false
  override:
    enabled: replace

This disables the lab rule when supported by the installed version’s override semantics. Do not apply it during the positive/negative labs, and do not globally disable Falco because one particular rule has too many alerts. In production, tune or restrict the specific rule(s), use mature/stable rulesets deliberately, and test changes with a controlled rollout.

Falco’s upstream rules have different maturity levels: stable rules are commonly loaded by default; incubating or sandbox rules require an explicit decision to enable and tune. A noisy experiment should not silently redefine the detection baseline across all clusters.

Severity, alert suppression and production noise

Increasing a rule’s severity does not make its logic more accurate. Priority supports incident routing and response, but the condition determines what is detected. Good severity policy considers impact, confidence and the identity of the workload. A critical asset running a shell unexpectedly can deserve more attention than a developer sandbox, even when the underlying syscall is identical.

Keep a tuning register:

Field	Example
Rule	LAB Shell Started in Container
False-positive cause	Approved diagnostic session
Workload	Development test container only
Proposed exception	Verified maintenance actor + image + command restrictions
Owner	Runtime security team
Review date	Next monthly detection review
Negative-control test	Unknown process still alerts

Event drops: alert noise vs missing data

Alert count is not the same thing as captured event count. A noisy rule can cause alert volume, while kernel event drops can indicate that the capture pipeline cannot keep up with load, creating blind spots.

Falco metrics can measure event rates, kernel-side drop counters, Falco output queue drops, resource utilization and rule counters. With Falco 0.45’s modern eBPF driver, additional auxiliary-map drop counters can help distinguish causes of loss.

Low-risk inspection in the lab (no config modifications):

sudo falco --version
sudo grep -n -A20 '^metrics:' /etc/falco/falco.yaml | head -n 30
sudo journalctl -u falco-modern-bpf.service --since '15 minutes ago' --no-pager | \
  grep -iE 'drop|buffer|backpressure|metric' | tail -n 25

Production metrics planning (config fragment, not a full falco.yaml):

metrics:
  enabled: true
  interval: 1m
  output_rule: true
  rules_counters_enabled: true
  resource_utilization_enabled: true
  state_counters_enabled: true
  kernel_event_counters_enabled: true

Read Chapter 11 before applying monitoring changes. High event rates need observation and benchmark-driven tuning, not arbitrary bigger buffers. For example, scap.n_drops captures kernel-side event losses; changes should be studied alongside event rate, CPU, memory, alert queue, and application performance. Increasing ring-buffer capacity consumes memory and does not fix every kind of drop.

Undo the exception lab

# Remove only the experiment's override (the base two lab detections remain)
sudo rm -f /etc/falco/rules.d/95-falco-training-exceptions.yaml
sudo falco --dry-run -c /etc/falco/falco.yaml
sudo systemctl restart falco-modern-bpf.service
sudo systemctl is-active falco-modern-bpf.service

Knowledge check

• Q: Is every alert a real security incident? A: No; legitimate activity can match a rule.
• Q: Why use process AND file path in an exception? A: To keep the exemption narrow and limit missed threats.
• Q: What is the main risk of a broad exemption? A: False negatives/security blind spots.
• Q: What does a kernel event drop indicate? A: Some monitored events could not be delivered; it is not an ordinary low-severity alert or a mere cosmetic issue.

Completion checkpoint: You can add a narrowly scoped exception, prove that it suppresses only one allowed behavior and still catches a different actor, then roll it back safely.

Official references: https://falco.org/docs/concepts/rules/exceptions/ ; https://falco.org/docs/concepts/rules/overriding/ ; https://falco.org/docs/concepts/metrics/ ; https://falco.org/docs/troubleshooting/dropping/

---

Chapter 11. Operations, Troubleshooting and Next Steps – 10 minutes

Overview and learning outcomes

Core goal: Run practical health checks, understand falcoctl artifact management, diagnose the most common failures, and sketch a safe path from lab deployment to production operation.

The five operational questions

1. Is Falco running? Is the process/service active on every intended node?
2. Is it monitoring? Has the syscall engine opened successfully and loaded the intended rules?
3. Can it detect? Does a controlled positive test still create an alert?
4. Can it deliver? Does the alert reach a retained/logged/forwarded destination?
5. Is it losing data? What are the rates of dropped kernel events and output queue drops?

A working Falco pod alone is not proof of an effective deployment. Pods may be ready while rules are absent, enrichment is missing, integrations are misconfigured or selected events are being dropped.

Daily operational command runbook: Ubuntu host

# Software and host baseline
sudo falco --version
uname -r

# Daemon readiness and unit state
sudo systemctl is-active falco-modern-bpf.service
sudo systemctl is-enabled falco-modern-bpf.service

# Recent failure signatures and rule-loading messages
sudo journalctl -u falco-modern-bpf.service -n 120 --no-pager

# Inspect active configuration; this does not require running a second sensor
sudo falco --dry-run -c /etc/falco/falco.yaml

# List the relevant services, including optional artifact updater
systemctl list-units --all | grep -E 'falco|falcoctl'

Daily operational command runbook: Kubernetes

# Validate context before every set of cluster commands
kubectl config current-context

# Helm-based deployment checks
helm list -n falco
kubectl get daemonsets,pods -n falco -o wide
kubectl get events -n falco --sort-by=.lastTimestamp | tail -n 25
kubectl logs -n falco -l app.kubernetes.io/name=falco \
  -c falco --since=10m --prefix=true

# When using the Operator instead of the plain chart
kubectl get falco,plugins,rulesfiles,configs,components -n falco
kubectl get pods -n falco-operator

Expected: every intended node has a ready sensor, rules/plugins are available, recent logs contain no persistent startup/reload failures, and the output path works. If an operator resource reports a reconciliation failure, inspect its details with kubectl describe and operator controller logs.

falcoctl – artifact and rules lifecycle

falcoctl accompanies Falco and can manage rule artifacts, plugins and (where relevant) drivers. Falco packaging may install falcoctl-artifact-follow.service for compatible rule updates. Its job is different from Falco: falcoctl manages artifacts; Falco loads and evaluates rules. Do not confuse an up-to-date rule artifact with an up-to-date Falco binary or compatible kernel driver.

# Confirm tool availability and inspect the installed command surface
falcoctl --help
falcoctl artifact --help
falcoctl --version

# Inspect related systemd units without changing their behavior
systemctl list-unit-files | grep -E 'falco|falcoctl'
sudo systemctl status falcoctl-artifact-follow.service --no-pager

The falcoctl-artifact-follow unit might not be installed/enabled in every deployment, especially container/Operator setups; treat an absent unit as a packaging/deployment difference, not immediate proof of failure. Use falcoctl artifact search or the configured artifact index to discover available rule versions. Before upgrading rules or plugins in production, confirm compatibility with the engine, review release notes, test local overrides, and roll out in staging.

Version-specific note: Falco 0.45.0 uses a driver API 11.0.0. Older kernel modules/drivers are not necessarily compatible. Falco 0.45.0 also introduced incubating /reload reporting and optional Unix-socket reload control; these are not needed for the beginner core and should not be treated as universally enabled public management endpoints.

Operational health metrics

Falco’s native metrics are off by default and can be enabled in falco.yaml or appropriate Helm/Operator configuration. The official documentation supports JSON metrics output and Prometheus exposure through an enabled webserver.

# Config fragment: merge into your verified configuration, not replace the entire file
metrics:
  enabled: true
  interval: 1m
  output_rule: true
  kernel_event_counters_enabled: true
  resource_utilization_enabled: true
  rules_counters_enabled: true
  state_counters_enabled: true

# Enable HTTP metrics only when the listener is appropriately secured.
webserver:
  enabled: true
  prometheus_metrics_enabled: true

When enabled, /metrics shares the Falco health webserver port. Protect the endpoint’s network exposure through binding, networking rules and monitoring access controls; do not open a management/metrics port to the internet. Observe Falco CPU/memory, sensor availability, event throughput, kernel drops (scap.n_drops), output-queue drops and forwarding errors. For modern eBPF 0.45, also examine auxiliary-map counters when present. Always relate changes in metrics to cluster load and workload events.

Troubleshooting matrix

Symptom	Diagnose	Likely cause	Correction	Verify
Falco package unavailable	apt-cache policy falco	Missing apt source, DNS/proxy or key issue	Correct signed official repo and refresh apt	Candidate appears
falco-modern-bpf inactive	systemctl status, journal	Service/startup failure	Fix reported error, then restart in lab	is-active returns active
eBPF BTF/ring buffer error	uname -r, /sys/kernel/btf/vmlinux, journal	Kernel feature or VM limitations	Use a kernel/VM with required features or supported alternate driver	Logs show source opened
Permission denied loading BPF	Journal, LSM policy and caps	Missing BPF privileges or host restrictions	Use documented privileges in authorized environment	Sensor initializes
Rule YAML fails validation	falco -V and --dry-run	Bad YAML, unknown field/macro, wrong order	Fix exact parser error	Validation returns zero
No custom alert	rules_files, startup logs, positive/negative trigger	Rule not loaded, wrong source, shadowed/mismatched condition	Correct path/order/condition, rerun controlled test	Expected rule name appears
No container name/image	Container plugin and runtime socket setup	Missing plugin/CRI access or metadata unavailable	Configure compatible metadata enrichment	Container context populated
No Kubernetes pod name	k8smeta/collector setup	Missing pod metadata enrichment	Configure supported metadata plugin/collector	Pod fields populated
Falco DaemonSet Pending	kubectl describe pod	Privileges, scheduling, image, node OS	Correct compatible node/scheduling/security constraints	Sensor pod Ready
Many false positives	Analyze alert samples	Overly broad logic	Narrow and test exception; record policy	Normal behavior excluded, malicious test alerts
Event-drop warnings	Metrics and event rate	Capture pipeline overloaded or specific driver issue	Profile and tune buffer/rules with testing	Drops decrease under realistic load
Forwarder receives nothing	HTTP output and destination logs	Incorrect URL, firewall, auth or TLS	Repair and secure output path	Receiver sees controlled alert
Upgrade unexpectedly changes service	Package and driver release notes	Version-specific lifecycle changes	Pin/test versions and driver selection	Post-upgrade sensor healthy

Production-ready deployment checklist

Before operating Falco for a real business service, ensure:

• [ ] Every desired Linux node/compute type can run an approved Falco driver and privileges.
• [ ] Base engine, chart/operator, rules, plugin and driver versions are recorded and upgraded in a tested sequence.
• [ ] Custom rules are code-reviewed, version controlled, validated and tested with positive/negative scenarios.
• [ ] Every critical rule has an owner, a defined severity and an investigation runbook.
• [ ] Rule exceptions are specific, reviewed and revisited after workload changes.
• [ ] Container and Kubernetes metadata are available as needed for incident triage.
• [ ] Alerts are routed over secured connections with secrets stored properly and field-level privacy reviewed.
• [ ] Sensor restarts, node coverage, dropped events and forwarding failures are monitored.
• [ ] An operator can reliably answer “what happened, where, who, and what should happen next?”
• [ ] Rollback and disaster recovery are rehearsed; Falco outages do not silently remove all detection coverage.
• [ ] Production response automation is separately designed and protected against accidental disruption.

Suggested next study milestones (after the 3-hour course)

Intermediate project (half day): create a five-rule pack for a sample backend app: unexpected shell, protected config writes, unexpected process execution, suspicious network connection, and restricted Kubernetes namespace behavior. Test each with a positive and negative control and document alert triage.

Intermediate-to-advanced project (1-2 days): install Falco Operator on an approved test cluster, manage rules through declarative resources, add Falcosidekick to a secure receiver, enable metrics, test node replacements, and validate detection throughput/noise during a sample application deployment.

Production-specialization project: define version pinning, GitOps integration, security controls, staging rollout, exceptions review, alert-to-incident correlation, system-level reliability SLOs, and incident response workflows. These require team agreement and environment-specific validation.

Final knowledge check

1. Q: What proves sensor readiness? A: Active process, initialized event source, loaded rules, successful controlled alert and functioning delivery path.
2. Q: What is falcoctl for? A: Managing Falco ecosystem artifacts and drivers, not directly evaluating every syscall rule.
3. Q: What must you check when updating a kernel module? A: Compatibility with the kernel and Falco driver API, build/signing and required privileges.
4. Q: Why monitor kernel event drops? A: Dropped events can undermine detection accuracy and alert completeness.
5. Q: What can you now demonstrate at intermediate foundation level? A: Installing Falco, triggering events, writing/tuning custom rules, inspecting outputs, deploying to a Linux-node cluster and performing basic operational checks.

Completion checkpoint: You can implement, verify and troubleshoot a lab Falco deployment and discuss production limitations without treating the demo setup as automatically production-ready.

Official references: https://falco.org/docs/troubleshooting/start-up-error/ ; https://falco.org/docs/troubleshooting/dropping/ ; https://falco.org/docs/concepts/metrics/ ; https://falco.org/docs/concepts/rules/default-custom/ ; https://falco.org/blog/falco-0-45-0/

---

Consolidated quick reference

Commands to memorize

Goal	Command (run on an authorized Linux host unless noted)
Check binary version	sudo falco --version
Check driver service	sudo systemctl status falco-modern-bpf.service --no-pager
Follow host alerts	sudo journalctl -fu falco-modern-bpf.service
List rules	sudo falco -L
List fields	sudo falco --list=syscall -N
List event types	sudo falco --list-events
Validate rules	sudo falco -V /path/to/rules.yaml
Validate configuration	sudo falco --dry-run -c /etc/falco/falco.yaml
Restart lab service	sudo systemctl restart falco-modern-bpf.service
Examine falcoctl	falcoctl artifact --help
Add Helm repo	helm repo add falcosecurity https://falcosecurity.github.io/charts
View Kubernetes sensors	kubectl get pods,ds -n falco -o wide
View Kubernetes alerts	kubectl logs -n falco -l app.kubernetes.io/name=falco -c falco --since=10m

How to decide whether a rule is good enough

A minimum acceptable custom rule should include:

• An explicit security objective and expected malicious behavior.
• A reasonably bounded positive event expression, normally including evt.type.
• Useful but privacy-conscious output fields for investigation.
• A suitable priority and searchable tags.
• A positive test that proves the behavior is detected.
• A negative test that proves an allowed behavior is not mistaken for that behavior.
• A documented false-positive boundary and an owner for future tuning.
• A compatible deployment/rollback method.

Course-wide self-assessment rubric

Score each competency 0 (cannot explain), 1 (can explain), 2 (can demonstrate with guidance), 3 (can demonstrate independently):

Competency	Expected result
Explain purpose	Contrast Falco, scanning, policy controls and SIEM
Understand kernel evidence	Identify process, syscall, file and container context
Explain architecture	Modern eBPF/driver -> libs -> rules -> alert
Host install	Start Falco on a supported Ubuntu VM
Trigger events	Detect sensitive-file access and inspect an alert
Read rules	Explain rule, macro, list, conditions and output fields
Author rules	Write and validate two original detections
Forward output	Explain JSON output and Falcosidekick
Kubernetes	Deploy/read a per-node sensor and trigger a pod event
Tune rules	Add one narrow exception while preserving another detection
Operate	Find health, coverage, rule, plugin and event-drop problems

Passing target for the three-hour foundation: level 2 or above in at least 9 of the 11 competencies. Level 3 in rules, tuning and operations usually requires additional sessions.

Verified references and version notes

The following are official project/vendor resources relevant to the precise claims in this guide, accessed/checked for the 11 October 2026 reference baseline:

1. Falco 0.45.0 release announcement – 21 September 2026; driver API 11.0.0; bundled falcoctl 0.14.2; falco-rules 5.2.0; 0.45 changes.
2. Falco Linux quickstart – official apt instructions, Lima example, modern eBPF and /etc/shadow demo.
3. Falco Kubernetes quickstart – Linux-node cluster, Helm commands, demo pod event.
4. Falco Operator deployment – current recommended Kubernetes management model; CRDs; install/cleanup ordering and Kubernetes prerequisites.
5. Falco Operator version matrix – validate the default Falco image deployed by an operator release.
6. Falco kernel event sources and modern eBPF – supported drivers and feature prerequisites.
7. Kernel event architecture – driver-to-userspace event pipeline.
8. Falco rules syntax and supported fields – conditions, lists, macros and rule attributes.
9. Falco rules style guide – efficient rule construction, positive event selection, output conventions.
10. Falco rule exceptions and overriding – tuning and current override syntax (old append: true is deprecated).
11. Falco CLI options – -V, --dry-run, -r, --list and logging options.
12. Outputs and Falcosidekick – standard channels, JSON, HTTP forwarding and ecosystem integrations.
13. Falco metrics – operational counters and optional Prometheus endpoint.
14. Falco event-drop troubleshooting – buffers, dropped syscall events and modern eBPF 0.45 counters.
15. Canonical Multipass installation – Windows virtual-machine path.
16. Falco 0.44.0 breaking changes – removed legacy eBPF, gVisor engine and gRPC output.

Final caveat: This training document supplies copy-ready commands and complete core YAML examples, but its sample workloads were not executed in a real Linux kernel/Kubernetes environment during authoring. Running them yourself and validating actual results against the installed Falco/driver/rules versions is part of the learning process. In production, prefer vendor-tested deployment choices, staged rollouts and an explicit rollback plan.