Icinga Trainer

How the daemon actually works, because almost every advanced topic depends on it. The feature and library model, the check scheduler and its execution flow, result processing and state machines, the event and notification pipeline, and the cluster protocol that carries all of it between endpoints.

Topics: Icinga 2 daemon structure, features and libraries · The check scheduler: scheduling, latency, execution and reaping · Check result processing and the state machine · Soft and hard states, max_check_attempts and retry intervals · Event handling and the notification pipeline · Cluster communication: the JSON-RPC protocol, endpoints and connections · Icinga DB, Redis and the relational backend; migrating from IDO · The API listener and the event stream · Icinga 2 compared with Nagios, Zabbix and Prometheus · Where Icinga is the wrong tool

• Assignments: (1) Trace one check from schedule to stored state and identify every component it passed; (2) Measure check latency and explain where the time was spent
• Capstone: Produce an architecture note proposing an Icinga topology for a specific estate

The heart of Icinga 2 and the reason it replaced its predecessor. Object types and their relationships, templates and multiple inheritance, custom variables as typed dictionaries and arrays, then the language features — functions, conditionals, scoping, operators — that let configuration express intent instead of enumerating facts.

Topics: Object types: Host, Service, CheckCommand, Notification, User, TimePeriod, Dependency · Templates and multiple inheritance, and resolution order · Custom variables: scalars, arrays and dictionaries · Variable scoping, globals and constants · Functions, conditionals, loops and operators in the DSL · Configuration file layout, zones.d and includes · Runtime macros and macro resolution order · icinga2 daemon -C validation and reading its errors · Configuration as code: structure that survives a team · Common configuration anti-patterns and their cost

• Assignments: (1) Refactor a set of static host definitions into templates plus custom variables; (2) Fix a configuration that validates but resolves the wrong macro at runtime
• Capstone: Produce a configuration layout and naming standard another engineer could extend safely

The technique that separates a maintainable Icinga estate from an unmaintainable one. Apply rules with assign and ignore expressions, apply-for loops that iterate a dictionary to generate one service per element, applying notifications and dependencies, and the design pattern of describing a host by its data and letting the rules do the rest.

Topics: apply Service with assign where and ignore where · Matching on custom variables, groups, names and regular expressions · apply for: generating services from a dictionary · Per-element thresholds and command arguments inside apply for · Applying notifications, dependencies and scheduled downtimes · Group membership from apply rules · Ordering, precedence and rule conflicts · Designing the host data model that the rules consume · Debugging why a rule matched, or did not · Scaling patterns for thousands of hosts and tens of thousands of services

• Assignments: (1) Generate one disk service per mount point from a host dictionary with per-mount thresholds; (2) Replace a hundred hand-written service definitions with three apply rules
• Capstone: Deliver a configuration where adding a fully monitored host means adding one object

The layer that touches the monitored system. CheckCommand definitions with typed arguments, conditional arguments and macros; the Monitoring Plugins catalogue and the plugin output contract; performance data format; and writing custom plugins in Bash and Python that behave correctly under every failure the target can produce.

Topics: The plugin contract: exit codes, output line and long output · Performance data format, units, warning, critical, min and max · CheckCommand objects: arguments, set_if, order and repeat_key · Command arguments driven by custom variables · The Monitoring Plugins catalogue and what is worth using · Writing a check plugin in Bash · Writing a check plugin in Python with argument parsing and timeouts · Handling unreachable targets, timeouts and partial data · Threshold ranges and the standard threshold syntax · Testing plugins outside Icinga and inside it · Distributing plugins across agents

• Assignments: (1) Write a Python plugin that emits correct performance data and handles timeouts; (2) Wrap an existing internal script as a CheckCommand with typed arguments
• Capstone: Deliver an internal plugin plus CheckCommand that another team could adopt unchanged

Everything that decides whether a state change becomes an alert. Check intervals and retry behaviour, soft and hard state transitions, flapping detection and its thresholds, acknowledgements with and without expiry, fixed and flexible downtimes, and dependencies that suppress the consequences of a single root cause.

Topics: check_interval, retry_interval and max_check_attempts · Soft versus hard states and when notifications fire · Flapping detection: thresholds, history and suppression · Acknowledgements: sticky, expiring and notification behaviour · Fixed and flexible downtimes, and scheduled recurring downtime · Downtime propagation to child objects · Host and service dependencies, and disable_notifications · Parent-child host relationships and network reachability · Volatile services and their use cases · Event commands for automatic remediation · Freshness checking and passive check results

• Assignments: (1) Configure dependencies so a router failure produces one alert rather than forty; (2) Build a recurring maintenance downtime that survives configuration reloads
• Capstone: Deliver a state-processing policy that measurably reduces alert volume without hiding failures

The hardest part of Icinga and the reason most teams book training. Zone hierarchy design, endpoints and their connection direction, the certificate authority and signing workflow, configuration synchronisation from master downward, high availability with check and notification split, and a structured method for diagnosing a cluster that has gone quiet.

Topics: Zones, endpoints and the parent-child tree · Master, satellite and agent roles · Connection direction, NAT and firewall constraints · The built-in CA: node wizard, signing requests, ticket-based and on-demand signing · Certificate renewal, expiry and revocation · Config sync: zones.d, sync direction and what must not be synced · High availability: two endpoints in a zone, check and notification split · Split-brain behaviour and reconnection · Top-down command endpoint versus top-down config sync · Agent-based checks and the local execution model · Diagnosing cluster faults: logs, cluster-zone checks, connection state · Scaling a hierarchy across many sites

• Assignments: (1) Build a master, satellite and agent hierarchy with certificates issued end to end; (2) Break the cluster deliberately in three ways and diagnose each from logs alone
• Capstone: Deliver a multi-site distributed design with HA at the master and a documented failure runbook

Routing state changes to people, correctly and only once. Notification objects and how they are applied, users and user groups, time periods including exclusions and holidays, notification intervals and escalation by time, state and type filters, and integration with email, chat and incident management platforms.

Topics: Notification objects and applying them with rules · Users, user groups and per-user notification preferences · TimePeriod objects, ranges, exclusions and holiday handling · States and types filters: what actually triggers a notification · Notification interval, and stopping repeat notifications · Escalations by begin and end time · NotificationCommand definitions and custom notification scripts · Email formatting and useful macros in messages · Integrating Slack, Microsoft Teams, PagerDuty and webhooks · Suppressing notifications during downtime and acknowledgement · Auditing what was sent and to whom

• Assignments: (1) Build a three-tier escalation across time periods with a holiday exclusion; (2) Integrate an incident platform so problems open and close automatically
• Capstone: Deliver a notification policy covering business hours, out of hours, escalation and incident integration

Configuration from a source of truth rather than from files. Director's object model and templates, import sources from CMDB, cloud APIs, databases, LDAP and CSV, property modifiers, sync rules that create and update objects automatically, then deployment, change history and rollback — and how Director coexists with file-based configuration and Git.

Topics: Icinga Director architecture, database and deployment model · Director host and service templates, and template inheritance · Service sets and apply rules in Director · Import sources: SQL, LDAP, REST, CSV, cloud provider APIs · Property modifiers and transforming imported data · Sync rules: create, update, purge and merge policies · Data fields, data lists and custom variable validation · Deployment, activity log, diffing and rollback · Director and file-based configuration side by side · Git-based configuration workflows and review · Director basket import and export for promotion between environments · The Director API and automating the automation

• Assignments: (1) Import hosts from a CMDB or cloud account and sync them into fully monitored objects; (2) Promote a Director configuration between two environments using baskets
• Capstone: Deliver a pipeline where the source of truth changes and monitoring follows without human action

The interface and everything built on it. Icinga Web 2 navigation, filters, custom dashboards and the module system; graphing by writing performance data to Graphite or InfluxDB and rendering it inline; business process modelling that expresses service health as a tree of dependencies; and reporting with SLA views.

Topics: Icinga Web 2: views, filters, URL-based filtering and saved dashboards · Custom dashlets and navigation items · The module system and useful community modules · Writing performance data: Graphite writer, InfluxDB writer, Icinga DB · Grafana and Graphite integration for inline graphs · Retention and downsampling for performance data · Business Process Modelling: nodes, operators and simulation · Modelling a customer-facing service as a dependency tree · Reporting module: scheduled reports and SLA calculation · Defining availability and what the numbers actually mean · Multi-tenant views for application teams

• Assignments: (1) Write performance data to a time-series store and render inline graphs in the interface; (2) Model one customer-facing service as a business process and simulate a component failure
• Capstone: Deliver an interface set with per-team dashboards, inline graphing and an SLA report

Locking down a system that can execute commands on every host it monitors. API user configuration and permission filters, certificate-based authentication, role-based access in Icinga Web 2 with restrictions on hosts and services, secure agent communication, and hardening the installation as a whole.

Topics: The Icinga 2 API: objects, actions, queries and the event stream · ApiUser objects, permissions and object filters · Client certificate authentication to the API · Automating with the API from scripts and CI · Icinga Web 2 roles, permissions and restrictions · Restricting hosts and services by filter for multi-tenancy · Authentication back ends: database, LDAP, Active Directory, SSO · Securing agent communication and certificate hygiene · Protecting the CA and the signing workflow · Hardening: file permissions, listener exposure, command execution risk · Audit trail and change accountability

• Assignments: (1) Create an API user restricted to one zone and prove it cannot act outside it; (2) Configure role restrictions so an application team sees only its own services
• Capstone: Deliver a security model covering API access, web roles, agent certificates and hardening

Keeping a large installation healthy. Tuning check concurrency and intervals for a stated check volume, Icinga DB and Redis sizing, history growth and retention, then a structured troubleshooting method covering checks, notifications, cluster faults and web performance — and finally upgrade and migration strategy including IDO to Icinga DB.

Topics: Measuring check latency, execution time and scheduler saturation · Tuning concurrent checks, intervals and check distribution · Icinga DB and Redis sizing, and history retention · Database growth, cleanup and partitioning · Monitoring Icinga itself: the built-in checks worth using · Debugging checks: debug log, console, and running a command by hand · Debugging notifications that did not fire · Debugging cluster faults and certificate problems · Log analysis and common error signatures · Recovery scenarios: lost master, corrupted config, expired certificates · Upgrade strategy across major versions · Migrating from IDO to Icinga DB, and from Nagios to Icinga 2

• Assignments: (1) Diagnose four injected faults — a check, a notification, a cluster link and a certificate; (2) Tune an installation whose check latency is climbing, and prove the improvement
• Capstone: Deliver an operations runbook: tuning targets, retention plan, failure signatures and upgrade path