Prometheus

Recording rules and Alerting rules explained in Prometheus!!!

March 31, 2020 comments off

What is rules in Prometheus?

Time series queries can quickly become quite complicated to remember and type using the Expression Browser in the default Prometheus

User Interface. Prometheus rule is way to run promql expression at certain interval and store a value in Prometheus time series

database for future use such as to store a some manipulative values in TSDB or alerting needs.

Why to use rules in Prometheus?

Time series queries can quickly become quite complicated to remember and type using the Expression Browser in the default Prometheus User Interface. 

100 - (100 * node_memory_MemFree_bytes / node_memory_MemTotal_bytes)

This is not so bad, it describes how much percent of memory is free on your server running the Prometheus Node Exporter. So, rather than remembering and typing this query every time we want to know that answer, we can create a recording rule that will run at a chosen interval and make the data available as a time series.

Types of rules in Prometheus?

Prometheus supports two types of rules which may be configured and then evaluated at regular intervals:

  • Recording rules are for pre-calculating frequently used or computationally expensive queries. The results of those rules are saved into their own time series.
  • Alerting rules on the other hand enable you to specify the conditions that an alert should be fired to an external service like Slack. These are based on PromQL queries.

How to add Recording rules and Alerting rules in Prometheus?

Step 1 – You can create prometheus.rules.yml file in the same directory where prometheus.yml is stored, e.g.

/etc/prometheus/prometheus.rules.yml.
Step 2 – Now lets add the prometheus_rules.yml reference to the prometheus.yml rule_files section.
Step 3 – and restart the prometheus service.
Step 4 – Refresh the Prometheus user interface and check the drop down.

How to check rules config file?
$ promtool check rules /etc/prometheus/prometheus.rules.yml

Step to enable prometheus Rule

Example of prometheus Recording Rule

Example of prometheus alerting rules

Rajesh Kumar
Follow me
Latest posts by Rajesh Kumar (see all)