Prometheus PromQL Example Query: node exporter

For your Linux Node Exporter target, these are the most important PromQL queries for CPU, RAM, Disk, Network, Load, Uptime. Your uploaded metrics include Node Exporter metrics like node_cpu_seconds_total, node_memory_MemAvailable_bytes, node_filesystem_size_bytes, node_disk_*, and node_network_*.

1. Server UP / DOWN

up{job="linux"}
Code language: JavaScript (javascript)

1 = server is up
0 = server is down

2. CPU Usage %

100 - (avg by(instance) (rate(node_cpu_seconds_total{job="linux",mode="idle"}[5m])) * 100)
Code language: JavaScript (javascript)

For one server:

100 - (avg(rate(node_cpu_seconds_total{job="linux",instance="172.31.2.24:9100",mode="idle"}[5m])) * 100)
Code language: JavaScript (javascript)

3. CPU Idle %

avg by(instance) (rate(node_cpu_seconds_total{job="linux",mode="idle"}[5m])) * 100
Code language: JavaScript (javascript)

4. CPU System Usage %

avg by(instance) (rate(node_cpu_seconds_total{job="linux",mode="system"}[5m])) * 100
Code language: JavaScript (javascript)

5. CPU User Usage %

avg by(instance) (rate(node_cpu_seconds_total{job="linux",mode="user"}[5m])) * 100
Code language: JavaScript (javascript)

6. CPU I/O Wait %

avg by(instance) (rate(node_cpu_seconds_total{job="linux",mode="iowait"}[5m])) * 100
Code language: JavaScript (javascript)

Very useful for finding slow disk/storage issues.

7. RAM Usage %

100 * (1 - (node_memory_MemAvailable_bytes{job="linux"} / node_memory_MemTotal_bytes{job="linux"}))
Code language: JavaScript (javascript)

8. RAM Available GB

node_memory_MemAvailable_bytes{job="linux"} / 1024 / 1024 / 1024
Code language: JavaScript (javascript)

9. RAM Total GB

node_memory_MemTotal_bytes{job="linux"} / 1024 / 1024 / 1024
Code language: JavaScript (javascript)

10. RAM Used GB

(node_memory_MemTotal_bytes{job="linux"} - node_memory_MemAvailable_bytes{job="linux"}) / 1024 / 1024 / 1024
Code language: JavaScript (javascript)

11. Disk Usage %

100 * (1 - (
  node_filesystem_avail_bytes{job="linux",fstype!~"tmpfs|overlay|squashfs"}
  /
  node_filesystem_size_bytes{job="linux",fstype!~"tmpfs|overlay|squashfs"}
))
Code language: JavaScript (javascript)

12. Disk Usage % Only Root /

100 * (1 - (
  node_filesystem_avail_bytes{job="linux",mountpoint="/"}
  /
  node_filesystem_size_bytes{job="linux",mountpoint="/"}
))
Code language: JavaScript (javascript)

13. Disk Free GB

node_filesystem_avail_bytes{job="linux",fstype!~"tmpfs|overlay|squashfs"} / 1024 / 1024 / 1024
Code language: JavaScript (javascript)

14. Disk Total GB

node_filesystem_size_bytes{job="linux",fstype!~"tmpfs|overlay|squashfs"} / 1024 / 1024 / 1024
Code language: JavaScript (javascript)

15. Disk Read MB/s

rate(node_disk_read_bytes_total{job="linux"}[5m]) / 1024 / 1024
Code language: JavaScript (javascript)

16. Disk Write MB/s

rate(node_disk_written_bytes_total{job="linux"}[5m]) / 1024 / 1024
Code language: JavaScript (javascript)

17. Disk Read IOPS

rate(node_disk_reads_completed_total{job="linux"}[5m])
Code language: JavaScript (javascript)

18. Disk Write IOPS

rate(node_disk_writes_completed_total{job="linux"}[5m])
Code language: JavaScript (javascript)

19. Network Receive MB/s

rate(node_network_receive_bytes_total{job="linux",device!="lo"}[5m]) / 1024 / 1024
Code language: JavaScript (javascript)

20. Network Transmit MB/s

rate(node_network_transmit_bytes_total{job="linux",device!="lo"}[5m]) / 1024 / 1024
Code language: JavaScript (javascript)

21. Network Receive Packets/s

rate(node_network_receive_packets_total{job="linux",device!="lo"}[5m])
Code language: JavaScript (javascript)

22. Network Transmit Packets/s

rate(node_network_transmit_packets_total{job="linux",device!="lo"}[5m])
Code language: JavaScript (javascript)

23. Network Errors

rate(node_network_receive_errs_total{job="linux",device!="lo"}[5m])
+
rate(node_network_transmit_errs_total{job="linux",device!="lo"}[5m])
Code language: JavaScript (javascript)

24. Network Drops

rate(node_network_receive_drop_total{job="linux",device!="lo"}[5m])
+
rate(node_network_transmit_drop_total{job="linux",device!="lo"}[5m])
Code language: JavaScript (javascript)

25. Load Average 1 Minute

node_load1{job="linux"}
Code language: JavaScript (javascript)

26. Load Average 5 Minutes

node_load5{job="linux"}
Code language: JavaScript (javascript)

27. Load Average 15 Minutes

node_load15{job="linux"}
Code language: JavaScript (javascript)

28. Running Processes

node_procs_running{job="linux"}
Code language: JavaScript (javascript)

29. Blocked Processes

node_procs_blocked{job="linux"}
Code language: JavaScript (javascript)

30. Server Uptime

time() - node_boot_time_seconds{job="linux"}
Code language: JavaScript (javascript)

In hours:

(time() - node_boot_time_seconds{job="linux"}) / 3600
Code language: JavaScript (javascript)

In days:

(time() - node_boot_time_seconds{job="linux"}) / 86400
Code language: JavaScript (javascript)

Best 10 PromQL for Linux Dashboard

Use these first:

up{job="linux"}
Code language: JavaScript (javascript)

100 - (avg by(instance) (rate(node_cpu_seconds_total{job="linux",mode="idle"}[5m])) * 100)
Code language: JavaScript (javascript)

100 * (1 - (node_memory_MemAvailable_bytes{job="linux"} / node_memory_MemTotal_bytes{job="linux"}))
Code language: JavaScript (javascript)

100 * (1 - (node_filesystem_avail_bytes{job="linux",mountpoint="/"} / node_filesystem_size_bytes{job="linux",mountpoint="/"}))
Code language: JavaScript (javascript)

rate(node_disk_read_bytes_total{job="linux"}[5m]) / 1024 / 1024
Code language: JavaScript (javascript)

rate(node_disk_written_bytes_total{job="linux"}[5m]) / 1024 / 1024
Code language: JavaScript (javascript)

rate(node_network_receive_bytes_total{job="linux",device!="lo"}[5m]) / 1024 / 1024
Code language: JavaScript (javascript)

rate(node_network_transmit_bytes_total{job="linux",device!="lo"}[5m]) / 1024 / 1024
Code language: JavaScript (javascript)

node_load1{job="linux"}
Code language: JavaScript (javascript)

(time() - node_boot_time_seconds{job="linux"}) / 86400
Code language: JavaScript (javascript)

For Grafana variables, replace:

job="linux"
Code language: JavaScript (javascript)

with:

job="$job", instance="$instance"
Code language: JavaScript (javascript)

Example PromQL for Prometheus Server

prometheus_http_requests_total
prometheus_http_requests_total{job="prometheus",group="canary"}
prometheus_http_requests_total{environment=~"staging|testing|development",method!="GET"}
prometheus_http_requests_total{job=~".*"} # Bad!
prometheus_http_requests_total{job=~".+"} # Good!
prometheus_http_requests_total{job=~".*",method="get"} # Good!
prometheus_http_requests_total[5m]
prometheus_http_requests_total{job="prometheus",group="canary"}[2h]
prometheus_http_requests_total{environment=~"staging|testing|development",method!="GET"}[60m]Code language: PHP (php)

Prometheus PromQL Example Query: node exporter

Metrics specific to the Node Exporter are prefixed with node_ and include metrics like node_cpu_seconds_total and node_exporter_build_info.

Modern CPUs don't run at one constant frequency.

To save power CPUs can reduce the frequency they run at, which is quite useful for battery based devices like laptops. So while CPU metrics give you the proportion of time in each mode, one second of user time isn't always represent same amount of work as another second of user time. This can be a problem when running benchmarks.

Linux provides information about this under /sys/devices/system/cpu/*/cpufreq/, and on my desktop the node exporter produces:

# HELP node_cpu_frequency_max_hertz Maximum cpu thread frequency in hertz.
# TYPE node_cpu_frequency_max_hertz gauge
node_cpu_frequency_max_hertz{cpu="0"} 3.4e+09
node_cpu_frequency_max_hertz{cpu="1"} 3.4e+09

# HELP node_cpu_frequency_min_hertz Minimum cpu thread frequency in hertz.
# TYPE node_cpu_frequency_min_hertz gauge
node_cpu_frequency_min_hertz{cpu="0"} 1.6e+09
node_cpu_frequency_min_hertz{cpu="1"} 1.6e+09

# HELP node_cpu_scaling_frequency_hertz Current scaled cpu thread frequency in hertz.
# TYPE node_cpu_scaling_frequency_hertz gauge
node_cpu_scaling_frequency_hertz{cpu="0"} 2.352192e+09
node_cpu_scaling_frequency_hertz{cpu="1"} 2.243048e+09

# HELP node_cpu_scaling_frequency_max_hertz Maximum scaled cpu thread frequency in hertz.
# TYPE node_cpu_scaling_frequency_max_hertz gauge
node_cpu_scaling_frequency_max_hertz{cpu="0"} 3.4e+09
node_cpu_scaling_frequency_max_hertz{cpu="1"} 3.4e+09

# HELP node_cpu_scaling_frequency_min_hertz Minimum scaled cpu thread frequency in hertz.
# TYPE node_cpu_scaling_frequency_min_hertz gauge
node_cpu_scaling_frequency_min_hertz{cpu="0"} 1.6e+09
node_cpu_scaling_frequency_min_hertz{cpu="1"} 1.6e+09

# Linux provides scheduling metrics in /proc/schedstat, which the node exporter uses.
# The node exporter current exposes three of these:

# HELP node_schedstat_running_seconds_total Number of seconds CPU spent running a process.
# TYPE node_schedstat_running_seconds_total counter
node_schedstat_running_seconds_total{cpu="0"} 1.093032217430793e+06
node_schedstat_running_seconds_total{cpu="1"} 1.07527722232456e+06

# HELP node_schedstat_timeslices_total Number of timeslices executed by CPU.
# TYPE node_schedstat_timeslices_total counter
node_schedstat_timeslices_total{cpu="0"} 5.965185464e+09
node_schedstat_timeslices_total{cpu="1"} 5.266658269e+09

# HELP node_schedstat_waiting_seconds_total Number of seconds spent by processing waiting for this CPU.
# TYPE node_schedstat_waiting_seconds_total counter
node_schedstat_waiting_seconds_total{cpu="0"} 217918.365216207
node_schedstat_waiting_seconds_total{cpu="1"} 218559.331226331

Using the group() aggregator in PromQL

If you wanted to count the number of unique values a label has, such as say the number of values the cpu label had in node_cpu_seconds_total per instance the standard pattern is:

count without(cpu) (
  count without(mode) (node_cpu_seconds_total)
)

count without(cpu) (
  group without(mode) (node_cpu_seconds_total)
)

Network interface metrics from the node exporter

Along with many others, the node exporter exposes network interface metrics.

Network interface metrics have the prefix node_network_ on the node exporter's /metrics, and a device label. These are distinct from the node_netstat_ metrics which are about the kernel's network subsystem in general.

  rate(node_network_transmit_errs_total[5m]) 
/ 
  rate(node_network_transmit_packets_total[5m])
Code language: PHP (php)

CPU scheduling metrics from the node exporter

CPU frequency scaling metrics from the node exporter

ARP cache metrics from the node exporter

Using the group() aggregator in PromQL

Linux software RAID metrics from the node exporter

Time metric from the node exporter

Using Letsencrypt with the node exporter

Conntrack metrics from the node exporter

Kernel file descriptor metrics from the node exporter

Temperature and hardware monitoring metrics from the node exporter

Network interface metrics from the node exporter

Filesystem metrics from the node exporter

Analyse a metric by kernel version

Mapping iostat to the node exporter’s node_disk_* metrics

Using the textfile collector from a shell script

New Features in Node Exporter 0.16.0

Using group_left to calculate label proportions

Understanding Machine CPU usage

Monitoring directory sizes with the Textfile Collector

Rajesh Kumar

I’m a DevOps/SRE/DevSecOps/Cloud Expert passionate about sharing knowledge and experiences. I have worked at Cotocus. I share tech blog at DevOps School, travel stories at Holiday Landmark, stock market tips at Stocks Mantra, health and fitness guidance at My Medic Plus, product reviews at TrueReviewNow , and SEO strategies at Wizbrand.

Do you want to learn Quantum Computing?

Please find my social handles as below;

Rajesh Kumar Personal Website
Rajesh Kumar at YOUTUBE
Rajesh Kumar at INSTAGRAM
Rajesh Kumar at X
Rajesh Kumar at FACEBOOK
Rajesh Kumar at LINKEDIN
Rajesh Kumar at WIZBRAND

Rajesh Kumar DailyLogs