{"id":22351,"date":"2021-07-12T08:48:01","date_gmt":"2021-07-12T08:48:01","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=22351"},"modified":"2021-10-01T10:24:54","modified_gmt":"2021-10-01T10:24:54","slug":"prometheus-promql-example-query-node-exporter","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/prometheus-promql-example-query-node-exporter\/","title":{"rendered":"Prometheus PromQL Example Query: node exporter"},"content":{"rendered":"\n

Example PromQL for Prometheus Server<\/h2>\n\n\n\n
\n\n\n
prometheus_http_requests_total\r\nprometheus_http_requests_total{job=\"prometheus\"<\/span>,group=\"canary\"<\/span>}\r\nprometheus_http_requests_total{environment=~\"staging|testing|development\"<\/span>,method!=\"GET\"<\/span>}\r\nprometheus_http_requests_total{job=~\".*\"<\/span>} # Bad!<\/span>\r\nprometheus_http_requests_total{job=~\".+\"<\/span>} # Good!<\/span>\r\nprometheus_http_requests_total{job=~\".*\"<\/span>,method=\"get\"<\/span>} # Good!<\/span>\r\nprometheus_http_requests_total[5<\/span>m]\r\nprometheus_http_requests_total{job=\"prometheus\"<\/span>,group=\"canary\"<\/span>}[2<\/span>h]\r\nprometheus_http_requests_total{environment=~\"staging|testing|development\"<\/span>,method!=\"GET\"<\/span>}[60<\/span>m]<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Prometheus PromQL Example Query: node exporter<\/h2>\n\n\n\n
Metrics specific to the Node Exporter are prefixed with node_ and include metrics like node_cpu_seconds_total and node_exporter_build_info.<\/p>\n\n\n\n
\n\n\n\n
Metric<\/th>Meaning<\/th><\/tr><\/thead>
rate(node_cpu_seconds_total{mode=”system”}[1m])<\/td>The average amount of CPU time spent in system mode, per second, over the last minute (in seconds)<\/td><\/tr>
node_filesystem_avail_bytes<\/td>The filesystem space available to non-root users (in bytes)<\/td><\/tr>
rate(node_network_receive_bytes_total[1m])<\/td>The average network traffic received, per second, over the last minute (in bytes)<\/td><\/tr>
<\/td><\/td><\/tr>
<\/td><\/td><\/tr>
<\/td><\/td><\/tr>
<\/td><\/td><\/tr>
<\/td><\/td><\/tr>
<\/td><\/td><\/tr>
<\/td><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n
Modern CPUs don't run at one constant frequency.\n\nTo save power CPUs can reduce the frequency they run at, which is quite useful for battery based devices like laptops. So while CPU metrics give you the proportion of time in each mode, one second of user time isn'<\/span>t always represent same amount of work as<\/span> another second of user time. This can be a problem when running benchmarks.\n\nLinux provides information about this under \/sys\/devices\/system\/cpu\/*\/cpufreq\/, and on my desktop the node exporter produces:\n\n# HELP node_cpu_frequency_max_hertz Maximum cpu thread frequency in hertz.\n# TYPE node_cpu_frequency_max_hertz gauge\nnode_cpu_frequency_max_hertz{cpu=\"0\"} 3.4e+09\nnode_cpu_frequency_max_hertz{cpu=\"1\"} 3.4e+09\n\n# HELP node_cpu_frequency_min_hertz Minimum cpu thread frequency in hertz.\n# TYPE node_cpu_frequency_min_hertz gauge\nnode_cpu_frequency_min_hertz{cpu=\"0\"} 1.6e+09\nnode_cpu_frequency_min_hertz{cpu=\"1\"} 1.6e+09\n\n# HELP node_cpu_scaling_frequency_hertz Current scaled cpu thread frequency in hertz.\n# TYPE node_cpu_scaling_frequency_hertz gauge\nnode_cpu_scaling_frequency_hertz{cpu=\"0\"} 2.352192e+09\nnode_cpu_scaling_frequency_hertz{cpu=\"1\"} 2.243048e+09\n\n# HELP node_cpu_scaling_frequency_max_hertz Maximum scaled cpu thread frequency in hertz.\n# TYPE node_cpu_scaling_frequency_max_hertz gauge\nnode_cpu_scaling_frequency_max_hertz{cpu=\"0\"} 3.4e+09\nnode_cpu_scaling_frequency_max_hertz{cpu=\"1\"} 3.4e+09\n\n# HELP node_cpu_scaling_frequency_min_hertz Minimum scaled cpu thread frequency in hertz.\n# TYPE node_cpu_scaling_frequency_min_hertz gauge\nnode_cpu_scaling_frequency_min_hertz{cpu=\"0\"} 1.6e+09\nnode_cpu_scaling_frequency_min_hertz{cpu=\"1\"} 1.6e+09\n\n# Linux provides scheduling metrics in \/proc\/schedstat, which the node exporter uses.\n# The node exporter current exposes three of these:\n\n# HELP node_schedstat_running_seconds_total Number of seconds CPU spent running a process.\n# TYPE node_schedstat_running_seconds_total counter\nnode_schedstat_running_seconds_total{cpu=\"0\"} 1.093032217430793e+06\nnode_schedstat_running_seconds_total{cpu=\"1\"} 1.07527722232456e+06\n\n# HELP node_schedstat_timeslices_total Number of timeslices executed by CPU.\n# TYPE node_schedstat_timeslices_total counter\nnode_schedstat_timeslices_total{cpu=\"0\"} 5.965185464e+09\nnode_schedstat_timeslices_total{cpu=\"1\"} 5.266658269e+09\n\n# HELP node_schedstat_waiting_seconds_total Number of seconds spent by processing waiting for this CPU.\n# TYPE node_schedstat_waiting_seconds_total counter\nnode_schedstat_waiting_seconds_total{cpu=\"0\"} 217918.365216207\nnode_schedstat_waiting_seconds_total{cpu=\"1\"} 218559.331226331\n\nUsing the group() aggregator in PromQL\n\nIf you wanted to count the number of unique values a label has, such as say the number of values the cpu label had in node_cpu_seconds_total per instance the standard pattern is:\n\ncount without(cpu) (\n  count without(mode) (node_cpu_seconds_total)\n)\n\ncount without(cpu) (\n  group without(mode) (node_cpu_seconds_total)\n)\n\nNetwork interface metrics from the node exporter\n\nAlong with many others, the node exporter exposes network interface metrics.\n\nNetwork interface metrics have the prefix node_network_ on the node exporter's \/metrics, and a device label. These are distinct from the node_netstat_ metrics which are about the kernel's network subsystem in general.\n\n  rate(node_network_transmit_errs_total[5m]) \n\/ \n  rate(node_network_transmit_packets_total[5m])\n<\/span><\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
<\/h2>\n\n\n\n
CPU scheduling metrics from the node exporter<\/a><\/p>\n\n\n\n
CPU frequency scaling metrics from the node exporter<\/a><\/p>\n\n\n\n
ARP cache metrics from the node exporter<\/a><\/p>\n\n\n\n
Using the group() aggregator in PromQL<\/a><\/p>\n\n\n\n
Linux software RAID metrics from the node exporter<\/a><\/p>\n\n\n\n
Time metric from the node exporter<\/a><\/p>\n\n\n\n
Using Letsencrypt with the node exporter<\/a><\/p>\n\n\n\n
Conntrack metrics from the node exporter<\/a><\/p>\n\n\n\n
Kernel file descriptor metrics from the node exporter<\/a><\/p>\n\n\n\n
Temperature and hardware monitoring metrics from the node exporter<\/a><\/p>\n\n\n\n
Network interface metrics from the node exporter<\/a><\/p>\n\n\n\n
Filesystem metrics from the node exporter<\/a><\/p>\n\n\n\n
Analyse a metric by kernel version<\/a><\/p>\n\n\n\n
Mapping iostat to the node exporter\u2019s node_disk_* metrics<\/a><\/p>\n\n\n\n
Using the textfile collector from a shell script<\/a><\/p>\n\n\n\n
New Features in Node Exporter 0.16.0<\/a><\/p>\n\n\n\n
Using group_left to calculate label proportions<\/a><\/p>\n\n\n\n
Understanding Machine CPU usage<\/a><\/p>\n\n\n\n
Monitoring directory sizes with the Textfile Collector<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
Example PromQL for Prometheus Server Prometheus PromQL Example Query: node exporter Metrics specific to the Node Exporter are prefixed with node_ and include metrics like node_cpu_seconds_total and node_exporter_build_info. Metric Meaning… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[5013],"tags":[],"class_list":["post-22351","post","type-post","status-publish","format-standard","hentry","category-prometheus"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/22351","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=22351"}],"version-history":[{"count":6,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/22351\/revisions"}],"predecessor-version":[{"id":23973,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/22351\/revisions\/23973"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=22351"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=22351"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=22351"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}