{"id":23894,"date":"2021-09-26T17:52:05","date_gmt":"2021-09-26T17:52:05","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=23894"},"modified":"2021-09-26T17:52:06","modified_gmt":"2021-09-26T17:52:06","slug":"prometheus-promql-example-query","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/prometheus-promql-example-query\/","title":{"rendered":"Prometheus PromQL Example Query"},"content":{"rendered":"\n

This should return a number of different time series (along with the latest value recorded for each), all with the metric name promhttp_metric_handler_requests_total, but with different labels. These labels designate different requests statuses.<\/p>\n\n\n\n

Instant vector and Instant vector with Filter<\/h2>\n\n\n
promhttp_metric_handler_requests_total\r\nscrape_duration_seconds{instance=\"localhost:9100\"<\/span>}\r\npromhttp_metric_handler_requests_total{code=\"200\"<\/span>,instance=\"localhost:9100\"<\/span>,job=\"node-exporter\"<\/span>}\r\nscrape_duration_seconds\r\nscrape_duration_seconds{instance=\"172.31.20.76:9182\"<\/span>,job=\"win-exporter\"<\/span>}\r\nnode_cpu_seconds_total\r\nnode_cpu_seconds_total{cpu=\"0\"<\/span>,instance=\"localhost:9100\"<\/span>,job=\"node-exporter\"<\/span>,mode=\"nice\"<\/span>}\r\n# If we were only interested in requests that resulted in HTTP code 200, we could use this query to retrieve that information:<\/span>\r\npromhttp_metric_handler_requests_total{code=\"200\"<\/span>}\r\npromhttp_metric_handler_requests_total{code=\"200\"<\/span>,job=\"node-exporter\"<\/span>}<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Instant vector and Instant vector with Filter using regular Expresson<\/h2>\n\n\n
scrape_duration_seconds{instance=\"localhost:9100\"<\/span>}\r\nscrape_duration_seconds{instance=~\".*9100\"<\/span>}\r\npromhttp_metric_handler_requests_total{code=\"200\"<\/span>,instance=\"localhost:9100\"<\/span>,job=\"node-exporter\"<\/span>}\r\npromhttp_metric_handler_requests_total{code=\"200\"<\/span>,instance=\"localhost:9100\"<\/span>,job=~\".*exporter\"<\/span>}\r\nscrape_duration_seconds{instance=\"172.31.20.76:9182\"<\/span>,job=\"win-exporter\"<\/span>}\r\nscrape_duration_seconds{instance=\"172.31.20.76:9182\"<\/span>,job=~\"win.*\"<\/span>}\r\nnode_cpu_seconds_total\r\nnode_cpu_seconds_total{mode=\"irq\"<\/span>}\r\nnode_cpu_seconds_total{mode=~\".*irq\"<\/span>}\r\nprometheus_http_requests_total{job=~\".*server\"<\/span>}<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Range Vector with filter<\/h2>\n\n\n
promhttp_metric_handler_requests_total[10<\/span>m]\r\nscrape_duration_seconds{instance=\"localhost:9100\"<\/span>}[10<\/span>m]\r\npromhttp_metric_handler_requests_total{code=\"200\"<\/span>,instance=\"localhost:9100\"<\/span>,job=\"node-exporter\"<\/span>}[200<\/span>s]\r\nscrape_duration_seconds[200<\/span>s]\r\nscrape_duration_seconds{instance=\"172.31.20.76:9182\"<\/span>,job=\"win-exporter\"<\/span>}[200<\/span>s]\r\nnode_cpu_seconds_total[3<\/span>h]\r\nnode_cpu_seconds_total{cpu=\"0\"<\/span>,instance=\"localhost:9100\"<\/span>,job=\"node-exporter\"<\/span>,mode=\"nice\"<\/span>}[3<\/span>h]\r\n# If we were only interested in requests that resulted in HTTP code 200, we could use this query to retrieve that information:<\/span>\r\npromhttp_metric_handler_requests_total{code=\"200\"<\/span>}[1<\/span>d]\r\npromhttp_metric_handler_requests_total{code=\"200\"<\/span>,job=\"node-exporter\"<\/span>}[1<\/span>d]<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
offset example<\/h2>\n\n\n
prometheus_http_requests_total\r\nprometheus_http_requests_total{code=\"200\"<\/span>,handler=\"\/metrics\"<\/span>,instance=\"localhost:9090\"<\/span>,job=\"prometheus\"<\/span>}\r\nprometheus_http_requests_total{code=\"200\"<\/span>,handler=\"\/metrics\"<\/span>,instance=\"localhost:9090\"<\/span>,job=\"prometheus\"<\/span>} offset 5<\/span>m\r\nprometheus_http_requests_total offset 5<\/span>m\r\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Scalar Example<\/h2>\n\n\n
sum<\/span>(rate<\/span>(prometheus_http_requests_total<\/span>[5m]<\/span> offset<\/span> 5m<\/span>))<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Sub Queries<\/h2>\n\n\n
node_netstat_Tcp_InSegs\r\nnode_netstat_Tcp_InSegs{instance=\"localhost:9100\"<\/span>,job=\"node-exporter\"<\/span>}\r\nrate(node_netstat_Tcp_InSegs{instance=\"localhost:9100\"<\/span>,job=\"node-exporter\"<\/span>}[5<\/span>m])\nrate(node_netstat_Tcp_InSegs[1<\/span>m])\r\nceil(rate(node_netstat_Tcp_InSegs[1<\/span>m]))\r\nderiv(ceil(rate(node_netstat_Tcp_InSegs[1<\/span>m]))[1<\/span>m:])<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
aggregation operators<\/h2>\n\n\n\n
To count the number of returned time series, you could write:
promhttp_metric_handler_requests_total
count(promhttp_metric_handler_requests_total)<\/p>\n\n\n\n
Functions<\/h2>\n\n\n
rate(promhttp_metric_handler_requests_total)\t#Error<\/span>\r\nrate(promhttp_metric_handler_requests_total{code=\"200\"<\/span>}[1<\/span>m])\r\n# The average amount of CPU time spent in system mode, per second, over the last minute (in seconds)<\/span>\r\nrate(node_cpu_seconds_total{mode=\"system\"<\/span>}[1<\/span>m])\r\n\r\n# The average network traffic received, per second, over the last minute (in bytes)<\/span>\r\nrate(node_network_receive_bytes_total[1<\/span>m])\r\n\r\nrate(scrape_duration_seconds{instance=\"localhost:9100\"<\/span>}[1<\/span>m:20<\/span>s])\r\n\r\n# Usability of such graphs is close to zero, because they show hard-to-interpret constantly growing counter values, while we need graphs for network bandwidth \u2014 see MB\/s on the left of the graph. PromQL has a magic function for this \u2014 rate(). It calculates per-second rate for all the matching time series:<\/span>\r\nrate(node_network_receive_bytes_total[5<\/span>m])<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
aggregation operatos with Functions<\/h2>\n\n\n
sum<\/span> by<\/span> (job<\/span>) (\r\n  rate<\/span>(prometheus_http_requests_total<\/span>[5m]<\/span>)\r\n)<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
A set of time series containing a range of data points over time for each time series. Return a whole range of scrape_duration_seconds (in this case 5 minutes) for the same vector, making it a range vector.<\/p>\n\n\n
promhttp_metric_handler_requests_total[5<\/span>m]\r\nnode_netstat_Tcp_InSegs{instance=\"localhost:9100\"<\/span>}[5<\/span>m] \r\nnode_netstat_Tcp_InSegs{instance=\"localhost:9100\"<\/span>}[1<\/span>m] \r\nnode_netstat_Tcp_InSegs{instance=\"localhost:9100\"<\/span>}[30<\/span>s]\r\n\r\nReturn<\/span> all time series with the metric http_requests_total:\r\nprometheus_http_requests_total\r\n\r\nReturn<\/span> all time series with the metric http_requests_total and<\/span> the given job and<\/span> handler labels:\r\nprometheus_http_requests_total{job=\"apiserver\"<\/span>, handler=\"\/api\/comments\"<\/span>}\r\n\r\nReturn<\/span> a whole range of time (in this case<\/span> 5<\/span> minutes) for<\/span> the same vector, making it a range vector:\r\nprometheus_http_requests_total{job=\"apiserver\"<\/span>, handler=\"\/api\/comments\"<\/span>}[5<\/span>m]\r\n\r\nNote that an expression resulting in a range vector cannot be graphed directly, but viewed in the tabular (\"Console\"<\/span>) view of the expression browser.\r\n\r\nUsing regular expressions, you could select time series only for<\/span> jobs whose name match a certain pattern, in this case<\/span>, all jobs that end with server:\r\n\r\n\r\nTo select all HTTP status codes except 4<\/span>xx ones, you could run:\r\nprometheus_http_requests_total{status!~\"4..\"<\/span>}\r\n\r\nSubquery- Return<\/span> the 5<\/span>-minute rate of the http_requests_total metric for<\/span> the past 30<\/span> minutes, with a resolution of 1<\/span> minute.\r\nrate(prometheus_http_requests_total[5<\/span>m])[30<\/span>m:1<\/span>m]\r\n\r\nThis is an example of a nested subquery. The subquery for<\/span> the deriv function<\/span> uses<\/span> the<\/span> default<\/span> resolution<\/span>. Note<\/span> that<\/span> using<\/span> subqueries<\/span> unnecessarily<\/span> is<\/span> unwise<\/span>.\r\nmax_over_time<\/span>(deriv(rate(distance_covered_total[5<\/span>s])<\/span>[30<\/span>s:5<\/span>s])<\/span>[10<\/span>m:])<\/span>\r\n\r\nUsing<\/span> functions<\/span>, operators<\/span>, etc<\/span> - Return<\/span> the<\/span> per<\/span>-second<\/span> rate<\/span> for<\/span> all<\/span> time<\/span> series<\/span> with<\/span> the<\/span> http_requests_total<\/span> metric<\/span> name<\/span>, as<\/span> measured<\/span> over<\/span> the<\/span> last<\/span> 5 minutes<\/span>:\r\nrate<\/span>(prometheus_http_requests_total[5<\/span>m])<\/span>\r\n\r\nAssuming<\/span> that<\/span> the<\/span> http_requests_total<\/span> time<\/span> series<\/span> all<\/span> have<\/span> the<\/span> labels<\/span> job<\/span> (fanout by job name)<\/span> and<\/span> instance<\/span> (fanout by instance of the job)<\/span>, we<\/span> might<\/span> want<\/span> to<\/span> sum<\/span> over<\/span> the<\/span> rate<\/span> of<\/span> all<\/span> instances<\/span>, so<\/span> we<\/span> get<\/span> fewer<\/span> output<\/span> time<\/span> series<\/span>, but<\/span> still<\/span> preserve<\/span> the<\/span> job<\/span> dimension<\/span>:\r\nsum<\/span> by<\/span> (job)<\/span> (\r\n  rate(prometheus_http_requests_total[5<\/span>m])<\/span>\r\n)<\/span><\/span><\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
The filesystem space available to non-root users (in bytes)<\/h2>\n\n\n\n
node_filesystem_avail_bytes<\/p>\n\n\n\n
Filtering by label- A single metric name may correspond to multiple time series with distinct label sets as in the example above. How to select time series matching only {device=”eth1″}? Just mention the required label in the query:
node_network_receive_bytes_total{device=”eth1″}<\/p>\n\n\n\n
If you want selecting all the time series for devices other than eth1, then just substitute = with != in the query:
node_network_receive_bytes_total{device!=”eth1″}<\/p>\n\n\n\n
How to select time series for devices starting with eth? Just use regular expressions:
node_network_receive_bytes_total{device=~”eth.+”}<\/p>\n\n\n\n
For selecting all the time series for devices not starting with eth, the =~ must be substituted with !~:
node_network_receive_bytes_total{device!~”eth.+”}<\/p>\n\n\n\n
Filtering by multiple labels
Label filters may be combined. For instance, the following query would return only time series on the instance node42:9100 for devices starting with eth:
node_network_receive_bytes_total{instance=”node42:9100″, device=~”eth.+”}<\/p>\n\n\n\n
Filtering by regexps on metric name
Sometimes it is required returning all the time series for multiple metric names. Metric name is just an ordinary label with a special name \u2014 name<\/strong>. So filtering by multiple metric names may be performed by applying regexps on metric names. For instance, the following query returns all the time series with node_network_receive_bytes_total or node_network_transmit_bytes_total metric names:
{name<\/strong>=~”node_network_(receive|transmit)_bytes_total”}<\/p>\n\n\n\n
Comparing current data with historical data
PromQL allows querying historical data and combining \/ comparing it to the current data. Just add offset to the query. For instance, the following query would return week-old data for all the time series with node_network_receive_bytes_total name:
node_network_receive_bytes_total offset 7d<\/p>\n\n\n\n
If we have two different metrics with the same dimensional labels, we can apply binary operators to them and elements on both sides with the same label set will get matched and propagated to the output. For example, this expression returns the unused memory in MiB for every instance (on a fictional cluster scheduler exposing these metrics about the instances it runs):
(instance_memory_limit_bytes – instance_memory_usage_bytes) \/ 1024 \/ 1024<\/p>\n\n\n\n
The same expression, but summed by application, could be written like this:
sum by (app, proc) (
instance_memory_limit_bytes – instance_memory_usage_bytes
) \/ 1024 \/ 1024<\/p>\n\n\n\n
we could get the top 3 CPU users grouped by application (app) and process type (proc) like this:
topk(3, sum by (app, proc) (rate(instance_cpu_time_ns[5m])))<\/p>\n\n\n\n
Assuming this metric contains one time series per running instance, you could count the number of running instances per application like this:
count by (app) (instance_cpu_time_ns)<\/p>\n\n\n\n
Reference<\/h1>\n\n\n\n
  • https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/examples\/<\/li>
  • https:\/\/timber.io\/blog\/promql-for-humans\/<\/li>
  • https:\/\/prometheus.io\/docs\/introduction\/first_steps\/<\/li>
  • https:\/\/medium.com\/@valyala\/promql-tutorial-for-beginners-9ab455142085<\/li>
  • https:\/\/prometheus.io\/docs\/prometheus\/latest\/querying\/basics\/<\/li>
  • https:\/\/sbcode.net\/prometheus\/example-queries\/<\/li>
  • https:\/\/files.timber.io\/pdfs\/PromQL+Cheatsheet.pdf<\/li><\/ul>\n","protected":false},"excerpt":{"rendered":"
    This should return a number of different time series (along with the latest value recorded for each), all with the metric name promhttp_metric_handler_requests_total, but with different labels. These labels designate… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[2],"tags":[],"class_list":["post-23894","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/23894","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=23894"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/23894\/revisions"}],"predecessor-version":[{"id":23895,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/23894\/revisions\/23895"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=23894"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=23894"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=23894"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}