{"id":23932,"date":"2021-09-28T02:45:45","date_gmt":"2021-09-28T02:45:45","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=23932"},"modified":"2021-09-28T05:30:33","modified_gmt":"2021-09-28T05:30:33","slug":"using-expression-browser-in-prometheus","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/using-expression-browser-in-prometheus\/","title":{"rendered":"Using Expression browser in Prometheus"},"content":{"rendered":"\n<p>The expression browser is available at \/graph on the Prometheus server, allowing you to enter any expression and see its result either in a table or graphed over time.<\/p>\n\n\n\n<p>This is primarily useful for ad-hoc queries and debugging. For graphs, use Grafana or Console templates.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"708\" height=\"531\" src=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/Expression-browser-prometheus-1.jpg\" alt=\"\" class=\"wp-image-23934\" srcset=\"https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/Expression-browser-prometheus-1.jpg 708w, https:\/\/www.devopsschool.com\/blog\/wp-content\/uploads\/2021\/09\/Expression-browser-prometheus-1-300x225.jpg 300w\" sizes=\"auto, (max-width: 708px) 100vw, 708px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Using the expression browser<\/h2>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-1\" data-shcb-language-name=\"JavaScript\" data-shcb-language-slug=\"javascript\"><span><code class=\"hljs language-javascript\">As you can gather <span class=\"hljs-keyword\">from<\/span> localhost:<span class=\"hljs-number\">9090<\/span>\/metrics, one metric that Prometheus exports about itself is named prometheus_target_interval_length_seconds (the actual amount <span class=\"hljs-keyword\">of<\/span> time between target scrapes). Enter the below into the expression <span class=\"hljs-built_in\">console<\/span> and then click <span class=\"hljs-string\">\"Execute\"<\/span>:\n\n-------------------------------------------\nprometheus_target_interval_length_seconds\n-------------------------------------------\n\nThis should <span class=\"hljs-keyword\">return<\/span> a number <span class=\"hljs-keyword\">of<\/span> different time series (along <span class=\"hljs-keyword\">with<\/span> the latest value recorded <span class=\"hljs-keyword\">for<\/span> each), each <span class=\"hljs-keyword\">with<\/span> the metric name prometheus_target_interval_length_seconds, but <span class=\"hljs-keyword\">with<\/span> different labels. These labels designate different latency percentiles and target group intervals.\n\nIf we are interested only <span class=\"hljs-keyword\">in<\/span> <span class=\"hljs-number\">99<\/span>th percentile latencies, we could use <span class=\"hljs-keyword\">this<\/span> query:\n\n-------------------------------------------\nprometheus_target_interval_length_seconds{quantile=<span class=\"hljs-string\">\"0.99\"<\/span>}\n-------------------------------------------\n\nTo count the number <span class=\"hljs-keyword\">of<\/span> returned time series, you could write:\n\n-------------------------------------------\ncount(prometheus_target_interval_length_seconds)\n-------------------------------------------\n\n<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-1\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">JavaScript<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">javascript<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<h2 class=\"wp-block-heading\">Using the graphing interface<\/h2>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-2\" data-shcb-language-name=\"JavaScript\" data-shcb-language-slug=\"javascript\"><span><code class=\"hljs language-javascript\">To graph expressions, navigate to http:<span class=\"hljs-comment\">\/\/localhost:9090\/graph and use the \"Graph\" tab.<\/span>\n\nFor example, enter the following expression to graph the per-second rate <span class=\"hljs-keyword\">of<\/span> chunks being created <span class=\"hljs-keyword\">in<\/span> the self-scraped Prometheus:\n\n-------------------------------------------\nrate(prometheus_tsdb_head_chunks_created_total&#91;<span class=\"hljs-number\">1<\/span>m])\n-------------------------------------------\n\n\nExperiment <span class=\"hljs-keyword\">with<\/span> the graph range parameters and other settings.<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-2\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">JavaScript<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">javascript<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>\n\n\n<h2 class=\"wp-block-heading\">Configure Prometheus to monitor the sample targets such as Node Exporter<\/h2>\n\n\n\n<figure class=\"wp-block-embed is-type-wp-embed is-provider-devopsschool-com wp-block-embed-devopsschool-com\"><div class=\"wp-block-embed__wrapper\">\n<blockquote class=\"wp-embedded-content\" data-secret=\"FTladNUErL\"><a href=\"https:\/\/www.devopsschool.com\/blog\/install-and-configure-prometheus-server-and-node-exporter-in-rhel\/\">Install and Configure Prometheus Server and Node Exporter in RHEL<\/a><\/blockquote><iframe loading=\"lazy\" class=\"wp-embedded-content\" sandbox=\"allow-scripts\" security=\"restricted\" style=\"position: absolute; clip: rect(1px, 1px, 1px, 1px);\" title=\"&#8220;Install and Configure Prometheus Server and Node Exporter in RHEL&#8221; &#8212; DevOpsSchool.com\" src=\"https:\/\/www.devopsschool.com\/blog\/install-and-configure-prometheus-server-and-node-exporter-in-rhel\/embed\/#?secret=FTladNUErL\" data-secret=\"FTladNUErL\" width=\"600\" height=\"338\" frameborder=\"0\" marginwidth=\"0\" marginheight=\"0\" scrolling=\"no\"><\/iframe>\n<\/div><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Example queries that aggregate over thousands<\/h2>\n\n\n<pre class=\"wp-block-code\" aria-describedby=\"shcb-language-3\" data-shcb-language-name=\"JavaScript\" data-shcb-language-slug=\"javascript\"><span><code class=\"hljs language-javascript\">Though not a problem <span class=\"hljs-keyword\">in<\/span> our example, queries that aggregate over thousands <span class=\"hljs-keyword\">of<\/span> time series can <span class=\"hljs-keyword\">get<\/span> slow when computed ad-hoc. To make this more efficient, Prometheus can prerecord expressions into new persisted time series via configured recording rules. Let's say we are interested in recording the per-second rate of cpu time (node_cpu_seconds_total) averaged over all cpus per instance (but preserving the job, instance and mode dimensions) as measured over a window of 5 minutes. We could write this as:\n\n-------------------------------------------\navg by (job, instance, mode) (rate(node_cpu_seconds_total&#91;5m]))\n-------------------------------------------\n\n<\/code><\/span><small class=\"shcb-language\" id=\"shcb-language-3\"><span class=\"shcb-language__label\">Code language:<\/span> <span class=\"shcb-language__name\">JavaScript<\/span> <span class=\"shcb-language__paren\">(<\/span><span class=\"shcb-language__slug\">javascript<\/span><span class=\"shcb-language__paren\">)<\/span><\/small><\/pre>","protected":false},"excerpt":{"rendered":"<p>The expression browser is available at \/graph on the Prometheus server, allowing you to enter any expression and see its result either in a table or graphed over time. This&#8230; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[2],"tags":[],"class_list":["post-23932","post","type-post","status-publish","format-standard","hentry","category-uncategorised"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/23932","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=23932"}],"version-history":[{"count":5,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/23932\/revisions"}],"predecessor-version":[{"id":23950,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/23932\/revisions\/23950"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=23932"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=23932"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=23932"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}