{"id":75277,"date":"2026-04-29T08:12:32","date_gmt":"2026-04-29T08:12:32","guid":{"rendered":"https:\/\/www.devopsschool.com\/blog\/?p=75277"},"modified":"2026-04-29T08:12:34","modified_gmt":"2026-04-29T08:12:34","slug":"grafana-13-x-recording-rules-lab-manual-with-graphite","status":"publish","type":"post","link":"https:\/\/www.devopsschool.com\/blog\/grafana-13-x-recording-rules-lab-manual-with-graphite\/","title":{"rendered":"Grafana 13.x Recording Rules Lab Manual with Graphite"},"content":{"rendered":"\n

Using Graphite + Telegraf Linux Metrics<\/h2>\n\n\n\n

You are on the correct screen now: Alerting \u2192 Alert rules \u2192 New recording rule<\/strong>.<\/p>\n\n\n\n

But one important point first: Grafana recording rules are not stored back into Graphite.<\/strong> In Grafana 13, a Grafana-managed recording rule can read from a supported data source such as Graphite, but the recorded result must be written into a Prometheus-compatible data source<\/strong>. Grafana does not include its own time-series database for recording rule output. (Grafana Labs<\/a>)<\/p>\n\n\n\n

Your Graphite data is valid for this lab. The real metric prefix is:<\/p>\n\n\n

telegraf<\/span>.linux-demo<\/span>.*\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
The uploaded metric output also confirms that servers.*<\/code> returns empty, so this lab will not use servers.*<\/code>. It will only use the real captured Telegraf metrics from Graphite.<\/p>\n\n\n\n
\n\n\n\n
1. What is a Grafana Recording Rule?<\/h1>\n\n\n\n
A recording rule<\/strong> pre-calculates a query result and saves it as a new metric.<\/p>\n\n\n\n
Example:<\/p>\n\n\n\n
Instead of repeatedly querying this Graphite metric:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.cpu<\/span>.usage_active<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Grafana can evaluate it every 1 minute and store the result as a new Prometheus-style metric:<\/p>\n\n\n
linux_cpu_usage_active_percent\n<\/code><\/span><\/pre>\n\n\n
Then dashboards and alerts can query the new recorded metric.<\/p>\n\n\n\n
Simple explanation for students<\/h2>\n\n\n
Normal dashboard query:\nGrafana asks Graphite every time the dashboard loads.\n\nRecording rule:\nGrafana runs the query on schedule and stores the result as<\/span> a new<\/span> metric.\n\nAlert rule:\nGrafana checks a metric and fires an alert when a condition is true<\/span>.\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
2. Very Important: Recording Rule vs Alert Rule<\/h1>\n\n\n\n
Option<\/th>Purpose<\/th>Use in this lab?<\/th><\/tr><\/thead>
New alert rule<\/strong><\/td>Creates alert condition like CPU > 90%<\/td>Yes<\/td><\/tr>
New recording rule<\/strong><\/td>Creates a new precomputed metric<\/td>Yes, for this lab<\/td><\/tr>
New recording rule with CloudWatch<\/strong><\/td>Not for your Graphite lab<\/td>No<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
In your screenshot, the query section currently shows CloudWatch<\/strong>. That is why you see an error about CloudWatch statistics. For this lab, students must select the Graphite data source<\/strong> as the query source.<\/p>\n\n\n\n
\n\n\n\n
3. Architecture for This Recording Rule Lab<\/h1>\n\n\n\n
The flow should be explained like this:<\/p>\n\n\n
Telegraf\n   \u2193\nGraphite\n   \u2193\nGrafana recording rule reads Graphite metric\n   \u2193\nGrafana writes recorded result into Prometheus-compatible data source\n   \u2193\nDashboard and<\/span> alert can use<\/span> the<\/span> recorded<\/span> metric<\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
So we need two Grafana data sources:<\/p>\n\n\n
1.<\/span> Graphite\n   Used as<\/span> the source query data source.\n\n2.<\/span> Prometheus-compatible data source\n   Used as<\/span> the target data source to store recording rule output.\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Grafana documentation says Grafana-managed recording rules can query alerting-supported data sources, but the result must be stored in a Prometheus-compatible database. (Grafana Labs<\/a>)<\/p>\n\n\n\n
\n\n\n\n
4. Prerequisites for Students<\/h1>\n\n\n\n
Before starting this lab, students should already have:<\/p>\n\n\n
Grafana<\/span> 13.x<\/span> running<\/span> on<\/span> port<\/span> 3000\nGraphite<\/span> running<\/span> on<\/span> port<\/span> 8080\nTelegraf<\/span> sending<\/span> Linux<\/span> metrics<\/span> to<\/span> Graphite<\/span>\nGraphite<\/span> data<\/span> source<\/span> added<\/span> in<\/span> Grafana<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Current valid Graphite metrics include:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.cpu<\/span>.*\ntelegraf<\/span>.linux-demo<\/span>.mem<\/span>.*\ntelegraf<\/span>.linux-demo<\/span>.disk<\/span>.*\ntelegraf<\/span>.linux-demo<\/span>.diskio<\/span>.*\ntelegraf<\/span>.linux-demo<\/span>.net<\/span>.*\ntelegraf<\/span>.linux-demo<\/span>.kernel<\/span>.*\ntelegraf<\/span>.linux-demo<\/span>.processes<\/span>.*\ntelegraf<\/span>.linux-demo<\/span>.swap<\/span>.*\ntelegraf<\/span>.linux-demo<\/span>.system<\/span>.*\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Do not use:<\/p>\n\n\n
servers.*\n<\/code><\/span><\/pre>\n\n\n
Do not use .wsp<\/code> in Grafana queries.<\/p>\n\n\n\n
Correct:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.cpu<\/span>.usage_active<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Wrong:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.cpu<\/span>.usage_active<\/span>.wsp<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
5. Add a Prometheus-Compatible Target for Recording Rules<\/h1>\n\n\n\n
If your Target data source<\/strong> dropdown is empty or does not show a Prometheus-compatible data source, you need to add one.<\/p>\n\n\n\n
For a classroom lab, the simplest option is to run a small Prometheus container as the recording-rule target.<\/p>\n\n\n\n
Step 1: Create a Prometheus config file<\/h2>\n\n\n\n
On the Linux VM, run:<\/p>\n\n\n
mkdir -p ~\/prometheus-recording-lab\ncd ~\/prometheus-recording-lab\n<\/code><\/span><\/pre>\n\n\n
Create a file:<\/p>\n\n\n
cat > prometheus.yml <<'EOF'<\/span>\nglobal<\/span>:\n  scrape_interval: 15<\/span>s\n  evaluation_interval: 15<\/span>s\n\nscrape_configs:\n  - job_name: prometheus\n    static_configs:\n      - targets: [\"localhost:9090\"<\/span>]\nEOF\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Step 2: Start Prometheus container<\/h2>\n\n\n
docker run -d \\\n  --name prometheus-recording \\\n  --restart unless-stopped \\\n  -p 9090<\/span>:9090<\/span> \\\n  -v ~\/prometheus-recording-lab\/<\/span>prometheus.yml:\/etc\/<\/span>prometheus\/prometheus.yml \\\n  prom\/prometheus:latest \\\n  --config.file=\/etc\/<\/span>prometheus\/prometheus.yml \\\n  --web.enable-remote-write-receiver \\\n  --web.enable-lifecycle\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Prometheus can receive remote-write data when the remote write receiver is enabled, and its receiver endpoint is \/api\/v1\/write<\/code>. (Prometheus<\/a>)<\/p>\n\n\n\n
Step 3: Verify Prometheus is running<\/h2>\n\n\n
docker ps\n<\/code><\/span><\/pre>\n\n\n
Expected container:<\/p>\n\n\n
prometheus-recording\n<\/code><\/span><\/pre>\n\n\n
Test from the Linux VM:<\/p>\n\n\n
curl \"http:\/\/localhost:9090\/-\/ready\"<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Expected output:<\/p>\n\n\n
Prometheus Server is Ready.\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
6. Add Prometheus as a Grafana Data Source<\/h1>\n\n\n\n
In Grafana 13.x left menu:<\/p>\n\n\n
Connections \u2192 Add new<\/span> connection\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Search:<\/p>\n\n\n
Prometheus\n<\/code><\/span><\/pre>\n\n\n
Click:<\/p>\n\n\n
Add new<\/span> data source\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Use this name:<\/p>\n\n\n
prometheus-recording\n<\/code><\/span><\/pre>\n\n\n
Use this URL if Grafana is installed directly on the same VM:<\/p>\n\n\n
http:\/\/localhost:9090<\/span>\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
If Grafana is running inside Docker, localhost<\/code> may not work because it points to the Grafana container itself. In that case, use the VM private IP, public IP, or Docker network name.<\/p>\n\n\n\n
Click:<\/p>\n\n\n
Save & test\n<\/code><\/span><\/pre>\n\n\n
Expected result:<\/p>\n\n\n
Successfully queried the Prometheus API.\n<\/code><\/span><\/pre>\n\n\n
In Prometheus data source settings, Grafana has an alerting option called Allow as recording rules target<\/strong>, which allows that data source to appear as a target when creating Grafana-managed recording rules. (Grafana Labs<\/a>)<\/p>\n\n\n\n
\n\n\n\n
7. Verify Graphite Query Before Creating Recording Rule<\/h1>\n\n\n\n
Before creating recording rules, students should test the Graphite query.<\/p>\n\n\n\n
In Grafana:<\/p>\n\n\n
Explore\n<\/code><\/span><\/pre>\n\n\n
Select data source:<\/p>\n\n\n
Graphite\n<\/code><\/span><\/pre>\n\n\n
Run:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.cpu<\/span>.usage_active<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
If data appears, the query is correct.<\/p>\n\n\n\n
Also test:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.mem<\/span>.used_percent<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n
telegraf<\/span>.linux-demo<\/span>.disk<\/span>.used_percent<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
8. Create First Recording Rule: CPU Usage<\/h1>\n\n\n\n
Now go to:<\/p>\n\n\n
Alerting \u2192 Alert rules \u2192 New<\/span> recording rule\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Your screen has these main sections:<\/p>\n\n\n
1.<\/span> Enter recording rule and metric name\n2.<\/span> Define recording rule\n3.<\/span> Add folder and labels\n4.<\/span> Set<\/span> evaluation behavior\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Section 1: Enter Recording Rule and Metric Name<\/h2>\n\n\n\n
Fill the fields like this:<\/p>\n\n\n\n
Name<\/h3>\n\n\n
record_cpu_usage_active_percent\n<\/code><\/span><\/pre>\n\n\n
Metric<\/h3>\n\n\n
linux_cpu_usage_active_percent\n<\/code><\/span><\/pre>\n\n\n
Target data source<\/h3>\n\n\n\n
Select:<\/p>\n\n\n
prometheus-recording\n<\/code><\/span><\/pre>\n\n\n
Important: the metric name must follow Prometheus metric naming style and must not contain spaces. Grafana\u2019s recording rule documentation also says the metric name must be a Prometheus metric name with no whitespace. (Grafana Labs<\/a>)<\/p>\n\n\n\n
Do not use dots in the recorded metric name.<\/p>\n\n\n\n
Wrong:<\/p>\n\n\n
linux<\/span>.cpu<\/span>.usage<\/span>.active<\/span>.percent<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Correct:<\/p>\n\n\n
linux_cpu_usage_active_percent\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Section 2: Define Recording Rule<\/h2>\n\n\n\n
In the query area, change the data source from CloudWatch to:<\/p>\n\n\n
Graphite\n<\/code><\/span><\/pre>\n\n\n
Use this Graphite query:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.cpu<\/span>.usage_active<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Set time range:<\/p>\n\n\n
10m to now\n<\/code><\/span><\/pre>\n\n\n
Click:<\/p>\n\n\n
Run queries\n<\/code><\/span><\/pre>\n\n\n
You should see data in the preview graph.<\/p>\n\n\n\n
Now add an expression:<\/p>\n\n\n
Add expression \u2192 Reduce\n<\/code><\/span><\/pre>\n\n\n
Set:<\/p>\n\n\n
Input<\/span>: A\nFunction<\/span>: Last\nMode<\/span>: Strict\n<\/code><\/span>Code language:<\/span> HTTP<\/span> (<\/span>http<\/span>)<\/span><\/small><\/pre>\n\n\n
Click:<\/p>\n\n\n
Preview\n<\/code><\/span><\/pre>\n\n\n
Then click:<\/p>\n\n\n
Set<\/span> as<\/span> recording rule output\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Use the Reduce output<\/strong> as the recording rule output.<\/p>\n\n\n\n
Why Reduce? Because Graphite returns a time series. The recording rule should store a clean value at every evaluation interval.<\/p>\n\n\n\n
\n\n\n\n
Section 3: Add Folder and Labels<\/h2>\n\n\n\n
Folder:<\/p>\n\n\n
Linux Monitoring Lab\n<\/code><\/span><\/pre>\n\n\n
If it does not exist, click:<\/p>\n\n\n
New<\/span> folder\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Add labels:<\/p>\n\n\n
source = graphite\nhost = linux-demo\ncollector = telegraf\nlab = grafana13\n<\/code><\/span><\/pre>\n\n\n
Labels help students search, filter, and organize rules.<\/p>\n\n\n\n
\n\n\n\n
Section 4: Set Evaluation Behavior<\/h2>\n\n\n\n
Create or select evaluation group:<\/p>\n\n\n
graphite-recording-rules-1m\n<\/code><\/span><\/pre>\n\n\n
Evaluation interval:<\/p>\n\n\n
1m\n<\/code><\/span><\/pre>\n\n\n
This means Grafana will run the recording rule every 1 minute.<\/p>\n\n\n\n
Click:<\/p>\n\n\n
Save rule\n<\/code><\/span><\/pre>\n\n\n
or:<\/p>\n\n\n
Save rule and<\/span> exit<\/span>\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
9. Verify the Recorded Metric<\/h1>\n\n\n\n
Wait 1\u20132 minutes.<\/p>\n\n\n\n
Go to:<\/p>\n\n\n
Explore\n<\/code><\/span><\/pre>\n\n\n
Select data source:<\/p>\n\n\n
prometheus-recording\n<\/code><\/span><\/pre>\n\n\n
Run:<\/p>\n\n\n
linux_cpu_usage_active_percent\n<\/code><\/span><\/pre>\n\n\n
If labels were added, you can also query:<\/p>\n\n\n
linux_cpu_usage_active_percent{host=\"linux-demo\"<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Expected result:<\/p>\n\n\n
A new<\/span> Prometheus-style metric appears.\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
This metric was created from the original Graphite metric:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.cpu<\/span>.usage_active<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
10. Required Recording Rules for Student Lab<\/h1>\n\n\n\n
Create these recording rules one by one. They only use the real Graphite metrics captured in your environment.<\/p>\n\n\n\n
Rule 1: CPU Active Usage<\/h2>\n\n\n
Name<\/span>:\nrecord_cpu_usage_active_percent<\/span>\n\nMetric<\/span>:\nlinux_cpu_usage_active_percent<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.cpu<\/span>.usage_active<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 2: Memory Used Percent<\/h2>\n\n\n
Name<\/span>:\nrecord_memory_used_percent<\/span>\n\nMetric<\/span>:\nlinux_memory_used_percent<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.mem<\/span>.used_percent<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 3: Disk Used Percent<\/h2>\n\n\n
Name<\/span>:\nrecord_disk_used_percent<\/span>\n\nMetric<\/span>:\nlinux_disk_used_percent<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.disk<\/span>.used_percent<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 4: Disk Inodes Used Percent<\/h2>\n\n\n
Name<\/span>:\nrecord_disk_inodes_used_percent<\/span>\n\nMetric<\/span>:\nlinux_disk_inodes_used_percent<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.disk<\/span>.inodes_used_percent<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 5: Swap Used Percent<\/h2>\n\n\n
Name<\/span>:\nrecord_swap_used_percent<\/span>\n\nMetric<\/span>:\nlinux_swap_used_percent<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.swap<\/span>.used_percent<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 6: System Load 1 Minute<\/h2>\n\n\n
Name<\/span>:\nrecord_system_load1<\/span>\n\nMetric<\/span>:\nlinux_system_load1<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.system<\/span>.load1<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 7: Running Processes<\/h2>\n\n\n
Name<\/span>:\nrecord_processes_running<\/span>\n\nMetric<\/span>:\nlinux_processes_running<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.processes<\/span>.running<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 8: Zombie Processes<\/h2>\n\n\n
Name<\/span>:\nrecord_processes_zombies<\/span>\n\nMetric<\/span>:\nlinux_processes_zombies<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.processes<\/span>.zombies<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 9: Blocked Processes<\/h2>\n\n\n
Name<\/span>:\nrecord_processes_blocked<\/span>\n\nMetric<\/span>:\nlinux_processes_blocked<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.processes<\/span>.blocked<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 10: Disk I\/O Utilization<\/h2>\n\n\n
Name<\/span>:\nrecord_diskio_util_percent<\/span>\n\nMetric<\/span>:\nlinux_diskio_util_percent<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.diskio<\/span>.io_util<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 11: Network Receive Bytes per Second<\/h2>\n\n\n
Name<\/span>:\nrecord_network_receive_bytes_per_second<\/span>\n\nMetric<\/span>:\nlinux_network_receive_bytes_per_second<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\nperSecond<\/span>(telegraf<\/span>.linux-demo<\/span>.net<\/span>.bytes_recv<\/span>)\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 12: Network Transmit Bytes per Second<\/h2>\n\n\n
Name<\/span>:\nrecord_network_transmit_bytes_per_second<\/span>\n\nMetric<\/span>:\nlinux_network_transmit_bytes_per_second<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\nperSecond<\/span>(telegraf<\/span>.linux-demo<\/span>.net<\/span>.bytes_sent<\/span>)\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 13: Disk Read Bytes per Second<\/h2>\n\n\n
Name<\/span>:\nrecord_disk_read_bytes_per_second<\/span>\n\nMetric<\/span>:\nlinux_disk_read_bytes_per_second<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\nperSecond<\/span>(telegraf<\/span>.linux-demo<\/span>.diskio<\/span>.read_bytes<\/span>)\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 14: Disk Write Bytes per Second<\/h2>\n\n\n
Name<\/span>:\nrecord_disk_write_bytes_per_second<\/span>\n\nMetric<\/span>:\nlinux_disk_write_bytes_per_second<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\nperSecond<\/span>(telegraf<\/span>.linux-demo<\/span>.diskio<\/span>.write_bytes<\/span>)\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 15: Network Input Errors<\/h2>\n\n\n
Name<\/span>:\nrecord_network_input_errors<\/span>\n\nMetric<\/span>:\nlinux_network_input_errors<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.net<\/span>.err_in<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 16: Network Output Errors<\/h2>\n\n\n
Name<\/span>:\nrecord_network_output_errors<\/span>\n\nMetric<\/span>:\nlinux_network_output_errors<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\ntelegraf<\/span>.linux-demo<\/span>.net<\/span>.err_out<\/span>\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Rule 17: Kernel Context Switches per Second<\/h2>\n\n\n
Name<\/span>:\nrecord_kernel_context_switches_per_second<\/span>\n\nMetric<\/span>:\nlinux_kernel_context_switches_per_second<\/span>\n\nSource<\/span> data<\/span> source<\/span>:\nGraphite<\/span>\n\nQuery<\/span>:\nperSecond<\/span>(telegraf<\/span>.linux-demo<\/span>.kernel<\/span>.context_switches<\/span>)\n\nExpression<\/span>:\nReduce<\/span> \u2192 Last<\/span>\n\nTarget<\/span> data<\/span> source<\/span>:\nprometheus-recording<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
11. Recommended Student Practice Sequence<\/h1>\n\n\n\n
Do not ask students to create all rules immediately. Use this classroom sequence:<\/p>\n\n\n\n
Beginner Practice<\/h2>\n\n\n\n
Create only these 5 recording rules:<\/p>\n\n\n
linux_cpu_usage_active_percent\nlinux_memory_used_percent\nlinux_disk_used_percent\nlinux_system_load1\nlinux_processes_zombies\n<\/code><\/span><\/pre>\n\n\n
Intermediate Practice<\/h2>\n\n\n\n
Add these:<\/p>\n\n\n
linux_swap_used_percent\nlinux_diskio_util_percent\nlinux_network_receive_bytes_per_second\nlinux_network_transmit_bytes_per_second\n<\/code><\/span><\/pre>\n\n\n
Advanced Practice<\/h2>\n\n\n\n
Add these:<\/p>\n\n\n
linux_disk_read_bytes_per_second\nlinux_disk_write_bytes_per_second\nlinux_network_input_errors\nlinux_network_output_errors\nlinux_kernel_context_switches_per_second\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
12. Create Dashboard Panels from Recorded Metrics<\/h1>\n\n\n\n
After recording rules are working, create a new dashboard.<\/p>\n\n\n\n
Go to:<\/p>\n\n\n
Dashboards \u2192 New<\/span> \u2192 New<\/span> dashboard \u2192 Add visualization\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
Select data source:<\/p>\n\n\n
prometheus-recording\n<\/code><\/span><\/pre>\n\n\n
Now use the recorded metrics, not Graphite.<\/p>\n\n\n\n
Panel 1: Recorded CPU Usage<\/h2>\n\n\n\n
Panel title:<\/p>\n\n\n
Recorded CPU Usage %\n<\/code><\/span><\/pre>\n\n\n
Query:<\/p>\n\n\n
linux_cpu_usage_active_percent{host=\"linux-demo\"<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Visualization:<\/p>\n\n\n
Stat\n<\/code><\/span><\/pre>\n\n\n
Unit:<\/p>\n\n\n
Percent\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Panel 2: Recorded Memory Usage<\/h2>\n\n\n
linux_memory_used_percent{host=\"linux-demo\"<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Visualization:<\/p>\n\n\n
Gauge\n<\/code><\/span><\/pre>\n\n\n
Unit:<\/p>\n\n\n
Percent\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Panel 3: Recorded Disk Usage<\/h2>\n\n\n
linux_disk_used_percent{host=\"linux-demo\"<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Visualization:<\/p>\n\n\n
Gauge\n<\/code><\/span><\/pre>\n\n\n
Unit:<\/p>\n\n\n
Percent\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Panel 4: Recorded System Load<\/h2>\n\n\n
linux_system_load1{host=\"linux-demo\"<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Visualization:<\/p>\n\n\n
Time series\n<\/code><\/span><\/pre>\n\n\n
Unit:<\/p>\n\n\n
None\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Panel 5: Recorded Network Traffic<\/h2>\n\n\n\n
Queries:<\/p>\n\n\n
linux_network_receive_bytes_per_second{host=\"linux-demo\"<\/span>}\nlinux_network_transmit_bytes_per_second{host=\"linux-demo\"<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Visualization:<\/p>\n\n\n
Time series\n<\/code><\/span><\/pre>\n\n\n
Unit:<\/p>\n\n\n
Bytes\/sec\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
13. Create Alerts from Recorded Metrics<\/h1>\n\n\n\n
Now students can create alert rules using the recorded metrics<\/strong>.<\/p>\n\n\n\n
Go to:<\/p>\n\n\n
Alerting \u2192 Alert rules \u2192 New<\/span> alert rule\n<\/code><\/span>Code language:<\/span> PHP<\/span> (<\/span>php<\/span>)<\/span><\/small><\/pre>\n\n\n
This time use New alert rule<\/strong>, not New recording rule.<\/p>\n\n\n\n
\n\n\n\n
Alert 1: High Recorded CPU Usage<\/h2>\n\n\n\n
Name:<\/p>\n\n\n
High CPU Usage - Recorded\n<\/code><\/span><\/pre>\n\n\n
Data source:<\/p>\n\n\n
prometheus-recording\n<\/code><\/span><\/pre>\n\n\n
Query:<\/p>\n\n\n
linux_cpu_usage_active_percent{host=\"linux-demo\"<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Condition:<\/p>\n\n\n
WHEN Last OF QUERY IS ABOVE 90\n<\/code><\/span><\/pre>\n\n\n
Evaluate every:<\/p>\n\n\n
1m\n<\/code><\/span><\/pre>\n\n\n
Pending period:<\/p>\n\n\n
5m\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Alert 2: High Recorded Memory Usage<\/h2>\n\n\n\n
Query:<\/p>\n\n\n
linux_memory_used_percent{host=\"linux-demo\"<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Condition:<\/p>\n\n\n
IS ABOVE 90\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Alert 3: High Recorded Disk Usage<\/h2>\n\n\n\n
Query:<\/p>\n\n\n
linux_disk_used_percent{host=\"linux-demo\"<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Condition:<\/p>\n\n\n
IS ABOVE 90\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Alert 4: Zombie Processes Detected<\/h2>\n\n\n\n
Query:<\/p>\n\n\n
linux_processes_zombies{host=\"linux-demo\"<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Condition:<\/p>\n\n\n
IS ABOVE 0\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
Alert 5: Network Input Errors<\/h2>\n\n\n\n
Query:<\/p>\n\n\n
linux_network_input_errors{host=\"linux-demo\"<\/span>}\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Condition:<\/p>\n\n\n
IS ABOVE 0\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
14. Why Your Screenshot Shows an Error<\/h1>\n\n\n\n
Your screenshot shows this error:<\/p>\n\n\n
query must have either statistic or statistics field\n<\/code><\/span><\/pre>\n\n\n
That error is from CloudWatch<\/strong>, not Graphite.<\/p>\n\n\n\n
It happened because the query data source is currently set to:<\/p>\n\n\n
cloudwatch\n<\/code><\/span><\/pre>\n\n\n
For this lab, change it to:<\/p>\n\n\n
Graphite\n<\/code><\/span><\/pre>\n\n\n
Then use a Graphite metric such as:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.cpu<\/span>.usage_active<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Also make sure the Target data source<\/strong> at the top is:<\/p>\n\n\n
prometheus-recording\n<\/code><\/span><\/pre>\n\n\n
Not CloudWatch.<\/p>\n\n\n\n
Not Graphite.<\/p>\n\n\n\n
The target is where Grafana stores the newly recorded metric.<\/p>\n\n\n\n
\n\n\n\n
15. Troubleshooting Guide<\/h1>\n\n\n\n
Problem: Target data source dropdown is empty<\/h2>\n\n\n\n
Reason:<\/p>\n\n\n
No Prometheus-compatible data source is available as<\/span> recording-rule target.\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Fix:<\/p>\n\n\n
Add Prometheus as<\/span> a data source.\nEnable or keep enabled: Allow as<\/span> recording rules target.\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Problem: Graphite does not appear in query data source<\/h2>\n\n\n\n
Reason:<\/p>\n\n\n
Graphite data source may not be configured or may not support alerting in<\/span> this<\/span> Grafana setup.\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Fix:<\/p>\n\n\n
Connections \u2192 Data sources \u2192 Graphite \u2192 Save & test\n<\/code><\/span><\/pre>\n\n\n
Then try again.<\/p>\n\n\n\n
\n\n\n\n
Problem: Rule saves but metric does not appear<\/h2>\n\n\n\n
Wait:<\/p>\n\n\n
1 to 2 minutes\n<\/code><\/span><\/pre>\n\n\n
Then query in Explore using Prometheus:<\/p>\n\n\n
linux_cpu_usage_active_percent\n<\/code><\/span><\/pre>\n\n\n
Also check:<\/p>\n\n\n
Alerting \u2192 Alert rules\n<\/code><\/span><\/pre>\n\n\n
Make sure the rule is not paused.<\/p>\n\n\n\n
\n\n\n\n
Problem: Student used .wsp<\/code><\/h2>\n\n\n\n
Wrong:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.cpu<\/span>.usage_active<\/span>.wsp<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Correct:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.cpu<\/span>.usage_active<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
\n\n\n\n
Problem: Student used servers.*<\/code><\/h2>\n\n\n\n
Wrong:<\/p>\n\n\n
servers<\/span>.cpu<\/span>.usage<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Correct:<\/p>\n\n\n
telegraf<\/span>.linux-demo<\/span>.cpu<\/span>.usage_active<\/span>\n<\/code><\/span>Code language:<\/span> CSS<\/span> (<\/span>css<\/span>)<\/span><\/small><\/pre>\n\n\n
Reason:<\/p>\n\n\n
Your Graphite data does not contain servers.* metrics.\n<\/code><\/span><\/pre>\n\n\n
\n\n\n\n
16. Final Student Lab Summary<\/h1>\n\n\n\n
By the end of this lab, students should understand:<\/p>\n\n\n
1.<\/span> Graphite stores the original Telegraf Linux metrics.\n2.<\/span> Grafana recording rules can read Graphite metrics.\n3.<\/span> Recording rule output must be stored in<\/span> a Prometheus-compatible data source.\n4.<\/span> The recorded metric name must use Prometheus-style naming.\n5.<\/span> Dashboards and alerts can use the new<\/span> recorded metric.\n6.<\/span> Recording rules are useful for<\/span> frequently used or expensive queries.\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\n
Most important classroom rule:<\/p>\n\n\n
Source query:\nUse real Graphite metrics from<\/span> telegraf.linux-demo.*\n\nTarget output:\nStore as<\/span> Prometheus-style metrics such as<\/span> linux_cpu_usage_active_percent\n\nNever use:\nservers.*\n\nNever include:\n.wsp\n<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>","protected":false},"excerpt":{"rendered":"
Using Graphite + Telegraf Linux Metrics You are on the correct screen now: Alerting \u2192 Alert rules \u2192 New recording rule. But one important point first: Grafana… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_joinchat":[],"footnotes":""},"categories":[11138],"tags":[],"class_list":["post-75277","post","type-post","status-publish","format-standard","hentry","category-best-tools"],"_links":{"self":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/75277","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=75277"}],"version-history":[{"count":1,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/75277\/revisions"}],"predecessor-version":[{"id":75278,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/75277\/revisions\/75278"}],"wp:attachment":[{"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=75277"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=75277"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.devopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=75277"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}