SRE teams commonly use Prometheus and Grafana together as part of their observability stack. Prometheus collects and stores time-series metrics, while Grafana queries those metrics and turns them into dashboards, visualizations, and alerts.
The important point is that SREs should not use these tools simply to create hundreds of dashboards. They should use them to answer questions such as:
- Is the service healthy?
- Are users experiencing errors or high latency?
- Are we meeting our SLO?
- What changed before the incident?
- When should an engineer be paged?
Prometheus for Metrics Collection
Prometheus typically collects metrics from applications, servers, Kubernetes components, exporters, and other monitored systems.
For example, an SRE might track:
- CPU and memory utilization
- Request rate
- Error rate
- Request latency
- Pod restarts
- HTTP status codes
- Database connections
- Queue depth
- Service availability
Prometheus stores these measurements as time-series data and allows engineers to query them using PromQL.
Grafana for Visualization
Grafana connects to Prometheus as a data source and uses PromQL queries to visualize the metrics.
An SRE dashboard might contain panels for:
Service Health
- Request rate
- Error percentage
- Latency
Infrastructure
Kubernetes
- Pod restarts
- Container CPU/memory
- Deployment status
- Node health
Grafana also provides Explore functionality, which is useful when engineers need to investigate a problem interactively instead of relying only on predefined dashboards.
Alerting Is More Important Than Pretty Dashboards
A common mistake is creating dashboards with too many metrics but no clear operational purpose.
SRE teams should design alerts around user-impacting symptoms rather than every possible infrastructure condition. Prometheus documentation recommends keeping alerts focused on symptoms and avoiding pages where there is nothing actionable to do.
For example, instead of paging an engineer every time CPU reaches 80%, a better alert might be based on sustained high latency or elevated error rates.
Prometheus alerting rules can evaluate PromQL expressions and send firing alerts to Alertmanager.
Using SLOs and SLIs
This is where Prometheus becomes particularly valuable for SRE.
An SRE team might define:
SLI: Percentage of successful requests
SLO: 99.9% successful requests over a defined period
Prometheus can collect the underlying request metrics, while Grafana can display the SLI, SLO status, trends, and error-budget consumption.
This gives the team a much better reliability signal than simply monitoring CPU and memory.
Reducing Alert Noise
Alert fatigue is one of the biggest problems in monitoring.
A good SRE approach is to:
- Remove duplicate alerts
- Alert on symptoms rather than every cause
- Use appropriate evaluation windows
- Add severity labels
- Include useful annotations
- Group related alerts
- Use recording rules for expensive or frequently reused queries
- Review noisy alerts after incidents
Alertmanager can deduplicate, group, route, silence, and inhibit alerts, which is especially useful during large incidents when many related alerts can fire simultaneously.
Using Metrics During Incident Response
Suppose customers report that an API has become slow.
An SRE could use Grafana to examine:
Request rate → Error rate → Latency → Application resources → Database metrics → Recent deployment
Prometheus provides the underlying time-series data, while Grafana makes it easier to compare these signals over the same time period.
This helps engineers move from “something is wrong” to “what changed and where should I investigate?”
A Practical SRE Workflow
A simple production workflow could look like:
Application / Infrastructure → Prometheus → PromQL → Grafana Dashboard
For alerting:
Prometheus Alert Rules → Alertmanager → Notification / On-call System
Grafana can also query Prometheus data directly and create alert rules based on time-series data.
Final Takeaway
Prometheus and Grafana are most effective when they are connected to an SRE methodology.
Prometheus provides the metrics and querying, Grafana provides visualization and investigation, and alerting components provide operational notification and routing.
The real SRE goal is not to monitor everything. It is to monitor the signals that matter, connect them to SLIs and SLOs, create actionable alerts, and give engineers enough information to troubleshoot incidents quickly.
A small set of well-designed dashboards and alerts is usually much more valuable than a huge monitoring setup that nobody trusts.