Introduction of Prometheus Operator

$ kubectl api-resources | grep monitoring
alertmanagerconfigs                                  monitoring.coreos.com/v1alpha1                 true         AlertmanagerConfig
alertmanagers                                        monitoring.coreos.com/v1                       true         Alertmanager
podmonitors                                          monitoring.coreos.com/v1                       true         PodMonitor
probes                                               monitoring.coreos.com/v1                       true         Probe
prometheuses                                         monitoring.coreos.com/v1                       true         Prometheus
prometheusrules                                      monitoring.coreos.com/v1                       true         PrometheusRule
servicemonitors                                      monitoring.coreos.com/v1                       true         ServiceMonitor
thanosrulers                                         monitoring.coreos.com/v1                       true         ThanosRuler
Code language: JavaScript (javascript)

The list is of Kubernetes Custom Resource Definitions (CRDs) related to the Prometheus Operator provided by CoreOS. The Prometheus Operator simplifies the deployment and operation of Prometheus, Alertmanager, and related monitoring components in Kubernetes environments.

Here’s a brief overview of each CRD:

1. AlertmanagerConfig (v1alpha1): This CRD allows for defining alert routing and receivers for Alertmanager in a more modular fashion.
2. Alertmanager (v1): Represents an Alertmanager cluster. You can define the version, replicas, and other specifications for running Alertmanager in a cluster.
3. PodMonitor (v1): Allows you to define how Prometheus should monitor specific pods based on labels and annotations.
4. Probe (v1): Defines how Prometheus should perform “blackbox” probes using the Blackbox Exporter.
5. Prometheus (v1): Represents a Prometheus instance in your cluster. You can define specifications like the version of Prometheus, storage configurations, etc.
6. PrometheusRule (v1): Represents a set of Prometheus alerting and/or recording rules.
7. ServiceMonitor (v1): A higher-level way to define how Prometheus instances should monitor Kubernetes services. Rather than writing raw Prometheus scrape configs, you can use a ServiceMonitor to automatically generate them based on service labels and annotations.
8. ThanosRuler (v1): Represents a Thanos Ruler instance, which can evaluate recording and alerting rules against Thanos Query data, especially useful in multi-cluster Thanos deployments.

These CRDs allow Kubernetes administrators and users to define monitoring configurations natively in Kubernetes manifests. Once these resources are applied, the Prometheus Operator ensures that the actual services (Prometheus, Alertmanager, etc.) are correctly configured and running as expected.

What is ServiceMonitor?

ServiceMonitor is a Custom Resource Definition (CRD) introduced by the Prometheus Operator, which simplifies the deployment and configuration of Prometheus on Kubernetes. It’s designed to help in automatically discovering and scraping metrics from Kubernetes services without having to manually write and update Prometheus scrape configurations.

You do not need Prometheus to scrape your service by modifying prometheus.yaml file.

ServiceMonitor  is A higher-level way to define how Prometheus instances should monitor Kubernetes services. Rather than writing raw Prometheus scrape configs, you can use a ServiceMonitor to automatically generate them based on service labels and annotations.

How ServiceMonitor works:

1. Service Discovery:
   • ServiceMonitor works by selecting Kubernetes services based on label selectors. This allows Prometheus to automatically discover the services and start scraping metrics without explicit configurations.
2. Configuration:
   • Once a Kubernetes service is selected, ServiceMonitor provides details on how to scrape metrics from the service’s pods. This includes the port, path (/metrics by default), interval, and other scrape configurations.
3. Integration with Prometheus:
   • When you create a Prometheus custom resource, you specify which ServiceMonitors it should look at, usually by specifying label selectors. This Prometheus instance will then automatically generate appropriate scrape configs based on the ServiceMonitors it’s observing.

ServiceMonitor works in the following way:

1. When a ServiceMonitor is created, the Prometheus Operator adds the ServiceMonitor configuration to the Prometheus scrape configuration.
2. Prometheus starts scraping the service endpoints specified in the ServiceMonitor configuration.
3. If the Prometheus Operator detects that the service has changed, it updates the Prometheus scrape configuration accordingly.
4. Prometheus continues to scrape the service endpoints specified in the Prometheus scrape configuration.

Rajesh Kumar

I’m Rajesh Kumar, a DevOps, SRE, DevSecOps, Cloud, and Platform Engineering expert passionate about sharing practical knowledge, real-world experiences, and industry best practices. I have worked at Cotocus and regularly write about technology, travel, investing, health, product reviews, and digital marketing through my various platforms.

I publish technical articles at DevOps School, travel stories at Holiday Landmark, stock market insights at Stocks Mantra, health and fitness guidance at My Medic Plus, product reviews at TrueReviewNow, and SEO and digital marketing strategies at Wizbrand.