Docker

Install and Configure Prometheus for Collecting Docker metrics

September 26, 2021 comments off

How to install Install Docker

Configure docker for exposing a Metrices at end point

To configure the Docker daemon as a Prometheus target, you need to specify the metrics-address. The best way to do this is via the daemon.json, which is located at one of the following locations by default. If the file does not exist, create it.

Linux: /etc/docker/daemon.json
Windows Server: C:\ProgramData\docker\config\daemon.json
Docker Desktop for Mac / Docker Desktop for Windows: Click the Docker icon in the toolbar, select Preferences, then select Daemon. Click Advanced.

If the file is currently empty, paste the following:

{
  "metrics-addr" : "127.0.0.1:9323",
  "experimental" : true
}

Save the file, or in the case of Docker Desktop for Mac or Docker Desktop for Windows, save the configuration. Restart Docker.

$ sudo systemctl restart docker

Docker now exposes Prometheus-compatible metrics on port 9323.

Configure and run Prometheus

# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'codelab-monitor'

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first.rules"
  # - "second.rules"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'docker'
         # metrics_path defaults to '/metrics'
         # scheme defaults to 'http'.

    static_configs:
      - targets: ['localhost:9323']

Verify that the Docker target is listed at http://localhost:9090/targets/.

Use Prometheus for Swarm only

Create a graph. Click the Graphs link in the Prometheus UI. Choose a metric from the combo box to the right of the Execute button, and click Execute. The screenshot below shows the graph for
engine_daemon_network_actions_seconds_count.

The above graph shows a pretty idle Docker instance. Your graph might look different if you are running active workloads.

To make the graph more interesting, create some network actions by starting a service with 10 tasks that just ping Docker non-stop (you can change the ping target to anything you like):

$ docker service create –replicas 10 –name ping_service alpine ping docker.com

Wait a few minutes (the default scrape interval is 15 seconds) and reload your graph.

When you are ready, stop and remove the ping_service service, so that you are not flooding a host with pings for no reason.

docker service remove ping_service
Wait a few minutes and you should see that the graph falls back to the idle level.

How to secure Prometheus Docker Endpoint after enabling through metrics-addr in daemon.json

[Experiment – NOT Tested]

If you need to access the Docker daemon remotely, you need to enable the tcp Socket. Beware that the default setup provides un-encrypted and un-authenticated direct access to the Docker daemon – and should be secured either

  1. Using the built in HTTPS encrypted socket, or
  2. By putting a secure web proxy in front of it.

If you need to access the Docker daemon remotely, you need to enable the tcp Socket. Beware that the default setup provides un-encrypted and un-authenticated direct access to the Docker daemon – and should be secured either using the built in HTTPS encrypted socket, or by putting a secure web proxy in front of it.

Note: If you’re using an HTTPS encrypted socket, keep in mind that only TLS1.0 and greater are supported. Protocols SSLv3 and under are not supported anymore for security reasons.

Method 1 – Protect or Secure the Docker daemon socket
https://docs.docker.com/engine/security/https/
https://docs.docker.com/config/daemon/
https://docs.docker.com/engine/security/https/
https://gist.github.com/kekru/b9e4da822514df93e6fdf2f7d3d90d8a

Method 2 – secure web proxy
One option to help secure our Prometheus server is to put it behind a reverse proxy so that we can later add SSL and an Authentication layer over the default unrestricted Prometheus web interface.

Example of daemon.json

{
  "metrics-addr" : "127.0.0.1:9323",
  "experimental" : true
}

We will use Nginx.

$ sudo apt install nginx
# CD to the Nginx sites-enabled folder
cd /etc/nginx/sites-enabled
# Create a new Nginx configuration from Prometheus
$ sudo nano prometheus
And copy/paste the example below
server {
    listen       443;   

    location / {
        proxy_pass           http://localhost:9323/;
    }
}
# Save and restart Nginx
$ sudo service nginx restart
$ sudo service nginx status
Rajesh Kumar
Follow me
Latest posts by Rajesh Kumar (see all)