Prometheus in Docker: SSH Tunnel to Target Nodes

I am currently using Prometheus to collect time-series machine metrics from servers operating in various VPC’s.  I decided to run Prometheus (and Grafana for dashboards) in Docker to allow for easy deployment using Docker Compose.  Configuring the two applications to communicate is made easy by Docker networking in bridge mode.  Since I’m using Docker Compose, I can reach each container via their container name as Docker Compose configures every container to be reachable in the Docker network. However, getting Prometheus to retrieve metrics from instances that are in a private subnet that Prometheus is not a part of can require some extra work when the Prometheus instance is running in a Docker container.  Essentially, I want to achieve the following: Use a utility like Autossh to keep a persistent SSH connection live to a jumpbox on the destination’s network and port forward to the host that we want to scrape metrics from (the host will be running an application like Node Exporter to serve machine metrics).

At this point you have a few options:

  1. Run Autossh on the Docker host and have the Prometheus Docker container communicate via the Docker host’s network interface
  2. Provision a Docker container which runs Autossh and have Prometheus reference that container

SSH Tunnel via Docker Host

Install Autossh on the Docker host:

sudo apt install autossh

Before configuring Autossh, you need to know which network interface to bind to when tunnelling. Execute ifconfig and locate the network interface configured by Docker (it will be named something like docker0). The IP address on my Docker host is 172.17.0.1.

Configure a SystemD service to run Autossh:

sudo nano /etc/systemd/system/autossh.service

The unit file should be defined as follows:

[Unit]
Description=Keeps a tunnel to a jumpbox open and tunnels to instances running Node Exporter
After=network-online.target

[Service]
User=ubuntu

ExecStart=/usr/bin/autossh -M 0 -N -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -l autossh prometheus@<public-ip-of-jumpbox> -L 172.17.0.1:19100:<private-ip-of-node-exporter>:9100 -i /home/ubuntu/prometheus.key

[Install]
WantedBy=multi-user.target

As you can see, after SSHing into the jumpbox (with the user prometheus in my case), we are binding the IP address of our Docker network interface, and forwarding this connection to the destination host and port.

After saving the unit file, you will need to reload the systemd process and enable the service:

sudo systemctl daemon-reload
sudo systemctl enable autossh.service
sudo systemctl start autossh.service

We can now configure our Prometheus configuration file to reference this endpoint:

...
static_configs:
- targets:
  - 172.17.0.1:19100
...

Restarting the Prometheus Docker container will make these changes take effect. Prometheus will now be scraping metrics from the destination instance via the Docker host, which has a persistent SSH connection to the jumpbox on the destination instance’s network, and a tunnel to the destination instance.

Run Autossh in Docker Container

Another option is to create a Docker image which uses Autossh and connect to that container from Prometheus.  Plenty of Dockerized Autossh repos exist for use.  Essentially you want your Autossh to be executing with similar parameters:

autossh -M 0 -g -N -o StrictHostKeyChecking=no -o ServerAliveInterval=5 -o ServerAliveCountMax=1 -i /root/.ssh/id_rsa prometheus@<public-ip-of-jumpbox> -L 0.0.0.0:19100:10.0.3.104:9100

Very similar to the configuration where we used the Docker host. The only main difference is we are configuring the container to listen on all network interfaces.  We will need to add this new Docker image to our Docker Compose file.

...
services:
  autossh:
    image: mkezz/autossh
...

We can now modify our Prometheus config file to include the target by its discoverable name:

...
static_configs:
- targets:
  - autossh:19100
...

After starting up our containers via Docker Compose we should start seeing metrics pour in from our tunnelled endpoint.

Advertisements

One thought on “Prometheus in Docker: SSH Tunnel to Target Nodes

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s