I am currently using Prometheus to collect time-series machine metrics from servers operating in various VPC’s. I decided to run Prometheus (and Grafana for dashboards) in Docker to allow for easy deployment using Docker Compose. Configuring the two applications to communicate is made easy by Docker networking in bridge mode. Since I’m using Docker Compose, I can reach each container via their container name as Docker Compose configures every container to be reachable in the Docker network. However, getting Prometheus to retrieve metrics from instances that are in a private subnet that Prometheus is not a part of can require some extra work when the Prometheus instance is running in a Docker container. Essentially, I want to achieve the following: Use a utility like Autossh to keep a persistent SSH connection live to a jumpbox on the destination’s network and port forward to the host that we want to scrape metrics from (the host will be running an application like Node Exporter to serve machine metrics).
At this point you have a few options:
- Run Autossh on the Docker host and have the Prometheus Docker container communicate via the Docker host’s network interface
- Provision a Docker container which runs Autossh and have Prometheus reference that container
SSH Tunnel via Docker Host
Install Autossh on the Docker host:
sudo apt install autossh
Before configuring Autossh, you need to know which network interface to bind to when tunnelling. Execute ifconfig and locate the network interface configured by Docker (it will be named something like docker0). The IP address on my Docker host is 172.17.0.1.
Configure a SystemD service to run Autossh:
sudo nano /etc/systemd/system/autossh.service
The unit file should be defined as follows:
[Unit]
Description=Keeps a tunnel to a jumpbox open and tunnels to instances running Node Exporter
After=network-online.target
[Service]
User=ubuntu
ExecStart=/usr/bin/autossh -M 0 -N -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -l autossh prometheus@<public-ip-of-jumpbox> -L 172.17.0.1:19100:<private-ip-of-node-exporter>:9100 -i /home/ubuntu/prometheus.key
[Install]
WantedBy=multi-user.target
As you can see, after SSHing into the jumpbox (with the user prometheus in my case), we are binding the IP address of our Docker network interface, and forwarding this connection to the destination host and port.
After saving the unit file, you will need to reload the systemd process and enable the service:
sudo systemctl daemon-reload
sudo systemctl enable autossh.service
sudo systemctl start autossh.service
We can now configure our Prometheus configuration file to reference this endpoint:
...
static_configs:
- targets:
- 172.17.0.1:19100
...
Restarting the Prometheus Docker container will make these changes take effect. Prometheus will now be scraping metrics from the destination instance via the Docker host, which has a persistent SSH connection to the jumpbox on the destination instance’s network, and a tunnel to the destination instance.
Run Autossh in Docker Container
Another option is to create a Docker image which uses Autossh and connect to that container from Prometheus. Plenty of Dockerized Autossh repos exist for use. Essentially you want your Autossh to be executing with similar parameters:
autossh -M 0 -g -N -o StrictHostKeyChecking=no -o ServerAliveInterval=5 -o ServerAliveCountMax=1 -i /root/.ssh/id_rsa prometheus@<public-ip-of-jumpbox> -L 0.0.0.0:19100:10.0.3.104:9100
Very similar to the configuration where we used the Docker host. The only main difference is we are configuring the container to listen on all network interfaces. We will need to add this new Docker image to our Docker Compose file.
...
services:
autossh:
image: mkezz/autossh
...
We can now modify our Prometheus config file to include the target by its discoverable name:
...
static_configs:
- targets:
- autossh:19100
...
After starting up our containers via Docker Compose we should start seeing metrics pour in from our tunnelled endpoint.