Configure JDBC SSL Mode with RDS

In order to enable SSL mode on a JDBC connection with RDS one must update the keystore the application is using to use the relevant root certificate from AWS. If you’re following this AWS article, be careful. The instructions in the article explain updating the javax.net.ssl.trustStore property. This will work, however, if your application uses SSL for other services, they will break. For example, if you are using ElastiCache (Redis) for session management in a Spring Boot app you will receive SSL handshake errors if your Redis instance/cluster requires SSL. The reason for this is overriding the javax.net.ssl.trustStore will no longer reference the default JRE keystore.

Instead of managing a separate keystore, simply update the default JRE keystore with the root certificate from AWS:

  1. Download the cert from: https://s3.amazonaws.com/rds-downloads/rds-ca-2019-root.pem
  2. Import the cert into the default cacerts keystore: keytool -import -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -noprompt -alias rds-ca-2019-root -file rds-ca-2019-root.pem

High Availability Prometheus and Alertmanager

I wanted to demonstrate how to configure Prometheus with Alertmanager (using AWS SES as a notification channel) in a high availability architecture.  Both applications are deployed as Docker containers using Docker Compose for provisioning.  Ideally in a production environment you would use a production grade container orchestration platform like Kubernetes to manage the containers.  I have the Vagrant file and associated resources available in a GitHub repo found here.

The architecture that I wanted to mimic is as follows:

It is important to remember some key characteristics about Prometheus.  Since it is a pull based monitoring platform, your exporters do not need to be aware of the Prometheus instances or care how many of them are deployed.  As long as the metrics are being served and the Prometheus instance can access the node you can provision as many Prometheus nodes as needed.  Since each Prometheus instance maintains its own persistence layer it does not need to be made aware of other Prometheus instances in the cluster.  If changes need to be applied to any of the configuration options in Prometheus, a configuration management platform like Ansible would be ideal for these kinds of tasks.

Where the Prometheus instances need to be aware of one another, the Alertmanager is not exempt from this requirement.  In order to properly configure the cluster of Alertmanager nodes, the service must be configured with a list of peers that are in the cluster.  This cluster mode prevents Alertmanager from sending duplicate notifications.

When Alertmanager receives a triggered alert from one or more Prometheus instances, by design it has been developed to handle these duplicates and not erroneously report multiple active alerts with the same criteria.  This is an effective design that allows any number of Prometheus instances to be reporting to the same Alertmanager.

This setup is very easy to configure and very important if using Prometheus and Alertmanager in a production environment.  Here is my GitHub repo again demonstrating how to provision the nodes and services: https://github.com/mkez00/prom-alertmanager-ha

Migrating a SQL Server DB on RDS from AWS Commercial to AWS GovCloud

Recently I was tasked with migrating a relational database from an AWS Commercial account to an AWS GovCloud account. The task seemed trivial with the variety of services AWS has to offer. The caveat being that some features (for example Cross-Account Snapshot Sharing) do not work with GovCloud accounts. AWS DMS is another option but unfortunately I have never had any luck using that service. Regardless, there is still hope to complete this task. The instructions/checklist below will make this migration a possibility:

    1. The destination SQL Server instance MUST match major versions on the source instance. For example: SQL Server Standard Edition 11.00.2100.60.v1 on the source will be able to restore to a version like SQL Server Standard Edition 11.00.5058.0.v1
    2. Create an S3 bucket in both accounts (AWS GovCloud and AWS Commercial) that RDS will use to dump the backup file to and eventually read from. My example uses the following buckets:
      1. Bucket name: my-rds-backup on the AWS Commercial account
      2. Bucket name: my-rds-restore on the AWS GovCloud account
    3. An option group must be applied to the source and destination instances which will allow for backup and restore. If this option is not applied to the instances you will not be able to execute the required stored procedures for backup and restore. Follow the steps below to create the option group and apply it to your RDS instances (one for your source instance on the commercial cloud, one for your destination instance on the GovCloud). Starting on your commercial cloud account:
      1. Go to RDS in the AWS web console
      2. Select Option Group from the side menu
      3. Select Create Group located at the top of the table view
      4. Give the group a descriptive name and description. Select the Engine and Major Engine Version that matches your instance
      5. Click Create
      6. Select your newly created Option Group and and click Add option
      7. The default option selected is SQLSERVER_BACKUP_RESTORE. This is the option we need
      8. Select the S3 bucket where the backup will be dumped to (if doing these steps for restore, it will be the S3 bucket that will be for restoring the DB)
      9. Change Apply Immediately to Yes
      10. Click Add option
      11. Modify the RDS instance and apply the new Option Group
      12. Repeat steps above for restore (to be done on AWS GovCloud account)
    4. Provision an EC2 instance with Windows Server that also has SQL Server Management Studio installed
    5. RDP into the Windows Server instance and connect to your source database from SQL Server Management Studio
    6. Run the following stored procedure on the source database:
      exec msdb.dbo.rds_backup_database
      @source_db_name='ProductionDatabase',
      @s3_arn_to_backup_to='arn:aws:s3:::my-rds-backup/prod.bak',
      @overwrite_S3_backup_file=1,
      @type='FULL';

      Execute the following stored procedure to check the status:

      exec msdb.dbo.rds_task_status @db_name='ProductionDatabase'
    1. Upon completion the .bak file will be available in S3 in the commercial cloud bucket configured for backing up your database. Login to S3 (this step and the next should be done with a script) and download the bak file
    2. Login to the AWS GovCloud console and upload the bak file to the S3 bucket created for restoring the DB
    3. Connect to your destination instance using SQL Server Management Studio
    4. Execute the following stored procedure:
      exec msdb.dbo.rds_restore_database
      @restore_db_name='ProductionDatabase',
      @s3_arn_to_restore_from='arn:aws:s3:::my-rds-restore/prod.bak';

      Execute the following to check the status:

      exec msdb.dbo.rds_task_status @db_name='ProductionDatabase';

 

Upon completion your database will be available on your AWS GovCloud RDS instance.

Configuring a Simple Kubernetes Cluster on Ubuntu

There are a lot of exciting trends in tech right now.  One of the most popular trends and one that I have been keen on is container orchestration.  Kubernetes is the beast in this space.  After spending a good amount of time with Docker the past year I wanted my focus to include Kubernetes and really dedicate some time getting a stronger understanding of this powerful tool.  Minikube is a handy project that will help you learn the basic concepts of deployments, pods, services, etc.  But if you are like me, this isn’t enough.  You want some insight on how to scale out a multi-node cluster on the Linux distribution of your choice.  I couldn’t find a solid tutorial that worked well when using VirtualBox as a hypervisor (could be that they are out of date) and used Ubuntu so here is my contribution.

Here is a link to the GitHub repository which has the Vagrant file and all required configs for this setup.

I wanted to elaborate on some of the key configuration changes that were needed in order to make this setup operational.  Kubeadm does a great job of automating the majority of the setup for you, however, you will need to be more involved with the setup depending on your networking setup.

1) Hostname

As mentioned here, you will need to ensure that hostname -i returns a routable IP address (i.e. for Vagrant configurations it would be the IP on the second network interface, not the first one). By default, it doesn’t do this and kubelet ends-up using the first non-loopback network interface, which is usually NATed.

2) CNI Implementation

Choosing and configuring a CNI implementation can be a hurdle when setting up your cluster.  Most users new to Kubernetes just want something that works initially and then they can investigate what flavour of plugin suits their needs.  I chose to use Flannel as my choice of CNI implementation.  In addition to the recommended steps to configure referenced on Kubernetes’ tutorial page (here), there are some other changes required for this setup.  With Flannel, it is important to remember that if your nodes have multiple network interfaces (like the Vagrant definition linked in my project) you need to specify an interface that can communicate with the other nodes in your cluster (reference).  Simply using the default Flannel configuration will not work (at least with the setup defined in my Vagrant file).  For my setup, you need to override a configuration parameter set in the Flannel config file (see here).

Testing and Debugging

An important tip to remember when troubleshooting network issues is that pods will always have visibility to one another.  If you are trying to debug other specific issues remember that if pod communication is not verified, you should start with that.  I included a set of steps in the Read Me of my project to test this.

Conclusion

This post was designed to summarize what required changes I needed to get my setup to work.  My GitHub repositories Read Me has the steps laid out and the Shell provisioners in the Vagrant file contain comments on the changes.  I wanted to use this post as an additional source of documentation that elaborates a little more on certain pain points.  I hope it helps.

Link to GitHub Repository

Prepping DigiCert Issued Certificate with OpenSSL for IIS

Recently I had to update my IIS server with a renewed SSL/TLS certificate signed by DigiCert. Converting the P7B file to the required PFX file gave me some headaches since DigiCert exports the P7B file with Windows line endings and I was doing the P7B file to PFX file conversion from my Mac. Since this is a task that is required of me to do every year or so, I wanted to document it for future me when this time of year rolls around again.

Step 1. Convert the P7B to CER

openssl pkcs7 -print_certs -in certificatename.p7b -out certificatename.cer

***NOTE:  If you get the following error:

unable to load PKCS7 object
140735856653192:error:0906D066:PEM routines:PEM_read_bio:bad end line:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22/libressl/crypto/pem/pem_lib.c:814:

Check the file if it has incompatible line endings:

file certificatename.p7b

If the output resembles the following:

certificatename.p7b: ASCII text, with CRLF, CR line terminators

The CRLF and CR line terminators will give you issues when using OpenSSL to convert the file. Use a utility like dos2unix to convert to a format compatible for OpenSSL. To install using Homebrew:

brew install dos2unix

After installing, convert the file using the following command:

dos2unix certificatename.p7b

Check the file again:

file certificatename.p7b

If the output resembles the following:

certificatename.p7b: ASCII text, with CRLF line terminators

The CRLF line terminators will still throw errors when trying to use with OpenSSL. We will need to convert the file again using dos2unix:

dos2unix certificatename.p7b

After checking the file again:

file certificatename.p7b

The output should resemble the following:

certificatename.p7b: ASCII text

At this point, try Step 1 again. If you still get an error like the following:

unable to load PKCS7 object
140735856653192:error:0906D066:PEM routines:PEM_read_bio:bad end line:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-22/libressl/crypto/pem/pem_lib.c:814:

Manually inspect the file in a text editor. If you notice the “—–END” indicator is not prefixed with a line break and looks similar to this:

lZVt3jEA-----END PKCS #7 SIGNED DATA-----

Manually add a line break before the leading hyphens:

lZVt3jEA
-----END PKCS #7 SIGNED DATA-----

Try Step 1 again.

Step 2. Convert CER and Private Key to PFX

openssl pkcs12 -export -in certificatename.cer -inkey privateKey.key -out certificatename.pfx

Create a password for the PKCS#12 (PFX) archive when prompted. You can now take the PFX archive and import it into IIS Server Certificates.

Prometheus Command-Line Flags in Docker Service

Recently I’ve come across an issue that took me a bit of time to figure out. The issue was deploying a Prometheus container from a Docker service and overriding the command-line flags on service create. The TLDR; of the issue and correct command is:

docker service create --name prometheus --publish 9090:9090/tcp prom/prometheus --storage.tsdb.retention 30d --config.file /etc/prometheus/prometheus.yml --storage.tsdb.path /prometheus

What I was trying to accomplish was simply extending how long Prometheus retains the time-series database (the default being 15 days). The problem was when setting the command line flag for the retention period it was changing the default location of the config file. To demonstrate the steps required to reproduce the issue, start by getting the latest Docker image of Prometheus:

docker pull prom/prometheus

Create a Docker service from the image leaving everything defaulted:

docker service create --name prometheus --publish 9090:9090/tcp prom/prometheus

At this point the service and Docker container are running OK.  Below is the list of command line flags for this Prometheus instance (located at /flags):

Command Line Flag Default Command Line Arguments
alertmanager.notification-queue-capacity 10000
alertmanager.timeout 10s
completion-bash FALSE
completion-script-bash FALSE
completion-script-zsh FALSE
config.file /etc/prometheus/prometheus.yml
help FALSE
help-long FALSE
help-man FALSE
log.level info
query.lookback-delta 5m
query.max-concurrency 20
query.timeout 2m
storage.tsdb.max-block-duration 36h
storage.tsdb.min-block-duration 2h
storage.tsdb.no-lockfile FALSE
storage.tsdb.path /prometheus
storage.tsdb.retention 15d
version FALSE
web.console.libraries /usr/share/prometheus/console_libraries
web.console.templates /usr/share/prometheus/consoles
web.enable-admin-api FALSE
web.enable-lifecycle FALSE
web.external-url
web.listen-address 0.0.0.0:9090
web.max-connections 512
web.read-timeout 5m
web.route-prefix /
web.user-assets

If we want to override one of these flags, first we’ll need to stop and remove the service.  After doing so, create the service again and override a parameter (in my case I wanted to extend the retention of the time series database):

docker service create --name prometheus --publish 9090:9090/tcp prom/prometheus --storage.tsdb.retention 30d

Notice that the service is no longer running correctly:

docker service ls

Output:

ID NAME MODE REPLICAS IMAGE PORTS
no0x27gcmnw0 prometheus replicated 0/1 prom/prometheus:latest *:9090->9090/tcp

Analyze the logs to get a better idea of what happened

docker service logs no0x27gcmnw0

Output:

prometheus.1.vb2wbvg0nkwd@simple | level=error ts=2017-11-13T14:14:39.819967148Z caller=main.go:356 msg="Error loading config" err="couldn't load configuration (--config.file=prometheus.yml): open prometheus.yml: no such file or directory"
prometheus.1.os2rqn85w44z@simple | level=error ts=2017-11-13T14:14:34.060338685Z caller=main.go:356 msg="Error loading config" err="couldn't load configuration (--config.file=prometheus.yml): open prometheus.yml: no such file or directory"
prometheus.1.rf8wnrk9kvxs@simple | level=error ts=2017-11-13T14:14:45.676759398Z caller=main.go:356 msg="Error loading config" err="couldn't load configuration (--config.file=prometheus.yml): open prometheus.yml: no such file or directory"

Prometheus could not start because the default config location is now being set to prometheus.yml.  If you go to the original table of flags, the default was  /etc/prometheus/prometheus.yml.  To fix this, you’ll need to explicitly set the config file location when creating the Docker service (delete the service before running the command below):

docker service create --name prometheus --publish 9090:9090/tcp prom/prometheus --storage.tsdb.retention 30d --config.file /etc/prometheus/prometheus.yml

Prometheus now starts correctly.  However, it is worth noting that some of the default command line arguments are different.  Below is a comparison of the defaults and the flags after overriding the config.file and storage.tsdb.retention flags:

Command Line Flag After Setting Command Line Arguments Default Command Line Arguments
alertmanager.notification-queue-capacity 10000 10000
alertmanager.timeout 10s 10s
completion-bash FALSE FALSE
completion-script-bash FALSE FALSE
completion-script-zsh FALSE FALSE
config.file /etc/prometheus/prometheus.yml /etc/prometheus/prometheus.yml
help FALSE FALSE
help-long FALSE FALSE
help-man FALSE FALSE
log.level info info
query.lookback-delta 5m 5m
query.max-concurrency 20 20
query.timeout 2m 2m
storage.tsdb.max-block-duration 3d 36h
storage.tsdb.min-block-duration 2h 2h
storage.tsdb.no-lockfile FALSE FALSE
storage.tsdb.path data/ /prometheus
storage.tsdb.retention 30d 15d
version FALSE FALSE
web.console.libraries console_libraries /usr/share/prometheus/console_libraries
web.console.templates consoles /usr/share/prometheus/consoles
web.enable-admin-api FALSE FALSE
web.enable-lifecycle FALSE FALSE
web.external-url
web.listen-address 0.0.0.0:9090 0.0.0.0:9090
web.max-connections 512 512
web.read-timeout 5m 5m
web.route-prefix / /
web.user-assets

At this point you can manually override the parameters when creating the Docker service that have discrepancies if you want to maintain the defaults.

Spring Boot: Define Primary Bean at Runtime

When building an application, it is not out of the ordinary to offer multiple implementations of a specific utility or feature and have the primary implementation be defined when the application is deployed.

For example, if we are building an application that uses SMS messaging we may want to offer multiple implementations for different providers: Twilio and Sinch for example.  If we need to define which implementation of an SMS service to specify at runtime this can be easily done by an IoC framework like the Spring Framework.

I have created a sample project which demonstrates this functionality with an implementation of a service (PillService) which has two distinct implementations (BluePillServiceImpl and RedPillServiceImpl).  The magic for defining which implementation to use is accomplished by overriding the BeanDefinitionRegistryPostProcessor bean:

@Component
public class BeanDefinitionRegistryPostProcessorImpl implements BeanDefinitionRegistryPostProcessor, EnvironmentAware {

    private static final String PILL_SERVICE_ENVIRONMENT_VARIABLE = "pillService";

    Environment environment;

    @Override
    public void postProcessBeanDefinitionRegistry(BeanDefinitionRegistry beanDefinitionRegistry) throws BeansException {

    }

    @Override
    public void postProcessBeanFactory(ConfigurableListableBeanFactory configurableListableBeanFactory) throws BeansException {
        String implementation = environment.getProperty(PILL_SERVICE_ENVIRONMENT_VARIABLE);
        configurableListableBeanFactory.getBeanDefinition(implementation).setPrimary(true);
    }

    @Override
    public void setEnvironment(Environment environment) {
        this.environment = environment;
    }
}

Overriding the postProcessBeanFactory and setting the specific implementation as the primary implementation of the interface will allow Spring to correctly wire the correct bean to the interface.

Implementing the EnvironmentAware interface allows us to inject the specific implementation via an environment variable.

See my GitHub repository for the source code: GitHub Repository

Prometheus in Docker: SSH Tunnel to Target Nodes

I am currently using Prometheus to collect time-series machine metrics from servers operating in various VPC’s.  I decided to run Prometheus (and Grafana for dashboards) in Docker to allow for easy deployment using Docker Compose.  Configuring the two applications to communicate is made easy by Docker networking in bridge mode.  Since I’m using Docker Compose, I can reach each container via their container name as Docker Compose configures every container to be reachable in the Docker network. However, getting Prometheus to retrieve metrics from instances that are in a private subnet that Prometheus is not a part of can require some extra work when the Prometheus instance is running in a Docker container.  Essentially, I want to achieve the following: Use a utility like Autossh to keep a persistent SSH connection live to a jumpbox on the destination’s network and port forward to the host that we want to scrape metrics from (the host will be running an application like Node Exporter to serve machine metrics).

At this point you have a few options:

  1. Run Autossh on the Docker host and have the Prometheus Docker container communicate via the Docker host’s network interface
  2. Provision a Docker container which runs Autossh and have Prometheus reference that container

SSH Tunnel via Docker Host

Install Autossh on the Docker host:

sudo apt install autossh

Before configuring Autossh, you need to know which network interface to bind to when tunnelling. Execute ifconfig and locate the network interface configured by Docker (it will be named something like docker0). The IP address on my Docker host is 172.17.0.1.

Configure a SystemD service to run Autossh:

sudo nano /etc/systemd/system/autossh.service

The unit file should be defined as follows:

[Unit]
Description=Keeps a tunnel to a jumpbox open and tunnels to instances running Node Exporter
After=network-online.target

[Service]
User=ubuntu

ExecStart=/usr/bin/autossh -M 0 -N -q -o "ServerAliveInterval 60" -o "ServerAliveCountMax 3" -l autossh prometheus@<public-ip-of-jumpbox> -L 172.17.0.1:19100:<private-ip-of-node-exporter>:9100 -i /home/ubuntu/prometheus.key

[Install]
WantedBy=multi-user.target

As you can see, after SSHing into the jumpbox (with the user prometheus in my case), we are binding the IP address of our Docker network interface, and forwarding this connection to the destination host and port.

After saving the unit file, you will need to reload the systemd process and enable the service:

sudo systemctl daemon-reload
sudo systemctl enable autossh.service
sudo systemctl start autossh.service

We can now configure our Prometheus configuration file to reference this endpoint:

...
static_configs:
- targets:
  - 172.17.0.1:19100
...

Restarting the Prometheus Docker container will make these changes take effect. Prometheus will now be scraping metrics from the destination instance via the Docker host, which has a persistent SSH connection to the jumpbox on the destination instance’s network, and a tunnel to the destination instance.

Run Autossh in Docker Container

Another option is to create a Docker image which uses Autossh and connect to that container from Prometheus.  Plenty of Dockerized Autossh repos exist for use.  Essentially you want your Autossh to be executing with similar parameters:

autossh -M 0 -g -N -o StrictHostKeyChecking=no -o ServerAliveInterval=5 -o ServerAliveCountMax=1 -i /root/.ssh/id_rsa prometheus@<public-ip-of-jumpbox> -L 0.0.0.0:19100:10.0.3.104:9100

Very similar to the configuration where we used the Docker host. The only main difference is we are configuring the container to listen on all network interfaces.  We will need to add this new Docker image to our Docker Compose file.

...
services:
  autossh:
    image: mkezz/autossh
...

We can now modify our Prometheus config file to include the target by its discoverable name:

...
static_configs:
- targets:
  - autossh:19100
...

After starting up our containers via Docker Compose we should start seeing metrics pour in from our tunnelled endpoint.

Updating AWS WAF ACL with Spamhaus Blacklist

Using AWS Application Load Balancing is easy to use, and quick to setup.  However, if you require any routing rules applied to the load balancer your options are limited.  AWS remedied this by introducing the Web Application Firewall service (or WAF) late last year (2016).  Although useable, there are some caveats when attempting to use this service to maintain a blacklist of unwanted IP addresses:

  1. Only supports /8, /16, /24, and /32 IP address ranges
  2. Each Rule limit is 10,000 (better than the previous 1000)
  3. Limited to very few AWS regions

Using a block list by a provider like Spamhaus with AWS WAF is difficult because Spamhaus uses the full spectrum of address ranges to represent all identified IP addresses.   There is also no import utility from the AWS Console to use to easily update a list of IP addresses.  To bridge this gap, I decided to create a Python script which uses the AWS CLI to translate the block list provided by Spamhaus to valid AWS IP sets using the handy netaddr library.  The script is setup to be idempotent so it can be run as a scheduled task in a CI server without requiring additional tasks to tear down the IP Set.

Here is the link to the GitHub repository

Setup Ubuntu 16.04 on VirtualBox with Host-only Adapter

This post is really meant to be a point of reference for me when setting up networking on a fresh Ubuntu 16.04 install in VirtualBox that needs Internet access and access from the host.

Network Adapters

When configuring your VM in VirtualBox, two (2) network adapters will be needed:

Adapter 1 – NAT

The primary adapter should be set to use NAT mode.  This will give the VM access to the Internet.

Adapter 2 – Host-only Adapter

This mode allows for connectivity among the host and the virtual machines.  I will not go through the setup of this adapter from VirtualBox.

Interface Configuration

When the adapters above have been configured they will not automatically be assigned on boot.  The additional interface will need to be manually configured.  To do so, boot the VM and configure the machines network interface:

sudo nano /etc/network/interface

Add the following network interface to the end of the file:

auto enp0s8
iface enp0s8 inet dhcp

This will assign a dynamic IP address to this interface.  Probably not ideal if you plan on accessing the server via SSH.  Change the dhcp attribute to static to set a static IP address. In addition to this attribute additional attributes need to be set. The attribute values for the interface will differ from the example below and will depend on your Host-only Adapter configuration:

auto enp0s8
iface enp0s8 inet static
address 192.168.56.101
netmask 255.255.255.0
network 192.168.56.0
broadcast 192.168.56.255

At this point the network service will need to be restarted:

sudo service networking restart

Executing an ifconfig at this point should validate that both interfaces are available.