It's been a while since I wrote my last docker related post and even longer since I last discussed docker on AWS and a lot has changed since then. For production deployment there are now the Amazon EC2 Container Service (ECS) as well as Google Container Engine (GCE) which can both help you easily deploy microservices based on docker with ease.

If, however, you need to comply with additional auditable security requirements the situation is a bit more murky and those services which can ease development might not be as acceptable to your auditors. I am going to outline my setup for a HIPAA compliant docker architecture on AWS. In some ways it is a bit of a throwback, but in other ways it allows you to embrace recent developments such as Docker Machine and Docker Compose. When it comes to HIPAA my main concerns are:

  1. Encryption of all data, including PHI, in motion.
  2. Encryption of all data, including PHI, at rest.
  3. Solid isolation of decrypted data from other programs running in AWS's cloud.

Fundamentally it is important to remember that on AWS you are always using a shared network, and unless you explicitly specify otherwise you are using shared servers. Therefore it is important both within and outside your VPC to treat all PHI as though it were being transferred and stored in a publicly accessible network.

Encryption in Motion

When running on AWS, I use a Virtual Private Cloud (VPC) to create a logical isolation of my internal network. I create public and private zones, and I keep PHI in the private zones. My private EC2 instances can create connections to the outside world via a NAT instance in the public zone of our network, and I can connect to my private EC2 instances using a Bastion server in my public zone. There are a lot of parameters to make a good VPC, including access control lists, security policies, and default tenancy requirements. In order to avoid any mistakes I use a CloudFormation definition of my VPC and update it as necessary. I can write an entire post about this process, however it is not the topic of this section.

The main thing to understand about VPC is that it creates a logical isolation of your resources. VPC does not encrypt your traffic as it is sent on the physical wires that connect real hosts. For this reason I encrypt traffic that is internal to my VPC as well as traffic which goes to and from the public internet. Traffic moving between docker containers on the same EC2 instance is allowed to be unencrypted.

I encrypt my network traffic using two mechanisms:

  • HTTP over TLS
  • SSH Tunneling

HTTP over TLS

I primarily make use of a microservice based architecture with RESTful HTTP APIs. It is very straightforward to add a reverse-proxy Nginx container in front of each microservice to manage the TLS layer, which is typically what I do, as shown in the architecture diagram below.

Secure Microservice

In order to facilitate TLS connections, I create a certificate authority as well as a certificate revocation list and bundle the root certificate and CRL in the operating system-level base container image on which I build my microservice containers. I then provide each Nginx container with a separate certificate signed by my certificate authority. Should one of the certificate private keys escape into the wild, I can revoke the certificate, create a new certificate to replace the old one, and rebuild my containers.

I also create client certificates, signed by my certificate authority, to make sure that only authorized clients are connecting to the microservices I have deployed. Nginx checks for the presence of a client certificate, and passes on information about the certificate, if found and verified, to the microservice itself. The microservice can choose to respond if a certificate is verified, or if a particular certificate fingerprint is present. The particular configuration lines I use in Nginx to facilitate this are:

ssl_client_certificate /etc/nginx/certs/root-ca.crt;
ssl_crl /etc/nginx/certs/ca.crl;
ssl_verify_client optional;

location / {
    proxy_pass  http://service:8080/;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Client-Verified $ssl_client_verify;
    proxy_set_header X-Client-DN $ssl_client_s_dn;
    proxy_set_header X-Client-Serial $ssl_client_serial;
    proxy_set_header X-Client-Fingerprint $ssl_client_fingerprint;
}

SSH Tunneling

Most of the microservices I create can be hosted on a single EC2 instance, however there are times when I wish to separate resources that are part of one service on to multiple EC2 instances. A common example is when I wish to use multiple database instances located on redundant EC2 instances in separate availability zones.

In these cases I use a variation on the ambassador pattern which embeds the links between EC2 instances in SSH tunnels managed by autossh. Each EC2 instance is already running an OpenSSH server, so it may seem natural to use those existing servers to manage the connections between EC2 instances, however I have chosen to roll the SSH server into the ambassador containers in order to provide an additional layer of separation between the infrastructure layer and application layer. The ambassador containers use their own SSH keys and certificates separate from those on the EC2 instances themselves. This somewhat limits the potential damage should an SSH key move beyond our control.

SSH Tunnels

I call these ambassador containers crypto-ambassadors. Their creation is coordinated by a master EC2 node in the network and a redis store on that node is used to coordinate the relationship between Docker endpoints and EC2 instance ports and addresses. From the point of view of the client and server containers on the separate EC2 instances, the links appear as links on the local docker bridge and they do not need to concern themselves with any cryptography.

Encryption at Rest

In order to encrypt the data at rest I use encryption at the application layer as well as at the infrastructure layer. The application layer encryption varies depending on the application, but the infrastructure layer encryption is handled using Amazon's Encrypted Elastic Block Storage. One important thing to note is that the root volume of an EC2 instance is never encrypted, so I always mount at least one additional EBS data volume with encryption enabled as in the figure below.

Encrypted EBS Data Volume

Since my data is in containers it is important that the docker volumes end up on the encrypted data EBS rather than the unencrypted root EBS. I facilitate this by creating the EBS in our CloudFormation script and creating a symbolic link from /var/lib/docker to /data/docker in the User-Data section of the CloudFormation script. Although conceptually simple, the devil is in the details as it so often is with CloudFormation, so I have provided a sample template.

The template will allow you to create a single EC2 instance with an unencrypted root volume, an encrypted data volume with a docker directory. It will have access to an S3 bucket you specify, which I typically use to store backups. You need to provide the VPC and subnet ids in which to place the EC2 instance, the name of the S3 bucket you wish for the instance to access, and the SSH key you wish for the instance to use. These must be pre-created. The launch link above launches the template in the us-west-2 region, but the template is written to support all the AWS regions.

I switch the SSH port from port 22 to port 2222 to avoid spurious SSH traffic if you leave the SSH port accessible from 0.0.0.0/0, however you can restrict the IP addresses allowed to access you server's SSH and docker ports by specifying a more restrictive AdminCidrIp.

To complete installation of docker you must install Docker Machine on the host you will use to manage this new EC2 server. I run Docker Machine on the management node in my network's private subnet, however when experimenting with the template above you can use your laptop. From your management system, you install docker on the new EC2 instance using the following command:

docker-machine create -d generic \
    --generic-ip-address IP_ADDRESS \
    --generic-ssh-key SSH_KEY \
    --generic-ssh-port 2222 \
    --generic-ssh-user ubuntu \
    MACHINE_NAME

where IP_ADDRESS is the IP address of the new EC2 instance, SSH_KEY is the file containing the SSH key you selected, and MACHINE_NAME is the name you choose for the new instance. Note that this takes a long time to run, and the only feedback you get is a message stating, "Importing SSH key...".

Once installation is complete, I recommend you SSH into the machine and reboot. You can connect using docker-machine with the following command.

docker-machine ssh MACHINE_NAME

When you delete the EC2 instance you must stop the docker process by SSHing in and running the command

sudo stop docker

One issue I have encountered is that the docker init script does not wait for remote filesystems to be mounted before starting. I recommend editing /etc/init/docker.conf so that the start on line reads

start on (remote-filesystems and net-device-up IFACE!=lo)

which is a bit redundant.

Isolation of Decrypted Data

At some point the data must be decrypted. This occurs within the EC2 instance containers. Docker's containerization helps to isolate the decrypted data from other process groups on the instance, however in order to be completely safe we use dedicated tenancy instances. This assures us that only our processes are running on our instance's physical processor. Occasionally there are reported exploits which suggest that there is the possibility of an attack due to leakage of data from one virtual machine to another on the same physical hardware.

Conclusion

The cloud is an amazing resource which allows us to access a tremendous amount of processing power at a minimal expense, however we must remain aware of the fact that in using the cloud we are using a shared space. It was much easier to be certain that a connection was secure when we could trace the physical wire. It was easier to see that our network was isolated when we could disconnect the firewall from the internet. In the cloud even our isolated networks are shared and we should build with that in mind.

i