HIPAA Microservices and Encryption with AWS Key Management Service

In my last HIPAA related post, I discussed an architecture involving the use of Docker containers to provide a platform of microservices along with two mechanisms for encryption in motion:

  1. The use of TLS over HTTP with a private certificate authority.
  2. The use of sustained SSH connections between EC2 instances established by crypto-ambassador containers.

Both of these solutions have some drawbacks. In the case of TLS over HTTPS I use an open source nginx proxy container to terminate SSL on each EC2 instance, and then forward the requests internally to each individual service on that same EC2 instance. This generally means that when deploying a new microservice, the nginx container has to be restarted to reload the configuration files. This is not that onerous when you have a handfull of EC2 instances, but with many EC2 instances, each with their own nginx proxy container, it becomes more onerous. The other issue I have is that whenever we rotate server certificates or update the certificate revocation list, I have to deploy these changes to all the relevant EC2 instances. Bash scripting the CA steps, and using Ansible to manage deployment helps streamline this operation, but it's a delicate dance and when a deployment failed on one EC2 instance it took a while to realize that there was a problem and to debug the problem.

SSH connections work well. I use autossh to maintain and monitor SSH connections among the crypto-ambassador containers, and the connections generally persist for months between EC2 instances in the same zone. However, when one fails it can be like looking for a needle in a haystack to find which one is malfunctioning. It is important to set the environment variable AUTOSSH_DEBUG=1 so that the warnings are logged to the container logs. In the end, as the SSH web grows I find this method of communication between containers between EC2 instances to be quite a headache when something goes wrong. I think that more can be done to add monitoring to this system, but I have managed a more reliable solution.

A Third Option for Encryption in Motion on AWS

I wanted a solution which would minimize deployment complexity, but allow for encrypted communication using a standard that would work in a variety of programming languages. I tend to use python, but architect services written in several languages, so I wanted an encryption standard which had multiple libraries and a simple implementation. I also decided that I would use a symmetric algorithm with a centralized key management service. I decided to use the Fernet Specification which is implemented in python's cryptography library.

I wont go into much detail on the Fernet specification, but essentially it uses AES-128 CBC encryption using a 128 bit encryption key; signs the message and some associated metadata using a SHA256 HMAC with a 128 bit signing key; and base64url encodes the encrypted, signed output. Since the output is a base64url encoded string, it can easily be sent as part of a message passed to a microservice.

Symmetric encryption is only half the problem. The other part is key management, and this is where AWS offers a service which is so useful and inexpensive that it overcomes my aversion towards vendor lock in. That service is AWS Key Managment Service (KMS).

AWS KMS Architectue

AWS KMS is a hardened security appliance (HSA) backed service for generating cryptographically random encryption keys and for encrypting and decrypting small amounts of data (up to 4 KB). The details of AWS KMS are described in the AWS Key Management Service whitepaper which should be reviewed to be sure it complies with your HIPAA policies. Essentially AWS KMS allows the creation of customer master keys. These keys are not exported from the hardened security appliances, but can be used to encrypt and decrypt up to 4 KB of data. For our purposes we can request that AWS KMS produce a cryptographically random data key and encrypt that data key with a customer master key. The GenerateDataKey API call will conveniently generate a data key and send back both the unencrypted and encrypted versions of that key. That data key can be used to encrypt data using the Fernet specification, and then the Fernet encrypted message and the encrypted data key can both be sent over the wire to a microservice. The microservice, on receiving the encrypted data key, can use the Decrypt API call to decrypt the data key, and then use that data key to verify and decrypt the Fernet message.

KMS is not a HIPAA certified service, so you should not send PHI to KMS servers to be encrypted, however you can encrypt PHI using the keys generated by KMS provided you keep the encrypted PHI on HIPAA certified AWS services. KMS pricing is very low. It costs $1 per month per key plus an additional $1 per month if you enable automated key rotation. The first 20,000 API requests are free each month and then the rate is $0.03 per 10,000 API calls after that.

Sample Key Creation and Use

Key creation is easily accomplished using the command line. The aws kms create-key command will create a new KMS customer master key which can be used to generate data keys. Each key has an ARN, however it is also useful to create an alias for ease of use. Aliases can also be used to refer to different keys in different regions, which can makes it easier to deploy microservices in different regions.

$ aws kms create-key
{
    "KeyMetadata": {
        "KeyId": "f20c924b-e52c-444f-a3ca-1463b1751651", 
        "Description": "", 
        "Enabled": true, 
        "KeyUsage": "ENCRYPT_DECRYPT", 
        "CreationDate": 1487749152.65, 
        "Arn": "arn:aws:kms:us-west-2:4XXXXXXXXXXX:key/f20c924b-e52c-444f-a3ca-1463b1751651", 
        "AWSAccountId": "4XXXXXXXXXXX"
    }
}

$ aws kms create-alias --alias-name alias/hipaa_test --target-key-id "arn:aws:kms:us-west-2:4XXXXXXXXXXX:key/f20c924b-e52c-444f-a3ca-1463b1751651"

The following function can be used to request a data key, and a version of that key encrypted by the customer master key kms_key_id. It then uses that data key to encrypt the bytestream message using the Fernet specification. Since Fernet requires a 128 bit encryption key and a 128 bit signing key, I request 256 bits from KMS. The function returns a dictionary containing the encrypted_data_key and the encrypted_message.

import base64
import boto3
from cryptography.fernet import Fernet

def encrypt_message(kms_key_id, message):
    kms_client = boto3.client('kms')

    keystruct = kms_client.generate_data_key(KeyId=kms_key_id,
                                             KeySpec='AES_256')

    data_key = base64.urlsafe_b64encode(keystruct.get('Plaintext'))
    encrypted_data_key = base64.urlsafe_b64encode(keystruct.get('CiphertextBlob'))

    fernet_encryptor = Fernet(data_key)
    encrypted_message = fernet_encryptor.encrypt(message)

    return {'encrypted_data_key': encrypted_data_key,
            'encrypted_message': encrypted_message}

The function below decrypts the encrypted message given the encrypted_data_key and encrypted_message returned by the function above. 

import base64
import boto3
from cryptography.fernet import Fernet

def decrypt_message(encrypted_data_key, encrypted_message):
    kms_client = boto3.client('kms')

    encrypted_data_key_decoded = base64.urlsafe_b64decode(encrypted_data_key)

    keystruct = kms_client.decrypt(CiphertextBlob=encrypted_data_key_decoded)
    data_key = base64.urlsafe_b64encode(keystruct.get('Plaintext'))

    fernet_decryptor = Fernet(data_key)
    plaintext = fernet_decryptor.decrypt(encrypted_message)

    return plaintext

Below is an example of a session in ipython which shows use of the functions above to use the alias/hipaa_test customer master key to generate a data key to encrypt the bytestream Hello World!. The resulting encrypted data key and encrypted message are then decrypted with the above decryption function.

In [13]: encrypted_items = encrypt_message('alias/hipaa_test', 'Hello World!')

In [14]: encrypted_items
Out[14]: 
{'encrypted_data_key': 'AQEDAHiZVz_oviQnu3c_B12Hnr4J7T0m-ESVJjW2vN2vQTmh4wAAAH4wfAYJKoZIhvcNAQcGoG8wbQIBADBoBgkqhkiG9w0BBwEwHgYJYIZIAWUDBAEuMBEEDJzbVXJ0G4YqmMNbEwIBEIA7hhtUTFKL80ZPshA5ODak5uTcy-dDLZUkem2JFu7eMOacRcjVTkJ1cE4ec7-foKlu2H46bh0Nx-SzBck=',
 'encrypted_message': 'gAAAAABYrQqpREhmWPtR3w5ptbp44K1R_csyIpbhkYTXfCdO8s8334ar945-8aOwrVGvrfZeobCth_6d8pevkEGoIM3e-lu6Tg=='}

In [15]: decrypt_message(**encrypted_items)
Out[15]: 'Hello World!'

KMS Access Control

Access to KMS services can be granted to EC2 instances using IAM roles. In particular the kms:Decrypt and kms:GenerateDataKey* actions need to be enabled for each key that the EC2 instance requires access to. As an example, to use the above key to generate keys and decrypt, one would attach the following policy to the EC2 instance's IAM role during creation.

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "kms:GenerateDataKey*",
      "kms:Decrypt"
    ],
    "Resource": [
      "arn:aws:kms:us-west-2:4XXXXXXXXXXX:key/f20c924b-e52c-444f-a3ca-1463b1751651"
    ]
  }
}

AWS Key Management Service is a great resource for generating and managing encryption keys. By allowing EC2 instances access to the appropriate KMS keys it is possible to provide a simple avenue for encrypted communication among microservices in a HIPAA compliant system on AWS.