Inside the Enclave, Part 2

Footprint’s mission is to bring trust back to the internet. To do that, we’d like to first earn your trust by sharing how we think about security and systems design — and specifically how we built the core systems at Footprint that power our vaulting infrastructure to securely store and process highly sensitive data. If you haven’t already, first take a look at Inside the Enclave: Part 1 where we survey the most common ways companies secure data today and introduce a new cutting-edge model — confidential computing — that Footprint uses as a core building block of our vaulting infrastructure. In this post, we’ll jump into the nitty gritty details of how Nitro Enclaves work, what guarantees they provide, and finally how Footprint uses Nitro Enclaves in production.

What’s in an enclave?

Simply put: an enclave is a “trusted” isolated execution environment. Isolation is a key technique when it comes to security because it lets us isolate processes and system components that touch sensitive objects like PII or cryptographic keys. Trust is equally important — we need to be able to ensure that what’s running inside this isolated environment is not only our signed code, but that it’s also running in a non-tampered environment.

Isolation

There are many threat vectors to consider when considering isolation as a security control for sensitive code.

• Memory Isolation. It’s critical that your isolated process’s memory cannot be read or written by external actors. Trivially, an adversary with memory read access could extract keys in memory or other sensitive data as it’s being processed.
• CPU. It’s important to dedicate separate CPUs for the execution environments to defend against attacks that try to overwhelm the entire system’s CPU capacity and thereby could prevent correct functionality of the isolated environment and cause unintended side effects. A more complex attack could use classes of timing attacks to aid cryptographic attacks on encryption schemes used by the isolated environment.
• Network. Isolating network access — sometimes called a “network-gapped” environment — is critical for a wide range of security vulnerabilities where dependencies are hijacked or malicious code is injected and tries to exfiltrate sensitive data and keys to external counter-parties. Isolating or “gapping” the network access effectively squashes this class of attack vector.

Trust

How can you prove isolation without trust? It’s important to know that the environment is running your untampered code — otherwise you may be sending your sensitive data and keys to the wrong party. The primary way to assert trust is cryptographic signatures with root of trust. This is a core building block of a Trusted Execution Environments (TEE): it can produce a signature that asserts both the underlying environment has not been tampered with and that the running application is signed by a private key controlled by the application developer. Using these system assertions (and of course trusting that the root of trust signing key has not been compromised nor the underlying cryptographic co-processor.)

Meet AWS’s Nitro Enclave

Combining isolation with trust in a hardware-backed platform is not an easy task, but more so making it extensible to run arbitrary workloads is even harder. AWS’s Confidential Computing services (built on the Nitro system) enables us to build efficient and secure workloads that provide Trust and Isolation, our two key ingredients.

Isolation

One of the big “move to cloud” problems early on was tenant-based isolation: how can I guarantee another cloud customer won’t break out of their context and into mine? AWS’s Nitro System was built and improved over many years to harden isolation on a tenant level. However, deploying software in the cloud is more complicated than ever: we run our code in multiple service, vendor code, dependencies and even customer-provided code/logic. There are many more adversaries to protect against.

Nitro Enclaves let us further isolate within our private instance. Enclaves operate their own separate kernel and exclusive access to memory and CPU resources. No network connectivity and no non-ephemeral storage. The only way to push data in and get data out is through a local virtual socket (VSOCK) connection that lives within the EC2 instance.

This means we can carve out a single process — or crucially sensitive code — and run it in an environment that has a dedicated CPU, memory, and is inaccessible to the network. Any other applications, processes, services, or even logged-in users on the system cannot inspect, tamper, or otherwise access the sensitive data or code running in the enclave.

Trust

How do we know that this sensitive code is running inside of a secure enclave? When provisioning a Nitro Enclave, the Nitro Hypervisor creates a signed attestation document. This document creates a number “controls” called PCRs (Platform Configuration Registers) that are cryptographically attested to guarantee certain properties about the enclave. See the table below for the full list of PCR types.

Source: https://docs.aws.amazon.com/enclaves/latest/user/set-up-attestation.html

Notably PCR8 and PCR0 let us attest that a specific image hash or an image signed by a trusted signing key was used to create the enclave.

Once the enclave is built, the attestation document prevents further modification — otherwise the PCR measures will be invalid. Critically: the measures are contiguously monitored and computed for attestation by the hypervisor. A Nitro Enclave can ask the Nitro Hypervisor to generate an attestation document which proves its legitimacy and certifies PCR measures in real-time, all signed by AWS’s Nitro Enclave Attestation root of trust.

Integration with KMS

One of the most useful parts of AWS’s Nitro Enclave attestation document is that AWS’s Key Management Service (KMS) supports policies based on these attestations. Fundamentally, this allows us to harden the security of KMS keys by binding secret key material for use only within the Nitro Enclave. Policies that restrict raw key material to specific Nitro Enclaves with specified PCRX values lets you control EXACTLY which verifiably signed code can access extremely sensitive keys. This is a core building block for how Footprint utilizes Nitro Enclaves in our Enclave Vault Architecture.

How to deploy a Nitro Enclave

While this post won’t go into the development steps of how to actually go about taking some code and deploying inside an of Nitro Enclave, it’s important to cover the basic steps.

1. Build your software as a binary capable of executing on a simple base image, like alpine

2. Package your binary as a docker image, for example create a docker file like follows and build it docker build -t enclave:latest

FROM alpine:latest
COPY ./enclave /enclave
CMD ["/enclave"]

3. Use AWS’s nitro-cli to turn your docker image into a signed EIF (Enclave Image File)

nitro-cli build-enclave --docker-uri enclave:latest --output-file enclave.eif --private-key ../enclavekey.pem --signing-certificate ../cert.pem

4. Now that you have a signed EIF we are able to deploy this to a Nitro Enclave running on a compatible EC2 machine:

nitro-cli run-enclave --eif-path enclave.eif --cpu-count 2 --memory 256 --enclave-cid 16

These 4 steps are the key ingredients to deploying Nitro Enclaves, but there are several important considerations to keep in mind:

• Steps 3 and 4 must be run on a compatible EC2 machine with enclaveOptions enabled. In order to build (and of course run) EIFs we must the available Nitro Enclave hardware and associated nitro enclave kernel loaded.
• You may need to run additional services like a VSOCK proxy (to communicate to AWS KMS over VSOCK). Nitro Enclaves are completely gapped from network interfaces.
• Step 1 may require building your code in such a way that it’s capable of invoking the C AWS Nitro Enclave SDK or otherwise replicates the functionality.

Footprint’s enclave architecture

Now that we know what Nitro Enclaves are and how to deploy them, what kind of code does Footprint run in Nitro Enclaves? Why are Nitro Enclave such useful security building blocks for our core vaulting infrastructure?

Talk to a Nitro Enclave

Footprint vaulting infrastructure is built on three distinct layers: the core APIs, the Enclave Proxy, and Nitro Enclaves running on EC2 machines. The API is largely responsible for two types of actions: (1) encrypting incoming data to be vaulted and (2) preparing requests for the enclave to process previously vaulted data (decryption or functional decryption). The enclave proxy translates requests from the API component all the way down to our deepest layer, the Nitro Enclave.

In theory, the API layer can encapsulate the enclave proxy functionality. The API containers can run on the same EC2 machine as the Nitro Enclaves — this is technically safe due to the isolation properties! However this can pose a problem — as we scale the number of requests that our system can process, we will need to scale up the resources that a single machine has because Nitro Enclaves have dedicated CPU and Memory. The issue is that we are coupling API instances with Enclave instances, which means every time we want to deploy an API container we are forced to deploy Enclaves alongside it. For higher performance and flexibility, we can actually separate the Enclave Proxy from the API layer, and simplify our infrastructure as two high level services: the API service and the Enclave Service

In the diagram above, the Enclave Proxy runs on each EC2 machine along side one or more Nitro Enclaves. A load balancer fronts this group of EC2 machines. This set of machines running on private subnet, frontend by a load balancer encapsulates our enclave service. It’s locked down to only be accessible by subsystems like the API container.

The core invariant

The core invariant of our enclave architecture is that sensitive data is encrypted in a such a way that it can only be decrypted inside of a Nitro Enclave. This is accomplished using public-key cryptography. While the private key is only accessible within the enclave, the public key can be stored safely outside of the enclave and is used to encrypt data as it enters our systems.

In order to decrypt attributes and compute functions over the underlying plaintext, the enclave service defined above, via the enclave proxy, must be used.

A user vault is represented as a set of database rows with encrypted content where the key to the content is protected by the enclave service. A compromise of the database therefore does not compromise the underlying data as you also must compromise the enclave service and its nitro enclave hypervisor.

Initializing a user vault

When a new user enters Footprint, we must initialize the “user vault” to secure their data attributes. The vault record is maintained in a typical transactional database, but the fields (columns) are encrypted with a per-user-vault public key. The private key is backed by AWS’s Key Management Service (KMS), and the policy binds this key to only being accessed with the an Attested Nitro Enclave running our signed code.

1. Our API accepts a new user request and invokes the KMS GenerateDataKeypairWithoutPlaintext. Note that this does NOT return the private key in plain-text. This is important since this operation can be called outside of the enclave context.
2. KMS returns the newly created public key and the Encrypted private key. This is envelope encryption. The private key is encrypted with Nitro-Enclave-Backed KMS root key. This lets us use an indirection to bind the nitro enclave to a root key while using separate, per-user vault key pairs. The decryption operation on this key is protected with a special policy that requires Nitro Enclave attestation with PCR registers set to assert that the Nitro Enclave code is running our signed code.
3. Finally we store the PublicKey and the EncryptedPrivateKey in a database record which houses our encrypted user vault.

Adding data to the user vault

When sensitive new user data enters Footprint, the user’s public key is used to encrypt the data attribute and the resulting encrypted attribute is stored on the user record.

Getting data from the user vault

When entities request functional decryption of user data attributes, Footprint must route the request through the nitro enclave to perform the actual decryption.

1. An external request for a data attribute like birthdate— or a function of that data attribute — age = yearsAgo(birthdate) — is requested from the footprint API on behalf of a user
2. The user’s EncryptedPrivateKey, denoted ePrivKey, and the encrypted attributed (i.e. the encrypted SSN), denoted eAttr is fetched from the user vault record db.
3. An API request with these items is transmitted to the enclave proxy along with the specified function to compute on the plaintext data (i.e. lastFourSsn). The enclave proxy forwards this to the nitro enclave over the RPC-based VSOCK channel
4. First, the ePrivKey is decrypted using KMS (protected by nitro enclave attestation based key policy, transmitted over a VSOCK proxy) to get the PrivKey
5. Next, the PrivKey is used to decrypt the eAttr attribute to get the plaintext Attr attribute
6. Finally, a function (possibly the identity function) is computed on the Attr and the result is sent back to the enclave proxy
7. the enclave proxy communicates this back as a response to the API request from the footprint API
8. Finally, the resulting function of the plaintext data attribute is transmitted back as the response to the original external request

Cryptographic Algorithms

Thus far, we’ve been talking about “encrypting to a public-key” but we haven’t defined the algorithm that we’re actually using to encrypt -sized user data to a public key.

At Footprint, we use the Elliptic Curve Integrated Encryption Scheme (ECIES) Scheme (P256 X9.63-KDF SHA256 AES-GCM). This is a NIST-recommended (SECP-256) elliptic curve for elliptic curve-based secret key exchange (ECDH), with X9.63 SHA-256 key derivation, with data encryption using AES GCM. This is a well-studied and NIST-recommended encryption scheme for ECC.

Source: https://cryptobook.nakov.com/asymmetric-key-ciphers/ecies-public-key-encryption

Public-key encryption algorithms typically only allow for encrypting a small fix-sized block. To encrypt arbitrarily-sized data, we employ a hybrid scheme. To encrypt some data M to a user vault public key:

1. First generate an ephemeral (random) public-private key pair: EPK, ESK
2. Next, perform elliptic curve Diffie-Hellman Key Exchange (ECDH) — also known as secure key agreement — using the ephemeral private key ESK and the user vault public-key, this produces some shared secret K
3. Use a Key Derivation Function (KDF) to uniformly distribute bits of K. In our scheme we use X9.63 ANSI standard with the SHA-256 hash function. This produces SK.
4. Use an Authentication Encryption with Additional Data (AEAD) symmetric encryption scheme to encrypt M. In our case, we use the standard AES-GCM (with 256-bit keys) to produce our cipher-text C. This scheme is designed to encrypt arbitrary sized data and authenticate it so it is also tamper proof, the cipher text contains both the encrypted bytes and the signature.
5. Finally, we package our cryptogram to include: EPK || C

What’s next?

In this post we just covered the surface of why Nitro Enclaves are special and how Footprint uniquely leverages them for our core vaulting infrastructure.

Over the next few months we’ll release follow-up articles that cover topics like

• Improving decryption performance throughput with a second layer of indirection
• Diving deeper into the PCR registers and attestation
• Externally verifiable attestation: how we can prove to our users and customers that certain operations are performed by the Enclave.
• Enclave re-encryption: safely decrypt data by re-encrypting data to a new public key inside the enclave
• User-provided functions for more sophisticated computation on encryption data: allow customers to provide code that runs securely inside the enclave context on sensitive data
• Comparing Nitro Enclaves to Google’s Confidential Computing environment

These are just a few of the privacy, systems security, and cryptography areas we’re working on at Footprint. Stay tuned to learn more!