Exporting DynamoDB to S3 — cross-account and SSE-KMS encryption

Most of us who have worked with DynamoDB have had this requirement of exporting data to S3. Recently, I also had the same requirement and it turned out it was not as simple as I had thought it to be.

Requirement:

Export DynamoDB to a cross-account S3 bucket on an overnight basis. The destination S3 bucket expects objects to have SSE-KMS encryption.

My first reaction:

This is a simple requirement (isn’t it always the case). It is going to take me like a few hours. So, I had a couple of approaches in mind and I started evaluating them.

Important point: AWS had not released the native Export DynamoDB table to S3 feature then.

Approach 1: Lambda

Well, lambda is the bash of AWS. It is a panacea of almost all problems in AWS. But, I wanted to try other approaches first. So, kept on hold.

Approach 2: Data Pipeline

When I Google, this seems to be one of the popular solutions for this problem. However, it had a few cons

• Too complicated, especially writing the Cloudormation template for this
• The DynamoDB table was a small one and the idea of launching an EMR cluster for this sounded like an overkill

Hence, rejected the solution.

Approach 3: Glue

Now, here I was excited a bit as AWS glue is in principle a serverless solution and I have a “Serverless First” approach in my project. Moreover, I had personally not worked properly on glue yet. So, I thought this would be a good project for me to get a taste of it.

However, the project went on a back burner because of a change in priorities and I had to move this back to my backlog. Few weeks down the line, AWS released the native DynamoDB Export Data to S3 feature and almost at the same time, the project came out of the backlog. Again, I was super excited about this, since this is exactly what we wanted.

Approach 4: Native DynamoDB export to S3

All I had to do now was write a simple lambda ( I know!!! ) to export the data and trigger the lambda using Cloudwatch Event Rule daily. The export API expected the table to have point in time recovery, which was fine by us. The API also took S3BucketOwner and S3SseKmsKeyId as a parameter, so the cross-account and SSE-KMS support seemed to be already there. Alas, it was not to be. The approach would fail because the ExportTableToPointInTime API would not change the object ownership to the new account (argh!!!). A chat with the AWS support reveals that it is a known issue and they are working on it and they obviously had no ETA when it would get fixed.

So, next, I had to export the table to the same account where the DynamoDB was. Then, I had to somehow copy the exported objects to the other bucket and transfer the ownership at the same time. However, that would require me to write two lambdas. Moreover, the table export API was async in nature and does not wait for the export operation to be over (it just returns the ExportArn). Additionally, since the export generates multiple files, it is difficult to know when the export is really complete to start the copy process.

Finally, S3 replication came to the rescue. S3 had already introduced the Same Region Replication, which definitely helped in deciding as both the source and destination buckets were in the same region. Alas, it was not so easy and straight forward given that source bucket was using SSE-S3 encryption and the destination bucket was expecting SSE-KMS encryption. The objects did get replicated, but with SSE-S3 replication. And after hours of investigation and another chat with the nice people at AWS support, I learnt that if the destination object expects the object to have SSE-KMS encryption, the source bucket has to use the same (albeit, the KMS keys can be different, just the encryption method need to be SSE-KMS). So, finally, after changing the encryption in the source bucket to SSE-KMS, the objects got replicated successfully. I do not know why this is not documented. The nice people at AWS Support assured that they will request the S3 team to clarify it in the documentation.

In a nutshell, the below steps were required

• Export DynamoDB to an S3 bucket in the same account
• Use SSE-KMS as the mode for encrypting objects in the source bucket
• Setup S3 replication to copy the objects to the destination bucket

Hopefully, AWS resolves the issue. The ExportTableToPointInTime is a great feature and can save lots of man-hours for people like us.

This article has been written as part of the Spice Program @ Futurice (an open-source and social impact program)