Store Amazon SES Delivery Notifications in DynamoDB using SNS and Lambdas
I spent way too much time configuring my Amazon AWS account to store Simple Email Service SES Delivery Notifications to a DynamoDB Table. I thought I'd put this guide together to hopefully help future developers.
This guide does not currently cover these topics, but I may include them in the future:
1. Setting up your Amazon AWS Account.
2. Setting up Amazon SES. (dkim, domain/email identities, etc.)
3. Integration with Amazon SES from your app.
4. SNS Topic creation/configuration.
Prerequisites:
1. Amazon AWS account has been created/configured.
2. Amazon SES has been created/configured.
3. Amazon SNS Topic has been created/configured.
4. Delivery Notifications for the Amazon SES Domain Identity have been configured to be sent to the SNS Topic.
5. Your app is integrated with Amazon SES and successfully sending email.
Step by Step Guide to Storing SES Delivery Notification messages into a DynamoDB Table:
So at this point, you're successfully integrated with Amazon Simple Email Service SES and sending email, and you can prove that the Delivery notifications are being sent to a Simple Notification Service SNS Topic....
NOTE: All these services (SES, SNS, Lambda, DynamoDB) should be in the same region. The links I include in this guide are for us-west-2 Oregon.
We'll be using AWS Lambdas to process the SNS message and store the value into a DynamoDB Table.
1. Create a DynamoDB Table. (NoSQL db, more info)
Start here: https://us-west-2.console.aws.amazon.com/dynamodb/home
- Create a table named: "ses-delivery"
- Primary key (partition key, hash key) "messageId", type string. (each SNS message has a unique UUID, that will be our partition key)
- Note: Depending on your use case, you may want to use a different value as the partition key and/or set up a global secondary index over email address.
Note: it will have 5 read and write capacity units by default, you'll need to bump those up to whatever values are appropriate for your app. (I may add more info on capacity units here later... tbd)
2. Create a new Lambda function. (light weight compute, sub-second metering, more info)
Start here: https://us-west-2.console.aws.amazon.com/lambda/home
- Create a new lambda function (click the button)
- In the Select Blueprint section, search for SNS
- Choose the "sns-message" template
- on the next screen, choose the event source of SNS, and choose the correct topic.
- hit the next button.
- on the next screen, give your lambda a name (I go with "ses-delivery-sns-to-dynamo")
- keep the runtime as node.js
- leave the default code, for now
- leave the handler value "index.handler:
- choose a new role, choose the "Create new role" > "* basic execution role" (we'll add dynamodb access to the IAM role it creates in a couple steps)
- leave memory of 128MB (more than enough)
- leave timeout around 3-5 secs (and yes, that is seconds, the entire write/compute takes ~500ms)
- hit NEXT
- review everything, leave the radio "enable later" selected, hit CREATE function button.
3. Configure Lambda Role
Start here: https://console.aws.amazon.com/iam/home
- click on "Roles" on the left hand side.
- click on the role that was just created, most likely "lambda_basic_execution"
- click on the Permissions tab, and click the "Attach Policy" button.
- search for DynamoDB, and select "AmazonDynamoDBFullAccess"
4. Configure Lambda Function
You now have a brand new lambda function, let's set it up.
Start here: https://us-west-2.console.aws.amazon.com/lambda/home
- By default there will be a single event source tied to the SNS topic - It will be deleted when we publish the first version of this lambda (I'll explain in a bit, and remind you again to verify below).
- Notice there is a little arrow on the far left side. Hit it so the versions of the lambda expand.
- Notice that we are on "$LATEST" this is the only version that you can modify the code, you'll publish a version to eventually use and tie to the actual SNS Topic. You'll see published versions listed here eventually.
- Go here, and copy paste the code into your lambda: https://gist.github.com/aaronhoffman/c2b7ef532b7a28e99aae#file-sns-to-dynamodb-lambda-js
- Go to the Actions dropdown and select "configure test event"
- Choose any of the defaults, then overwrite it with this (that is an actual SES>SNS>Lambda message, some values obfuscated to protect the innocent)
- Choose Save. Then click the Test button.
- The output shows up below the code (there should be a nice little green check mark next to the word "success" -- you have no idea how long it took me to get here... and I don't want to say)
5. Publish and Configure Lambda Version
Start here: (where we left off in #4) https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/ses-delivery-sns-to-dynamo
- Click on the Actions dropdown button and select "Publish new version"
- A new version has been created, and the event source on $LATEST should now be deleted (please verify - this is your reminder.)
- With your new version selected, choose the "event source" tab, click the "Add event source" button.
- Event Source type: SNS
- SNS topic (likely "ses-delivery")
- Choose "Enable now"
- Submit!
If successful, we should see logs right away. To verify, go to CloudWatch
Within CloudWatch choose the log group that matches your lambda name
You should see a list of some log streams. (Note: the logs for executions of multiple SNS messages may be included in a single log stream.)
Look in these logs for errors... hopefully there aren't any.
Go back to your dynamodb table and scan for records:
https://us-west-2.console.aws.amazon.com/dynamodb/home?region=us-west-2#tables:selected=ses-delivery
Next Steps
You may want to set up notifications for Bounce and Complaint rates. See here for a good walk-through: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/reputationdashboard-cloudwatch-alarm.html
And that's it, hope this helps!
Aaron
p.s. hilariously, AWS published an article 6 days later on how to do this: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-sns-ses-dynamodb/
This guide does not currently cover these topics, but I may include them in the future:
1. Setting up your Amazon AWS Account.
2. Setting up Amazon SES. (dkim, domain/email identities, etc.)
3. Integration with Amazon SES from your app.
4. SNS Topic creation/configuration.
Prerequisites:
1. Amazon AWS account has been created/configured.
2. Amazon SES has been created/configured.
3. Amazon SNS Topic has been created/configured.
4. Delivery Notifications for the Amazon SES Domain Identity have been configured to be sent to the SNS Topic.
5. Your app is integrated with Amazon SES and successfully sending email.
Step by Step Guide to Storing SES Delivery Notification messages into a DynamoDB Table:
So at this point, you're successfully integrated with Amazon Simple Email Service SES and sending email, and you can prove that the Delivery notifications are being sent to a Simple Notification Service SNS Topic....
NOTE: All these services (SES, SNS, Lambda, DynamoDB) should be in the same region. The links I include in this guide are for us-west-2 Oregon.
We'll be using AWS Lambdas to process the SNS message and store the value into a DynamoDB Table.
1. Create a DynamoDB Table. (NoSQL db, more info)
Start here: https://us-west-2.console.aws.amazon.com/dynamodb/home
- Create a table named: "ses-delivery"
- Primary key (partition key, hash key) "messageId", type string. (each SNS message has a unique UUID, that will be our partition key)
- Note: Depending on your use case, you may want to use a different value as the partition key and/or set up a global secondary index over email address.
Note: it will have 5 read and write capacity units by default, you'll need to bump those up to whatever values are appropriate for your app. (I may add more info on capacity units here later... tbd)
2. Create a new Lambda function. (light weight compute, sub-second metering, more info)
Start here: https://us-west-2.console.aws.amazon.com/lambda/home
- Create a new lambda function (click the button)
- In the Select Blueprint section, search for SNS
- Choose the "sns-message" template
- on the next screen, choose the event source of SNS, and choose the correct topic.
- hit the next button.
- on the next screen, give your lambda a name (I go with "ses-delivery-sns-to-dynamo")
- keep the runtime as node.js
- leave the default code, for now
- leave the handler value "index.handler:
- choose a new role, choose the "Create new role" > "* basic execution role" (we'll add dynamodb access to the IAM role it creates in a couple steps)
- leave memory of 128MB (more than enough)
- leave timeout around 3-5 secs (and yes, that is seconds, the entire write/compute takes ~500ms)
- hit NEXT
- review everything, leave the radio "enable later" selected, hit CREATE function button.
3. Configure Lambda Role
Start here: https://console.aws.amazon.com/iam/home
- click on "Roles" on the left hand side.
- click on the role that was just created, most likely "lambda_basic_execution"
- click on the Permissions tab, and click the "Attach Policy" button.
- search for DynamoDB, and select "AmazonDynamoDBFullAccess"
4. Configure Lambda Function
You now have a brand new lambda function, let's set it up.
Start here: https://us-west-2.console.aws.amazon.com/lambda/home
- By default there will be a single event source tied to the SNS topic - It will be deleted when we publish the first version of this lambda (I'll explain in a bit, and remind you again to verify below).
- Notice there is a little arrow on the far left side. Hit it so the versions of the lambda expand.
- Notice that we are on "$LATEST" this is the only version that you can modify the code, you'll publish a version to eventually use and tie to the actual SNS Topic. You'll see published versions listed here eventually.
- Go here, and copy paste the code into your lambda: https://gist.github.com/aaronhoffman/c2b7ef532b7a28e99aae#file-sns-to-dynamodb-lambda-js
- Go to the Actions dropdown and select "configure test event"
- Choose any of the defaults, then overwrite it with this (that is an actual SES>SNS>Lambda message, some values obfuscated to protect the innocent)
- Choose Save. Then click the Test button.
- The output shows up below the code (there should be a nice little green check mark next to the word "success" -- you have no idea how long it took me to get here... and I don't want to say)
5. Publish and Configure Lambda Version
Start here: (where we left off in #4) https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/ses-delivery-sns-to-dynamo
- Click on the Actions dropdown button and select "Publish new version"
- A new version has been created, and the event source on $LATEST should now be deleted (please verify - this is your reminder.)
- With your new version selected, choose the "event source" tab, click the "Add event source" button.
- Event Source type: SNS
- SNS topic (likely "ses-delivery")
- Choose "Enable now"
- Submit!
If successful, we should see logs right away. To verify, go to CloudWatch
Within CloudWatch choose the log group that matches your lambda name
You should see a list of some log streams. (Note: the logs for executions of multiple SNS messages may be included in a single log stream.)
Look in these logs for errors... hopefully there aren't any.
Go back to your dynamodb table and scan for records:
https://us-west-2.console.aws.amazon.com/dynamodb/home?region=us-west-2#tables:selected=ses-delivery
Next Steps
You may want to set up notifications for Bounce and Complaint rates. See here for a good walk-through: https://docs.aws.amazon.com/ses/latest/DeveloperGuide/reputationdashboard-cloudwatch-alarm.html
And that's it, hope this helps!
Aaron
p.s. hilariously, AWS published an article 6 days later on how to do this: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-sns-ses-dynamodb/
Comments