Serverless Replacement for Basic Email Services

If you are like me you've accumulated a number of domain names over the years for various projects and in most cases there is a need to send and receive emails. Many online services consumed by your website or application depend on email to verify ownership or send notifications, so even with very light email volumes the requirement for mail exchange often exists. The traditional route is to sign up with a hosting provider that offers mail services but there is another way that leverages the emerging serverless architecture approach.

A guide to using Amazon Web Services Simple Email Service and Lambda to receive and forward emails.

What is Amazon SES?

Amazon Web Services (AWS) Simple Email Service (SES) is capable of sending and receiving email using your domain names. When SES was launched in January 2011 the target use case was an organization sending large volumes of email that didn't want the responsibility of building and managing mail servers. Technology solutions such as DKIM and SPF and government legislation (e.g. CAN-SPAM Act) had been introduced to attempt to address the proliferation of unsolicited email, but these developments also made it more challenging to manage an in-house email platform.

Organizations using SES could send almost any scale of transactional emails and correspondence while only having to pay for the number of emails sent and amount of data transferred. Currently the charge is $0.10 USD for every 1,000 emails sent which includes content filtering, handling of bounce backs, delivery attempt success notifications, and processing spam complaints.

In September 2015 AWS added the capability to receive email with SES. Beyond the basics of communicating with other mail servers SES also scans incoming email for spam and viruses and can reject mail from untrusted sources.

As an integrated member of Amazon's portfolio of services, SES can also chain together actions upon receiving an email, such as storing it in an S3 bucket, publishing it to a Simple Notification Service (SNS) topic, calling custom Lambda functions, or transferring the message to WorkMail.

What is AWS Lambda?

AWS Lambda is an event-driven, serverless computing platform. You upload your code in a supported language (currently Node.js, Java, and Python), and Lambda runs the code (called a function), on a high-availability compute fleet that automatically scales from a few requests per day to thousands per second. All of the typical administration tasks of managing compute resources, networking, capacity management, and logging are handled by Lambda.

AWS Lambda executes your code only when triggered by configurable event sources, and you are charged based on the number of requests for your functions, the time your code executes, and if your function utilizes other AWS services or transfers data. Event sources that can currently trigger Lambda functions are Amazon's API Gateway, IoT, Alexa Skills Kit, Alexa Smart Home, CloudWatch, Cognito, Dynamo DB, Kinesis, S3, and SNS.

If you are interested in learning more about Lambda the 3 minute video produced by Amazon and embedded in this post provides an introductory overview. There is also a very informative article on Serverless Architectures on MartinFowler.com.

What Can We Build?

This post started with a need to provide email services for domain names that are primarily associated with an application or project. Rather than building and maintaining a mail server or paying a fixed monthly rate to a hosting provider I used Amazon SES, S3, and Lambda to build a low-cost, serverless mail receiving and forwarding solution. It is a novel solution to meet a ongoing need, but more importantly, a practical application of the associated technologies and concepts.

This scenario assumes the following:

  • The domain name associated with your project is project.com, and you've signed up for a web analytics service that needs to send a verification email to admin@project.com.
  • The business or personal mailbox you regularly use to send and receive emails is you@yourdomain.com.
  • You with to have any emails sent to admin@project.com forwarded to you@yourdomain.com.
  • You have an Amazon Web Services account to follow along with the guide.
  • You have a DNS service already setup for the project.com domain. It can be AWS Route 53 or from another provider.
  • You have a beginner level of proficiency with JavaScript.

Here is a diagram to help visualize the architecture and flow once the build steps have been completed:

After you have registered a new account with the web analytics company and their verification process kicks off the following high-level steps will occur:

  1. The company's mail server will perform an MX record lookup for the project.com domain.
  2. The company's mail server will transmit the verification email to the Amazon SES SMTP server specified in the MX record.
  3. Amazon SES will store the email in MIME format in an S3 bucket.
  4. SES will invoke a Lambda function.
  5. The Lambda function will begin executing and read in the MIME email from the S3 bucket.
  6. After processing the email the Lambda function will pass the email to SES to send.
  7. SES will perform an MX record lookup for yourdomain.com and then transmit the email to the specified mail server. At this point you can now retrieve the email from the you@yourdomain.com mailbox using your preferred mail client.

Here is an outline of the steps needed to configure your DNS service, SES, Lambda, and IAM.

Step One - Verify Your Domain for Amazon SES Email Receiving

  1. Sign in to the AWS Management Console and open the Amazon SES console at https://console.aws.amazon.com/ses.
  2. In the navigation pane, under Identity Management, choose Domains.
  3. Click the Verify a New Domain button.
  4. In the Verify a New Domain dialog box, enter the domain name project.com. Select the Generate DKIM Settings option. Click the Verify This Domain button.
  5. In the Verify a New Domain dialog box, you will see a Domain Verification Record Set and a DKIM Record Set containing Name, Type, and Value sets. It will also list an Email Receiving Record that is needed to route incoming mail to Amazon SES.
  6. If you use Amazon Route 53 to provide DNS service for your domain follow these steps:
    • At the bottom of the Verify a New Domain dialog box under Amazon Route 53 Customers click the Use Route 53 button.
    • In the Use Route 53 dialog box verify that Domain Verification Record, DKIM Record Set, and Email Receiving Record are selected. Under Hosted Zones verify that there is a check mark next to your domain.
    • Click the Create Record Sets button.
  7. If you use a DNS service other than Route 53 you need to add the displayed TXT records to your domain's DNS server. It may take up to 72 hours for domain verification to complete.
  8. When verification is complete, the domain's status in the Amazon SES console will change from pending verification to verified.

Step Two - Configure the SES Email Receiving Rule Set

  1. From the Amazon SES navigation pane, under Email Receiving, choose Rule Sets.
  2. Click the Create a Receipt Rule button.
  3. On the Step 1: Recipients page enter the recipient email address admin@project.com in Recipient table and click on the Add Recipient button.
  4. Confirm that the Verification status column says Verified for the recipient you just added and then click the Next Step.
  5. On the Step 2: Actions page click the Select an action type menu and select S3.
  6. For the S3 Bucket click the menu and select Create S3 Bucket. Enter a unique Bucket Name. For this guide we'll use project-emails. Click the Create Bucket button.
  7. For the Object key prefix enter a value to group all of the email objects in the bucket. The prefix value can be thought of as a directory to store similar data. Typically a trailing forward slash (/) is appended to the prefix. You can leave this field blank or set it to admin/ so that emails to admin@project.com are grouped separately in the event you want to store other mail in the same S3 bucket.
  8. Leave the Encrypt Message option deselected for this walk-through.
  9. Leave the SNS topic set to <None>.
  10. Click the Next Step button.
  11. On the Step 3: Rule Details page enter a Rule name, ensure Enabled and Enable spam and virus scanning are selected, set Rule set to default-rule-set, and set Insert after rule to <Beginning>. Click the Next Step button.
  12. On the Step 4: Review page review the rule settings and then click Create Rule.

Step 3 - Create the Lambda Function

  1. Open the Amazon Lambda console at https://console.aws.amazon.com/lambda
  2. Click Get Started Now button
  3. Scroll down to the bottom and click the Skip button
  4. Enter sesForwarderProject for the Name and whatever you'd like for the Description. Make sure Runtime is set to Node.js 4.3. Paste the contents of the AWS Lambda SES Email Forwarder index.js file into the Lambda function code area.
  5. Modify the values in the config object at the top of the code to specify the S3 bucket you created earlier, "project-emails", and object prefix you used (either "" or "admin/") for locating emails stored by SES. Also provide the email forwarding mapping from original destination (admin@project.com) to new destination (you@yourdomain.com). It should look like this when you are done assuming you didn't use a prefix for your S3 bucket:
  6. Ensure Handler is set to index.handler.
  7. Select Basic Execution Role from the role list. A new window will appear to grant Lambda permissions to other AWS resources.
  8. Choose Create a new IAM Role from the IAM Role drop down and provide the Role Name LambdaSESForwarder. Click View Policy Document and then Edit to edit the policy document. Copy and paste the policy document document below, also taken from the AWS Lambda SES Email Forwarder repository, into the text area:
  9. Click the Allow button.
  10. Memory can be left at 128 MB, but set Timeout to 10 seconds to be safe. The task usually takes about 30 MB and a few seconds. After testing the task, you may be able to reduce the Timeout limit.
  11. Click the Next button.
  12. Click the Create Function button.

Step Four - Add the Lambda Function to the SES Rule Set

  1. Open the Amazon Lambda console at https://console.aws.amazon.com/ses.
  2. From the navigation pane, under Email Receiving, choose Rule Sets.
  3. Click the View Active Rule Set button.
  4. Click the Rule Name you created previously in Step 2.
  5. On the Edit Rule page scroll down to the Add action row and select <Lambda> from the menu.
  6. For the newly added Lambda action choose the sesForwarderProject function. Leave the Invocation Type set to Event and the SNS Topic set to <none>.
  7. Click the Save Rule button.
  8. If you are asked for SES to attempt to add permissions to access lambda:InvokeFunction, agree to it.

Step Five - Verify Email Address

Amazon places new SES users in a sandbox that has the following restrictions:

  • You can only send mail to the Amazon SES mailbox simulator and to verified email addresses and domains.
  • You can only send mail from verified email addresses and domains.
  • You can send a maximum of 200 messages per 24-hour period.
  • Amazon SES can accept a maximum of one message from your account per second.

Details on how to have these restrictions lifted can be found in Moving Out of the Amazon SES Sandbox.

In order to use the Lambda/SES/S3 forwarder you have to verify you@yourdomain.com so that you can forward emails sent to admin@project.com.

  1. From the Amazon SES navigation pane, under Identity Management, choose Email Addresses.
  2. Click on Verify a New Email Address at the top of the screen.
  3. Enter you@yourdomain.com and click Verify This Email Address.
  4. Once you receive an email from AWS, click on the link to complete the verification.

Ongoing Maintenance

If you need to add another email address that you want forwarded you simply need to make modifications in two areas of AWS. For demonstration purposes we'll say that we want sales@project.com to be forwarded to jdoe@yourdomain.com.

Update SES Rule

  1. Open the Amazon Lambda console at https://console.aws.amazon.com/ses.
  2. From the navigation pane, under Email Receiving, choose Rule Sets.
  3. Click the View Active Rule Set button.
  4. Click the Rule Name you created.
  5. Under Recipient enter the new email address you wish to have forwarded (sales@project.com) and click the Add Recipient button.

Update Lambda Function

  1. Open the Amazon Lambda console at https://console.aws.amazon.com/lambda
  2. Click the Function name you created.
  3. On the Code tab inline editor change the defaultConfig hash variable to:
  4. Scroll up and click the Save button.

Summary

If you've made it this far then you should have a working email forwarding system. Sending an email to the address you wish to have forwarded (admin@project.com in this guide) should be received by AWS SES, stored in an S3 bucket, processed by a Lambda function, and then transmitted to your mailbox (e.g. you@yourdomain.com) in under 30 seconds.

To take this a step further you could extend the functionality by storing the fromEmail and forwardMapping values in a DynamoDB table and presenting a simple form for CRUD operations. I personally don't modify the mailboxes and forwarding all that frequently, so manual steps involved are not overly cumbersome.

If you ran into any issues please leave a comment, and I'll try to help.