19 May 2015

Sending emails from Rails is not that hard, but keeping them out of user's spam folders can be difficult. The first time I set up a production rails app as an MVP for a project, a significant number of users didn't complete sign up because the account confirmation email went straight to their spam folder. There are a lot of reasons this can happen, and a fair number of tools online that will tell you what they are. Fixing the issues often involves sending mail through a delivery partner so that bounces and sender reputation are handled cleanly, but the providers vary a lot in what they offer.

Sendgrid offers a lot of nice features, but is relatively expensive, so why use it if you're just sending simple emails and don't need the extra features? SES on AWS is roughly ten times cheaper, so can be a very good option, but the setup is a lot more complex. Your app requires specialist bounce handling before they'll let you send real emails outside of the sandbox, so you need to write a fair bit of code. The idea is that when an email address is dead, your app hears about it via the bounce notification and stops sending emails to it. Without this, both you and AWS are going to get a reputation as spammers, and your deliverability will suffer so it's a good thing that they insist on this.

This post will show you how to get it working. Before you start, you'll need:

An AWS account with admin access
A production Rails app hosted live under the production domain name
HTTPS set up and working for your production app with a valid certificate
Administrator access to the domain registrar account for the production domain.
Access to emails that are sent to the 'from' address that is set up in your app

Step 1: Set the IAM SMTP user and add the credentials to your action mailer config.

To send email from Rails, you need an SMTP username and password.

To get this for SES, start by logging into the AWS management console and head to the SES section. Then, go to SMTP Settings and click the Create My SMTP Credentials button. Working through the steps it presents, you can accept the defaults and it will give you a new IAM user with a username and password combination (More info here). Make a note of these, and add them to your production application's SMTP settings.

Step 2: Verify your domain and enable DKIM.

Now we have to prove to AWS and others that your domain is actually yours and that you're not a spammer. This is key to keeping your mails in the inboxes of your recipients where they belong.

Add your production domain to the list on the Domains page within the SES section. To do this, click the Verify a New Domain button, enter the domain name, and make sure you tick the Generate DKIM Settings box. If you don't know why DKIM matters, you can get a pretty good overview from Wikipedia here. In short, it makes it harder for spammers to send messages that look like they're from your domain when they're not.

In the next stage, AWS will show you some DNS records that you will need to add to your existing DNS records with your registrar. Your registrar will have their own instructions for how you can do this, so the next step is to find these instructions and follow them. More information on the process is in the AWS docs here.

Domain verification is automatic and can take some time. For me it was a couple of hours. Once it's done, make sure you turn on the DKIM setting for the domain in the SES settings.

Step 3: Verify your from address.

Similarly, AWS need to know that you're not maliciously sending email with someone else's account as the from address.

You should have the address you want to use set up in Rails already. Something like [email protected]. This appears in the SMTP settings (or in your mailers). Go to the Email addressed page in the SES settings and click the Verify a New Email Address button. This is pretty straightforwards - you just click on the confirmation link in the email you receive. However, it's possible that the address does not have a mailbox, so you will need to make one first, or get it to forward messages to your personal address temporarily.

Once this is done, turn on DKIM for that address too.

Step 4: Add an SNS topic and add your own email address to it

We now need a way for the bounces to be routed to your app, and SNS is the way AWS handles this. Adding your own email to start with means that bounce will be forwarded to you so you can verify that the SNS topic is working.

Go to the SNS section of the AWS console, click on the Topics link, and then Create a new topic. Do this twice, once for yourapp_bounces, and once for yourapp_complaints.

For each of them, click on the link in the list that starts with 'arn', then click Create subscription, then choose email and enter your personal email address. This is just for testing, so remember to remove this afterwards.

You are now ready to test that the bounce and complaint handling will pick up correctly when your app sends mail.

Go to your app and do whatever it takes to get an email sent to the bounce and complaint testing addresses that you can find here. You should find that the SNS topic sends a notification to your personal email address informing you of the problem.

Step 5: Add bounce handling in your app.

To stop future emails being send to dead addresses, your app needs to store a record of the email having bounced and will mark the address as invalid. The process is:

SNS should call a HTTPS post endpoint in your app (instead of sending an email to you)
The endpoint verifies that it's really AWS using the AWS SDK (certificate signing)
The email address is then marked as being a bounce in your app's DB
Your app's mailers always check this list before sending

It's a good idea to make sure that you and your users can see the bounces and complaints in the application UI somewhere, otherwise you'll get users filing bug reports if they've misspelled their email.

Here's the code I used to implement the bounce mechanism:

Controller Tests

These are just a guide. You may not be using Rspec or VCR (maybe you should :), but this shows the general idea.

require 'rails_helper'

describe EmailResponsesController do
  let(:user) { create :user }
  let(:other_user) { create :user }

  describe 'POST bounce' do

    let(:bounce_json) do
      %q{
        {
          "Type" : "Notification",
          "MessageId" : "2d222228-90e1-5ba1-9784-9c60222222e5",
          "TopicArn" : "arn:aws:sns:eu-west-1:863122222282:yourapp_bounces",
          "Message" : "{\"notificationType\":\"Bounce\",\"bounce\":{\"bounceSubType\":\"General\",\"bounceType\":\"Permanent\",\"bouncedRecipients\":[{\"status\":\"5.1.1\",\"action\":\"failed\",\"diagnosticCode\":\"smtp; 550 5.1.1 user unknown\",\"emailAddress\":\"[email protected]\"}],\"reportingMTA\":\"dsn; a6-26.smtp-out.eu-west-1.amazonses.com\",\"timestamp\":\"2015-01-21T00:37:34.429Z\",\"feedbackId\":\"00000122222225fb-9a217d45-8eed-4403-b7d9-172222222be7-000000\"},\"mail\":{\"timestamp\":\"2015-01-21T00:37:33.000Z\",\"source\":\"[email protected]\",\"destination\":[\"[email protected]\"],\"messageId\":\"000222222222429c-49e59593-7f16-44e3-b5a2-1a222222222c6-000000\"}}",
          "Timestamp" : "2015-01-21T00:37:34.452Z",
          "SignatureVersion" : "1",
          "Signature" : "ZmtUp+wmfSNUW0dGkzVN9A9Q7R88KwfU32zXTHn43pppZAd6pJlwKRdH/B9Ui4M1Sd1gC1Zi2WgLtC2Xi7kf60bdi66a222222222222QKeRy1GbqWfT9kcOvm4aAFlRy0FoDu2FD9iyGPu62RsABlNzLYGNI1YnmIKwHSXXy/St4MMwwGjGLprZsRSSimM/B9VJr5WCzwMmKYr/kWGVr3pN8B5WMRuFw0pYBIemJoZugkHElEypz8KNVRxSwRN3Iz7me38mMDPCs1xiv/6IyNQXu1pdtEWkAPb26feK3SuBpPNpTQtbnaZvt7wNSw==",
          "SigningCertURL" : "https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-d6d622222222222222221f4f9e198.pem",
          "UnsubscribeURL" : "https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:82222222222282:yourapp_bounces:222222224f-7a0a-4e07-b34d-4a662222222226"
        }

      }
    end

    let(:invalid_bounce_json) do
      %q{
        {
          "Type" : "Notification",
          "MessageId" : "2222222222228-90e1-5ba1-9784-9c2222222222225",
          "TopicArn" : "arn:aws:sns:eu-west-1:862222222282:yourapp_bounces",
          "Message" : "{\"notificationType\":\"Bounce\",\"bounce\":{\"bounceSubType\":\"General\",\"bounceType\":\"Permanent\",\"bouncedRecipients\":[{\"status\":\"5.1.1\",\"action\":\"failed\",\"diagnosticCode\":\"smtp; 550 5.1.1 user unknown\",\"emailAddress\":\"[email protected]\"}],\"reportingMTA\":\"dsn; a6-26.smtp-out.eu-west-1.amazonses.com\",\"timestamp\":\"2015-01-21T00:37:34.429Z\",\"feedbackId\":\"0000022222222222fb-9a217d45-8eed-4403-b7d9-17222222222227-000000\"},\"mail\":{\"timestamp\":\"2015-01-21T00:37:33.000Z\",\"source\":\"[email protected]\",\"destination\":[\"[email protected]\"],\"messageId\":\"0000022222222222c-49e59593-7f16-44e3-b5a2-1222222222226-000000\"}}",
          "Timestamp" : "2015-01-21T00:37:34.452Z",
          "SignatureVersion" : "1",
          "Signature" : "ZmtUp+wmfSNUW0dGkzVN9A9Q7R88KwfU3222222222222gC1Zi2WgLtC2Xi7kf60bdi66aY/R76EV80xL/Znfn9YYQKeRy1GbqWfT9kcOvm4aAFlRy0FoDu2FD9iyGPu62RsABlNzLYGNI1YnmIKwHSXXy/St4MMwwGjGLprZsRSSimM/B9VJr5WCzwMmKYr/kWGVr3pN8B5WMRuFw0pYBIemJoZugkHElEypz8KNVRxSwRN3Iz7me38mMDPCs1xiv/6IyNQXu1pdtEWkAPb26feK3SuBpPNpTQtbnaZvt7wNSw==",
          "SigningCertURL" : "https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-d6d6792222222222222222222229e198.pem",
          "UnsubscribeURL" : "https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:8622222222222222:yourapp_bounces:28222222222224f-7a0a-4e07-b34d-422222222222f6"
        }

      }
    end

    it 'creates a new bounce record' do
      expect {
        VCR.use_cassette 'check_sns_message' do
          post :bounce, bounce_json, format: :json
        end
      }.to change(EmailResponse, :count).by(1)
    end

    it 'saves the information it should' do
      allow(EmailResponse).to receive(:create)
      expect(EmailResponse).to receive(:create).with(hash_including(email: '[email protected]', response_type: 'bounce', extra_info: "status: 5.1.1, action: failed, diagnosticCode: smtp; 550 5.1.1 user unknown"))
      VCR.use_cassette 'check_sns_message' do
        post :bounce, bounce_json, format: :json
      end
    end

    it 'fails if it cannot verify the message' do
      expect {
        VCR.use_cassette 'check_sns_message' do
          post :bounce, invalid_bounce_json, format: :json
        end
      }.to change(EmailResponse, :count).by(0)
    end
  end

  describe 'POST complaint' do

    let(:complaint_json) do
      %q{
        {
          "Type" : "Notification",
          "MessageId" : "092cf767-9156-5244-b2d5-ba2222225436",
          "TopicArn" : "arn:aws:sns:eu-west-1:8622222222243:yourapp_email_complaints",
          "Message" : "{\"notificationType\":\"Complaint\",\"complaint\":{\"complaintFeedbackType\":\"abuse\",\"complainedRecipients\":[{\"emailAddress\":\"[email protected]\"}],\"userAgent\":\"Amazon SES Mailbox Simulator\",\"timestamp\":\"2015-01-21T12:56:53.000Z\",\"feedbackId\":\"0000014b022222e7-f934da45-a45c-11e4-9559-192222eb80f1-000000\"},\"mail\":{\"timestamp\":\"2015-01-21T12:56:52.000Z\",\"source\":\"[email protected]\",\"destination\":[\"[email protected]\"],\"messageId\":\"0000014b0c911fd2-c7eb049f-f610-4b56-6541-2e2222249521-000000\"}}",
          "Timestamp" : "2015-01-21T12:56:54.396Z",
          "SignatureVersion" : "1",
          "Signature" : "u7u/zeRG5KC3K7CpXll22222fwbrAGMn9rZZTTxmy2vrIyioEpIXtCaT6MhjUum2erYBi0Doo8K0/nmD+vNJMK43+IGtqsjQZeEwtr6cWDDJyrxoX53a18fp9YqBNTzwvu9TOkTNn3fUhqbumw9fH1+ltQ3qeDRP1DrpkJczQ080cZPmkF2xeDL2222IDlZJJkWpvivIrt9ZBS/lW4HU0UpjvHVAZhxgZyUoWuRMOM7j3q3aRh/RB9aHOOAw8wdfg5ie8vHSbcEOVdj2fakGfUM3kCVrgm983AjJt2SA==",
          "SigningCertURL" : "https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-d6d679a1d2222522222222222198.pem",
          "UnsubscribeURL" : "https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:863132222282:yourapp_email_complaints:0222222c-d183-45d8-b0ac-787d02222084"
        }
      }
    end

    let(:invalid_complaint_json) do
      %q{
        {
          "Type" : "Notification",
          "MessageId" : "092cf767-9156-5244-b2d5-ba5222222d80",
          "TopicArn" : "arn:aws:sns:eu-west-1:863122222282:yourapp_email_complaints",
          "Message" : "{\"notificationType\":\"Complaint\",\"complaint\":{\"complaintFeedbackType\":\"abuse\",\"complainedRecipients\":[{\"emailAddress\":\"[email protected]\"}],\"userAgent\":\"Amazon SES Mailbox Simulator\",\"timestamp\":\"2015-01-21T12:56:53.000Z\",\"feedbackId\":\"0000014b0c9126e7-f922222b-a16c-11e4-9559-122221eb80f1-000000\"},\"mail\":{\"timestamp\":\"2015-01-21T12:56:52.000Z\",\"source\":\"[email protected]\",\"destination\":[\"[email protected]\"],\"messageId\":\"0000014b022222d2-c722222f-f610-4b56-1237-2e2222249521-000000\"}}",
          "Timestamp" : "2015-01-21T12:56:54.396Z",
          "SignatureVersion" : "1",
          "Signature" : "u7u/zeRG5KC3K7CpXllfwbrAGMqcHsp/01d3JAwxwPZYuOU22222222Gn9rZZTTxmy2vrIyioEpIXtCaT6MhjUum2erYBi0Doo8K03+IGtqsjQZeEwtr6cWDDJyrxoX53a18fp9YqBNTzwvu9TOkTqbumw9fH1+ltQ3qeDRP1DrpkJczQ080cZPmkF2xeDLlt5aIDlZJJkWpvivIrt9ZBS/lW4HU0UpjvHVAZhxgZyUoWuRMOM7j3q3aRh/RB9aHOOAw8wdfg5ie8vHSbcEOVIfOziENBEgYCctpVMgZDjdj2fakGfUM3kCVrgm983AjJt2SA==",
          "SigningCertURL" : "https://sns.eu-west-1.amazonaws.com/SimpleNotificationService-d6d679a1d2222222222e198.pem",
          "UnsubscribeURL" : "https://sns.eu-west-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:eu-west-1:863132222282:yourapp_email_complaints:0222222c-d183-45d8-b0ac-787222222084"
        }
      }
    end

    it 'creates a new complaint record' do
      expect {
        VCR.use_cassette 'check_sns_message' do
          post :complaint, complaint_json, format: :json
        end
      }.to change(EmailResponse, :count).by(1)
    end

    it 'saves the information it should' do
      allow(EmailResponse).to receive(:create)
      expect(EmailResponse).to receive(:create).with(hash_including(email: '[email protected]', response_type: 'complaint', extra_info: "complaintFeedbackType: abuse"))
      VCR.use_cassette 'check_sns_message' do
        post :complaint, complaint_json, format: :json
      end
    end

    it 'fails if it cannot verify the message' do
      expect {
        VCR.use_cassette 'check_sns_message' do
          post :complaint, invalid_complaint_json, format: :json
        end
      }.to change(EmailResponse, :count).by(0)
    end
  end
end

DB schema

# In a migration file in db/migrations. Use rails g migration to make one

class CreateEmailResponses < ActiveRecord::Migration
  def change
    create_table :email_responses do |t|
      t.string :email
      t.text :extra_info
      t.integer :response_type

      t.timestamps null: false
    end
  end
end

Model

# In app/models/email_response.rb

class EmailResponse < ActiveRecord::Base
  enum response_type: [ :bounce, :complaint ]

  validates_presence_of :email
end

Controller

# In app/controllers/email_responses_controller.rb

require 'json'

class EmailResponsesController < ApplicationController

  skip_authorization_check
  skip_before_action :verify_authenticity_token

  before_action :log_incoming_message

  def bounce
    return render json: {} unless aws_message.authentic?

    if type != 'Bounce'
      Rails.logger.info "Not a bounce - exiting"
      return render json: {}
    end

    bounce = message['bounce']
    bouncerecps = bounce['bouncedRecipients']
    bouncerecps.each do |recp|
      email = recp['emailAddress']
      extra_info  = "status: #{recp['status']}, action: #{recp['action']}, diagnosticCode: #{recp['diagnosticCode']}"
      Rails.logger.info "Creating a bounce record for #{email}"

      EmailResponse.create ({ email: email, response_type: 'bounce', extra_info: extra_info})
    end

    render json: {}
  end

  def complaint
    return render json: {} unless aws_message.authentic?

    if type != 'Complaint'
      Rails.logger.info "Not a complaint - exiting"
      return render json: {}
    end

    complaint = message['complaint']
    recipients = complaint['complainedRecipients']
    recipients.each do |recp|
      email = recp['emailAddress']
      extra_info = "complaintFeedbackType: #{complaint['complaintFeedbackType']}"
      EmailResponse.create ( { email: email, response_type: 'complaint', extra_info: extra_info } )
    end

    render json: {}
  end

  protected

  def aws_message
    @aws_message ||= AWS::SNS::Message.new request.raw_post
  end

  def log_incoming_message
    Rails.logger.info request.raw_post
  end

  # Weirdly, AWS double encodes the JSON.
  def message
    @message ||= JSON.parse JSON.parse(request.raw_post)['Message']
  end

  def type
    message['notificationType']
  end
end

Routes

# In config/routes.rb

post 'email_responses/bounce' => 'email_responses#bounce'
post 'email_responses/complaint' => 'email_responses#complaint'

Mailer intercept

# In config/initializers/invalid_email_interceptor.rb

class BouncedEmailInterceptor
  def self.delivering_email(message)
    if EmailResponse.exists? email: message.to
      message.perform_deliveries = false
    end
  end
end

ActionMailer::Base.register_interceptor(BouncedEmailInterceptor)

You may be tempted to just use a column in the user table for marking the email address as invalid or not, but you may end up having more than one bounce/complaint per email address, and the information might be very useful in knowing whether you want to unblock the address (server may be down, mailbox full, etc). You may also have multiple addresses per user over time, so a separate table is better.

Step 6: Add the HTTPS endpoint to your SNS topic

This is where you actually get the SNS topic to ping you app when a bounce happens.

Go back to the SNS section of the AWS console, and click on the bounces topic you created and then click the Create subscription button. Choose https and enter https://www.yourdomain.com/email_responses/bounce. Do the same for the complaints topic, altering the URL as appropriate.

Now repeat whatever action in your app will send an email to the bounce test address. You should find that a new record is created in the database. Repeat the action, and the logs should show that no email was sent this time.

Then do the same test for the complaints test address.

Once you're happy this works, remove the subscription that copies the emails to your personal email for each of the SNS topics, and you're all done.

Step 7: Ask AWS to enable production SES access.

Now that your app can handle bounces and complaints, you are OK to ask AWS to take you out of the sandbox and enable actual emails to be sent from your app. There are instructions on how to do this here.

All done!

That's it. You should be able to send transactional emails cheaply and easily now using the AWS infrastructure. Let's hope Amazon keep dropping their prices like they have done in the past!

1 2

Mountain Road Software