Services like SendGrid, Mailgun, or Mandrill are a great way to handle outbound email from a web application, and each will certainly allow you to transmit email directly to their mail servers. However, any kind of outage in between your app server and your mail transfer service could easily result in mail getting routed to /dev/null with no easy way to identify and resend the lost mail. This is particularly problematic for emails sent from a worker process since they will often keep right on churning away even if other things are down. For some applications, it may be a perfectly acceptable risk to drop a few emails, as hopefully such service interuptions are rare, but losing emails would be a major problem for the project I’m working on and I suspect many others.

To mitigate this, I’m configuring Postfix on each app server to relay out to Mandrill. You can do the same for SendGrid or Mailgun, and while this post is going to be mostly Mandrill-specific, since that’s what I’m using, the equivalent for SendGrid/Mailgun ought to be pretty close as all three services use SASL authentication for relay configurations.

This is a decent solution because Postfix, being a full mail transfer agent in its own right, has useful things like automatic retry logic built-in. Rather than reimplement all that in my app server, no doubt in a much less reliable way, I’m going to set up Postfix running on each app server’s localhost. By doing it that way, I can leverage the fact that it’s hard to screw up communication over loopback and just outsource my email reliability problem to software that’s been around for just shy of two decades. Postfix uses very little in the way of resources, so it’s not a problem to run it on each app server.

The Opscode Postfix cookbook is the main tool we need. I’m using OpsWorks, but this cookbook isn’t specific to OpsWorks and should work in any Chef setup.

The custom Chef JSON for a Postfix null client relay looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

{
  "postfix": {
    "mail_type": "master",
    "main": {
      "relayhost": "smtp.mandrillapp.com",
      "myhostname": "myapp.com",
      "mydomain": "myapp.com",
      "mynetworks": [
        "127.0.0.0/8",
        "[::1]/128",
        "[fe80::]/10"
      ],
      "mydestination": "",
      "inet_interfaces": "loopback-only",
      "smtp_use_tls": "yes",
      "smtp_tls_CAfile": "/etc/ssl/certs/ca-certificates.crt",
      "smtpd_tls_CAfile": "/etc/ssl/certs/ca-certificates.crt",
      "smtp_tls_security_level": "encrypt",
      "smtp_sasl_auth_enable": "yes"
    },
    "sasl": {
      "smtp_sasl_user_name": "[redacted]",
      "smtp_sasl_passwd": "[redacted]"
    }
  }
}

The main attributes in the JSON config that we care about are the values for smtp_tls_security_level and smtp_sasl_auth_enable. The yes value for smtp_sasl_auth_enable switches on SASL authentication, which is how Mandrill knows which account is sending mail. Setting the value to encrypt for smtp_tls_security_level forces TLS for everything. We ought to just care about encrypting outbound mail since this is a null client, but, especially in the world in which BCP 188 was necessary…

Since we’re using SASL for authentication, postfix::sasl_auth will be the recipe we want to use. For OpsWorks, this can go into your ‘Setup’ lifecycle event.

I’m using Ubuntu on EC2. The Postfix cookbook doesn’t do a great job of locating the CA certificate bundle for that platform, so you may need to specify the correct location for that or alternatively download a bundle and place it the fallback location of /etc/postfix/cacert.pem. You can test that it’s working via telnet.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

$ telnet localhost 25
Trying 127.0.0.1...
Connected to myhostname.localdomain.
Escape character is '^]'.
220 myapp.com ESMTP Postfix
EHLO test.com
250-myapp.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN
STARTTLS
220 2.0.0 Ready to start TLS
QUIT

If you send STARTTLS and in response get a message like 454 4.7.0 TLS not available due to local problem, that usually means Postfix couldn’t find your CA bundle.

Finally, verify that relaying works with sendmail.

1
2
3
4
5

$ sendmail your.email@myapp.com
From: admin@myapp.com
Subject: Testing from Postfix
This is a test email
.