May 10, 2020

Getting mail event data with Mailgun webhooks

This post explains how to use Mailgun webhooks to get mail event data for things like tracking deliveries, opens and clicks. The post originally appeared here, but has been updated to reflect changes to Mailgun.

Tracking mail events like opens, clicks, failures and complaints can be done on Mailgun using Mailgun webhooks. Although this data is available in the Mailgun account dashboard, there are many reasons using the webhooks may come in handy. Because the webhook is a realtime push to you, you can collect the information to do things like visualisation, reporting or presentation of the mail data in any form or through any channel you want. There is also an API to get the events but using the webhooks has more advantages.

  1. You don’t have to make repeated API requests. This means lesser resources.
  2. Updates are more realtime because it is pushed as soon as available on the server.

Setting it up

There are two ways to set up webhooks in Mailgun. It can either be done through the Mailgun dashboard or API. The more straightforward way to do it is through the dashboard. Once logged into your dashboard, a Webhooks link is available on the navigation bar.

On the webhooks page, you can click the Add webhook button to set the URL the event data for the different events can be sent to.

Handling the data

To handle the event data sent to our webhook URL, we must know what the data looks like in the first place. Parameters sent via POST are available in the API documentation. We can go a step further and confirm this by using a test webhook URL that will log the data from Mailgun. We can use Mailgun’s Postbin or Webhook.site. These services will generate a unique endpoint we can use in the Mailgun dashboard to get sample event data. We will be using webhook.site for this post.

Let’s go ahead to create a test endpoint and see what the event data look like compared to what is in the documentation.

  • Visit webhook.site
  • Copy the generated URL and access the Webhooks section of your Mailgun dashboard.
  • Paste the URL in the URL to test input field of the Test webhook section and click the Test Webhook button. This will send sample event data to the URL.
  • Repeat this for all the events you are interested in. Ignore the ones labeled Legacy.

Now that we know what the parameters are for each event types and the content type they may come in, it is easy to write the code that will handle the sent data. Here is a simple code that will output details of complaints and dropped emails.

const express = require(‘express’)
    , bodyParser = require(‘body-parser’)
    , multer = require(‘multer’)
    ;

const app = express();
app.use(bodyParser.urlencoded({extended: false}));
app.listen(process.env.PORT || 3000);

app.post(‘/webhook’, multer().none(), function(req, res) {
  const email = req.body.recipient;
  const event = req.body.event;

  if (event == ‘complained’) {
    console.log(`${email} complained about your mail`);
  }
  else if (event == ‘dropped’) {
    console.log(`Mail to ${email} dropped. ${event.description}`);
  }
  else if (event == ‘bounced’) {
    console.log(`Error ${event.code}: Mail to ${email} bounced. ${event.error}`);
  }

  res.end();
});

Making it secure

Nothing stops anyone that knows our webhook URL from crafting a false event data and sending it to the URL. Luckily, Mailgun signs each request sent and posts the following parameters along:

  • timestamp (Number of seconds passed since January 1, 1970)
  • token (Randomly generated string with length 50)
  • signature (Hexadecimal string generated by HMAC algorithm)

To verify the token;

  • Concatenate the values of timestamp and token.
  • Encode the resulting string with HMAC, using your Mailgun API key as the key and Sha256 as the algorithm.
  • The result should be the same as the signature.

Here is what it looks like in Node.js:

const value = event_data_timestamp+event_data_token;
      const hash = crypto.createHmac(‘sha256’, apikey)
                        .update(value)
                        .digest(‘hex’);
      if (hash !== event_data_signature) {
        console.log(‘Invalid signature’);
        return;
      }

If we add that to our original code example, we will have something like this:

const express = require(‘express’)
    , crypto = require(‘crypto’)
    , multer = require(‘multer’)
    , bodyParser = require(‘body-parser’)
    ;

const app = express();
app.use(bodyParser.urlencoded({extended: false}));
app.listen(process.env.PORT || 3000);

app.get(‘/webhook’, multer().none(), function(req, res) {
  // Validate signature
  const value = req.body.timestamp+req.body.token;
  const hash = crypto.createHmac(‘sha256’,
              process.env.API_KEY)
                  .update(value)
                  .digest(‘hex’);
  if (hash !== req.body.signature) {
    console.log(‘Invalid signature’);
    return res.end();
  }

  // Log status of event
  const email = req.body.recipient;
  const event = req.body.event;

  if (event == ‘complained’) {
    console.log(`${email} complained about your mail`);
  }
  else if (event == ‘dropped’) {
    console.log(`Mail to ${email} dropped. ${event.description}`);
  }
  else if (event == ‘bounced’) {
    console.log(`Error ${event.code}: Mail to ${email} bounced. ${event.error}`);
  }

  res.end();
});

We can even step this up and:

  1. For every request, check against a token cache to prevent use of the same token. Every token will be stored there. This will prevent replay attacks.
  2. Check if the timestamp is not too far from the current time.

Making it scalable

If you send lots of emails and you are expecting lots of events, putting your webhook script on a server that can’t scale automatically is a bad idea. Even if you are not expecting lots of events, unexpected things can lead to a surge in events. Having a server that can scale automatically is really useful for instances like this.

Enter Serverless computing. In simple terms, the idea is that you can delegate the execution of your code and everything related to another service. Because multiple instances of your code can be executed in parallel and you can adjust computing resources like RAM and execution time on the fly, it is highly scalable. You are also charged based on consumed resources and execution time so it can be really cheap.

There are a couple of serverless computing providers. One I use and recommend is Google Cloud Functions because of the ease of setting up HTTP functions. An HTTP function is a code block wrapped as a function that can be triggered by visiting a URL. This is exactly what we need as our webhook.

To create this function, we need to write a JavaScript function that will be exported as a Node.js module. The function takes HTTP-specific arguments: `request` and `response`.

exports.webhook = function(request, response) {
  // Handle event data here
  response.send({status:”ok”});
}

Based on the request content-type, the body of the request is automatically passed and available in the body parameter of the request object.

exports.webhook = function(request, response) {
  let event = request.body.event; // delivered
  // Handle event data here
  // …
  response.send({status:”ok”});
}

Next, we can publish the function to Cloud Functions. An easy way to do this is to do it from the Cloud Functions dashboard.

  • Go to your Google Cloud Console (if you don’t have an account yet, create one).
  • Enable Cloud Functions in the dashboard.
  • Click on Create Function.
  • Enter a name for your function (e.g mailgun-webhook).
  • In the trigger section, select HTTP trigger. Note the URL, that will be your webhook URL.
  • Copy your event data handling code to the index.js section of the Cloud function.
  • Copy the content of your package.json and paste in the package.json section.
  • Select or create a Stage bucket. The stage bucket is simply where the code is staged. You can use anything here.
  • In Function to execute, enter the name of your function (e.g webhook).
  • Save.

You can now use the function’s URL in Mailgun as your webhook URL.

Conclusion

Working with Mailgun’s webhooks is easy. There are many ways the event data can be used to enrich your applications outside Mailgun. If for example you allow your users send emails from your web for any reason and you use Mailgun, you can use this to provide analytics for them. Or maybe you want to send your email analytics to another platform. Or maybe you want to be notified of failures in your Slack account. Or maybe not even that. Maybe you just want more detailed analytics than what is available on the Mailgun dashboard. Whatever the use case is, the event data is available for you.

If you are interested in a more detailed analytics, reporting and business intelligence for your transactional emails through Mailgun, without setting all these yourself, create a free Mailintel account.

Follow the conversation: @mailintel · RSS