Simon J.K. Pedersen - Azure & Tech

In this blog post I will describe how you can leverage SendGrids WebHooks to send email messages to an azure queue. You might ask, why do I need that? Well, I can’t answer that, but my use case was to parse an email I recieve on daily basis which contains a link to a zip file containing updates from the Danish CVR Register (Company Database). Every day I want to download this file to keep track of changes, and unfortunately the only way they distribute updates are via a link in an email, so I had to find a way to automate the download.

Here’s the flow I devised for it.

Obviously, this setup is prepared to handle a lot more than just a single email a day. In fact it’s pretty ridiculous to use a queue for my use case I could just have done everything in the WebAPI endpoint, but I wanted to build a framework that can scale to thoundsands of emails per day, and for that purpose queues are the way to go.

SendGrid Setup

First you need to get yourself af sendgrid account, so head over to https://sendgrid.com and sign-up for a free account. When you are signed in click the “Developer” menu item in the top menu, and hit “Parse Incomming Mails” in the right menu. That will take you to the following screen

Here you will need to add the hostname you want to recieve mails on, and the Url you want sendgrid to post the content of the mail to.

To create the hostname you need a domain name where you are able to configure a MX record to point to sendgrids servers. For my test purpose I created a subdomain mx.sjkp.dk, and configured the MX record the following way.

Remember to set the TTL to a low value, unless you don’t mind waiting 48hours before you can test your setup.

Now when you email e.g. [email protected] the mail will be received by SendGrids servers and they will forward the email content to the Url of my WebAPI REST endpoint configured, in my case http://cvrapi.azurewebsites.net/parsecvremails/api/email

WebAPI Endpoint for Receiving Emails

The WebAPI endpoint we need is pretty simple, it needs to be able to receive a multipart/formdata HTTP POST, with the content specified in their Inbound Parse Webhook documentation.

I’m not gonna go into all details needed to build this, it can either be done directly in the ApiController, or a custom HttpParameterBinding can be created and connfigured for the purpose. I decided to go with the later approach as it is more clean, but the first works equally well.

The code for my complete solution can be found at my github repository https://github.com/sjkp/ParseCVREmails. If you download it, make sure that you add a connection string to a Azure storage account in web.config in the appsetting StorageAccount, in the format: DefaultEndpointsProtocol=https;AccountName=[YOUR_ACCOUNTNAME];AccountKey=[YOUR_KEY]. The same storage account must be added to the App.config of the ParseCVREmails.WebJob project.

The interesting classes are:

Lets take a quick peek at the controller Web API action
[csharp]
public async Task<HttpResponseMessage> Post([MultipartFormData(typeof(EmailData))] EmailData data)
{
await _queueManager.SendMessage(Map(data), await _queueManager.GetQueue("emails"));
return Request.CreateResponse(HttpStatusCode.OK);
}
[/csharp]

When SendGrid POST the multipart/formdata to our WebAPI endpoint this method gets called. Because the data parameter is attributed by [MultipartFormData] the special HttpParameterBinding implementation MultipartFormDataHttpParameterBinding gets used. This Parameter binding takes the formdata and inserts it into the strongly typed EmailData class that I have created to directly correspond to the contents of the email SendGrid posts to the Url. This way we have a nice strongly typed object that we can work with in our controller. The uncommented code block shows how the implementation could have been done without using HttpParameterBinding.

The EmailData object is mapped to another Lib.EmailData object before it’s serialized and added to the queue, but that is just an implementation detail, that I have choosen because I want to do something special with the attachments later on. Right now the attachments are not added to the queue, later I want to save them to BLOB storage and save the url to the location in BLOB storage to the queue.

The Azure WebJob (ParseCVREmails.WebJob) that is listening to new messages on the queue simply runs a regular expression on the email body text, to find the URL from which it then downloads the zip file and saves it in Azure BLOB storage.
[fsharp]
open Microsoft.Azure.WebJobs
open ParseCVREmails.Lib
open System.Configuration
open System.Text.RegularExpressions
open System.Runtime.InteropServices
open System.Net
open System.Net.Http
open System.IO
open System.Threading.Tasks

let findUrl text =
let r = new Regex("kan hentes her: (.*)?For at", RegexOptions.Singleline)
let m = r.Match(text)
m.Groups.[1].Value

//let Download ([<QueueTriggerAttribute(queueName="test")>] downloadUrl : string, [<Blob("downloaded-files/download", FileAccess.Write)>]output : System.IO.Stream) =

let Download ([<QueueTriggerAttribute(queueName="downloads")>] downloadUrl : string, binder : IBinder ) =
async {
let output = binder.Bind<Stream>(new BlobAttribute("downloaded-files/"+(downloadUrl.Split([|’/’|]) |> Seq.last),FileAccess.Write))
let handler = new HttpClientHandler()
handler.Credentials <- new NetworkCredential(ConfigurationManager.AppSettings.["CvrUserName"], ConfigurationManager.AppSettings.["CvrPassword"])
use client = new HttpClient(handler)
let! r = client.GetStreamAsync(downloadUrl) |> Async.AwaitTask
do! r.CopyToAsync(output) |> Async.AwaitIAsyncResult |> Async.Ignore
}

let ProcessMessage ([<QueueTriggerAttribute(queueName = "emails")>] message : EmailData, [<QueueAttribute(queueName="downloads")>] [<Out>] downloads : byref<string>) =
let url = findUrl message.Text
downloads <- url

[<EntryPoint>]
let main argv =
printfn "%A" argv
use jobHost = new JobHost()
jobHost.RunAndBlock()
0 // return an integer exit code
[/fsharp]

I hope this post gives some ideas on how SendGrid together with Azure Queues can be leveraged to create a robust solution for receiving emails.