Tuesday, April 19, 2016

UTM Zone WGS84 Latitude Longitude Bounding Box

We are using proj.net to calculate the UTM Easting/Northing values for a given pair of WGS84 Latitude/Longitude coordinates. But I couldn't find an easy way to determine the correct UTM Zone for a WGS84 coordinate pair. The Zone had to be provided for the library to calculate the Easting/Northing values. I wanted the library to decide the best UTM zone based on the lat/lng coordinates.

To assist with this, I created the following table containing the "bounding box" lat/lng values for each UTM Zone in North America.

Test Code and CSV file:
https://gist.github.com/aaronhoffman/699d3659e1f68991e85d05e5637e1c5a

The easiest way I found to determine these values was to use the centroid point for each UTM Zone, then add +/- 4 degrees Latitude (North/South) and +/- 3 degrees Longitude (East/West).

I have also provided some C# code to convert these into arrays, and then an example using a "point in polygon" function to determine the appropriate UTM Zone.



Hope this helps,
Aaron




Monday, March 14, 2016

Azure HTTP Web Server Logs to SQL

If you enable File System Web Server logs for your Azure App Service...

Settings > Diagnostics logs > Web server logging > File System





You'll start to see logs on the file system for your app service here:

(you can find your FTP host and credentials in the publish profile file)

/LogFiles/http/RawLogs




Alternatively, you can see these logs through the Kudu UI:

https://{yourappservicename}.scm.azurewebsites.net/DebugConsole/?shell=powershell



Instead of downloading these one and a time and parsing through them, you can use the following library to assist in loading them into a relational database.

You could even run this as a webjob within your app service.


Get the code here:
https://github.com/Stonefinch/AzureWebServerLogsToSql


**Update: This repo now includes a web project that references the console app as a WebJob.

You may also want to check out the LogParser tool by Microsoft https://blogs.msdn.microsoft.com/friis/2014/02/06/how-to-analyse-iis-logs-using-logparser-logparser-studio/


Hope this helps,
Aaron

Saturday, March 5, 2016

Store Amazon SES Delivery Notifications in DynamoDB using SNS and Lambdas

I spent way too much time configuring my Amazon AWS account to store Simple Email Service SES Delivery Notifications to a DynamoDB Table. I thought I'd put this guide together to hopefully help future developers.


This guide does not currently cover these topics, but I may include them in the future:
1. Setting up your Amazon AWS Account.
2. Setting up Amazon SES. (dkim, domain/email identities, etc.)
3. Integration with Amazon SES from your app.
4. SNS Topic creation/configuration.



Prerequisites:
1. Amazon AWS account has been created/configured.
2. Amazon SES has been created/configured.
3. Amazon SNS Topic has been created/configured.
4. Delivery Notifications for the Amazon SES Domain Identity have been configured to be sent to the SNS Topic.
5. Your app is integrated with Amazon SES and successfully sending email.



Step by Step Guide to Storing SES Delivery Notification messages into a DynamoDB Table:

So at this point, you're successfully integrated with Amazon Simple Email Service SES and sending email, and you can prove that the Delivery notifications are being sent to a Simple Notification Service SNS Topic....

NOTE: All these services (SES, SNS, Lambda, DynamoDB) should be in the same region. The links I include in this guide are for us-west-2 Oregon.

We'll be using AWS Lambdas to process the SNS message and store the value into a DynamoDB Table.



1. Create a DynamoDB Table. (NoSQL db, more info)

Start here: https://us-west-2.console.aws.amazon.com/dynamodb/home
 - Create a table named: "ses-delivery"
 - Primary key (partition key, hash key) "messageId", type string. (each SNS message has a unique UUID, that will be our partition key)
 - Note: Depending on your use case, you may want to use a different value as the partition key and/or set up a global secondary index over email address.


Note: it will have 5 read and write capacity units by default, you'll need to bump those up to whatever values are appropriate for your app. (I may add more info on capacity units here later... tbd)



2. Create a new Lambda function. (light weight compute, sub-second metering, more info)

Start here: https://us-west-2.console.aws.amazon.com/lambda/home
 - Create a new lambda function (click the button)
 - In the Select Blueprint section, search for SNS
 - Choose the "sns-message" template



 - on the next screen, choose the event source of SNS, and choose the correct topic.
 - hit the next button.
 - on the next screen, give your lambda a name (I go with "ses-delivery-sns-to-dynamo")
 - keep the runtime as node.js
 - leave the default code, for now
 - leave the handler value "index.handler:
 - choose a new role, choose the "Create new role" > "* basic execution role" (we'll add dynamodb access to the IAM role it creates in a couple steps)
 - leave memory of 128MB (more than enough)
 - leave timeout around 3-5 secs (and yes, that is seconds, the entire write/compute takes ~500ms)
 - hit NEXT
 - review everything, leave the radio "enable later" selected, hit CREATE function button.



3. Configure Lambda Role

Start here: https://console.aws.amazon.com/iam/home
 - click on "Roles" on the left hand side.
 - click on the role that was just created, most likely "lambda_basic_execution"
 - click on the Permissions tab, and click the "Attach Policy" button.
 - search for DynamoDB, and select "AmazonDynamoDBFullAccess"



4. Configure Lambda Function

You now have a brand new lambda function, let's set it up.

Start here: https://us-west-2.console.aws.amazon.com/lambda/home

 - By default there will be a single event source tied to the SNS topic - It will be deleted when we publish the first version of this lambda (I'll explain in a bit, and remind you again to verify below).


 - Notice there is a little arrow on the far left side. Hit it so the versions of the lambda expand.



 - Notice that we are on "$LATEST" this is the only version that you can modify the code, you'll publish a version to eventually use and tie to the actual SNS Topic. You'll see published versions listed here eventually.

 - Go here, and copy paste the code into your lambda: https://gist.github.com/aaronhoffman/c2b7ef532b7a28e99aae#file-sns-to-dynamodb-lambda-js

 - Go to the Actions dropdown and select "configure test event"



 - Choose any of the defaults, then overwrite it with this (that is an actual SES>SNS>Lambda message, some values obfuscated to protect the innocent)

- Choose Save. Then click the Test button.

- The output shows up below the code (there should be a nice little green check mark next to the word "success" -- you have no idea how long it took me to get here... and I don't want to say)



5. Publish and Configure Lambda Version

Start here: (where we left off in #4) https://us-west-2.console.aws.amazon.com/lambda/home?region=us-west-2#/functions/ses-delivery-sns-to-dynamo


 - Click on the Actions dropdown button and select "Publish new version"
 - A new version has been created, and the event source on $LATEST should now be deleted (please verify - this is your reminder.)
 - With your new version selected, choose the "event source" tab, click the "Add event source" button.




 - Event Source type: SNS
 - SNS topic (likely "ses-delivery")
 - Choose "Enable now"
 - Submit!


If successful, we should see logs right away. To verify, go to CloudWatch

Within CloudWatch choose the log group that matches your lambda name

You should see a list of some log streams. (Note: the logs for executions of  multiple SNS messages may be included in a single log stream.)

Look in these logs for errors... hopefully there aren't any.


Go back to your dynamodb table and scan for records:

https://us-west-2.console.aws.amazon.com/dynamodb/home?region=us-west-2#tables:selected=ses-delivery


And that's it, hope this helps!
Aaron