Using AWS Lambda for free Mongo backups

DevOps CSC Blogs

Welcome to Just DevOps, a blog for exploring the intersection of development and operations.

For my first post, I want to discuss a recent project my team has been working on to demonstrate CSC’s DevOps capabilities.

I wrote a small javascript Web application using the Meteor framework, MongoDB, AngularJS and Google’s Material Design visual language. When code was checked in to the Master branch of our git repository, the application was automatically tested, built, containerised and deployed across cloud providers.

We decided to showcase these capabilities in the real world by releasing the app to a small boutique gym here in Sydney and using our own Continuous Integration and Continuous Deployment capabilities to update the app automatically. That is, a code push caused a new version of the application to be rolled out across all production instances. There’s nothing new here; this is the way most startups work today.

Now we have about 100 users relying on the application to track and store their workout data, and we are continuously pushing updates out. The engagement and feedback have been fantastic.

But what about the data???

In any scenario, but particularly one in which people trust you with their data, you need to have a solid backup plan in place. Backing up Mongo is simple enough — Mongo has a dump utility that will back up all your data from the command line. But what do you do with that data, and how do you ensure that it’s being reliably backed up?

Amazon’s Simple Storage Service (S3) is a very cheap object store that is reliable and easy to configure, so it was perfect for this purpose. But how do you call the Mongo instance to dump its data to S3? Luckily I wasn’t the first person to ask this question and was able to use a great node library.

I briefly considered creating a CRON job to run on my laptop and dump the data to S3, but that’s not reliable and certainly not an enterprise-grade solution, as there is no guarantee that my laptop will be running reliably.

Next I thought about firing up an AWS EC2 instance in order to set up a CRON job to run there, but having an entire instance running 24×7 for one CRON job seemed like overkill to me.

Enter Amazon Web Service’s Lambda — Amazon’s foray into serverless code. In its simplest form, Lambda allows you to run code without any specific servers. Amazon calls each instance of this a Function. Functions can be triggered by a number of events, or configured to run on a CRON schedule. Lambda also supports a variety of languages, so chances are you can achieve most simple tasks there. But the best part is that (at the time of writing…) the first million Lambda calls per month are free.

I was lucky enough to find a Node library that someone else had written to back Mongo to S3, so all I needed to do was compile it on the same version of Node and type of Linux that the AWS zone I wanted to use was running, and voila!

The Mongo DB instance we are using to track user workout data is now being backed up to an S3 Bucket every eight hours, via a Lambda function. That means, not only is the data backed up, but our client doesn’t need to worry about servers or CRON jobs.

The best part of all is the cost. The average backup is only a couple of hundred kilobytes in size, and the ~93 Lambda Function calls are way below the million monthly call threshold, keeping us on the AWS Free Tier for at least a year.

Can’t beat free!

Find this interesting? Got any questions? Hit me up in the comments section below.

RELATED LINKS

Creating a virtual service desk agent – a DevOps experiment

Puppet: One of the best cloud DevOps tools

Does your enterprise have the DevOps skills for agile mobile development?

Comments

  1. great use of lambda!

    Like

  2. davidmadams says:

    Would you mind posting your Lambda function code (obfuscate credentials of course) for those of us who would like to implement this? Thanks in advance.

    Like

  3. Hey David – sorry mate I was out on holiday. My code is below, but you will need to compile your binaries to work on the linux instance used for the AWS Lambda region you want to run the function in, and upload as a zip file.

    exports.handler = function (event, context) {
    var mongoBackup = require(‘mongo-backup-to-s3’);
    console.log(‘creating config’);
    var config = {
    mongodb:{
    url: ‘mongodb://:@’
    },
    s3:{
    bucket:”,
    folder:”,
    key: ”,
    secret: ”
    }
    };
    mongoBackup.dumpToS3(config);
    context.succeed();
    }

    Like

  4. Wow. That is so elegant and logical and clearly explained. Brilliantly goes through what could be a complex process and makes it obvious.

    Like

  5. Nice Article. How it help to developer in terms of balance the day to day life.

    Like

  6. Thank’s it was very helpful .. i just have one doubt : How can i restore my database from DMP file ?

    Like

  7. Nice Blog! We have read a few of the interesting content on your website now, and we really like your style. Thanks a million and please keep it up the effective work.
    Digital Media Recruitment

    Like

  8. Alon Harel says:

    Nice, But the lib of mongo-backup-to-s3 is DEPRECATED!
    I created new lambda function that use the original mongodump util. If you want to use you can look at my github , https://github.com/alonhar/lambda-mongodb-s3-backup.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: