Database backup to Amazon S3

Posted on Wednesday, January 30th, 2013 in site by Ian


A while ago we posted a simple database dump and FTP backup script we use to backup all the site databases daily. Since jet lag is setting in pretty bad I added an extra option to upload the script to Amazon’s S3 storage system at 4:30 this morning.

The latest version will always be on the wiki.

You need a command line app to get the database dumps into S3. For our Debian server I installed s3cmd with the usual apt-get install s3cmd.

Next setup s3cmd with s3cmd –configure to create a profile. You’ll need the Access and Secret keys from your Amazon account (under security credentials). We made a new set just for backup. I didn’t bother setting the PGP or HTTPS stuff for now. S3cmd documentation is here.

Do a s3cmd ls to list all buckets and make sure the connection is working. If you haven’t already, create a bucket for your backups using the S3 web interface or the s3cmd utility as described in the documentation.

s3cmd put back_$i-$day-$ s3://dpdbbackup/back_$i-$day-$

To upload to S3 just add a single line to the old backup script processing loop. Replace dpdbbackup with your bucket name (the version on the wiki now uses a variable to set this).

Run the script with ./ (or whatever you named it) to test.

This entry was posted on Wednesday, January 30th, 2013 at 8:15 am and is filed under site. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.

One Response to “Database backup to Amazon S3”

  1. James says:

    I have managed a number of websites on EC2 instances, the EBS snapshot feature makes keeping full system incremental backups a doddle, my setup keeps one backup for each of the preceding 7 days, and one 1 backup for each of the preceding 12 months.

    Restoring a backup is as simple as creating a new EBS volume from the appropriate snapshot and mounting it.

    One of the sites I don’t mind if it goes down for an hour now and then so I run it on a spot priced instance for a ridiculously small amount of money, in the fairly rare occurance that it gets outbid and shuts down I get an email, and then when the price reduces amazon starts it up again.

    I also have the sites setup so I can quickly load balance in case of very high traffic, just spin up a “slave” instance or three and they register with the amazon load balancer.

    EC2 (and the other AWS services) are simply brilliant, so convenient, often I’ve spun up an instance for an hour or two for experimenting on something, or even spun up 50 or more systems to do some big job in parallel. Nothing really compares.

Leave a Reply

Notify me of followup comments via e-mail. You can also subscribe without commenting.

Recent Comments

  • KH: Well, I'm sure the HDD can handle a little vibration *cough*. What I am more interested in is this: the fan appears to be placed...
  • Alan: The Si5351 runs on 3.3V, and several FPGA dev boards now include "Arduino compatible" headers [if you stick to 3.3V logic levels]. Maybe it's time...
  • hli: Sunday++
  • JB: Hi
  • Peter: Sunday