gpg backups
This commit is contained in:
Danny Berger
@@ -2,3 +2,4 @@ safe: true
|
||||
auto: true
|
||||
server: true
|
||||
pygments: true
|
||||
markdown: redcarpet
|
||||
|
||||
173
blog/_posts/2013-02-08-automating-backups-to-the-cloud.md
Normal file
173
blog/_posts/2013-02-08-automating-backups-to-the-cloud.md
Normal file
@@ -0,0 +1,173 @@
|
||||
---
|
||||
title: Automating Backups to the Cloud
|
||||
layout: post
|
||||
tags: backup gpg s3
|
||||
description: Combining gpg, Amazon S3 and IAM policies.
|
||||
---
|
||||
|
||||
Backups are extremely important and I've been experimenting with a few different methods. My concerns are always focused
|
||||
on maintaining data integrity, security, and availability. One of my current methods involves using asymmetric keys for
|
||||
secure storage and object versioning to ensure backup data can't undesirably be overwritten.
|
||||
|
||||
|
||||
### Encryption Keys
|
||||
|
||||
For encryption and decryption I'm using asymmetric keys via [`gpg`][1]. This way, any server can generate and encrypt
|
||||
the data, but only administrators who have the private key could actually decrypt the data. Generating the
|
||||
administrative key looks like:
|
||||
|
||||
{% highlight console %}{% raw %}
|
||||
$ gpg --gen-key
|
||||
gpg (GnuPG) 1.4.11; Copyright (C) 2010 Free Software Foundation, Inc.
|
||||
This is free software: you are free to change and redistribute it.
|
||||
There is NO WARRANTY, to the extent permitted by law.
|
||||
|
||||
... [snip] ...
|
||||
|
||||
gpg: key CEFAF45B marked as ultimately trusted
|
||||
public and secret key created and signed.
|
||||
|
||||
gpg: checking the trustdb
|
||||
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
|
||||
gpg: depth: 0 valid: 4 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 4u
|
||||
pub 2048R/CEFAF45B 2013-02-08
|
||||
Key fingerprint = 46DF 2951 7E2D 41D7 F7B5 EB16 20C2 1C03 CEFA F45B
|
||||
uid Danny Berger (secret-project-backup) <dpb587@gmail.com>
|
||||
sub 2048R/765C4556 2013-02-08
|
||||
{% endraw %}{% endhighlight %}
|
||||
|
||||
To actually use the public key on servers, it can be exported and copied...
|
||||
|
||||
{% highlight console %}{% raw %}
|
||||
$ gpg --armor --export 'Danny Berger (secret-project-backup) <dpb587@gmail.com>'
|
||||
-----BEGIN PGP PUBLIC KEY BLOCK-----
|
||||
Version: GnuPG v1.4.11 (Darwin)
|
||||
... [snip] ...
|
||||
-----END PGP PUBLIC KEY BLOCK-----
|
||||
{% endraw %}{% endhighlight %}
|
||||
|
||||
Then pasted and imported on the machine(s) that will be encrypting data...
|
||||
|
||||
{% highlight console %}{% raw %}
|
||||
$ cat | gpg --import
|
||||
gpg: directory `/home/app-devtools/.gnupg' created
|
||||
gpg: new configuration file `/home/app-devtools/.gnupg/gpg.conf' created
|
||||
gpg: WARNING: options in `/home/app-devtools/.gnupg/gpg.conf' are not yet active during this run
|
||||
gpg: keyring `/home/app-devtools/.gnupg/secring.gpg' created
|
||||
gpg: keyring `/home/app-devtools/.gnupg/pubring.gpg' created
|
||||
... [snip] ... Ctrl-D ...
|
||||
gpg: /home/app-devtools/.gnupg/trustdb.gpg: trustdb created
|
||||
gpg: key CEFAF45B: public key "Danny Berger (secret-project-backup) <dpb587@gmail.com>" imported
|
||||
gpg: Total number processed: 1
|
||||
gpg: imported: 1 (RSA: 1)
|
||||
{% endraw %}{% endhighlight %}
|
||||
|
||||
And then marked as "ultimately trusted" with the `trust` command (otherwise it always wants to confirm before using the
|
||||
key)...
|
||||
|
||||
{% highlight console %}{% raw %}
|
||||
$ gpg --edit-key 'Danny Berger (secret-project-backup) <dpb587@gmail.com>'
|
||||
... [snip] ...
|
||||
|
||||
pub 2048R/CEFAF45B created: 2013-02-08 expires: never usage: SC
|
||||
trust: ultimate validity: unknown
|
||||
sub 2048R/765C4556 created: 2013-02-08 expires: never usage: E
|
||||
[ unknown] (1). Danny Berger (secret-project-backup) <dpb587@gmail.com>
|
||||
Please note that the shown key validity is not necessarily correct
|
||||
unless you restart the program.
|
||||
|
||||
Command> quit
|
||||
{% endraw %}{% endhighlight %}
|
||||
|
||||
|
||||
### Amazon S3
|
||||
|
||||
In my case, I wanted to regularly send the encrypted backups offsite and [S3][2] seemed like a flexible, effective
|
||||
storage place. This involved a couple steps:
|
||||
|
||||
**Create a new S3 bucket** (e.g. `backup.secret-project.example.com`) - this will just hold all the different backup
|
||||
types and files for the project.
|
||||
|
||||
**Enable Object Versioning** on the S3 bucket - whenever a new backup gets dropped off, previous backups will remain.
|
||||
This provides for additional security (e.g. a compromised server could not overwrite the backup with an empty file) and
|
||||
more complex retention policies than Amazon's Glacier lifecycle rules.
|
||||
|
||||
**Create a new IAM user** (e.g. `com-example-secret-project-backup`) - the user and it's Access Key will be responsible
|
||||
for uploading the backup files to the bucket.
|
||||
|
||||
**Add a User Policy** to the IAM user - the only permission it needs is `PutObject` for the bucket:
|
||||
|
||||
{% highlight javascript %}
|
||||
{
|
||||
"Statement" : [
|
||||
{
|
||||
"Sid" : "Stmt0123456789",
|
||||
"Action" : [
|
||||
"s3:PutObject"
|
||||
],
|
||||
"Effect" : "Allow",
|
||||
"Resource" : [
|
||||
"arn:aws:s3:::backup.secret-project.example.com/*"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
**Upload Method** - instead of depending on third-party libraries for uploading the backup files, I wanted to try simply
|
||||
using `curl` with Amazon S3's Browser-Based Upload functionality. This involved creating and signing the appropriate
|
||||
policy via the [sample][3] policy builder for a particular backup type. My simple policy looked like:
|
||||
|
||||
{% highlight javascript %}
|
||||
{
|
||||
"expiration" : "2016-01-01T12:00:00.000Z",
|
||||
"conditions" : [
|
||||
{ "bucket" : "backup.secret-project.example.com" },
|
||||
{ "acl" : "private" },
|
||||
{ "key" : "database.sql.gz.enc" },
|
||||
]
|
||||
}
|
||||
{% endhighlight %}
|
||||
|
||||
|
||||
### All Together
|
||||
|
||||
Putting everything together, a single command could be used to backup the database, compress, encrypt, and upload:
|
||||
|
||||
{% highlight console %}
|
||||
$ mysqldump ... \
|
||||
| gzip -c \
|
||||
| gpg --recipient 'Danny Berger (secret-project-backup) <dpb587@gmail.com>' --encrypt \
|
||||
| curl \
|
||||
-F key=database.sql.gz.enc \
|
||||
-F acl=private \
|
||||
-F AWSAccessKeyId=AKIA99076E3F28E55AF85 \
|
||||
-F policy=ewogICJleHBpcmF0aW9uIiA6ICIyMDE2LTAxLTAxVDEyOjAwOjAwLjAwMFoiLAogICJjb25kaXRpb25zIiA6IFsKICAgIHsgImJ1Y2tldCIgOiAiYmFja3VwLnNlY3JldC1wcm9qZWN0LmV4YW1wbGUuY29tIiB9LAogICAgeyAiYWNsIiA6ICJwcml2YXRlIiB9LAogICAgeyAia2V5IiA6ICJkYXRhYmFzZS5zcWwuZ3ouZW5jIiB9LAogIF0KfQ== \
|
||||
-F signature=937ca778e4d44db7b804cfdd70d= \
|
||||
-F file=@- \
|
||||
https://s3.amazonaws.com/backup.secret-project.example.com
|
||||
{% endhighlight %}
|
||||
|
||||
And then to download, decrypt, decompress, and reload the database from an administrative machine:
|
||||
|
||||
{% highlight console %}
|
||||
$ wget -qO- 'https://s3.amazonaws.com/backup.secret-project.example.com/database.sql.tgz.enc?versionId=c0b55912f42c4142bcb44c3eb1376f35&AWSAccessKeyId=AKIA99076E3F28E55AF85&...' \
|
||||
| gpg -d \
|
||||
| gunzip \
|
||||
| mysql ...
|
||||
{% endhighlight %}
|
||||
|
||||
The only task remaining is creating a cleanup script using the S3 API to monitor the different backup versions and
|
||||
delete them as they expire.
|
||||
|
||||
|
||||
### Summary
|
||||
|
||||
While it has a bit of overhead to get things set up properly, using `gpg` makes secure backups trivial and S3 provides
|
||||
the flexible storage strategy to ensure data is safe.
|
||||
|
||||
|
||||
[1]: http://www.gnupg.org/
|
||||
[2]: http://aws.amazon.com/s3/
|
||||
[3]: http://s3.amazonaws.com/doc/s3-example-code/post/post_sample.html
|
||||
Reference in New Issue
Block a user