Introduction

duplicity supports backups to AWS S3. There are a lot of howtos out there but most assume that you want to backup to the (default) US region. It gets more complicated for other regions and your mostly left on your own with strange errors from boto, the library that talks to AWS. Below I document how to use duplicity to do GPG encrypted backups to S3 in the EU (Ireland) (eu-west-1) region.

Setting up S3 and IAM

We'll create an S3-bucket and an IAM user with full access to that bucket. Make sure to use EU (Ireland) region so that the backup-script below works.

1. Go to the S3 console and create a new bucket in region EU (Ireland). Do NOT use dots in the bucket-name (#2836). Let's say we use MY-S3-BUCKET-NAME.
2. Next go to Identity and Access Management (IAM), create a new user with programmatic access and write down the access key and secret access key.
3. Still in IAM create a new policy with the content below and attach it to the user. Make sure to replace <MY-S3-BUCKET-NAME> with the actual bucket name.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::<MY-S3-BUCKET-NAME>",
                "arn:aws:s3:::<MY-S3-BUCKET-NAME>/*"
            ]
        }
    ]
}

Server Setup

Assuming Ubuntu we'll install the latest duplicity version from the ppa:

root@server:~# add-apt-repository ppa:duplicity-team/ppa
root@server:~# apt clean && apt update && apt install -y duplicity python-boto

Next we'll need a gpg key to encrypt the backup stored in the cloud:

user@server:~# sudo -i # we'll run the backup as root. so make the gpg key using user root too
root@server:~# gpg --gen-key # use kind "RSA and RSA (default)", name "duplicity" and set a passphrase
root@server:~# gpg --list-keys
/root/.gnupg/pubring.gpg
------------------------
pub   4096R/<The-GPG-Key-ID> 2017-12-05
uid                  duplicity (...) <....>
sub   4096R/....... 2017-12-05
# export the gpg keys and keep them at a secure location:
root@server:~# gpg -a --export duplicity
root@server:~# gpg -a --export-secret-key duplicity

AGAIN: Save the gpg key at a secure location and not only on the server. If the server goes down you will not be able to restore your backups without that key!

Backup Script

On the server we'll create two files: /root/.duplicity_secrets with the S3 access keys and /root/backup_to_s3.sh which will be the backup script.

/root/.duplicity_secrets

This file will hold the S3 access keys and the passphrase to the gpg key:

export PASSPHRASE="<THE-PASSPHRASE-TO-YOUR-GPG-KEY>"
export AWS_ACCESS_KEY_ID="<YOUR-IAM-USERS-ACCESS-KEY>"
export AWS_SECRET_ACCESS_KEY="<YOUR-IAM-USERS-SECRET-ACCESS-KEY>"

/root/backup_to_s3.sh

The backup script will source the .duplicity_secrets file and then backup to S3. Below is an example how to backup /etc into a subdirectory server-name/etc in the S3-bucket. Make sure to configure <The-GPG-Key-ID> and <MY-S3-BUCKET-NAME> in the following script:

source /root/.duplicity_secrets

# duplicity args
# 
s3_bucket="s3://s3.eu-west-1.amazonaws.com/<MY-S3-BUCKET-NAME>/${HOSTNAME}"
encrypt_key="<The-GPG-Key-ID>"
aws_args="--s3-european-buckets --s3-use-new-style"
# experimental setting that may speed up the backup: --asynchronous-upload

# backup /etc to <MY-S3-BUCKET-NAME>/hostname/etc
/usr/bin/duplicity $aws_args --encrypt-key $encrypt_key -v 4 \
    incr --full-if-older-than 14D \
    /etc ${s3_bucket}/etc

unset PASSPHRASE
unset AWS_ACCESS_KEY_ID
unset AWS_SECRET_ACCESS_KEY

Some Duplicity commands

Checkout: https://help.ubuntu.com/community/DuplicityBackupHowto

Some examples:

root@server:~# source /root/.duplicity_secrets 
root@server:~# export S3_BUCKET="s3://s3.eu-west-1.amazonaws.com/<MY-S3-BUCKET-NAME>/<SET-YOUR-SERVER-HOSTNAME-HERE>"
# list files in the backup
root@server:~# duplicity list-current-files $S3_BUCKET/etc
# restore /etc/fstab to /tmp/restored_fstab
root@server:~# duplicity --file-to-restore fstab $S3_BUCKET/etc /tmp/restored_fstab 

Troubleshooting

We're connecting duplicity directly to s3.eu-west-1.amazonaws.com. This is EU (Ireland) region and will therefore only work if you setup the bucket in the correct region. If you're using another region you may have to change the URL and/or play with $aws_args in the backup script. If you're getting ACCESS DENIED errors make sure to check the IAM policy and that it is attached to the user.

Credits

A lot of this post is based on Igor Cicimov's Duplicity encrypted backups to Amazon S3 post. I followed the steps described there but ran into S3 upload problems with the URL-schema used there. That's why I'm using eu-west-1 and directly connect to that endpoint.