Handling backups with Trellis

trellis
1

I know this is one of those topics that comes up every couple of months but now we’re working more and more with Trellis at our agency, this is becoming more of a talking point. Previously, we worked with managed hosts who handled backups automatically for us and which always worked flawlessly. It also was reassuring for us (and our clients) to have a great backup policy in place given that we work with a number of corporates who need websites as a regulatory requirement.

Anyway… this is a contentious topic i.e. should things like backups be handled natively in Trellis, or is this more of a “figure out what works best for you” responsibility. For me, backups are just as important as security for software. Ultimately, no software is 100% secure or reliable and backups ensure we can mitigate these risks. Following this line of thinking, I personally believe it would be great to have backups handled natively within Trellis. There’s the pull request here: https://github.com/roots/trellis/pull/650 based on the Ansible role Stouts.backup, in turn based on Duplicity. This seems like a pretty solid approach as it supports incremental backups and backups to S3, SFTP, local etc. I’m not sure how ease restores are.

Other approaches I see are:

  • Use an external service: I’ve tried Deploy-Tantra which works well but these services always have their drawbacks.
  • Use VPS provider server-level backups: Digital Ocean for example offer backups/snapshots. However, these offer completely inflexible/short schedules. I’ve also found these backup are majorly resource intensive on smaller droplets.

I’m sure I’m missing some options but I see completely necessary requirements as:

  • Totally automated post-setup
  • Flexible scheduling
  • Purging/retention policies
  • Easy restoration

Thoughts? What’s your backup strategy? Ideas?

4 Likes
2

+1 for this. I know that the things we really need to backup is the Database and the uploads folders where if you use Offload S3 or any other similar plugin, you can skip.

If you use RDS for the backup I think you’ve covered. But we all know that a backup is only good if the restoration really works when needed. But I’d love to have a peace of mind for this.

I think if you cover those two, you can always spin up a new app and everything should work normally.

But a daily automatic DB + uploads (S3 can get deleted by your co-worker and it will trust me). would be pretty nice to have a peace of mind.

One thing I’d love to see would be a daily(or set interval) “test” that you run that says backup restoration is ok etc. So I know that if the time comes the restoration will be a success.

4 Likes
3

I think there’s also a danger of “the perfect being the enemy of the good” in planning a feature like this. For most sites, two rolling weeks of backups local to the droplet, plus Digital Ocean’s backup feature which takes full-droplet snapshots, should be enough of a DR plan.

There are definitely sites that need more complex solutions, but those are also sites where you’re going make a specific DR plan for the client. Having something basic, and automatic (or at least strongly recommended) helps normalize a DR solution in the minds of Trellis users.

1 Like
4

I agree this is an issue that is critical to the long term adoption rate of Trellis and possibly Bedrock.

Before Trellis, I found perfect peace with Revisr. Which backs up my entire site and database to my Git repository, incrementally. You can even specify a separate branch, which you can later manually prune so your repo doesn’t get oversized.

Revisr is not compatible, however, with Trellis because the local files structure differs. This means that when Revisr asks me what folders to include, it gets confused because the development path to the uploads folder, for example, differs from the production path.

I’m actively working on solving this issue (possibly going to submit a patch to Revisr that allows you to configure different paths for different locations) and will definitely post a solution here if I arrive at one. But I wanted to share this in case someone has settled on a solution since and would like to share.

@MWDelaney, I saw your solution (great work!) but the reason I am not adopting it currently is because:

  1. If Git contains a backup of the site already, why create yet another server to backup too later?
  2. Seems like restoring the database and uploads directory would be a manual process?

Also, I am using (for the first time today) Mergebot to sync my Production and Development databases. But it’s not a backup solution, strictly merging.

5

Well that’s debatable. And maybe take a look at the emotional manipulation part of this article.

This is my solution for backing up sites to an S3 bucket:

site/scripts/backup.sh:

#!/bin/bash
eval $(cat ../.env | sed 's/^/export /')
export AWS_CONFIG_FILE="/home/web/.aws/config"
SITE="${DB_USER//_/.}"
ENVIRONMENT="$WP_ENV"
TIMESTAMP=`env TZ=America/Denver date +%Y-%m-%d-%H%M`
ARCHIVE_PATH=/tmp/$SITE-$ENVIRONMENT-$TIMESTAMP
ARCHIVE_FILENAME=$SITE-$ENVIRONMENT-$TIMESTAMP.tar.gz
mkdir -p $ARCHIVE_PATH &&
cd /srv/www/$SITE/current && wp db export $ARCHIVE_PATH/db.sql &&
rsync -kavzP --exclude web/wp/ --exclude web/wp-config.php /srv/www/$SITE/current/web $ARCHIVE_PATH &&
rsync -kavzP /srv/www/$SITE/shared/uploads $ARCHIVE_PATH/web/app &&
tar -C $ARCHIVE_PATH -czf /tmp/$ARCHIVE_FILENAME . &&
/usr/local/bin/aws s3 cp /tmp/$ARCHIVE_FILENAME s3://site-backups/$SITE/$ARCHIVE_FILENAME &&
rm -rf $ARCHIVE_PATH &&
rm /tmp/$ARCHIVE_FILENAME

Add to wordpress-setup/tasks/main.yml:

- name: Setup nightly cron for S3 backup
  cron:
    name: "{{ item.key }} nightly S3 backup"
    minute: "0"
    hour: "12"
    user: "{{ web_user }}"
    job: "cd {{ www_root }}/{{ item.key }}/{{ item.value.current_path | default('current') }}/scripts && ./backup-to-s3.sh > /dev/null 2>&1"
    cron_file: "backup-nightly-{{ item.key | replace('.', '_') }}"
  with_dict: "{{ wordpress_sites }}"
  when: env != 'development'

Add to trellis/requirements.yml:

- name: aws-cli
  src: https://github.com/dstil/ansible-aws-cli
  version: 5c1c14d9b0e6264605bd475c86d4ea67e7d5896f

Add to trellis/server.yml:

    - { role: aws-cli, tags: [aws-cli] }

Add to trellis/group_vars/all/vault.yml:

vault_aws_access_key_id: xxxxxxx
vault_aws_secret_access_key: "xxxxxxx"
17 Likes
6

Thanks for the solution Ben. This looks very well thought out! I will probably end up at least trying this.

Regarding the article and specifically the the emotional manipulation part of it: that’s a great article and extremely well articulated.

After reading it, I suppose how you, or others, may interpret my comment as emotional manipulation. But I want to ask you’d reconsider your interpretation of it for the following reason:

I’m under the assumption that it’s the desire of the Roots community to see WordPress core teams adopt the practices we example here. Or at least some variation of them. WordPress backup Plugins are one of the most varied and downloaded (if not the most downloaded) categories of Plugins in the WordPress community (contrast that to downloads of JetPack - which I believe is the most downloaded WordPress Plugin). This tells us that it’s of great importance to that community. So if Roots stack doesn’t offer a backup solution, and isn’t compatible with virtually any other backup Plugins, then would you disagree that this will affect the long term adoption rate?

To clarify, I’m not arguing that my proposed solution is the best. I’m just saying that without a best-practice backup solution in place, described in the docs, and simplified as much as is possible, it seems there is enough evidence to assume it will impact the adoption rate.

7

Maybe you haven’t listened to any of the podcasts or the Changelog episode that Scott and I were on… but we don’t exactly have any sort of faith in WordPress core.

And now you’re just making things up. Not only have I personally used several backup plugins with Trellis and Bedrock, but many other users in the community as well.

This is opinionated and the docs already reference user contributed extensions

8

I’m sorry Ben. I seem to have offended you here.

No I haven’t heard your podcasts. But I will subscribe now. I’m sorry to hear you guys don’t have much faith in WordPresss Core. It’s definitely a pain to deal with, but I hope that you both, and everyone involved in this community, never gives up on influencing that group because Roots essentially forks WordPress. If you guys as leaders choose to avoid working with WordPress then doesn’t that seem like it makes the fate of the Roots stack extremely unpredictable? I, for one, will do my best to serve both of these communities and hopefully that example and the results that come from it will speak loudly enough to affect change in that area.

I’m sorry you feel I’m making things up. I would never intentionally do that. I’ll rephrase what I said and say: based on my experience, I don’t know of any backup Plugins that are compatible with Roots - not to say there are none. Compatible, to me, means:

  1. It recognizes the environment it’s in and does not backup Staging or Development if the developer wills it.
  2. It backs up the database, uploads folder, and anything else I may be forgetting of that the Roots community would all agree should be.
  3. It only backs up what has changed since the last backup (incremental).
  4. This is debatable as a best practice I suppose, but I believe it should back these up to the Git repository that already exists for the Roots stack (into a dedicated branch) because the rest of the application / site is there.
  5. Restoration is automatic apart from the need to possible specify configuration parameters of some kind. This way you don’t have to manually move your uploads folder, or others, into /site during a restore.

See above.

9

Thanks, we have been doing this for many years now.

Yep, it is, and you’re still doing it. Please stop immediately.

1 Like
11

And then run ansible-galaxy install -r requirements.yml again.

2 Likes