I rolled back production to a staging release and now it's broken

trellis
1

I’ve made so many mistakes today I don’t know where to begin…

I was trying to apply a simple code fix to my site. I edited the files, committed the changes to the “staging” branch in my repo and deployed to the Digital Ocean staging server. Everything seemed to work.

Then, I ran a deploy to production, which is set to pull from the master branch. Nothing happened because I hadn’t merged the code yet. I went ahead and did this.

There were some other changes in the code that ended up breaking stuff in the site. Important stuff. I don’t know why it broke or what files to look at.

I panicked and tried first to reset my commits and then redeployed (multiple times). Unsuccessfully.

Then I attempted to roll back using Trellis to a release that I knew worked. Unfortunately I didn’t have one in production because it only keeps the last few.

I copied a release folder from staging to production and edited the .env file to match others in the production.

The site is semi-working but I think there are broken symlinks to things like the uploads folder and that is preventing a needed plugin from working.

My developer is away in Europe until next week and my mistakes are affecting all my customers. Is there anyone out there that can help me?

Tom

2

Hey Tom,

First off, I’m not certain why your staging and production environments would be in different branches. Default, Trellis handles both simultaneously.

Secondly, unfortunately, I’ve found the best solution for issues like these are to reclone the default project, and tying my customizations back in slowly until I can figure out where the issue was.

Last week I had some crazy obscure errors I was getting because I stupidly updated my Python version. Resorted to reformatting my entire hard drive and re-spinning up a roots project.

I’ve found that to actually be a time saver if you don’t have any specific details on what exactly is wrong.

Perhaps one of the other guys has a better tip, but that’s my usual approach.

You can always run the deploy in verbose mode and report the error log back.

3

It’s normal to use different branches for environments.

Use git to revert back to the last working commit that was deployed to production, then re-deploy.

4

Thanks Ben. I’ve already hard reset the repos to the last commit that I know works. There are just some deployment issues to deal with now. I think I can solve my issues if I can move the uploads folder from an older release to another. What’s the best way to do this?