How to change droplets on DO with SSL and minimal downtime?

jcollin · February 7, 2020, 9:58am

Hi,

Currently, I have a DO droplet (Ubuntu 16.04) hosting a website for testing at staging.mysite.com. I created a new droplet (Ubuntu 18.04) with various updates.

I would like to be able to provision this new droplet, test that all is working, then flip the old staging droplet to the new one without any downtime. The problem is, I cannot enable SSL because Let’s Encrypt requires that the canonical hosts have DNS that points to the new droplet before provisioning. This means bringing the current staging droplet offline before I can start provisioning the new droplet.

I will need to do the same process for our production droplet, and cannot have this extensive downtime during provisioning, then a vanilla website up while we import the database, media, etc.

I feel like a floating IP would have resolved this had we originally had the floating IP when creating the first droplet and SSL cert. However, I’m not sure how to address this now.

In summary, I just want to be able to provision and deploy, test it, then flip the DNS from the old droplet to the new one without any major downtime (or any at all preferably).

Any ideas?

Thanks in advance.

Xilonz · February 7, 2020, 10:45am

When using letsencrypt you can use a manual ssl certificate first which you create using certbot (in manual mode, DNS-verification)

I’ve been using Cloudflare and https://github.com/TypistTech/trellis-cloudflare-origin-ca, I believe this doesn’t need the domain to resolve to the right ip.

You’ll see Trellis error-out if it didn’t work

When going for minimal downtime:
Change TTL of the A record to lowest value possible first.
Disable SSL, provision the server, set everything up.
Set the A record to the new ip
Provision with --tags letsencrypt (and maybe wordpress, I don’t remember). So it skips all the other tasks.

craigpearson · February 7, 2020, 10:48am

I believe the SSL Certificate isn’t bound to the IP address (I could be wrong). If you copy the certificates from your production server to your new server and then switch your DNS I believe you should still be able to access over HTTPS with no down time. It’s getting the certificate from Lets Encrypt which relies on the IP(?)

You can attempt to do this by copying all the required certificate files, then making an entry for www.mysite.com in your host file pointing to the new IP with certificates in place. This would allow you to test the theory.

All of the directories where certificate data is stored can be found here:

github.com

roots/trellis/blob/master/roles/letsencrypt/defaults/main.yml

sites_using_letsencrypt: "[{% for name, site in wordpress_sites.items() | list if site.ssl.enabled and site.ssl.provider | default('manual') == 'letsencrypt' %}'{{ name }}',{% endfor %}]"
site_uses_letsencrypt: ssl_enabled and item.value.ssl.provider | default('manual') == 'letsencrypt'
missing_hosts: "{{ site_hosts | difference((current_hosts.results | selectattr('item.key', 'equalto', item.key) | selectattr('stdout_lines', 'defined') | sum(attribute='stdout_lines', start=[]) | map('trim') | list | join(' ')).split(' ')) }}"
letsencrypt_cert_ids: "{ {% for item in (generate_cert_ids | default({'results':[{'skipped':True}]})).results if item is not skipped %}'{{ item.item.key }}':'{{ item.stdout }}', {% endfor %} }"

acme_tiny_repo: 'https://github.com/diafygi/acme-tiny.git'
acme_tiny_commit: 'cb094cf3efa34acef8c7139c8480e2135422e755'

acme_tiny_software_directory: /usr/local/letsencrypt
acme_tiny_data_directory: /var/lib/letsencrypt
acme_tiny_challenges_directory: "{{ www_root }}/letsencrypt"

# Path to the local file containing the account key to copy to the server.
# Secure this file using Git-crypt for example.
# Leave this blank to generate a new account key that will need to be registered manually with Letsencrypt.org
#letsencrypt_account_key_source_file: /my/account.key

# Content of the account key to copy to the server.
# Secure this key using Ansible Vault for example.
# Leave this blank to generate a new account key that will need to be registered manually with Letsencrypt.org

This file has been truncated. show original

And Here:

github.com

roots/trellis/blob/master/roles/letsencrypt/tasks/nginx.yml

---
- name: Create Nginx conf for challenges location
  template:
    src: acme-challenge-location.conf.j2
    dest: "{{ nginx_path }}/acme-challenge-location.conf"

- name: Get list of hosts in current Nginx conf
  shell: |
    [ ! -f {{ nginx_path }}/sites-enabled/{{ item.key }}.conf ] ||
    sed -n -e "/listen 80/,/server_name/{s/server_name \(.*\);/\1/p}" {{ nginx_path }}/sites-enabled/{{ item.key }}.conf
  register: current_hosts
  changed_when: false
  when: site_uses_letsencrypt
  with_dict: "{{ wordpress_sites }}"

- name: Create needed Nginx confs for challenges
  template:
    src: nginx-challenge-site.conf.j2
    dest: "{{ nginx_path }}/sites-available/letsencrypt-{{ item.key }}.conf"
  register: challenge_site_confs

This file has been truncated. show original

And Here:

github.com

roots/trellis/blob/master/roles/letsencrypt/tasks/main.yml

---
- import_tasks: setup.yml
- import_tasks: nginx.yml
- import_tasks: certificates.yml

- name: Install cronjob for key generation
  cron:
    cron_file: letsencrypt-certificate-renewal
    name: letsencrypt certificate renewal
    user: root
    job: cd {{ acme_tiny_data_directory }} && ./renew-certs.py && /usr/sbin/service nginx reload
    day: "{{ letsencrypt_cronjob_daysofmonth }}"
    hour: 4
    minute: 30
    state: present

Before doing this you may need to do some things such originally deploy/provision on a staging URL such as temp.mysite.com to allow Trellis scripts to complete successfully. Then SSH into the server to change appropriate nginx configs, from temp.mysite.com to www.mysite.com and overwrite any certificate files.

system · March 20, 2020, 9:58am

This topic was automatically closed after 42 days. New replies are no longer allowed.