Let's Encrypt fails to renew domain

Hi,

I have three sites running on one server on Digital Ocean running Trellis (latest commit 186a1d7)

I discovered this morning that none of the three sites had renewed the certificate from Let’s Encrypt. Since I had HSTS enabled none of them are accessible at the moment.

What I have tried:

  • Manually running renew_certs.py with the following output.
Certificate file /etc/nginx/ssl/letsencrypt/dinbuss.se.cert already exists
Generating certificate for dinbuss.se
Error while generating certificate for dinbuss.se
Parsing account key...
Parsing CSR...
Registering account...
Already registered!
Verifying dinbuss.se...
Traceback (most recent call last):
  File "/usr/local/letsencrypt/acme_tiny.py", line 198, in <module>
    main(sys.argv[1:])
  File "/usr/local/letsencrypt/acme_tiny.py", line 194, in main
    signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca)
  File "/usr/local/letsencrypt/acme_tiny.py", line 123, in get_crt
    wellknown_path, wellknown_url))
ValueError: Wrote file to /srv/www/letsencrypt/bwsSQDWEC4y7coHulz-NdF-iWsNarHSejdTPeMj79Ds, but couldn't download http://dinbuss.se/.well-known/acme-challenge/bwsSQDWEC4y7coHulz-NdF-iWsNarHSejdTPeMj79Ds
  • Running the letscenrypt task with Trellis (with -vvv). The following error was output for all three sites (showing only one here)
Could not access the challenge file for the hosts/domains: dinbuss.se,
www.dinbuss.se. Let's Encrypt requires every domain/host be publicly
accessible. Make sure that a valid DNS record exists for dinbuss.se,
www.dinbuss.se and that they point to this server's IP. If you don't want
these domains in your SSL certificate, then remove them from `site_hosts`.
See https://roots.io/trellis/docs/ssl for more details.

failed: [178.62.204.175] (item={'_ansible_parsed': True, u'changed': False, '_ansible_no_log': False, 'failed': True, '_ansible_item_result': True, u'rc': 1, 'item': {'value': {u'multisite': {u'enabled': False}, u'env': {u'symfony_env': u'prod'}, u'cache': {u'enabled': True}, u'repo': u'git@bitbucket.org:arashohadi/sfsite-dinbuss.git', u'ssl': {u'enabled': True, u'provider': u'letsencrypt'}, u'local_path': u'../DinBuss', u'branch': u'master', u'site_hosts': [{u'redirects': [u'www.dinbuss.se'], u'canonical': u'dinbuss.se'}]}, 'key': u'dinbuss.se'}, u'failed_hosts': [u'dinbuss.se', u'www.dinbuss.se'], 'invocation': {'module_name': u'test_challenges', u'module_args': {u'path': u'.well-known/acme-challenge', u'hosts': [u'dinbuss.se', u'www.dinbuss.se'], u'file': u'ping.txt'}}}) => {
    "failed": true,
    "invocation": {
        "module_args": {
            "msg": "Could not access the challenge file for the hosts/domains: dinbuss.se, www.dinbuss.se. Let's Encrypt requires every domain/host be publicly accessible. Make sure that a valid DNS record exists for dinbuss.se, www.dinbuss.se and that they point to this server's IP. If you don't want these domains in your SSL certificate, then remove them from `site_hosts`. See https://roots.io/trellis/docs/ssl for more details.\n"
        },
        "module_name": "fail"
    },
    "item": {
        "changed": false,
        "failed": true,
        "failed_hosts": [
            "dinbuss.se",
            "www.dinbuss.se"
        ],
        "invocation": {
            "module_args": {
                "file": "ping.txt",
                "hosts": [
                    "dinbuss.se",
                    "www.dinbuss.se"
                ],
                "path": ".well-known/acme-challenge"
            },
            "module_name": "test_challenges"
        },
        "item": {
            "key": "dinbuss.se",
            "value": {
                "branch": "master",
                "cache": {
                    "enabled": true
                },
                "env": {
                    "symfony_env": "prod"
                },
                "local_path": "../DinBuss",
                "multisite": {
                    "enabled": false
                },
                "repo": "git@bitbucket.org:arashohadi/sfsite-dinbuss.git",
                "site_hosts": [
                    {
                        "canonical": "dinbuss.se",
                        "redirects": [
                            "www.dinbuss.se"
                        ]
                    }
                ],
                "ssl": {
                    "enabled": true,
                    "provider": "letsencrypt"
                }
            }
        },
        "rc": 1
    }
}

Questions:

  • It is obviously telling me that it can’t access the files for verification but how I can I solve this?
  • Is it because HSTS was enabled and now it is not allowing incoming connections since the certificate has expired?
  • I have many other sites that I manage and I am afraid that this might affect all of them eventually (once the certificate expires). The closest one in time is a site (on another server) with a certificate that expires June 19th. Looking at the crontab script it should have renewed by now right?

Best regards,
aoe

It puzzles me that your output suggests that Trellis is connecting to the IP 178.62.204.175 but dinbus.se seems to be assigned a different IP:

# What is IP for dinbuss.se
$ nslookup dinbuss.se
Non-authoritative answer:
Name:	dinbuss.se
Address: 188.166.200.79

Perhaps you have changed the DNS since posting?

Right now I get a real site loading at http://dinbuss.se/ but the server doesn’t seem to be listening at https for this domain. Maybe you’ve just turned off SSL for now.

I also get at 200 successful connection to the acme ping test:

# Success in accessing acme challenge ping.txt test
$ curl -s -o /dev/null -I -w "%{http_code}" http://dinbuss.se/.well-known/acme-challenge/ping.txt
200%

This suggests that the Let’s Encrypt verification should work, if you run the Trellis letsencrypt role on the other server at 188.166.200.79.

You could check your hosts/production to ensure you’re connecting to the correct IP/server when running Trellis.

There has been one Let’s Encrypt-related update to Trellis since the version you are using, but I don’t think it would affect simple LE cert renewal, the issue at hand.

You probably already know this, but just in case…
While investigating these issues, it’s useful to clear your browser’s HSTS headers for the domain (and its subdomains like www) and view the site in an “incognito” or “private browsing” window (less cached info, fewer headers saved, etc.).

Thank you for the detailed suggestions!

188.166.200.79 is a floating IP which is connected to the droplet at 178.62.204.175. I’ll give it a try and see if it works with the floating IP (but there shouldn’t be a difference right?).

Yes, I have reprovisioned the server with http for now. This is actually because I was following your outlined steps in another post for reprovisioning an already provisioned server. That seems to be what you have to do prior to the pull request you mention.

I will also try to clear the HSTS headers. I have since posting this successfully managed to manually create HTTPS certificates using the letsencrypt repository directly (e.g. creating acme files manually). I have just not been able to get the Trellis task to go through yet.

I hadn’t touched or provisioned this server for a few months. Before then the automatic renewals of certificates worked fine so I’m a little stumped as to why it didn’t renew and now can’t reprovision with https.

1 Like

A follow up:

I never managed to make a provision pass the Test Acme challenges task. Looking in /srv/www/letsencrypt while the provisioning was being carried out, I could not see a file being generated, which meant the task failed.

Ultimately I ended up creating a new droplet and pointed the floating ip to that one instead. It worked without problems on the first try. There was obviously something wrong with the previous droplet but I did not manage to find out what exactly.

1 Like