Server.yml failing on letsencrypt Generate the certificates

Server.yml is failing on the following task:

TASK [letsencrypt : Generate the certificates] 

This is in the error message:

File \"/usr/lib/python2.7/urllib2.py\", line 556, in http_error_default", "    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)", "urllib2.HTTPError: HTTP Error 403: Forbidden"]

Any ideas how to fix it?

On a hunch: Are you using python2 instead python3?

Yes. I use v2 for roots projects, via Anaconda.

I have a hunch that if I can pass headers in it’ll run, as I have this in the error, too:

File \"/usr/lib/python2.7/urllib2.py\", line 556, in http_error_default", "    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)"

Any idea how I would go about that in the relevant .yml file (roles/letsencrypt/tasks/certificates.yml)?

Can you run ansible with python3 and see whether it works then?

That is my end goal, but at the moment that’s not working.

The complete stack trace of that error would be useful.
That’s what Trellis (latest) is using in the certificate renewal script:

non-zero return code
fatal: [example.com]: FAILED! => {
    "changed": false, "cmd": ["./renew-certs.py"], 
    "delta": "0:00:00.550123", 
    "end": "2021-02-11 11:11:39.949698", 
    "failed": true, 
    "rc": 1, 
    "start": "2021-02-11 11:11:39.399575", 
    "stderr": "", 
    "stderr_lines": [], 
    "stdout": "Certificate file /etc/nginx/ssl/letsencrypt/example.com-270d171.cert already exists\n
        Generating certificate for example.com\n
        Error while generating certificate for example.com\n
        Traceback (most recent call last):\n
            File \"/usr/local/letsencrypt/acme_tiny.py\", line 198, in <module>\n 
                main(sys.argv[1:])\n  
            File \"/usr/local/letsencrypt/acme_tiny.py\", line 194, in main\n    
                signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca)\n  
            File \"/usr/local/letsencrypt/acme_tiny.py\", line 85, in get_crt\n    
                \"agreement\": json.loads(urlopen(CA + \"/directory\").read().decode('utf8'))['meta']['terms-of-service'],\n  
            File \"/usr/lib/python2.7/urllib2.py\", line 154, in urlopen\n    
                return opener.open(url, data, timeout)\n  
            File \"/usr/lib/python2.7/urllib2.py\", line 435, in open\n    
                response = meth(req, response)\n  
            File \"/usr/lib/python2.7/urllib2.py\", line 548, in http_response\n    
                'http', request, response, code, msg, hdrs)\n  
            File \"/usr/lib/python2.7/urllib2.py\", line 473, in error\n    
                return self._call_chain(*args)\n  
            File \"/usr/lib/python2.7/urllib2.py\", line 407, in _call_chain\n    
                result = func(*args)\n  
            File \"/usr/lib/python2.7/urllib2.py\", line 556, in http_error_default\n    
                raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)\nurllib2.HTTPError: HTTP Error 403: Forbidden", "stdout_lines": [
                    "Certificate file /etc/nginx/ssl/letsencrypt/example.com-270d171.cert already exists", 
                    "Generating certificate for example.com", 
                    "Error while generating certificate for example.com", 
                    "Traceback (most recent call last):", "  
                        File \"/usr/local/letsencrypt/acme_tiny.py\", line 198, in <module>", "    
                            main(sys.argv[1:])", "  
                        File \"/usr/local/letsencrypt/acme_tiny.py\", line 194, in main", "    
                            signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca)", "  
                        File \"/usr/local/letsencrypt/acme_tiny.py\", line 85, in get_crt", "    
                            \"agreement\": json.loads(urlopen(CA + \"/directory\").read().decode('utf8'))['meta']['terms-of-service'],", "  
                        File \"/usr/lib/python2.7/urllib2.py\", line 154, in urlopen", "    
                            return opener.open(url, data, timeout)", "  
                        File \"/usr/lib/python2.7/urllib2.py\", line 435, in open", "    
                            response = meth(req, response)", "  
                        File \"/usr/lib/python2.7/urllib2.py\", line 548, in http_response", "    
                            'http', request, response, code, msg, hdrs)", "  
                        File \"/usr/lib/python2.7/urllib2.py\", line 473, in error", "    
                            return self._call_chain(*args)", "  
                        File \"/usr/lib/python2.7/urllib2.py\", line 407, in _call_chain", "    
                            result = func(*args)", "  
                        File \"/usr/lib/python2.7/urllib2.py\", line 556, in http_error_default", "    
                            raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)", "urllib2.HTTPError: HTTP Error 403: Forbidden"
                ] 
}

OK, so acme-tiny tries to validate the URL (and fails) before renewing the certificate via Let’s Encrypt using the HTTP-01 challenge.
Are the IPv4 (A) and IPv6 (AAAA) (if any) DNS records set correctly for that domain?
Can yiou reach that domain via plain HTTP (no HTTPS)?

I can’t reach the domain either way. It’s using IPv4 only.

Let’s Encrypt requires some kind of validation. Trellis uses the commonly used HTTP-01 challenge which requires these site domains to be reachable via HTTP (not HTTPS) from the outside.
So there is an underlying issue here, your domain has to be reachable via plain HTTP first.
Have you provisioned the Trellis system with that domain and have you deployed a functioning WordPress site to that domain?

Yep, the site’s been running for 3 years over https.

We are often pushing updates to the site, often several times a month. I re-ran server.yml because we needed to update the php version from 7.3 to 7.4.

So this is a typo and you meant tha the site is reachable via HTTP.

I can only access it via ssh at the moment.

It must be accessible at least by Let’s Encrypt for the certificate signing to succeed.
You could temporarily skip the certificate generation in wordpress-sites.yml:

wordpress_sites:
  example.com:
    ssl:
      enabled: false
      provider: self-signed

provider must be self-signed, otherwise some Let’s Encrypt configuration will still be applied.

Does this fix the plain HTTP issue when you reapply the playbook?

Nope. The playbook ran successfully, but the site wouldn’t load.

Are there other sites on that Trellis system, do these load?
Is the nginx listening on the system (netstat -tulpn | grep :80)?
Is the DNS A record the same as of the Trellis system public interface?
Is a hardware/cloud firewall involved (as on Ionos for example), that also blocks the HTTP 80 port by default?

I have other sites (including a staging site for this client) that use the same configs.

Thanks for this. This has temporarily helped me, as the site goes through Cloudflare, where I am able to apply an SSL until I can fix this properly.

You may now even try to re-enable Let’s Encrypt and re-apply the playbook:

wordpress_sites:
  example.com:
    ssl:
      enabled: true
      provider: letsencrypt

From previous experience the approach to disable Let’s Encrypt and then re-enabling it often resolves these kinds of issues.

1 Like