Roots Discourse

Certificate expires, issues running renew-certs.py

More and more of my older Trellis projects that I have created 2-3 years ago are now coming up with the same expired certificate issue: NET::ERR_CERT_AUTHORITY_INVALID. Today’s one marks the fourth one that has experienced the very same issue in the last two weeks, and while I just created new droplets for the others with a new installation and migrate over, I would love to know if there’s some method to resolving this without having to do that for each one, as some of them are smaller projects that I cannot designate that much time towards.

  • All of them are built on Ubuntu 16.04 with Python 2.7 — which I’m sure is now being deprecated and is causing the issues.
  • For some reason all of them reference July 28th as the expiry date, although they were deployed at different times.
  • The cron.d file contains the correct letsencrypt-certificate-renewal file with the proper paths, but the renew-certs.py fails every time citing a forbidden error:
Error while generating certificate for x.com
Traceback (most recent call last):
  File "/usr/local/letsencrypt/acme_tiny.py", line 198, in <module>
    main(sys.argv[1:])
  File "/usr/local/letsencrypt/acme_tiny.py", line 194, in main
    signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca)
  File "/usr/local/letsencrypt/acme_tiny.py", line 85, in get_crt
    "agreement": json.loads(urlopen(CA + "/directory").read().decode('utf8'))['meta']['terms-of-service'],
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 435, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 548, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 473, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 556, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden
  • As suggested in another thread, re-running the provision with the letsencrypt tag comes up with a similar error:
fatal: [ea]: FAILED! => {
    "changed": false,
    "cmd": [
        "./renew-certs.py"
    ],
    "delta": "0:00:00.185746",
    "end": "2021-08-04 00:16:46.597302",
    "invocation": {
        "module_args": {
            "_raw_params": "./renew-certs.py",
            "_uses_shell": false,
            "chdir": "/var/lib/letsencrypt",
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "rc": 1,
    "start": "2021-08-04 00:16:46.411556",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "Generating certificate for x.com\nError while generating certificate for x.com\nTraceback (most recent call last):\n  File \"/usr/local/letsencrypt/acme_tiny.py\", line 198, in <module>\n    main(sys.argv[1:])\n  File \"/usr/local/letsencrypt/acme_tiny.py\", line 194, in main\n    signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca)\n  File \"/usr/local/letsencrypt/acme_tiny.py\", line 85, in get_crt\n    \"agreement\": json.loads(urlopen(CA + \"/directory\").read().decode('utf8'))['meta']['terms-of-service'],\n  File \"/usr/lib/python2.7/urllib2.py\", line 154, in urlopen\n    return opener.open(url, data, timeout)\n  File \"/usr/lib/python2.7/urllib2.py\", line 435, in open\n    response = meth(req, response)\n  File \"/usr/lib/python2.7/urllib2.py\", line 548, in http_response\n    'http', request, response, code, msg, hdrs)\n  File \"/usr/lib/python2.7/urllib2.py\", line 473, in error\n    return self._call_chain(*args)\n  File \"/usr/lib/python2.7/urllib2.py\", line 407, in _call_chain\n    result = func(*args)\n  File \"/usr/lib/python2.7/urllib2.py\", line 556, in http_error_default\n    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)\nurllib2.HTTPError: HTTP Error 403: Forbidden",
    "stdout_lines": [
        "Generating certificate for x.com",
        "Error while generating certificate for x.com",
        "Traceback (most recent call last):",
        "  File \"/usr/local/letsencrypt/acme_tiny.py\", line 198, in <module>",
        "    main(sys.argv[1:])",
        "  File \"/usr/local/letsencrypt/acme_tiny.py\", line 194, in main",
        "    signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca)",
        "  File \"/usr/local/letsencrypt/acme_tiny.py\", line 85, in get_crt",
        "    \"agreement\": json.loads(urlopen(CA + \"/directory\").read().decode('utf8'))['meta']['terms-of-service'],",
        "  File \"/usr/lib/python2.7/urllib2.py\", line 154, in urlopen",
        "    return opener.open(url, data, timeout)",
        "  File \"/usr/lib/python2.7/urllib2.py\", line 435, in open",
        "    response = meth(req, response)",
        "  File \"/usr/lib/python2.7/urllib2.py\", line 548, in http_response",
        "    'http', request, response, code, msg, hdrs)",
        "  File \"/usr/lib/python2.7/urllib2.py\", line 473, in error",
        "    return self._call_chain(*args)",
        "  File \"/usr/lib/python2.7/urllib2.py\", line 407, in _call_chain",
        "    result = func(*args)",
        "  File \"/usr/lib/python2.7/urllib2.py\", line 556, in http_error_default",
        "    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)",
        "urllib2.HTTPError: HTTP Error 403: Forbidden"
    ]
}

Ideally I would just love to manually issue a certificate on the remote server, or figure out a direct patch to the Trellis files in question. I’ve tried to manually update Ubuntu with another project, which ran into a series of apt package issues. Removing the cert has obvious issues with the HSTS headers so I would love to hear what the best tactic here would be for me to quickly get the site back up and running with a certificate as I anticipate this will continue to happen to the older/unmaintained projects I have out in the wild.

Appreciate it, as always!

Can you try updating trellis/roles/letsencrypt/defaults/main.yml based on https://github.com/roots/trellis/pull/1114?

- acme_tiny_commit: '4ed13950c0a9cf61f1ca81ff1874cde1cf48ab32'
+ acme_tiny_commit: 'cb094cf3efa34acef8c7139c8480e2135422e755'
- letsencrypt_ca: 'https://acme-v01.api.letsencrypt.org'
+ letsencrypt_ca: 'https://acme-v02.api.letsencrypt.org'
3 Likes

@benword — that was exactly what I was looking for, worked beautifully! Thank you so much, for anyone else who runs into a similar issue here is the summary for the fix:

SOLUTION: Update the main.yml file,
change the value for letsencrypt_ca to:
https://acme-v02.api.letsencrypt.org

and acme_tiny_commit to:
cb094cf3efa34acef8c7139c8480e2135422e755

then run:
ansible-playbook server.yml -e env=production -K --tags letsencrypt

Appreciate all the help, as always!

2 Likes

I ran into the same problem on an older site. Solution worked for me too. And then running ansible-playbook server.yml -e env=production --tags letsencrypt Thanks for sharing

1 Like

This topic was automatically closed after 42 days. New replies are no longer allowed.