Staging Provision Issues

I have a setup that hosts multiple websites. It does not use multisite but they are all separate installs on the same server. Everything has been working fine but I just went to add a new website and provision the staging server and cannot get past the SSL generation. I turn the ssl off for the one website but everything breaks once the code has been deployed and I try to run the wordpress install. Everything locally is working as expected. I appreciate any help.

Ansible: 2.7.5
Node: 10.16.3

Here is the error that is currently showing. Please let me know if you need any additional info.

System info:
  Ansible 2.7.5; Darwin
  Trellis version (per changelog): "Update wp-cli to 2.0.1"
---------------------------------------------------
non-zero return code
fatal: [104.248.218.91]: FAILED! => {
    "changed": false, 
    "cmd": [
        "./renew-certs.py"
    ], 
    "delta": "0:00:00.976674", 
    "end": "2020-07-15 17:17:49.992593", 
    "invocation": {
        "module_args": {
            "_raw_params": "./renew-certs.py", 
            "_uses_shell": false, 
            "argv": null, 
            "chdir": "/var/lib/letsencrypt", 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "stdin": null, 
            "warn": true
        }
    }, 
    "rc": 1, 
    "start": "2020-07-15 17:17:49.015919", 
    "stderr": "", 
    "stderr_lines": [], 
    "stdout": "Certificate file /etc/nginx/ssl/letsencrypt/rustinconcrete.com-3c49826.cert already exists\n  The certificate is younger than 60 days. Not creating a new certificate.\n\nCertificate file /etc/nginx/ssl/letsencrypt/conagg-mo.com-745cf75.cert already exists\n  The certificate is younger than 60 days. Not creating a new certificate.\n\nCertificate file /etc/nginx/ssl/letsencrypt/greenamericarecycling.com-4a57f48.cert already exists\n  The certificate is younger than 60 days. Not creating a new certificate.\n\nCertificate file /etc/nginx/ssl/letsencrypt/nrhamm.com-bc62510.cert already exists\n  The certificate is younger than 60 days. Not creating a new certificate.\n\nCertificate file /etc/nginx/ssl/letsencrypt/continentalcementmulti.com-364790b.cert already exists\n  The certificate is younger than 60 days. Not creating a new certificate.\n\nCertificate file /etc/nginx/ssl/letsencrypt/americanmaterialsco.com-89bcd7e.cert already exists\n  The certificate is younger than 60 days. Not creating a new certificate.\n\nCertificate file /etc/nginx/ssl/letsencrypt/razorbackconcrete.com-5a58987.cert already exists\n  The certificate is younger than 60 days. Not creating a new certificate.\n\nGenerating certificate for midwest.com\nError while generating certificate for midwest.com\nTraceback (most recent call last):\n  File \"/usr/local/letsencrypt/acme_tiny.py\", line 198, in <module>\n    main(sys.argv[1:])\n  File \"/usr/local/letsencrypt/acme_tiny.py\", line 194, in main\n    signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca)\n  File \"/usr/local/letsencrypt/acme_tiny.py\", line 104, in get_crt\n    raise ValueError(\"Error requesting challenges: {0} {1}\".format(code, result))\nValueError: Error requesting challenges: 403 {\n  \"type\": \"urn:acme:error:unauthorized\",\n  \"detail\": \"Error creating new authz :: Validations for new domains are disabled in the V1 API (https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430)\",\n  \"status\": 403\n}\n\nCertificate file /etc/nginx/ssl/letsencrypt/georgiastoneproducts.com-85984a8.cert already exists\n  The certificate is younger than 60 days. Not creating a new certificate.\n\nCertificate file /etc/nginx/ssl/letsencrypt/rdje.com-079311b.cert already exists\n  The certificate is younger than 60 days. Not creating a new certificate.\n\nCertificate file /etc/nginx/ssl/letsencrypt/mainlandcmmulti.com-17a3218.cert already exists\n  The certificate is younger than 60 days. Not creating a new certificate.", 
    "stdout_lines": [
        "Certificate file /etc/nginx/ssl/letsencrypt/rustinconcrete.com-3c49826.cert already exists", 
        "  The certificate is younger than 60 days. Not creating a new certificate.", 
        "", 
        "Certificate file /etc/nginx/ssl/letsencrypt/conagg-mo.com-745cf75.cert already exists", 
        "  The certificate is younger than 60 days. Not creating a new certificate.", 
        "", 
        "Certificate file /etc/nginx/ssl/letsencrypt/greenamericarecycling.com-4a57f48.cert already exists", 
        "  The certificate is younger than 60 days. Not creating a new certificate.", 
        "", 
        "Certificate file /etc/nginx/ssl/letsencrypt/nrhamm.com-bc62510.cert already exists", 
        "  The certificate is younger than 60 days. Not creating a new certificate.", 
        "", 
        "Certificate file /etc/nginx/ssl/letsencrypt/continentalcementmulti.com-364790b.cert already exists", 
        "  The certificate is younger than 60 days. Not creating a new certificate.", 
        "", 
        "Certificate file /etc/nginx/ssl/letsencrypt/americanmaterialsco.com-89bcd7e.cert already exists", 
        "  The certificate is younger than 60 days. Not creating a new certificate.", 
        "", 
        "Certificate file /etc/nginx/ssl/letsencrypt/razorbackconcrete.com-5a58987.cert already exists", 
        "  The certificate is younger than 60 days. Not creating a new certificate.", 
        "", 
        "Generating certificate for midwest.com", 
        "Error while generating certificate for midwest.com", 
        "Traceback (most recent call last):", 
        "  File \"/usr/local/letsencrypt/acme_tiny.py\", line 198, in <module>", 
        "    main(sys.argv[1:])", 
        "  File \"/usr/local/letsencrypt/acme_tiny.py\", line 194, in main", 
        "    signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca)", 
        "  File \"/usr/local/letsencrypt/acme_tiny.py\", line 104, in get_crt", 
        "    raise ValueError(\"Error requesting challenges: {0} {1}\".format(code, result))", 
        "ValueError: Error requesting challenges: 403 {", 
        "  \"type\": \"urn:acme:error:unauthorized\",", 
        "  \"detail\": \"Error creating new authz :: Validations for new domains are disabled in the V1 API (https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430)\",", 
        "  \"status\": 403", 
        "}", 
        "", 
        "Certificate file /etc/nginx/ssl/letsencrypt/georgiastoneproducts.com-85984a8.cert already exists", 
        "  The certificate is younger than 60 days. Not creating a new certificate.", 
        "", 
        "Certificate file /etc/nginx/ssl/letsencrypt/rdje.com-079311b.cert already exists", 
        "  The certificate is younger than 60 days. Not creating a new certificate.", 
        "", 
        "Certificate file /etc/nginx/ssl/letsencrypt/mainlandcmmulti.com-17a3218.cert already exists", 
        "  The certificate is younger than 60 days. Not creating a new certificate."
    ]
}

RUNNING HANDLER [common : disable temporary challenge sites] ***********************************************************************************************************************************************
task path: /Users/mikewalcott/Sites/summitopco/trellis/roles/common/tasks/disable_challenge_sites.yml:2
Using module file /Library/Python/2.7/site-packages/ansible/modules/files/file.py
<104.248.218.91> ESTABLISH SSH CONNECTION FOR USER: admin
<104.248.218.91> SSH: EXEC ssh -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=admin -o ConnectTimeout=10 -o ControlPath=/Users/mikewalcott/.ansible/cp/1ca9331dd4 104.248.218.91 '/bin/sh -c '"'"'sudo -H -S  -p "[sudo via ansible, key=cfkfuukkelplcvdbygrrhnpfgafydwpr] password: " -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-cfkfuukkelplcvdbygrrhnpfgafydwpr; /usr/bin/python'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<104.248.218.91> (0, '\n{"invocation": {"module_args": {"directory_mode": null, "force": false, "remote_src": null, "_original_basename": null, "path": "/etc/nginx/sites-enabled/letsencrypt-rustinconcrete.com.conf", "owner": null, "follow": true, "group": null, "unsafe_writes": null, "state": "absent", "content": null, "serole": null, "selevel": null, "setype": null, "access_time": null, "access_time_format": "%Y%m%d%H%M.%S", "modification_time": null, "regexp": null, "src": null, "seuser": null, "recurse": false, "_diff_peek": null, "delimiter": null, "mode": null, "modification_time_format": "%Y%m%d%H%M.%S", "attributes": null, "backup": null}}, "path": "/etc/nginx/sites-enabled/letsencrypt-rustinconcrete.com.conf", "state": "absent", "changed": false}\n', '')

This seems to be the underlying error:

ACME v1 API is now phased out in favour of ACME v2 API. Only existing domains can still be renewed over the v1 API, so legacy systems that use LE certificates don’t break.

Are you using an outdated version of Trellis?

1 Like

Yes this is an older version. I have over 10 sites on this install and havent found a simple way to update it. Is there any way to make this version work or do I need to update it?

Well, you are trying to register new domains with LE using the old ACME v1 API, which isn’t possible anymore. While you could try to update just the ACME related part in Trellis, I think you should update your whole Trellis setup and test an update (reprovision) on a staging server. When everything went well and the sites are still available on staging, it should also work on production.

For my part, I cloned the Trellis repository and added all modifications/adjustments to it on a separate branch. Then I can pull in and merge new changes from time to time.
In you case you could just put your whole Trellis code into Version control (git) and then overwrite it with the latest Trellis files - or clone the Trellis repository, create a new branch and put your changed files on top of it and resolve all conflicts.
Always test on a staging system that already has been provisioned with your old Trellis setup, so you can verify the updates all work correctly!

1 Like

@strarsis’s replies are great, but it is quite simple to just apply a patch and fix this issue:

Note that we’ve had a sticking post about this since October: Trellis and Lets Encrypt v1 end of life

1 Like

@strarsis and @swalkinshaw - Thank you for the help. I just got the patch applied and everything is working as expected now.

Should this patch also work for production. I am now having the same issue even after applying the patch.

I just setup another website and tested the provision on staging. This works with no problem. But when I provision the production server I get this. Is there something that I am missing between production and staging for change from V1 to V2.

System info:
  Ansible 2.5.3; Darwin
  Trellis version (per changelog): "Update wp-cli to 2.0.1"
---------------------------------------------------
non-zero return code
Error while generating certificate for peakranchresource.com
Traceback (most recent call last):
  File "/usr/local/letsencrypt/acme_tiny.py", line 198, in <module>
    main(sys.argv[1:])
  File "/usr/local/letsencrypt/acme_tiny.py", line 194, in main
    signed_crt = get_crt(args.account_key, args.csr, args.acme_dir,
log=LOGGER, CA=args.ca, disable_check=args.disable_check,
directory_url=args.directory_url, contact=args.contact)
  File "/usr/local/letsencrypt/acme_tiny.py", line 143, in get_crt
    raise ValueError("Wrote file to {0}, but couldn't download {1}:
{2}".format(wellknown_path, wellknown_url, e))
ValueError: Wrote file to
/srv/www/letsencrypt/BE4T5_5c1Zy6po4fsbCAQnV35mhXag5bPLRUcLAUpjA, but
couldn't download http://www.peakranchresource.com/.well-known/acme-
challenge/BE4T5_5c1Zy6po4fsbCAQnV35mhXag5bPLRUcLAUpjA:
fatal: [138.68.5.112]: FAILED! => {
    "changed": false, 
    "cmd": [
        "./renew-certs.py"
    ], 
    "delta": "0:00:02.588395", 
    "end": "2020-08-03 19:51:22.969678", 
    "invocation": {
        "module_args": {
            "_raw_params": "./renew-certs.py", 
            "_uses_shell": false, 
            "chdir": "/var/lib/letsencrypt", 
            "creates": null, 
            "executable": null, 
            "removes": null, 
            "stdin": null, 
            "warn": true
        }
    }, 
    "rc": 1, 
    "start": "2020-08-03 19:51:20.381283", 
    "stderr_lines": [
        "Error while generating certificate for peakranchresource.com", 
        "Traceback (most recent call last):", 
        "  File \"/usr/local/letsencrypt/acme_tiny.py\", line 198, in <module>", 
        "    main(sys.argv[1:])", 
        "  File \"/usr/local/letsencrypt/acme_tiny.py\", line 194, in main", 
        "    signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca, disable_check=args.disable_check, directory_url=args.directory_url, contact=args.contact)", 
        "  File \"/usr/local/letsencrypt/acme_tiny.py\", line 143, in get_crt", 
        "    raise ValueError(\"Wrote file to {0}, but couldn't download {1}: {2}\".format(wellknown_path, wellknown_url, e))", 
        "ValueError: Wrote file to /srv/www/letsencrypt/BE4T5_5c1Zy6po4fsbCAQnV35mhXag5bPLRUcLAUpjA, but couldn't download http://www.peakranchresource.com/.well-known/acme-challenge/BE4T5_5c1Zy6po4fsbCAQnV35mhXag5bPLRUcLAUpjA: "
    ], 
    "stdout": "Certificate file /etc/nginx/ssl/letsencrypt/greenamericarecycling.com-72821e5-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.\nCertificate file /etc/nginx/ssl/letsencrypt/nrhamm.com-0f5af2e-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.\nCertificate file /etc/nginx/ssl/letsencrypt/buildex.com-dfdf251-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.\nCertificate file /etc/nginx/ssl/letsencrypt/continentalcementmulti.com-57de851-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.\nCertificate file /etc/nginx/ssl/letsencrypt/americanmaterialsco.com-32de6d5-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.\nCertificate file /etc/nginx/ssl/letsencrypt/razorbackconcrete.com-9164c9e-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.\nCertificate file /etc/nginx/ssl/letsencrypt/georgiastoneproducts.com-e74bb25-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.\nCertificate file /etc/nginx/ssl/letsencrypt/rdje.com-a6dee22-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.\nCertificate file /etc/nginx/ssl/letsencrypt/mainlandcmmulti.com-774b24d-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.", 
    "stdout_lines": [
        "Certificate file /etc/nginx/ssl/letsencrypt/greenamericarecycling.com-72821e5-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.", 
        "Certificate file /etc/nginx/ssl/letsencrypt/nrhamm.com-0f5af2e-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.", 
        "Certificate file /etc/nginx/ssl/letsencrypt/buildex.com-dfdf251-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.", 
        "Certificate file /etc/nginx/ssl/letsencrypt/continentalcementmulti.com-57de851-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.", 
        "Certificate file /etc/nginx/ssl/letsencrypt/americanmaterialsco.com-32de6d5-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.", 
        "Certificate file /etc/nginx/ssl/letsencrypt/razorbackconcrete.com-9164c9e-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.", 
        "Certificate file /etc/nginx/ssl/letsencrypt/georgiastoneproducts.com-e74bb25-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.", 
        "Certificate file /etc/nginx/ssl/letsencrypt/rdje.com-a6dee22-bundled.cert already exists and is younger than 60 days. Not creating a new certificate.", 
        "Certificate file /etc/nginx/ssl/letsencrypt/mainlandcmmulti.com-774b24d-bundled.cert already exists and is younger than 60 days. Not creating a new certificate."
    ]
}

It looks like this was an issue on the engineering side. They added the DNS for the normal domain but with www it was pointing to another ip. I will reply again once the DNS has propagated.

I just ran the provision again with the DNS changes and still having the same error that is showing above. I dont understand why this works fine on staging but now on the production server

There’s no difference from Trellis’ point of view for any environment (staging vs production); it’s really all the same. The differences, or errors, could come from either Let’s Encrypt, DNS, or something on the server getting into a weird state.

In this case it looks like you have a lot of domains which all work fine except for one. You could try seeing if there’s any files/certificates for that domain only left behind and manually delete them in the following paths:

  • etc/nginx/ssl/letsencrypt/
  • /var/lib/letsencrypt
1 Like

Would I remove those locally or out on the production server?

You’d remove them on the remote production server.

Yes I just did that and provisioned again and this solved my issue. Thank you for the help saved me a major headache.

1 Like

This topic was automatically closed after 42 days. New replies are no longer allowed.