Trellis deploy fails with Composer commands, issue with SSH-config?

Hi there,

we are currently trying to deploy to a non-Trellis server (running Plesk) and are facing issues when it comes to the TASK [deploy : Run composer check]

The error isn’t tied to a specific composer task as we have already set the composer_platform_requirements_check to false and then the subsequent composer task will fail which leads us to believe that there could be something off with regards to the SSH connection?

In order to better debug I have provisioned a regular Trellis-based server instance (where the commands work fine) so we can directly compare the outputs:

Run composer check (FAIL)
TASK [deploy : Run composer check] *********************************************
task path: /local/path/htdocs/dskv/trellis/roles/deploy/hooks/build-after.yml:55
redirecting (type: modules) ansible.builtin.composer to community.general.composer
redirecting (type: modules) ansible.builtin.composer to community.general.composer
Using module file /local/path/htdocs/dskv/trellis/.trellis/virtualenv/lib/python3.9/site-packages/ansible_collections/community/general/plugins/modules/composer.py
Pipelining is enabled.
<our-staging-domain.xyz> ESTABLISH SSH CONNECTION FOR USER: dskv
<our-staging-domain.xyz> SSH: EXEC ssh -vvv -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o HostKeyAlgorithms=ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="dskv"' -o ConnectTimeout=10 -o 'ControlPath="/local/user/.ansible/cp/f5faf09ba7"' our-staging-domain.xyz '/bin/sh -c '"'"'/usr/bin/python3 && sleep 0'"'"''
<our-staging-domain.xyz> (1, b'', b'OpenSSH_8.6p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /local/user/.ssh/config\r\ndebug1: /local/user/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/local/user/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/local/user/.ssh/known_hosts2\'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 285\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\nTraceback (most recent call last):\n  File "<stdin>", line 107, in <module>\n  File "<stdin>", line 99, in _ansiballz_main\n  File "<stdin>", line 47, in invoke_module\n  File "/usr/lib/python3.10/runpy.py", line 224, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code\n    exec(code, run_globals)\n  File "/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible_collections/community/general/plugins/modules/composer.py", line 275, in <module>\n  File "/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible_collections/community/general/plugins/modules/composer.py", line 221, in main\n  File "/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible_collections/community/general/plugins/modules/composer.py", line 169, in get_available_options\n  File "/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible/module_utils/basic.py", line 1469, in from_json\n  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads\n    return _default_decoder.decode(s)\n  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode\n    raise JSONDecodeError("Expecting value", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n')
<our-staging-domain.xyz> Failed to connect to the host via ssh: OpenSSH_8.6p1, LibreSSL 3.3.6
debug1: Reading configuration data /local/user/.ssh/config
debug1: /local/user/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/local/user/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/local/user/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 285
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
Traceback (most recent call last):
  File "<stdin>", line 107, in <module>
  File "<stdin>", line 99, in _ansiballz_main
  File "<stdin>", line 47, in invoke_module
  File "/usr/lib/python3.10/runpy.py", line 224, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/usr/lib/python3.10/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible_collections/community/general/plugins/modules/composer.py", line 275, in <module>
  File "/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible_collections/community/general/plugins/modules/composer.py", line 221, in main
  File "/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible_collections/community/general/plugins/modules/composer.py", line 169, in get_available_options
  File "/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible/module_utils/basic.py", line 1469, in from_json
  File "/usr/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1
fatal: [our-staging-domain.xyz]: FAILED! => {
    "changed": false,
    "module_stderr": "OpenSSH_8.6p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /local/user/.ssh/config\r\ndebug1: /local/user/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/local/user/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/local/user/.ssh/known_hosts2'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 285\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\nTraceback (most recent call last):\n  File \"<stdin>\", line 107, in <module>\n  File \"<stdin>\", line 99, in _ansiballz_main\n  File \"<stdin>\", line 47, in invoke_module\n  File \"/usr/lib/python3.10/runpy.py\", line 224, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/lib/python3.10/runpy.py\", line 96, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/lib/python3.10/runpy.py\", line 86, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible_collections/community/general/plugins/modules/composer.py\", line 275, in <module>\n  File \"/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible_collections/community/general/plugins/modules/composer.py\", line 221, in main\n  File \"/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible_collections/community/general/plugins/modules/composer.py\", line 169, in get_available_options\n  File \"/tmp/ansible_composer_payload_qwqerfva/ansible_composer_payload.zip/ansible/module_utils/basic.py\", line 1469, in from_json\n  File \"/usr/lib/python3.10/json/__init__.py\", line 346, in loads\n    return _default_decoder.decode(s)\n  File \"/usr/lib/python3.10/json/decoder.py\", line 337, in decode\n    obj, end = self.raw_decode(s, idx=_w(s, 0).end())\n  File \"/usr/lib/python3.10/json/decoder.py\", line 355, in raw_decode\n    raise JSONDecodeError(\"Expecting value\", s, err.value) from None\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}
Run composer check (SUCCESS)
TASK [deploy : Run composer check] *********************************************
task path: /local/path/htdocs/dskv/trellis/roles/deploy/hooks/build-after.yml:55
redirecting (type: modules) ansible.builtin.composer to community.general.composer
redirecting (type: modules) ansible.builtin.composer to community.general.composer
Using module file /local/path/htdocs/dskv/trellis/.trellis/virtualenv/lib/python3.9/site-packages/ansible_collections/community/general/plugins/modules/composer.py
Pipelining is enabled.
<188.34.xxx.xxx> ESTABLISH SSH CONNECTION FOR USER: dskv
<188.34.xxx.xxx> SSH: EXEC ssh -vvv -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o HostKeyAlgorithms=ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="dskv"' -o ConnectTimeout=10 -o 'ControlPath="/local/user/.ansible/cp/3b7368bb7f"' 188.34.xxx.xxx '/bin/sh -c '"'"'/usr/bin/python3 && sleep 0'"'"''
<188.34.xxx.xxx> (0, b'\n{"changed": true, "msg": "composer-plugin-api 2.3.0 success ext-ctype * success provided by symfony/polyfill-ctype ext-pcre 8.1.21 success php 8.1.21 success No vendor dir present, checking non-dev platform requirements from the lock file", "stdout": "composer-plugin-api 2.3.0    success                                    \\next-ctype           *        success provided by symfony/polyfill-ctype \\next-pcre            8.1.21   success                                    \\nphp                 8.1.21   success                                    \\nNo vendor dir present, checking non-dev platform requirements from the lock file\\n", "invocation": {"module_args": {"command": "check-platform-reqs", "working_dir": "/var/www/vhosts/staging-dskv.de/httpdocs/deutscherskatverband.de/staging/dskv/releases/20230721091609", "arguments": "", "global_command": false, "prefer_source": false, "prefer_dist": false, "no_dev": true, "no_scripts": false, "no_plugins": false, "apcu_autoloader": false, "optimize_autoloader": true, "classmap_authoritative": false, "ignore_platform_reqs": false, "executable": null, "composer_executable": null}}}\n', b"OpenSSH_8.6p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /local/user/.ssh/config\r\ndebug1: /local/user/.ssh/config line 1: Applying options for *\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 188.34.xxx.xxx is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/local/user/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/local/user/.ssh/known_hosts2'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 97631\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
changed: [188.34.xxx.xxx] => {
    "changed": true,
    "invocation": {
        "module_args": {
            "apcu_autoloader": false,
            "arguments": "",
            "classmap_authoritative": false,
            "command": "check-platform-reqs",
            "composer_executable": null,
            "executable": null,
            "global_command": false,
            "ignore_platform_reqs": false,
            "no_dev": true,
            "no_plugins": false,
            "no_scripts": false,
            "optimize_autoloader": true,
            "prefer_dist": false,
            "prefer_source": false,
            "working_dir": "/remote/path/staging/dskv/releases/20230721091609"
        }
    },
    "msg": "composer-plugin-api 2.3.0 success ext-ctype * success provided by symfony/polyfill-ctype ext-pcre 8.1.21 success php 8.1.21 success No vendor dir present, checking non-dev platform requirements from the lock file",
    "stdout": "composer-plugin-api 2.3.0    success                                    \next-ctype           *        success provided by symfony/polyfill-ctype \next-pcre            8.1.21   success                                    \nphp                 8.1.21   success                                    \nNo vendor dir present, checking non-dev platform requirements from the lock file\n",
    "stdout_lines": [
        "composer-plugin-api 2.3.0    success                                    ",
        "ext-ctype           *        success provided by symfony/polyfill-ctype ",
        "ext-pcre            8.1.21   success                                    ",
        "php                 8.1.21   success                                    ",
        "No vendor dir present, checking non-dev platform requirements from the lock file"
    ]
}

Running the SSH: EXEC-command directly yields the following:

In both cases the Authentication succeeded (publickey), the only noticable difference IMO when diffing both outputs is that the hostkeys / server keys resolve differently –

hostkeys / server keys (FAIL)

debug3: client_input_hostkeys: 2 server keys: 1 new, 0 retained, 1 incomplete match. 0 to remove
debug1: client_input_hostkeys: host key found matching a different name/address, skipping UserKnownHostsFile update

hostkeys / server keys (SUCCESS)

debug3: client_input_hostkeys: 2 server keys: 0 new, 2 retained, 0 incomplete match. 0 to remove
debug1: client_input_hostkeys: no new or deprecated keys from server

Anybody have an idea what could be the culprit here?

We compared the sshd_config and found some differentiating values but merging those also didn’t do the trick…

Your input is greatly appreciated! :pray:

Just a guess, but have you checked and removed any matching entries in ~/.ssh/known_hosts?

It seems like there may be a fingerprint match, but a different address / hostname in known_hosts.

Thanks @talss89 for chipping in!

I’ve checked and removed any previous entries in my local ~/.ssh/known_hosts-file but the error upon deployment unfortunately persists…

1 Like

If you’re using Trellis to deploy to a non-Trellis server (other than, I think, Kinsta) that’s a practice misapplication. Trellis isn’t meant to do this.

If you figure this out, please share! I ran into this exact issue trying to test using trellis only for deploying to a client’s server and could never figure it out.

So far I have successfully used it to deploy to both kinsta and a server setup with runcloud.

I am fully aware of the conceptual misalignment :sweat_smile:

We want to use a dedicated machine already in place that hosts a number of other sites and it should be able to work like this…

Assuming the issue is a minor one (some packages misconfigured or similar) then we are real close to making this happen.

OK, interesting to hear that we are not alone here :face_with_peeking_eye:

Will share any learnings for sure!

In the end we switched to using a Trellis-provisioned server – which works fine :wink:

Still wondering what the core issue is here and whether that can be overcome by tweaking the config at some point. If anyone solves this or has some more insights then please share, thanks. :pray:

Take a look at trellis-github-deployment for some quality-of-life features: