Trellis deploy fails on "Clone project files" with the 1Password SSH agent

I’m very perplexed by this. I’ve been switching over to using 1Password’s SSH agent on my Mac. Most things work fine, but deploying my Trellis projects now fails at the “TASK [deploy : Clone project files]” step with a message signing failed for ED25519 \"GitHub/GitLab SSH Auth Key\" from agent: communication with agent failed" . This is happening on multiple servers, including ones that used to work fine before I tried to put my GitHub key in 1P.

What I’ve tested and know so far:

  • I can SSH into the servers manually with the 1Password agent, no problem.
  • While in the server, I can ssh -T git@github.com and 1Password prompts me to authorize the proper key, and it connects just fine — so agent forwarding works.
  • I can also (in the server) run a git clone or pull from my repo manually, and the SSH key from 1Password is used.
  • trellis provision <environment> works perfectly.
  • Debugging with trellis deploy --verbose shows that Ansible is somehow seeing the proper key from 1Password, but it can’t communicate with the agent.

Here’s the relevant portion of my ~/.ssh/config:

Host 161.35.188.103
  IdentityAgent ~/.1password/agent.sock
  ForwardAgent yes

# USED BY GIT
Host github.com gitlab.com
  IdentityAgent ~/.1password/agent.sock

And full debug output from the failing task:

TASK [deploy : Clone project files] ********************************************
task path: /Users/andron/Code/andronocean/trellis-wp-sites/trellis/roles/deploy/tasks/update.yml:26
Using module file /Users/andron/Code/andronocean/trellis-wp-sites/trellis/.trellis/virtualenv/lib/python3.10/site-packages/ansible/modules/git.py
Pipelining is enabled.
<161.35.188.103> ESTABLISH SSH CONNECTION FOR USER: web
<161.35.188.103> SSH: EXEC ssh -vvv -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o HostKeyAlgorithms=ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="web"' -o ConnectTimeout=10 -o 'ControlPath="/Users/andron/.ansible/cp/5778993b96"' 161.35.188.103 '/bin/sh -c '"'"'/usr/bin/python3 && sleep 0'"'"''
<161.35.188.103> (1, b'\n{"cmd": "/usr/bin/git ls-remote git@github.com:andronocean/wordpress-sandbox-bedrock.git -h refs/heads/master", "rc": 128, "stdout": "", "stderr": "sign_and_send_pubkey: signing failed for ED25519 \\"GitHub/GitLab SSH Auth Key\\" from agent: communication with agent failed\\r\\ngit@github.com: Permission denied (publickey).\\r\\nfatal: Could not read from remote repository.\\n\\nPlease make sure you have the correct access rights\\nand the repository exists.\\n", "failed": true, "msg": "sign_and_send_pubkey:********@github.com: Permission denied (publickey).\\r\\nfatal: Could not read from remote repository.\\n\\nPlease make sure you have the correct access rights\\nand the repository exists.", "invocation": {"module_args": {"repo": "git@github.com:andronocean/wordpress-sandbox-bedrock.git", "dest": "/srv/www/sandbox.andronocean.com/shared/source", "version": "master", "accept_hostkey": true, "force": true, "remote": "origin", "clone": true, "update": true, "verify_commit": false, "gpg_whitelist": [], "accept_newhostkey": false, "bare": false, "recursive": true, "single_branch": false, "track_submodules": false, "refspec": null, "reference": null, "depth": null, "key_file": null, "ssh_opts": null, "executable": null, "umask": null, "archive": null, "archive_prefix": null, "separate_git_dir": null}}}\n', b"OpenSSH_9.3p1, OpenSSL 1.1.1t  7 Feb 2023\r\ndebug1: Reading configuration data /Users/andron/.ssh/config\r\ndebug1: /Users/andron/.ssh/config line 30: Applying options for 161.35.188.103\r\ndebug1: Reading configuration data /usr/local/etc/ssh/ssh_config\r\ndebug2: resolve_canonicalize: hostname 161.35.188.103 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/andron/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/andron/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 8962\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n")
<161.35.188.103> Failed to connect to the host via ssh: OpenSSH_9.3p1, OpenSSL 1.1.1t  7 Feb 2023
debug1: Reading configuration data /Users/andron/.ssh/config
debug1: /Users/andron/.ssh/config line 30: Applying options for 161.35.188.103
debug1: Reading configuration data /usr/local/etc/ssh/ssh_config
debug2: resolve_canonicalize: hostname 161.35.188.103 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/andron/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/andron/.ssh/known_hosts2'
debug1: auto-mux: Trying existing master
debug2: fd 3 setting O_NONBLOCK
debug2: mux_client_hello_exchange: master version 4
debug3: mux_client_forwards: request forwardings: 0 local, 0 remote
debug3: mux_client_request_session: entering
debug3: mux_client_request_alive: entering
debug3: mux_client_request_alive: done pid = 8962
debug3: mux_client_request_session: session request sent
debug1: mux_client_request_session: master session id: 2
debug3: mux_client_read_packet: read header failed: Broken pipe
debug2: Received exit status from master 1
fatal: [161.35.188.103]: FAILED! => {
    "changed": false,
    "cmd": "/usr/bin/git ls-remote git@github.com:andronocean/wordpress-sandbox-bedrock.git -h refs/heads/master",
    "invocation": {
        "module_args": {
            "accept_hostkey": true,
            "accept_newhostkey": false,
            "archive": null,
            "archive_prefix": null,
            "bare": false,
            "clone": true,
            "depth": null,
            "dest": "/srv/www/sandbox.andronocean.com/shared/source",
            "executable": null,
            "force": true,
            "gpg_whitelist": [],
            "key_file": null,
            "recursive": true,
            "reference": null,
            "refspec": null,
            "remote": "origin",
            "repo": "git@github.com:andronocean/wordpress-sandbox-bedrock.git",
            "separate_git_dir": null,
            "single_branch": false,
            "ssh_opts": null,
            "track_submodules": false,
            "umask": null,
            "update": true,
            "verify_commit": false,
            "version": "master"
        }
    },
    "msg": "sign_and_send_pubkey:********@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.",
    "rc": 128,
    "stderr": "sign_and_send_pubkey: signing failed for ED25519 \"GitHub/GitLab SSH Auth Key\" from agent: communication with agent failed\r\ngit@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n",
    "stderr_lines": [
        "sign_and_send_pubkey: signing failed for ED25519 \"GitHub/GitLab SSH Auth Key\" from agent: communication with agent failed",
        "git@github.com: Permission denied (publickey).",
        "fatal: Could not read from remote repository.",
        "",
        "Please make sure you have the correct access rights",
        "and the repository exists."
    ],
    "stdout": "",
    "stdout_lines": []
}
...ignoring

TASK [deploy : Failed connection to remote repo] *******************************
task path: /Users/andron/Code/andronocean/trellis-wp-sites/trellis/roles/deploy/tasks/update.yml:37
fatal: [161.35.188.103]: FAILED! => {
    "changed": false,
    "msg": "Git repo git@github.com:andronocean/wordpress-sandbox-bedrock.git on branch master cannot be accessed. Please verify the repository/branch are correct and you have SSH forwarding set up correctly.\nMore info:\n> https://roots.io/trellis/docs/deploys/#ssh-keys\n> https://roots.io/trellis/docs/ssh-keys/#cloning-remote-repo-using-ssh-agent-forwarding\n\nError:\nsign_and_send_pubkey:********@github.com: Permission denied (publickey).\r\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n"
}

I’ve tried on projects with Trellis v1.21.0 and v1.20.1. trellis-cli is version 1.11.0.

Has anyone else tried the 1Password SSH agent and gotten it to work? Right now I’m thinking this is an incompatibility between it and something in Ansible’s git system…

1 Like

Great detail in your post :+1:.

If ssh works, I can’t imagine this problem is an unfixable compatibility issue with 1Password.

This sounds very similar to this ansible issue from 2017.

Seems like people have had success with setting ControlPersist.

Or @straris’ fix by toggling pipelining.

There appear to be issues with newer SSH libraries connecting to older ones. Perhaps 1Password has bumped the library version?

:thinking: This may be a bit on a tangent (as this is about KeeAgent (KeePass) + WSL), but the SSH debugging tips in this guide may be helpful.

So you would try ssh-add -l in the terminal from which you would also run ansible.
Does that work?

Changing ControlPersist and pipelining don’t make any difference, unfortunately. I get the same exact error.

You could be on to something with the SSH version incompatibility though. ssh -V on my Mac shows OpenSSH_9.3p1, OpenSSL 1.1.1t 7 Feb 2023 (I get mine from Homebrew since Apple’s bundled ssh used to be outdated and nonstandard).

Meanwhile both my servers are on Ubuntu 20.04, and they show ssh -V as OpenSSH_8.2p1 Ubuntu-4ubuntu0.5, OpenSSL 1.1.1f 31 Mar 2020.

One of them is on Kinsta so I can’t update that :disappointed:

When I have a chance I’ll see if updating the server SSH changes anything.

Unfortunately not. By default it works with the normal ssh-agent, not the 1Password one. I did wonder whether this was part of the problem, so I tried changing SSH_AUTH_SOCK manually before a deploy to point to the 1P agent (just in case IdentityAgent in ssh config wasn’t doing the trick), but it still doesn’t work:

export SSH_AUTH_SOCK=~/Library/Group\ Containers/2BUA8C4S2C.com.1password/t/agent.sock
trellis deploy etc....

What do you get when you invoke ssh-add -l to check the connection to the SSH agent (run that command in the terminal you use for ansible)?

OK, some progress! I’ve isolated the problem further. I created a new local SSH key in ~/.ssh, added it to GitHub and the server, added it to my normal SSH agent, and tested permutations of my ~/.ssh/config

If I set the server host config to use 1Password’s agent, deployment fails regardless of whether I have the GitHub connection using 1Password or my normal SSH agent.

If I set my server to use the normal SSH agent, then the SSH config for github.com can be pointed to 1Password’s agent, and deploy succeeds. New non-1P key works too.

In other words, this ~/.ssh/config works:

Host 161.35.188.103
  AddKeysToAgent yes
  UseKeychain yes
  IdentityFile ~/.ssh/id_ed25519
  ForwardAgent yes

# USED BY GIT
Host github.com gitlab.com
  IdentityAgent ~/.1password/agent.sock

However Ansible is setting up the initial SSH connection, somehow it doesn’t play well with 1Password’s agent forwarding. I’m kind of doubtful now that it’s a compatibility problem between my ssh and the server’s, since if that were the case it seems like this should also fail. I think I need to take this to 1Password support.


@strarsis:

What do you get when you invoke ssh-add -l to check the connection to the SSH agent (run that command in the terminal you use for ansible)?

By default, nothing (no keys added… well at least before I did what I described in this post). If I switch to the 1Password socket (SSH_AUTH_SOCK=~/Library/Group\ Containers/2BUA8C4S2C.com.1password/t/agent.sock ssh-add -l then I get a list of all my keys in 1Password’s agent.

1 Like

Edit: ForwardAgent=yes is already set for Trellis: