Failed to connect to the host via ssh: admin@xxx.xxx.xxx.xxx: Permission denied (publickey)

Been trying to set up a staging server on a brand new Droplet for a while now. Cannot get it to provision while the production server and local site has been up and running for some time now. I get the error that the public key is not acccepted for the user admin. Well DO only sets up root initially and Trellis would set up admin no? So why can’t I get passed all this:

 TASK [connection : Load become password] ************************************************************
task path: /Users/jasper/webdesign/domain.com/trellis/roles/connection/tasks/main.yml:50
ok: [xxx.xxx.xxx.xxx] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
META: ran handlers
META: ran handlers

PLAY [Install prerequisites] ************************************************************************
META: ran handlers

TASK [Install Python 2.x] ***************************************************************************
task path: /Users/jasper/webdesign/domain.com/trellis/server.yml:17
<xxx.xxx.xxx.xxx> ESTABLISH SSH CONNECTION FOR USER: admin
<xxx.xxx.xxx.xxx> SSH: EXEC ssh -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o HostKeyAlgorithms=ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=admin -o ConnectTimeout=10 -o ControlPath=/Users/jasper/.ansible/cp/3fbaecdf22 -tt xxx.xxx.xxx.xxx 'sudo -H -S  -p "[sudo via ansible, key=bpnidmdvrffjwxyldycmsqwpubhbmryr] password: " -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-bpnidmdvrffjwxyldycmsqwpubhbmryr; which python || sudo apt-get update && sudo apt-get install -qq -y python-simplejson'"'"''
<xxx.xxx.xxx.xxx> (255, '', 'admin@xxx.xxx.xxx.xxx: Permission denied (publickey).\r\n')
System info:
  Ansible 2.3.2.0; Darwin
  Trellis at "Accommodate deploy hook vars formatted as lists of includes"
---------------------------------------------------
Failed to connect to the host via ssh: admin@xxx.xxx.xxx.xxx: Permission
denied (publickey).

fatal: [xxx.xxx.xxx.xxx]: UNREACHABLE! => {
    "changed": false, 
    "unreachable": true
}
	to retry, use: --limit @/Users/jasper/webdesign/publiqly.com/trellis/server.retry

PLAY RECAP ******************************************************************************************
xxx.xxx.xxx.xxx            : ok=4    changed=0    unreachable=1    failed=0   
localhost                  : ok=0    changed=0    unreachable=0    failed=0

And here the verbose log:

Failed to connect to the host via ssh: OpenSSH_7.6p1, LibreSSL 2.6.2
debug1: Reading configuration data /Users/jasper/.ssh/config
debug1: /Users/jasper/.ssh/config line 69: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Control socket "/Users/jasper/.ansible/cp/7067d6e622" does not exist
debug2: resolving "xxx.xxx.x.xxx" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to xxx.xxx.x.xxx [xxx.xxx.x.xxx] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 9483 ms remain after connect
debug1: identity file /Users/jasper/.ssh/id_rsa type 0
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jasper/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jasper/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jasper/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jasper/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jasper/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jasper/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/jasper/.ssh/id_ed25519-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2
Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to xxx.xxx.x.xxx:22 as 'admin'
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-
sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-
exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-
hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ssh-rsa-
cert-v01@openssh.com,ssh-ed25519,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes
256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes
256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-
sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256
,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-
sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256
,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-
sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-
hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-
sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes
256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes
256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-
sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256
,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-
sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256
,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC:
<implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC:
<implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-ed25519
SHA256:OjjAsHwOui3Ql1XN+VQBNIB/dR9zgU60rrH4oHl8SnM
debug3: hostkeys_foreach: reading file "/Users/jasper/.ssh/known_hosts"
debug3: record_hostkey: found key type ED25519 in file
/Users/jasper/.ssh/known_hosts:85
debug3: load_hostkeys: loaded 1 keys from xxx.xxx.x.xxx
debug1: Host 'xxx.xxx.x.xxx' is known and matches the ED25519 host key.
debug1: Found key in /Users/jasper/.ssh/known_hosts:85
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /Users/jasper/.ssh/id_rsa (0x7ff284500150), agent
debug2: key: /Users/jasper/.ssh/id_dsa (0x0)
debug2: key: /Users/jasper/.ssh/id_ecdsa (0x0)
debug2: key: /Users/jasper/.ssh/id_ed25519 (0x0)
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred: ,gssapi-keyex,hostbased,publickey
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: RSA
SHA256:ArFCaVQs4Kf9z+k6cTecYuHO61GpaS3LgJEZMpOdZvM /Users/jasper/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: /Users/jasper/.ssh/id_dsa
debug3: no such identity: /Users/jasper/.ssh/id_dsa: No such file or
directory
debug1: Trying private key: /Users/jasper/.ssh/id_ecdsa
debug3: no such identity: /Users/jasper/.ssh/id_ecdsa: No such file or
directory
debug1: Trying private key: /Users/jasper/.ssh/id_ed25519
debug3: no such identity: /Users/jasper/.ssh/id_ed25519: No such file or
directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
admin@xxx.xxx.x.xxx: Permission denied (publickey).

fatal: [xxx.xxx.x.xxx]: UNREACHABLE! => {
    "changed": false, 
    "unreachable": true
}
	to retry, use: --limit @/Users/jasper/webdesign/domain.com/trellis/server.retry

PLAY RECAP ******************************************************************************************
xxx.xxx.x.xxx              : ok=4    changed=0    unreachable=1    failed=0   
localhost                  : ok=0    changed=0    unreachable=0    failed=0

PS with ansible_user=root I do manage, but that causes other issues later on…

When I remove known hosts and root into the server I get

ssh root@staging.domain.com
The authenticity of host 'staging.domain.com (xxx.xxx.x.xxx)' can't be established.
ECDSA key fingerprint is SHA256:rrOwnKUvZ825dQKvm93QClAedbcnAvbN9mUCLdQ8FE8.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'staging.domain.com,xxx.xxx.x.xxx' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-103-generic x86_64)

and then on trying to provision I get

WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

If this change in host keys is expected (e.g., if you rebuilt the server
or if the Trellis sshd role made changes recently), then run the following
command to clear the old host key from your known_hosts.

  ssh-keygen -R xxx.xxx.x.xxx

Then try your Trellis playbook or SSH connection again.

If the change is unexpected, cautiously consider why the host identification
may have changed and whether you may be victim to a man-in-the-middle attack.

---------------------------------------------------
The fingerprint for the ED25519 key sent by the remote host is

SHA256:OjjAsHwOui3Ql1XN+VQBNIB/dR9zgU60rrH4oHl8SnM.

Add correct host key in /Users/jasper/.ssh/known_hosts to get rid of this
message.

Offending ECDSA key in /Users/jasper/.ssh/known_hosts:84

ED25519 host key for xxx.xxx.x.xxx has changed and you have requested strict
checking.

Host key verification failed.

fatal: [xxx.xxx.x.xxx]: FAILED! => {
    "changed": false, 
    "failed": true
}

So it seems there are two different fingerprints… very odd… And when I accept if for root and try to provision again post ssh-keygen -R xxx.xxx.x.xxx I get to hear access is denied as the earlier post showed.

Also , in the auth.log I saw Dec 27 07:49:12 domain-staging sshd[2013]: input_userauth_request: invalid user admin [preauth]

The “Trellis at” portion serves as an indicator of your Trellis version. That particular entry in the Trellis CHANGELOG is older than the entry “Ansible 2.3 compatibility (#813).” That PR’s comment mentions…

One commit prevents Ansible’s new dense.py callback from causing Trellis ssh connection tests to always show root as failing to connect.

It appears you are using Ansible 2.3.2 without related compatibility updates for Trellis. I think you either need to rollback your Ansible using pip install ansible==2.2.3.0 or update your Trellis. Then do the ssh-keygen -R again for both the IP and domain and try to provision.

When you SSHed manually to staging.domain.com, your ssh client defaults to loading the server’s ECDSA key into your local machine’s known_hosts, specifically in an entry for staging.domain.com. Then when you provisioned with Trellis, it saw only that it was connecting to a certain IP (in your hosts/staging file) and that this IP was not already in your known_hosts (only the domain was there). So Trellis asked the server to provide the more secure ED25519 key type but your ssh client realized now you were trying two different host keys and gave you the warning. If you use versions of Ansible and Trellis that are compatible, I doubt you will encounter this issue again.

1 Like

Thank you so much @fullyint ! Doing a setup with Ansible downgrade now. And it is running! Never thought it was related to the Trellis connect test always showing root as failing to connect.

Will upgrade to latest Trellis so I can avoid these issues and use the latest Ansible in the future. And to have a peace of mind. Been doing so much yesterday with ssh-add -l, ssh-add -D, ssh-add -K, ssh-keygen -R, ECDSA and reading on SSH and SSHD that I was going mental…Thanks again!

2 Likes