Trellis 0.9.7 connecting as admin on initial server.yml. Fails - unreachable

I was trying to update Trellis from 0.9.6 to 0.9.7 and ran into a problem when attempting to provision the server on a brand new droplet. I’ve been rebuilding droplets like crazy trying to figure this out. It seems that with 0.9.7 on the initial provisioning TASK [remote-user] chooses to connect as admin and from there fails to connect via SSH.

SSH keys are initialized on the droplet and have been working fine on my Trellis 0.9.6 setup.

I’ve tried this using both

use sshd_permit_root_login:true

and

use sshd_permit_root_login: false

I run:

ansible-playbook server.yml -e env=staging

Resulting in:

PLAY [Ensure necessary variables are defined] **********************************

TASK [Ensure environment is defined] *******************************************
skipping: [localhost]

PLAY [Determine Remote User] ***************************************************

TASK [remote-user : Determine whether to connect as root or admin_user] ********
The authenticity of host 'staging.bigsignalmusic.com (104.131.140.76)' can't be established.
ECDSA key fingerprint is SHA256:/U8UXf5VioAi94krQimTwJEIUbeVaXEMcmKuMap24MM.
Are you sure you want to continue connecting (yes/no)? yes
ok: [staging.bigsignalmusic.com -> localhost]

TASK [remote-user : Set remote user for each host] *****************************
ok: [staging.bigsignalmusic.com]

TASK [remote-user : Announce which user was selected] **************************
ok: [staging.bigsignalmusic.com] => {
    "msg": "Note: Ansible will attempt connections as user = root"
}

All good!

However on 0.9.7 with the exact same settings (I think!) I’m getting this:

PLAY [Ensure necessary variables are defined] **********************************

TASK [Ensure environment is defined] *******************************************
skipping: [localhost]

PLAY [Determine Remote User] ***************************************************

TASK [remote-user : Determine whether to connect as root or admin_user] ********
ok: [staging.bigsignalmusic.com -> localhost]

TASK [remote-user : Set remote user for each host] *****************************
ok: [staging.bigsignalmusic.com]

TASK [remote-user : Announce which user was selected] **************************
Note: Ansible will attempt connections as user = admin
ok: [staging.bigsignalmusic.com]

PLAY [WordPress Server - Install LEMP Stack with PHP 7.0 and MariaDB MySQL] ****

TASK [setup] *******************************************************************
The authenticity of host 'staging.bigsignalmusic.com (104.131.140.76)' can't be established.
ECDSA key fingerprint is SHA256:/U8UXf5VioAi94krQimTwJEIUbeVaXEMcmKuMap24MM.
Are you sure you want to continue connecting (yes/no)? yes
System info:
  Ansible 2.0.0.2; Darwin
  Trellis at "Add connection-related cli options to ping command"
---------------------------------------------------
ERROR! SSH encountered an unknown error during the connection. We recommend
you re-run the command using -vvvv, which will enable SSH debugging output to
help diagnose the issue
fatal: [staging.bigsignalmusic.com]: UNREACHABLE! => {"changed": false, "unreachable": true}

PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=0   
staging.bigsignalmusic.com : ok=3    changed=0    unreachable=1    failed=0

Any ideas? Thanks

Think there’s been fixes related to this since:

I’d try merging in those and see if it works.

Thanks for the reply Scott. I just tried pulling down the very latest Trellis Latest commit 30ec2d9 from github but no dice. The same error persists for me. I guess I’ll stick with my older version that is working for the moment.

I’m having the same issue on a project that was deploying to a DO droplet successfully earlier before the upgrade - just pulled all the latest changes up to 30ec2d9f and have the same situation.

Here’s the -vvvv output on the error:

ERROR! SSH encountered an unknown error. The output was: OpenSSH_6.9p1, LibreSSL 2.1.8 debug1: Reading configuration data /Users/USERNAME/.ssh/config debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 20: Applying options for * debug1: /etc/ssh/ssh_config line 102: Applying options for * debug1: auto-mux: Trying existing master debug1: Control socket "/Users/USERNAME/.ansible/cp/ansible- ssh-45.XXX.XXX.XXX-22-admin" does not exist debug2: ssh_connect: needpriv 0 debug1: Connecting to 45.XXX.XXX.XXX [45.XXX.XXX.XXX] port 22. debug2: fd 3 setting O_NONBLOCK debug1: fd 3 clearing O_NONBLOCK debug1: Connection established. debug3: timeout: 9854 ms remain after connect debug1: identity file /Users/USERNAME/.ssh/id_rsa type 1 debug1: key_load_public: No such file or directory debug1: identity file /Users/USERNAME/.ssh/id_rsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/USERNAME/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/USERNAME/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/USERNAME/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/USERNAME/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/USERNAME/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/USERNAME/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.9 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6 debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6 pat OpenSSH_6.6.1* compat 0x04000000 debug2: fd 3 setting O_NONBLOCK debug1: Authenticating to 45.XXX.XXX.XXX:22 as 'admin' debug3: hostkeys_foreach: reading file "/Users/USERNAME/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /Users/USERNAME/.ssh/known_hosts:152 debug3: load_hostkeys: loaded 1 keys from 45.XXX.XXX.XXX debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa- sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com ,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa- sha2-nistp384,ecdsa-sha2-nistp521 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256 ,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256 ,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie- hellman-group1-sha1 debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa- sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com ,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh- ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss- cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss- cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ct r,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour 128,aes128-cbc,3des-cbc,blowfish- cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ct r,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour 128,aes128-cbc,3des-cbc,blowfish- cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com ,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac- sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256 ,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac- ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac- md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com ,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com ,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac- sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256 ,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac- ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac- md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com ,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256 ,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256 ,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie- hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519 debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfou r128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh. com,aes128-cbc,3des-cbc,blowfish- cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfou r128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh. com,aes128-cbc,3des-cbc,blowfish- cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac- sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac- sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac- ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac- md5-96-etm@openssh.com,hmac-md5,hmac- sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512 ,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac- sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac- sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac- ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac- md5-96-etm@openssh.com,hmac-md5,hmac- sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512 ,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: none,zlib@openssh.com debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug1: kex: server->client chacha20-poly1305@openssh.com <implicit> zlib@openssh.com debug1: kex: client->server chacha20-poly1305@openssh.com <implicit> zlib@openssh.com debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ecdsa-sha2-nistp256 SHA256:SeQWdzr28OgIETfSudcL01ZTBt7xR2b4qcLjJzfVh20 debug3: hostkeys_foreach: reading file "/Users/USERNAME/.ssh/known_hosts" debug3: record_hostkey: found key type ECDSA in file /Users/USERNAME/.ssh/known_hosts:152 debug3: load_hostkeys: loaded 1 keys from 45.XXX.XXX.XXX debug1: Host '45.XXX.XXX.XXX' is known and matches the ECDSA host key. debug1: Found key in /Users/USERNAME/.ssh/known_hosts:152 debug2: set_newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug2: set_newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: SSH2_MSG_SERVICE_REQUEST sent debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug2: key: /Users/USERNAME/.ssh/id_rsa (0x7fac8b600f20), debug2: key: /Users/USERNAME/.ssh/id_dsa (0x0), debug2: key: /Users/USERNAME/.ssh/id_ecdsa (0x0), debug2: key: /Users/USERNAME/.ssh/id_ed25519 (0x0), debug1: Authentications that can continue: publickey,password debug3: start over, passed a different list publickey,password debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey debug3: authmethod_lookup publickey debug3: remaining preferred: ,gssapi-keyex,hostbased,publickey debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /Users/USERNAME/.ssh/id_rsa debug3: send_pubkey_test debug2: we sent a publickey packet, wait for reply debug1: Authentications that can continue: publickey,password debug1: Trying private key: /Users/USERNAME/.ssh/id_dsa debug3: no such identity: /Users/USERNAME/.ssh/id_dsa: No such file or directory debug1: Trying private key: /Users/USERNAME/.ssh/id_ecdsa debug3: no such identity: /Users/USERNAME/.ssh/id_ecdsa: No such file or directory debug1: Trying private key: /Users/USERNAME/.ssh/id_ed25519 debug3: no such identity: /Users/USERNAME/.ssh/id_ed25519: No such file or directory debug2: we did not send a packet, disable method debug1: No more authentication methods to try. Permission denied (publickey,password).

I’ve double checked that the public key is set correctly in group_vars and ssh-ing directly to the droplet works fine. It looks like it’s trying out the public key but not getting a response for some reason?

The similar fix that jankups reported here Server.yml and SSH error also doesn’t work in my case, unfortunately.

@davidmichelruddy and @nickkeenan: If possible, could you upgrade to Ansible 2.0.2 and see if that resolves the problem?

In Ansible 2.0.2, Ansible’s internal variable ansible_user is either undefined or is a real user name. This is the behavior Trellis expects and for which Trellis has a default user. However, in 2.0.0.2 and 2.0.1, instead of being undefined, ansible_user == ' ' so Trellis doesn’t supply the default root. This means the test connection uses the username ' ', which fails, so Trellis falls back to the admin_user, etc.

We’ll push a fix shortly to either require Ansible 2.0.2+ or replace the ' ' user with root. Thanks for reporting the problem!

1 Like

Upgraded to Ansible 2.0.2 and all is working as expected. Thank you!

1 Like

Thanks, @fullyint! That was the issue on my end, and with Ansible 2.0.2 I’m back deploying like a champ. Seriously doing the happy dance over here (we were upgrading to take advantage of the Let’s Encrypt feature, and having that work the first time was just… really really satisfying.)

2 Likes

Trellis now requires Ansible >= 2.0.2.0: https://github.com/roots/trellis/pull/579