SSH key in users.yml being ingnored

aaronjpitts · November 8, 2017, 11:21pm

Hi guys, I’m trying to provision my first server. I have trellis working fine locally and I can SSH into my server on Azure just fine. But it just keeps failing to authenticate when running ansible-playbook server.yml -e env=production
my users.yml looks like the below:

# Documentation: https://roots.io/trellis/docs/ssh-keys/
admin_user: myadminusername

# Also define 'vault_users' (`group_vars/staging/vault.yml`, `group_vars/production/vault.yml`)
users:
  - name: "{{ web_user }}"
groups:
  - "{{ web_group }}"
keys:
  - "{{ lookup('file', '~/.ssh/azure.pub') }}"
  # - https://github.com/username.keys
  - name: "{{ admin_user }}"
groups:
  - sudo
keys:
  - "{{ lookup('file', '~/.ssh/azure.pub') }}"
  # - https://github.com/username.keys

web_user: web
web_group: www-data
web_sudoers:
  - "/usr/sbin/service php7.1-fpm *"

When running ansible-playbook server.yml -e env=production -vvvv I get the below:

Failed to connect to the host via ssh: OpenSSH_7.4p1, LibreSSL 2.5.0
debug1: Reading configuration data /Users/aaronpitts/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: auto-mux: Trying existing master
debug1: Control socket "/Users/aaronpitts/.ansible/cp/a5e2bd1b5c" does not
exist
debug2: resolving "40.115.59.138" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 40.115.59.138 [40.115.59.138] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 9969 ms remain after connect
debug1: identity file /Users/aaronpitts/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aaronpitts/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aaronpitts/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aaronpitts/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aaronpitts/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aaronpitts/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aaronpitts/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/aaronpitts/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2
Ubuntu-4ubuntu2.2
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 40.115.59.138:22 as 'papayapods'
debug3: hostkeys_foreach: reading file "/Users/aaronpitts/.ssh/known_hosts"
debug3: record_hostkey: found key type ED25519 in file
/Users/aaronpitts/.ssh/known_hosts:23
debug3: load_hostkeys: loaded 1 keys from 40.115.59.138
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-
ed25519-cert-v01@openssh.com,ssh-ed25519
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-
sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-
exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512
,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-
hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ssh-ed25519
,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-
sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com
,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-
sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes
256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,a
es256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes
256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,a
es256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-
sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256
,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-
sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256
,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-
sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-
hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-
sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes
256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes
256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-
sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256
,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-
sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256
,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC:
<implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC:
<implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ssh-ed25519
SHA256:PUjNuHeXREeJYJOfVh7RUU+yFNRTRvJZphT+9v9SVOs
debug3: hostkeys_foreach: reading file "/Users/aaronpitts/.ssh/known_hosts"
debug3: record_hostkey: found key type ED25519 in file
/Users/aaronpitts/.ssh/known_hosts:23
debug3: load_hostkeys: loaded 1 keys from 40.115.59.138
debug1: Host '40.115.59.138' is known and matches the ED25519 host key.
debug1: Found key in /Users/aaronpitts/.ssh/known_hosts:23
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug2: key: /Users/aaronpitts/.ssh/id_rsa (0x7ff1b37000d0), agent
debug2: key: /Users/aaronpitts/.ssh/id_dsa (0x0)
debug2: key: /Users/aaronpitts/.ssh/id_ecdsa (0x0)
debug2: key: /Users/aaronpitts/.ssh/id_ed25519 (0x0)
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred: ,gssapi-keyex,hostbased,publickey
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/aaronpitts/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: /Users/aaronpitts/.ssh/id_dsa
debug3: no such identity: /Users/aaronpitts/.ssh/id_dsa: No such file or
directory
debug1: Trying private key: /Users/aaronpitts/.ssh/id_ecdsa
debug3: no such identity: /Users/aaronpitts/.ssh/id_ecdsa: No such file or
directory
debug1: Trying private key: /Users/aaronpitts/.ssh/id_ed25519
debug3: no such identity: /Users/aaronpitts/.ssh/id_ed25519: No such file or
directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

fatal: [40.115.59.138]: UNREACHABLE! => {
    "changed": false, 
    "unreachable": true
}

I see no mention of the SSH key I defined in users.yml (azure.pub) in the error log, what am I missing?

Many thanks

aaronjpitts · November 9, 2017, 11:30am

So if I use the standard id_rsa.pub key it connects fine, so the issue here is using a key other than id_rsa.pub

fullyint · November 10, 2017, 12:59am

I imagine the indentation formatting in your group_vars/all/users.yml is actually correct and just didn’t come through when posting here, but you might double-check. However, that wouldn’t be related to the initial inability to connect.

I think that yes, attempting to use a custom key is part of the issue, but not because Trellis won’t accommodate a different key. Rather, I suspect that your SSH agent doesn’t know to use whichever private key is associated with the custom azure.pub you had on the server initially. In contrast, I think the id_rsa id_rsa.pub pair worked because that is most likely one of the keys that your agent tries by default.

When you add a key to users.yml, you are telling Trellis, “Once you’ve managed to connect, please add these public keys to the server’s authorized_keys file for the respective user(s).” In contrast, adding a key to users.yml does not tell Trellis, “When connecting to my server, please try private keys corresponding to the public keys I’ve listed here.”

So, how does Trellis know which key to use when connecting to your server? Usually Trellis (and Ansible) just leaves key selection up to your local machine’s SSH agent. Hopefully the only bit you need from the SSH keys troubleshooting info is the suggestion to add your custom key to your SSH agent:

ssh-add may be enough
ssh-add /path/to/key may be required if it seems your agent didn’t pick up the custom key
ssh-add -l will show you the keys your agent knows of – your custom key must be listed
ssh-add -K or ssh-add -K /path/to/key is important to run if you are on a mac. Also, be sure your ssh agent isn’t “forgetting” your keys:
- more info
- and more info

Figuring out your SSH agent’s key selection is really worth it for your sanity as a dev (see also SSH keys docs). But if you don’t want to leave key selection up to the SSH agent, you can tell Trellis/Ansible which key to use:

ansible-playbook server.yml -e env=production --private-key=/path/to/private_key

Or, automate that by indicating the key in your ansible.cfg.

Alternatively, you could add an IdentityFile parameter to your specific host in your .ssh/config.

Yep, that works. You could just leave it at that and not try to troubleshoot connections via your custom azure key pair.