Fatal un reachable when deploying to staging

I setup my droplet with SSH key for staging. I added all my code to repo but when I try to run:

ansible-playbook server.yml -e env=staging

I get

fatal: [45.55.167.58]: UNREACHABLE! => {"changed": false, "unreachable": true}

On another site that I am using the same process everything works fine. I can ssh into the new droplet with no problems. Any ideas what might be happening here?

I also pinged staging and get the correct ip back for my droplet. Could this be a DNS issue if I just added the A Record.

This happens on staging and production droplets for this server

I ended up solving this. I had to change the admin user to root and it worked.

I don’t think it’s a DNS issue, particularly because the error message fatal: [45.55.167.58] suggests that Trellis/Ansible is trying to connect to the IP directly (vs. trying to connect to a domain that DNS must associate with an IP).


The problem with admin_user: root is that if you ever change and apply sshd_permit_root_login: false, you will have no user who can ssh into your server.

Now that root appears to be able to connect for you, could you change back to admin_user: admin and try running server.yml and see if you don’t in fact see the message below, followed by a successful (“reachable”) connection and a successful run of server.yml?

TASK [remote-user : Announce which user was selected] **************************
Note: Ansible will attempt connections as user = root

The only way I was able to reproduce UNREACHABLE! was following these steps:

  • build a droplet
  • connect to droplet by ssh, whether via manual ssh or via server.yml (now I have a hostkey for server in my local known_hosts)
  • rebuild droplet (now my local hostkey is out-of-date)
  • run server.yml again without removing out-of-date hostkey
  • out-of-date hostkey causes test-connection for root to fail, so Trellis uses admin_user (i.e., admin) as fallback, but admin hasn’t been created yet because server.yml hasn’t run on the rebuilt droplet.

Any chance that may have been your scenario? If so, the solution would be to remove the hostkey from known_hosts after rebuilding the droplet (and before running server.yml): ssh-keygen -R hostname

If you find that the issue you face is not an out-of-date hostkey after rebuilding your droplet, I’d love if you could share a detailed list of steps to reproduce the problem, assuming you discover the problem is with Trellis. Thanks!

If you’re not yet running 2.0.2, upgrading may solve the problem.

After looking further, I suspect that Ansible’s change of handling its internal ansible_user variable may be the cause. I’ll soon post details over at Trellis 0.9.7 connecting as admin on initial server.yml. Fails - unreachable

I replicated the exact steps and still run into the issue.

1.) I removed all known hosts with ssh-keygen -R hostname
2.) I changed the user from admin to root
3.) Built the droplet
4.) Connected to it through SSH and it did not ask for a password
5.) Then went into /trellis and ran ansible-playbook server.yml -e env=staging

This still returns the connection issue.

SSH encountered an unknown error during the connection. We recommend you re-
run the command using -vvvv, which will enable SSH debugging output to help
diagnose the issue
fatal: [45.55.167.58]: UNREACHABLE! => {"changed": false, "unreachable": true}

Below I have attached all the files that I have changed and the output of ansible-playbook server.yml -e env=staging -vvvv

staging/vault.yml

# Documentation: https://roots.io/trellis/docs/vault/
vault_mysql_root_password: stagingpw

# Documentation: https://roots.io/trellis/docs/security/
vault_sudoer_passwords:
  admin: adminppassword
# Variables to accompany `group_vars/staging/wordpress_sites.yml`
# Note: the site name (`example.com`) must match up with the site name in the above file.
vault_wordpress_sites:
  mywebsite.com:
    env:
      db_password: dbpassword
      # Generate your keys here: https://roots.io/salts.html
      auth_key: "< generated salts >"
      secure_auth_key: "< generated salts >"
      logged_in_key: "< generated salts >"
      nonce_key: "< generated salts >"
      auth_salt: "< generated salts >"
      secure_auth_salt: "< generated salts >"
      logged_in_salt: "< generated salts >"
      nonce_salt: "< generated salts >"

staging/wordpress_sites.yml

# Documentation: https://roots.io/trellis/docs/remote-server-setup/
# `wordpress_sites` options: https://roots.io/trellis/docs/wordpress-sites
# Define accompanying passwords/secrets in group_vars/production/vault.yml

wordpress_sites:
  mywebsite.com:
    site_hosts:
      - staging.mywebsite.com
    local_path: ../site # path targeting local Bedrock site directory (relative to Ansible root)
    repo: mygitrepo.git # replace with your Git repo URL
    repo_subtree_path: site # relative path to your Bedrock/WP directory in your repo
    branch: master
    multisite:
      enabled: false
    ssl:
      enabled: false
      provider: letsencrypt
    cache:
      enabled: false

hosts/staging

I updated the ip address to match my droplet

Here is the output of ansible-playbook server.yml -e env=staging -vvvv

SH encountered an unknown error. The output was:
OpenSSH_6.9p1, LibreSSL 2.1.8
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 20: Applying options for *
debug1: /etc/ssh/ssh_config line 102: Applying options for *
debug1: auto-mux: Trying existing master
debug1: Control socket "/Users/<user>/.ansible/cp/ansible-
ssh-45.55.167.58-22-admin" does not exist
debug2: ssh_connect: needpriv 0
debug1: Connecting to 45.55.167.58 [45.55.167.58] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 9940 ms remain after connect
debug1: identity file /Users/<user>/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<user>/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<user>/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<user>/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<user>/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<user>/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<user>/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /Users/<user>/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1
Ubuntu-2ubuntu2.7
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.7 pat OpenSSH_6.6.1* compat
0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 45.55.167.58:22 as 'admin'
debug3: hostkeys_foreach: reading file "/Users/<user>/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file
/Users/<user>/.ssh/known_hosts:29
debug3: load_hostkeys: loaded 1 keys from 45.55.167.58
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-
sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com
,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-
sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256
,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256
,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-
hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-
sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com
,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-
ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-
cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-
cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ct
r,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour
128,aes128-cbc,3des-cbc,blowfish-
cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ct
r,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,arcfour256,arcfour
128,aes128-cbc,3des-cbc,blowfish-
cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com
,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256
,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-
ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-
md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com
,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: umac-64-etm@openssh.com,umac-128-etm@openssh.com
,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256
,hmac-sha2-512,hmac-sha1,hmac-md5-etm@openssh.com,hmac-
ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-
md5-96-etm@openssh.com,hmac-md5,hmac-ripemd160,hmac-ripemd160@openssh.com
,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit: zlib@openssh.com,zlib,none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256
,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256
,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-
hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfou
r128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.
com,aes128-cbc,3des-cbc,blowfish-
cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfou
r128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.
com,aes128-cbc,3des-cbc,blowfish-
cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-
sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-
md5-96-etm@openssh.com,hmac-md5,hmac-
sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512
,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-
sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-
sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-
ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-
md5-96-etm@openssh.com,hmac-md5,hmac-
sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512
,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug1: kex: server->client chacha20-poly1305@openssh.com <implicit>
zlib@openssh.com
debug1: kex: client->server chacha20-poly1305@openssh.com <implicit>
zlib@openssh.com
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256
SHA256:mjLBEhUfjiRzHY1uN1IZcX6zCzYtTBl0wI+z89raQjI
debug3: hostkeys_foreach: reading file "/Users/<user>/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file
/Users/<user>/.ssh/known_hosts:29
debug3: load_hostkeys: loaded 1 keys from 45.55.167.58
debug1: Host '45.55.167.58' is known and matches the ECDSA host key.
debug1: Found key in /Users/<user>/.ssh/known_hosts:29
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /Users/<user>/.ssh/id_rsa (0x7faed3e02d90),
debug2: key: /Users/<user>/.ssh/id_dsa (0x0),
debug2: key: /Users/<user>/.ssh/id_ecdsa (0x0),
debug2: key: /Users/<user>/.ssh/id_ed25519 (0x0),
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred: ,gssapi-keyex,hostbased,publickey
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /Users/<user>/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Trying private key: /Users/<user>/.ssh/id_dsa
debug3: no such identity: /Users/<user>/.ssh/id_dsa: No such file or
directory
debug1: Trying private key: /Users/<user>/.ssh/id_ecdsa
debug3: no such identity: /Users/<user>/.ssh/id_ecdsa: No such file or
directory
debug1: Trying private key: /Users/<user>/.ssh/id_ed25519
debug3: no such identity: /Users/<user>/.ssh/id_ed25519: No such file or
directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

fatal: [45.55.167.58]: UNREACHABLE! => {"changed": false, "unreachable": true}

I’m currently Ansible 2.0.1.0. I have not worked with Ansible up until recently. I have googled updating Ansible and a lot of the responses recommend to uninstall and reinstall. What would you recommend to make this process as smooth as possible?

Thank you for your help. Once I have this process down I will try to contribute more instead of asking the questions.

Thank you for so carefully sharing all that info. Sorry about all that work when the problem probably can just be solved by updating Ansible.

I am on a Mac. Initially I managed Ansible versions using homebrew but I switched to pip because 1) it seems to more quickly make available the latest Ansible releases and 2) it was easier for me to switch between versions. Brew can switch very easily between versions if they are already downloaded on your machine, but I found it takes a number of steps to obtain older Ansible versions if you don’t already have them in your Brew. I’m sure a Brew pro could demo how to do it very easily or has written a custom script, but I found pip to work great so I switched.

So, if you didn’t install Ansible originally via pip, then I think it would in fact be a good idea to uninstall Ansible using whatever means you used to install it. For example, if you installed via Brew, then uninstall Ansible completely via Brew.

Now your system is ready to let pip take over managing your Ansible versions. Run the two easy commands Ansible suggests to get pip and install Ansible. (I think I did some work to avoid using sudo with the second command, and possibly the first. Sorry I’m not a pip pro.)

Then, going forward you can…

  • Upgrade to the latest Ansible: pip install ansible --upgrade
  • Switch to a particular Ansible version (use real version #): pip install ansible==2.x.x

Of course, if you initially installed Ansible via pip, don’t uninstall; just use the upgrade command above.

Ok I will give this a try and post my results after. Thank you for the help

Trellis now requires Ansible >= 2.0.2.0: https://github.com/roots/trellis/pull/579

2 Likes

Took me a few days to get back to this project. Upgrading Ansible did the trick. Thanks @swalkinshaw & @fullyint. Now that I have the workflow down I can’t believe I didn’t make the switch earlier. Better late than never.

1 Like