AWS SSH Host Unreachable

I’m having a bit of trouble connecting to an AWS EC2 instance that I have set up for a staging environment. I always get a failed to connect to host via SSH when running ansible-playbook server.yml -e env=staging. I’m hoping that someon here much smarter than me can figure this out. I have set up my user inside of all/users.yml and have verified my SSH keys are available using ssh-add -l. If I manually ssh into the host by using ssh -t ubuntu@ec2-11-11-111-11@us-west-2.compute.amazonaws.com it works fine. I tried googling and looking around in this discourse for answers but none of the threads I found seemed to help me at all.

Thanks for your time. Also the host IP has of course been changed.

Using /vagrant/ansible.cfg as config file
Loaded callback output of type stdout, v2.0

PLAYBOOK: server.yml ***********************************************************
4 plays in server.yml

PLAY [Ensure necessary variables are defined] **********************************

TASK [Ensure environment is defined] *******************************************
task path: /vagrant/variable-check.yml:8
skipping: [localhost] => {"changed": false, "skip_reason": "Conditional check failed", "skipped": true}

PLAY [Determine Remote User] ***************************************************

TASK [remote-user : Require manual definition of remote-user] ******************
task path: /vagrant/roles/remote-user/tasks/main.yml:2
skipping: [ec2-11-11-111-11.us-west-2.compute.amazonaws.com] => {"changed": false, "skip_reason": "Conditional check failed", "skipped": true}

TASK [remote-user : Check whether Ansible can connect as root] *****************
task path: /vagrant/roles/remote-user/tasks/main.yml:9
ESTABLISH LOCAL CONNECTION FOR USER: vagrant
localhost EXEC /bin/sh -c '( umask 22 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1475433623.38-178984940592333 `" && echo "` echo $HOME/.ansible/tmp/ansible-tmp-1475433623.38-178984940592333 `" )'
localhost PUT /tmp/tmpS7h5ey TO /home/vagrant/.ansible/tmp/ansible-tmp-1475433623.38-178984940592333/command
localhost EXEC /bin/sh -c 'LANG=en_US LC_ALL=en_US LC_MESSAGES=en_US /usr/bin/python /home/vagrant/.ansible/tmp/ansible-tmp-1475433623.38-178984940592333/command; rm -rf "/home/vagrant/.ansible/tmp/ansible-tmp-1475433623.38-178984940592333/" > /dev/null 2>&1'
ok: [ec2-11-11-111-11.us-west-2.compute.amazonaws.com -> localhost] => {"changed": false, "cmd": ["ansible", "ec2-11-11-111-11.us-west-2.compute.amazonaws.com", "-m", "raw", "-a", "whoami", "-u", "root", "--connection=smart", "--timeout=10", "--inventory-file=hosts"], "delta": "0:00:00.729792", "end": "2016-10-02 18:40:24.151463", "failed": false, "failed_when_result": false, "invocation": {"module_args": {"_raw_params": "ansible ec2-11-11-111-11.us-west-2.compute.amazonaws.com -m raw -a whoami -u root --connection='smart' --timeout='10' --inventory-file='hosts'", "_uses_shell": false, "chdir": null, "creates": null, "executable": null, "removes": null, "warn": true}, "module_name": "command"}, "rc": 3, "start": "2016-10-02 18:40:23.421671", "stderr": "", "stdout": "\u001b[1;31mec2-11-11-111-11.us-west-2.compute.amazonaws.com | UNREACHABLE! => {\n    \"changed\": false, \n    \"msg\": \"Failed to connect to the host via ssh.\", \n    \"unreachable\": true\n}\u001b[0m", "stdout_lines": ["\u001b[1;31mec2-11-11-111-11.us-west-2.compute.amazonaws.com | UNREACHABLE! => {", "    \"changed\": false, ", "    \"msg\": \"Failed to connect to the host via ssh.\", ", "    \"unreachable\": true", "}\u001b[0m"], "warnings": []}

TASK [remote-user : Set remote user for each host] *****************************
task path: /vagrant/roles/remote-user/tasks/main.yml:16
ok: [ec2-11-11-111-11.us-west-2.compute.amazonaws.com] => {"ansible_facts": {"ansible_user": "ubuntu"}, "changed": false, "invocation": {"module_args": {"ansible_user": "ubuntu"}, "module_name": "set_fact"}}

TASK [remote-user : Announce which user was selected] **************************
task path: /vagrant/roles/remote-user/tasks/main.yml:21
Note: Ansible will attempt connections as user = ubuntu
ok: [ec2-11-11-111-11.us-west-2.compute.amazonaws.com] => {}

TASK [remote-user : Load become password] **************************************
task path: /vagrant/roles/remote-user/tasks/main.yml:25
ok: [ec2-11-11-111-11.us-west-2.compute.amazonaws.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}

PLAY [Install prerequisites] ***************************************************

TASK [Install Python 2.x] ******************************************************
task path: /vagrant/server.yml:17
<ec2-11-11-111-11.us-west-2.compute.amazonaws.com> ESTABLISH SSH CONNECTION FOR USER: ubuntu
<ec2-11-11-111-11.us-west-2.compute.amazonaws.com> SSH: EXEC ssh -C -vvv -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r -tt ec2-11-11-111-11.us-west-2.compute.amazonaws.com 'sudo -H -S -n -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-aexzcqscexezclrvxdnhovxsltsvzhbg; sudo apt-get install -qq -y python-simplejson'"'"''
System info:
  Ansible 2.0.2.0; Linux
  Trellis at "Enable per-site setup for permalink structure"
---------------------------------------------------
Failed to connect to the host via ssh.
fatal: [ec2-11-11-111-11.us-west-2.compute.amazonaws.com]: UNREACHABLE! => {"changed": false, "unreachable": true}

PLAY [WordPress Server - Install LEMP Stack with PHP 7.0 and MariaDB MySQL] ****
        to retry, use: --limit @server.retry

PLAY RECAP *********************************************************************
ec2-11-11-111-11.us-west-2.compute.amazonaws.com : ok=4    changed=0    unreachable=1    failed=0
localhost                  : ok=0    changed=0    unreachable=0    failed=0
1 Like

I’m looking into the exact same issue. For me this seems to be brought on by the change of IP on my deployment machine. I.e. if I provision the server from one public IP and deploy also, I cannot deploy/provision from a separate IP. However I don’t think this is fail2ban related as I can also SSH into my staging machine, this is strictly an Ansible issue by the looks of things

Some avenues I’m exploring are any kind of caches / fact gathering issues and also updating the whitelist for fail2ban and it’s configuration

I’ll add more information as soon as I know more

So my particular issue was actually a ssh config issue. In order to debug it I broke away from the ansible script and ran the last command which ansible failed on. Then the OS gave me a little more feedback on to where the problem stemmed from.

Although not a solution perhaps running the last ansible command you received may help. For example in your case:

ssh -C -vvv -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 -o ControlPath=/home/vagrant/.ansible/cp/ansible-ssh-%h-%p-%r -tt ec2-11-11-111-11.us-west-2.compute.amazonaws.com 'sudo -H -S -n -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-aexzcqscexezclrvxdnhovxsltsvzhbg; sudo apt-get install -qq -y python-simplejson'"'"''

@devotoare

2 Likes