Unable to provision on Amazon EC2 when provisioning via ansible-playbook server.yml command

I’m trying to connect to an Amazon AWS EC2 instance via ansible ansible-playbook server.yml -e env=staging with no luck.

In case it is needed, I am using Cygwin64 on Windows 10. I have the VM spun up (VirtualBox 5.1.14) and can access my local site via <site_name>.dev. (actual site name replaced with <site_name> for security)

To connect to the instance on Amazon I use ubuntu@ec2-34-250-211-85.eu-west-1.compute.amazonaws.com (via port 22). I use the Private Key C:\Users\<my_name>\.ssh\Lenovo-EC2-IE.ppk

Using Cygwin64 (always run as administrator) I can SSH without any difficulty via ssh ubuntu@ec2-34-250-211-85.eu-west-1.compute.amazonaws.com => Response: Welcome to Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-64-generic x86_64)

Running ssh-add -L from Cygwin64 show the following loaded keys:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC5I2AIXqC5FnVwoHVhW7wq1NY3g8/VQQCODsvE97J3+9GKeWwdNdBxQ44DpGenvqges9+m3plHT4/NNroDxjhCzhXzKf4eP2GATBpQQP2d7s20+XawrvVNNDIpHuRSskYjj0VoS3xX9sapxK1nqcW34Rg7pMKcHjU562D4AKryfCPGw+jpId7nWotU9W9FjvRQuC/tO0jtyc0GMWKuZd8oUCxQRsYBOemtWG8oeWT06qMalwI1SoCq6XgWLscQLoYeGmJm8GFYN6RXHQluUyXcI1wan2PlYEaWv03TolQjx2bBQQVkiaV9xqShZ4nX9/OqrVPjyciKTu8qGTSvQ7X3jJVRSC3TalX3epvvzPm6OlF6uPIdtu3RcSZLpxKXqPqmOxZlF2Z8Ql4psK+Tlbng4xq8tgjjbdJnSQTmeuVwd68sC0Card5ljvHHq3CqAtltr0UEV3oowAbiRDkYC0/aOgmTLfUe4k9QDq++cfg8kWo3vr6ZtNv0ztXrg/28jvJAKSA501RjcIBaBz+eFvGEDjR2hI8u1M4Yls5CIV889nklZCKczWmAUim6MQ6eSfo51wYC9PqzrF6KxFSN8A7fTtFIjWkTGAYZcZBMSAQ+QZiUZcu4eHFgD0Rfzp6+a2NZamGLN2psDfZnLsPUlkO+Sr6aWozBXU/tmas/0CECqQ== /cygdrive/c/Users/<my_name>/.ssh/id_rsa
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOOipKt5V5jK7nvCaCw1NSqnRiLHMKBzcXEx6W28krVbS/O2lz84IBGmvZ/GwtvmDd3qQYvZi6ltfJxoCzNq8unF6KRgYCnf6a1YO8Kikf7QodlSOtzlsp5mJwP9PR794FIo5gA2Ft7xCKyZ9wOoQAwgLxRM3UYul5i/pBVH3z9YDC7DA8/DBntd6g7SDdIpxwGglv2zfBVEBKV9mFoHugeQKPt3yXc+K8Pk0yVsvxzJViHy4BYk5Cv/W0txPyZzQvFFzAJ6g7YfFqimfZd4efMInkHx3mfd37e+MMXWCogU64jcYZiODWKZmmmNMtjAEA9klAytuYGqnsP2KkVuP .ssh/Lenovo-EC2-IE.pem

group_vars/all/users.yml has admin_user set as ubuntu

admin_user: ubuntu

Also define ‘vault_users’ (group_vars/staging/vault.yml, group_vars/production/vault.yml)

users:

  • name: “{{ web_user }}”
    groups:
    • “{{ web_group }}”
      keys:

    - “{{ lookup(‘file’, ‘~/.ssh/Lenovo-EC2-IE.pem’) }}”

  • name: “{{ admin_user }}”
    groups:
    • sudo
      keys:

    - “{{ lookup(‘file’, ‘~/.ssh/Lenovo-EC2-IE.pem’) }}”

web_user: web
web_group: www-data
web_sudoers:

  • “/usr/sbin/service php7.1-fpm *”

group_vars/staging/wordpress_sites.yml:

wordpress_sites:
ec2-34-250-211-85.eu-west-1.compute.amazonaws.com:
site_hosts:
- canonical: staging.<site_name>.com
# redirects:
# - otherdomain.com
local_path: …/site # path targeting local Bedrock site directory (relative to Ansible root)
repo: git@github.com:<site_name>/<site_name>.com.git
repo_subtree_path: site # relative path to your Bedrock/WP directory in your repo
branch: master
multisite:
enabled: false
ssl:
enabled: false
provider: letsencrypt
cache:
enabled: false

group_vars/staging.vault.yml:

vault_mysql_root_password: stagingpw

vault_users:

  • name: “{{ admin_user }}”
    password: example_password
    salt: “generateme”

vault_wordpress_sites:
ec2-34-250-211-85.eu-west-1.compute.amazonaws.com:
env:
db_password: example_dbpassword
# Generate your keys here: https://roots.io/salts.html
auth_key: “generateme”
secure_auth_key: “generateme”
logged_in_key: “generateme”
nonce_key: “generateme”
auth_salt: “generateme”
secure_auth_salt: “generateme”
logged_in_salt: “generateme”
nonce_salt: “generateme”

hosts/staging:

[staging]
ec2-34-250-211-85.eu-west-1.compute.amazonaws.com

[web]
ec2-34-250-211-85.eu-west-1.compute.amazonaws.com

Here is the debug info from Cygwin64 executing ansible-playbook server.yml -vvv -e env=staging

Using /cygdrive/c/Users/<my_name>/Web/example.com/trellis/ansible.cfg as config file
[DEPRECATION WARNING]: Included file ‘/cygdrive/c/Users/<my_name>/Web/example.com/trellis/disable_challenge_sites.yml’ not found, however since this
include is not explicitly marked as ‘static: yes’, we will try and include it dynamically later. In the future, this will be an error unless
‘static: no’ is used on the include task. If you do not want missing includes to be considered dynamic, use ‘static: yes’ on the include or set
the global ansible.cfg options to make all inclues static for tasks and/or handlers.
This feature will be removed in a future release.
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: Included file ‘/cygdrive/c/Users/<my_name>/Web/example.com/trellis/reload_nginx.yml’ not found, however since this include is
not explicitly marked as ‘static: yes’, we will try and include it dynamically later. In the future, this will be an error unless ‘static: no’
is used on the include task. If you do not want missing includes to be considered dynamic, use ‘static: yes’ on the include or set the global
ansible.cfg options to make all inclues static for tasks and/or handlers.
This feature will be removed in a future release. Deprecation warnings
can be disabled by setting deprecation_warnings=False in ansible.cfg.
statically included: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/users/tasks/connection-warnings.yml
statically included: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/vendor/roles/composer/tasks/global-require.yml
statically included: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/letsencrypt/tasks/setup.yml
statically included: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/letsencrypt/tasks/nginx.yml
statically included: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/letsencrypt/tasks/certificates.yml
statically included: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/wordpress-setup/tasks/database.yml
statically included: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/wordpress-setup/tasks/self-signed-certificate.yml
statically included: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/wordpress-setup/tasks/nginx-includes.yml
statically included: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/wordpress-setup/tasks/nginx.yml

PLAYBOOK: server.yml ****************************************************************************************************************************
4 plays in server.yml

PLAY [Ensure necessary variables are defined] ***************************************************************************************************
META: ran handlers

TASK [Ensure environment is defined] ************************************************************************************************************
task path: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/variable-check.yml:8
skipping: [localhost] => {
“changed”: false,
“skip_reason”: “Conditional result was False”,
“skipped”: true
}
META: ran handlers
META: ran handlers

PLAY [Determine Remote User] ********************************************************************************************************************
META: ran handlers

TASK [remote-user : Require manual definition of remote-user] ***********************************************************************************
task path: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/remote-user/tasks/main.yml:2
skipping: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com] => {
“changed”: false,
“skip_reason”: “Conditional result was False”,
“skipped”: true
}

TASK [remote-user : Check whether Ansible can connect as root] **********************************************************************************
task path: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/remote-user/tasks/main.yml:9
Using module file /opt/ansible/lib/ansible/modules/commands/command.py
ESTABLISH LOCAL CONNECTION FOR USER: <my_name>
EXEC /bin/sh -c ‘/usr/bin/python && sleep 0’
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com → localhost] => {
“changed”: false,
“cmd”: [
“ansible”,
ec2-34-250-211-85.eu-west-1.compute.amazonaws.com”,
“-m”,
“raw”,
“-a”,
“whoami”,
“-u”,
“root”,
“–connection=smart”,
“–timeout=10”,
“–inventory-file=hosts”
],
“delta”: “0:00:10.622844”,
“end”: “2017-03-04 20:54:04.001474”,
“failed”: false,
“failed_when_result”: false,
“invocation”: {
“module_args”: {
“_raw_params”: “ansible ec2-34-250-211-85.eu-west-1.compute.amazonaws.com -m raw -a whoami -u root --connection=‘smart’ --timeout=‘10’ --inventory-file=‘hosts’”,
“_uses_shell”: false,
“chdir”: null,
“creates”: null,
“executable”: null,
“removes”: null,
“warn”: true
}
},
“rc”: 4,
“start”: “2017-03-04 20:53:53.378630”,
“stderr”: “”,
“stderr_lines”: ,
“stdout”: “\u001b[1;31mec2-34-250-211-85.eu-west-1.compute.amazonaws.com | UNREACHABLE! => {\u001b[0m\n\u001b[1;31m "changed": false, \u001b[0m\n\u001b[1;31m "msg": "Failed to connect to the host via ssh: Permission denied (publickey).\r\n", \u001b[0m\n\u001b[1;31m "unreachable": true\u001b[0m\n\u001b[1;31m}\u001b[0m”,
“stdout_lines”: [
“\u001b[1;31mec2-34-250-211-85.eu-west-1.compute.amazonaws.com | UNREACHABLE! => {\u001b[0m”,
“\u001b[1;31m "changed": false, \u001b[0m”,
“\u001b[1;31m "msg": "Failed to connect to the host via ssh: Permission denied (publickey).\r\n", \u001b[0m”,
“\u001b[1;31m "unreachable": true\u001b[0m”,
“\u001b[1;31m}\u001b[0m”
]
}

TASK [remote-user : Set remote user for each host] **********************************************************************************************
task path: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/remote-user/tasks/main.yml:17
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com] => {
“ansible_facts”: {
“ansible_user”: “ubuntu”
},
“changed”: false
}

TASK [remote-user : Announce which user was selected] *******************************************************************************************
task path: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/remote-user/tasks/main.yml:22
Note: Ansible will attempt connections as user = ubuntu
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com] => {
“changed”: false
}

TASK [remote-user : Load become password] *******************************************************************************************************
task path: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/remote-user/tasks/main.yml:26
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com] => {“censored”: “the output has been hidden due to the fact that ‘no_log: true’ was specified for this result”}
META: ran handlers
META: ran handlers

PLAY [Install prerequisites] ********************************************************************************************************************
META: ran handlers

TASK [Install Python 2.x] ***********************************************************************************************************************
task path: /cygdrive/c/Users/<my_name>/Web/example.com/trellis/server.yml:17
<ec2-34-250-211-85.eu-west-1.compute.amazonaws.com> ESTABLISH SSH CONNECTION FOR USER: ubuntu
<ec2-34-250-211-85.eu-west-1.compute.amazonaws.com> SSH: EXEC ssh -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o HostKeyAlgorithms=ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 -o ControlPath=/cygdrive/c/Users/<my_name>/.ansible/cp/cab8d17f01 -tt ec2-34-250-211-85.eu-west-1.compute.amazonaws.com ‘sudo -H -S -p “[sudo via ansible, key=jdkjcgnfthcggwotfyfrbcjbenbpyzts] password: " -u root /bin/sh -c '”’“‘echo BECOME-SUCCESS-jdkjcgnfthcggwotfyfrbcjbenbpyzts; which python || sudo apt-get update && sudo apt-get install -qq -y python-simplejson’”‘"’’
<ec2-34-250-211-85.eu-west-1.compute.amazonaws.com> (255, ‘’, ‘Permission denied (publickey).\r\n’)
System info:
Ansible 2.3.0; CYGWIN_NT-10.0
Trellis at “Allow for per-project packagist.com authentication”

Failed to connect to the host via ssh: Permission denied (publickey).

fatal: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]: UNREACHABLE! => {
“changed”: false,
“unreachable”: true
}
to retry, use: --limit @/cygdrive/c/Users/<my_name>/Web/example.com/trellis/server.retry

PLAY RECAP **************************************************************************************************************************************
ec2-34-250-211-85.eu-west-1.compute.amazonaws.com : ok=4 changed=0 unreachable=1 failed=0
localhost : ok=0 changed=0 unreachable=0 failed=0

Running

ssh -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o HostKeyAlgorithms=ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 -o ControlPath=/cygdrive/c/Users/<my_name>/.ansible/cp/cab8d17f01 -tt ec2-34-250-211-85.eu-west-1.compute.amazonaws.com

twice in a row returns 2 different results, in order:

<my_name>@THINKPAD-UPDATE ~/Web/example.com/trellis
$ ssh -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o HostKeyAlgorithms=ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 -o ControlPath=/cygdrive/c/Users/<my_name>/.ansible/cp/cab8d17f01 -tt ec2-34-250-211-85.eu-west-1.compute.amazonaws.com
mux_client_request_session: read from master failed: Connection reset by peer
Failed to connect to new control master

<my_name>@THINKPAD-UPDATE ~/Web/example.com/trellis
$ ssh -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o HostKeyAlgorithms=ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,ssh-rsa -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 -o ControlPath=/cygdrive/c/Users/<my_name>/.ansible/cp/cab8d17f01 -tt ec2-34-250-211-85.eu-west-1.compute.amazonaws.com
mm_send_fd: sendmsg(2): Connection reset by peer
mux_client_request_session: send fds failed

Any help anyone could give would be greatly appreciated.

Thanks for the detailed thread :slight_smile:

I don’t use Windows, but this seems like a confusing between private and public keys?

EC2 requires a private key when connecting to it. It seems like everything that’s been on the Trellis/Ansible side is with the public key. Trellis itself is only concerned with public keys (like in group_vars/all/users.yml).

When running an Ansible command, you can specify the --private-key option (and a path to it). Or you can add it to your hosts file after the IP/domain like ansible_ssh_private_key_file=C:\Users\<my_name>\.ssh\Lenovo-EC2-IE.pem

Thanks for the response @swalkinshaw :smile:

After setting ansible_ssh_private_key_file in hosts/staging as suggested, I was finally able to connect to the server and begin provisioning. However I instantly ran into a subsequent issue which appears to be related to having **ControlMaster ** set to auto in ansible.cfg

ansible-playbook server.yml -e env=staging

PLAY [Ensure necessary variables are defined] ***************************************************************************************************

TASK [Ensure environment is defined] ************************************************************************************************************
skipping: [localhost]

PLAY [Determine Remote User] ********************************************************************************************************************

TASK [remote-user : Require manual definition of remote-user] ***********************************************************************************
skipping: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]

TASK [remote-user : Check whether Ansible can connect as root] **********************************************************************************
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com → localhost]

TASK [remote-user : Set remote user for each host] **********************************************************************************************
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]

TASK [remote-user : Announce which user was selected] *******************************************************************************************
Note: Ansible will attempt connections as user = ubuntu
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]

TASK [remote-user : Load become password] *******************************************************************************************************
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]

PLAY [Install prerequisites] ********************************************************************************************************************

TASK [Install Python 2.x] ***********************************************************************************************************************
System info:
Ansible 2.3.0; CYGWIN_NT-10.0
Trellis at “Allow for per-project packagist.com authentication”

Failed to connect to the host via ssh: mm_send_fd: sendmsg(2): Connection
reset by peer
mux_client_request_session: send fds failed

fatal: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]: UNREACHABLE! => {“changed”: false, “unreachable”: true}
to retry, use: --limit @/cygdrive/c/Users/<my_name>/Web/example.com/trellis/server.retry

PLAY RECAP **************************************************************************************************************************************
ec2-34-250-211-85.eu-west-1.compute.amazonaws.com : ok=4 changed=0 unreachable=1 failed=0
localhost : ok=0 changed=0 unreachable=0 failed=0

Changing the following setting from auto to no in ansible.cfg

[ssh_connection]
ssh_args = -o ControlMaster=no

and then running: ansible-playbook server.yml -e env=staging results in:

PLAY [Ensure necessary variables are defined] ***************************************************************************************************

TASK [Ensure environment is defined] ************************************************************************************************************
skipping: [localhost]

PLAY [Determine Remote User] ********************************************************************************************************************

TASK [remote-user : Require manual definition of remote-user] ***********************************************************************************
skipping: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]

TASK [remote-user : Check whether Ansible can connect as root] **********************************************************************************
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com → localhost]

TASK [remote-user : Set remote user for each host] **********************************************************************************************
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]

TASK [remote-user : Announce which user was selected] *******************************************************************************************
Note: Ansible will attempt connections as user = ubuntu
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]

TASK [remote-user : Load become password] *******************************************************************************************************
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]

PLAY [Install prerequisites] ********************************************************************************************************************

TASK [Install Python 2.x] ***********************************************************************************************************************
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]

PLAY [WordPress Server - Install LEMP Stack with PHP 7.1 and MariaDB MySQL] *********************************************************************

TASK [Gathering Facts] **************************************************************************************************************************
ok: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]

TASK [common : Validate Ansible version] ********************************************************************************************************
[WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: ansible_version is not defined or
false in [{% for item in ansible_requirements %}{{ ansible_version.full | version_compare(item.version, item.operator) }},{% endfor %}]

skipping: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]

TASK [common : Validate format of site_hosts] ***************************************************************************************************
System info:
Ansible 2.3.0; CYGWIN_NT-10.0
Trellis at “Allow for per-project packagist.com authentication”

The conditional check ‘item.value.site_hosts | rejectattr(‘canonical’,
‘defined’) | list | count’ failed. The error was: template error while
templating string: no filter named ‘rejectattr’. String: {% if
item.value.site_hosts | rejectattr(‘canonical’, ‘defined’) | list | count %}
True {% else %} False {% endif %}

The error appears to have been in
‘/cygdrive/c/Users/<my_name>/Web/example.com/trellis/roles/common/tasks/main.yml’:
line 13, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

  • name: Validate format of site_hosts
    ^ here

fatal: [ec2-34-250-211-85.eu-west-1.compute.amazonaws.com]: FAILED! => {“failed”: true}
to retry, use: --limit @/cygdrive/c/Users/<my_name>/Web/example.com/trellis/server.retry

PLAY RECAP **************************************************************************************************************************************
ec2-34-250-211-85.eu-west-1.compute.amazonaws.com : ok=6 changed=0 unreachable=0 failed=1
localhost : ok=0 changed=0 unreachable=0 failed=0

Anyone got any ideas?

@oranblackwell I see that you’re running Ansible 2.3.0, which hasn’t been released yet. I can reproduce some the unusual behavior above if I run Ansible 2.3.0 from source.

I’d recommend using pip to install the latest stable Ansible (2.2) instead (maybe this will help with pip on Cygwin).

If pip won’t work for you and you need to run from source, you might try this to be sure you are cloning the latest stable branch, instead of the devel branch:

git clone git://github.com/ansible/ansible.git --recursive -b stable-2.2

If you don’t want to re-clone, you could try these commands from your local Ansible directory:

git checkout v2.2.1.0-1
git submodule update --init --recursive
1 Like