Trellis SSH Permission Denied (AWS)

I’m very interested in using Trellis and gave it a go this morning. I got the development environment set up no problem and i’m following the example project. When I go to provision my AWS Ubuntu 14.04 server, however, I get SSH Error: Permission denied (publickey). The server is set up to use a pem key and I’m on a vpn, but I can ssh in on my own with no problem. I’ve even used a very basic Ansible setup that I built with no problem on a server just like this… Any suggestions on what to look for would be great! My staging/users.yml file is:

admin_user: ubuntu

users:
  - name: "{{ web_user }}"
    groups:
      - "{{ web_group }}"
    keys:
      - "{{ lookup('file', '~/.ssh/myprecious.pem') }}"
      # - https://github.com/username.keys
  - name: "{{ admin_user }}"
    groups:
      - sudo
    keys:
      - "{{ lookup('file', '~/.ssh/myprecious.pem') }}"
      # - https://github.com/username.keys

web_user: web
web_group: www-data
web_sudoers:
  - "/usr/sbin/service php5-fpm *"

My hosts/staging file is:

[web]
10.0.11.187

[staging:children]
web

The -vvvv edition of my command is:

PLAY [Determine Remote User] **************************************************

TASK: [remote-user | Determine whether to connect as root or admin_user] ******
<127.0.0.1> REMOTE_MODULE command ansible 10.0.11.187 -m ping -i hosts/staging -u root
<127.0.0.1> EXEC ['/bin/sh', '-c', 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1443459446.82-146080007544667 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1443459446.82-146080007544667 && echo $HOME/.ansible/tmp/ansible-tmp-1443459446.82-146080007544667']
<127.0.0.1> PUT /var/folders/tp/jk7zh0yx1jzg4_xpq17mnv140000gn/T/tmpm0icWi TO /Users/swain/.ansible/tmp/ansible-tmp-1443459446.82-146080007544667/command
<127.0.0.1> EXEC ['/bin/sh', '-c', u'LANG=en_US.UTF-8 LC_CTYPE=en_US.UTF-8 /usr/bin/python /Users/swain/.ansible/tmp/ansible-tmp-1443459446.82-146080007544667/command; rm -rf /Users/swain/.ansible/tmp/ansible-tmp-1443459446.82-146080007544667/ >/dev/null 2>&1']
ok: [10.0.11.187 -> 127.0.0.1] => {"changed": false, "cmd": ["ansible", "10.0.11.187", "-m", "ping", "-i", "hosts/staging", "-u", "root"], "delta": "0:00:00.398513", "end": "2015-09-28 12:57:27.279631", "failed": false, "failed_when_result": false, "rc": 3, "start": "2015-09-28 12:57:26.881118", "stderr": "10.0.11.187 | FAILED => SSH Error: Permission denied (publickey).\n    while connecting to 10.0.11.187:22\nIt is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.", "stdout": "", "stdout_lines": [], "warnings": []}

TASK: [remote-user | Set remote user for each host] ***************************
<10.0.11.187> ESTABLISH CONNECTION FOR USER: swain
ok: [10.0.11.187] => {"ansible_facts": {"ansible_ssh_user": "ubuntu"}}

TASK: [remote-user | Announce which user was selected] ************************
<10.0.11.187> ESTABLISH CONNECTION FOR USER: ubuntu
ok: [10.0.11.187] => {
    "msg": "Note: Ansible will attempt connections as user = ubuntu"
}

PLAY [WordPress Server - Install LEMP Stack with PHP 5.6 and MariaDB MySQL] ***

GATHERING FACTS ***************************************************************
<10.0.11.187> ESTABLISH CONNECTION FOR USER: ubuntu
<10.0.11.187> REMOTE_MODULE setup
<10.0.11.187> EXEC ssh -C -tt -v -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o ControlPath="/Users/swain/.ansible/cp/ansible-ssh-%h-%p-%r" -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=ubuntu -o ConnectTimeout=10 10.0.11.187 /bin/sh -c 'mkdir -p $HOME/.ansible/tmp/ansible-tmp-1443459447.32-135083032644451 && chmod a+rx $HOME/.ansible/tmp/ansible-tmp-1443459447.32-135083032644451 && echo $HOME/.ansible/tmp/ansible-tmp-1443459447.32-135083032644451'
fatal: [10.0.11.187] => SSH Error: Permission denied (publickey).
    while connecting to 10.0.11.187:22
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.

TASK: [common | Validate Ansible version] *************************************
FATAL: no hosts matched or all hosts have already failed -- aborting


PLAY RECAP ********************************************************************
           to retry, use: --limit @/Users/swain/server.retry

10.0.11.187                : ok=3    changed=0    unreachable=1    failed=0

Thanks!

I haven’t tried using a pem key yet with Trellis, but according to the Ansible docs, you do need to set --private-key if you want to use a pem key: http://docs.ansible.com/ansible/intro_getting_started.html#your-first-commands

I haven’t tried this though, so perhaps someone who’s used a pem key can chime in. I know others have successfully provisioned an AWS instance with Trellis.

Ugh, answer was to ssh-add -K my pem key

2 Likes

For anyone setting up on a AWS VPC using pem keys I had to do a couple of things:

  1. Add a hostname, or else SSMTP wouldn’t install
  2. Switch from pem key to rsa or else the User tasks would fail in the authorized_key module
3 Likes

@swaincreates this is exactly the problem I’m having as well. Glad to see you got it to work. I’m somewhat new to all this. Can you (or anyone) be more specific about how to make the changes you listed:

  1. Add a hostname, or else SSMTP wouldn’t install
  2. Switch from pem key to rsa or else the User tasks would fail in the authorized_key module

Thank you

So if haven’t set up an Ansible way to do this yet, but here are some solutions:

  1. Check your /etc/hostname file to find out your hostname (change it if you want). Then in /etc/hosts/ add the hostname to the line:
127.0.0.1 localhost [your hostname here]

Oh, and if you change the hostname you need to reboot.

  1. So if you add your rsa public key to the list of authorized keys for the root/ubuntu user it will use that
cat ~/.ssh/id_rsa.pub | ssh user@hostname 'cat >> .ssh/authorized_keys'

That command will copy your public key and add it to the authorized_kekys

2 Likes

Thank you @swaincreates

I’ll try this. I’ve been researching a good way to set up a proper deployment process for AWS, including VPC, Route53, security groups, IAM, and roles, etc. and came across this:

I’m currently exploring ways to combine Trellis with a similar approach. If you’re interested, we could collaborate on this.

Also, I noticed there is no inventory file included with Trellis. Does anyone (@kalenjohnson) know what the thinking is for not including it? I’m new to Ansible, so I am curious to understand different configuration decision.

Thanks!

You mean allowing a PEM key for AWS? Others have gotten it going, I haven’t personally tried. Trellis is mostly tested with Digital Ocean, although any stock Ubuntu 14.04 install with standard root SSH access should work.

Although AWS support specifically isn’t supported out of the box, I would assume there’s probably options on Ansible Galaxy, which should be pretty easy to install alongside Trellis.

Hi @kalenjohnson

actually regarding the inventory file, I was referring to running this command:

ansible all --list-hosts

which results in:

ERROR: Unable to find an inventory file, specify one with -i ?

Here is a link to the Ansible documentation for the Inventory file setup:

http://docs.ansible.com/ansible/intro_inventory.html#splitting-out-host-and-group-specific-data

It looks to me like in Trellis there are three separate host files (production, staging, development), no host_vars, and no inventory file. So, I want to make sure I understand the design. Are the three host files the inventory files (just called something else)?

The primary reason I’m asking is so that I understand the design well enough to modify it for a more comprehensive aws setup.

Thanks again!

hosts = inventory (Ansible uses both terms unfortunately).

With AWS you can either put the static IPs in your staging/production files or use the dynamic AWS inventory script which is in the Ansible docs.

Thank you! I’ll be giving that a try.

I need to do this from the aws instance or my dev vm or my local machine???

basically, this says “don’t use the pem file, use my personal pub file”. It does not solve the issue, it bypasses it by adding my personal pub key to the authorized keys. :slight_smile:

I’m looking for a clear way to make Trellis work on AWS

1 Like

Got it! So: it is sort of said in the documentation: found out a .pem file is not valid - you need a public key. You can generate one like this:

ssh-keygen -y -f private_key1.pem > public_key1.pub 

then use that .pub file to connect to your AWS instance via Ansible. Will update the doc.