SSH Forwarding Broken

trellis
1

I recently provisioned and deployed a site successfully. Everything was working great, until this week, I got rid of my old laptop and got a new one.
I never thought about my SSH keys at the time.
Since Trellis set my remote server up SSH password authentication and root login both disabled, I couldn’t SSH in.

So, here’s what I did:

  • I generated a new SSH key on my new laptop
  • Added it to my GitHub account
  • Deleted my old SSH key from my GitHub account, since I thought it was no longer needed. (I probably shouldn’t have done this?)
  • Opened a console for my droplet on the Digitalocean website and managed to enable root login to my server.
  • Copied my local key by using clip < ~/.ssh/id_rsa.pub
  • Manually SSH’d in to the server and manually added my key to both ~/web/.ssh/authorized_keys and ~/admin/.ssh/authorized_keys
  • Disabled root login again

After doing some reading, these steps were probably not needed. Could I have just reprovisioned with trellis to add my new key? But I still would have had to enable root login I think.

Anyway, it seemed to work, I could SSH from my new local machine to my server using my new key, and I can commit to Github from local with no issues.

However, yesterday I tried to deploy some changes using ./deploy.sh production mysite.com , and I get an error reading permission denied (publickey)

After doing some more reading, it seems like this is an issue with SSH forwarding.
So I followed all the steps recommended here: Redirecting...

  • My keys work locally
  • Local ssh-agent is running
  • SSH agent forwarding is allowed, I created and added the following to ~/.ssh/config (IS Trellis supposed to create this file for me?)

Host example.com
ForwardAgent yes

The only step that gave me trouble was making my key available to ssh-agent.
ssh-add id_rsa works, but when I try ssh-add id_rsa.pub it asks me for my ssh passphrase, even though I left that blank on setup.

Anyway, when I run ssh -T git@github.com I’m still getting permission denied (publickey)
Running echo "$SSH_AUTH_SOCK" on both local and remote gives different outputs, which means forwarding is still not working I guess.

So, I’m out of ideas basically. Is there a way to have Trellis set up SSH keys again for me? Can I just enable root login, delete my authorized_keys files, and re-provision my server?

2

When ssh forwarding stops working apparently for no reason, and you’re on a Mac, it is very often solved by this note from the docs:

OSX users. Remember to import your SSH key password into Keychain by running ssh-add -K.

I realize you’ve been attentive to your ssh-agent, but just double checking that you’ve used -K.

1 Like
3

Thanks,
I noticed that, but I’m running Linux locally actually (Ubuntu 16.04), so I assume that doesn’t apply?

4

Right, the -K for keychain wouldn’t apply given that you’re running Linux.

Yep, once able to connect (e.g., as root that you enabled), the Trellis server.yml playbook would have added the pub key to authorized_keys for admin and web.

I’m pretty sure you only need to add your private key to your ssh-agent.[quote=“greene48, post:1, topic:8184”]
I created and added the following to ~/.ssh/config (IS Trellis supposed to create this file for me?)
[/quote]

Trellis does not create or modify your local machine’s ~/.ssh/config but includes ForwardAgent=yes to enable SSH Forwarding for its connections (e.g., when you run a playbook). You would only need the forwarding enabled in your ~/.ssh/config for forwarding to function as part of your manual ssh commands, e.g., when you manually ssh to your remote and run ssh -T git@github.com.

Be sure the keys for web include your new key, and the keys for admin include your new key, then try to let Trellis do its own pass at setting up the users

ansible-playbook server.yml -e env=production --tags users

If the above connection succeeds, try the ./deploy.sh command again. Maybe that will have straightened things out for the web user (who makes the deploy connection).

If the above server.yml connection fails, see if this manual connection works:

ssh admin@example.com

If the manual connection ssh admin@example.com succeeds, try adding something like --private-key=/home/greene48/.ssh/id_rsa to the end of the ansible-playbook command to be sure it is using the right key. If that resolves the issue, you could always use the --private-key option, or specify private_key_file in your ansible.cfg.

However, if the manual ssh admin@example.com fails, try specifying the key file:

ssh -i /home/greene48/.ssh/id_rsa admin@example.com

If that still fails, it really comes down to a basic ssh issue rather than Trellis issue and you may have more luck on other forums. Perhaps you need to flush out the ssh-agent and reload keys, or just restart ssh-agent, etc.

You might also spin up a fresh test DO droplet to test whether all the connections work. If the connections don’t work, that suggests the problem is with your local ssh setup. If the connections do work, it suggests there is just a problem with your existing site’s server.

Someone may be able to help a little more if you post your verbose output:

ansible-playbook deploy.yml -e env=production -e site=mysite.com -vvvv
2 Likes
5

Wow, thanks for the detailed response.

Done

The server.yml playbook succeeded. I tried the ./deploy.sh command again and it still fails though.

This works.

I tried this, and still get the same permission denied error.

I’ll try spinning up a new DO droplet. I guess I’ll have to go through all my config files and point things to the new IP.

6

Ok, I’ve got it working. It turns out it was as simple as this:

ssh-add -k

Apparently for linux you use -k rather than -K, as per this serverfault answer: ubuntu - Why is ssh agent forwarding not working? - Server Fault

Funny that there is no mention of that on the github troubleshooting guide.
Anyway, obviously not a Trellis issue, sorry for wasting your time, but thanks a lot for the help anyway.

2 Likes