Server.yml and SSH error

trellis
1

Hi guys,

I’m a bit stuck, and things have been piling on. I’m having trouble identifying the issue.

Firstly: deploying the site works.

When running ansible-playbook server.yml -e env=production , I receive the following message:

fatal: [37.139.24.247]: FAILED! => {"changed": false, "failed": true, "module_stderr": "", "module_stdout": "sudo: a password is required\r\n", "msg": "MODULE FAILURE", "parsed": false}

When running ansible-playbook server.yml -e env=production --ask-become-pass, I receive the following message:

fatal: [37.139.24.247]: FAILED! => {"failed": true, "msg": "Incorrect sudo password"}

I have used the security docs and generated a password (vault_sudoer_passwords).

I’m using a DO droplet, and was in the proces of disabling Root login but could not get it to work. The current status of the root_login is:

sshd_permit_root_login: true
sshd_password_authentication: false

Using ssh root@IP gives me: Permission denied (publickey).

In group_vars/all/users.yml:

admin_user: admin

Using ssh admin@IP gives me: Permission denied (publickey).

Any ideas what the issue might be ? I’m thinking something with the SSHD settings.

2

Perhaps you have another user that can connect, but if not, you may have lost SSH access to the server.

If the server/site can stand the downtime, and you have copies of files and data, you could destroy/rebuild the server (with whatever setting you want for sshd_permit_root_login).

If the server must stay up, you can restore root access on Digital Ocean.

In the unlikely event that you have staging and production on the same server (not recommended) and have edited the admin sudoer password for staging or production, note that it’s the same admin user on a single machine, so admin should have the same sudoer password in your Trellis configs for both environments.

3

Thanks for you reply, i have tried to restore the root access. Unfortunately i ran into a “port 22: Connection refused” error.

I decided to rebuild the server. I can SSH into the server again, which is great.

Unfortunately, server.yml can’t run yet.

Is there anything I need to restore after the rebuild? I’m getting the following error message:

fatal: [37.139.24.247]: UNREACHABLE! => {"changed": false, "msg": "SSH encountered an unknown error. The output was:\nOpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011\ndebug1: Reading configuration data /Users/Jan/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh_config\r\ndebug1: /etc/ssh_config line 20: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/Users/Jan/.ansible/cp/ansible-ssh-37.139.24.247-22-admin\" does not exist\r\ndebug2: ssh_connect: needpriv 0\r\ndebug1: Connecting to 37.139.24.247 [37.139.24.247] port 22.\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug1: fd 3 clearing O_NONBLOCK\r\ndebug1: Connection established.\r\ndebug3: timeout: 9990 ms remain after connect\r\ndebug3: Incorrect RSA1 identifier\r\ndebug3: Could not load \"/Users/Jan/.ssh/id_rsa\" as a RSA1 public key\r\ndebug1: identity file /Users/Jan/.ssh/id_rsa type 1\r\ndebug1: identity file /Users/Jan/.ssh/id_rsa-cert type -1\r\ndebug1: identity file /Users/Jan/.ssh/id_dsa type -1\r\ndebug1: identity file /Users/Jan/.ssh/id_dsa-cert type -1\r\ndebug1: Enabling compatibility mode for protocol 2.0\r\ndebug1: Local version string SSH-2.0-OpenSSH_6.2\r\ndebug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6\r\ndebug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6 pat OpenSSH*\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug3: load_hostkeys: loading entries for host \"37.139.24.247\" from file \"/Users/Jan/.ssh/known_hosts\"\r\ndebug3: load_hostkeys: found key type RSA in file /Users/Jan/.ssh/known_hosts:5\r\ndebug3: load_hostkeys: loaded 1 keys\r\ndebug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa\r\ndebug1: SSH2_MSG_KEXINIT sent\r\ndebug1: SSH2_MSG_KEXINIT received\r\ndebug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-dss\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se\r\ndebug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: zlib@openssh.com,zlib,none\r\ndebug2: kex_parse_kexinit: zlib@openssh.com,zlib,none\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1\r\ndebug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se\r\ndebug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se\r\ndebug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96\r\ndebug2: kex_parse_kexinit: none,zlib@openssh.com\r\ndebug2: kex_parse_kexinit: none,zlib@openssh.com\r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: \r\ndebug2: kex_parse_kexinit: first_kex_follows 0 \r\ndebug2: kex_parse_kexinit: reserved 0 \r\ndebug2: mac_setup: found hmac-md5-etm@openssh.com\r\ndebug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com zlib@openssh.com\r\ndebug2: mac_setup: found hmac-md5-etm@openssh.com\r\ndebug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com zlib@openssh.com\r\ndebug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent\r\ndebug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP\r\ndebug2: dh_gen_key: priv key bits set: 127/256\r\ndebug2: bits set: 521/1024\r\ndebug1: SSH2_MSG_KEX_DH_GEX_INIT sent\r\ndebug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY\r\ndebug1: Server host key: RSA 3f:48:8a:34:1f:98:f0:93:12:ec:71:35:64:79:36:c7\r\ndebug3: load_hostkeys: loading entries for host \"37.139.24.247\" from file \"/Users/Jan/.ssh/known_hosts\"\r\ndebug3: load_hostkeys: found key type RSA in file /Users/Jan/.ssh/known_hosts:5\r\ndebug3: load_hostkeys: loaded 1 keys\r\ndebug1: Host '37.139.24.247' is known and matches the RSA host key.\r\ndebug1: Found key in /Users/Jan/.ssh/known_hosts:5\r\ndebug2: bits set: 547/1024\r\ndebug1: ssh_rsa_verify: signature correct\r\ndebug2: kex_derive_keys\r\ndebug2: set_newkeys: mode 1\r\ndebug1: SSH2_MSG_NEWKEYS sent\r\ndebug1: expecting SSH2_MSG_NEWKEYS\r\ndebug2: set_newkeys: mode 0\r\ndebug1: SSH2_MSG_NEWKEYS received\r\ndebug1: Roaming not allowed by server\r\ndebug1: SSH2_MSG_SERVICE_REQUEST sent\r\ndebug2: service_accept: ssh-userauth\r\ndebug1: SSH2_MSG_SERVICE_ACCEPT received\r\ndebug2: key: /Users/Jan/.ssh/id_rsa (0x7f9662403f90),\r\ndebug2: key: /Users/Jan/.ssh/github_rsa (0x7f9662406e90),\r\ndebug2: key: /Users/Jan/.ssh/id_rsa (0x7f9662400190),\r\ndebug2: key: /Users/Jan/.ssh/id_dsa (0x0),\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug3: start over, passed a different list publickey,password\r\ndebug3: preferred gssapi-with-mic,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_lookup publickey\r\ndebug3: remaining preferred: ,gssapi-keyex,hostbased,publickey\r\ndebug3: authmethod_is_enabled publickey\r\ndebug1: Next authentication method: publickey\r\ndebug1: Offering RSA public key: /Users/Jan/.ssh/id_rsa\r\ndebug3: send_pubkey_test\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug1: Offering RSA public key: /Users/Jan/.ssh/github_rsa\r\ndebug3: send_pubkey_test\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug1: Offering RSA public key: /Users/Jan/.ssh/id_rsa\r\ndebug3: send_pubkey_test\r\ndebug2: we sent a publickey packet, wait for reply\r\ndebug1: Authentications that can continue: publickey,password\r\ndebug1: Trying private key: /Users/Jan/.ssh/id_dsa\r\ndebug3: no such identity: /Users/Jan/.ssh/id_dsa: No such file or directory\r\ndebug2: we did not send a packet, disable method\r\ndebug1: No more authentication methods to try.\r\nPermission denied (publickey,password).\r\n", "unreachable": true}

4

I can’t think of anything you’d need to change after the rebuild, but here are some responses to your output.

I see debug1: Connection established and debug3: load_hostkeys: found key type RSA in file /Users/Jan/.ssh/known_hosts:5 which suggests that your machine is connecting to the server just fine, but the SSH authentication then fails, of course. So, it’s not a known_hosts issue or some unreachable problem. It seems to be an authentication problem.

I see multiple instances of…

debug1: Offering RSA public key: /Users/Jan/.ssh/keyname:
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply

This seems to indicate that your ssh keys are being found and offered but rejected. It makes me suspect that the droplet does not have a public ssh key (in root user’s authorized_keys, for example) that corresponds to one of the private keys on your local machine. This hypothesis may also be supported by your report of Connection refused even after setting PermitRootLogin True.

You may want to rebuild the droplet and be sure you load up or associate your public SSH key with the new droplet.

Then be sure your ssh-agent is running on your local machine so it can/will offer your keys during the ssh connection:

ssh-agent bash

and your key is loaded into the agent (again on local machine):

ssh-add ~/.ssh/private_key_name

and if you’re on a mac add the key password to Keychain:

ssh-add -K

Then try running server.yml. If it fails again, could you paste the output of a verbose SSH attempt as root (assuming it fails)?

ssh -v root@37.139.24.247
1 Like
5

Thanks again!

Apparently the SSH authentication indeed failed. Adding the the private keys again fixed the issue.
I have been able to add the SSH keys, and run server.yml for this droplet. Great!

I’m hoping to find the cause, because connecting to another (different IP) Droplet gives me again: Permission denied (publickey).

This droplet also has PermitRootLogin yes , and I am able to login as root on the droplet’s Console. Is there any way to use ssh-copy-id while not being able to connect?

What i’ve tried to do is copy the public key from .ssh/id_rsa.pub to the servers .ssh/authorized_keys, without luck. I can’t copy the id_rsa into the browser window.

6

That’s frustrating. I don’t have any knowledge or experience regarding this next challenge. It doesn’t seem to be a Trellis issue, per se, so you might find the best forum response over at DigitalOcean Questions, stackoverflow, serverfault etc.

However, I succumbed to temptation and searched “copy paste into digital ocean console.” I clicked only one result and didn’t review it closely, but it might help: “How to Copy and Paste into the Digital Ocean VNC Console.”

7

I have managed to fix the issue, had to set PasswordAuthentication to yes inside sshd_config. Thanks a million for helping me get to the solution.

I’m unsure how or when this has changed. Perhaps when I wanted to lock down root access.

For future reference:
The error is Permission denied (publickey)when connecting via SSH, also unable to run Server.yml

How to fix:

  1. DigitalOcean web GUI > Access > Console Access
  2. Login with root
  3. vim /etc/ssh/sshd_config
  4. PermitRootLogin yes
  5. PasswordAuthentication yes
  6. save and exit vim editor (:w (save) , :q (quit))
  7. service ssh restart
2 Likes