Can't deploy -

I’ve been trying to deploy with Trellis.
No errors when run:
ansible-playbook -i hosts/production server.yml

But when I try to deploy, I get this:

TASK [deploy : Clone project files] ********************************************
Failed to checkout master
fatal: [mysite.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
...ignoring

TASK [deploy : Failed connection to remote repo] *******************************
Git repo git@bitbucket.org:comp-dev-ar/trellis-test.git cannot be
accessed. Please verify the repository exists and you have SSH forwarding set
up correctly.
More info:
> https://roots.io/trellis/docs/deploys/#ssh-keys
> https://roots.io/trellis/docs/ssh-keys/#cloning-remote-repo-using-ssh-
agent-forwarding

fatal: [mysite.com]: FAILED! => {"changed": false, "failed": true}
	to retry, use: --limit @deploy.retry

PLAY RECAP *********************************************************************
mysite.com             : ok=7    changed=0    unreachable=0    failed=1   
localhost                  : ok=0    changed=0    unreachable=0    failed=0   

I’ve tried to use bitbucket and github… I get the same error.

Then…

`ssh -T git@github.com`
Hi user/test1! You've successfully authenticated, but GitHub does not provide shell access.

and…

ssh -T git@bitbucket.com
Warning: Permanently added 'bitbucket.com,104.192.143.7' (RSA) to the list of known hosts.
authenticated via a deploy key.
You can use git or hg to connect to Bitbucket. Shell access is disabled.
This deploy key has read access to the following repositories:
comp-dev-ar/trellis-test: id_rsa -- root@macuser.local

I don’t know what could be the problem…
Thanks.

Were you testing the commands like ssh -T git@bitbucket.com under the web user? That’s the user which deploys are run under.

You mean in the server?
I’ve tried and it fail… Do I need to add it manually on the server? I assume that trellis added that for me.
Thanks.

Deploys are run by the web user so you need to SSH in via that user to properly test the commands like ssh -T git@bitbucket.com. Testing them via the root user won’t actually tell you what’s going on during a deploy.

What failed? What exactly did you try?

2 Likes

I didn’t test in the server with web user… I’ve just do it:

Warning: Permanently added the RSA host key for IP address '104.192.143.8' to the list of known hosts.
Permission denied (publickey).

Do I have to add it on the provisioned server?

Follow our docs on SSH keys. You can add your keys as detailed then re-provision.

I’ve run:

ansible-playbook server.yml -e env=production -K

Some part of the provisioning:

TASK [users : Setup users] *****************************************************
changed: [mysite.com] => (item={u'keys': [u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+EsnZb//gctYBrYDS4yNMtAA1f+jriUnKj41Im5yg+m1/wk7kCRLnVU8nLFQ9cPhEyLXLc086jkfea62/Mi5hnpBymT1nqQasKNtY6ZhK/3AVqMNuF2Ff5pQ8CxkQNXvXSabYFt8O3qR9KJubIuzX75J+oyeXiUKl8TkNpvkvlImZIM8yVXyP66Dl3g7HeUVwAwVgAg2HqOgt/dYfKZEk1MTV1EO8FPsA0E9YxS2QN1CHW9qFfncyafePr6Sc+2iaCV6f8GQu2NVp1bE/e7OXqVdp/C1qQdtZUbTPh/adwpn1mKvuwgBDiArmvq9xzXsdbAf82SOO94yBc7z/31px macuser@osx.local', u'https://github.com/mygithubuser.keys'], u'name': u'web', u'groups': [u'www-data']})
changed: [mysite.com] => (item={u'keys': [u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+EsnZb//gctYBrYDS4yNMtAA1f+jriUnKj41Im5yg+m1/wk7kCRLnVU8nLFQ9cPhEyLXLc086jkfea62/Mi5hnpBymT1nqQasKNtY6ZhK/3AVqMNuF2Ff5pQ8CxkQNXvXSabYFt8O3qR9KJubIuzX75J+oyeXiUKl8TkNpvkvlImZIM8yVXyP66Dl3g7HeUVwAwVgAg2HqOgt/dYfKZEk1MTV1EO8FPsA0E9YxS2QN1CHW9qFfncyafePr6Sc+2iaCV6f8GQu2NVp1bE/e7OXqVdp/C1qQdtZUbTPh/adwpn1mKvuwgBDiArmvq9xzXsdbAf82SOO94yBc7z/31px macuser@osx.local', u'https://github.com/mygithubuser.keys'], u'name': u'admin', u'groups': [u'sudo']})

TASK [users : Add web user sudoers items for services] *************************
ok: [mysite.com]

TASK [users : Add SSH keys] ****************************************************
ok: [mysite.com] => (item=({u'name': u'web', u'groups': [u'www-data']}, u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+EsnZb//gctYBrYDS4yNMtAA1f+jriUnKj41Im5yg+m1/wk7kCRLnVU8nLFQ9cPhEyLXLc086jkfea62/Mi5hnpBymT1nqQasKNtY6ZhK/3AVqMNuF2Ff5pQ8CxkQNXvXSabYFt8O3qR9KJubIuzX75J+oyeXiUKl8TkNpvkvlImZIM8yVXyP66Dl3g7HeUVwAwVgAg2HqOgt/dYfKZEk1MTV1EO8FPsA0E9YxS2QN1CHW9qFfncyafePr6Sc+2iaCV6f8GQu2NVp1bE/e7OXqVdp/C1qQdtZUbTPh/adwpn1mKvuwgBDiArmvq9xzXsdbAf82SOO94yBc7z/31px macuser@osx.local'))
ok: [mysite.com] => (item=({u'name': u'web', u'groups': [u'www-data']}, u'https://github.com/mygithubuser.keys'))
ok: [mysite.com] => (item=({u'name': u'admin', u'groups': [u'sudo']}, u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+EsnZb//gctYBrYDS4yNMtAA1f+jriUnKj41Im5yg+m1/wk7kCRLnVU8nLFQ9cPhEyLXLc086jkfea62/Mi5hnpBymT1nqQasKNtY6ZhK/3AVqMNuF2Ff5pQ8CxkQNXvXSabYFt8O3qR9KJubIuzX75J+oyeXiUKl8TkNpvkvlImZIM8yVXyP66Dl3g7HeUVwAwVgAg2HqOgt/dYfKZEk1MTV1EO8FPsA0E9YxS2QN1CHW9qFfncyafePr6Sc+2iaCV6f8GQu2NVp1bE/e7OXqVdp/C1qQdtZUbTPh/adwpn1mKvuwgBDiArmvq9xzXsdbAf82SOO94yBc7z/31px macuser@osx.local'))
ok: [mysite.com] => (item=({u'name': u'admin', u'groups': [u'sudo']}, u'https://github.com/mygithubuser.keys'))



PLAY RECAP *********************************************************************
mysite.com                : ok=93   changed=5    unreachable=0    failed=0   
localhost                  : ok=0    changed=0    unreachable=0    failed=0   

Then I run…

./deploy.sh production mysite.com

and I get:

TASK [deploy : Clone project files] ********************************************
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

fatal: [mysite.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
...ignoring

TASK [deploy : Failed connection to remote repo] *******************************
Git repo git@github.com:mygithubuser/test1.git cannot be accessed. Please
verify the repository exists and you have SSH forwarding set up correctly.
More info:
> https://roots.io/trellis/docs/deploys/#ssh-keys
> https://roots.io/trellis/docs/ssh-keys/#cloning-remote-repo-using-ssh-
agent-forwarding

fatal: [mysite.com]: FAILED! => {"changed": false, "failed": true}
	to retry, use: --limit @deploy.retry

PLAY RECAP *********************************************************************
mysite.com                : ok=7    changed=0    unreachable=0    failed=1   
localhost                  : ok=0    changed=0    unreachable=0    failed=0   

In my local environment I can get a ssh connection to my github account, but I don’t understand where I need to create the keys for web user… Apparently this is made in users.yml but I’ve already re-provision again with this file:

# Documentation: https://roots.io/trellis/docs/ssh-keys/
admin_user: admin

# Also define sudoer_passwords in group_vars/<environment>/main.yml
users:
  - name: "{{ web_user }}"
    groups:
      - "{{ web_group }}"
    keys:
      - "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
      - https://github.com/mygithubaccount.keys
  - name: "{{ admin_user }}"
    groups:
      - sudo
    keys:
      - "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
      - https://github.com/mygithubaccount.keys

web_user: web
web_group: www-data
web_sudoers:
  - "/usr/sbin/service php7.0-fpm *"

Looks fine to me at a quick glance. I’d verify your ssh-agent is working properly. See here: https://help.github.com/articles/generating-a-new-ssh-key-and-adding-it-to-the-ssh-agent/#adding-your-ssh-key-to-the-ssh-agent

I thought it too…

admin@ps-ubuntu-06:~$ eval "$(ssh-agent -s)"
Agent pid 12468
admin@ps-ubuntu-06:~$ ls -al
total 44
drwxr-xr-x 5 admin sudo 4096 Apr 10 23:40 .
drwxr-xr-x 4 root       root 4096 Apr 10 15:18 ..
drwxr-xr-x 3 admin sudo 4096 Apr 10 15:15 .ansible
-rw------- 1 admin sudo 2315 Apr 11 11:37 .bash_history
-rw-r--r-- 1 admin sudo  220 Apr  7 18:20 .bash_logout
-rw-r--r-- 1 admin sudo 3637 Apr  7 18:20 .bashrc
drwx------ 2 admin sudo 4096 Apr  7 18:29 .cache
-rw-rw-r-- 1 admin sudo    0 Apr  7 18:29 .cloud-locale-test.skip
-rw------- 1 root       root   24 Apr  7 20:16 .nano_history
-rw-r--r-- 1 admin sudo  675 Apr  7 18:20 .profile
drwx------ 2 admin sudo 4096 Apr 10 23:40 .ssh
-rw-r--r-- 1 admin sudo  470 Apr 10 23:13 dead.letter
admin@ps-ubuntu-06:~$ ls -al .ssh/
total 16
drwx------ 2 admin sudo 4096 Apr 10 23:40 .
drwxr-xr-x 5 admin sudo 4096 Apr 10 23:40 ..
-rw------- 1 admin sudo  397 Apr  7 20:15 authorized_keys
-rw-r--r-- 1 admin sudo 2210 Apr 11 14:38 known_hosts
admin@ps-ubuntu-06:~$ 

There is no id_rsa in my provisioned server… Do I need to create it myself?

No, your public keys are added to authorized_keys. You could verify that they are indeed in there.

Did you run this locally? ssh-add ~/.ssh/id_rsa

3 Likes

Yes. Locally, doing:

ssh-add -L

I get:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+EsnZb//gctYBrYDS4yNMtAA1f +jriUnKj41Im5yg+m1/wk7kCRLnVU8nLFQ9cPhEyLXLc086jkfea62/Mi5hnpBymT1nqQasKNtY6ZhK/3AVqMNuF2Ff5pQ8CxkQNXvXSabYFt8O3qR9KJubIuzX75J+oyeXiUKl8TkNpvkvlImZIM8yVXyP66Dl3g7HeUVwAwVgAg2HqOgt/dYfKZEk1MTV1EO8FPsA0E9YxS2QN1CHW9qFfncyafePr6Sc+2iaCV6f8GQu2NVp1bE/e7OXqVdp/C1qQdtZUbTPh/adwpn1mKvuwgBDiArmvq9xzXsdbAf82SOO94yBc7z/31px /var/root/.ssh/id_rsa

On provisioned server, doing:

cat .ssh/authorized_keys

I get:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+EsnZb//gctYBrYDS4yNMtAA1f +jriUnKj41Im5yg+m1/wk7kCRLnVU8nLFQ9cPhEyLXLc086jkfea62/Mi5hnpBymT1nqQasKNtY6ZhK/3AVqMNuF2Ff5pQ8CxkQNXvXSabYFt8O3qR9KJubIuzX75J+oyeXiUKl8TkNpvkvlImZIM8yVXyP66Dl3g7HeUVwAwVgAg2HqOgt/dYfKZEk1MTV1EO8FPsA0E9YxS2QN1CHW9qFfncyafePr6Sc+2iaCV6f8GQu2NVp1bE/e7OXqVdp/C1qQdtZUbTPh/adwpn1mKvuwgBDiArmvq9xzXsdbAf82SOO94yBc7z/31px macuser@osx.local

Here’s the way I think about Trellis/Ansible’s process of cloning the repo on deploy.

The final section of the ssh keys docs mentions this:

All the SSH connections discussed above apply to Trellis connecting from your local machine to your server. It is a different type of connection, however, when Trellis clones a remote private repo during deployment. In this case, your remote server is allowed to forward your local machine’s SSH credentials to the remote repo to authorize the connection.

It could state more explicitly that this means that the list of users (and their keys) is actually irrelevant. Similarly, any keys present or absent on your remote are also irrelevant to your issue of failed connection to the repo.

You could think of it as an authentication of your local machine to your git repo. Your remote server just happens to be in the middle, forwarding the authentication credentials from your local machine to the git repo. (I get it though, that you could think of it as the remote connecting to the git host, but asking your local machine “can we just use your ssh key/credentials?”)

The required ssh credentials are managed by your local ssh-agent, so the whole thing is called ssh-agent forwarding. The process relies on the 1) correct keys being used (on local and git host) and 2) the ssh-agent forwarding being set up correctly.

Keys

Local machine has private key. Git repo has public key. Keys are really a pair.

Be sure your private key and public key are really a pair. Run ssh-keygen -y on local machine then enter in the absolute path to private key file. This prints the corresponding public key to the log. Ensure that the printed public key is among the keys loaded into the repo (e.g., this key displays at https://github.com/user_who_owns_target_repo.keys).

Key is being handled by ssh-agent.

Because you are a mac user, add your private key to your keychain by running

ssh-add -K ~/.ssh/id_rsa

Note the -K that is here, but was not in this command as it appeared earlier in the thread.

And yes, run ssh-add -L on local machine to verify that the public key that prints out is the one loaded on your git host (e.g., GitHub or Bitbucket).

SSH-agent forwarding

Local machine allows forwarding.

Make sure your version of Trellis has ansible.cfg (in Trellis project root) and that it includes ssh_args = -o ForwardAgent=yes. This instructs Ansible to allow your local ssh-agent credentials to be forwarded through the remote and on to the git host.

Note that if you test ssh-agent forwarding by running

ssh web@example.com 'ssh -T git@github.com'

this command won’t see the ForwardAgent setting in your ansible.cfg so it will fail unless you add this to your ~/.ssh/config:

# this host is your remote, not the host of your git repo
Host example.com
  ForwardAgent yes

Remote machine allows forwarding.

The remote must also be set up to allow ssh-agent forwarding. This is enabled by default on a bare Ubuntu install you would get from DigitalOcean. If your remote is not running Ubuntu, or is with a VPS provider that might adjust sshd default settings, let us know.

Try running this on your local machine to output the sshd configs:

ssh root@example.com 'less /etc/ssh/sshd_config'

Do you see a setting for AllowAgentForwarding? If not, the default is yes so you should be ok.

Get more debug info

If the problem/solution doesn’t surface as you work through the notes above, please share more verbose log output from the failed command. You could comment out no_log: true so no info is suppressed and rerun the deploy with -vvvv:

ansible-playbook deploy.yml -e "env=production site=example.com" -vvvv

You might also work through each step here (some steps you’ve covered already). One challenge is that Ansible introduces another variable to the equation.

5 Likes

I’ve read all… and made all you’ve told me… and re-provision.

In local I run:

ssh web@mysite.com 'ssh -T git@github.com'

I get:

Hi githubuser! You've successfully authenticated, but GitHub does not provide shell access.

But when I run the deployment:

deploy -e "env=production site=mysite.com" -vvvv

I get:

TASK [deploy : Clone project files] ********************************************
task path: /Library/WebServer/Document/mysite.com/trellis/roles/deploy/tasks/update.yml:24
<mysite.com> ESTABLISH SSH CONNECTION FOR USER: web
<mysite.com> SSH: EXEC ssh -C -vvv -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=web -o ConnectTimeout=10 -o ControlPath=/private/var/root/.ansible/cp/ansible-ssh-%h-%p-%r -tt mysite.com '/bin/sh -c '"'"'( umask 22 && mkdir -p "` echo $HOME/.ansible/tmp/ansible-tmp-1460502900.47-25772355765552 `" && echo "` echo $HOME/.ansible/tmp/ansible-tmp-1460502900.47-25772355765552 `" )'"'"''
<mysite.com> PUT /tmp/tmp01c1qj TO /home/web/.ansible/tmp/ansible-tmp-1460502900.47-25772355765552/git
<mysite.com> SSH: EXEC sftp -b - -C -vvv -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=web -o ConnectTimeout=10 -o ControlPath=/private/var/root/.ansible/cp/ansible-ssh-%h-%p-%r '[mysite.com]'
<mysite.com> ESTABLISH SSH CONNECTION FOR USER: web
<mysite.com> SSH: EXEC ssh -C -vvv -o ForwardAgent=yes -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=web -o ConnectTimeout=10 -o ControlPath=/private/var/root/.ansible/cp/ansible-ssh-%h-%p-%r -tt mysite.com '/bin/sh -c '"'"'LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 /usr/bin/python /home/web/.ansible/tmp/ansible-tmp-1460502900.47-25772355765552/git; rm -rf "/home/web/.ansible/tmp/ansible-tmp-1460502900.47-25772355765552/" > /dev/null 2>&1'"'"''
Failed to checkout master
fatal: [mysite.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}
...ignoring

TASK [deploy : Failed connection to remote repo] *******************************
task path: /Library/WebServer/Document/mysite.com/trellis/roles/deploy/tasks/update.yml:34
Git repo git@github.com:comp-dev-ar/trellis-test.git cannot be accessed. Please
verify the repository exists and you have SSH forwarding set up correctly.
More info:
> https://roots.io/trellis/docs/deploys/#ssh-keys
> https://roots.io/trellis/docs/ssh-keys/#cloning-remote-repo-using-ssh-
agent-forwarding

fatal: [mysite.com]: FAILED! => {"changed": false, "failed": true, "invocation": {"module_args": {"msg": "Git repo git@github.com:comp-dev-ar/trellis-test.git cannot be accessed. Please verify the repository exists and you have SSH forwarding set up correctly.\nMore info:\n> https://roots.io/trellis/docs/deploys/#ssh-keys\n> https://roots.io/trellis/docs/ssh-keys/#cloning-remote-repo-using-ssh-agent-forwarding\n"}, "module_name": "fail"}}
	to retry, use: --limit @deploy.retry

PLAY RECAP *********************************************************************
mysite.com                : ok=7    changed=0    unreachable=0    failed=1   
localhost                  : ok=0    changed=0    unreachable=0    failed=0   

On remote host , /etc/ssh/sshd_config doesn’t have the AllowAgentForwarding, so I assume that is on default value = yes… But then I added manually.

AllowAgentForwarding yes

Then try on provisioned server:

ssh -T git@github.com
Warning: Permanently added the RSA host key for IP address 'xxx.xxx.xxx.xxx' to the list of known hosts.
debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
debug1: channel 1: new [authentication agent connection]
debug1: confirm auth-agent@openssh.com
debug1: channel 1: FORCE input drain
Hi githubuser! You've successfully authenticated, but GitHub does not provide shell access.
debug1: channel 1: free: authentication agent connection, nchannels 2

So… Im lost…

Your output includes this:

Git repo git@github.com:comp-dev-ar/trellis-test.git cannot be accessed.

https://github.com/comp-dev-ar gives a 404, which suggests to me that comp-dev-ar does not have a GitHub account. Could you confirm that this is a real account and repo on the GitHub servers (not just on your local machine)? If it is not, note that you’ll need to create a real account and repo on GitHub servers and list it in wordpress_sites.

You might also compare whether these commands work from your local machine:

ssh web@example.com 'cd /tmp && git clone git@github.com:roots/bedrock.git'
ssh web@example.com 'cd /tmp && git clone git@github.com:comp-dev-ar/trellis-test.git'

That’s because Im replacing all the real information (IP, users, keys, etc). It’s created and working just fine…
It’s just empty:

running…
ssh web@mysite.com 'cd /tmp && git clone git@github.com:comp-dev-ar/trellis-test.git'

get…
Cloning into 'trellis-test'... warning: You appear to have cloned an empty repository.

Hmm… I’m guessing you created the comp-dev-ar/trellis-test repo/project using the GitHub GUI but haven’t pushed your actual repo from your local machine. Try this in your local machine Trellis directory:

# add github repo as a remote
git remote add github git@github.com:comp-dev-ar/trellis-test.git

# push the latest repo changes up to GitHub
git push github master

# test whether cloned repo is no longer empty
ssh web@mysite.com 'cd /tmp && git clone git@github.com:comp-dev-ar/trellis-test.git test2'

If that last command just responds Cloning into 'test2'... I think that means success and you can try your deploy again.

If that doesn’t resolve it, could you please do this:

Otherwise, I think the critical debug info is being censored, as per your error message:
FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}

4 Likes

Thanks. I will… may be was obvious that I need to initiate git… :frowning:
I have this:

mysite.com
site
trellis

Do I need to initiate git in mysite.com directory or in trellis directory?

Thanks.

In the project root folder so mysite.com. Run git init and then git push to push it to GitHub.

Yes, thank you for checking @pushstudio, and for clarifying @swalkinshaw, you’ll be pushing up your mysite.com repo/project. I was wrong when I suggested pushing from your Trellis directory. You’ll be deploying mysite.com so that is what you need to push to a git host.

Thanks to all… It was only that. I can’t believe I didn’t see it :frowning:
Thanks. It deploy. The database is empty, so I will read more about that. Thanks to all!
:clap:

2 Likes