# Ansible upgrade breaks access to all other trellis deploys

**URL:** https://discourse.roots.io/t/ansible-upgrade-breaks-access-to-all-other-trellis-deploys/16733
**Category:** trellis
**Created:** 2019-10-02T15:21:25Z
**Posts:** 35

## Post 1 by @morcth — 2019-10-02T15:21:25Z

Hello,

I created a new trellis site the other day and saw that the requirements step had changed slightly and that when I went through it, it upgrade my version of ansible.

I just tried to deploy and older site and received: ‘PlaybookCLI’ object has no attribute ‘options’.

After searching around, I realized it was the ansible upgrade so I downgraded and all is well.

I will need to create more sites soon and am not sure what to do here.

I found this post about virtualenvs: [Updated to Ansible 2.4, deploys broken, now what?](https://discourse.roots.io/t/updated-to-ansible-2-4-deploys-broken-now-what/10588/6)

Is this the best way to deal with different sites that use different ansible versions or is there a quick way to upgrade all my older sites or what?

Any help is appreciated.

Thanks!  
Josh

---

## Post 2 by @swalkinshaw — 2019-10-02T15:26:03Z

It depends on a few things such as:

- how many projects you have
- how old they are
- how customized they are

Trellis did get support for Ansible 2.8 in [https://github.com/roots/trellis/pull/1103](https://github.com/roots/trellis/pull/1103) but you’d have to apply those changes to all your projects.

Using virtualenv for different versions is likely the best way to proceed.

Should also mention that [trellis-cli](https://github.com/roots/trellis-cli) just got a new feature to manage virtualenvs + dependencies automatically. I think the process would be:

1. manually add a `requirements.txt` to all projects that don’t have it (based on [https://github.com/roots/trellis/blob/master/requirements.txt](https://github.com/roots/trellis/blob/master/requirements.txt)) but set the version of Ansible you want.
2. run `trellis init` within that project dir and a virtualenv will be created + pip dependencies installed
3. use `trellis` cli for subsequent commands (or `trellis exec <any ansible command>`) to run it within the virtual env

---

## Post 3 by @morcth — 2019-10-02T15:59:21Z

Thank you Scott!

Trying this out now… do I run init in the trellis dir or in the root of the proj that contains site and trellis dir?

---

## Post 4 by @swalkinshaw — 2019-10-02T16:20:12Z

:thinking: either works if I remember correctly. trellis-cli detects project structures and handles things properly (at least it should!)

---

## Post 5 by @morcth — 2019-10-02T17:53:50Z

Hey Scott,

I initiated trellis to one of my projects and now am redoing the requirements step of my newer version and ran into this error which I did not experience the first time my Ansible was updated:

Found existing installation: ansible 2.7.5  
Uninstalling ansible-2.7.5:  
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: ‘/usr/local/lib/python2.7/site-packages/ansible-2.7.5-py2.7.egg-info/PKG-INFO’  
Consider using the `--user` option or check the permissions.

Any idea why this is happening now? My command was: pip install -r requirements.txt

Thank you!

---

## Post 6 by @morcth — 2019-10-02T18:02:33Z

I guess I can just add --user to the command but definitely didn’t have to a week ago when I ran the same pip install command in my trellis directory.

---

## Post 7 by @swalkinshaw — 2019-10-02T18:06:30Z

You just ran `pip install -r requirements.txt` on its own? If you could list the steps you took/commands you ran in order it might help figure out what happened.

---

## Post 8 by @morcth — 2019-10-02T19:01:59Z

Sure can:

```
**trellis3** ] $ pip install -r requirements.txt

Collecting ansible<2.9,>=2.7.12 (from -r requirements.txt (line 1))

Requirement already satisfied: passlib in /usr/local/lib/python2.7/site-packages (from -r requirements.txt (line 2)) (1.7.1)

Requirement already satisfied: cryptography in /usr/local/lib/python2.7/site-packages (from ansible<2.9,>=2.7.12->-r requirements.txt (line 1)) (2.7)

Requirement already satisfied: jinja2 in /usr/local/lib/python2.7/site-packages (from ansible<2.9,>=2.7.12->-r requirements.txt (line 1)) (2.10.1)

Requirement already satisfied: PyYAML in /usr/local/lib/python2.7/site-packages (from ansible<2.9,>=2.7.12->-r requirements.txt (line 1)) (5.1.2)

Requirement already satisfied: enum34; python_version < "3" in /usr/local/lib/python2.7/site-packages (from cryptography->ansible<2.9,>=2.7.12->-r requirements.txt (line 1)) (1.1.6)

Requirement already satisfied: cffi!=1.11.3,>=1.8 in /usr/local/lib/python2.7/site-packages (from cryptography->ansible<2.9,>=2.7.12->-r requirements.txt (line 1)) (1.12.3)

Requirement already satisfied: six>=1.4.1 in /usr/local/lib/python2.7/site-packages (from cryptography->ansible<2.9,>=2.7.12->-r requirements.txt (line 1)) (1.11.0)

Requirement already satisfied: asn1crypto>=0.21.0 in /usr/local/lib/python2.7/site-packages (from cryptography->ansible<2.9,>=2.7.12->-r requirements.txt (line 1)) (0.24.0)

Requirement already satisfied: ipaddress; python_version < "3" in /usr/local/lib/python2.7/site-packages (from cryptography->ansible<2.9,>=2.7.12->-r requirements.txt (line 1)) (1.0.22)

Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python2.7/site-packages (from jinja2->ansible<2.9,>=2.7.12->-r requirements.txt (line 1)) (1.1.1)

Requirement already satisfied: pycparser in /usr/local/lib/python2.7/site-packages (from cffi!=1.11.3,>=1.8->cryptography->ansible<2.9,>=2.7.12->-r requirements.txt (line 1)) (2.19)

Installing collected packages: ansible

Found existing installation: ansible 2.7.5

Uninstalling ansible-2.7.5:

Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/lib/python2.7/site-packages/ansible-2.7.5-py2.7.egg-info/PKG-INFO'

Consider using the `--user` option or check the permissions.
```

Ok, I just realized when I downgraded ansible with: sudo pip install ansible==2.7.5

This caused all my older deploys to work again but it might not have been the best way to downgrade?

So maybe I should uninstall with sudo and run a requirements.txt?

Thanks!

---

## Post 9 by @swalkinshaw — 2019-10-02T19:09:22Z

Oh yeah `sudo` can break things. Ideally you should never run `pip` with `sudo`. I’m not entirely sure how to get out of that broken permission state once you’re in it.

However, if you ran `trells init` then you can run `trellis exec pip install -r requirements.txt` instead to make sure you’re in the activated virtualenv

edit: `trellis init` runs pip install automatically too

---

## Post 10 by @morcth — 2019-10-02T19:46:05Z

Ok so I took a chance and just sudo pip uninstall and then ran pip requirements from the new version of trellis and installed the new version of ansible fine.

I then did a deploy how I did before for an older trellis site and received the error I received yesterday… so I was back to where I wanted to be.

I then ran trellis deploy and received this:  
Running command =\> ./bin/deploy.sh staging -----.com

```
PLAY [Ensure necessary variables are defined] **********************************

 **[WARNING]: Failure using method (v2_playbook_on_play_start) in callback plugin**

**(<ansible.plugins.callback./Users/----/Documents/projects/----/web**

**sites/trellis/lib/trellis/plugins/callback/vars.CallbackModule object at**

**0x7fc000267c50>): 'dict' object has no attribute 'iteritems'**

 **[WARNING]: You Ansible version is 2.7.13 but this version of Trellis has only**

**been tested for compatability with Ansible 2.4.0.0 -> 2.4.3.0. It is advisable**

**to check for Trellis updates or downgrade your Ansible version.**

TASK [Ensure environment is defined] *******************************************

 **[WARNING]: Failure using method (v2_runner_on_skipped) in callback plugin**

**(<ansible.plugins.callback.output.CallbackModule object at 0x7fc0086ebd68>):**

**name 'unicode' is not defined**

PLAY [Test Connection] *********************************************************

TASK [connection : Require manual definition of remote-user] *******************

TASK [connection : Specify preferred HostKeyAlgorithms for unknown hosts] ******

/bin/sh: grep: command not found

 **[WARNING]: Failure using method (v2_runner_on_failed) in callback plugin**

**(<ansible.plugins.callback.output.CallbackModule object at 0x7fc0086ebd68>):**

**name 'unicode' is not defined**

to retry, use: --limit @/Users/-----/Documents/projects/-----/websites/trellis/deploy.retry

PLAY RECAP *********************************************************************

------ : ok=0 changed=0 unreachable=0 failed=1

localhost : ok=0 changed=0 unreachable=0 failed=0

2019/10/02 15:30:15 exit status 2
```

Because it mentioned ansible 2.4.3.0, I downgraded in my projects requirements.txt and ran the trellis exec command you posted before. It downgraded fine.

When I ran deploy again, I received:

```
Running command => ./bin/deploy.sh staging ------.com

ERROR! Unexpected Exception, this is probably a bug: invalid syntax ( __init__.py, line 96)

to see the full traceback, use -vvv

2019/10/02 15:36:12 exit status 250
```

I feel like we are close… any idea why neither ansible version is working?

Thanks again!

---

## Post 11 by @morcth — 2019-10-02T19:59:43Z

Just wanted to summarize here for easier reading.

So If I do a full install of ansible 2.7.5, this trellis site deploys fine. If I upgrade, it stops working.

If I use trellis cli and requirements.txt with ansible==2.7.5 and passlib, I get further but also receive the errors above.

Maybe I need to put more in the requirements.txt? There is needed functionality missing in the virtualenv?

Thanks!

---

## Post 12 by @morcth — 2019-10-02T20:03:01Z

virtualenv python version is 3.7 and outside of it is 2.7.15.

That probably the prob?

Should I put python==2.7.15 in requirements.txt as well?

---

## Post 13 by @morcth — 2019-10-02T21:17:26Z

Ok, it looks like a virtualenv needs to be created with the wanted python version in the command. I guess I have python 3 installed on here but 2 is still the default version but trellis-cli ound 3 and used 3.

How can I make trellis-cli create a virtualenv with python 2.7 without deleting python 3?

Thanks again!

---

## Post 14 by @swalkinshaw — 2019-10-02T21:39:26Z

> [@morcth](#):
>
> How can I make trellis-cli create a virtualenv with python 2.7 without deleting python 3?

Unfortunately not yet. It defaults to python3 if installed. Although the Python version itself shouldn’t really matter :thinking:

> Ok, it looks like a virtualenv needs to be created with the wanted python version in the command.

I’m not sure what command you mean. Is it running deploy? If so, you should use trellis-cli’s `deploy` command. I wasn’t really clear on this, but if you’re trying to use it’s built-in venv feature, then _all_ commands should be run through it as well.

It has almost every common command built-in and for those that aren’t, you can run anything through `trellis exec <anything>`.

So in theory just using trellis-cli should make everything work without having to care about python version, ansible, etc.

---

## Post 15 by @morcth — 2019-10-02T21:46:00Z

Yep I am running trellis-cli’s deploy and getting the errors above.

I looked into ‘dict’ object has no attribute ‘iteritems’\*\* and _name ‘unicode’ is not defined_\* and it seems that these commands are in python 2 but not 3.

I checked python version on my mac with python --version and got 2.7.15 and when I use trellis exec python --version I get 3.7.0

I’m thinking that could be the issue?

Did the python version that trellis uses change in the last year to year and a half?

Thanks!

---

## Post 16 by @morcth — 2019-10-02T21:49:00Z

Also, from reading your latest reply, it is possible you only saw my last response… i had written a few as I found new things in my research… so you can see the error messages and extra info above when you have the time… and really thanks a lot for helping me get through it.

---

## Post 17 by @swalkinshaw — 2019-10-02T22:03:55Z

> Did the python version that trellis uses change in the last year to year and a half?

Okay yeah this makes sense, sorry. I think I assumed your projects couldn’t be that old :slightly_frowning_face: Definitely looks like they are old enough _not_ to have the Python 3 support we’ve added over time. For a while now Trellis supports both versions.

Sorry I’ve led you down a bit of a rabbit-hole then without asking some initial questions. Let me add an option to specify the python used for `init` quickly. I’ll push a new version tonight hopefully.

In the meantime you can always try to get `pip` working as usual without trellis-cli

---

## Post 18 by @morcth — 2019-10-02T22:21:28Z

; ) Yeah the one I was deploying to was probably created June-Aug 18. I never updated these projects because they always worked and didn’t want to break anything.

Thank you very much for being so responsive on this and please don’t go crazy trying to fix anything too fast. I can always upgrade and downgrade as needed if I have to work on any new projects or mess with virtualenv myself and that post I linked to in my initial post.

And also glad you pointed me to trellis-cli which is going to make my life so much easier for new projects and also to manage my older projects and virtual environments… freakin awesome that you created it!

Josh

---

## Post 19 by @swalkinshaw — 2019-10-03T14:10:09Z

I did take a look at this last night but of course it wasn’t as easy as I originally thought :pensive: I need to think about it a bit more.

Thanks for trying out trellis-cli!

---

## Post 20 by @morcth — 2019-10-03T15:38:06Z

Yeah, it’s never as quick/easy as we think sadly.

I’m watching the project now and let me know if I can help in any way.

Thanks!

---

## Post 21 by @swalkinshaw — 2019-10-04T00:09:12Z

I found a nice improvement to make which also kind of solves your use case: [https://github.com/roots/trellis-cli/pull/73](https://github.com/roots/trellis-cli/pull/73)

1. install virtualenv via `pip install virtualenv`
2. `virtualenv` command should work globally now
3. run `export VIRTUALENV_PYTHON=<path to python2 binary>`
4. use `trellis init`

I’ll release this in a new version soon.

---

## Post 22 by @morcth — 2019-10-04T19:33:55Z

Thanks Scott! Sadly, still having one error.

Ok so I installed virtualenv and found where my python path was… I believe this should work: /usr/local/Cellar/python@2/2.7.15\_1/bin

I ran export command and then trellis init and then trellis deploy and had the same issues. I then deleted the .trellis folder and ran trellis init again and trellis deploy… and same issues.

I then realized I still hadn’t upgraded trellis-cli yet! So I got v0.6.1 and removed the .trellis folder and ran trellis init and received an error:

```
[**----** @------MBP: **~/Documents/projects/-----/websites/------.com/trellis** **(staging)**] $ trellis init

Creating virtualenv in /Users/-----/Documents/projects/------/websites/-------.com/trellis/.trellis/virtualenv

Error creating virtualenv: exit status 3
```

an empty .trellis folder is created but nothing in it.

edit: the trellis init command works fine if I uninstall virtualenv… when I reinstall and run trellis init again… i get the error.

Any idea what could be going on?

---

## Post 23 by @swalkinshaw — 2019-10-04T23:25:32Z

Interesting, thanks for trying it out.

Does `virtualenv` work fine if you run it manually? Oh if you literally used the value `/usr/local/Cellar/python@2/2.7.15_1/bin` then that’s probably the problem. You also need the binary itself:

```
VIRTUALENV_PYTHON=/usr/local/Cellar/python@2/2.7.15_1/bin/python virtualenv .trellis/virtualenv
```

Assuming that works, `trellis init` should…

btw you can run `which python` or `which python3` to easily get the full path. You can even do ‘export VIRTUALENV\_PYTHON=`which python`’

---

## Post 24 by @morcth — 2019-10-05T20:53:34Z

Thanks Scott… that’s hopefully it. I’ll try that out on Monday.

Am I putting “export” before the command you wrote and virtualenv .trellis/virtualenv after the location in the same command?

Thanks!

---

## Post 25 by @swalkinshaw — 2019-10-06T16:46:40Z

It’s an either or:

```
export VIRTUALENV_PYTHON=/usr/local/Cellar/python@2/2.7.15_1/bin/python
```

This sets the env var for the duration of that shell/terminal session

```
VIRTUALENV_PYTHON=/usr/local/Cellar/python@2/2.7.15_1/bin/python virtualenv .trellis/virtualenv
```

This sets the env var for that command _only_

---

## Post 26 by @morcth — 2019-10-07T15:05:55Z

Ok… super close here I think.

I made sure I had the newest ansible on my system and ran a deploy on old site and it did not work.

I then ran ‘export VIRTUALENV\_PYTHON=/usr/local/bin/python’ and trellis init and no issues. I then ran ‘trellis exec ansible --version’ and got the older version of trellis I put in requirements.txt… AWESOME!

Then I ran ‘trellis deploy staging [site.com](http://site.com)’ and got the response and error:  
Running command =\> ./bin/deploy.sh staging [site.com](http://site.com)

```
PLAY [Ensure necessary variables are defined] **********************************
 [WARNING]: You Ansible version is 2.7.5 but this version of Trellis has only
been tested for compatability with Ansible 2.4.0.0 -> 2.4.3.0. It is advisable
to check for Trellis updates or downgrade your Ansible version.

TASK [Ensure environment is defined] *******************************************
skipping: [localhost]

PLAY [Test Connection] *********************************************************

TASK [connection : Require manual definition of remote-user] *******************
skipping: [xxx.xxx.124.81]

TASK [connection : Specify preferred HostKeyAlgorithms for unknown hosts] ******
/bin/sh: grep: command not found
System info:
  Ansible 2.7.5; Darwin
  Trellis at "Support git url format `ssh://user@host/path/to/repo`"
---------------------------------------------------
An unhandled exception occurred while templating '{{ openssh_6_5_plus |
ternary('ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-
ed25519,ssh-rsa', 'ssh-rsa-cert-v01@openssh.com,ssh-rsa') }}'. Error was a
<class 'ansible.errors.AnsibleError'>, original message: An unhandled
exception occurred while templating '{{ (lookup('pipe', 'ssh -V 2>&1')) |
regex_replace('(.*OpenSSH_([\d\.]*).*)', '\2') | version_compare('6.5', '>=')
}}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An
unhandled exception occurred while running the lookup plugin 'pipe'. Error
was a <class 'ansible.errors.AnsibleError'>, original message:
lookup_plugin.pipe(ssh -V 2>&1) returned 127
fatal: [xxx.xxx.124.81]: FAILED! => {}
	to retry, use: --limit @/Users/------/Documents/projects/-------/websites/trellis/deploy.retry

PLAY RECAP *********************************************************************
xxx.xxx.124.81 : ok=0 changed=0 unreachable=0 failed=1   
localhost : ok=0 changed=0 unreachable=0 failed=0   

2019/10/07 10:39:10 exit status 2
```

So I tried to activate the virtualenv myself by going into **/trellis/.trellis/virtualenv** and running ‘source bin/activate’ And it was working because it said (virtualenv) in front of my bash location line or whatever it’s called.

I ran ansible --version and got the correct one I needed and then cd …/…/ back into my trellis folder and ran: ‘ansible-playbook deploy.yml -e “[site=site.com](http://site=site.com) env=staging”’

This worked fine with no errors!

So it seems like trellis deploy is having some sort of ssh issue while ansible deploy after activated virtualenv is not having that issue.

Any ideas?

Thanks!

---

## Post 27 by @TangRufus — 2019-10-07T18:59:44Z

`/bin/sh: grep: command not found` is interesting.  
What are the outputs of `which grep` on your local machine and the remote server?

---

## Post 28 by @morcth — 2019-10-07T19:11:40Z

Locally in virtualenv and after I deactivated… /usr/bin/grep  
Remote… /bin/grep

Also… not sure if it matters but server is running Ubuntu 16.04.5

Thanks!

---

## Post 29 by @TangRufus — 2019-10-07T19:22:27Z

how about `ssh -V` on your local machine and the remote server?

---

## Post 30 by @morcth — 2019-10-07T20:11:43Z

local - ssh -V OpenSSH\_7.9p1, LibreSSL 2.7.3  
virtualenv - ssh -V OpenSSH\_7.9p1, LibreSSL 2.7.3  
remote - ssh -V OpenSSH\_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g 1 Mar 2016

Thanks,  
Josh

---

## Post 31 by @TangRufus — 2019-10-07T20:18:58Z

Maybe related:

With Trellis [b53f5e6](https://github.com/roots/trellis/commit/b53f5e6d212851fa531f6464b85cd984ab7b4dde) plus this change:

```
// vagrant.default.yml
- vagrant_box: 'bento/ubuntu-18.04'
+ vagrant_box: 'bento/ubuntu-16.04'
```

`vagrant up` stuck at

```
==> default: Bindfs seems to not be installed on the virtual machine, installing now
```

But it works fine if reverting to `bento/ubuntu-18.04`

ping @swalkinshaw

---

## Post 32 by @morcth — 2019-10-07T21:01:40Z

I’m not sure how to rule out the version of ubuntu right now… have to think of that for a minute.

Also, why would it work fine when manually activating virtualenv but not when using trellis-cli deploy?

---

## Post 33 by @swalkinshaw — 2019-10-07T21:53:43Z

Oh this might be due to [https://github.com/roots/trellis-cli/pull/74](https://github.com/roots/trellis-cli/pull/74)

Try upgrading to `v0.6.2` which I released

---

## Post 34 by @morcth — 2019-10-08T15:09:01Z

SUCCESS!!!

Upgrading to 0.6.2 worked. SO it was the PATH issue.

Thank you so much for working with me on this!

So using requirements.txt and trellis-cli seems like a great and easy option to use sites that use older versions of trellis and newer sites on the same system.

---

## Post 35 by @swalkinshaw — 2019-10-08T23:27:56Z

:tada: yay I’m glad you got it all working. Thanks for testing it out and you helped identify a few bugs!

---

## Post 36 by @system — 2019-11-13T15:21:29Z

This topic was automatically closed after 42 days. New replies are no longer allowed.