Ansible_local version mismatch

[conductdesign]

Hi everyone,
I’ve recently had version mismatch issues with the ansible_local provisioner vagrant box. Same as reported here. This is the error I get when running trellis up with a fresh install via trellis new test4.com :

...
==> default: Running provisioner: ansible_local...
    default: Installing Ansible...
    default: Installing pip... (for Ansible installation)
The requested Ansible version (2.10.7) was not found on the guest.
Please check the Ansible installation on your Vagrant guest system (currently: 2.10.17),
or adapt the provisioner `version` option in your Vagrantfile.
See https://docs.vagrantup.com/v2/provisioning/ansible_common.html#version
for more information.
exit status 1

I’ve sshed into the box to see what’s been installed:

vagrant@test4:~$ ansible --version
ansible 2.10.17
  config file = None
  configured module search path = ['/home/vagrant/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0]
vagrant@test4:~$ pip list
Package                Version
---------------------- ---------
ansible                2.10.7
ansible-base           2.10.17
...

Ansible versioning appears to be a mess now and the Vagrant’s version check seems to be getting it wrong. This issue has been documented on the Vagrant github page.
It looks like the fix will be added in Vagrant 2.3.5…?

Here is a workaround that skips the version check altogether.

My setup:
MBP M1 2020
macOS Ventura (13.3.1)
python 3.9.6 (from command line tools, I believe)
pip 21.2.4 (also from command line tools)
Parallels Desktop 18.2.0
Vagrant 2.3.4
trellis-cli 1.10.1

I had success after switching the ansible version to latest in vagrant.default.yml (maybe overriding it in vagrant.local.yml is better practice:

vagrant_ansible_version: 'latest'

and starting over with a fresh box.

Hopefully this info will spare someone else the time it took to test and read up.

(Does anyone know if it’s possible to get a pre-release of Vagrant 2.3.5 via homebrew? Looks unlikely, but thought I’d ask here anyway.)

[swalkinshaw]

Did you see my reply in the other thread?

Ansible version error on vagrant up

Any issue with Vagrant and the ansible version is likely because you don’t have Ansible installed on your host machine. When Trellis’ Vagrantfile does not find an Ansible installation, it uses ansible_local (confusingly named) mode which installs and runs Ansible on the Vagrant VM and that’s what is broken due to that Vagrant bug. The recommend way to install Ansible and any other necessary Trellis dependencies is with trellis-cli by running the trellis init command in your project.

[conductdesign]

Hi @swalkinshaw and @ben ,
Thanks for your quick replies. I have been using trellis-cli for a while now and this issue just came out of nowhere (though I want to somehow blame my upgrade to Ventura). That helps clear things up a bit, but why can’t my Vagrant file find the ansible in the virtualenv?
Everything looks good on a brand new site until it tries to find the local/host ansible.

Does anything below strike you as suspicious?
(Hope I haven’t cut away anything relevant)

MBP-2020:Sites conduct$ ansible --version
-bash: ansible: command not found
MBP-2020:Sites conduct$ trellis new example.com
Creating new Trellis project in /Users/conduct/Sites/example.com

Site domain [example.com]: 

✔ example.com
Initializing project...

[✓] Created virtualenv (/Users/conduct/Sites/example.com/trellis/.trellis/virtualenv)
[✓] Ensure pip is up to date
[✓] Dependencies installed
Starting galaxy role install process
- downloading role 'composer', owned by geerlingguy
[... other roles here. No problems ...]
- mailpit (v1.0.0) was installed successfully

example.com project created with versions:
  Trellis v1.21.0
  Bedrock v1.22.0

MBP-2020:Sites conduct$ cd example.com/
[trellis] activated env
MBP-2020:example.com conduct$ ansible --version
ansible [core 2.14.4]
  config file = None
  configured module search path = ['/Users/conduct/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/lib/python3.9/site-packages/ansible
  ansible collection location = /Users/conduct/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/bin/ansible
  python version = 3.9.6 (default, Mar 10 2023, 20:16:38) [Clang 14.0.3 (clang-1403.0.22.14.1)] (/Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/bin/python3)
  jinja version = 3.1.2
  libyaml = True
MBP-2020:example.com conduct$ trellis up             
Starting galaxy role install process
- composer (1.9.0) is already installed, skipping.
- ntp (2.3.1) is already installed, skipping.
- logrotate (v0.0.5) is already installed, skipping.
- swapfile (v2.0.36) is already installed, skipping.
- mailpit (v1.0.0) is already installed, skipping.

Running command => vagrant up
Bringing machine 'default' up with 'parallels' provider...
==> default: Registering VM image from the base box 'bento/ubuntu-22.04-arm64'...
==> default: Creating new virtual machine as a linked clone of the box image...

[ boots without issue until... ]

==> default: Running provisioner: ansible_local...
    default: Installing Ansible...
    default: Installing pip... (for Ansible installation)
The requested Ansible version (2.10.7) was not found on the guest.
Please check the Ansible installation on your Vagrant guest system (currently: 2.10.17),
or adapt the provisioner `version` option in your Vagrantfile.
See https://docs.vagrantup.com/v2/provisioning/ansible_common.html#version
for more information.
exit status 1

Are there any other tests/checks I could run?

Thanks again!

@ben, I’ve tested Lima and had success after sorting out some networking issues. I’ll try again with trellis-cli 1.11 and hope to switch over before I have to renew my Parallels subscription!

[swalkinshaw]

Thanks for those details. Though something is definitely off

Based on your ansible --version output, it’s definitely being run from the virtualenv as it should be.

executable location = /Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/bin/ansible

And yet it’s using the ansible_local provisioning mode. Here’s the relevant part of the Vagrantfile:

roots/trellis/blob/02cfc360911c545611d624c01ce4d97f197c6d00/Vagrantfile#L121-L130

Which runs this method:

https://github.com/roots/trellis/blob/02cfc360911c545611d624c01ce4d97f197c6d00/lib/trellis/vagrant.rb#L53-L55

So there’s 3 possibilities:

• you’re on Windows (we can rule that out)
• which('ansible-playbook') is not returning true
• FORCE_ANSIBLE_LOCAL env var is set

I assume FORCE_ANSIBLE_LOCAL isn’t set, but can you double check by running echo $FORCE_ANSIBLE_LOCAL? (it should display nothing)

The next thing to check is that which method. The easiest way is to make this one addition:

diff --git lib/trellis/vagrant.rb lib/trellis/vagrant.rb
index ef3c6bd13a..392962e95e 100644
--- lib/trellis/vagrant.rb
+++ lib/trellis/vagrant.rb
@@ -88,6 +88,8 @@ def remote_site_path(site_name, site)
 end

 def which(cmd)
+  puts "which #{cmd}"
+
   exts = ENV['PATHEXT'] ? ENV['PATHEXT'].split(';') : ['']

   paths = ENV['PATH'].split(File::PATH_SEPARATOR).flat_map do |path|
@@ -96,6 +98,7 @@ def which(cmd)

   paths.any? do |path|
     next unless File.executable?(path) && !File.directory?(path)
+    puts path
     system("#{path} --help", %i(out err) => File::NULL)
   end
 end

And then run trellis up. You should see something like this at the start:

$ trellis up
Starting galaxy role install process
- composer (1.9.0) is already installed, skipping.
- ntp (2.3.1) is already installed, skipping.
- logrotate (v0.0.5) is already installed, skipping.
- swapfile (v2.0.36) is already installed, skipping.
- mailpit (v1.0.0) is already installed, skipping.

Running command => vagrant up
which ansible-playbook
/Users/scottwalkinshaw/dev/trellis/.trellis/virtualenv/bin/ansible-playbook
which ansible-playbook
/Users/scottwalkinshaw/dev/trellis/.trellis/virtualenv/bin/ansible-playbook
which ansible-playbook
/Users/scottwalkinshaw/dev/trellis/.trellis/virtualenv/bin/ansible-playbook

And finally, for a temporary workaround, you can just force the normal provisioning mode:

diff --git Vagrantfile Vagrantfile
index ea5178a36a..52c30a1dda 100644
--- Vagrantfile
+++ Vagrantfile
@@ -118,7 +118,7 @@ Vagrant.configure('2') do |config|
     end
   end

-  provisioner = local_provisioning? ? :ansible_local : :ansible
+  provisioner = :ansible
   provisioning_path = local_provisioning? ? ANSIBLE_PATH_ON_VM : ANSIBLE_PATH

   config.vm.provision provisioner do |ansible|

[conductdesign]

Thanks @swalkinshaw.
The additions to the which method returned (as expected/hoped)

MBP-2020:example.com conduct$ trellis up
Starting galaxy role install process
- ...

Running command => vagrant up
which ansible-playbook
/Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/bin/ansible-playbook
...

But it still used ansible_local.
Indeed, echo $FORCE_ANSIBLE_LOCAL returns nothing, so that’s good.

I won’t pretend to be a ruby guy, but I thought I’d try this in the local_provisioning method:

def local_provisioning?
  puts "test local_provisioning"
  @local_provisioning = which('ansible-playbook')
  puts @local_provisioning
end

which spits out this:

...
Running command => vagrant up
test local_provisioning
which ansible-playbook
/Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/bin/ansible-playbook
false
...

Looks like the path is being output fine from within the which method, but not returned? Again, I don’t know ruby, so maybe my test is meaningless.

Speaking of ruby. Not sure if it’s relevant or not but…

MBP-2020:example.com conduct$ ruby -v
ruby 3.0.0p0 (2020-12-25 revision 95aff21468) [arm64-darwin22]

I’ll use your workaround in the meantime. Thanks!

[swalkinshaw]

Even stranger…

@conductdesign

Looks like the path is being output fine from within the which method, but not returned? Again, I don’t know ruby, so maybe my test is meaningless.

Yeah that change won’t works since Ruby implicitly uses the last value as the return value, and by having a puts statement last, you’ve made it return nil (which is a “falsey” value in Ruby). So funny enough, you made local_provisioning? return false which is basically what you want in a roundabout way

Did that actually fix it? It should have output:

==> default: Running provisioner: ansible...

Instead of ansible_local

My workaround wasn’t 100% correct in hindsight. It’s better to just hardcode that method to false to be certain:

def local_provisioning?
  return false # short-circuit as a workaround
  @local_provisioning ||= Vagrant::Util::Platform.windows? || !which('ansible-playbook') || ENV['FORCE_ANSIBLE_LOCAL']
end

To be 100% sure, here’s a better debugging change you can try:

diff --git lib/trellis/vagrant.rb lib/trellis/vagrant.rb
index ef3c6bd13a..392962e95e 100644
--- lib/trellis/vagrant.rb
+++ lib/trellis/vagrant.rb
def local_provisioning?
+ puts "Windows? => #{Vagrant::Util::Platform.windows?}"
+ puts "ansible? => #{which('ansible-playbook')}"
+ puts "override? => #{ENV['FORCE_ANSIBLE_LOCAL']}"
  @local_provisioning ||= Vagrant::Util::Platform.windows? || !which('ansible-playbook') || ENV['FORCE_ANSIBLE_LOCAL']
end

I’d also run this with vagrant up --debug for more verbose output.

Thanks for trying all this.

[conductdesign]

Thanks for the suggestions. I added the debug puts as suggested and got this:

(had to strip out a bunch of the uninteresting bits. Scroll to bottom for the output from local_provisioning)

Running command => vagrant up --debug
 INFO global: Vagrant version: 2.3.4
 INFO global: Ruby version: 2.7.6
 INFO global: RubyGems version: 3.1.6
 INFO global: VAGRANT_INSTALLER_ENV="1"
 INFO global: VAGRANT_EXECUTABLE="/opt/vagrant/embedded/gems/2.3.4/gems/vagrant-2.3.4/bin/vagrant"
 INFO global: VAGRANT_INSTALLER_VERSION="2"
 INFO global: VAGRANT_INSTALLER_EMBEDDED_DIR="/opt/vagrant/embedded"
 INFO global: VAGRANT_LOG="debug"
 WARN global: resolv replacement has not been enabled!
 INFO manager: Registered plugin: NFS synced folders
 INFO manager: Registered plugin: RSync synced folders
 INFO manager: Registered plugin: SMB synced folders
 INFO manager: Registered plugin: kernel
 INFO manager: Registered plugin: ftp
 INFO manager: Registered plugin: noop
 INFO manager: Registered plugin: heroku
 INFO manager: Registered plugin: local-exec
 INFO manager: Registered plugin: atlas
 INFO manager: Registered plugin: docker-provider
 INFO manager: Registered plugin: Hyper-V provider
 INFO manager: Registered plugin: VirtualBox provider
 INFO manager: Registered plugin: kernel
 INFO manager: Registered plugin: podman
 INFO manager: Registered plugin: docker
 INFO manager: Registered plugin: file
 INFO manager: Registered plugin: shell
 INFO manager: Registered plugin: chef
 INFO manager: Registered plugin: ansible
 INFO manager: Registered plugin: container
 INFO manager: Registered plugin: salt
 INFO manager: Registered plugin: CFEngine Provisioner
 INFO manager: Registered plugin: puppet
 INFO manager: Registered plugin: provision command
 INFO manager: Registered plugin: destroy command
 INFO manager: Registered plugin: init command
 INFO manager: Registered plugin: snapshot command
 INFO manager: Registered plugin: validate command
 INFO manager: Registered plugin: box command
 INFO manager: Registered plugin: autocomplete command
 INFO manager: Registered plugin: winrm command
 INFO manager: Registered plugin: resume command
 INFO manager: Registered plugin: suspend command
 INFO manager: Registered plugin: plugin command
 INFO manager: Registered plugin: halt command
 INFO manager: Registered plugin: provider command
 INFO manager: Registered plugin: rdp command
 INFO manager: Registered plugin: status command
 INFO manager: Registered plugin: ssh-config command
 INFO manager: Registered plugin: winrm-config command
 INFO manager: Registered plugin: reload command
 INFO manager: Registered plugin: powershell command
 INFO manager: Registered plugin: cap command
 INFO manager: Registered plugin: push command
 INFO manager: Registered plugin: list-commands command
 INFO manager: Registered plugin: ssh command
 INFO manager: Registered plugin: version command
 INFO manager: Registered plugin: up command
 INFO manager: Registered plugin: vagrant-cloud
 INFO manager: Registered plugin: global-status command
 INFO manager: Registered plugin: start Vagrant server
 INFO manager: Registered plugin: port command
 INFO manager: Registered plugin: vagrant-login
 INFO manager: Registered plugin: package command
 INFO manager: Registered plugin: help command
 INFO manager: Registered plugin: upload command
 INFO manager: Registered plugin: Slackware host
 INFO manager: Registered plugin: BSD host
 INFO manager: Registered plugin: FreeBSD host
 INFO manager: Registered plugin: null host
 INFO manager: Registered plugin: Void host
 INFO manager: Registered plugin: Linux host
 INFO manager: Registered plugin: Arch host
 INFO manager: Registered plugin: Mac OS X host
 INFO manager: Registered plugin: Windows host
 INFO manager: Registered plugin: Red Hat Enterprise Linux host
 INFO manager: Registered plugin: Gentoo host
 INFO manager: Registered plugin: SUSE host
 INFO manager: Registered plugin: ALT Platform host
 INFO manager: Registered plugin: winrm communicator
 INFO manager: Registered plugin: windows ssh communicator
 INFO manager: Registered plugin: ssh communicator
 INFO manager: Registered plugin: Alpine guest
 INFO manager: Registered plugin: Solaris 11 guest.
 INFO manager: Registered plugin: DragonFly BSD guest
 INFO manager: Registered plugin: PLD Linux guest
 INFO manager: Registered plugin: Amazon Linux guest
 INFO manager: Registered plugin: Mint guest
 INFO manager: Registered plugin: Debian guest
 INFO manager: Registered plugin: Rocky guest
 INFO manager: Registered plugin: Slackware guest
 INFO manager: Registered plugin: CentOS guest
 INFO manager: Registered plugin: ESXi guest.
 INFO manager: Registered plugin: NetBSD guest
 INFO manager: Registered plugin: OmniOS guest.
 INFO manager: Registered plugin: Alma guest
 INFO manager: Registered plugin: BSD-based guest
 INFO manager: Registered plugin: Haiku guest
 INFO manager: Registered plugin: Fedora guest
 INFO manager: Registered plugin: TinyCore Linux guest.
 INFO manager: Registered plugin: Atomic Host guest
 INFO manager: Registered plugin: FreeBSD guest
 INFO manager: Registered plugin: Trisquel guest
 INFO manager: Registered plugin: SmartOS guest.
 INFO manager: Registered plugin: Kali guest
 INFO manager: Registered plugin: Linux guest.
 INFO manager: Registered plugin: Arch guest
 INFO manager: Registered plugin: OpenBSD guest
 INFO manager: Registered plugin: OpenWrt guest
 INFO manager: Registered plugin: NixOS guest
 INFO manager: Registered plugin: VMware Photon guest
 INFO manager: Registered plugin: Darwin guest
 INFO manager: Registered plugin: Funtoo guest
 INFO manager: Registered plugin: Windows guest.
 INFO manager: Registered plugin: Red Hat Enterprise Linux guest
 INFO manager: Registered plugin: Solaris guest.
 INFO manager: Registered plugin: Elementary guest
 INFO manager: Registered plugin: Gentoo guest
 INFO manager: Registered plugin: SUSE guest
 INFO manager: Registered plugin: CoreOS guest
 INFO manager: Registered plugin: Ubuntu guest
 INFO manager: Registered plugin: ALT Platform guest
 INFO vagrant: `vagrant` invoked: ["up", "--debug"]
DEBUG vagrant: Creating Vagrant environment
 INFO environment: Environment initialized (#<Vagrant::Environment:0x00007f7ac1da64c8>)
 INFO environment:   - cwd: /Users/conduct/Sites/example.com/trellis
 INFO environment: Home path: /Users/conduct/.vagrant.d
DEBUG environment: Effective local data path: /Users/conduct/Sites/example.com/trellis/.vagrant
 INFO environment: Local data path: /Users/conduct/Sites/example.com/trellis/.vagrant
DEBUG environment: Creating: /Users/conduct/Sites/example.com/trellis/.vagrant
DEBUG manager: Enabling localized plugins
 INFO manager: Plugins:
DEBUG solution_file: new solution file instance plugin_file=/Users/conduct/Sites/example.com/trellis/.vagrant/plugins.json solution_file=/Users/conduct/Sites/example.com/trellis/.vagrant/bundler/local.sol
DEBUG solution_file: missing file so skipping loading
DEBUG bundler: solution file in use for init: <Vagrant::Bundler::SolutionFile:/Users/conduct/Sites/example.com/trellis/.vagrant/plugins.json:/Users/conduct/Sites/example.com/trellis/.vagrant/bundler/local.sol:invalid>
DEBUG bundler: Generating new builtin set instance.
DEBUG bundler: Generating new plugin set instance. Skip gems - []
DEBUG bundler: generating solution set for configured plugins
DEBUG bundler: Current generated plugin dependency list: []
DEBUG bundler: resolving solution from available specification set
DEBUG bundler: solution set for configured plugins has been resolved
DEBUG bundler: activating solution set
DEBUG bundler: Activating solution set: []
DEBUG solution_file: plugin file does not exist, not storing solution
DEBUG bundler: solution set stored to - <Vagrant::Bundler::SolutionFile:/Users/conduct/Sites/example.com/trellis/.vagrant/plugins.json:/Users/conduct/Sites/example.com/trellis/.vagrant/bundler/local.sol:invalid>
 INFO manager: Loading plugins...
DEBUG manager: Enabling globalized plugins
 INFO manager: Plugins:
 INFO manager:   - vagrant-bindfs = [installed: 1.2.0 constraint: > 0]
 INFO manager:   - vagrant-goodhosts = [installed: 1.1.5 constraint: > 0]
 INFO manager:   - vagrant-hostmanager = [installed: 1.8.9 constraint: > 0]
 INFO manager:   - vagrant-parallels = [installed: 2.3.1 constraint: > 0]
 INFO manager:   - vagrant-trellis-cert = [installed: 0.5.2 constraint: > 0]
DEBUG solution_file: new solution file instance plugin_file=/Users/conduct/.vagrant.d/plugins.json solution_file=/Users/conduct/Sites/example.com/trellis/.vagrant/bundler/global.sol
DEBUG solution_file: reading solution file - /Users/conduct/Sites/example.com/trellis/.vagrant/bundler/global.sol
DEBUG solution_file: solution validation check CHECKSUM c3ec7e06d7349e5f0f5b4a2aad6ad9a3adde27e7c8836e49c7c8ff667c3896f8 <-> c3ec7e06d7349e5f0f5b4a2aad6ad9a3adde27e7c8836e49c7c8ff667c3896f8 VERSION 2.3.4 <-> 2.3.4
DEBUG solution_file: loading solution dependency list
DEBUG solution_file: solution dependency list: [#<Gem::Resolver::DependencyRequest:0x00007f7ac2a94fe0 @dependency=<Gem::Dependency type=:runtime name="vagrant-bindfs" requirements="= 1.2.0">, @requester=nil>, #<Gem::Resolver::DependencyRequest:0x00007f7ac2a94db0 @dependency=<Gem::Dependency type=:runtime name="os" requirements=">= 0">, @requester=nil>, #<Gem::Resolver::DependencyRequest:0x00007f7ac2a94b08 @dependency=<Gem::Dependency type=:runtime name="vagrant-goodhosts" requirements="= 1.1.5">, @requester=nil>, #<Gem::Resolver::DependencyRequest:0x00007f7ac2a94860 @dependency=<Gem::Dependency type=:runtime name="vagrant-hostmanager" requirements="= 1.8.9">, @requester=nil>, #<Gem::Resolver::DependencyRequest:0x00007f7ac2a945b8 @dependency=<Gem::Dependency type=:runtime name="racc" requirements="~> 1.4">, @requester=nil>, #<Gem::Resolver::DependencyRequest:0x00007f7ac2a94388 @dependency=<Gem::Dependency type=:runtime name="nokogiri" requirements=">= 0">, @requester=nil>, #<Gem::Resolver::DependencyRequest:0x00007f7ac2a940e0 @dependency=<Gem::Dependency type=:runtime name="vagrant-parallels" requirements="= 2.3.1">, @requester=nil>, #<Gem::Resolver::DependencyRequest:0x00007f7ac1bb2dd8 @dependency=<Gem::Dependency type=:runtime name="vagrant-trellis-cert" requirements="= 0.5.2">, @requester=nil>]
DEBUG bundler: solution file in use for init: <Vagrant::Bundler::SolutionFile:/Users/conduct/.vagrant.d/plugins.json:/Users/conduct/Sites/example.com/trellis/.vagrant/bundler/global.sol:valid>
DEBUG bundler: Generating new builtin set instance.
DEBUG bundler: Generating new plugin set instance. Skip gems - []
DEBUG bundler: loading cached solution set
DEBUG bundler: activating solution set
DEBUG bundler: Activating solution set: ["vagrant-bindfs-1.2.0", "os-1.1.4", "vagrant-goodhosts-1.1.5", "vagrant-hostmanager-1.8.9", "racc-1.4.16", "nokogiri-1.14.2-x86_64-darwin", "vagrant-parallels-2.3.1", "vagrant-trellis-cert-0.5.2"]
DEBUG bundler: Activating gem vagrant-bindfs-1.2.0
DEBUG bundler: Activating gem os-1.1.4
DEBUG bundler: Activating gem vagrant-goodhosts-1.1.5
DEBUG bundler: Activating gem vagrant-hostmanager-1.8.9
DEBUG bundler: Activating gem racc-1.4.16
DEBUG bundler: Activating gem nokogiri-1.14.2-x86_64-darwin
DEBUG bundler: Activating gem vagrant-parallels-2.3.1
DEBUG bundler: Activating gem vagrant-trellis-cert-0.5.2
 INFO manager: Loading plugins...
DEBUG manager: Successfully loaded plugin `vagrant-bindfs`.
DEBUG manager: Successfully loaded plugin `vagrant-goodhosts`.
DEBUG manager: Successfully loaded plugin `vagrant-hostmanager`.
DEBUG manager: Successfully loaded plugin `vagrant-parallels`.
DEBUG manager: Successfully loaded plugin `vagrant-trellis-cert`.
DEBUG loader: Load procs for pathname: /Users/conduct/Sites/example.com/trellis/Vagrantfile
 INFO root: Version requirements from Vagrantfile: [">= 2.1.0"]
 INFO root:   - Version requirements satisfied!
 INFO loader: Loading configuration in order: [:home, :root]
DEBUG loader: Loading from: root (evaluating)
Windows? => false
ansible? => false
override? => 
Windows? => false
ansible? => false
override? => 
DEBUG provisioner: Provisioner defined: 
Windows? => false
ansible? => false
override? => 
Windows? => false
ansible? => false
override? =>

which method still returns false instead of a positive.

And here’s the ansible_local bit for completeness, though not terribly helpful:

 INFO interface: info: Running provisioner: ansible_local...
 INFO interface: info: ==> default: Running provisioner: ansible_local...
==> default: Running provisioner: ansible_local...
 INFO environment: Running hook: provisioner_run
 INFO runner: Running action: provisioner_run #<Method: Vagrant::Action::Builtin::Provision#run_provisioner(env) /opt/vagrant/embedded/gems/2.3.4/gems/vagrant-2.3.4/lib/vagrant/action/builtin/provision.rb:137>
 INFO warden: Calling IN action: #<Proc:0x00007f7aa58ec598 /opt/vagrant/embedded/gems/2.3.4/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:126 (lambda)>
DEBUG ssh: Re-using SSH connection.
 INFO ssh: Execute: test -f '/home/vagrant/trellis/dev.yml' (sudo=false)
DEBUG ssh: Exit status: 0
 INFO ansible_guest: Checking for Ansible installation...
DEBUG ssh: Checking whether SSH is ready...
 INFO ssh: SSH is ready!
DEBUG ssh: Re-using SSH connection.
 INFO ssh: Execute:  (sudo=false)
DEBUG ssh: Exit status: 0
DEBUG guest: Searching for cap: ansible_installed
...
DEBUG guest: Checking in: linux
DEBUG guest: Found cap: ansible_installed in linux
DEBUG ssh: Checking whether SSH is ready...
DEBUG ssh: Re-using SSH connection.
 INFO ssh: SSH is ready!
DEBUG ssh: Re-using SSH connection.
 INFO ssh: Execute:  (sudo=false)
DEBUG ssh: Exit status: 0
DEBUG guest: Searching for cap: ansible_installed
DEBUG guest: Checking in: linux
DEBUG guest: Found cap: ansible_installed in linux
 INFO guest: Execute capability: ansible_installed [#<Vagrant::Machine: default (VagrantPlugins::Parallels::Provider)>, "2.10.7"] (ubuntu)
DEBUG ssh: Re-using SSH connection.
 INFO ssh: Execute: test -x "$(command -v ansible)"&& ansible --version | grep 'ansible 2.10.7' (sudo=false)
DEBUG ssh: stderr: 41e57d38-b4f7-4e46-9c38-13873d338b86-vagrant-ssh
DEBUG ssh: Exit status: 1
 INFO interface: detail: Installing Ansible...
 INFO interface: detail:     default: Installing Ansible...
    default: Installing Ansible...
DEBUG ssh: Checking whether SSH is ready...
 INFO ssh: SSH is ready!
 INFO ssh: Execute:  (sudo=false)
DEBUG ssh: stderr: 41e57d38-b4f7-4e46-9c38-13873d338b86-vagrant-ssh
DEBUG ssh: Exit status: 0
DEBUG guest: Searching for cap: ansible_install
DEBUG guest: Checking in: ubuntu
DEBUG guest: Found cap: ansible_install in ubuntu
 INFO guest: Execute capability: ansible_install [#<Vagrant::Machine: default (VagrantPlugins::Parallels::Provider)>, :pip, "2.10.7", "", "sudo apt-get install -y -qq python3-pip"] (ubuntu)
 INFO ssh: Execute: apt-get update -y -qq (sudo=true)
 INFO ssh: Execute: echo; printf $SSH_AUTH_SOCK (sudo=false)
 INFO ssh: Setting SSH_AUTH_SOCK remotely: /tmp/ssh-XXXXv56ZEk/agent.1119
DEBUG ssh: Exit status: 0
 INFO ssh: Execute: apt-cache show python-dev-is-python3 (sudo=false)
DEBUG ssh: Exit status: 0
 INFO ssh: Execute: DEBIAN_FRONTEND=noninteractive apt-get install -y -qq --option "Dpkg::Options::=--force-confold" build-essential curl git libssl-dev libffi-dev python-dev-is-python3 (sudo=true)

 INFO interface: detail: Installing pip... (for Ansible installation)
 INFO interface: detail:     default: Installing pip... (for Ansible installation)
    default: Installing pip... (for Ansible installation)
DEBUG ssh: Re-using SSH connection.
 INFO ssh: [...stripped to save space...] 
DEBUG ssh: stderr: WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
DEBUG ssh: Exit status: 0
ERROR warden: Error occurred: The requested Ansible version (2.10.7) was not found on the guest.
Please check the Ansible installation on your Vagrant guest system (currently: 2.10.17),
or adapt the provisioner `version` option in your Vagrantfile.
See https://docs.vagrantup.com/v2/provisioning/ansible_common.html#version
for more information.
 INFO warden: Beginning recovery process...
...

I’m way out of my depth here, but I took a look at the system() call in the which method and tried running in Terminal for both ‘trellis’ and ‘ansible-playbook’:

MBP-2020:example.com conduct$ ruby -e 'system("trellis --help", %i(out err) => File::NULL)'
MBP-2020:example.com conduct$ ruby -e 'system("ansible-playbook --help", %i(out err) => File::NULL)'
MBP-2020:example.com conduct$ 

I read that the File:NULL at the end is to suppress the output, so I guess this is the expected result.

I tried again without suppression and got the following error for ansible-playbook. Trellis returned the help text as expected:

MBP-2020:example.com conduct$ ruby -e 'system("ansible-playbook --help")'
ImportError: dlopen(/Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so, 0x0002): tried: '/Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/System/Volumes/Preboot/Cryptexes/OS/Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so' (no such file), '/Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/lib/python3.9/site-packages/_cffi_backend.cpython-39-darwin.so' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64'))
thread '<unnamed>' panicked at 'Python API call failed', /Users/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.15.2/src/err/mod.rs:582:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/bin/ansible-playbook", line 5, in <module>
    from ansible.cli.playbook import main

  [... stripped other tracebacks to save space...]

  File "/Users/conduct/Sites/example.com/trellis/.trellis/virtualenv/lib/python3.9/site-packages/cryptography/hazmat/primitives/padding.py", line 11, in <module>
    from cryptography.hazmat.bindings._rust import (
pyo3_runtime.PanicException: Python API call failed

Architecture problem?!? Maybe this explains why this issue popped up around the time I upgraded to Ventura.

I could test this by running an x86 version of python ('cause I’m dying to know), but think I’ll look to the future and put my efforts into the Lima migration!

Thanks @swalkinshaw for your quick replies and troubleshooting suggestions!

[swalkinshaw]

I had looked into those system calls but didn’t get you to debug them for some reason, so great idea. That seems to be the problem

Python should work fine regardless of x86; you just need to re-install it from scratch I think.

[swalkinshaw]

@conductdesign I moved all these replies to your other thread to keep it together. Your problem is likely different than some other people’s.

[talss89]

Weird how things line up. I’ve just spent the weekend cross-compiling Python for termux / aarch64…

I’m not sure if you’ve ever run in Rosetta (perhaps opened a terminal in Rosetta, then tried to install Python / Trellis), but it’s possible pip has downloaded or built the x86_64 wheel.

Making sure you’re not running in Rosetta (check with uname -m !== x86_64), then uninstall and reinstall cffi.

[conductdesign]

@swalkinshaw, indeed a fresh install of Python solved it.

I’ll loosely summarize in case anyone else encounters the same issue:

1. Install pyenv via homebrew (Follow the post-install setup).
2. (Re)Install python via pyenv. I did a ‘global’ install.
3. It may not be necessary to destroy the vagrant box, but I did anyway.
4. Force re-initialize the trellis project: trellis init --force.
5. trellis up (and reimport the database for existing project).

Windows? => false
ansible? => true
override? => 

Wooot!

==> default: Running provisioner: ansible...

Boom. works.
(But strangely all of my debugging puts had piled up in my ~/.ssh/config… which threw an error. But way easier to track down and fix.)

@talss89, I double-checked the Rosetta thing. Terminal was not using rosetta, but my whole system was migrated from my older laptop, so who knows what’s been dragged over slipped through the cracks.

Thanks guys – I can move on with my life now