Vagrant up failing at: nginx_test failure

peterorpete · July 7, 2016, 2:47pm

Hi I’m a very new to trellis and bedrock so apologies if i dont explain myself clearly.

Been trying to use Vagrant up, but it fails at the nginx_test failure stage the error message is as follows:

The conditional check 'nginx_test | success' failed. The error was: |failed
expects a dictionary

The error appears to have been in '/Users/USERNAME/Sites/EXAMPLE.COM/ansible/roles/common/tasks/reload_nginx.yml': line 7, column
3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: reload nginx
  ^ here

fatal: [default]: FAILED! => {"failed": true}

Not too sure how to fix the above, tried to install nginx via brew install nginx but didnt work.
Any help would be appreciated.
The below is the content of that file

---
- name: test nginx conf
  command: nginx -t
  register: nginx_test
  changed_when: false

- name: reload nginx
  service:
    name: nginx
    state: reloaded
  when: nginx_test | success
  changed_when: false

fullyint · July 8, 2016, 12:31am

You won’t need to brew install nginx on your control machine because the issue appears to be with the nginx on the managed machine (the Vagrant VM).

As you work toward your first successful vagrant up, seeing errors along the way, it often helps to vagrant destroy and start over with a fresh VM via a new vagrant up.

If recreating the VM doesn’t resolve it, could you scroll up higher in the output to see if there was a failed task up higher, before the - name: reload nginx handler task failed, and share the output?

Normally the output will only ever include one failed task because Ansible aborts after a single failed task. However, after a task fails, Ansible still runs handlers (tasks designated as such, including the reload nginx task giving you trouble), so you may have both a failed task and a failed handler task.

If you don’t find an earlier failing task, could you provide more debug info via these two steps?

1 - Temporarily add a debug task here between the test nginx conf task and reload nginx task.

- name: debug nginx_test var
  debug:
    msg: "{{ nginx_test | to_nice_json }}"

2 - With the Vagrant VM already up and running, run the playbook with greater verbosity via the -vvvv option. I think you’re on OS X so this command should do it:

ansible-playbook dev.yml -i .vagrant/provisioners/ansible/inventory/vagrant_ansible_inventory -vvvv

Please share the full output for the debug task you created
(will appear with header RUNNING HANDLER [common : debug])
and the task before and the task after.
If there is a task that fails earlier in the playbook, please share its output too, as a bonus

rpk · August 5, 2016, 12:15pm

I ran into the same issue like @peterorpete . I did a fresh install of the latest version of Trellis with Ansible 2.1.1 ( which has fixed the issue which made it impossible to provision with Ansible 2.1).

I’ve tried the vagrant destroy in the first place like @fullyint suggested. Same result. Then I’ve added the debug statement into the reload_ngnix.ymland did a vagrant destroy and vagrant up again. But i don’t get any debug related output. Just the same like before:

RUNNING HANDLER [common : reload nginx] ****************************************
System info:
  Ansible 2.1.1.0; Darwin
  Trellis at "Upgrade Ubuntu from 14.04 Trusty to 16.04 Xenial"
---------------------------------------------------
The conditional check 'nginx_test | success' failed. The error was: |failed
expects a dictionary

The error appears to have been in '/Users/danton/Sites/trellisexample/trellis
/roles/common/tasks/reload_nginx.yml': line 11, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: reload nginx
  ^ here

fatal: [default]: FAILED! => {"failed": true}

Then I tried the second suggested step and ran the ansible-playbook command. But I got the following failure:

PLAY [WordPress Server: Install LEMP Stack with PHP 7.0 and MariaDB MySQL] *****

TASK [setup] *******************************************************************
System info:
  Ansible 2.1.1.0; Darwin
  Trellis at "Upgrade Ubuntu from 14.04 Trusty to 16.04 Xenial"
---------------------------------------------------
Failed to connect to the host via ssh.
fatal: [default]: UNREACHABLE! => {"changed": false, "unreachable": true}
 [WARNING]: Could not create retry file 'dev.retry'.         [Errno 2] No such file or directory: ''


PLAY RECAP *********************************************************************
default                    : ok=0    changed=0    unreachable=1    failed=0

But the machines status in contrast is:

Current machine states:

default                   running (virtualbox)

The VM is running. To stop this VM, you can run `vagrant halt` to
shut it down forcefully, or you can run `vagrant suspend` to simply
suspend the virtual machine. In either case, to restart it again,
simply run `vagrant up`.

Unsure how to proceed. Just let me know if you need any further informations. Best regards r.

fullyint · August 5, 2016, 3:28pm

@rpk
Could you share the output from doing a vagrant ssh into your VM and running

nginx -v && sudo nginx -t

That should be the output loaded into nginx_test that seems to be giving the trouble.

Could you also tell us your Vagrant version?

vagrant -v

If the output above doesn’t reveal the problem/solution, could you

make your output verbose by temporarily inserting
ansible.verbose = 'vvvv'
into your Vagrantfile between these two lines
comment out when: nginx_test | success in roles/common/tasks/reload_nginx.yml
vagrant destroy -f && vagrant up

Hopefully then the debug will print this time and the vvvv verbose output may reveal something helpful.

rpk · August 5, 2016, 4:46pm

thanks for the fast feedback! ok a quick reply on the first part of your answer. when i ssh into the machine i get the following output:

vagrant@trellis:~$ nginx -v && sudo nginx -t
nginx version: nginx/1.11.2
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Nothing failing or giving any clues apparently. And the Vagrant version i am using is:

$> vagrant -v
Vagrant 1.8.5

And about the commenting out in reload_nginx.ymlshall the debug insert remain? and shall the changed_whenalso be commented out like?

---
- name: test nginx conf
  command: nginx -t
  register: nginx_test
  changed_when: false

#- name: debug nginx_test var
#  debug:
#    msg: "{{ nginx_test | to_nice_json }}"

- name: reload nginx
  service:
    name: nginx
    state: reloaded
  #when: nginx_test | success
  #changed_when: false

fullyint · August 5, 2016, 5:01pm

Retain the debug. I anticipate Its output will be the most useful of anything.
The changed_when doesn’t matter either way, so…

---
- name: test nginx conf
  command: nginx -t
  register: nginx_test
  changed_when: false

- name: debug nginx_test var
  debug:
    msg: "{{ nginx_test | to_nice_json }}"

- name: reload nginx
  service:
    name: nginx
    state: reloaded
  #when: nginx_test | success
  changed_when: false

With when: nginx_test | success commented out, it should no longer cause the yml parsing error the aborts the playbook before the debug output can print.

When you’ve received the error in question, are there any other failing tasks prior, or does it appear that the playbook runs all the way through the wordpress-install role?

rpk · August 5, 2016, 5:55pm

Ok I did as you suggested. With a forced vagrant destroy and the spin up afterwards everything worked without an error.

PLAY RECAP *********************************************************************
default                    : ok=102  changed=78   unreachable=0    failed=0


==> default: Machine 'default' has a post `vagrant up` message. This is a message
==> default: from the creator of the Vagrantfile, and not from Vagrant itself:
==> default:
==> default: Your Trellis Vagrant box is ready to use!
==> default: * Composer and WP-CLI commands need to be run on the virtual machine.
==> default: * You can SSH into the machine with `vagrant ssh`.
==> default: * Then navigate to your WordPress sites at `/srv/www`.

Afterwards I’ve uncommented the when: nginx_test | success part and did a vagrant destroy -f and vagrant upagain. Then the error reappeared. But no further information again (the vvvv setting for ansible doesnt provide anything in this chase nor the debug statement in the yml file)

RUNNING HANDLER [common : reload nginx] ****************************************
System info:
  Ansible 2.1.1.0; Darwin
  Trellis at "Upgrade Ubuntu from 14.04 Trusty to 16.04 Xenial"
---------------------------------------------------
The conditional check 'nginx_test | success' failed. The error was: |failed
expects a dictionary

The error appears to have been in '/Users/myuser/Sites/trellisexample/trellis
/roles/common/tasks/reload_nginx.yml': line 11, column 3, but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:


- name: reload nginx
  ^ here

fatal: [default]: FAILED! => {"failed": true}

fullyint · August 5, 2016, 6:48pm

Thanks for your work on this @rpk

Once I update to Ansible 2.1.1.0, I can reproduce the problem on vagrant 1.8.5 (running on OS X).
If I roll Ansible back to 2.0.2.0 the problem stops.

In addition, on Ansible 2.1.1.0, the debug task msg seems to not print when run as part of the handler, but it does print if I put the reload nginx include in a regular playbook task file:

- include: "{{ playbook_dir }}/roles/common/tasks/reload_nginx.yml"

I think the success filter had some modifications in Ansible 2.1, which may be responsible for what we’re seeing here. I’ll look into it a bit.

I also tried to reproduce on a staging/production remote with DigitalOcean. If I use Ansible 2.1.1.0 I get other yml errors, which disappear if I roll back to 2.0.2.0.

Could you try rolling your Ansible version back to 2.0.2.0 and see if that solves it for now? I guess that’s the only version the Trellis README reports as supported. I thought only 2.1.0.0 was problematic, but I actually didn’t know anything about 2.1.1.0 till testing it now.

fullyint · August 6, 2016, 3:08am

@rpk and @peterorpete

Thank you for reporting this strange Trellis/Ansible behavior and for your endurance in testing and reporting back. I think roots/trellis#631 should sort things out, enabling Trellis to run successfully on Ansible 2.1.1.0. If you have any inclination to test this update, your help and feedback would be very welcome.

rpk · August 6, 2016, 9:57pm

PLAY RECAP *********************************************************************
default                    : ok=103  changed=80   unreachable=0    failed=0

wohooo awesome! thanks a lot for the quick fix! works like a charm as you can see above! I’ve redone the whole Trellis set up process with 2.1.1. to write up slight flaws i ran into before which haven’t been show stoppers.

First I’ve removed the vendorfolder and redid ansible-galaxy install -r requirements.yml. The download works without any problem. Only Ansible is throwing a deprecation warning and afterwards it is noted that daemonize is already installed. But overall all roles get downloaded and installed:

- downloading role 'composer', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-composer/archive/1.2.7.tar.gz
- extracting composer to /Users/myuser/Sites/trellisexample/trellis/vendor/roles/composer
- composer was installed successfully
- downloading role 'ntp', owned by resmo
- downloading role from https://github.com/resmo/ansible-role-ntp/archive/0.3.0.tar.gz
- extracting ntp to /Users/myuser/Sites/trellisexample/trellis/vendor/roles/ntp
- ntp was installed successfully
- downloading role 'logrotate', owned by nickhammond
- downloading role from https://github.com/nickhammond/ansible-logrotate/archive/fc3ea4.tar.gz
- extracting logrotate to /Users/myuser/Sites/trellisexample/trellis/vendor/roles/logrotate
- logrotate was installed successfully
- downloading role 'swapfile', owned by kamaln7
- downloading role from https://github.com/kamaln7/ansible-swapfile/archive/0.4.tar.gz
- extracting swapfile to /Users/myuser/Sites/trellisexample/trellis/vendor/roles/swapfile
- swapfile was installed successfully
- downloading role 'daemonize', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-daemonize/archive/1.1.0.tar.gz
- extracting geerlingguy.daemonize to /Users/myuser/Sites/trellisexample/trellis/vendor/roles/geerlingguy.daemonize
- geerlingguy.daemonize was installed successfully
- downloading role 'mailhog', owned by geerlingguy
- downloading role from https://github.com/geerlingguy/ansible-role-mailhog/archive/1.0.5.tar.gz
- extracting mailhog to /Users/myuser/Sites/trellisexample/trellis/vendor/roles/mailhog
- mailhog was installed successfully
[DEPRECATION WARNING]: The comma separated role spec format, use the yaml/explicit format instead..
This feature will be removed in a future release.
Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
- dependency geerlingguy.daemonize is already installed, skipping.

When running the Ansible Playbook on vagrant up I get two more deprecation warnings:

TASK [logrotate : nickhammond.logrotate | Setup logrotate.d scripts] ***********
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your
playbooks so that the environment value uses the full variable syntax
('{{logrotate_scripts}}').
This feature will be removed in a future release.
Deprecation warnings can be disabled by setting deprecation_warnings=False in
ansible.cfg.
changed: [default] => (item={u'path': u'/srv/www/**/logs/*.log', u'scripts': {u'postrotate': u'service nginx rotate', u'prerotate': u'if [ -d /etc/logrotate.d/httpd-prerotate ]; then \\\n      run-parts /etc/logrotate.d/httpd-prerotate; \\\n    fi \\\n'}, u'options': [u'weekly', u'maxsize 50M', u'missingok', u'rotate 8', u'compress', u'delaycompress', u'notifempty', u'create 0640 vagrant www-data', u'sharedscripts'], u'name': u'wordpress-sites'})

TASK [composer : Install configured globally-required packages.] ***************
[DEPRECATION WARNING]: Using bare variables is deprecated. Update your
playbooks so that the environment value uses the full variable syntax
('{{composer_global_packages}}').
This feature will be removed in a future
release. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
changed: [default] => (item={u'name': u'hirak/prestissimo'})

And one file is not found but that isn’t related to Ansible. It’s includes.d:

TASK [wordpress-setup : Template files out to includes.d] **********************
/bin/sh: line 0: cd: /Users/myuser/Sites/trellisexample/trellis/roles/wordpress-setup/templates/includes.d: No such file or directory

That is everything I found. Best regards r.

fullyint · December 1, 2016, 10:26pm

To close this out in my mind…

the various deprecation warnings were fixed in roots/trellis#681
the file not found warning was fixed in roots/trellis#687