WordPress 6.0 update deploy failed

Thanks, that did the trick :slight_smile:

Summoning @tGeorgel, @Mike_Wheeler:
The deployment task for multisite sites now doesn’t use temporary PHP constants anymore and
should work now in any case (initial multisite site deployment, already configured multisite site, with WordPress < 6.0 and with WordPress ≥ 6.0): Exempt from `is-installed` check the DB error dump PHP warning for not yet set up multisite sites by strarsis · Pull Request #1391 · roots/trellis · GitHub

2 Likes

Just a quick note for people like me using the roles trellis_flush_rewrite_rules_during_deploy and/or trellis-backup-during-deploy:

Both of them mirrored the “WP installed?”-steps from before and will throw the same errors when invoked…

They have been updated to reflect the current changes (thanks @codepuncher !) and you may need to manually update and install these in your galaxy.yml.

And a big thanks to @strarsis for the work done on this!

2 Likes

Hm, could common tasks like WP Installed? be re-used by Trellis Roles from Trellis core?
This reduces code duplication and ensure that always one and most recent task is used.

1 Like

Please pardon me for adding to a closed thread.

I’m hitting a problem in a previous Trellis droplet I’ve rebased over latest Trellis codebase.

On initial deploy, WordPress Installed (non-multisite) fails.

On the server itself, where I would expect wp core --is-installed to return, “Error: The site you have requested is not installed.”, it returns nothing.

Wp is installed: wp --info returns:

OS:	Linux 5.4.0-122-generic #138-Ubuntu SMP Wed Jun 22 15:00:31 UTC 2022 x86_64
Shell:	/bin/bash
PHP binary:	/usr/bin/php8.0
PHP version:	8.0.21
php.ini used:	/etc/php/8.0/cli/php.ini
MySQL binary:	/usr/bin/mysql
MySQL version:	mysql  Ver 15.1 Distrib 10.5.16-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
SQL modes:
WP-CLI root dir:	phar://wp-cli.phar/vendor/wp-cli/wp-cli
WP-CLI vendor dir:	phar://wp-cli.phar/vendor
WP_CLI phar path:	/srv/www/example.com/releases/20220803202003
WP-CLI packages dir:
WP-CLI global config:
WP-CLI project config:	/srv/www/example.com/releases/20220803202003/wp-cli.yml
WP-CLI version:	2.6.0

Provisioning seems to have run successfully. What am I missing here? Did I miss, or neglect to remove a file or variable?

FYI:

non-zero return code
fatal: [143.198.161.xxx]: FAILED! => {
    "changed": false,
    "cmd": [
        "wp",
        "core",
        "is-installed",
        "--skip-plugins",
        "--skip-themes"
    ],
    "delta": "0:00:00.302903",
    "end": "2022-08-03 21:15:44.909393",
    "failed_when_result": true,
    "invocation": {
        "module_args": {
            "_raw_params": "wp core is-installed --skip-plugins --skip-themes",
            "_uses_shell": false,
            "argv": null,
            "chdir": "/srv/www/example.com/releases/20220803211520",
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": false
        }
    },
    "rc": 255,
    "start": "2022-08-03 21:15:44.606490",
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}

Exit code 255 indicates a fatal PHP error (the exit code is from the PHP interpreter itself by the way). Can you check the PHP error logs (site-specific and system-wide) to find out what exactly caused it? It may be some plugin or theme.

1 Like

Thanks, starsis. This is embarrassing, but, where are those php error logs? I’m only finding:

/srv/www/example.com/logs/error.log (empty)
/srv/www/example.com/logs/access.log (empty)
/var/log/php8.0-fpm.log
[03-Aug-2022 20:42:38] NOTICE: fpm is running, pid 49585
[03-Aug-2022 20:42:38] NOTICE: ready to handle connections
[03-Aug-2022 20:42:38] NOTICE: systemd monitor interval set to 10000ms
[03-Aug-2022 20:43:42] NOTICE: Reloading in progress ...
[03-Aug-2022 20:43:42] NOTICE: reloading: execvp("/usr/sbin/php-fpm8.0", {"/usr/sbin/php-fpm8.0", "--nodaemonize", "--fpm-config", "/etc/php/8.0/fpm/php-fpm.conf"})
[03-Aug-2022 20:43:42] NOTICE: using inherited socket fd=7, "/run/php/php8.0-fpm.sock"
[03-Aug-2022 20:43:42] NOTICE: using inherited socket fd=7, "/run/php/php8.0-fpm.sock"
[03-Aug-2022 20:43:42] NOTICE: fpm is running, pid 49585
[03-Aug-2022 20:43:42] NOTICE: ready to handle connections
[03-Aug-2022 20:43:42] NOTICE: systemd monitor interval set to 10000ms
/var/log/nginx/access.log
/var/log/nginx/error.log (empty)

And some that don’t seem relevant:

/var/log/mail.log
/var/log/dpkg.log
/var/log/kern.log
/var/log/cloud-init-output.log
/var/log/cloud-init.log
/var/log/fontconfig.log
/var/log/ubuntu-advantage-timer.log
/var/log/alternatives.log
/var/log/auth.log
/var/log/unattended-upgrades/unattended-upgrades.log
/var/log/unattended-upgrades/unattended-upgrades-dpkg.log
/var/log/unattended-upgrades/unattended-upgrades-shutdown.log
/var/log/php8.1-fpm.log
/var/log/apt/history.log
/var/log/apt/term.log
/var/log/landscape/sysinfo.log
/var/log/fail2ban.log
/var/log/droplet-agent.update.log
/sys/kernel/tracing/error_log
/sys/kernel/debug/tracing/error_log

Can you turn PHP error reporting directly on when using the PHP CLI?

php -d display_errors=on $(which wp)

(Note: (which wp prints the path to the wp binary and $() interpolates it so the php interpreter binary runs it, allowing to pass additional options as for force-enabling (hopefully) the PHP error reporting.)

Invoke this command inside the current/release directory of the troubling site on the server.
Does it show the PHP fatal error?

Well, there is no “current” directory as the deploy didn’t get that far. Running as web user. No output. I’m going to try deploying a basic Bedrock site…

Got further!

Finalize the deploy...
rmtree failed: [Errno 13] Permission denied: 'views-for-wpforms'

rmtree failed is related to a Trellis deploy issue:

@mZoo: Were you able to fix/find out the reason for this issue?

Thanks a lot for asking man. Not yet. I spun up a local vagrant box to see if there were php errors and there aren’t (though a bunch of depreciation warnings that pop-up with current WP codebase and php 8.1).

Still getting the 255 error.

Running any php -d display_errors=on $(which wp) command returns nothing. Same after removing all plugins and themes from web/app directories.

Man this just keeps on going. Provisioned a new droplet and after three or four deploys of wpackagist plugin composer failures, back to the original problem from a few days ago:

non-zero return code
sudo: a terminal is required to read the password; either use the -S option
to read from standard input or configure an askpass helper
fatal: [67.205.178.xxx]: FAILED! => {
    "changed": true,
    "cmd": "sudo service php8.0-fpm reload",
    "delta": "0:00:00.013650",
    "end": "2022-08-05 03:40:43.968472",
    "invocation": {
        "module_args": {
            "_raw_params": "sudo service php8.0-fpm reload",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": false
        }
    },
    "rc": 1,
    "start": "2022-08-05 03:40:43.954822",
    "stderr_lines": [
        "sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper"
    ],
    "stdout": "",
    "stdout_lines": []
}

And on the server:

root@...:/home/admin# cat /etc/sudoers.d/web-services
# Ansible managed

web ALL=(root) NOPASSWD: /usr/sbin/service.php8.0-fmp.*

And

root@/home/admin# php --version
PHP 8.0.21 (cli) (built: Aug  1 2022 07:40:52) ( NTS )

and

web@...:/home/admin$ sudo service php8.0-fpm reload
[sudo] password for web:

Additionally

root@:/home/admin# ls -l $(which php)
lrwxrwxrwx 1 root root 21 Aug  5 01:15 /usr/bin/php -> /etc/alternatives/php
root@:/home/admin# ls -l /etc/alternatives/ | grep php
lrwxrwxrwx 1 root root  15 Aug  5 01:15 php -> /usr/bin/php8.0
lrwxrwxrwx 1 root root  22 Aug  5 01:16 php-config -> /usr/bin/php-config8.0
lrwxrwxrwx 1 root root  38 Aug  5 01:16 php-config.1.gz -> /usr/share/man/man1/php-config8.0.1.gz
lrwxrwxrwx 1 root root  24 Aug  5 01:16 php-fpm.sock -> /run/php/php8.0-fpm.sock
lrwxrwxrwx 1 root root  31 Aug  5 01:15 php.1.gz -> /usr/share/man/man1/php8.0.1.gz
lrwxrwxrwx 1 root root  18 Aug  5 01:16 phpize -> /usr/bin/phpize8.0
lrwxrwxrwx 1 root root  34 Aug  5 01:16 phpize.1.gz -> /usr/share/man/man1/phpize8.0.1.gz

Migrating this back to my original post about reload of php-fmp.

1 Like

Resolved the fmp (where should have been fpm) mixup, back to no output from

php -d display_errors=on $(which wp) core

This is with no “current” dir, all plugins and themes removed. Barebones Bedrock composer file.

  1. You are provisioning a bare-bones DO droplet,
    using a minimal Ubuntu 20.04 LTS image?
  2. You are using the latest Trellis release or the latest Trellis commit?
  3. The deployment only fails for a particular Bedrock site, or fail for all Bedrock sites?
    Are there Bedrock sites (also simple ones without extra theme, just for testing) that also fail deployment, except for this particular site?
  4. Do only subsequent deployments fail or also at initial deployment?
  5. Would the failing deployments actually succeed when you use the bare Trellis, directly from current upstream, on a test DO droplet/VM

For further debugging it makes sense setting up a VM and provision it and deploy to it. Create snapshots between those steps so you can revert and find out what changes break it.

There is a similar issue for the install sub-command (not is-installed),
where wp CLI exits silently with exit code 255:

The approach is the same for your issue:
If necessary, you need to manually create the current symlink to the failed release (where wp CLI exits with code 255) and then request the site over HTTP(S). Then check the PHP-FPM global and site-wide logs (again).

Also note that Sage can indeed catch and log some PHP errors in its own log file, so those may not end up in the default PHP-FPM log files:
https://discourse.roots.io/t/tip-http-500-but-no-php-nginx-error-logs/22349

Not sure what you mean by minimal. I have been generating the “Premium AMD with NVMe SSD”, which is, apparently the dashboard default. I was wondering why the cost seemed to go up from what I’ve been used to! Spinning one up now on the “Regular with SSD” plan.

What I have done now is:

  1. Create a brand new project with latest trellis-cli, matching the project url.
  2. Move the .git directory from the previous (trellis and site) codebase.
  3. Make a new branch git checkout -b fresh, and commit it.
  4. Replace my trellis dir with content of my previously rebased codebase and using git status and the wish/gitk app, compare the two. Then, one file at a time, either commit my updates or git restore the/file to the new trellis-cli one.

Here’s what I found (to be significant):

-# Created by trellis-cli v0.9.0
+# Created by trellis-cli v1.7.0

This is the big one:

----------------------- trellis/group_vars/all/users.yml -----------------------
 web_sudoers:
-  - "/usr/sbin/service.php{{ php_version }}-fpm.*"
+  - "/usr/sbin/service php{{ php_version }}-fpm *"

Not sure where to dot/periods came from in the top line. Was I typing this up by hand?

----------------------- trellis/group_vars/all/helpers.yml -----------------------
-composer_authentications: "{{ vault_wordpress_sites[site].composer_authentications | default([]) }}"
+# For backward compatibility, to be removed in Trellis v2.
+site_packagist_org_authentications:
+  - { hostname: repo.packagist.com, username: token, password: "{{ vault_wordpress_sites[site].packagist_token | default('') }}" }
+site_composer_authentications: "{{ vault_wordpress_sites[site].composer_authentications | default([]) }}"
+composer_authentications: "{{ site_packagist_org_authentications + site_composer_authentications }}"

I see that depreciated site_packagist_org_authentications was removed two weeks ago. Not sure how trellis-cli included it yesterday, as I thought it would be generating from the master codebase.

  1. Then I provisioned and deployed, successfully with bare Bedrock site, pointing to the temporary “fresh” branch in wordpress_sites.yml branch: fresh

  2. Reinstated my composer.json and deployed again.

  3. Reinstated my themes and deployed again.

  4. Reinstated my plugins in the codebase and deployed.

  5. Point example.com to the new IP in etc/hosts and reset DNS

  6. Used Delicious Brains’ Migrate DB to pull in db and images from live site.

  7. Hard reset the ‘development’ branch to the ‘fresh’ content.

  8. Reinstantiate branch: development (from fresh)

Unfortunately I’m still unsure why I was getting the failure on wp core is-installed.

At least, once I have confirmed the site is in good shape and have updated the public DNS, I can move on to writing a poem or something. :grinning:

Again, thank you so much for the sage input. :brain:

Cool! I wasn’t familiar with using Linux dollar question mark echo $? to echo the error code of previous command.

What do you mean by that?

(This actually isn’t a Sage-based site.)

PHP-FPM has site-specific log files and global ones. Maybe the error somehow ended up there.

When this isn’t a Sage theme, then maybe that theme still somehow catches the PHP error and logs it somewhere else. So despite the 255 exit of WP CLI for those subsequent releases the site loads fine in frontend backend?

Alas. At this point, I have destroyed the problematic droplet(s), but thanks to you, I go forward with awareness of /var/log/php8.0-fpm.log, /var/log/auth.log and echo $?, among other things.

Roots University continues to succeed. :woman_student:

1 Like