WordPress database error Disk full

trellis
1

I have a roots trellis/sage 10 setup on aws ec2. The deploys initially worked fine but now the site consistently goes down (the DB anyway) and I am not sure what is hogging all the space.
This is my error.log

sudo tail -n 10 error.log
2023/06/12 16:25:00 [error] 2813200#2813200: *2219857 access forbidden by rule, client: 178.159.37.10, server: saharvest.org, request: "HEAD /wp-config.php~~~ HTTP/1.1", host: "saharvest.org"
2023/06/12 16:25:00 [error] 2813200#2813200: *2219860 access forbidden by rule, client: 178.159.37.10, server: saharvest.org, request: "HEAD /wp-config.php~ HTTP/1.1", host: "saharvest.org"
2023/06/12 16:25:00 [error] 2813200#2813200: *2219877 access forbidden by rule, client: 178.159.37.10, server: saharvest.org, request: "HEAD /wp-config~ HTTP/1.1", host: "saharvest.org"
2023/06/12 16:25:00 [error] 2813200#2813200: *2219881 access forbidden by rule, client: 178.159.37.10, server: saharvest.org, request: "HEAD /wp-config.swp HTTP/1.1", host: "saharvest.org"
2023/06/12 16:25:00 [error] 2813200#2813200: *2219883 access forbidden by rule, client: 178.159.37.10, server: saharvest.org, request: "HEAD /.wp-config.php.swp HTTP/1.1", host: "saharvest.org"
2023/06/12 16:25:00 [error] 2813200#2813200: *2219884 access forbidden by rule, client: 178.159.37.10, server: saharvest.org, request: "HEAD /.wp-config.swp HTTP/1.1", host: "saharvest.org"
2023/06/12 16:25:00 [error] 2813200#2813200: *2219887 access forbidden by rule, client: 178.159.37.10, server: saharvest.org, request: "HEAD /.wp-config.php.swo HTTP/1.1", host: "saharvest.org"
2023/06/12 16:25:00 [error] 2813201#2813201: *2219896 access forbidden by rule, client: 178.159.37.10, server: saharvest.org, request: "HEAD /wp-config-sample.php.bak HTTP/1.1", host: "saharvest.org"
2023/06/13 12:36:11 [error] 2813201#2813201: *2223764 FastCGI sent in stderr: "PHP message: WordPress database error Disk full (/tmp/#sql-temptable-776c-1116b8-103.MAI); waiting for someone to free some space... (errno: 28 "No space left on device") for query SHOW COLUMNS FROM `wp_aioseo_posts` made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/saharvest/index.php'), language_attributes, get_language_attributes, apply_filters('language_attributes'), WP_Hook->apply_filters, AIOSEO\Plugin\Common\Social\Social->addAttributes, AIOSEO\Plugin\Common\Social\Facebook->getObjectType, AIOSEO\Plugin\Common\Meta\MetaData->getMetaData, AIOSEO\Plugin\Common\Models\Post::getPost, AIOSEO\Plugin\Common\Models\Model->__construct, AIOSEO\Plugin\Common\Models\Model->getColumns, AIOSEO\Plugin\Common\Utils\Database->executePHP message: WordPress database error Disk full (/tmp/#sql-temptable-776c-1116b8-104.MAI); waiting for someone to free some space... (errno: 28 "No space left on device") for query SHOW COLUMNS FROM `wp_aioseo_posts` made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/saharvest/index.php'), language_attributes, get_language_attributes, apply_filters('language_attributes'), WP_Hook->apply_filters, AIOSEO\Plugin\Common\Social\Social->addAttributes, AIOSEO\Plugin\Common\Social\Facebook->getObjectType, AIOSEO\Plugin\Common\Meta\MetaData->getMetaData, AIOSEO\Plugin\Common\Models\Post::getPost, AIOSEO\Plugin\Common\Utils\Database->model, AIOSEO\Plugin\Common\Models\Model->__construct, AIOSEO\Plugin\Common\Models\Model->getColumns, AIOSEO\Plugin\Common\Utils\Database->execute" while reading upstream, client: 105.245.168.145, server: saharvest.org, request: "GET /become-a-beneficiary/ HTTP/1.1", upstream: "fastcgi://unix:/var/run/php-fpm-wordpress.sock:", host: "saharvest.org"
2023/06/13 12:36:11 [error] 2813201#2813201: *2223764 FastCGI sent in stderr: "PHP message: WordPress database error Disk full (/tmp/#sql-temptable-776c-1116b8-106.MAI); waiting for someone to free some space... (errno: 28 "No space left on device") for query SHOW FULL COLUMNS FROM `wp_options` made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/saharvest/index.php'), wp_head, do_action('wp_head'), WP_Hook->do_action, WP_Hook->apply_filters, wp_enqueue_scripts, do_action('wp_enqueue_scripts'), WP_Hook->do_action, WP_Hook->apply_filters, wp_enqueue_global_styles, wp_get_global_stylesheet, get_transient, delete_optionPHP message: WordPress database error Disk full (/tmp/#sql-temptable-776c-1116b8-107.MAI); waiting for someone to free some space... (errno: 28 "No space left on device") for query SHOW FULL COLUMNS FROM `wp_options` made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/saharvest/index.php'), wp_head, do_action('wp_head'), WP_Hook->do_action, WP_Hook->apply_filters, wp_enqueue_scripts, do_action('wp_enqueue_scripts'), WP_Hook->do_action, WP_Hook->apply_filters, wp_enqueue_global_styles, wp_get_global_stylesheet, get_transient, delete_optionPHP message: WordPress database error Disk full (/tmp/#sql-temptable-776c-1116b8-108.MAI); waiting for someone to free some space... (errno: 28 "No space left on device") for query SHOW FULL COLUMNS FROM `wp_options` made by require('wp-blog-header.php'), require_once('wp-includes/template-loader.php'), include('/themes/saharvest/index.php'), wp_head, do_action('wp_head'), WP_Hook->do_action, WP_Hook->apply_filters, wp_enqueue_scripts, do_action('wp_enqueue_scripts'), WP_Hook->do_action, WP_Hook->apply_filters, wp_enqueue_global_styles, wp_get_global_stylesheet, set_transient, update_option" while reading upstream, client: 105.245.168.145, server: saharvest.org,

I have previously increased the physical ssd space on the ec2 instance and this worked but now I have the same problem and I have not done any deploys since.
Any help would be appreciated.

2

The error is being thrown when writing to /tmp, which is likely a different size partition to /. Does restarting MySQL temporarily solve the issue?

That said, an EC2 instance should have sensible defaults set for /tmp, so this would indicate an issue with a query or your MySQL configuration. Increasing the /tmp size would likely just push the problem further down the road.

What does df -h return? Can you see large sql-temptable type files in /tmp?

Depending on your MySQL version, system resources and query usage, you may have success by forcing MySQL to place temporary tables in RAM, instead of /tmp. You may wish to try:

SET GLOBAL internal_tmp_mem_storage_engine=MEMORY

More information on how MySQL uses temporary storage is here.

1 Like
3

Thanks for the feedback. So it turns out that I although I had extended the volume I had not extended the filesystem.
The following details are for an ubuntu linux ec2 instance.
First create a snapshot of your volume in case there is a problem.
SSH into your instance and run the following command (my filesystem is in /dev/root)

sudo resize2fs /dev/root

then check that the filesystem has indeed been extended

sudo df -h

Hopefully my disk full issues are now resolved but time will tell. I will update this post if there are any further developments.