If your Drupal website is suddenly slow, showing errors, or even inaccessible, and your server reports full disk space, you're facing a common challenge. This article will delve into the diagnosis process for a Drupal 7 website's full disk space issue and provide a set of comprehensive solutions, from emergency cleanup to long-term optimization.
1) The Problem Emerges: Drush Command Fails with "No space left on device"
It all started with a seemingly simple cache clear command: drush cc all. However, Drush returned an error:
Drush command terminated abnormally due to an unrecoverable error.
PDOException: SQLSTATE[HY000]: General error: 3 Error writing file '/tmp/MYfd=295' (OS errno 28 - No space left on device)
This error message clearly states: insufficient server disk space. MySQL couldn't create a temporary file in the /tmp directory, leading to a failed database query, which in turn prevented Drupal from functioning correctly and clearing its cache.
A quick check with the df -h command confirmed the root partition /dev/vda2 was at 100% usage, with 0 available space. This was the root cause of all subsequent problems.
2) Phase One: Emergency Stopgap – Rapid Disk Space Release
When disk space reaches 100%, the system becomes extremely unstable and can even crash. The immediate priority is to free up space.
2.1) 1. Check the Backup Directory (/var/backups)
This is one of the most common culprits for large disk space consumption. Running ls -lh /var/backups often reveals numerous backup files, for example:
- 7.6G drupal7_backup_2025-02-23.tar.gz
- 4.7G database-backup20240725ws.sql
- 4.3G database-backup.sql
- 3.9G wishiny_01bc20220813.sql
These massive backup files collectively occupy tens of gigabytes of space.
Solution:
Carefully delete old backups that are no longer needed. Prioritize deleting the oldest or largest files.
Example commands:
cd /var/backups/
rm database-backup20240725ws.sql
rm drupal7_backup_2025-02-23.tar.gz # Only if you have confirmed other copies or no longer need itAfter each deletion, immediately run df -h to check if disk space has been freed. The goal is to bring /dev/vda2 usage below 90%.
2.2) 2. Locate Rapidly Growing Active Files:
If deleting old backups isn't enough, or if disk space is quickly filled up again, it's highly likely that an active process is generating a large number of new files.
Diagnostic command: Find files larger than 10MB that have been modified within the last 60 minutes:
sudo find / -type f -size +10M -mmin -60 -print0 | xargs -0 du -h | sort -rhDiagnostic Results (Example):
The output clearly points to abnormal growth of MySQL database files as the primary cause of rapid disk space exhaustion:
- 51G /var/lib/mysql/chogory_db/cache_form.ibd: An astounding 51GB! Drupal's form cache table should not be this large. It indicates a potential misconfiguration or a process writing excessive data.
- 1.9G /var/lib/mysql/drupal7/semaphore.ibd: Drupal's semaphore table, indicating a large number of concurrent operations or improper lock releases.
- 1.2G /var/lib/mysql/chogory_db/watchdog.ibd: Drupal's log table, showing extremely frequent logging with no cleanup mechanism.
- 485M /var/lib/mysql/drupal7/watchdog.ibd: Another watchdog table from a different Drupal database, also significantly large.
- 785M /var/lib/mysql/undo_002 and 385M /var/lib/mysql/undo_001: InnoDB undo log files, a product of extensive transaction operations or long-running transactions.
- 58M /var/log/mysql/mysql-slow.log: MySQL slow query log, indicating numerous time-consuming queries being recorded.
- 111M /var/log/btmp: Failed login attempts log, possibly suggesting brute-force attacks.
- Numerous 32M /var/lib/mysql/#innodb_redo/#ib_redoXXXXX_tmp files: InnoDB redo log temporary files, indicating a high volume of write activity.
Solution: Targeted Truncation of Database Files and Logs:
Warning: TRUNCATE TABLE permanently deletes all data in a table. For cache and log tables, this is generally safe as Drupal can regenerate them or their loss does not affect core functionality. Understand the risks before proceeding.
Empty abnormally large Drupal cache tables:
Connect via MySQL client and execute the commands:
mysql -u root -p
-- For chogory_db database
USE chogory_db;
TRUNCATE TABLE cache_form; -- The 51GB culprit, prioritize this
TRUNCATE TABLE watchdog; -- Massive log table
TRUNCATE TABLE cache_views;
TRUNCATE TABLE cache_menu;
TRUNCATE TABLE cache_field;
TRUNCATE TABLE cache;
TRUNCATE TABLE cache_advagg_aggregates;-- For drupal7 database
USE drupal7;
TRUNCATE TABLE semaphore; -- To address concurrency/lock issues
TRUNCATE TABLE watchdog;
TRUNCATE TABLE advagg_aggregates;
TRUNCATE TABLE advagg_aggregates_versions;
TRUNCATE TABLE cache_update;After execution, immediately run df -h to check space release.
Clear large log files:
MySQL slow query log:
sudo truncate -s 0 /var/log/mysql/mysql-slow.logFailed login log (btmp):
sudo truncate -s 0 /var/log/btmp
These operations safely empty file contents without deleting the files themselves, preventing service interruption.
1) Phase Two: Root Cause Resolution – Website Performance Optimization & Prevention
Simply clearing files is a temporary fix. To prevent recurrence and fundamentally improve website performance, systemic optimization is required.
1.0.1) 1. Memory Configuration Optimization: Address Missing Swap & MySQL Memory Bottleneck
Your top output shows high server load, no Swap space, and MySQL (mysqld) consuming 137.5% CPU and 44.5% memory (approx. 2.6GB), making it the primary bottleneck.
Configure Swap Space (Urgent & Critical!)
Lack of Swap leads to system crashes when memory runs out.
sudo fallocate -l 4G /swapfile # Create a 4GB Swap file
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0'
sudo tee -a /etc/fstab # Auto-mount on boot1.0.2) Optimize MySQL Configuration (my.cnf)
Modify the configuration file, typically at /etc/mysql/my.cnf or /etc/my.cnf.
- innodb_buffer_pool_size: The most critical parameter. For a 5.9GB server, if MySQL is the primary application, set it to 50%-70% of physical memory, e.g., innodb_buffer_pool_size = 3G.
- max_connections: Adjust based on actual concurrent needs, avoiding excessively high values that waste memory.
- tmp_table_size and max_heap_table_size: For complex queries and temporary tables, increase these to prevent writes to disk.
- expire_logs_days: If binary logging (log_bin) is enabled, set it to automatically purge old logs, e.g., expire_logs_days = 7.
- query_cache_type = 0 / query_cache_size = 0: For active Drupal sites, it's generally recommended to disable the query cache, as it can degrade performance under high concurrency.
- After modification, restart the MySQL service: sudo systemctl restart mysql
1.0.3) PHP-FPM Process Optimization
- Multiple PHP-FPM processes consume significant CPU and memory.
- Adjust PHP-FPM Pool Configuration (www.conf or relevant pool file)
- pm.max_children: The maximum number of child processes. Set this reasonably based on each PHP-FPM process's memory consumption (around 80-100MB) and total server memory. For 5.9GB RAM, try 15-20.
- pm = dynamic: Balances memory usage and response time by dynamically creating processes as needed.
- Standardize PHP Version: Your server runs both PHP 7.0 and 7.4. Consolidate to the latest version (PHP 7.4) to simplify management and optimize resources.
- After modification, restart the PHP-FPM service: sudo systemctl restart php7.4-fpm
1.0.4) Drupal-Specific Optimizations
- Cache Configuration: Enable all built-in caches on Drupal's Performance page (/admin/config/development/performance): Cache pages for anonymous users, Cache blocks, and set reasonable cache lifetimes.
- Utilize Memcached as the Cache Backend: Ensure your Drupal settings.php is correctly configured to store cache data in Memcached instead of the database. This significantly reduces database load.
- Regularly run drush cc all to clear Drupal caches.
- Log Management: Enable the Syslog module: This is the best practice for redirecting Drupal's watchdog logs from the database to system log files, drastically reducing database pressure.
- Configure Logrotate: Ensure all system and application logs (including MySQL logs, web server logs, and Syslog) have proper rotation policies to prevent indefinite growth.
- Disable Unnecessary Modules & Optimize Views: Review your Drupal module list and disable any unused modules. Optimize Views queries and enable caching for high-traffic Views.
- PHP OPcache: Verify that your PHP installation has the OPcache extension installed and enabled. This is crucial for PHP performance.
Image Optimization and CDN: Optimize and compress website images, and consider using a Content Delivery Network (CDN) to serve static files, reducing server load.
1.1) Phase Three: Continuous Monitoring & Prevention
- Install Fail2ban: Given the high number of failed login attempts in btmp, install and configure Fail2ban to automatically ban malicious IPs, improving server security and reducing log growth.
- Monitoring Tools: Implement professional server monitoring tools (like Zabbix, Prometheus, etc.) or simple Cron scripts to continuously monitor disk space, CPU, memory, and MySQL status, alerting you when thresholds are met.
- Regular Maintenance: Establish a routine for cleaning logs, checking backups, and optimizing your database.
By implementing this comprehensive set of solutions, you can not only resolve immediate disk space issues but also significantly enhance your Drupal website's performance and stability, preventing future recurrences of these problems.
Back to top
Comments