On our Ubuntu 22.04 server (using CyberPanel), we encountered an issue where the system reported "No space left on device" even though there was ample disk space available. Upon further inspection, we discovered that the inodes were completely exhausted.

Steps to Reproduce:

1. Check disk usage using the df -h command:

   df -h

   The system shows sufficient disk space, but the error still occurs.

2. Check inode usage using the df -i command:

   df -i

   Inodes show 100% usage for the root partition (/).

Expected Behavior:

There should be sufficient inodes available for the system to function normally, even if disk space is used up.

Actual Behavior:

The system cannot create new files, and the "No space left on device" error appears even though there is available disk space.

Cause:

The root partition had run out of inodes due to a large number of PHP session files, especially in the /var/lib/lsphp/session/ directory, which contained millions of session files from multiple PHP versions (e.g., lsphp74 and lsphp81).

Solution:

1. Delete PHP Session Files:
   The excessive number of PHP session files can be safely deleted to free up inodes.
   Run the following command to remove all session files:

   sudo rm -rf /var/lib/lsphp/session/*

   Or, if you want to remove only session files older than 1 day:

   sudo find /var/lib/lsphp/session/ -type f -mtime +1 -delete

2. Prevent Future Inode Exhaustion:
   Set up a cron job to automatically clean up old PHP session files:

   • Edit the crontab:

     sudo crontab -e

   • Add the following line to delete session files older than 1 day every day at 3 AM:

     0 3 * * * find /var/lib/lsphp/session/ -type f -mtime +1 -delete

3. Monitor Inode Usage:
   Regularly check inode usage with the command:

   df -i

   This will help detect any future inode exhaustion before it impacts the system.

Additional Information:

• Server: Ubuntu 22.04
• Disk Usage: /dev/sda1 (38GB) — 49% used
• Inodes Exhausted: /var/lib/lsphp/session/ (2.4 million inodes used)

This solution resolved the issue and restored system functionality.