When the Pieces Don’t Add Up: The Mystery of the Full File System

If you boil down all of the reasons that I love what I do it all comes down to one thing. I like solving puzzles. This is the one profession that I have found that not only allows me to do that, but actually handcrafts bespoke puzzles for my enjoyment almost daily.

This puzzle started with a warning that a server’s file system was running out of disk space. This is usually simple enough to diagnose and fix. I’ve seen this happen because of failed log rotation, runaway backup scripts, and several other reasons over the years.

I started simply. Just run df -h. The output showed that all of the partitions had room to spare. Now, this was confusing. I’d seen the warning. There should be some indication from that command’s output that should point me in the right direction.

Now, let’s try this again. I run some variations of the du command until I can see that there are some directories in /var/backup that have several large files in them. It turns out that is where the database backups are being stored. I can see that these directories are not even close to showing as full.

Then I notice that the /var/backup directory is a remote file system. There is an entry in /etc/fstab pointing to a network location. That all seems legitimate.

Finally I start looking into the backups themselves. This particular server was creating database backups hourly. As I was looking through the list of all of the backups I noticed that there were some gaps where some were missing. Some of the gaps were for a few hours. Some were for several days. This looked concerning. Maybe the script that was running the backups was failing intermittently. I made a note to check into this after I had solved the disk space issue.

I continued to scour the file system to try and find the issue with no luck. Then I re-visited the /etc/fstab. There was an entry that had been commented out. Another network drive that may have been used at some point. Maybe this new backup drive had replaced it.

Then I noticed something. That commented network drive had been mounted at /var/database/backups. I remembered that I had seen that directory while looking at the other backups in /var/backups. I took another look. Inside /var/database/backups there were a couple of database backups that were dated earlier than any in the current backup directory. Only a few though.

This was the clue that allowed me to figure out what was happening.

I commented out the current network drive for the backups from the /etc/fstab and rebooted the server.

When I went to the /var/backups directory there were SQL dumps present. When I ran df -h this time it showed that the root partition was 100% full.

It turns out that if the network drive was not available when the server mounted all its file systems the backup script just wrote the backups to the directory that was created as a mount point. When the drive was available again it was mounted over that directory and hid the real disk usage.

I had solved the puzzle. Now I just had to move all of those backups to the network drive now mounted in a different location and free up the used disk space.