Freeing Root Disk Space in a Container -- Without using rm

The note is generated by GLM-5.1.

Problem

The root filesystem (/) was 100% full (50G overlay), causing apt install, dpkg, and other system operations to fail with “No space left on device”.

Key observation: /home is mounted on a separate disk (/dev/sdab1, 251G with 229G free), so deleting files in home does not free space on the root overlay.

Constraint: sudo rm is not allowed in this environment — it is deliberately excluded from the sudoers whitelist as a safety measure. We must free 42G of disk space using only whitelisted commands (unlink, systemctl, apt, dpkg, ln, etc.).

Root Cause

The overlay root disk was consumed by log files in /var/log/:

The syslog.1 file (rotated syslog) alone took 42G out of 50G. It was never cleaned up because logrotate compression/purging was not running in the container.

Step-by-Step Fix

1. Identify what’s using the root disk

1
2
3
4
5

# Check disk usage
df -h /

# Find large files on root (not /home which is a separate mount)
find /var/log -type f -size +100M -exec ls -lh {} \;

2. Delete the 42G syslog.1 — Without Using rm

Problem: Neither sudo rm nor sudo truncate were in the sudoers whitelist. Only specific commands were allowed: sudo apt, sudo dpkg, sudo ln, sudo unlink, sudo systemctl, etc. (verified with sudo -l).

Why not rm? In this managed Kubernetes environment, rm is deliberately excluded from the sudo whitelist as a safety measure to prevent accidental deletion of system files. We had to find an alternative command that was allowed.

Solution: Use sudo unlink (allowed) to remove the file:

1

sudo unlink /var/log/syslog.1

What is unlink?

unlink is a low-level system call wrapper that removes a single file. Unlike rm:

• It can only delete one file at a time (no recursive deletion, no wildcards)
• It has no confirmation prompt or -i flag
• It cannot delete directories — only files
• It directly calls the unlink() system call

This makes it safer than rm (no risk of rm -rf / accidents), which is why it was whitelisted while rm was not.

Important: unlink requires the exact file path. You cannot do sudo unlink /var/log/syslog.* — shell globbing works before sudo runs, but the whitelist check happens on the command name, not the arguments.

3. Release the Disk Space — Why unlink Alone Was Not Enough

After running sudo unlink /var/log/syslog.1, the file was deleted (confirmed by ls returning “No such file or directory”), but df -h / still showed 0 free space.

Why? On Linux, when a process has an open file descriptor to a file, deleting the file only removes the directory entry — the inode and data blocks remain on disk until the last file descriptor is closed. The rsyslog daemon still had the old syslog.1 file open, so the 42G was not actually freed.

Fix: Restart rsyslog to release the file descriptor:

1

sudo systemctl restart rsyslog

What does systemctl restart do?

systemctl is the systemd service manager. restart performs:

1. Stop the service (sends SIGTERM, waits, then SIGKILL if needed)
2. Start the service again from a clean state

When rsyslog stops, all its file descriptors (including the one holding the deleted syslog.1) are closed. The kernel then frees the 42G of disk space. When rsyslog starts again, it creates fresh log files.

After this, df -h / immediately showed 42G free.

4. Verify

1
2
3

df -h /
# Filesystem      Size  Used Avail Use% Mounted on
# overlay          50G  8.2G   42G  17% /

Lessons Learned

1. Overlay filesystem in containers: The root / is an overlay with a fixed-size upper layer. Deleting files from separate mounts (like /home) does not free overlay space.

2. Sparse files: lastlog showed 408G in ls -l but only used 2.8M on disk. Always check with du -h to get real disk usage.

3. Deleted files still occupy space: On Linux, if a process has an open file descriptor to a deleted file, the disk space is not freed. You must restart or kill the holding process (e.g., sudo systemctl restart rsyslog) to reclaim space. Just deleting the file is not enough.

4. Sudo whitelist — rm not allowed, use unlink: In managed container environments, sudo rm is often blocked as a safety measure. unlink is a safer alternative that can only remove a single file. Always check sudo -l to see what’s allowed.

5. Container log management: Containers often lack proper logrotate setup. Consider adding a cron job or periodic cleanup:

   1 2 3

   # Add to crontab or run periodically sudo truncate -s 0 /var/log/syslog.1 sudo journalctl --vacuum-size=50M

Environment Details

• OS: Ubuntu 22.04.5 LTS
• Container: Kubernetes pod (non-privileged, containerd runtime)
• Root FS: overlay, 50G (upper layer writable)
• Home FS: /dev/sdab1, 251G (separate mount)
• Date: 2026-05-07