In:
DevOps
From “No space left on device” panic to Docker storage mastery
Imagine this: You’re happily running Docker containers, then BAM—your Linux server screams “No space left on device”. You check df -h and see 50GB free space. What gives?
The culprit? /var/lib/docker/overlay2 is ballooning to tens of gigabytes. This guide demystifies exactly what overlay2 is, why it grows, and how to control it like a pro.
By the end, you’ll:
- Understand every file/folder in
/var/lib/docker/overlay2 - Build images and containers while watching overlay2 grow
- Master copy-on-write magic
- Clean up disk space safely
- Never fear “no space left” again
Chapter 1: Does Your System Support overlay2?
What is OverlayFS? (Dead Simple Analogy)
Think of OverlayFS like transparent plastic sheets stacked on a desk:
Bottom Sheet (Read-Only) ← Docker Image Layers
📁 /app
📄 config.txt = "default"
Top Sheet (Read-Write) ← Container Changes
📄 config.txt = "custom" ← Shadows the one below!
Combined View ← What container sees
📁 /app
📄 config.txt = "custom" ← Top sheet wins!Key Point: The bottom sheet (image) never changes. All your writes go to the top sheet.
🧪 Lab 1: Verify OverlayFS Support
# 1. Check if kernel supports overlay
grep -i overlay /proc/filesystems
# Expected: nodev overlay
# 2. Check kernel version (needs 4.0+)
uname -r
# Expected: 5.x or 6.x
# 3. Confirm Docker uses overlay2
docker info | grep "Storage Driver"
# Expected: Storage Driver: overlay2❌ If you see “overlay” instead of “overlay2”? You’re on the old driver—less efficient.
❌ If no overlay? Your kernel is too old. Update it!
Chapter 2: Inodes & Hard Links (Why overlay2 Saves Your Disk)
What Are Inodes?
Every file on Linux has an inode—like a file’s ID card:
📋 Inode contains:
- Permissions (rwxr-xr-x)
- Owner (root:root)
- Size (42 bytes)
- Timestamps
- Where data lives on diskProblem: Filesystems have limited inodes. You can fill them up even with free space!
# Check inode usage
df -i
# 90% used? You're in trouble!🧪 Lab 2: Play with Inodes
# Create a file
echo "Hello inodes!" > testfile.txt
# See its inode number
ls -i testfile.txt
# Output: 1234567 testfile.txt
# More details
stat testfile.txtWhat Are Hard Links?
Multiple names → Same inode/data
# Lab 3: Create hard link
echo "Original content" > original.txt
ln original.txt hardlink.txt # Same inode!
ls -li original.txt hardlink.txt
# 1234567 -rw-r--r-- 2 root root 16 original.txt
# 1234567 -rw-r--r-- 2 root root 16 hardlink.txt
# ↑ Same inode #1234567, Link count=2
# Modify one → BOTH change!
echo " changed" >> original.txt
cat hardlink.txt # Also changed!
# Delete one → data survives!
rm original.txt
cat hardlink.txt # Still works!
rm hardlink.txt # NOW data deletedWhy This Matters for Docker
❌ OLD overlay driver:
- Layer 1: /app/config.txt (1 inode)
- Container modifies → copies entire file (NEW inode!)
✅ overlay2 driver:
- Layer 1: /app/config.txt (inode #1234)
- Container modifies → hard link! (still inode #1234)Result: overlay2 uses 90% less disk space + inodes
Chapter 3: Build a Real Multi-Layer Image
🧪 Lab 4: Create 5-Layer Image
mkdir ~/docker-lab && cd ~/docker-lab
cat > Dockerfile << 'EOF'
FROM ubuntu:22.04
# Layer 2: Install packages
RUN apt-get update && apt-get install -y \
curl vim tree \
&& rm -rf /var/lib/apt/lists/*
# Layer 3: Create app structure
RUN mkdir -p /app/data /app/logs
# Layer 4: Config files
RUN echo "app_name=docker-demo" > /app/config.txt
RUN echo "debug=true" > /app/debug.txt
# Layer 5: Sample data
RUN echo "Layer 5 data only!" > /app/data/layer5.txt
WORKDIR /app
CMD ["/bin/bash"]
EOF
# Build (creates 5 layers!)
docker build -t demo-app:v1 See the Layers!
demo-app:v1IMAGE CREATED CREATED BY SIZE
abc123def 10 seconds ago /bin/sh -c #(nop) WORKDIR /app 0B
def456ghi 15 seconds ago /bin/sh -c echo "Layer 5 data only!"... 45B
... 5 total layers!Each RUN= 1 layer = 1 overlay2 directory
Chapter 4: overlay2 Directory Structure Explained
Location: /var/lib/docker/overlay2/
overlay2/
├── abc123def/ ← Layer directories (random hash names)
├── def456ghi/
├── l/ ← Short symlinks (avoids command line limits)
├── metadata/ ← Docker metadata
└── repositories.json ← Layer relationships🧪 Lab 5: Explore Real Structure
# See all layers
sudo ls -la /var/lib/docker/overlay2/ | head -10
# Symlinks
sudo ls -la /var/lib/docker/overlay2/l/ | head -5
# Pick ANY layer directory and explore
LAYER=$(sudo ls /var/lib/docker/overlay2/ | head -1)
sudo ls -la /var/lib/docker/overlay2/$LAYER/Inside each layer:
abc123def/
├── diff/ ← Files ADDED by this layer
├── lower ← Parent layer chain (lowerdir)
├── upper/ ← Writable layer (for containers)
├── work/ ← OverlayFS temp work
├── link ← Short name symlink
└── committed ← "This layer is finalized"Chapter 5: Containers & The Famous 4 Directories
🧪 Lab 6: Run Container + Inspect
docker run -dit --name my-container demo-app:v1 sleep 3600
# See the magic 4 dirs!
docker inspect my-container | grep -A5 -B5 DirOutput:
"LowerDir": "/var/lib/docker/overlay2/...;...;...", ← Image layers (read-only)
"UpperDir": "/var/lib/docker/overlay2/xyz789uvw/", ← Container writes
"MergedDir": "/var/lib/docker/overlay2/merged/...", ← What container SEES
"WorkDir": "/var/lib/docker/overlay2/work/..." ← OverlayFS scratchVisual:
LowerDir (Image Layers) ──┐
├─→ MergedDir ← What apps see!
UpperDir (Container) ─────┘
Chapter 6: Copy-on-Write DEMO (Watch It Happen!)
🧪 Lab 7: See Copy-on-Write Live
# Note your container's UpperDir
UPPER_DIR=$(docker inspect my-container | jq -r '.[0].GraphDriver.Data.UpperDir')
echo "UpperDir: $UPPER_DIR"
# 1. CREATE NEW FILE
docker exec my-container echo "hello world" > /tmp/newfile.txt
sudo ls -la "$UPPER_DIR/tmp/" # newfile.txt appears!
# 2. MODIFY EXISTING FILE (Copy-on-Write!)
docker exec my-container echo "CUSTOM!" >> /app/config.txt
# What happened?
sudo ls -la "$UPPER_DIR/app/"
# config.txt now exists here (copied from image layer)
sudo cat "$UPPER_DIR/app/config.txt"
# Contains original + your addition!
# 3. DELETE FILE (Whiteout magic!)
docker exec my-container rm /app/data/layer5.txt
sudo ls -la "$UPPER_DIR/app/data/"
# .wh.layer5.txt...c ← WHITEOUT FILE!Whiteout Files Explained
.wh.filename...c ← Special character device (0,0)
Meaning: "HIDE whatever is below me"
Original file still exists in image layer!Chapter 7: See OverlayFS Mount Live
# While container runs:
mount | grep overlay
# Output shows ALL 4 dirs in the mount command!Chapter 8: Safe Cleanup (The Right Way)
❌ NEVER DO THIS:
sudo rm -rf /var/lib/docker/overlay2/* # Docker breaks!✅ DO THIS INSTEAD:
# Stop all containers first
docker stop $(docker ps -aq)
# Cleanup in order:
docker container prune -f # 10s
docker image prune -a -f # 30s
docker volume prune -f # 5s
docker system prune -a --volumes # NUKES everything (use carefully)
# Check disk usage before/after
du -sh /var/lib/docker/overlay2/
Chapter 9: Complete End-to-End Example
Stage 1: Clean Slate + Pull Image
docker system prune -a --volumes
sudo du -sh /var/lib/docker/overlay2/ # ~100MB
docker pull nginx:alpine
sudo du -sh /var/lib/docker/overlay2/ # ~20MB → 35MB (+15MB layers)
sudo ls /var/lib/docker/overlay2/ | wc -l # +6 directories
Stage 2: Start Container
docker run -d --name nginx-test nginx:alpine
sudo ls /var/lib/docker/overlay2/ | wc -l # +1 directory (UpperDir!)Stage 3: Make Changes → Watch overlay2 Grow
# Inside container
docker exec nginx-test sh -c 'echo hello > /tmp/test.txt'
docker exec nginx-test sh -c 'echo custom >> /etc/nginx/nginx.conf'
docker exec nginx-test sh -c 'rm /usr/share/nginx/html/index.html'
# Check growth
sudo du -sh /var/lib/docker/overlay2/$(docker inspect nginx-test | jq -r '.[0].GraphDriver.Data.UpperDir')/diff/Stage 4: Cleanup Test
docker stop nginx-test
sudo du -sh /var/lib/docker/overlay2/ # Same size!
docker rm nginx-test
sudo du -sh /var/lib/docker/overlay2/ # Smaller! (UpperDir gone)Troubleshooting
| Problem | Cause | Fix |
| “No space left” but df -h shows space | Out of inodes | df -i, then docker system prune -a |
| overlay2 uses 100GB! | Stopped containers | docker container prune |
| Can’t build images | /var/lib/docker/tmp full | Clean /tmp, increase disk |
| “invalid layer” errors | Manual deletion | docker system prune -a –volumes |
Quick Reference Cheat Sheet
# Disk usage
du -sh /var/lib/docker/overlay2/
df -h /var/lib/docker/
df -i /var/lib/docker/
# Inspect running container
docker inspect <container> | jq '.[0].GraphDriver.Data'
# Nuclear cleanup
docker system prune -a --volumes
# See ALL layers
sudo ls /var/lib/docker/overlay2/ | wc -l
