Issue
Found out i’m using a lot of space where i’m not expecting it:
# du -hs /home/*
...
1.5G /home/website
Where is it coming from?
$ cd
$ du -hs .local/share/containers/storage/*
4.0K .local/share/containers/storage/defaultNetworkBackend
80K .local/share/containers/storage/libpod
4.0K .local/share/containers/storage/mounts
4.0K .local/share/containers/storage/networks
4.0K .local/share/containers/storage/secrets
4.0K .local/share/containers/storage/storage.lock
4.0K .local/share/containers/storage/tmp
0 .local/share/containers/storage/userns.lock
1.5G .local/share/containers/storage/vfs
48K .local/share/containers/storage/vfs-containers
48K .local/share/containers/storage/vfs-images
408K .local/share/containers/storage/vfs-layers
Now this user is running a standard nginx container which is not that big:
$ podman ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5d22e1b58e9d docker.io/library/nginx:latest nginx -g daemon o... 2 days ago Up 2 days ago 127.0.0.1:8000->80/tcp, 127.0.0.1:8000->80/udp deploy_website-e259076d-f130-0240-52fc-e56c40e4c059
From a quick look on docker hub, it’s about 70MB big but we’re using 1.5GBs…
Let’s check the directories:
$ du -hs .local/share/containers/storage/vfs/dir/* 2>/dev/null
195M .local/share/containers/storage/vfs/dir/0027a3dcfbf9f22273fa3ecf107bc4c2eb88777871b06ff9f19ce48ccacc7e75
195M .local/share/containers/storage/vfs/dir/2f8a3d547995b3736c230e4e7c688fdb5068dff0fade91f705e7356bd7f0087b
195M .local/share/containers/storage/vfs/dir/343bf4bf5698d27f55fcc706d71cf688d1b633574a65528a6f742aedce899a5a
195M .local/share/containers/storage/vfs/dir/9370a574774e908e8e20db68f4d32817cdcc07d49cc52db4685c129636b20d2a
82M .local/share/containers/storage/vfs/dir/98b5f35ea9d3eca6ed1881b5fe5d1e02024e1450822879e4c13bb48c9386d0ad
195M .local/share/containers/storage/vfs/dir/c94c5e1c3380613df5fe9e7a14966b12f8b3f10cdc65470cca4cb8f2f1416b51
195M .local/share/containers/storage/vfs/dir/d039ba323e95c72fe29ecee4c8b9ef29ba6295c538089e9ec0cff43cfe39dbbe
195M .local/share/containers/storage/vfs/dir/dbb1425572343693abc2b24ee89f50e7a9ce249f52135add21d811ab310fa790
Now if you’re paying attention to the output, then you already know what the problem is here. It seems that Podman 4.3.1. (on Debian 12) uses the VFS storage driver by default (where I expected overlayfs to be used…). The VFS is a super simple driver that does the following (source):
Each layer is a directory on disk, and there is no copy-on-write support. To create a new layer, a “deep copy” is done of the previous layer. This leads to lower performance and more space used on disk than other storage drivers. However, it is robust, stable, and works in every environment.
The directory names do not correlate with the image layer IDs shown in the docker pull command.
Now the documentation also explicitely states that the folders do not match the layer IDs of a container image. This confused me at first because I misinterpreted this statement to mean that there’s no correlation between the layers in the image and the directories. It’s only the name the name that has a unique (storage) ID but they do relate exactly to a layer in the image. Do note that not all statements in a Docker file will create a new layer but merely update the metadata JSON file (like LABEL, CMD, …).
But what if I do want to be able to map a VFS storage ID to a layer ID of the image?
Mapping layers to VFS storage folder IDs
First, let’s get the layer SHA256 sums of our container:
$ podman inspect nginx:latest
...
"RootFS": {
"Type": "layers",
"Layers": [
"sha256:98b5f35ea9d3eca6ed1881b5fe5d1e02024e1450822879e4c13bb48c9386d0ad",
"sha256:b33db0c3c3a85f397c49b1bf862e0472bb39388bd7102c743660c9a22a124597",
"sha256:63d7ce983cd5c7593c2e2467d6d998bb78ddbc2f98ec5fed7f62d730a7b05a0c",
"sha256:296af1bd28443744e7092db4d896e9fda5fc63685ce2fcd4e9377d349dd99cc2",
"sha256:8ce189049cb55c3084f8d48f513a7a6879d668d9e5bd2d4446e3e7ef39ffee60",
"sha256:6ac729401225c94b52d2714419ebdb0b802d25a838d87498b47f6c5d1ce05963",
"sha256:e4e9e9ad93c28c67ad8b77938a1d7b49278edb000c5c26c87da1e8a4495862ad"
]
},
...
We already note that one of the storage IDs matches one of the layers! The one starting with 98b5f35... is the same. This makes sense because it’s the initial layer, so the storage sha256 matches this layer. But for the second layer upwards it’s no longer matching because in the container file it’s a diff of changes, and the VFS does a deep copy and unions the diff with the lower layers.
lazy way
Presume there’s a file on all layers (if it’s in the first layer, it’s probably on all layers), we can stat the file and see which one is written first to get the order of layers:
$ stat {0027a3dcfbf9f22273fa3ecf107bc4c2eb88777871b06ff9f19ce48ccacc7e75,2f8a3d547995b3736c230e4e7c688fdb5068dff0fade91f705e7356bd7f0087b,343bf4bf5698d27f55fcc706d71cf688d1b633574a65528a6f742aedce899a5a,9370a574774e908e8e20db68f4d32817cdcc07d49cc52db4685c129636b20d2a,98b5f35ea9d3eca6ed1881b5fe5d1e02024e1450822879e4c13bb48c9386d0ad,c94c5e1c3380613df5fe9e7a14966b12f8b3f10cdc65470cca4cb8f2f1416b51,d039ba323e95c72fe29ecee4c8b9ef29ba6295c538089e9ec0cff43cfe39dbbe,dbb1425572343693abc2b24ee89f50e7a9ce249f52135add21d811ab310fa790}/bin/ls | grep -E '(File|Birth)'
File: 0027a3dcfbf9f22273fa3ecf107bc4c2eb88777871b06ff9f19ce48ccacc7e75/bin/ls
Birth: 2024-10-18 13:54:44.888388867 +0000
File: 2f8a3d547995b3736c230e4e7c688fdb5068dff0fade91f705e7356bd7f0087b/bin/ls
Birth: 2024-10-18 13:54:31.208524947 +0000
File: 343bf4bf5698d27f55fcc706d71cf688d1b633574a65528a6f742aedce899a5a/bin/ls
Birth: 2024-10-18 13:54:45.960378252 +0000
File: 9370a574774e908e8e20db68f4d32817cdcc07d49cc52db4685c129636b20d2a/bin/ls
Birth: 2024-10-18 13:54:37.648460737 +0000
File: 98b5f35ea9d3eca6ed1881b5fe5d1e02024e1450822879e4c13bb48c9386d0ad/bin/ls
Birth: 2024-10-18 13:54:27.316563880 +0000
File: c94c5e1c3380613df5fe9e7a14966b12f8b3f10cdc65470cca4cb8f2f1416b51/bin/ls
Birth: 2024-10-18 13:54:43.864399014 +0000
File: d039ba323e95c72fe29ecee4c8b9ef29ba6295c538089e9ec0cff43cfe39dbbe/bin/ls
Birth: 2024-10-18 21:38:22.481306713 +0000
File: dbb1425572343693abc2b24ee89f50e7a9ce249f52135add21d811ab310fa790/bin/ls
Birth: 2024-10-18 13:54:40.568431715 +0000
Or ordering them:
- 98b5f35ea9d3eca6ed1881b5fe5d1e02024e1450822879e4c13bb48c9386d0ad/bin/ls: 2024-10-18 13:54:27.316563880 +0000
- 2f8a3d547995b3736c230e4e7c688fdb5068dff0fade91f705e7356bd7f0087b/bin/ls: 2024-10-18 13:54:31.208524947 +0000
- 9370a574774e908e8e20db68f4d32817cdcc07d49cc52db4685c129636b20d2a/bin/ls: 2024-10-18 13:54:37.648460737 +0000
- dbb1425572343693abc2b24ee89f50e7a9ce249f52135add21d811ab310fa790/bin/ls: 2024-10-18 13:54:40.568431715 +0000
- c94c5e1c3380613df5fe9e7a14966b12f8b3f10cdc65470cca4cb8f2f1416b51/bin/ls: 2024-10-18 13:54:43.864399014 +0000
- 0027a3dcfbf9f22273fa3ecf107bc4c2eb88777871b06ff9f19ce48ccacc7e75/bin/ls: 2024-10-18 13:54:44.888388867 +0000
- 343bf4bf5698d27f55fcc706d71cf688d1b633574a65528a6f742aedce899a5a/bin/ls: 2024-10-18 13:54:45.960378252 +0000
- d039ba323e95c72fe29ecee4c8b9ef29ba6295c538089e9ec0cff43cfe39dbbe/bin/ls: 2024-10-18 21:38:22.481306713 +0000
Correct way
Sorry, i’m too lazy.
Moving to OverlayFS
Since moving to overlayfs is supported in kernel from 5.13 (before that fuse-overlayfs was needed) and we’re running a 6.1 kernel with Debian, it’s safe to move to overlayfs for all containers and configure a global setting that supports this.
I wrote the following to /usr/share/containers/storage.conf:
[storage]
driver = "overlay"
Then I killed all my podman containers (R.I.P.) and executed as the website user:
$ podman system reset
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve. May prevent use of images created by other tools
ERRO[0000] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve. May prevent use of images created by other tools
WARNING! This will remove:
- all containers
- all pods
- all images
- all networks
- all build cache
- all machines
- all volumes
- the graphRoot directory: "/home/website/.local/share/containers/storage"
- the runRoot directory: "/run/user/8000/containers"
Are you sure you want to continue? [y/N] y
ERRO[0002] User-selected graph driver "overlay" overwritten by graph driver "vfs" from database - delete libpod local files to resolve. May prevent use of images created by other tools
And then started the containers again (via Nomad).
If we now look at the size being used:
$ du -hs .local/*
198M .local/share
Much better!
Before
$ podman info
store:
configFile: /home/website/.config/containers/storage.conf
containerStore:
number: 1
paused: 0
running: 1
stopped: 0
graphDriverName: vfs
graphOptions: {}
graphRoot: /home/website/.local/share/containers/storage
graphRootAllocated: 20961435648
graphRootUsed: 3628384256
graphStatus: {}
imageCopyTmpDir: /var/tmp
imageStore:
number: 1
runRoot: /run/user/8000/containers
volumePath: /home/website/.local/share/containers/storage/volumes
Note that /home/website/.config/containers/storage.conf never has existed, it’s just the rootless configuration file that it would like to read. If it doesn’t exist, it looks at /etc/containers/storage.conf (also does not exist) and /usr/share/containers/storage.conf.
After
$ podman info
store:
configFile: /home/website/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/website/.local/share/containers/storage
graphRootAllocated: 20961435648
graphRootUsed: 2113007616
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "true"
Supports d_type: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 0
runRoot: /run/user/8000/containers
volumePath: /home/website/.local/share/containers/storage/volumes
Sources
Articles found: