While building my internal wiki called ‘Kribbel’, I was again confronted with the big sizes (>500MB) of docker containers, this time a result of my own doing. And I asked myself why a very simple Dancer2 app would need 500MB of “stuff” to run?
Around the same time (somewhere in 2024), I ran into Google’s distroless containers, and I put one and one together and asked myself if I could not make a distroless Perl container? I mean, how much work can it be? (this is foreshadowing)
But why?
- Only shipping what’s needed: I don’t want anything besides the components needed to run my web application.
- Less attack surface: having the bare minimum means less attack surface, less things that potentially have vulnerabilities (which hopefully translates in less emergency deploys)
- Faster pull from the container registry: Less data to transfer over the network, less data to read from disk when starting the container, …
And since I try to run all my container workloads on the smallest VPS that I can get away with, every byte counts (I have about 20 billion bytes on each VPS).
How do Google distroless containers work?
Despite being called “distroless”, it depends heavily on Debian packages to set up a minimal working environment.
That’s why I prefer to call it minimal or small containers instead of distroless.
A container image is itself a collection of layers that are TAR archives containing a filesystem structure. Debian packages contain a data.tar.xz file containing a filesystem structure of all the files that need to be installed on the OS. Since those two overlap, you can re-use Debian packages as a container layer for your container instead.
Not everything is created from a debian archive though. Certain files such as /etc/group and /etc/passwd are created manually.
You can find the parts that make up a distroless image back in the Google distroless repo.
But for our case, making Perl container images, they’re not provided by Google. The easy way is to go about this, is to use Bazel and make some minor tweaks to implement distroless containers that contain perl. But that would be too easy (heavy NIH syndrome from author).
Container::Builder
I recently released Container::Builder on CPAN which mimicks the Google Distroless approach of using Debian packages to do the hard work, and then stitch everything together to make a container image.
In this module, I’ve put an examples/ folder to show how it can be used to make container images.
And this brings us back to Kribbel. I added an example in the repository to make a Dancer2 but since that’s quite specific I couldn’t fully describe how to do the fatpacking for the Kribbel Dancer2 webapp in the module itself (it’s not the right place).
Speed comparisons
My incredible non-scientific measurements on my own desktop, using time in front of the commands. You can find the Dockerfiles and Container::Builder scripts at the bottom of this page.
| Steps | Time taken | Size |
|---|---|---|
| Pure Dockerfile | 43-50s | 532MB |
| Pure Dockerfile (cached from above) + one code change | 3.2s | 532MB |
| Fatpacking plackup+Kribbel | 1.6s | 4.78MB |
| Dockerfile to embed fatpacked plackup+Kribbel | 25.3s | 296MB |
| Container::Builder to make plackup base image + load into podman | 21.7s + 2.5s | 107MB (33MB compressed) |
| Container::Builder to make plackup base image with cache + load into podman | 6.1s + 0.2s | 107MB |
| Dockerfile pointing at plackup base Image + COPY fatpacked files | 4.2s | 113MB |
| Dockerfile pointing at plackup base Image (cached) + COPY fatpacked files | 4.2s | 113MB |
| One Container::Builder script to do everything (skip plackup container step) | 25.1s + 2.4s | 113MB (37MB compressed) |
| One Container::Builder script to do everything (skip plackup container step) with cache | 6.12s + 0.3s | 113MB |
From this data it shows that a well created cache and Dockerfile gives the fastest run but it also creates the biggest container image. Furthermore, if your cache is invalid you’ll wait about 45s for a build to happen. If you’re not sure the cache will still be there (in your CI/CD runners for example), might be better to use other options.
An intermediate solution would be to use a Dockerfile built from a minimal plackup base image and put the fatpacked app on top. This makes the runtime quite stable at about 5-6s (includes the fatpacking).
Creating the entire image with Container::Builder does not generate smaller images compared to the intermediate solution and also takes about 7-8s to run (incl. the fatpacking) but it allows the most control.
Other alternatives
In the links you can find a video from the German Perl Workshop where a small image was built upon Alpine using normal Dockerfiles.
Future goals for Container::Builder
Whilst building these containers, I’ve realized several issues with Container::Builder. One of the things that concern me the most is the hacky way in which I get XS dependencies in my container image. Since these are outside of fatpacking process, it can be that there is no Debian package for it (in which case I need to copy these files manually) or that the Debian package has a version that’s not compatible with my application dependency restrictions (too old, conflicting with related fatpacked modules, …). Since these issues are quite specifically related to fatpacking in Perl, I’m thinking to make a specific Container::Builder::Perl::Fatpacker (or whatever name I give it) so that files that are excluded from the fatpacking process automatically are picked up by the module and added as a layer in the Container.
I also want to investigate pp, the PAR packager, as it allows to ship XS modules, core modules and even the perl interpreter. It might be the way to go instead of dealing with the issues of App::Fatpacker as described above. But while I have experience with App::Fatpacker, I don’t have any with PAR so it’ll be something to explore.
Another goal is to provide more capabilities to filter out files and directories that would end up in the container as a result of using Debian packages underneath. We don’t really need those Pod files when the container is only meant for executing a workload. Similarly it would be super nice if I could determine which packages from libperl and perl-modules are being used by the application and trim accordingly. It might require some sort of dynamic analysis since (I think) App::Fatpacker doesn’t pack CORE modules but they might still be expected.
The last thing I want to support is to support basing of an existing container image, and only manipulating certain layers or adding a new layer on top. Much like FROM in a Dockerfile but more powerful (in that you can target specific layers to update, insert the file(s) into). This would speed up the normal development workflow (I want to get it down to less than 1 second) when you use Container::Builder.
Links
- Google distroless repo
- App::Fatpacker
- PAR Packager
- German Perl Workshop: perl containers using alpine
Resources
Pure Dockerfile
FROM docker.io/perl:5.38.2-slim-bookworm
WORKDIR /app
COPY cpanfile /app/cpanfile
RUN apt-get update && apt-get install -y curl gcc && curl -fsSL https://raw.githubusercontent.com/skaji/cpm/main/cpm > /bin/cpm && chmod +x /bin/cpm && apt-get autoremove -y && groupadd -g 1337 appie && useradd -m -g appie -u 1337 appie
RUN cpm install -g --show-build-log-on-failure && rm -rf /root/.perl-cpm
RUN chown appie:appie /app
USER appie
COPY --chown=appie:appie . /app/
CMD plackup bin/app.psgi
Fatpacker script
This is highly specific to my dancer2 application but it might give you an idea of the trouble one needs to go through to fatpack something with XS deps…
# CLEAN
rm -rf fatlib
# Get perl version
PERL_VERSION=`plenv version | cut -d ' ' -f1`
# FATPACK PLACKUP
fatpack trace /home/adri/.plenv/versions/${PERL_VERSION}/bin/plackup -E development bin/app.psgi
echo "Apache/LogFormat/Compiler.pm" >> fatpacker.trace
echo "POSIX/strftime/Compiler.pm" >> fatpacker.trace
fatpack packlists-for `cat fatpacker.trace` > packlists
fatpack tree `cat packlists`
# Avoid Kribbel being in dependencies
mv lib libtmp
mkdir lib
fatpack file /home/adri/.plenv/versions/${PERL_VERSION}/bin/plackup > fatpacked.plackup
# Replacing my homefolder in plackup -> my plackup is managed by plenv and my local directory is the first line of that file...
sed -i 's/home\/adri\/.*$/usr\/bin\/perl/' fatpacked.plackup
# Restore our Kribbel lib/
rmdir lib
mv libtmp lib
# CLEAN
rm -rf fatlib
# FATPACK OUR APP
fatpack trace --use=Template::Tiny bin/app.psgi
echo "Ref/Util/PP.pm" >> fatpacker.trace
echo "JSON/PP.pm" >> fatpacker.trace
echo "HTTP/Status.pm" >> fatpacker.trace
fatpack packlists-for `cat fatpacker.trace` > packlists
fatpack tree `cat packlists`
# Remove XS dependencies
rm fatlib/x86_64-linux/HTML/Parser.pm
rm fatlib/x86_64-linux/Clone.pm
rm fatlib/x86_64-linux/Ref/Util/XS.pm
rm -rf fatlib/x86_64-linux/Cpanel/JSON/
rm -rf fatlib/x86_64-linux/List/MoreUtils/
rm -rf fatlib/x86_64-linux/Template/Stash/XS.pm
fatpack file bin/app.psgi > bin/fatpacked.app.psgi
sed -i 's/Template::Stash::XS/Template::Stash/' bin/fatpacked.app.psgi
sed -i 's/fatpacked{"x86_64-linux\//fatpacked{"/g' bin/fatpacked.app.psgi
Dockerfile with embedded fatpacker
FROM docker.io/perl:5.38.2-slim-bookworm
WORKDIR /app
RUN apt-get update && apt-get install -y curl libperlio-utf8-strict-perl libdevel-stacktrace-perl libdevel-stacktrace-ashtml-perl libhtml-parser-perl libcrypt-bcrypt-perl && groupadd -g 1337 appie && useradd -m -g appie -u 1337 appie
RUN chown appie:appie /app
USER appie
COPY --chown=appie:appie . /app/
RUN chmod +x /app/fatpacked.plackup
CMD /app/fatpacked.plackup /app/bin/fatpacked.app.psgi
Container::Builder for plackup-trixie image
use v5.40;
use Container::Builder;
my $builder = Container::Builder->new(debian_pkg_hostname => 'debian.inf.tu-dresden.de', os_version => 'trixie', enable_packages_cache => 1, packages_file => 'Packages', cache_folder => 'artifacts');
$builder->create_directory('/', 0755, 0, 0);
$builder->create_directory('bin/', 0755, 0, 0);
$builder->create_directory('tmp/', 01777, 0, 0);
$builder->create_directory('root/', 0700, 0, 0);
$builder->create_directory('home/', 0755, 0, 0);
$builder->create_directory('home/larry/', 0700, 1337, 1337);
$builder->create_directory('etc/', 0755, 0, 0);
$builder->create_directory('app/', 0755, 1337, 1337);
# Base
$builder->add_deb_package('base-files');
$builder->add_deb_package('netbase');
$builder->add_deb_package('tzdata');
$builder->add_deb_package('media-types');
my $nsswitch = <<'NSS';
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.
passwd: compat
group: compat
shadow: compat
gshadow: files
hosts: files dns
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
netgroup: nis
NSS
$builder->add_file_from_string($nsswitch, '/etc/nsswitch.conf', 0644, 0, 0);
# C dependencies (to run a compiled executable)
$builder->add_deb_package('libc-bin');
$builder->add_deb_package('libc6');
$builder->add_deb_package('gcc-14-base');
$builder->add_deb_package('libgcc-s1');
$builder->add_deb_package('libgomp1');
$builder->add_deb_package('libstdc++6');
$builder->add_deb_package('ca-certificates');
# SSL support
$builder->add_deb_package('libssl3');
# Perl dependencies (to run a basic Perl program)
$builder->add_deb_package('libcrypt1');
$builder->add_deb_package('perl');
# My fatpack expects these to be already installed somehow
$builder->add_deb_package('libtry-tiny-perl');
$builder->add_deb_package('libdevel-stacktrace-perl');
$builder->add_deb_package('libdevel-stacktrace-ashtml-perl');
$builder->add_deb_package('libcrypt-bcrypt-perl');
# html::parser contains xs code so no can do with fatpack
$builder->add_deb_package('libhtml-parser-perl');
# same for Clone
$builder->add_deb_package('libclone-perl');
$builder->add_group('root', 0);
$builder->add_group('tty', 5);
$builder->add_group('staff', 50);
$builder->add_group('larry', 1337);
$builder->add_group('nobody', 65000);
$builder->add_user('root', 0, 0, '/sbin/nologin', '/root');
$builder->add_user('nobody', 65000, 65000, '/sbin/nologin', '/nohome');
$builder->add_user('larry', 1337, 1337, '/sbin/nologin', '/home/larry');
$builder->runas_user('larry');
$builder->set_env('PATH', '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin');
$builder->set_work_dir('/app');
$builder->add_file('fatpacked.plackup', '/app/plackup', 0755, 1337, 1337);
$builder->set_entry('/app/plackup');
$builder->build('05-plackup-trixie.tar');
say "Now run: podman load -i 05-plackup-trixie.tar";
say "Then run: podman tag " . substr($builder->get_digest(), 0, 12) . " localhost/plackup-trixie:latest";
Dockerfile based on Container::Builder plackup-base
FROM localhost/plackup-trixie:latest
WORKDIR /app
COPY --chown=1337:1337 bin/fatpacked.app.psgi /app/bin/app.psgi
COPY --chown=1337:1337 environments/ /app/environments/
COPY --chown=1337:1337 public/ /app/public/
COPY --chown=1337:1337 uploads/ /app/uploads/
COPY --chown=1337:1337 views/ /app/views/
COPY --chown=1337:1337 doc/ /app/doc/
COPY --chown=1337:1337 config.yml /app/config.yml
COPY --chown=1337:1337 log4perl.conf /app/log4perl.conf
USER larry
# Somehow our working directory is /app/bin/ and it looks for everything in there...
# So we have to hardcode where everything is...
COPY --chown=1337:1337 config.yml /app/bin/config.yml
ENV DANCER_ENVDIR=/app/environments/
ENV DANCER_VIEWS=/app/views/
ENV DANCER_PUBLIC=/app/public/
ENV DANCER_CONFIG_VERBOSE=1
ENV PWD=/app
EXPOSE 5000/tcp
# We take the entrypoint from plackup-trixie
Pure Container::Builder build
use v5.40;
use Container::Builder;
my $builder = Container::Builder->new(debian_pkg_hostname => 'debian.inf.tu-dresden.de', os_version => 'trixie', enable_packages_cache => 1, packages_file => 'Packages', cache_folder => 'artifacts');
$builder->create_directory('/', 0755, 0, 0);
$builder->create_directory('bin/', 0755, 0, 0);
$builder->create_directory('tmp/', 01777, 0, 0);
$builder->create_directory('root/', 0700, 0, 0);
$builder->create_directory('home/', 0755, 0, 0);
$builder->create_directory('home/larry/', 0700, 1337, 1337);
$builder->create_directory('etc/', 0755, 0, 0);
$builder->create_directory('app/', 0755, 1337, 1337);
# Base
$builder->add_deb_package('base-files');
$builder->add_deb_package('netbase');
$builder->add_deb_package('tzdata');
$builder->add_deb_package('media-types');
my $nsswitch = <<'NSS';
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.
passwd: compat
group: compat
shadow: compat
gshadow: files
hosts: files dns
networks: files
protocols: db files
services: db files
ethers: db files
rpc: db files
netgroup: nis
NSS
$builder->add_file_from_string($nsswitch, '/etc/nsswitch.conf', 0644, 0, 0);
# C dependencies (to run a compiled executable)
$builder->add_deb_package('libc-bin');
$builder->add_deb_package('libc6');
$builder->add_deb_package('gcc-14-base');
$builder->add_deb_package('libgcc-s1');
$builder->add_deb_package('libgomp1');
$builder->add_deb_package('libstdc++6');
$builder->add_deb_package('ca-certificates');
# SSL support
$builder->add_deb_package('libssl3');
# Perl dependencies (to run a basic Perl program)
$builder->add_deb_package('libcrypt1');
$builder->add_deb_package('perl');
# My fatpack expects these to be already installed somehow
$builder->add_deb_package('libtry-tiny-perl');
$builder->add_deb_package('libdevel-stacktrace-perl');
$builder->add_deb_package('libdevel-stacktrace-ashtml-perl');
$builder->add_deb_package('libcrypt-bcrypt-perl');
# html::parser contains xs code so no can do with fatpack
$builder->add_deb_package('libhtml-parser-perl');
# same for Clone
$builder->add_deb_package('libclone-perl');
$builder->add_group('root', 0);
$builder->add_group('tty', 5);
$builder->add_group('staff', 50);
$builder->add_group('larry', 1337);
$builder->add_group('nobody', 65000);
$builder->add_user('root', 0, 0, '/sbin/nologin', '/root');
$builder->add_user('nobody', 65000, 65000, '/sbin/nologin', '/nohome');
$builder->add_user('larry', 1337, 1337, '/sbin/nologin', '/home/larry');
$builder->runas_user('larry');
$builder->set_env('PATH', '/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin');
$builder->set_work_dir('/app');
# Adding fatpacked plackup
$builder->add_file('fatpacked.plackup', '/app/plackup', 0755, 1337, 1337);
# Our Kribbel files
$builder->create_directory('/app/bin', 0755, 1337, 1337);
$builder->add_file('bin/fatpacked.app.psgi', '/app/bin/app.psgi', 0644, 1337, 1337);
$builder->copy('environments', '/app/environments', 0755, 1337, 1337);
$builder->copy('public', '/app/public', 0755, 1337, 1337);
$builder->copy('uploads', '/app/uploads', 0755, 1337, 1337);
$builder->copy('views', '/app/views', 0755, 1337, 1337);
$builder->copy('doc', '/app/doc', 0755, 1337, 1337);
$builder->add_file('config.yml', '/app/config.yml', 0644, 1337, 1337);
$builder->add_file('config.yml', '/app/bin/config.yml', 0644, 1337, 1337);
$builder->add_file('log4perl.conf', '/app/log4perl.conf', 0644, 1337, 1337);
$builder->set_env('DANCER_ENVDIR', '/app/environments/');
$builder->set_env('DANCER_VIEWS', '/app/views/');
$builder->set_env('DANCER_PUBLIC', '/app/public/');
$builder->set_env('DANCER_CONFIG_VERBOSE', '1');
$builder->set_env('PWD', '/app');
# Set our entrypoint
$builder->set_entry('/app/plackup', '/app/bin/app.psgi');
$builder->build('kribbel.tar');
say "Now run: podman load -i kribbel.tar";
say "Then run: podman tag " . substr($builder->get_digest(), 0, 12) . " localhost/kribbel:ctrbuilder";