Apko: A Better Way To Build Containers?

Chainguard recently announced Wolfi, a set of container images that are built directly from Alpine packages and not Dockerfiles.

Ever since Docker released in 2013, we have been using Dockerfiles to build containers. They work well, but there are some problems:

• They make it hard to share components between images, a single base image is the only option. Code reuse is difficult.
• Some components like package managers are not required to run a container, but they are included for the convenience of the builder. There is no separation between build and runtime dependencies (although multi-stage builds can mostly achieve this).
• Docker does not support reproducible builds (but kaniko does).
• Layers serve no purpose for production builds, most images use one giant and messy RUN statement that does everything. That however slows down rebuilds during development.

Let's look at the solutions provided by Chainguard that try to address these (and more) problems.

melange#

Melange is a builder for Alpine packages. It uses pipelines similar to common CI/CD services, and it builds for multiple architectures by default. Here is a simplified example of a package build for the forum software NodeBB:

package:
  name: nodebb
  version: 2.5.3
  dependencies:
    runtime:
      - nodejs

environment:
  contents:
    packages:
      - alpine-baselayout
      - ca-certificates-bundle
      - nodejs
      - npm
      - git

pipeline:
  - uses: fetch
    with:
      uri: https://github.com/NodeBB/NodeBB/archive/refs/tags/v${{package.version}}.tar.gz
      expected-sha256: 92e390d7cda190e7f098833cbbbf03fbe1c50f25653656ad589ae97dc18a7684
      strip-components: 0
  - runs: |
      mkdir -p "${{targets.destdir}}/usr/share/nodebb"
      cd NodeBB-${{package.version}}
      cp install/package.json .
      npm install --omit=dev
      cp -a ./. "${{targets.destdir}}/usr/share/nodebb"

Under the environment section, we set up a build environment. Then we call the built-in fetch pipeline to fetch the source code bundle, verify the hash and extract it. There are a few more built-in pipelines, but we can also write our own.

Next, we run a short script to install dependencies, run a build and copy the resulting files into a destination directory. Everything under targets.destdir is copied into the package, which will only depend on nodejs, as we do not need npm or git at runtime.

Now, we can use melange to build an apk package. This isn't a container yet, but for that, we use the next tool.

apko#

apko takes apk packages and builds them into OCI images (aka Docker images). Sounds quite simple, because it is:

contents:
  repositories:
    - https://dl-cdn.alpinelinux.org/alpine/edge/main
  packages:
    - alpine-baselayout
    - nodebb

entrypoint:
  command: /usr/share/nodebb/nodebb start

Note that there is no base image being used, instead, we get exactly what's in the packages we list. alpine-baselayout adds busybox and the standard directory structure.

We can include other apko definitions as a base or create users and groups and set up file system permissions, but that's about it. apko lacks most capabilities of traditional OCI image builders as it's meant to be combined with melange, adding a files and folders directly is not supported. But what it does have is native support for s6 in case we want to run multiple services.

apko is quite fast, all it does is extract apk packages into OCI images and it doesn't run scripts. This is what enables fast rebuilds of images after an individual package is updated, and also makes them reproducible.

Using both tools, did we solve the problems we laid out before? Let's see.

• melange's pipelines enable modularity at the build level, and apko lets us combine multiple packages into one image. Code reuse is easy.
• Images built with apko contain exactly the packages we want, and nothing else. Images are smaller than comparable Docker images using the Alpine base image.
• Both melange and apko provide fully reproducible builds.
• Instead of layers, we use packages to keep build times fast and image sizes small.

Should I use this?#

We already use apko to create images for our own use (planned public release at a later stage). The technology works well, but we have questions. The state of glibc support isn't clear to us, and documentation is bare-bones. And while melange and apko are technically faster than Dockerfiles because they can be more selective about what needs to be rebuilt, there isn't tooling that takes advantage of that yet.

There are always the Wolfi images to use as base images in standard Dockerfiles. For the story behind the creation of Wolfi, this insightful Twitter thread by Ariadne Conill is worth a read.