The blog post has no new things, it's my personal note. If you have any suggestion for me or the post. Please feel free to tell me. I am glad to know.

I have to build a non-x86 docker image in my work. The problem lets me to study why the official docker image could support multi-arch pull? When you send a pull request to docker registry, it checks the arch of your machine and responses the corresponding images to you.

How to build a multi-arch docker image like this? If you build the image in mac OS, it's quite simple. Just build it since docker in macOS has hypervisor inside.

If in Linux? You have to provide a qemu binary to run non-x86 binary. The overview steps are

  1. Prepare static qemu binary
  2. Register binfmt_misc setting with static qemu binary
  3. build docker image with static qemu binary
  4. Pull image setting with manifest-tool

First, we have to know how to run non-x86 docker image in x86-64 linux platform. I would take aarch64 arch as the example in the blog post. You can replace aarch64 arch with any non-x86 arch.

Run aarch64 docker image in x86-64 linux

Run aarch64 binary in x86-64 linux

How to ruun aarch64 binary in x86 platform? Use qemu to simulate it. For example, there is a aarch64 binary named a.out and we want to run it with qemu. The command is qemu-aarch64 ./a.out.

In advanced, register binfmt with the aarch64 format to avoid to call qemu-aarch64 in every time. After register, you can run the ./a.out directly. When ./a.out is executed, the kernel checks the binary and knows the binary is for aarch64 then call qemu-aarch64 with it.

For example:

export QEMU_BIN_PATH=/usr/local/bin/qemu-aarch64

# Register binfmt for aarch64
echo ":qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:$QEMU_BIN_PATH:" > /proc/sys/fs/binfmt_misc/register

# Run the real command
./a.out

# (Optional) Unreigster binfmt for aarch64
echo -1 > /proc/sys/fs/binfmt_mist/qemu-aarch64

Run aarch64 docker image

The main concept is like the previous section but more complicated. The major idea is to mount qemu binary to the image. When a binary in container is executed, the kernel (the container uses the host kernel) checks the binary format and runs the binary with qemu binary. In container, the rootfs is container's rootfs so qemu has to be mounted in container. The solution sounds good but still has some problem. If the qemu is builded with dynamic linking, it would look for share libraries in the run time. If the qemu binary runs in the container, it could not find them. The simple solution is to build a static-linked qemu binary to avoid the behaviour.

To simple the example, I make a helper docker image (yen3/binfmt-register) to run.

export cpu=aarch64
export RUN_IMAGE=arm64v8/alpine

# Register binfmt
docker run --rm --privileged yen3/binfmt-register set ${cpu}

# Get qemu static binary
docker run --rm yen3/binfmt-register get ${cpu} > qemu-${cpu}-static
chmod +x qemu-${cpu}-static

# Run the image with the qemu static binary
export MOUNT_QEMU="-v $(pwd)/qemu-${cpu}-static:/usr/local/bin/qemu-${cpu}-static"
docker run -it --rm ${MOUNT_QEMU} ${RUN_IMAGE} uname -a

# Unregister binfmt
docker run --rm --privileged yen3/binfmt-register clear ${cpu}

# Remove the qemu static binary
rm -f qemu-${cpu}-static

The complete example is in here.

How to get static-linked qemu binary ?

  • In ubuntu, install qemu-user-static package (apt-get install -y qemu-user-static) to get. The static binary could be found in /usr/qemu-*-static
  • In alpine, install qemu-${cpu} package toget. The static binary could be found in /usr/qemu-${cpu}
  • Build qemu from source code. The Dockerfile provides a simple example to build qemu.

Build aarch64 docker image in x86-64 platform

In the previous section, we know how to run the aarch64 bianry and docker image in x86-64 platform. How to apply the concept to build aarch64 image?

The native thinking is to mount the binary when building docker image. Unfortunelly, docker could not mount in build. To resolve the problem, use multi-stage build to achieve the same effect.

  1. Prepared the dockerfile and copy qemu-static-binary to the docker image. After the build steps are finished, remove the binary.

    FROM yen3/binfmt-register:0.1 as builder
    
    FROM arm64v8/alpine
    
    # Add qemu binary to run the inmage in x86-64 platform
    # The binary is unused in macOS docker
    COPY --from=builder /qemu/qemu-aarch64-static /usr/local/bin/qemu-aarch64-static
    
    # Do something what you want to do
    # ...
    
    # Remove the binary. It's unused in the final result
    RUN rm -f /usr/local/bin/qemu-aarch64-static
    
  2. Run the docker file with the helper docker image (with privileged access to set binfmt)

    # Register binfmt for aarch64
    docker run --privileged yen3/binfmt-register set aarch64
    
    # Build the docker image
    docker build -t yen3/test:arm64 .
    
    # Unregisrer binfmt for aarch64
    docker run --privileged yen3/binfmt-register clear aarch64
    

Pull a multi-arch docker image

So far there are several docker images for different cpu archs. It's unnecessary to combine these images into one image. The manifest-tool could help us to create a image to reference these images.

Assume there are three images are yen3/test:amd64 for x86-64, yen3/test:arm32v7 for arm and yen3/test:arm64 for aarch64 (remember to push them to dockerhub first), the manifest-tool can create a image to reference these images.

  1. Write a config file named manifest.yml for manifest-tool

    image: yen3/test:latest
    manifests:
      - image: yen3/test:amd64
        platform:
          architecture: amd64
          os: linux
      - image: yen3/test:arm64
        platform:
          architecture: arm64
          os: linux
      - image: yen3/test:arm32v7
        platform:
          architecture: arm
          os: linux
    
  2. Run manifest-tool with the config file to create a image.

    ./manifest-tool push from-spec ./manifest.yml
    

That's all. Now we create a multi-arch docker image successfully.

In the future

docker manifeset command is still in discuss. The step may be changed in the near future.

Example

I build a yen3/docker-ubuntu-novnc docker image (original from fcwu/docker-ubuntu-vnc-desktop) with these concepts. It supports x86-64, arm and aarch64. You can see the Makefile, dockerfiles, manifest.yml to get more details.

Reference

  • walkerlee's help
  • https://en.wikipedia.org/wiki/Binfmt_misc
  • https://github.com/estesp/manifest-tool
  • https://container-solutions.com/multi-arch-docker-images/
  • https://github.com/moul/docker-binfmt-register
  • https://github.com/mikkeloscar/binfmt-manager
  • https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh
  • https://coldnew.github.io/5cecf128/
  • https://github.com/ColinHuang/docker-ubuntu-novnc-armhf
  • https://github.com/fcwu/docker-ubuntu-vnc-desktop