To efficiently ship custom container images to the ARM based Raspberry Pis in the Pilab Cluster it's necessary to build arm compatible images with the Gitlab CI. Docker Buildx is one tool that can compile container images for non-native build hosts.
With Docker Buildx it's possible to publish Docker Images that work on multiple CPU architectures. Buildx is still an experimental feature of the Docker CLI and requires some adjustments to the Host environment to work properly. This article focuses especially on how to use buildx with the Gitlab CI and a Docker-in-Docker setup.
Buildx can use native builder nodes running different architectures or make use of the QEMU processor emulator. QEMU is a pure software based solution and doesn't require us to install gitlab runners on different cpu architectures. This makes it possible to compile container images for the ARM architecture on a x86 host machine.
Install QEMU on the Host#
Even if we want to use buildx in a docker-in-docker setup we need to install QEMU on the host machine and register the supported binary flags in the kernel. The easiest way to accomplish that is by running a docker container that takes care of all the necessary steps and does not require to install additional packages on the host.
docker run --rm --privileged multiarch/qemu-user-static --reset -p yesEverytime the system reboots you need to run this command again. A simple solution to run this command on startup is to use a systemd service similar the following and place it in /etc/systemd/system/<name>.service
[Unit]Description=Run Command on BootAfter=docker.service
[Service]Type=oneshotExecStart=/usr/bin/docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
[Install]WantedBy=multi-user.targetDon't forget to enable the service:
systemctl enable <name>.serviceConfigure the Docker-in-Docker Runner#
Now we need to configure and register the actual runner with support for the docker-in-docker container. Informations on how to do this can be found in the Gitlab Documentation. --> here
Make sure that the buildx CI jobs always use the appropriate runner.
Configure the CI#
Since Buildx is still an experimental feature it is necessary to run some additional commands before buildx is actually working. The Docker in Docker Image does not include the experimental CLI Features. That means we need to explicit install buildx as an external plugin.
before_script: - mkdir -p ~/.docker/cli-plugins - wget -q -O ~/.docker/cli-plugins/docker-buildx https://github.com/docker/buildx/releases/download/v0.4.1/buildx-v0.4.1.linux-amd64 - chmod a+x ~/.docker/cli-plugins/docker-buildxNow it is possible to run the buildx command. The next step we need to do is creating a new builder.
before_script: - docker context create mybuilder - docker buildx create mybuilder --useAfter this it is possible to use all buildx functions on the CI pipeline. More Information about buildx can be found in the Docs. The following is an example for a working CI configuration:
.build_template: &build_template image: docker:latest variables: DOCKER_TLS_CERTDIR: "/certs" services: - name: docker:dind command: - "--experimental" before_script: - mkdir -p ~/.docker/cli-plugins - wget -q -O ~/.docker/cli-plugins/docker-buildx https://github.com/docker/buildx/releases/download/v0.4.1/buildx-v0.4.1.linux-amd64 - chmod a+x ~/.docker/cli-plugins/docker-buildx - docker context create mybuilder - docker buildx create mybuilder --use
build-hardware-test-arm: <<: *build_template stage: build variables: IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG script: - echo "$CI_REGISTRY_PASSWORD" | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY - docker buildx build --platform linux/arm64 -t $IMAGE_TAG --push .