Building multi-arch docker images with nix inside docker
2024-03-18During the preparation of moving some servers to aarch64, Some some nix-build docker images had to be changed from being x86-64 only to a multi-arch image.
Flocken worked wonderful for this purpose at first, until running the build from inside a forgejo action runner. Since the build process runs inside a docker container the buildah
process runs into some issue caused by the constrained environment.
Enabling the unshare
syscall
By default, docker blocks access to the unshare
syscall, which buildah
uses to set up a new user namespace, producing the following error:
Error during unshare(CLONE_NEWUSER): Operation not permitted
To work around this you can disable the seccomp to stop docker from blocked the syscall.
On NixOs this can be configured with the following option:
{
virtualisation.docker.daemon.settings.seccomp-profile = "unconfined";
}
On other distros you can set the option by adding
in /etc/docker/daemon.json
Disabling the seccomp filter does weaken the security isolation of docker a bit so if you often run less-trusted docker images this might not be advisable.
Creating a /etc/containers/policy.json
inside the builder
When pushing the image, buildah
looks for the security policy in /etc/containers/policy.json
. Since the docker image used for building doesn't have this file it will fail with:
open /etc/containers/policy.json: no such file or directory
If you're using nix to build your runner image you can add the default policy by changing the copyToRoot
section to:
buildEnv {
name = "image-root";
paths = [
coreutils-full
(writeTextDir "/etc/containers/policy.json" (builtins.toJSON {
"default" = [
{
"type" = "insecureAcceptAnything";
}
];
"transports" = {
"docker-daemon" = {
"" = [{"type" = "insecureAcceptAnything";}];
};
};
}))
];
pathsToLink = ["/bin" "/etc/containers"];
}
If you're not using nix you can add the policy.json
from the Dockerfile with the following content:
With these modifications to the docker host and images flocken should work as expected from inside the forgejo/gitea action runner.