Building multi-arch docker images with nix inside docker

During the preparation of moving some servers to aarch64, Some some nix-build docker images had to be changed from being x86-64 only to a multi-arch image.

Flocken worked wonderful for this purpose at first, until running the build from inside a forgejo action runner. Since the build process runs inside a docker container the buildah process runs into some issue caused by the constrained environment.

Enabling the unshare syscall

By default, docker blocks access to the unshare syscall, which buildah uses to set up a new user namespace, producing the following error:

Error during unshare(CLONE_NEWUSER): Operation not permitted

To work around this you can disable the seccomp to stop docker from blocked the syscall.

On NixOs this can be configured with the following option:

{
  virtualisation.docker.daemon.settings.seccomp-profile = "unconfined";
}

On other distros you can set the option by adding

{
  "seccomp-profile": "unconfined"
}

in /etc/docker/daemon.json

Disabling the seccomp filter does weaken the security isolation of docker a bit so if you often run less-trusted docker images this might not be advisable.

Creating a /etc/containers/policy.json inside the builder

When pushing the image, buildah looks for the security policy in /etc/containers/policy.json. Since the docker image used for building doesn't have this file it will fail with:

open /etc/containers/policy.json: no such file or directory

If you're using nix to build your runner image you can add the default policy by changing the copyToRoot section to:

buildEnv {
  name = "image-root";
  paths = [
    coreutils-full
    (writeTextDir "/etc/containers/policy.json" (builtins.toJSON {
      "default" = [
        {
          "type" = "insecureAcceptAnything";
        }
      ];
      "transports" = {
        "docker-daemon" = {
          "" = [{"type" = "insecureAcceptAnything";}];
        };
      };
    }))
  ];
  pathsToLink = ["/bin" "/etc/containers"];
}

If you're not using nix you can add the policy.json from the Dockerfile with the following content:

{
  "default": [
    {
      "type": "insecureAcceptAnything"
    }
  ],
  "transports": {
    "docker-daemon":
      {
        "": [{"type":"insecureAcceptAnything"}]
      }
  }
}

With these modifications to the docker host and images flocken should work as expected from inside the forgejo/gitea action runner.