Securing GitLab + Docker CI Pipelines

Intro

Continuous integration (CI) jobs often require interaction with Docker, either for building Docker images and/or deploying Docker containers.

TL;DR

The article is a bit long as the first half gives a detailed explanation of the security related problems for GitLab jobs that require interaction with Docker.

Contents

Security Problems with GitLab + Docker

It is common for CI jobs to require interaction with Docker, often to build container images and/or to deploy containers. These jobs are typically composed of steps executing Docker commands such as docker build, docker push, or docker run.

  • The Shell Executor
  • The Docker Executor

Security issues with the Shell Executor

When using the shell executor, the CI job is composed of shell commands executed in the same context as the GitLab runner.

build_image:
script:
- docker build -t my-docker-image .
- docker run my-docker-image /script/to/run/tests

Security issues with the Docker Executor

When using the Docker executor, the CI job runs within one or more Docker containers. This solves the functional problems of the shell executor described in the prior section because you get a clean environment prepackaged with your job’s dependencies.

Binding the host Docker Socket into the Job Container

This setup is shown below.

image: docker:19.03.12

build:
stage: build
script:
- docker build -t my-docker-image .
- docker run my-docker-image /script/to/run/tests
[[runners]]
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:19.03.12"
privileged = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]

Using a Docker-in-Docker Service Container

This setup is shown below.

image: docker:19.03.12

services:
- docker:19.03.12-dind

build:
stage: build
script:
- docker build -t my-docker-image .
- docker run my-docker-image /script/to/run/tests
[[runners]]
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = true
image = "docker:19.03.12"
privileged = true
disable_cache = false
volumes = ["/certs/client", "/cache"]

Solution: Using Docker + Sysbox

There is now a way to secure GitLab CI jobs that require interaction with Docker: using the new Sysbox container runtime.

A Secure DinD Service Container

The first approach is to use the Docker-in-Docker (DinD) service container described previously, but create the DinD container with Docker + Sysbox, as shown below.

[[runners]]
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = true
image = "docker:19.03.12"
privileged = false
disable_cache = false
volumes = ["/certs/client", "/cache"]
runtime = "sysbox-runc"
{
"default-runtime": "sysbox-runc",
"runtimes": {
"sysbox-runc": {
"path": "/usr/local/sbin/sysbox-runc"
}
}
}
image: docker:19.03.12

services:
- docker:19.03.12-dind

build:
stage: build
script:
- docker build -t my-docker-image .
- docker run my-docker-image /script/to/run/tests

GitLab Runner & Docker in a Container

The setup for this is shown below.

  • It allows the GitLab CI jobs to use the shell executor or Docker executor (either the DooD or DinD approaches) without compromising host security, because the system container provides a strong isolation boundary.
  • You can run many GitLab runners on the same host machine, in full isolation from one another. This way, you can easily deploy multiple customized GitLab runners on the same machine as you see fit, giving you more flexibility and improving machine utilization.
  • You can easily deploy this system container on bare-metal machines, VMs in the cloud, or any other machine where Linux, Docker, and Sysbox are running. It’s a self-contained and complete GitLab runner + Docker environment.
  • For CI jobs that interact with Docker, the isolation boundary is at the system container boundary rather than at the job level. That is, such a CI job could easily gain control of the system container and thus compromise the GitLab runner environment, but not the underlying host.
$ docker run --runtime=sysbox-runc -d --name gitlab-runner --restart always -v /srv/gitlab-runner/config:/etc/gitlab-runner nestybox/gitlab-runner-docker
$ docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register
[[runners]]
name = "syscont-runner-docker"
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:19.03.12"
privileged = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
$ docker restart gitlab-runner

Inner Docker Image Caching

One of the drawbacks of placing the Docker daemon inside a container is that containers are non-persistent by default, so any images downloaded by the containerized Docker daemon will be lost when the container is destroyed. In other words, the containerized Docker daemon’s cache is ephemeral.

[[runners]]
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = true
image = "docker:19.03.12"
privileged = false
disable_cache = false
volumes = ["/certs/client", "/cache", "/var/lib/docker"]
runtime = "sysbox-runc"
$ docker run --runtime=sysbox-runc -d --name gitlab-runner --restart always -v /srv/gitlab-runner/config:/etc/gitlab-runner -v inner-docker-cache:/var/lib/docker nestybox/gitlab-runner-docker
  • By making the containerized Docker’s image cache persistent, you are not just persisting images downloaded by the containerized Docker daemon; you are persisting the entire state of that Docker daemon (i.e., stopped containers, volumes, networks, etc.) Keep this in mind to make sure you CI jobs persist the state that you wish to persist, and explicitly cleanup any state you wish to not persist across CI jobs.
  • A given host volume bind-mounted into a system container’s /var/lib/docker must only be mounted on a single system container at any given time. This is a restriction imposed by the inner Docker daemon, which does not allow its image cache to be shared concurrently among multiple daemon instances. Sysbox will check for violations of this rule and report an appropriate error during system container creation.

Conclusion

If you have GitLab jobs that require interaction with Docker, be aware that these jobs have root-level access to the host on which they run, thus resulting in weak isolation and compromising the stability of your CI infrastructure and possibly beyond.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store