Securing GitLab + Docker CI Pipelines

October 21, 2020

Intro

Continuous integration (CI) jobs often require interaction with Docker, either for building Docker images and/or deploying Docker containers.

TL;DR

The article is a bit long as the first half gives a detailed explanation of the security related problems for GitLab jobs that require interaction with Docker.

Contents

Security Problems with GitLab + Docker

It is common for CI jobs to require interaction with Docker, often to build container images and/or to deploy containers. These jobs are typically composed of steps executing Docker commands such as docker build, docker push, or docker run.

  • The Docker Executor

Security issues with the Shell Executor

When using the shell executor, the CI job is composed of shell commands executed in the same context as the GitLab runner.

build_image:
script:
- docker build -t my-docker-image .
- docker run my-docker-image /script/to/run/tests

Security issues with the Docker Executor

When using the Docker executor, the CI job runs within one or more Docker containers. This solves the functional problems of the shell executor described in the prior section because you get a clean environment prepackaged with your job’s dependencies.

Binding the host Docker Socket into the Job Container

This setup is shown below.

image: docker:19.03.12

build:
stage: build
script:
- docker build -t my-docker-image .
- docker run my-docker-image /script/to/run/tests
[[runners]]
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:19.03.12"
privileged = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]

Using a Docker-in-Docker Service Container

This setup is shown below.

image: docker:19.03.12

services:
- docker:19.03.12-dind

build:
stage: build
script:
- docker build -t my-docker-image .
- docker run my-docker-image /script/to/run/tests
[[runners]]
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = true
image = "docker:19.03.12"
privileged = true
disable_cache = false
volumes = ["/certs/client", "/cache"]

Solution: Using Docker + Sysbox

There is now a way to secure GitLab CI jobs that require interaction with Docker: using the new Sysbox container runtime.

A Secure DinD Service Container

The first approach is to use the Docker-in-Docker (DinD) service container described previously, but create the DinD container with Docker + Sysbox, as shown below.

[[runners]]
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = true
image = "docker:19.03.12"
privileged = false
disable_cache = false
volumes = ["/certs/client", "/cache"]
runtime = "sysbox-runc"
{
"default-runtime": "sysbox-runc",
"runtimes": {
"sysbox-runc": {
"path": "/usr/local/sbin/sysbox-runc"
}
}
}
image: docker:19.03.12

services:
- docker:19.03.12-dind

build:
stage: build
script:
- docker build -t my-docker-image .
- docker run my-docker-image /script/to/run/tests

GitLab Runner & Docker in a Container

The setup for this is shown below.

  • You can run many GitLab runners on the same host machine, in full isolation from one another. This way, you can easily deploy multiple customized GitLab runners on the same machine as you see fit, giving you more flexibility and improving machine utilization.
  • You can easily deploy this system container on bare-metal machines, VMs in the cloud, or any other machine where Linux, Docker, and Sysbox are running. It’s a self-contained and complete GitLab runner + Docker environment.
$ docker run --runtime=sysbox-runc -d --name gitlab-runner --restart always -v /srv/gitlab-runner/config:/etc/gitlab-runner nestybox/gitlab-runner-docker
$ docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register
[[runners]]
name = "syscont-runner-docker"
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = false
image = "docker:19.03.12"
privileged = false
disable_cache = false
volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
$ docker restart gitlab-runner

Inner Docker Image Caching

One of the drawbacks of placing the Docker daemon inside a container is that containers are non-persistent by default, so any images downloaded by the containerized Docker daemon will be lost when the container is destroyed. In other words, the containerized Docker daemon’s cache is ephemeral.

[[runners]]
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = true
image = "docker:19.03.12"
privileged = false
disable_cache = false
volumes = ["/certs/client", "/cache", "/var/lib/docker"]
runtime = "sysbox-runc"
$ docker run --runtime=sysbox-runc -d --name gitlab-runner --restart always -v /srv/gitlab-runner/config:/etc/gitlab-runner -v inner-docker-cache:/var/lib/docker nestybox/gitlab-runner-docker
  • A given host volume bind-mounted into a system container’s /var/lib/docker must only be mounted on a single system container at any given time. This is a restriction imposed by the inner Docker daemon, which does not allow its image cache to be shared concurrently among multiple daemon instances. Sysbox will check for violations of this rule and report an appropriate error during system container creation.

Conclusion

If you have GitLab jobs that require interaction with Docker, be aware that these jobs have root-level access to the host on which they run, thus resulting in weak isolation and compromising the stability of your CI infrastructure and possibly beyond.

Founder and CEO of Nestybox, Inc.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store