Jan 10
Securing Kubernetes-in-Docker
January 10, 2022
Intro
This article will show how to run the K8s.io KinD tool inside an unprivileged container or pod launched with the Sysbox container runtime, as shown in the figure below.
The benefits of doing this are the following:
- Containerizing KinD itself (i.e., the KinD tool and all associated config is packaged in a container).
- Improving the isolation between the KinD cluster and the host machine, by surrounding the insecure “privileged” containers deployed by KinD with a secure (unprivileged) Sysbox container.
- Avoiding the limitations and complexity of KinD + Rootless Docker for the same.
- Running KinD inside Kubernetes pods securely (without privileged pods or heavier VM-based approaches like KubeVirt).
This setup is useful and gaining popularity, particularly when using containers as development environments or in CI pipelines, because it allows you to prepackage KinD and run it securely on your local host or in Kubernetes clusters.
Background on Sysbox
Sysbox is an open-source & free container runtime (a new type of “runc”) that enables containers to run “system workloads” such a systemd, Docker, Kubernetes, K3s, etc., seamlessly and securely (no tricky container setups or entrypoints, no privileged containers).
It’s pure container technology and does not use VMs (which makes it fast, efficient, and able to run across cloud environments without problem).
To learn more about Sysbox, refer to the Sysbox repo.
Background on K8s.io KinD
The K8s.io KinD tool allows users to easily deploy Kubernetes clusters using containers, as shown in the figure above.
That is, each container acts as a node in the Kubernetes cluster, and the cluster is made up of several such containers interconnected by a container network.
KinD is a very useful tool for testing on Kubernetes, as it allows you to provision clusters quickly and efficiently in your local machine or a cloud instance, using light-weight containers instead of heavier VMs.
Motivation for running KinD inside a Sysbox Container
Normally KinD runs on a host (e.g., bare-metal or VM), but in this article we are running KinD inside a Sysbox container.
The motivation for this is that KinD deploys the cluster using privileged Docker containers by default, and these are pretty insecure (they provide very weak isolation between the container and the host).
By placing the KinD cluster inside a Sysbox container, you can significantly harden the isolation because the privileged containers are only privileged inside the Sysbox container. The Sysbox container itself is unprivileged (aka rootless) and provides strong isolation between KinD and the underlying host.
Though there is an option to use KinD with Rootless Docker to harden isolation, this comes with the complexity and limitations associated with Rootless Docker. Plus it does not provide a solution when you want to run KinD inside Kubernetes pods securely (e.g., for CI setups).
With Sysbox, the approach is simpler, with fewer limitations, and allows you to run KinD securely inside Docker containers or Kubernetes pods.
The next two sections show how to do this.
Running KinD inside a Secure Docker Container
Ok here we go.
To run KinD inside an unprivileged container launched with Docker + Sysbox, follow these steps:
1) Install Sysbox on your host as described in the Sysbox User Guide.
For example, on my Ubuntu 20.04 (aka Focal) host, I installed Sysbox with:
2) Deploy a container using Docker and the Sysbox runtime:
This is the container inside of which the KinD tool and associated cluster will live, as shown in the figure at top of this article.
In this example I chose the image nestybox/ubuntu-focal-systemd-docker because it includes systemd and Docker inside the container ( see the Dockerfile), but you can choose another image if you wish. Just make sure the image includes Docker inside the container.
Once the container is running, you can exec into it and should see Docker running inside of it:
You can verify this is an unprivileged (rootless) container, where the root user inside the container is mapped to an unprivileged user at host level:
For example, in this container the root user (0) is mapped to unprivileged user 165536 on the host, thereby hardening isolation between the container and the host.
3) Install KinD inside the container, following the instructions in the K8s.io KinD website:
If all is good, you should be able to run the kind command inside the container:
We are now ready to deploy our first KinD cluster inside the unprivileged Sysbox container.
Unfortunately there is one wrinkle here regarding the Docker image that the KinD tool must use to deploy the Kubernetes cluster (inside the Sysbox container). The table below describes this:
[1] For KinD clusters that deploy Kubernetes < v1.22, the official kindest/node image does not work inside Sysbox containers due to a problem in the image's entrypoint. The nestybox/kindestnode image corrects this problem (see its Dockerfile).
[2] For KinD clusters that deploy Kubernetes >= v1.22, the official kindest/node image does work. However, you need Sysbox > v0.4.1 (to be released in late Jan '22) as there is a bug in Sysbox that prevents KinD from deploying the cluster.
With this in mind, let’s create a KinD cluster. In this example I am using image nestybox/kindestnode:v1.20.7.
Inside the Sysbox container, do the following:
If all goes well, in less than 2 minutes you’ll have a Kubernetes cluster running entirely inside the unprivileged Sysbox container!
You can then install kubectl inside the Sysbox container and start using the cluster as needed.
Refer to the k8s.io KinD website for more info on how to configure the cluster (e.g., create multi-node clusters, etc.) Note that if you create a multi-node cluster, then all nodes of the cluster would live inside the Sysbox container. In other words, the Sysbox container acts almost like a VM-like environment, but with pure container technology.
Running KinD inside an Unprivileged Pod
With Sysbox, you can also deploy KinD inside a Kubernetes pod securely, without using privileged pods.
This is useful when using KinD in Kubernetes-based CI pipelines or when running development environments inside Kubernetes pods.
To deploy KinD inside an unprivileged Kubernetes pod, follow these steps:
1) Install Sysbox on your Kubernetes cluster as described in the Sysbox User Guide.
For example, I created a Google Kubernetes Engine (GKE) cluster with Ubuntu-based nodes, and installed Sysbox with:
2) Deploy a pod using Docker and the Sysbox runtime:
This is the pod inside of which the KinD tool and associated cluster will live. The pod spec is below:
3) Verify it’s an unprivileged (rootless) pod:
This confirms this is an unprivileged pod: the root user inside the container (0) is mapped to an unprivileged user (362144) at host level. The mapping extends for a range of 65536 user IDs.
This means you can now deploy KinD inside the pod securely: the privileged containers that KinD will use to deploy the Kubernetes cluster are well isolated from the underlying host by the unprivileged Sysbox container.
4) Install KinD inside the pod:
5) Create the KinD cluster inside the pod:
That’s it! You can then install kubectl inside the Sysbox container and start using the cluster as needed.
Sysbox Container Image
In the examples above, after we deployed the Sysbox container using the nestybox/ubuntu-focal-systemd-dockerimage, we had to install KinD and kubectl inside the container manually.
We did this for illustration purposes, but in a real setup you would create an image for the Sysbox container that comes preconfigured with Docker, KinD, Kubectl, and any other tool you need. That way as soon as you deploy the Sysbox container, you can launch the KinD cluster inside of it.
As a reference on how to do this, there is a sample Dockerfile here.
Conclusion
This article showed you how to run the K8s.io KinD tool inside an unprivileged Docker container or Kubernetes pod, with help from the Sysbox runtime.
There are several benefits to doing so, such as containerizing an entire KinD cluster and hardening isolation from the underlying host. It’s very useful for local development or testing on Kubernetes without sacrificing security or functionality.
I hope you find this useful!
To learn more about Sysbox, go to the Sysbox repo, or join the Sysbox Slack Channel and ask questions in there too.
Resources
Originally published at https://blog.nestybox.com on January 10, 2022.
