Secure Docker-in-Kubernetes

Intro

Motivation

  1. Creating a pool of Docker engines on the cloud. Each user is assigned one such engine and connects remotely to it via the Docker CLI. Each Docker engine runs inside a Kubernetes pod (instead of a VM), so operators can leverage the power of Kubernetes to manage the pool’s resources.
  2. Running Docker inside Kubernetes-native CI jobs. Each job is deployed inside a pod and the job uses the Docker engine running inside the pod to build container images (e.g., Buildkit), push them to some repo, run them, etc.

Setup

  • Kubernetes will deploy the pods with the Sysbox runtime.
  • Each pod will run a Docker engine and SSH in it.
  • Each Docker engine will be assigned to a user (say a developer working from home with a laptop).
  • The user will connect remotely to her assigned Docker engine using the Docker CLI.

Why is the Sysbox Runtime Needed Here?

Kubernetes Cluster Creation

kubectl apply -f https://raw.githubusercontent.com/nestybox/sysbox/master/sysbox-k8s-manifests/sysbox-install.yaml

Defining the Pods (with Docker inside)

  • It creates 6 pods in parallel (see replicas and podManagementPolicy).
  • The pods are rootless by virtue of using Sysbox (see the cri-o annotation and sysbox-runc runtimeClassName).
  • Each pod exposes port 22 (ssh).
  • Each pod has a persistent volume mounted onto the pod’s /var/lib/docker directory (see next section).

Persistent Docker Cache

Deploying the Pods

$ kubectl apply -f gce-pd.yaml
$ kubectl apply -f local-storage.yaml
$ kubectl apply -f dockerd-statefulset.yaml

Verify the Pods are Working

Exposing the Pod’s IP Outside the Cluster

Connecting Remotely to the Pods

  1. Configure ssh access to the pod.
  2. Use the Docker CLI to connect to the pod remotely via ssh.

SSH config

  • Exec into one of the pods, and create a password for user root inside the pod:
  • Give the pod’s external IP address (see prior section) and password to the remote user in some secret way.
  • The remote user copies her machine’s public SSH key (e.g., generated with ssh-keygen) to the pod.

Docker CLI Access

Sharing Docker Images across Docker Engines

Scaling Pod Instances

Persistent Volume Removal

  1. Stop the pod using the persistent volume.
  2. List the persistent volume claims (pvc):

Docker Build Context

Conclusion

Resources

--

--

--

Founder and CEO of Nestybox, Inc.

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Cesar Talledo

Cesar Talledo

Founder and CEO of Nestybox, Inc.

More from Medium

How to safely drain the k8s node!

Continuous Build and Deployment of Go Applications with Google Cloud Build

Top Advantages and Disadvantages Of Kubernetes

Installation of Cloud-Native Application Stacks in Amazon EKS Cluster