rfc: trusted k8s images (#2648)

* rfc: trusted k8s images 

Co-authored-by: 3u13r <lc@edgeless.systems>
Co-authored-by: Malte Poll <1780588+malt3@users.noreply.github.com>
Co-authored-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

rfc/XXX-trusted-kubernetes-images.md

@@ -0,0 +1,148 @@
+# RFC XXX: Trusted Kubernetes Container Images
+
+Kubernetes control plane images should be verified by the Constellation installation.
+
+## The Problem
+
+When we bootstrap the Constellation Kubernetes cluster, `kubeadm` places a set
+of static pods for the control plane components into the filesystem. The
+manifests refer to images in a registry beyond the users' control, and the
+container image content is not reproducible.
+
+This is obviously a trust issue, because the Kubernetes control plane is
+part of Constellation's TCB, but it is also a problem when Constellation is set
+up in a restricted environment where this repo is not available.
+
+## Requirements
+
+1. In a default installation, Constellation must verify Kubernetes control plane images.
+
+Out of scope:
+
+- Customization of the image repository or image content.
+  - This is orthogonal to image verification and will be subject of a separate
+    RFC.
+- Reproducibility from github.com/kubernetes/kubernetes to registry.k8s.io and
+  the associated chain of trust.
+  - It is not clear whether Kubernetes images can be reproduced at all [1].
+    Either way, the likely threat is a machine-in-the-middle attack between
+    Constellation and registry.k8s.io. A desirable addition to this proposal
+    could be verification of image signatures [2].
+- Container registry trust & CA certificates.
+  - This is also orthogonal to image verification.
+
+[1]: https://github.com/kubernetes/kubernetes/blob/master/build/README.md#reproducibility
+[2]: https://kubernetes.io/docs/tasks/administer-cluster/verify-signed-artifacts/
+
+## Solution
+
+Kubernetes control plane images are going to be pinned by a hash, which is verified by
+the CRI. Image hashes are added to the Constellation codebase when support for
+a new version is added. During installation, the `kubeadm` configuration is
+modified so that images are pinned.
+
+### Image Hashes
+
+We are concerned with the following control plane images (tags for v1.27.7):
+
+- registry.k8s.io/kube-apiserver:v1.27.7
+- registry.k8s.io/kube-controller-manager:v1.27.7
+- registry.k8s.io/kube-scheduler:v1.27.7
+- registry.k8s.io/coredns/coredns:v1.10.1
+- registry.k8s.io/etcd:3.5.9-0
+
+When a new Kubernetes version is added to `/internal/versions/versions.go`, we
+generate the corresponding list of images with `kubeadm config images list` and
+probe their hashes on registry.k8s.io. Generating the list of images this way
+must happen offline to prevent `kubeadm` from being clever. These hashes are
+added to `versions.go` as a mapping from component to pinned image:
+
+```golang
+V1_27: {
+  ClusterVersion: "v1.27.7",
+  // ...
+  KubernetesImages: {
+    "kube-apiserver": "registry.k8s.io/kube-apiserver:v1.27.7@sha256:<...>", 
+    "kube-controller-manager": "registry.k8s.io/kube-controller-manager:v1.27.7@sha256:<...>", 
+    "kube-scheduler": "registry.k8s.io/kube-scheduler:v1.27.7@sha256:<...>", 
+    "etcd": "registry.k8s.io/etcd:3.5.9-0@sha256:<...>",
+    "coredns": "registry.k8s.io/coredns/coredns:v1.10.1@sha256:<...>",
+  },
+}
+```
+
+### Cluster Init
+
+During cluster initialization, we need to tell `kubeadm` that we want to use
+the embedded image references instead of the default ones. For that, we
+populate the
+[`InitConfiguration.Patches`](https://pkg.go.dev/k8s.io/kubernetes@v1.27.7/cmd/kubeadm/app/apis/kubeadm/v1beta3#InitConfiguration)
+with a list of [JSON Patch](https://datatracker.ietf.org/doc/html/rfc6902)
+files that replace the container image with the pinned alternative.
+
+The patches need to be written to the stateful filesystem by the
+bootstrapper. This is very similar to `components.Component`, which also
+place Kubernetes-related data onto the filesystem:
+
+```go
+type Component struct {
+  URL         string
+  Hash        string
+  InstallPath string
+  Extract     bool
+}
+```
+
+Components are handled by the installer, where the convention currently expects
+HTTP URLs that are to be downloaded. We can extend this by allowing other forms
+of URI schemes:
+[data URLs](https://developer.mozilla.org/en-US/docs/web/http/basics_of_http/data_urls).
+A patch definition as Component would look like this:
+
+```go
+patch := &components.Component{
+  URL: "data:application/json;base64,W3sib3AiOiJyZXBsYWNlIiwicGF0aCI6Ii9zcGVjL2NvbnRhaW5lcnMvMC9pbWFnZSIsInZhbHVlIjoicmVnaXN0cnkuazhzLmlvL215LWNvbnRyb2wtcGxhbmUtaW1hZ2U6djEuMjcuN0BzaGEyNTY6Li4uIn1dCg=="
+  InstallPath: "/opt/kubernetes/patches/kube-apiserver+json.json"
+  // Hash and Extract deliberately left empty.
+}
+```
+
+This method does not work for coredns, which can only be customized with the
+[`ClusterConfiguration.DNS`](https://pkg.go.dev/k8s.io/kubernetes@v1.27.7/cmd/kubeadm/app/apis/kubeadm/v1beta3#ClusterConfiguration)
+section. However, we can split the image into repository, path and tag+hash
+and use these values as `DNS.ImageMeta`.
+
+### Upgrade
+
+The upgrade agent currently receives a kubeadm URI and hash, and internally
+assembles this to a `Component`. We change the upgrade proto to accept
+a full `components.Components`, which then would also include the new patches.
+The components would be populated from the ConfigMap, as is already the case.
+
+The CoreDNS config would need to be updated in the `kube-system/kubeadm-config`
+ConfigMap.
+
+## Alternatives Considered
+
+### Exposing more of KubeadmConfig
+
+We could allow users to supply their own patches to `KubeadmConfig` for finer
+control over the installation. We don't want to do this because:
+
+1. It does not solve the problem of image verification - we'd still need to
+   derive image hashes from somewhere.
+2. It's easy to accidentally leave charted territory when applying config
+   overrides, and responsibilities are unclear in that case: should users be
+   allowed to configure network, etcd, etc.?
+3. The way Kubernetes exposes the configuration is an organically grown mess:
+   registries are now in multiple structs, path names are hard-coded to some
+   extent and versions come from somewhere else entirely (cf.
+   kubernetes/kubernetes#102502).
+
+### Ship the container images with the OS
+
+We could bundle all control plane images in our OS image and configure kubeadm
+to never pull images. This would make Constellation independent of external
+image resources at the expense of flexibility: overriding the control plane
+images in development setups would be harder, and we would not be able to
+support user-provided images anymore.