constellation/.github/docs/upgrade-kubernetes.md

Upgrading Kubernetes

Constellation is a Kubernetes distribution. As such, dependencies on Kubernetes versions exist in multiple places:

• The desired Kubernetes version deployed by kubeadm init
• Kubernetes resources (deployments made while initializing Kubernetes, including the cloud-controller-manager, cluster-autoscaler and more)
• Kubernetes go dependencies for the bootstrapper code

Understand what has changed

Before adding support for a new Kubernetes version, it is a very good idea to read the release notes and to identify breaking changes.

Upgrading Kubernetes resources

Everything related to Kubernetes versions is tracked in the versions file. Add a new ValidK8sVersion and fill out the VersionConfigs entry for that version. During cluster initialization, multiple Kubernetes resources are deployed. Some of these should be upgraded with Kubernetes. You can check available version tags for container images using the container registry tags API:

curl -q https://k8s.gcr.io/v2/autoscaling/cluster-autoscaler/tags/list | jq .tags
curl -q https://k8s.gcr.io/v2/cloud-controller-manager/tags/list | jq .tags
curl -q https://us.gcr.io/v2/k8s-artifacts-prod/provider-aws/cloud-controller-manager/tags/list | jq .tags
curl -q https://mcr.microsoft.com/v2/oss/kubernetes/azure-cloud-controller-manager/tags/list | jq .tags
curl -q https://mcr.microsoft.com/v2/oss/kubernetes/azure-cloud-node-manager/tags/list | jq .tags
# [...]

Upgrade go dependencies

The go.mod and go.sum files pin versions of the Kubernetes go packages. While these do not need to be on the exact versions used in the Kubernetes deployment, it is a good idea to keep them updated and on a similar version. Upgrade Kubernetes go dependencies by changing the versions of all packages in the k8s.io namespace from the old version to the new version in go.mod and run go mod tidy. Ensure that there are no other conflicts and test your changes. See the diff of this PR as an example of updating the go dependencies.

Test the new Kubernetes version

• Setup a Constellation cluster using the new image with the new bootstrapper binary and check if Kubernetes is deployed successfully.

  # should print the new k8s version for every node
  kubectl get nodes -o wide
  # read the logs for pods deployed in the kube-system namespace and ensure they are healthy
  kubectl -n kube-system get pods
  kubectl -n kube-system logs [...]
  kubectl -n kube-system describe pods
  

• Read the logs of the main Kubernetes components by getting a shell on the nodes and scan for errors / deprecation warnings:

  journalctl -u kubelet
  journalctl -u containerd
  

• Conduct e2e tests

  • Run the sonobuoy test suite against your branch
  • Run CI e2e tests