* bump K8s version in configmap/k8s-version to `vX.Y`
* [upgrade your cluster](https://docs.edgeless.systems/constellation/workflows/upgrade) with a new VM image
For more details on the first steps see the [official K8s documentation](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/). This upgrade [will create new kubelet certificates](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#automatic-certificate-renewal) but does [not rotate Kubernetes CA certificate](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/#certificate-authority-rotation).
## Cilium
Cilium is installed via helm. In the long term, we don't need to maintain our fork and Cilium can be updated independently using the official releases.
## Constellation microservices
All Constellation microservices will be bundled into and therefore updated via one helm chart.
The CLI will use a lookup table to map the Kubernetes version from the config to URLs and hashes. Those are sent over during `constellation init` and used by the first Bootstrapper. Then, the URLs and hashes are pushed to the `k8s-components-1.23.12` ConfigMap and the `k8s-components-1.23.12` ConfigMap is referenced by the `NodeVersion` CR named `constellation-version`.
The JoinService will look at the `k8s-components-1.23.12` ConfigMap in addition to the `NodeVersion` CR named `constellation-version`. Currently, the `k8s-version` ConfigMap is mounted into the JoinService pod. We will change that so that the JoinService requests the `kubernetesComponentsReference` from `constellation-version` and then uses this to look up the Kubernetes components.
Those components are then sent to any node requesting to join the cluster.
Additionally, each node trying to join the cluster is tracked as a `JoiningNode` CR.
The JoinService creates a `JoiningNode` CRD for each issued JoinTicket with the node's name and reference to the Kubernetes components ConfigMap it was sent. This `JoiningNode` CRD is consumed by the node operator.
During the cluster initialization we need to create the first ConfigMap with components and hashes.
We receive all necessary information from the CLI in the first place, since we need to download them to create a initialize the cluster in the first place.
To be able to even update singular components, we need to know if the set of components of a node is the desired one. To achieve that, the Bootstrapper calculates a hash of all the components' hashes.
Because of the length restriction for labels, we need to attach this information as an annotation to the node.
Annotations cannot be set during the join process (in contrast to node-labels).
We somehow need to download and execute `kubeadm upgrade plan` and `kubeadm upgrade apply vX.Y.Z` on the host system of a control plane node. For security reasons, we don't want those capabilities attached to any pod. Therefore, we opted for a simple and small agent, which exposes a narrow and predefined API as a socket on the control-plane host. This socket can then be mounted into the node operator pod running on a control plane node.
First, the node operator consumes the JoiningNode CRD. It watches on changes in the CRD list as well as changes in the node list. The controller reconciles the JoiningNode CRDs by trying to annotate the corresponding node. If successful, the controller deletes the CRD.
Second, we need to extend the node operator to also handle Kubernetes updates. The operator already receives information about the Kubernetes version of each node.
* After the agent returned successfully, update the components reference to `k8s-components-1.23.12` in the `NodeVersion` CRD named `constellation-version`.
Instead of having a separate `upgrade` section, we will opt for a declarative approach by updating the existing values of the config file. Since only parts of the config behave in a declarative way,
*`microserviceVersion` is a bundle of component versions which can be conceptually separated into two groups:
* Services that are versioned based on the constellation version: : KMS, JoinService, NodeMaintainanceOperator, NodeOperator, OLM, Verification, Cilium (for now).
There only exists one version of each service and it is compatible with all Kubernetes versions currently supported by Constellation.
The deployment and image version are the same for all three Kubernetes versions.
* Services that are versioned based on the kubernetes version: Autoscaler, CloudControllerManager, CloudNodeManager, GCP Guest Agent, Konnectivity.
There exist one version for each Kubernetes version.
The deployment is the same for all three Kuberenetes version, but the image is specific to the version.
Images are specified by the CLI upon loading the Helm chart, by inspeciting `constellation-conf.yaml`.
Deployment variations could be introduced into the Helm charts if they become necessary in the future.
When `constellation upgrade check` is called it checks if the current CLI includes helm charts and kubernetes components that are newer than the ones configured in `constellation-config.json`.
If this is the case, the CLI prints a list of all components that will be updated.
Moreover, it checks for new image patch versions via the update API (see: rfc/update-api.json).
Image patch versions are forward compatible within one minor version.
Lastly, the CLI checks if a newer CLI version is available via the update API (see: rfc/update-api.json). If this is the case, it will print the latest CLI version instead of the output described above.
If the current version and latest version diverge more than one minor version, it will also show the latest CLI of the next minor version, and suggest a way to download it.
Since any CLI can only upgrade from one minor version below to its own version, we need to perform the upgrade to `2.4.3` before upgrading to `2.5.0`.
If there are still microservice updates needed with the current CLI, we need to prompt the user to first install those before continuing with the next minor release.
We also print `In newer CLI versions there are even newer versions available.` if e.g. there is a newer patch version of Kubernetes available in one of the proposed minor versions.
Executing `constellation upgrade check --write-config` writes all new version values to `constellation-conf.json`.
This allows the user to execute `constellation upgrade apply` without manually modifying `constellation-conf.json`.
1. warn the user to create a Constellation/etcd backup before updating as documented in the [official K8s update docs](https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/#before-you-begin)
2. create a new `k8s-components-1.24.3` ConfigMap with the corresponding URLs and hashes from the lookup table in the CLI
3. update the measurements in the `join-config` ConfigMap
Since the service versions bundled inside a `microserviceVersion` are hidden, the CLI will print the changes taking place. We also print a warning to back up any important components when the upgrade necessitates a node replacement, i.e. on Kubernetes and VM image upgrades.
`constellation upgrade` has to handle the version of four components: CLI, image, microservices and Kubernetes.
To do this correctly and keep the cluster in a working condition some constraints are required.
- Constellation microservices, VM image and the CLI are versioned in lockstep, i.e. each release a new version of all components is released.
The Constellation version references all of these components.
- Each Constellation version is compatible with three or four Kubernetes versions.
When a new Kubernetes version is released, Constellation will support four versions for one release cycle; before phasing out the oldest Kubernetes version.
To learn if microserviceVersion A.B.C is compatible with Kubernetes version X.Y, one has to check whether the Constellation version is compatible with the Kubernetes version X.Y.
- Each Constellation version X.Y.Z is compatible with all patch versions of the next Constellation version X.Y+1.
- The individual versions of Microservice, image and CLI all have to be compatible with the targeted Kubernetes version.
Each component can be upgraded independently.
This means that during a Kubernetes upgrade the oldest Constellation component has to be compatible with the new Kubernetes version.
If not, this oldest component has to be updated first.
- For Kubernetes versions the version-target of an upgrade has to be supported by the current Constellation version.
The currently running Kubernetes version is not relevant for the Kubernetes upgrade.
These constraints are enforced by the CLI.
If users decide to change specific versions by changing the Kubernetes resources, nothing is stopping them.
The compatibility information should be separated from the enforcement code.
This way a minimal implementation can be created where the compatibility information is embedded into the CLI.
As a next step the information can be served through the [Constellation API](./apis.md).
By serving the compatibility information dynamically, faulty versions can be excluded from upgrade paths even after they have been released.
