From a9258eb9331cfcc2eb31711b38b4fd9b1054a58d Mon Sep 17 00:00:00 2001 From: flxflx Date: Thu, 26 Sep 2024 13:25:30 +0200 Subject: [PATCH] Restructure: Move attestation after "Cluster creation" --- dev-docs/security-overview.md | 34 ++++++++++++++++++---------------- 1 file changed, 18 insertions(+), 16 deletions(-) diff --git a/dev-docs/security-overview.md b/dev-docs/security-overview.md index dab61f533..d438dcc8e 100644 --- a/dev-docs/security-overview.md +++ b/dev-docs/security-overview.md @@ -15,7 +15,7 @@ Abstractly, the Constellation software comprises three core components: The CLI executable is signed by Edgeless Systems. To ensure non-repudiability for CLI releases, Edgeless Systems publishes corresponding signatures to the public ledger of the [Sigstore project](https://www.sigstore.dev/). There's a [step-by-step guide](https://docs.edgeless.systems/constellation/workflows/verify-cli) on how to verify CLI signatures based on Sigstore. -The CLI contains the [measurements](https://github.com/edgelesssys/constellation/blob/edc0c7068ef4527efeaf584a2a35e0f51f58c426/internal/attestation/measurements/measurements_enterprise.go#L21) of the latest Constellation node image for all supported cloud platforms. The CLI writes these measurements as part of the *attestation config* to a cluster's config file with the ["config generate" command](https://docs.edgeless.systems/constellation/workflows/config). Note that currently Constellation uses 16 TPM-based runtime measurements for each cloud platform. The purpose and source of the measurements are described in the [next section](#remote-attestation-of-nodes). In addition to the measurements, the attestation config contains expected patch levels for the CPU microcode and the X.509 certificate of the CPU vendor's remote attestation infrastructure. An example of an attestation config is given [below](#attestation-config). +The CLI contains the [measurements](https://github.com/edgelesssys/constellation/blob/edc0c7068ef4527efeaf584a2a35e0f51f58c426/internal/attestation/measurements/measurements_enterprise.go#L21) of the latest Constellation node image for all supported cloud platforms. The CLI writes these measurements as part of the *attestation config* to a cluster's config file with the ["config generate" command](https://docs.edgeless.systems/constellation/workflows/config). Note that Constellation currently uses 16 TPM-based runtime measurements for each cloud platform. The purpose and source of the measurements are described in the [next section](#remote-attestation-of-nodes). In addition to the measurements, the attestation config contains expected patch levels for the CPU microcode and the X.509 certificate of the CPU vendor's remote attestation infrastructure. An example of an attestation config is given [below](#attestation-config). In case a different version of the node image is to be used, the corresponding measurements can be fetched using the CLI's ["config fetch-measurements" command](reference/cli#constellation-config-fetch-measurements). This command downloads the measurements and the corresponding signature from Edgeless Systems from https://cdn.confidential.cloud. See for example the following files corresponding to node image v2.16.3: * [Measurements](https://cdn.confidential.cloud/constellation/v2/ref/-/stream/stable/v2.16.3/image/measurements.json) @@ -24,7 +24,22 @@ In case a different version of the node image is to be used, the corresponding m In addition to the attestation config, the CLI contains the following data in hardcoded form: * The [long-term public key of Edgeless Systems](https://github.com/edgelesssys/constellation/blob/edc0c7068ef4527efeaf584a2a35e0f51f58c426/internal/constants/constants.go#L264) to verify the signature of downloaded measurements. * The [hashes of the expected Kubernetes binaries](https://github.com/edgelesssys/constellation/blob/edc0c7068ef4527efeaf584a2a35e0f51f58c426/internal/versions/versions.go#L199). -* The [Helm charts used for the installation of services](https://github.com/edgelesssys/constellation/tree/main/internal/constellation/helm/charts), which include hashes of the respective containers. *Note: The Helm charts and the hashes are [generated at build time](https://github.com/edgelesssys/constellation/blob/main/internal/constellation/helm/imageversion/imageversion.go). A future version of the CLI will provide a command to print the Helm charts.* +* The [Helm charts used for the installation of services](https://github.com/edgelesssys/constellation/tree/main/internal/constellation/helm/charts), which include hashes of the respective containers. Note that the Helm charts and the hashes are [generated at build time](https://github.com/edgelesssys/constellation/blob/main/internal/constellation/helm/imageversion/imageversion.go). A future version of the CLI will provide a command to print the Helm charts. + +## Cluster creation + +When a cluster is [created](https://docs.edgeless.systems/constellation/workflows/create), the CLI interacts with the API of the respective infrastructure provider, for example Azure, and launches CVMs with the applicable node image. These CVMs are called *nodes*. On each node, the Bootstrapper is launched. + +The CLI automatically selects one of the nodes as *first node*. The CLI automatically verifies the first node's remote-attestation statement using the attestation config. Details on remote attestation are given in the [next section](#remote-attestation-of-nodes). + +Based on the remote-attestation statement, the CLI and the Bootstrapper running on the first node set up a temporary TLS connection between them. We refer to this type of connection as "attested TLS" (aTLS). This connection is mainly used for three things (see the the [interface definition](https://github.com/edgelesssys/constellation/blob/main/bootstrapper/initproto/init.proto) for a comprehensive list of exchanged data): +1. The CLI sends the hashes of the expected Kubernetes binaries to the first node. +2. The CLI generates the [master secret](../architecture/keys.md#master-secret) of the to-be-created cluster and sends it to the first node. +3. The first node generates a [kubeconfig file](https://www.redhat.com/sysadmin/kubeconfig) and sends it to the CLI. The kubeconfig file contains Kubernetes credentials for the CLI and the Kubernetes cluster's public key, among others. + +After this, the aTLS connection is closed and the Bootstrapper marks the node irrevocably as "initialized". This mechanism prevents a node from accidentally or maliciously joining different clusters. On all supported CVM platforms this is currently implemented by *extending* TPM register 15 with the unique ID of the cluster (`clusterID`). More information can be found in the [Constellation documentation](https://docs.edgeless.systems/constellation/architecture/keys#cluster-identity). + +For [launch digest-based attestation](#remote-attestation-of-nodes) on future CVM platforms, an alternative would be to extend `clusterID` to the so called RTMR registers of Intel TDX. TDX provides four RTMRs, which are automatically included in the `auxiliary data` part of a remote-attestation statement. For AMD SEV-SNP, a different solution exists. ## Remote attestation of nodes @@ -34,7 +49,7 @@ To identify themselves, nodes use the remote-attestation functionality of the un R = Sig-CPU(, , ) ``` -The `payload` is controlled by the software running inside the CVM. In the case of a Constellation node, the `payload` is always the public key of the respective Bootstrapper running inside the CVM. Thus, `R` can be seen as a certificate for that public key issued by the CPU. Based on this, nodes establish so called "attested TLS" (aTLS) connections. aTLS is used during [cluster creation](#cluster-creation) and when [growing a cluster](#cluster-growth). +The `payload` is controlled by the software running inside the CVM. In the case of a Constellation node, the `payload` is always the public key of the respective Bootstrapper running inside the CVM. Thus, `R` can be seen as a certificate for that public key issued by the CPU. Based on this, nodes establish attested TLS (aTLS) connections. aTLS is used during [cluster creation](#cluster-creation) and when [growing a cluster](#cluster-growth). ### Measurements @@ -50,19 +65,6 @@ With measured boot, Constellation only checks the 16 runtime measurements during Currently, on AWS and GCP the TPM implementation resides outside the CVM. On Azure, the TPM implementation is part of the injected firmware and resides inside the CVM. More information can be found in the [Constellation documentation](https://docs.edgeless.systems/constellation/overview/clouds). -## Cluster creation - -When a cluster is [created](https://docs.edgeless.systems/constellation/workflows/create), the CLI interacts with the API of the respective infrastructure provider, for example Azure, and launches CVMs with the applicable node image. These CVMs are called *nodes*. On each node, the Bootstrapper is launched. - -The CLI automatically selects one of the nodes as *first node*. The CLI automatically verifies the first node's remote-attestation statement using the attestation config. Based on this, the CLI and the Bootstrapper running on the first node set up a temporary aTLS connection. This connection is mainly used for three things (see the the [interface definition](https://github.com/edgelesssys/constellation/blob/main/bootstrapper/initproto/init.proto) for a comprehensive list of exchanged data): -1. The CLI sends the hashes of the expected Kubernetes binaries to the first node. -2. The CLI generates the [master secret](../architecture/keys.md#master-secret) of the to-be-created cluster and sends it to the first node. -3. The first node generates a [kubeconfig file](https://www.redhat.com/sysadmin/kubeconfig) and sends it to the CLI. The kubeconfig file contains Kubernetes credentials for the CLI and the Kubernetes cluster's public key, among others. - -After this, the aTLS connection is closed and the Bootstrapper marks the node irrevocably as "initialized". This mechanism prevents a node from accidentally or maliciously joining different clusters. On all supported CVM platforms this is currently implemented by *extending* TPM register 15 with the unique ID of the cluster (`clusterID`). More information can be found in the [Constellation documentation](https://docs.edgeless.systems/constellation/architecture/keys#cluster-identity). - -For [launch digest-based attestation](#remote-attestation-of-nodes) on future CVM platforms, an alternative would be to extend `clusterID` to the so called RTMR registers of Intel TDX. TDX provides four RTMRs, which are automatically included in the `auxiliary data` part of a remote-attestation statement. For AMD SEV-SNP, a different solution exists. - ## Kubernetes bootstrapping on the first node The Bootstrapper on the first node downloads and verifies the Kubernetes binaries, using the hashes it received from the CLI. These binaries include for example [kubelet](https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/), [kube-apiserver](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/), and [etcd](https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/). With these, the Bootstrapper creates a single-node Kubernetes cluster. Etcd is a distributed key-value store that Kubernetes uses to store configuration data for services. The etcd agent runs on each control-plane node of a cluster. The agents use mTLS for communication between them. Etcd uses the Raft protocol (over mTLS) to distribute state between nodes. All essential configuration data of a cluster is kept in etcd.