constellation/rfc/013-state-file.md

RFC 013: Constellation state file

During a cluster's lifecycle, Constellation needs to keep track of multiple different values and state information of the cluster, like loadbalancer IPs or UID of the cluster.

If this information is configurable by a user, and required to set up the cluster, the user saves it in Constellation's config file. Information that is not configurable, and that is generated by the cluster during the creation or init phase is either saved in Terraform output files, or in the cluster ID file (constellation-id.json).

This creates problems since the relevant information may be split over multiple files, and it is not always clear at what point certain information is available in the cluster ID file. Additionally, commands like upgrade and init currently depend on Terraform files being present in the working directory, which may not be the case if the cloud resources were manually created.

To fix these problems, we want to use a Constellation state file, which keeps track of cluster relevant information.

Design Goals

The state file should replace the current constellation-id.json file and be the single source of truth for this kind of cluster relevant information. The file should keep track of all information currently kept in constellation-id.json, as well as information we currently have to read from Terraform files during init or upgrade.

The file should be considered "read-only" by users, if Constellation's cloud resource were created using the Constellation CLI. If the cloud resources were created manually by a user, they will need to manually fill in some of the state file's fields.

This will allow us to more easily decouple resource creation/management from Constellation. Meaning it should be much easier for Constellation users to have different parties create and update cloud resources and IAM using their own tooling (e.g. Terraform), and only use the Constellation CLI for Kubernetes cluster management (Kubernetes versions, service versions, image upgrades etc.).

Format

The state file will be in annotated YAML format, and contain a version number to keep track of necessary migrations or breaking formats. The file should be split into multiple sections for each relevant resource. Annotations should be provided for all keys in the file. Where possible, links to documentation for specific keys should be provided to further explain their usage.

Each section and key should have a comment dedicated to explaining what it does, where to get it from, and if it is to be considered read-only, or if a user has to supplement a value themselves (if they manage their own infrastructure).

Information that is only used to upgrade Terraform resources (e.g. IAM) should not be part of this file.

The following is a list of proposed sections:

• Infrastructure

  Information about resources we currently create using Terraform. Essentially, we want to record the information we currently write to the cluster ID file during create, as well as the Terraform output during create.

  This field is expected to be manually set if cloud resources were manually deployed by a user. If cloud resource were created with the Constellation CLI (using constellation create), it will be populated automatically.

• Cluster values

  Cluster ID, measurement salt etc. This field will be automatically populated during constellation init. This entry should be marked with DO NOT EDIT to make it clear these values are expected to not be touched by a user manually.

YAML file example:

version: "v1" # version of the file format
# infrastructure contains values describing the cloud infrastructure for Constellation.
# Values may either be generated by the Constellation CLI during `constellation create`,
# or filled in manually if cloud resources are managed separately.
infrastructure:
    uid: "001122" # UID used to mark Constellation resources.
    publicIP: "192.0.2.1" # Public IP of the Cluster's API server endpoint.
    initSecret: c2VjcmV0Cg== # initSecret is used to authenticated bootstrapping requests.
    # apiServerCertSANs are subject alternative names to use for the Cluster's API Server.
    apiServerCertSANs:
        - "192.0.2.1"
        - "endpoint.example.com"
    # azure contains resources specific for Azure.
    azure:
        resourceGroup: "example-group" # resource group the resources are deployed to.
        subscriptionID: "subscription-id" # subscription ID the resources are deployed to.
        networkSecurityGroupName: "example-group-name" # network security group name of the cluster.
        loadBalancerName: "example-loadbalancer-name" # name of the loadbalancer in Azure.
        userAssignedIdentity: "identity" # name of the user assigned identity to use for the cluster.
        attestationURL: "attestation.example.com" # URL of the Microsoft Azure Attestation Service (MAA) to use for attestation validation.
    # gcp contains resources specific for GCP.
    gcp:
        projectID: "project-id" # project ID the cluster is deployed to.
        ipCidrNode: "192.0.2.0/24" # IP Cidr range of the cluster's nodes.
        ipCidrPod: "192.0.2.0/24" # IP Cidr range of the cluster's pods.
# clusterValues contains values generated by Constellation when bootstrapping the cluster.
# These values are automatically generated and filled in by Constellation.
# DO NOT TOUCH
clusterValues:
    clusterID: "00112233445566778899AABBCCDDEEFF" # cluster ID uniquely identifies this Constellation cluster.
    ownerID: "00112233445566778899AABBCCDDEEFF" # owner ID identifies this cluster as belonging to owner.
    measurementSalt: "c2VjcmV0Cg==" # measurement salt is used by nodes to derive their cluster ID.
    name: "constell-001122" # name of the cluster, as used in e.g. cluster resource naming.

Updates to the state file

Values depending on Terraform, that change during a Constellation upgrade, should be automatically updated by the CLI if the cloud resources are not manually managed by the user. If they are manually managed, the user has to update the values on their own.

Values depending purely on Kubernetes/node state should be automatically updated by the CLI.

Breaking format changes should be automatically handled for clusterValues. Breaking format changes should be automatically handled for infrastructure if the resources are managed by the Constellation CLI. Otherwise, we should provide documentation and/or tooling for migrating old file formats.

Migrating from terraform outputs + id file

The Constellation version implementing the state file should provide an upgrade path for moving the Terraform outputs and cluster id file content into the new state file. This should happen at the beginning of constellation upgrade apply.

We might want to implement a CLI command to automatically create the state file from an existing Terraform state. In that case, the migration path should make use of that command to set up the state file.