cli: use state file on init and upgrade (#2395)

* [wip] use state file in CLI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use state file in CLI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

take clusterConfig from IDFile for compat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

various fixes

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

wip

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add GCP-specific values in Helm loader test

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove unnecessary pointer

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* write ClusterValues in one step

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move stub to test file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove mention of id-file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move output to `migrateTerraform`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* unconditional assignments converting from idFile

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move require block in go modules file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fall back to id file on upgrade

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix linter check

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add notice to remove Terraform state check on manual migration

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add `name` field

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

fix name tests

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* return early if no Terraform diff

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* return infrastructure state even if no diff exists

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add TODO to remove comment

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use state-file in miniconstellation

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* cli: remove id-file (#2402)

* remove id-file from `constellation create`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add file renaming to handler

* rename id-file after upgrade

* use idFile on `constellation init`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation verify`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* linter fixes

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation mini`

* remove id-file from `constellation recover`

* linter fixes

* remove id-file from `constellation terminate`

* fix initSecret type

* fix recover argument precedence

* fix terminate test

* generate

* add TODO to remove id-file removal

* Update cli/internal/cmd/init.go

Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>

* fix verify arg parse logic

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add version test

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from docs

* add file not found log

* use state-file in miniconstellation

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation iam destroy`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `cdbg deploy`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>

* use state-file in CI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update orchestration docs

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>

docs/docs/architecture/orchestration.md

@@ -8,7 +8,7 @@ The CLI is also used for updating your cluster.
 ## Workspaces
 
 Each Constellation cluster has an associated *workspace*.
-The workspace is where data such as the Constellation state, config, and ID files are stored.
+The workspace is where data such as the Constellation state and config files are stored.
 Each workspace is associated with a single cluster and configuration.
 The CLI stores state in the local filesystem making the current directory the active workspace.
 Multiple clusters require multiple workspaces, hence, multiple directories.
@@ -21,14 +21,14 @@ To allow for fine-grained configuration of your cluster and cloud environment, C
 Altogether, the following files are generated during the creation of a Constellation cluster and stored in the current workspace:
 
 * a configuration file
-* an ID file
+* a state file
 * a Base64-encoded master secret
 * [Terraform artifacts](../reference/terraform.md), stored in subdirectories
 * a Kubernetes `kubeconfig` file.
 
-After the creation of your cluster, the CLI will provide you with a Kubernetes `kubeconfig` file.
+After the initialization of your cluster, the CLI will provide you with a Kubernetes `kubeconfig` file.
 This file grants you access to your Kubernetes cluster and configures the [kubectl](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) tool.
-In addition, the cluster's [identifier](orchestration.md#post-installation-configuration) is returned and stored in a file called `constellation-id.json`
+In addition, the cluster's [identifier](orchestration.md#post-installation-configuration) is returned and stored in the state file.
 
 ### Creation process details
 

docs/docs/reference/cli.md

@@ -380,7 +380,7 @@ Verify the confidential properties of a Constellation cluster
 ### Synopsis
 
 Verify the confidential properties of a Constellation cluster.
-If arguments aren't specified, values are read from `constellation-id.json`.
+If arguments aren't specified, values are read from `constellation-state.yaml`.
 
 ```
 constellation verify [flags]

docs/docs/workflows/create.md

@@ -65,14 +65,16 @@ terraform init
 terraform apply
 ```
 
-The Constellation [init step](#the-init-step) requires the already created `constellation-config.yaml` and the `constellation-id.json`.
-Create the `constellation-id.json` using the output from the Terraform state and the `constellation-conf.yaml`:
+The Constellation [init step](#the-init-step) requires the already created `constellation-config.yaml` and the `constellation-state.yaml`.
+Create the `constellation-state.yaml` using the output from the Terraform state and the `constellation-conf.yaml`:
 
 ```bash
 CONSTELL_IP=$(terraform output ip)
 CONSTELL_INIT_SECRET=$(terraform output initSecret | jq -r | tr -d '\n' | base64)
-CONSTELL_CSP=$(cat constellation-conf.yaml | yq ".provider | keys | .[0]")
-jq --null-input --arg cloudprovider "$CONSTELL_CSP" --arg ip "$CONSTELL_IP" --arg initsecret "$CONSTELL_INIT_SECRET" '{"cloudprovider":$cloudprovider,"ip":$ip,"initsecret":$initsecret}' > constellation-id.json
+touch constellation-state.yaml
+yq eval '.version ="v1"' --inplace constellation-state.yaml
+yq eval '.infrastructure.initSecret ="$CONSTELL_INIT_SECRET"' --inplace constellation-state.yaml
+yq eval '.infrastructure.clusterEndpoint ="$CONSTELL_IP"' --inplace constellation-state.yaml
 ```
 
 </tabItem>

docs/docs/workflows/recovery.md

@@ -125,7 +125,7 @@ This means that you have to recover the node manually.
 
 Recovering a cluster requires the following parameters:
 
-* The `constellation-id.json` file in your working directory or the cluster's load balancer IP address
+* The `constellation-state.yaml` file in your working directory or the cluster's endpoint
 * The master secret of the cluster
 
 A cluster can be recovered like this:

docs/docs/workflows/terminate.md

@@ -51,7 +51,7 @@ terraform destroy
 Delete all files that are no longer needed:
 
 ```bash
-rm constellation-id.json constellation-admin.conf
+rm constellation-state.yaml constellation-admin.conf
 ```
 
 Only the `constellation-mastersecret.json` and the configuration file remain.

docs/docs/workflows/verify-cluster.md

@@ -78,7 +78,7 @@ From the attestation statement, the command verifies the following properties:
 
 * The cluster is using the correct Confidential VM (CVM) type.
 * Inside the CVMs, the correct node images are running. The node images are identified through the measurements obtained in the previous step.
-* The unique ID of the cluster matches the one from your `constellation-id.json` file or passed in via `--cluster-id`.
+* The unique ID of the cluster matches the one from your `constellation-state.yaml` file or passed in via `--cluster-id`.
 
 Once the above properties are verified, you know that you are talking to the right Constellation cluster and it's in a good and trustworthy shape.