constellation/dev-docs/workflows/release.md

# Release Checklist

This checklist will prepare `v1.3.0` from `v1.2.0` (minor release) or `v1.3.1` from `v1.3.0` (patch release). Adjust your version numbers accordingly.

## Preparation

1. Search the code for TODOs and FIXMEs that should be resolved before releasing.
2. [Update titles and labels for all PRs relevant for this release](/dev-docs/workflows/pull-request.md) to aid in the [changelog generation](/.github/release.yml).

## Automated release

Releases should be performed using [the automated release pipeline](https://github.com/edgelesssys/constellation/actions/workflows/release.yml).

### Prepare temporary working branch

1. Create a temporary working branch to prepare the release. This branch should be based on main if preparing a minor release or be based on the existing release branch if it is a patch release.

   ```sh
   ver=v1.3.1 # replace me
   minor=$(echo ${ver} | cut -d '.' -f 1,2)
   # optional suffix to add to the temporary branch name. Can be empty: suffix=
   suffix=/foo
   # if preparing a patch release, checkout existing release branch as base
   git checkout release/${minor}
   # if preparing a minor release, branch out from main instead
   git checkout main
   git pull
   working_branch=tmp/${ver}${suffix}
   git checkout -b ${working_branch}
   git push origin ${working_branch}
   ```

### Patch release

1. `cherry-pick` (only) the required commits from `main`
   * Check PRs with label [needs-backport](https://github.com/edgelesssys/constellation/pulls?q=is%3Apr+is%3Aclosed+label%3A%22needs+backport%22) to find candidates that should be included in a patch release.
2. trigger the automated release pipeline from the working branch created above:

   ```sh
   gh workflow run release.yml --ref ${working_branch} -F version=${ver} -F kind=patch
   ```

3. wait for the pipeline to finish
4. Check the s3proxy PR in the [helm chart repository](https://github.com/edgelesssys/helm/pull/) and approve it if things (esp. the version) look correct. Correct example for reference: <https://github.com/edgelesssys/helm/pull/19/files>
5. while in editing mode for the release, clear the textbox, select the last patch release for the current release branch and click "Generate release notes".
6. look over the autogenerated draft release. When fixing the changelog, prioritize updating the PR title/labels/description and regenerating the changelog over fixing things in the final changelog. The changelog should be primarily aimed at users. Rule of thumb: first part of the sentence should describe what changed for the user, second part can describe what has been changed to achieve this.
7. in the GitHub release UI, make sure the tag to create on release is set to `$ver`, and the target commit is set to the temporary release branch.
8. publish.

### Minor release

1. Merge ready PRs
2. trigger the automated release pipeline from the working branch created above:

   ```sh
   gh workflow run release.yml --ref ${working_branch} -F version=${ver} -F kind=minor
   ```

3. wait for the pipeline to finish
4. upgrade the dogfooding cluster. Note that `upgrade check --update-config` will not yet show the new image. But you can manually set it in the config:

   ```sh
   ./constellation upgrade check --update-config
   yq eval -i '.image="vX.YY.Z"' constellation-conf.yaml
   ./constellation config fetch-measurements
   ./constellation apply --yes --debug
   ```

   Then wait until the node / Kubernetes upgrades are finished by periodically checking:

   ```sh
   ./constellation status
   ```

5. Check the s3proxy PR in the [helm chart repository](https://github.com/edgelesssys/helm/pull/) and approve it if things (esp. the version) look correct. Correct example for reference: <https://github.com/edgelesssys/helm/pull/19/files>
6. while in editing mode for the release, clear the textbox, select the last minor release and click "Generate release notes".
7. look over the autogenerated draft release. When fixing the changelog, prioritize updating the PR title/labels/description and regenerating the changelog over fixing things in the final changelog. The changelog should be primarily aimed at users. Rule of thumb: first part of the sentence should describe what changed for the user, second part can describe what has been changed to achieve this.
8. set the Target to `tmp/${ver}`
9. in the GitHub release UI, make sure the tag to create on release is set to `$ver`, and the target commit is set to the temporary release branch.
10. publish.

## Post release steps

1. Publish the [provider release](https://github.com/edgelesssys/terraform-provider-constellation/releases)
2. Merge the versioned docs PR
3. Close fixed "known issues"
4. Milestones management
   1. Create a new milestone for the next release
   2. Add the next release manager and an approximate release date to the milestone description
   3. Close the milestone for the release
   4. Move open issues and PRs from closed milestone to next milestone
5. If the release is a minor version release, bump the pre-release version in the `version.txt` file.
6. Update the `fromVersion` in `e2e-test-release.yml` and `e2e-test-weekly.yaml` to the newly released version. To check the current values, run: `grep "fromVersion: \[.*\]" -R .github`.
7. Reset `UpgradeRequiresIAMMigration`  in [`iamupgrade.go`](https://github.com/edgelesssys/constellation/blob/a88a731576184e3c5ee8527741c4a0cdaa4e9b24/cli/internal/cloudcmd/iamupgrade.go#L23).
8. Write an email to STACKIT to inform them of the new relase. For this, you require the name and UUID of the release image. You can find the email address in our internal [wiki](https://github.com/edgelesssys/wiki/blob/master/documentation/constellation/stackit.md):
    ```shell-session
    export OS_CLOUD=stackit
    openstack image list | grep constellation
    # the output should look similar to this, where the first column is the UUID and the second column is the name
    # | 25edf48d-161f-452b-b420-963c3a80abd8 | constellation-stable-v2.16.4-qemu-vtpm | active |
    ```

## Troubleshooting: Pipeline cleanup

No manual steps should be necessary anymore but in case you encounter issues, create a ticket to fix it. These are instructions to do some cleanup steps manually:

### General

Depending on how far the pipeline ran we need to delete:

* the working branch (remove automated commits made by the process, keep any cherry picks)
* (only minor releases) the branch to merge changes back to main: `feat/release/v1.3.0`

### GCP

1. Navigate to [Images](https://console.cloud.google.com/compute/images?tab=images&project=constellation-images) tab of the "constellation-images" project
2. Search for the image versions "v1-3-0-gcp-sev-es-stable" and "v1-3-0-gcp-sev-snp-stable"
3. Select the images and press "DELETE"

### Azure

1. Navigate to [Azure compute galleries](https://portal.azure.com/#view/HubsExtension/BrowseResource/resourceType/Microsoft.Compute%2Fgalleries)
2. Select "Constellation_CVM" (this is for confidential vms on AMD SEV SNP)
3. Select image definition "constellation"
4. Select "Versions" submenu and search for "1.3.0"
5. Press "Delete" button NEXT TO THE IMAGE VERSION TABLE. Do no delete the image definition.

### AWS

**Important:** You need to repeat the following steps for every region supported by Constellation!
Currently, this includes:

* Frankfurt (eu-central-1)
* Ireland (eu-west-1)
* Paris (eu-west-3)
* Ohio (us-east-2)
* Mumbai (ap-south-1)

#### Automated script

This is a script to automate the deletion but please be super careful to set the version correctly.

```shell
VERSION=vX.XX.X # !! DOUBLE CHECK CORRECTNESS!
regions=("eu-central-1" "eu-west-1" "eu-west-3" "us-east-2" "ap-south-1")
for region in "${regions[@]}"
do
    aws ec2 describe-images --filters "Name=name,Values=constellation-$VERSION-aws-sev-snp" --query "Images[0].ImageId" --output text --region "$region"  | xargs -I {{image_id}} aws ec2 deregister-image --image-id {{image_id}} --region "$region"
    aws ec2 describe-snapshots --filters Name=tag:Name,Values=constellation-$VERSION-aws-sev-snp --query 'Snapshots[].SnapshotId' --output text --region "$region" | xargs -n 1 aws ec2 delete-snapshot --snapshot-id --region "$region"
    aws ec2 describe-images --filters "Name=name,Values=constellation-$VERSION-aws-nitro-tpm" --query "Images[0].ImageId" --output text --region "$region"  | xargs -I {{image_id}} aws ec2 deregister-image --image-id {{image_id}} --region "$region"
    aws ec2 describe-snapshots --filters Name=tag:Name,Values=constellation-$VERSION-aws-nitro-tpm --query 'Snapshots[].SnapshotId' --output text --region "$region"  | xargs -n 1 aws ec2 delete-snapshot --snapshot-id --region "$region"
done
```

#### Manual GUI steps

1. Navigate to [AMI](https://eu-central-1.console.aws.amazon.com/ec2/home?region=eu-central-1#Images:visibility=owned-by-me)
2. Search for release version "constellation-v1.3.0" and select the AMIs for both variants ("constellation-v1.3.0-aws-sev-snp" and "constellation-v1.3.0-aws-nitro-tpm")
3. On the "Actions" button (top right) select "Deregister AMI"
4. Either follow the link on the deletion confirmation leading you to the [Snapshots](https://eu-central-1.console.aws.amazon.com/ec2/home?region=eu-central-1#Snapshots) panel or navigate there yourself
5. Search for a snapshot by the same name "constellation-v1.3.0" and select it
6. On the "Actions" button (top right) select "Delete snapshot"