constellation/dev-docs/workflows/release.md

Release Checklist

This checklist will prepare v1.3.0 from v1.2.0 (minor release) or v1.3.1 from v1.3.0 (patch release). Adjust your version numbers accordingly.

Preparation

1. Search the code for TODOs and FIXMEs that should be resolved before releasing.
2. Update titles and labels for all PRs relevant for this release to aid in the changelog generation.

Automated release

Releases should be performed using the automated release pipeline.

Prepare temporary working branch

1. Create a temporary working branch to prepare the release. This branch should be based on main if preparing a minor release or be based on the existing release branch if it is a patch release.

   ver=v1.3.1 # replace me
   minor=$(echo ${ver} | cut -d '.' -f 1,2)
   # optional suffix to add to the temporary branch name. Can be empty: suffix=
   suffix=/foo
   # if preparing a patch release, checkout existing release branch as base
   git checkout release/${minor}
   # if preparing a minor release, branch out from main instead
   git checkout main
   git pull
   working_branch=tmp/${ver}${suffix}
   git checkout -b ${working_branch}
   git push origin ${working_branch}
   

Patch release

1. cherry-pick (only) the required commits from main

   • Check PRs with label needs-backport to find candidates that should be included in a patch release.

2. trigger the automated release pipeline from the working branch created above:

   gh workflow run release.yml --ref ${working_branch} -F version=${ver} -F kind=patch
   

3. wait for the pipeline to finish

4. Check the s3proxy PR in the helm chart repository and approve it if things (esp. the version) look correct. Correct example for reference: https://github.com/edgelesssys/helm/pull/19/files

5. while in editing mode for the release, clear the textbox, select the last patch release for the current release branch and click "Generate release notes".

6. look over the autogenerated draft release. When fixing the changelog, prioritize updating the PR title/labels/description and regenerating the changelog over fixing things in the final changelog. The changelog should be primarily aimed at users. Rule of thumb: first part of the sentence should describe what changed for the user, second part can describe what has been changed to achieve this.

7. in the GitHub release UI, set the tag to create on publish to $ver.

8. publish.

Minor release

1. Merge ready PRs

2. trigger the automated release pipeline from the working branch created above:

   gh workflow run release.yml --ref ${working_branch} -F version=${ver} -F kind=minor
   

3. wait for the pipeline to finish

4. upgrade the dogfooding cluster. Note that upgrade check --update-config will not yet show the new image. But you can manually set it in the config:

   ./constellation upgrade check --update-config
   yq eval -i '.image="vX.YY.Z"' constellation-conf.yaml
   ./constellation config fetch-measurements
   ./constellation apply --yes --debug
   

   Then wait until the node / Kubernetes upgrades are finished by periodically checking:

   ./constellation status
   

5. Check the s3proxy PR in the helm chart repository and approve it if things (esp. the version) look correct. Correct example for reference: https://github.com/edgelesssys/helm/pull/19/files

6. while in editing mode for the release, clear the textbox, select the last minor release and click "Generate release notes".

7. look over the autogenerated draft release. When fixing the changelog, prioritize updating the PR title/labels/description and regenerating the changelog over fixing things in the final changelog. The changelog should be primarily aimed at users. Rule of thumb: first part of the sentence should describe what changed for the user, second part can describe what has been changed to achieve this.

8. in the GitHub release UI, set the tag to create on publish to $ver.

9. publish.

Post release steps

1. Close fixed "known issues"
2. Milestones management
   1. Create a new milestone for the next release
   2. Add the next release manager and an approximate release date to the milestone description
   3. Close the milestone for the release
   4. Move open issues and PRs from closed milestone to next milestone
3. If the release is a minor version release, bump the pre-release version in the version.txt file.
4. Update the fromVersion in e2e-test-release.yml and e2e-test-weekly.yaml to the newly released version. To check the current values, run: grep "fromVersion: \[.*\]" -R .github.
5. Reset upgradeRequiresIAMMigration in iamupgradeapply.go.

Troubleshooting: Pipeline cleanup

No manual steps should be necessary anymore but in case you encounter issues, create a ticket to fix it. These are instructions to do some cleanup steps manually:

General

Depending on how far the pipeline ran we need to delete:

• the working branch (remove automated commits made by the process, keep any cherry picks)
• (only minor releases) the branch to merge changes back to main: feat/release/v1.3.0

GCP

1. Navigate to Images tab of the "constellation-images" project
2. Search for the image versions "v1-3-0-gcp-sev-es-stable" and "v1-3-0-gcp-sev-snp-stable"
3. Select the images and press "DELETE"

Azure

1. Navigate to Azure compute galleries
2. Select "Constellation_CVM" (this is for confidential vms on AMD SEV SNP)
3. Select image definition "constellation"
4. Select "Versions" submenu and search for "1.3.0"
5. Press "Delete" button NEXT TO THE IMAGE VERSION TABLE. Do no delete the image definition.

AWS

Important: You need to repeat the following steps for every region supported by Constellation! Currently, this includes:

• Frankfurt (eu-central-1)
• Ireland (eu-west-1)
• Paris (eu-west-3)
• Ohio (us-east-2)
• Mumbai (ap-south-1)

Automated script

This is a script to automate the deletion but please be super careful to set the version correctly.

VERSION=vX.XX.X # !! DOUBLE CHECK CORRECTNESS!
regions=("eu-central-1" "eu-west-1" "eu-west-3" "us-east-2" "ap-south-1")
for region in "${regions[@]}"
do
    aws ec2 describe-images --filters "Name=name,Values=constellation-$VERSION-aws-sev-snp" --query "Images[0].ImageId" --output text --region "$region"  | xargs -I {{image_id}} aws ec2 deregister-image --image-id {{image_id}} --region "$region"
    aws ec2 describe-snapshots --filters Name=tag:Name,Values=constellation-$VERSION-aws-sev-snp --query 'Snapshots[].SnapshotId' --output text --region "$region" | xargs -n 1 aws ec2 delete-snapshot --snapshot-id --region "$region"
    aws ec2 describe-images --filters "Name=name,Values=constellation-$VERSION-aws-nitro-tpm" --query "Images[0].ImageId" --output text --region "$region"  | xargs -I {{image_id}} aws ec2 deregister-image --image-id {{image_id}} --region "$region"
    aws ec2 describe-snapshots --filters Name=tag:Name,Values=constellation-$VERSION-aws-nitro-tpm --query 'Snapshots[].SnapshotId' --output text --region "$region"  | xargs -n 1 aws ec2 delete-snapshot --snapshot-id --region "$region"
done

Manual GUI steps

1. Navigate to AMI
2. Search for release version "constellation-v1.3.0" and select the AMIs for both variants ("constellation-v1.3.0-aws-sev-snp" and "constellation-v1.3.0-aws-nitro-tpm")
3. On the "Actions" button (top right) select "Deregister AMI"
4. Either follow the link on the deletion confirmation leading you to the Snapshots panel or navigate there yourself
5. Search for a snapshot by the same name "constellation-v1.3.0" and select it
6. On the "Actions" button (top right) select "Delete snapshot"