constellation/image
Otto Bittner 30f2b332b3
api: restructure api pkg (#1851)
* api: rename AttestationVersionRepo to Client
* api: move client into separate subpkg for
clearer import paths.
* api: rename configapi -> attestationconfig
* api: rename versionsapi -> versions
* api: rename sut to client
* api: split versionsapi client and make it public
* api: split versionapi fetcher and make it public
* config: move attestationversion type to config
* api: fix attestationconfig client test

Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>
2023-06-02 09:19:23 +02:00
..
measured-boot image: fix pcr 12 calculation (#1706) 2023-05-02 12:01:30 +02:00
mkosi.cache Move mkosi folder to old image folder location 2022-10-21 11:04:25 +02:00
mkosi.conf.d todo responsibilities and cleanup (#1837) 2023-06-01 12:33:06 +02:00
mkosi.repart image: upgrade mkosi to support repart (#1684) 2023-04-25 18:22:40 +02:00
mkosi.reposdir Add AWS nvme udev rules (#351) 2022-10-21 14:55:13 +02:00
mkosi.skeleton image: upgrade mkosi to support repart (#1684) 2023-04-25 18:22:40 +02:00
pki_prod CI: Add secure boot prod keys (#462) 2022-11-04 16:48:52 +01:00
pki_testing Move mkosi folder to old image folder location 2022-10-21 11:04:25 +02:00
secure-boot image: pin aws uefivars version and install new deps (#1345) 2023-03-06 13:29:15 +01:00
upload api: restructure api pkg (#1851) 2023-06-02 09:19:23 +02:00
.gitignore image: add intermediate "image" verb to upload tool 2023-05-25 15:01:15 +02:00
BUILD.bazel image: implement idempotent upload of os images 2023-05-05 12:06:44 +02:00
Makefile image: add aws_aws-sev-snp variant 2023-06-01 11:25:31 +02:00
mkosi.finalize Preinstall kubelet systemd unit in OS images (#365) 2022-10-25 16:36:03 +02:00
mkosi.postinst image: fix shell code format 2023-01-16 14:49:33 +01:00
mkosi.prepare image: setup debugd as a separate systemd unit 2023-01-25 09:58:56 +01:00
README.md image: implement idempotent upload of os images 2023-05-05 12:06:44 +02:00

Setup

  • Install mkosi (from git):

    cd /tmp/
    git clone https://github.com/systemd/mkosi
    cd mkosi
    git checkout d8b32fbf3077b612db0024276e73cec3c2c87577
    tools/generate-zipapp.sh
    cp builddir/mkosi /usr/local/bin/
    
  • Build systemd tooling (from git):

    Ubuntu and Fedora ship outdated versions of systemd tools, so you need to build them from source:

    # Ubuntu
    echo "deb-src http://archive.ubuntu.com/ubuntu/ $(lsb_release -cs) main restricted universe multiverse" | sudo tee -a /etc/apt/sources.list
    sudo apt-get update
    sudo apt-get build-dep systemd
    sudo apt-get install libfdisk-dev
    # Fedora
    sudo dnf builddep systemd
    
    git clone https://github.com/systemd/systemd --depth=1
    meson systemd/build systemd -Drepart=true -Defi=true -Dbootloader=true
    BINARIES=(
        bootctl
        systemctl
        systemd-analyze
        systemd-dissect
        systemd-nspawn
        systemd-repart
        ukify
    )
    ninja -C systemd/build ${BINARIES[@]}
    SYSTEMD_BIN=$(realpath systemd/build)
    echo installed systemd tools to "${SYSTEMD_BIN}"
    
  • Install tools:

    Ubuntu / Debian
    sudo apt-get update
    sudo apt-get install --assume-yes --no-install-recommends \
        bubblewrap \
        coreutils \
        curl \
        dnf \
        e2fsprogs \
        efitools \
        jq \
        mtools \
        ovmf \
        python3-pefile \
        python3-pyelftools \
        python3-setuptools \
        qemu-system-x86 \
        qemu-utils \
        rpm \
        sbsigntool \
        squashfs-tools \
        systemd-container \
        util-linux \
        virt-manager
    
    Fedora
    sudo dnf install -y \
        bubblewrap \
        edk2-ovmf \
        systemd-container \
        qemu \
        e2fsprogs \
        squashfs-tools \
        efitools \
        sbsigntools \
        coreutils \
        curl \
        jq \
        util-linux \
        virt-manager
    
  • Prepare secure boot PKI (see secure-boot/genkeys.sh)

Build

When building your first image, prepare the secure boot PKI (see secure-boot/genkeys.sh) for self-signed, locally built images.

After that, you can build the image with:

# export SYSTEMD_BIN=<path to systemd tools>
# OPTIONAL: to create a debug image, export the following line
# export DEBUG=true
# OPTIONAL: to enable the serial console, export the following line
# export AUTOLOGIN=true
# OPTIONAL: symlink custom path to secure boot PKI to ./pki
# ln -s /path/to/pki/folder ./pki
sudo make EXTRA_SEARCH_PATHS="${SYSTEMD_BIN}" -j $(nproc)

Raw images will be placed in mkosi.output.<CSP>/fedora~37/image.raw.

Prepare Secure Boot

The generated images are partially signed by Microsoft (shim loader), and partially signed by Edgeless Systems (systemd-boot and unified kernel images consisting of the linux kernel, initramfs and kernel commandline).

For QEMU and Azure, you can pre-generate the NVRAM variables for secure boot. This is not necessary for GCP, as you can specify secure boot parameters via the GCP API on image creation.

libvirt / QEMU / KVM
secure-boot/generate_nvram_vars.sh mkosi.output.qemu/fedora~37/image.raw
Azure

These steps only have to performed once for a fresh set of secure boot certificates. VMGS blobs for testing and release images already exist.

First, create a disk without embedded MOK EFI variables.

# set these variables
export AZURE_SECURITY_TYPE=ConfidentialVM # or TrustedLaunch
export AZURE_RESOURCE_GROUP_NAME= # e.g. "constellation-images"

export AZURE_REGION=northeurope
export AZURE_REPLICATION_REGIONS=
export AZURE_DISK_NAME=constellation-$(date +%s)
export AZURE_SNAPSHOT_NAME=${AZURE_DISK_NAME}
export AZURE_RAW_IMAGE_PATH=${PWD}/mkosi.output.azure/fedora~37/image.raw
export AZURE_IMAGE_PATH=${PWD}/mkosi.output.azure/fedora~37/image.vhd
export AZURE_VMGS_FILENAME=${AZURE_SECURITY_TYPE}.vmgs
export AZURE_JSON_OUTPUT=${PWD}/mkosi.output.azure/fedora~37/image-upload.json
export BLOBS_DIR=${PWD}/blobs
upload/pack.sh azure "${AZURE_RAW_IMAGE_PATH}" "${AZURE_IMAGE_PATH}"
upload/upload_azure.sh --disk-name "${AZURE_DISK_NAME}-setup-secure-boot" ""
secure-boot/azure/launch.sh -n "${AZURE_DISK_NAME}-setup-secure-boot" -d --secure-boot true --disk-name "${AZURE_DISK_NAME}-setup-secure-boot"

Ignore the running launch script and connect to the serial console once available. The console shows the message "Verification failed: (0x1A) Security Violation". You can import the MOK certificate via the UEFI shell:

Press OK, then ENTER, then "Enroll key from disk". Select the following key: /EFI/loader/keys/auto/db.cer. Press Continue, then choose "Yes" to the question "Enroll the key(s)?". Choose reboot.

Extract the VMGS from the running VM (this includes the MOK EFI variables) and delete the VM:

secure-boot/azure/extract_vmgs.sh --name "${AZURE_DISK_NAME}-setup-secure-boot"
secure-boot/azure/delete.sh --name "${AZURE_DISK_NAME}-setup-secure-boot"

Upload to CSP

Warning! Never set --version to a value that is already used for a release image.

AWS
  • Install aws cli (see here)
  • Login to AWS (see here)
  • Choose secure boot PKI public keys (one of pki_dev, pki_test, pki_prod)
    • pki_dev can be used for local image builds
    • pki_test is used by the CI for non-release images
    • pki_prod is used for release images
# Warning! Never set `--version` to a value that is already used for a release image.
# Instead, use a `ref` that corresponds to your branch name.
bazel run //image/upload -- aws --verbose --raw-image mkosi.output.aws/fedora~37/image.raw --variant ""  --version ref/foo/stream/nightly/v2.7.0-pre-asdf
GCP
  • Install gcloud and gsutil (see here)
  • Login to GCP (see here)
  • Choose secure boot PKI public keys (one of pki_dev, pki_test, pki_prod)
    • pki_dev can be used for local image builds
    • pki_test is used by the CI for non-release images
    • pki_prod is used for release images
export GCP_RAW_IMAGE_PATH=${PWD}/mkosi.output.gcp/fedora~37/image.raw
export GCP_IMAGE_PATH=${PWD}/mkosi.output.gcp/fedora~37/image.tar.gz
upload/pack.sh gcp ${GCP_RAW_IMAGE_PATH} ${GCP_IMAGE_PATH}
# Warning! Never set `--version` to a value that is already used for a release image.
# Instead, use a `ref` that corresponds to your branch name.
bazel run //image/upload -- gcp --verbose --raw-image "${GCP_IMAGE_PATH}" --variant "sev-es"  --version ref/foo/stream/nightly/v2.7.0-pre-asdf
Azure

Note:

For testing purposes, it is a lot simpler to disable Secure Boot for the uploaded image! Disabling Secure Boot allows you to skip the VMGS creation steps above.

export AZURE_RAW_IMAGE_PATH=${PWD}/mkosi.output.azure/fedora~37/image.raw
export AZURE_IMAGE_PATH=${PWD}/mkosi.output.azure/fedora~37/image.vhd
upload/pack.sh azure "${AZURE_RAW_IMAGE_PATH}" "${AZURE_IMAGE_PATH}"
# Warning! Never set `--version` to a value that is already used for a release image.
# Instead, use a `ref` that corresponds to your branch name.
bazel run //image/upload -- azure --verbose --raw-image "${AZURE_IMAGE_PATH}" --variant "cvm"  --version ref/foo/stream/nightly/v2.7.0-pre-asdf
OpenStack

Note:

OpenStack is not one a global cloud provider, but rather a software that can be installed on-premises. This means we do not upload the image to a cloud provider, but to our CDN.

  • Install aws cli (see here)
  • Login to AWS (see here)
# Warning! Never set `--version` to a value that is already used for a release image.
# Instead, use a `ref` that corresponds to your branch name.
bazel run //image/upload -- openstack --verbose --raw-image mkosi.output.openstack/fedora~37/image.raw --variant "sev"  --version ref/foo/stream/nightly/v2.7.0-pre-asdf
QEMU
  • Install aws cli (see here)
  • Login to AWS (see here)
# Warning! Never set `--version` to a value that is already used for a release image.
# Instead, use a `ref` that corresponds to your branch name.
bazel run //image/upload -- qemu --verbose --raw-image mkosi.output.qemu/fedora~37/image.raw --variant "default"  --version ref/foo/stream/nightly/v2.7.0-pre-asdf