constellation/docs/versioned_docs/version-2.16/getting-started/first-steps-local.md

First steps with a local cluster

A local cluster lets you deploy and test Constellation without a cloud subscription. You have two options:

• Use MiniConstellation to automatically deploy a two-node cluster.
• For more fine-grained control, create the cluster using the QEMU provider.

Both options use virtualization to create a local cluster with control-plane nodes and worker nodes. They don't require hardware with Confidential VM (CVM) support. For attestation, they currently use a software-based vTPM provided by KVM/QEMU.

You need an x64 machine with a Linux OS. You can use a VM, but it needs nested virtualization.

Prerequisites

• Machine requirements:
  • An x86-64 CPU with at least 4 cores (6 cores are recommended)
  • At least 4 GB RAM (6 GB are recommended)
  • 20 GB of free disk space
  • Hardware virtualization enabled in the BIOS/UEFI (often referred to as Intel VT-x or AMD-V/SVM) / nested-virtualization support when using a VM
• Software requirements:
  • Linux OS with KVM kernel module
    • Recommended: Ubuntu 22.04 LTS
  • Docker
  • xsltproc
  • (Optional) virsh to observe and access your nodes

Software installation on Ubuntu

# install Docker
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install docker-ce
# install other dependencies
sudo apt install xsltproc
sudo snap install kubectl --classic
# install Constellation CLI
curl -LO https://github.com/edgelesssys/constellation/releases/latest/download/constellation-linux-amd64
sudo install constellation-linux-amd64 /usr/local/bin/constellation
# do not drop forwarded packages
sudo iptables -P FORWARD ACCEPT

Create a cluster

With the constellation mini command, you can deploy and test Constellation locally. This mode is called MiniConstellation. Conceptually, MiniConstellation is similar to MicroK8s, K3s, and minikube.

:::caution

MiniConstellation has specific soft- and hardware requirements such as a Linux OS running on an x86-64 CPU. Pay attention to all prerequisites when setting up.

:::

:::note

Since MiniConstellation runs on your local system, cloud features such as load balancing, attaching persistent storage, or autoscaling aren't available.

:::

The following creates your MiniConstellation cluster (may take up to 10 minutes to complete):

constellation mini up

This will configure your current directory as the workspace for this cluster. All constellation commands concerning this cluster need to be issued from this directory.

With the QEMU provider, you can create a local Constellation cluster as if it were in the cloud. The provider uses QEMU to create multiple VMs for the cluster nodes, which interact with each other.

:::caution

Constellation on QEMU has specific soft- and hardware requirements such as a Linux OS running on an x86-64 CPU. Pay attention to all prerequisites when setting up.

:::

:::note

Since Constellation on QEMU runs on your local system, cloud features such as load balancing, attaching persistent storage, or autoscaling aren't available.

:::

1. To set up your local cluster, you need to create a configuration file for Constellation first.

constellation config generate qemu

This creates a configuration file for QEMU called constellation-conf.yaml. After that, your current folder also becomes your workspace. All constellation commands for your cluster need to be executed from this directory.

2. Now you can create your cluster and its nodes. constellation apply uses the options set in constellation-conf.yaml.

constellation apply -y

The Output should look like the following:

$ constellation apply -y
Checking for infrastructure changes
The following Constellation cluster will be created:
  3 control-plane nodes of type 2-vCPUs will be created.
  1 worker node of type 2-vCPUs will be created.
Creating
Cloud infrastructure created successfully.
Your Constellation master secret was successfully written to ./constellation-mastersecret.json
Connecting
Initializing cluster
Installing Kubernetes components
Your Constellation cluster was successfully initialized.

Constellation cluster identifier  g6iMP5wRU1b7mpOz2WEISlIYSfdAhB0oNaOg6XEwKFY=
Kubernetes configuration          constellation-admin.conf

You can now connect to your cluster by executing:
        export KUBECONFIG="$PWD/constellation-admin.conf"

The cluster's identifier will be different in your output. Keep constellation-mastersecret.json somewhere safe. This will allow you to recover your cluster in case of a disaster.

:::info

Depending on your setup, constellation apply may take 10+ minutes to complete.

:::

3. Configure kubectl

export KUBECONFIG="$PWD/constellation-admin.conf"

Connect to the cluster

Your cluster initially consists of a single control-plane node:

$ kubectl get nodes
NAME              STATUS   ROLES           AGE   VERSION
control-plane-0   Ready    control-plane   66s   v1.24.6

Additional nodes will request to join the cluster shortly. Before each additional node is allowed to join the cluster, its state is verified using remote attestation by the JoinService. If verification passes successfully, the new node receives keys and certificates to join the cluster.

You can follow this process by viewing the logs of the JoinService:

$ kubectl logs -n kube-system daemonsets/join-service -f
{"level":"INFO","ts":"2022-10-14T09:32:20Z","caller":"cmd/main.go:48","msg":"Constellation Node Join Service","version":"2.1.0","cloudProvider":"qemu"}
{"level":"INFO","ts":"2022-10-14T09:32:20Z","logger":"validator","caller":"watcher/validator.go:96","msg":"Updating expected measurements"}
...

Once all nodes have joined your cluster, it may take a couple of minutes for all resources to become available. You can check on the state of your cluster by running the following:

$ kubectl get nodes
NAME              STATUS   ROLES           AGE     VERSION
control-plane-0   Ready    control-plane   2m59s   v1.24.6
worker-0          Ready    <none>          32s     v1.24.6

Deploy a sample application

1. Deploy the emojivoto app

kubectl apply -k github.com/BuoyantIO/emojivoto/kustomize/deployment

2. Expose the frontend service locally

kubectl wait --for=condition=available --timeout=60s -n emojivoto --all deployments
kubectl -n emojivoto port-forward svc/web-svc 8080:80 &
curl http://localhost:8080
kill %1

Terminate your cluster

Once you are done, you can clean up the created resources using the following command:

constellation mini down

This will destroy your cluster and clean up your workspace. The VM image and cluster configuration file (constellation-conf.yaml) will be kept and may be reused to create new clusters.

Once you are done, you can clean up the created resources using the following command:

constellation terminate

This should give the following output:

$ constellation terminate
You are about to terminate a Constellation cluster.
All of its associated resources will be DESTROYED.
This action is irreversible and ALL DATA WILL BE LOST.
Do you want to continue? [y/n]:

Confirm with y to terminate the cluster:

Terminating ...
Your Constellation cluster was terminated successfully.

This will destroy your cluster and clean up your workspace. The VM image and cluster configuration file (constellation-conf.yaml) will be kept and may be reused to create new clusters.

Troubleshooting

Make sure to use the latest release and check out the known issues.

VMs have no internet access / CLI remains in "Initializing cluster" state

iptables rules may prevent your VMs from accessing the internet. Make sure your rules aren't dropping forwarded packages.

List your rules:

sudo iptables -S

The output may look similar to the following:

-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER

If your FORWARD chain is set to DROP, you need to update your rules:

sudo iptables -P FORWARD ACCEPT