Co-authored-by: elchead <elchead@users.noreply.github.com>
4.1 KiB
Horizontal Pod Autoscaling
This example demonstrates Constellation's autoscaling capabilities. It's based on the Kubernetes HorizontalPodAutoscaler Walkthrough. During the following steps, Constellation will spawn new VMs on demand, verify them, add them to the cluster, and delete them again when the load has settled down.
Requirements
The cluster needs to be initialized with Kubernetes 1.23 or later. In addition, autoscaling must be enabled to enable Constellation to assign new nodes dynamically.
Just for this example specifically, the cluster should have as few worker nodes in the beginning as possible. Start with a small cluster with only one low-powered node for the control-plane node and one low-powered worker node.
:::info
We tested the example using instances of types Standard_DC4as_v5
on Azure and n2d-standard-4
on GCP.
:::
Setup
-
Install the Kubernetes Metrics Server:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
-
Deploy the HPA example server that's supposed to be scaled under load.
This manifest is similar to the one from the Kubernetes HPA walkthrough, but with increased CPU limits and requests to facilitate the triggering of node scaling events.
cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: php-apache spec: selector: matchLabels: run: php-apache replicas: 1 template: metadata: labels: run: php-apache spec: containers: - name: php-apache image: registry.k8s.io/hpa-example ports: - containerPort: 80 resources: limits: cpu: 900m requests: cpu: 600m --- apiVersion: v1 kind: Service metadata: name: php-apache labels: run: php-apache spec: ports: - port: 80 selector: run: php-apache EOF
-
Create a HorizontalPodAutoscaler.
Set an average CPU usage across all Pods of 20% to see one additional worker node being created later. Note that the CPU usage isn't related to the host CPU usage, but rather to the requested CPU capacities (20% of 600 milli-cores CPU across all Pods). See the original tutorial for more information on the HPA configuration.
kubectl autoscale deployment php-apache --cpu-percent=20 --min=1 --max=10
-
Create a Pod that generates load on the server:
kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- /bin/sh -c "while true; do wget -q -O- http://php-apache; done"
-
Wait a few minutes until new nodes are added to the cluster. You can monitor the state of the HorizontalPodAutoscaler, the list of nodes, and the behavior of the autoscaler.
-
To stop the load generator, press CTRL+C and run:
kubectl delete pod load-generator
-
The cluster-autoscaler checks every few minutes if nodes are underutilized. It will taint such candidates for removal and wait additional 10 minutes before the nodes are eventually removed and deallocated. The whole process can take about 20 minutes.
Monitoring
:::tip
For better observability, run the listed commands in different tabs in your terminal.
:::
You can watch the status of the HorizontalPodAutoscaler with the current CPU usage, the target CPU limit, and the number of replicas created with:
kubectl get hpa php-apache --watch
From time to time compare the list of nodes to check the behavior of the autoscaler:
kubectl get nodes
For deeper insights, see the logs of the autoscaler Pod, which contains more details about the scaling decision process:
kubectl logs -f deployment/constellation-cluster-autoscaler -n kube-system