mirror of
https://github.com/edgelesssys/constellation.git
synced 2025-05-02 06:16:08 -04:00
docs: add release v2.16.0
This commit is contained in:
malt3
Malte Poll
parent
93eb8f0694
commit
c4f27f62ee
72 changed files with 8740 additions and 0 deletions
|
|
@ -0,0 +1,102 @@
|
|||
# Application benchmarks
|
||||
|
||||
## HashiCorp Vault
|
||||
|
||||
[HashiCorp Vault](https://www.vaultproject.io/) is a distributed secrets management software that can be deployed to Kubernetes.
|
||||
HashiCorp maintains a benchmarking tool for vault, [vault-benchmark](https://github.com/hashicorp/vault-benchmark/).
|
||||
Vault-benchmark generates load on a Vault deployment and measures response times.
|
||||
|
||||
This article describes the results from running vault-benchmark on Constellation, AKS, and GKE.
|
||||
You can find the setup for producing the data discussed in this article in the [vault-benchmarks](https://github.com/edgelesssys/vault-benchmarks) repository.
|
||||
|
||||
The Vault API used during benchmarking is the [transits secret engine](https://developer.hashicorp.com/vault/docs/secrets/transit).
|
||||
This allows services to send data to Vault for encryption, decryption, signing, and verification.
|
||||
|
||||
## Results
|
||||
|
||||
On each run, vault-benchmark sends requests and measures the latencies.
|
||||
The measured latencies are aggregated through various statistical features.
|
||||
After running the benchmark n times, the arithmetic mean over a subset of the reported statistics is calculated.
|
||||
The selected features are arithmetic mean, 99th percentile, minimum, and maximum.
|
||||
|
||||
Arithmetic mean gives a general sense of the latency on each target.
|
||||
The 99th percentile shows performance in (most likely) erroneous states.
|
||||
Minimum and maximum mark the range within which latency varies each run.
|
||||
|
||||
The benchmark was configured with 1300 workers and 10 seconds per run.
|
||||
Those numbers were chosen empirically.
|
||||
The latency was stabilizing at 10 seconds runtime, not changing with further increase.
|
||||
Increasing the number of workers beyond 1300 leads to request failures, marking the limit Vault was able to handle in this setup.
|
||||
All results are based on 100 runs.
|
||||
|
||||
The following data was generated while running five replicas, one primary, and four standby nodes.
|
||||
All numbers are in seconds if not indicated otherwise.
|
||||
```
|
||||
========== Results AKS ==========
|
||||
Mean: mean: 1.632200, variance: 0.002057
|
||||
P99: mean: 5.480679, variance: 2.263700
|
||||
Max: mean: 6.651001, variance: 2.808401
|
||||
Min: mean: 0.011415, variance: 0.000133
|
||||
========== Results GKE ==========
|
||||
Mean: mean: 1.656435, variance: 0.003615
|
||||
P99: mean: 6.030807, variance: 3.955051
|
||||
Max: mean: 7.164843, variance: 3.300004
|
||||
Min: mean: 0.010233, variance: 0.000111
|
||||
========== Results C11n ==========
|
||||
Mean: mean: 1.651549, variance: 0.001610
|
||||
P99: mean: 5.780422, variance: 3.016106
|
||||
Max: mean: 6.942997, variance: 3.075796
|
||||
Min: mean: 0.013774, variance: 0.000228
|
||||
========== AKS vs C11n ==========
|
||||
Mean: +1.171577 % (AKS is faster)
|
||||
P99: +5.185495 % (AKS is faster)
|
||||
Max: +4.205618 % (AKS is faster)
|
||||
Min: +17.128781 % (AKS is faster)
|
||||
========== GKE vs C11n ==========
|
||||
Mean: -0.295851 % (GKE is slower)
|
||||
P99: -4.331603 % (GKE is slower)
|
||||
Max: -3.195248 % (GKE is slower)
|
||||
Min: +25.710886 % (GKE is faster)
|
||||
```
|
||||
|
||||
**Interpretation**: Latencies are all within ~5% of each other.
|
||||
AKS performs slightly better than GKE and Constellation (C11n) in all cases except minimum latency.
|
||||
Minimum latency is the lowest for GKE.
|
||||
Compared to GKE, Constellation had slightly lower peak latencies (99th percentile and maximum), indicating that Constellation could have handled slightly more concurrent accesses than GKE.
|
||||
Overall, performance is at comparable levels across all three distributions.
|
||||
Based on these numbers, you can use a similarly sized Constellation cluster to run your existing Vault deployment.
|
||||
|
||||
### Visualization
|
||||
|
||||
The following plots visualize the data presented above as [box plots](https://en.wikipedia.org/wiki/Box_plot).
|
||||
The whiskers denote the minimum and maximum.
|
||||
The box stretches from the 25th to the 75th percentile, with the dividing bar marking the 50th percentile.
|
||||
The circles outside the whiskers denote outliers.
|
||||
|
||||
<details>
|
||||
<summary>Mean Latency</summary>
|
||||
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>99th Percentile Latency</summary>
|
||||
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>Maximum Latency</summary>
|
||||
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>Minimum Latency</summary>
|
||||
|
||||
|
||||
|
||||
</details>
|
||||
204
docs/versioned_docs/version-2.16/overview/performance/io.md
Normal file
204
docs/versioned_docs/version-2.16/overview/performance/io.md
Normal file
|
|
@ -0,0 +1,204 @@
|
|||
# I/O performance benchmarks
|
||||
|
||||
To assess the overall performance of Constellation, this benchmark evaluates Constellation v2.6.0 in terms of storage I/O using [`fio`](https://fio.readthedocs.io/en/latest/fio_doc.html) and network performance using the [Kubernetes Network Benchmark](https://github.com/InfraBuilder/k8s-bench-suite#knb--kubernetes-network-be).
|
||||
|
||||
This benchmark tested Constellation on Azure and GCP and compared the results against the managed Kubernetes offerings AKS and GKE.
|
||||
|
||||
## Configurations
|
||||
|
||||
### Constellation
|
||||
|
||||
The benchmark was conducted with Constellation v2.6.0, Kubernetes v1.25.7, and Cilium v1.12.
|
||||
It ran on the following infrastructure configurations.
|
||||
|
||||
Constellation on Azure:
|
||||
|
||||
- Nodes: 3 (1 Control-plane, 2 Worker)
|
||||
- Machines: `DC4as_v5`: 3rd Generation AMD EPYC 7763v (Milan) processor with 4 Cores, 16 GiB memory
|
||||
- CVM: `true`
|
||||
- Region: `West US`
|
||||
- Zone: `2`
|
||||
|
||||
Constellation on GCP:
|
||||
|
||||
- Nodes: 3 (1 Control-plane, 2 Worker)
|
||||
- Machines: `n2d-standard-4`: 2nd Generation AMD EPYC (Rome) processor with 4 Cores, 16 GiB of memory
|
||||
- CVM: `true`
|
||||
- Zone: `europe-west3-b`
|
||||
|
||||
### AKS
|
||||
|
||||
On AKS, the benchmark used Kubernetes `v1.24.9` and nodes with version `AKSUbuntu-1804gen2containerd-2023.02.15`.
|
||||
AKS ran with the [`kubenet`](https://learn.microsoft.com/en-us/azure/aks/concepts-network#kubenet-basic-networking) CNI and the [default CSI driver](https://learn.microsoft.com/en-us/azure/aks/azure-disk-csi) for Azure Disk.
|
||||
|
||||
The following infrastructure configurations was used:
|
||||
|
||||
- Nodes: 2 (2 Worker)
|
||||
- Machines: `D4as_v5`: 3rd Generation AMD EPYC 7763v (Milan) processor with 4 Cores, 16 GiB memory
|
||||
- CVM: `false`
|
||||
- Region: `West US`
|
||||
- Zone: `2`
|
||||
|
||||
### GKE
|
||||
|
||||
On GKE, the benchmark used Kubernetes `v1.24.9` and nodes with version `1.24.9-gke.3200`.
|
||||
GKE ran with the [`kubenet`](https://cloud.google.com/kubernetes-engine/docs/concepts/network-overview) CNI and the [default CSI driver](https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/gce-pd-csi-driver) for Compute Engine persistent disk.
|
||||
|
||||
The following infrastructure configurations was used:
|
||||
|
||||
- Nodes: 2 (2 Worker)
|
||||
- Machines: `n2d-standard-4` 2nd Generation AMD EPYC (Rome) processor with 4 Cores, 16 GiB of memory
|
||||
- CVM: `false`
|
||||
- Zone: `europe-west3-b`
|
||||
|
||||
## Results
|
||||
|
||||
### Network
|
||||
|
||||
This section gives a thorough analysis of the network performance of Constellation, specifically focusing on measuring TCP and UDP bandwidth.
|
||||
The benchmark measured the bandwidth of pod-to-pod and pod-to-service connections between two different nodes using [`iperf`](https://iperf.fr/).
|
||||
|
||||
GKE and Constellation on GCP had a maximum network bandwidth of [10 Gbps](https://cloud.google.com/compute/docs/general-purpose-machines#n2d_machines).
|
||||
AKS with `Standard_D4as_v5` machines a maximum network bandwidth of [12.5 Gbps](https://learn.microsoft.com/en-us/azure/virtual-machines/dasv5-dadsv5-series#dasv5-series).
|
||||
The Confidential VM equivalent `Standard_DC4as_v5` currently has a network bandwidth of [1.25 Gbps](https://learn.microsoft.com/en-us/azure/virtual-machines/dcasv5-dcadsv5-series#dcasv5-series-products).
|
||||
Therefore, to make the test comparable, both AKS and Constellation on Azure were running with `Standard_DC4as_v5` machines and 1.25 Gbps bandwidth.
|
||||
|
||||
Constellation on Azure and AKS used an MTU of 1500.
|
||||
Constellation on GCP used an MTU of 8896. GKE used an MTU of 1450.
|
||||
|
||||
The difference in network bandwidth can largely be attributed to two factors.
|
||||
|
||||
- Constellation's [network encryption](../../architecture/networking.md) via Cilium and WireGuard, which protects data in-transit.
|
||||
- [AMD SEV using SWIOTLB bounce buffers](https://lore.kernel.org/all/20200204193500.GA15564@ashkalra_ubuntu_server/T/) for all DMA including network I/O.
|
||||
|
||||
#### Pod-to-Pod
|
||||
|
||||
In this scenario, the client Pod connects directly to the server pod via its IP address.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Node A
|
||||
Client[Client]
|
||||
end
|
||||
subgraph Node B
|
||||
Server[Server]
|
||||
end
|
||||
Client ==>|traffic| Server
|
||||
```
|
||||
|
||||
The results for "Pod-to-Pod" on Azure are as follows:
|
||||
|
||||
|
||||
|
||||
The results for "Pod-to-Pod" on GCP are as follows:
|
||||
|
||||
|
||||
|
||||
#### Pod-to-Service
|
||||
|
||||
In this scenario, the client Pod connects to the server Pod via a ClusterIP service. This is more relevant to real-world use cases.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
subgraph Node A
|
||||
Client[Client] ==>|traffic| Service[Service]
|
||||
end
|
||||
subgraph Node B
|
||||
Server[Server]
|
||||
end
|
||||
Service ==>|traffic| Server
|
||||
```
|
||||
|
||||
The results for "Pod-to-Pod" on Azure are as follows:
|
||||
|
||||
|
||||
|
||||
The results for "Pod-to-Pod" on GCP are as follows:
|
||||
|
||||
|
||||
|
||||
In our recent comparison of Constellation on GCP with GKE, Constellation has 58% less TCP bandwidth. However, UDP bandwidth was slightly better with Constellation, thanks to its higher MTU.
|
||||
|
||||
Similarly, when comparing Constellation on Azure with AKS using CVMs, Constellation achieved approximately 10% less TCP and 40% less UDP bandwidth.
|
||||
|
||||
### Storage I/O
|
||||
|
||||
Azure and GCP offer persistent storage for their Kubernetes services AKS and GKE via the Container Storage Interface (CSI). CSI storage in Kubernetes is available via `PersistentVolumes` (PV) and consumed via `PersistentVolumeClaims` (PVC).
|
||||
Upon requesting persistent storage through a PVC, GKE and AKS will provision a PV as defined by a default [storage class](https://kubernetes.io/docs/concepts/storage/storage-classes/).
|
||||
Constellation provides persistent storage on Azure and GCP [that's encrypted on the CSI layer](../../architecture/encrypted-storage.md).
|
||||
Similarly, upon a PVC request, Constellation will provision a PV via a default storage class.
|
||||
|
||||
For Constellation on Azure and AKS, the benchmark ran with Azure Disk storage [Standard SSD](https://learn.microsoft.com/en-us/azure/virtual-machines/disks-types#standard-ssds) of 400 GiB size.
|
||||
The [DC4as machine type](https://learn.microsoft.com/en-us/azure/virtual-machines/dasv5-dadsv5-series#dasv5-series) with four cores provides the following maximum performance:
|
||||
|
||||
- 6400 (20000 burst) IOPS
|
||||
- 144 MB/s (600 MB/s burst) throughput
|
||||
|
||||
However, the performance is bound by the capabilities of the [512 GiB Standard SSD size](https://learn.microsoft.com/en-us/azure/virtual-machines/disks-types#standard-ssds) (the size class of 400 GiB volumes):
|
||||
|
||||
- 500 (600 burst) IOPS
|
||||
- 60 MB/s (150 MB/s burst) throughput
|
||||
|
||||
For Constellation on GCP and GKE, the benchmark ran with Compute Engine Persistent Disk Storage [pd-balanced](https://cloud.google.com/compute/docs/disks) of 400 GiB size.
|
||||
The N2D machine type with four cores and pd-balanced provides the following [maximum performance](https://cloud.google.com/compute/docs/disks/performance#n2d_vms):
|
||||
|
||||
- 3,000 read IOPS
|
||||
- 15,000 write IOPS
|
||||
- 240 MB/s read throughput
|
||||
- 240 MB/s write throughput
|
||||
|
||||
However, the performance is bound by the capabilities of a [`Zonal balanced PD`](https://cloud.google.com/compute/docs/disks/performance#zonal-persistent-disks) with 400 GiB size:
|
||||
|
||||
- 2400 read IOPS
|
||||
- 2400 write IOPS
|
||||
- 112 MB/s read throughput
|
||||
- 112 MB/s write throughput
|
||||
|
||||
The [`fio`](https://fio.readthedocs.io/en/latest/fio_doc.html) benchmark consists of several tests.
|
||||
The benchmark used [`Kubestr`](https://github.com/kastenhq/kubestr) to run `fio` in Kubernetes.
|
||||
The default test performs randomized access patterns that accurately depict worst-case I/O scenarios for most applications.
|
||||
|
||||
The following `fio` settings were used:
|
||||
|
||||
- No Cloud caching
|
||||
- No OS caching
|
||||
- Single CPU
|
||||
- 60 seconds runtime
|
||||
- 10 seconds ramp-up time
|
||||
- 10 GiB file
|
||||
- IOPS: 4 KB blocks and 128 iodepth
|
||||
- Bandwidth: 1024 KB blocks and 128 iodepth
|
||||
|
||||
For more details, see the [`fio` test configuration](https://github.com/edgelesssys/constellation/blob/main/.github/actions/e2e_benchmark/fio.ini).
|
||||
|
||||
The results for IOPS on Azure are as follows:
|
||||
|
||||
|
||||
|
||||
The results for IOPS on GCP are as follows:
|
||||
|
||||
|
||||
|
||||
The results for bandwidth on Azure are as follows:
|
||||
|
||||
|
||||
|
||||
The results for bandwidth on GCP are as follows:
|
||||
|
||||
|
||||
|
||||
On GCP, the results exceed the maximum performance guarantees of the chosen disk type. There are two possible explanations for this. The first is that there may be cloud caching in place that isn't configurable. Alternatively, the underlying provisioned disk size may be larger than what was requested, resulting in higher performance boundaries.
|
||||
|
||||
When comparing Constellation on GCP with GKE, Constellation has similar bandwidth but about 10% less IOPS performance. On Azure, Constellation has similar IOPS performance compared to AKS, where both likely hit the maximum storage performance. However, Constellation has approximately 15% less read and write bandwidth.
|
||||
|
||||
## Conclusion
|
||||
|
||||
Despite the added [security benefits](../security-benefits.md) that Constellation provides, it only incurs a slight performance overhead when compared to managed Kubernetes offerings such as AKS and GKE. In most compute benchmarks, Constellation is on par with it's alternatives.
|
||||
While it may be slightly slower in certain I/O scenarios due to network and storage encryption, there is ongoing work to reduce this overhead to single digits.
|
||||
|
||||
For instance, storage encryption only adds between 10% to 15% overhead in terms of bandwidth and IOPS.
|
||||
Meanwhile, the biggest performance impact that Constellation currently faces is network encryption, which can incur up to 58% overhead on a 10 Gbps network.
|
||||
However, the Cilium team has conducted [benchmarks with Cilium using WireGuard encryption](https://docs.cilium.io/en/latest/operations/performance/benchmark/#encryption-wireguard-ipsec) on a 100 Gbps network that yielded over 15 Gbps.
|
||||
We're confident that Constellation will provide a similar level of performance with an upcoming release.
|
||||
|
||||
Overall, Constellation strikes a great balance between security and performance, and we're continuously working to improve its performance capabilities while maintaining its high level of security.
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
# Performance analysis of Constellation
|
||||
|
||||
This section provides a comprehensive examination of the performance characteristics of Constellation, encompassing various aspects, including runtime encryption, I/O benchmarks, and real-world applications.
|
||||
|
||||
## Impact of runtime encryption on performance
|
||||
|
||||
All nodes in a Constellation cluster are executed inside Confidential VMs (CVMs). Consequently, the performance of Constellation is inherently linked to the performance of these CVMs.
|
||||
|
||||
### AMD and Azure benchmarking
|
||||
|
||||
AMD and Azure have collectively released a [performance benchmark](https://community.amd.com/t5/business/microsoft-azure-confidential-computing-powered-by-3rd-gen-epyc/ba-p/497796) for CVMs that utilize 3rd Gen AMD EPYC processors (Milan) with SEV-SNP. This benchmark, which included a variety of mostly compute-intensive tests such as SPEC CPU 2017 and CoreMark, demonstrated that CVMs experience only minor performance degradation (ranging from 2% to 8%) when compared to standard VMs. Such results are indicative of the performance that can be expected from compute-intensive workloads running with Constellation on Azure.
|
||||
|
||||
### AMD and Google benchmarking
|
||||
|
||||
Similarly, AMD and Google have jointly released a [performance benchmark](https://www.amd.com/system/files/documents/3rd-gen-epyc-gcp-c2d-conf-compute-perf-brief.pdf) for CVMs employing 3rd Gen AMD EPYC processors (Milan) with SEV-SNP. With high-performance computing workloads such as WRF, NAMD, Ansys CFS, and Ansys LS_DYNA, they observed analogous findings, with only minor performance degradation (between 2% and 4%) compared to standard VMs. These outcomes are reflective of the performance that can be expected for compute-intensive workloads running with Constellation on GCP.
|
||||
|
||||
## I/O performance benchmarks
|
||||
|
||||
We evaluated the [I/O performance](io.md) of Constellation, utilizing a collection of synthetic benchmarks targeting networking and storage.
|
||||
We further compared this performance to native managed Kubernetes offerings from various cloud providers, to better understand how Constellation stands in relation to standard practices.
|
||||
|
||||
## Application benchmarking
|
||||
|
||||
To gauge Constellation's applicability to well-known applications, we performed a [benchmark of HashiCorp Vault](application.md) running on Constellation.
|
||||
The results were then compared to deployments on the managed Kubernetes offerings from different cloud providers, providing a tangible perspective on Constellation's performance in actual deployment scenarios.
|
||||
Loading…
Add table
Add a link
Reference in a new issue