Use terraform in CLI to create QEMU cluster (#172)

* Use terraform in CLI to create QEMU cluster

* Dont allow qemu creation on os/arch other than linux/amd64

* Allow usage of --name flag for QEMU resources

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
This commit is contained in:
Daniel Weiße 2022-09-26 15:52:31 +02:00 committed by GitHub
parent 2b32b79026
commit 804c173d52
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
41 changed files with 1066 additions and 182 deletions

View File

@ -1,4 +1,5 @@
# Process conventions
## Pull request process
Submissions should remain focused in scope and avoid containing unrelated commits.
@ -35,6 +36,7 @@ You should discuss larger changes and feature requests with the maintainers. Ple
[Run CI e2e tests](/.github/docs/README.md)
# Code conventions
## General
Adhere to the style and best practices described in [Effective Go](https://golang.org/doc/effective_go.html). Read [Common Review Comments](https://github.com/golang/go/wiki/CodeReviewComments) for further information.

View File

@ -1,10 +1,12 @@
# Build
The following are instructions for building all components in the constellation repository, except for images. A manual on how to build images locally can be found in the [image README](/image/README.md).
Prerequisites:
* 20 GB disk space
* [Go 1.18](https://go.dev/doc/install). Can be installed with these commands:
```sh
wget https://go.dev/dl/go1.18.linux-amd64.tar.gz && sudo rm -rf /usr/local/go && sudo tar -C /usr/local -xzf go1.18.linux-amd64.tar.gz && export PATH=$PATH:/usr/local/go/bin
```
@ -40,29 +42,35 @@ ctest -j `nproc`
You can limit the execution of tests to specific targets with e.g. `ctest -R unit-*` to only execute unit tests.
Some of the tests rely on libvirt and won't work if you don't have a virtualization capable CPU. You can find instructions on setting up libvirt in our [terraform README](/terraform/libvirt/README.md).
Some of the tests rely on libvirt and won't work if you don't have a virtualization capable CPU. You can find instructions on setting up libvirt in our [QEMU README](qemu.md).
# Deploy
> :warning: Debug images are not safe to use in production environments. :warning:
The debug images will open an additional **unsecured** port (4000) which accepts any binary to be run on the target machine. **Make sure that those machines are not exposed to the internet.**
## Cloud
To familiarize yourself with debugd and learn how to deploy a cluster using it, read [this](/debugd/README.md) manual.
If you want to deploy a cluster for production, please refer to our user documentation [here](https://docs.edgeless.systems/constellation/getting-started/first-steps#create-a-cluster).
## Locally
In case you want to have quicker iteration cycles during development you might want to setup a local cluster.
You can do this by utilizing our terraform setup.
Instructions on how to set it up can be found in it's [README](/terraform/libvirt/README.md).
You can do this by utilizing our QEMU setup.
Instructions on how to set it up can be found in the [QEMU README](qemu.md).
# Verification
In order to verify your cluster we describe a [verification workflow](https://docs.edgeless.systems/constellation/workflows/verify-cluster) in our official docs.
Apart from that you can also reproduce some of the measurements described in the [docs](https://docs.edgeless.systems/constellation/architecture/attestation#runtime-measurements) locally.
To do so we built a tool that creates a VM, collects the PCR values and reports them to you.
To run the tool execute the following command in `/hack/image-measurement`:
```
```sh
go run . -path <image_path> -type <image_type>
```
`<image_path>` needs to point to a valid image file.
The image can be either in raw or QEMU's `qcow2` format.
This format is specified in the `<image_type>` argument.
@ -74,21 +82,27 @@ Therefore, if you want to verify a cluster deployed with a release image you wil
After collecting the measurements you can put them into your `constellation-conf.yaml` under the `measurements` key in order to enforce them.
# Image export
To download an image you will have to export it first.
Below you find general instructions on how to do this for GCP and Azure.
You can find values for `<image_path>` in the `version_manifest.json` that is part of each constellation release.
## GCP
In order to download an image you will have to export it to a bucket you have access to:
- "Owner" permissions on the project
- "Storage Admin" permissions on the bucket
- Export with:
* "Owner" permissions on the project
* "Storage Admin" permissions on the bucket
* Export with:
```bash
gcloud compute images export --image=<image_path> --destination-uri=<bucket_uri> --export-format=qcow2 --project=<image_project>
```
- Click on "Download" on the created object
* Click on "Download" on the created object
## Azure
To download an image from Azure you will have to create a disk from the image and generate a download link for that disk:
```bash

View File

@ -17,8 +17,6 @@ Development components:
* [debugd](/debugd): Debug daemon and client
* [hack](/hack): Development tools
* [proto](/proto): Proto files generator
* [terraform](/terraform): Infrastructure management using terraform (instead of `constellation create/destroy`)
* [libvirt](/terraform/libvirt): Deploy local cluster using terraform, libvirt and QEMU
Additional repositories:

93
.github/docs/qemu.md vendored Normal file
View File

@ -0,0 +1,93 @@
# Local image testing with QEMU / libvirt
To create local testing clusters using QEMU, some prerequisites have to be met:
- [qcow2 constellation image](/image/README.md)
- [libvirt setup](#setup-libvirt)
- [qemu-metadata-api container image](/hack/qemu-metadata-api/README.md)
## Setup libvirt
<details>
<summary>Ubuntu</summary>
### Install required packages
[General reference](https://ubuntu.com/server/docs/virtualization-libvirt)
```shell-session
sudo apt install qemu-kvm libvirt-daemon-system xsltproc
sudo systemctl enable libvirtd
sudo usermod -a -G libvirt $USER
# reboot
```
### Setup emulated TPM
Using a virtual TPM (vTPM) with QEMU only works if swtpm is version 0.7 or newer!
Ubuntu 22.04 currently ships swtpm 0.6.3, so you need to install swtpm [from launchpad](https://launchpad.net/~stefanberger/+archive/ubuntu/swtpm-jammy/).
1. Uninstall current version of swtpm (if installed)
```shell-session
sudo apt remove swtpm swtpm-tools
```
2. Add ppa (this command shows the ppa for Ubuntu 22.04 jammy but others are available)
```shell-session
sudo add-apt-repository ppa:stefanberger/swtpm-jammy
sudo apt update
```
3. Install swtpm
```shell-session
sudo apt install swtpm swtpm-tools
```
4. Patch configuration under `/etc/swtpm_setup.conf`
```shell-session
# Program invoked for creating certificates
create_certs_tool = /usr/bin/swtpm_localca
```
5. Patch ownership of `/var/lib/swtpm-localca`
```shell-session
sudo chown -R swtpm:root /var/lib/swtpm-localca
```
</details>
<details>
<summary>Fedora</summary>
```shell-session
sudo dnf install -y dnf-plugins-core
sudo dnf -y install qemu-kvm libvirt-daemon-config-network libvirt-daemon-kvm xsltproc swtpm
sudo usermod -a -G libvirt $USER
# reboot
```
</details>
## Update libvirt settings
Open `/etc/libvirt/qemu.conf` and change the following settings:
```shell-session
security_driver = "none"
```
Then restart libvirt
```shell-session
sudo systemctl restart libvirtd
```
## Misc
- List all domains: `virsh list --all`
- Destroy domain with nvram: `virsh undefine --nvram <name>`

View File

@ -4,11 +4,13 @@ This checklist will prepare `v1.3.0` from `v1.2.0`. Adjust your version numbers
1. Merge ready PRs
2. Create docs release (new major or minor release)
```sh
cd docs
npm run docusaurus docs:version 1.3
# push upstream via PR
```
3. On the [CoreOS config repo](https://github.com/edgelesssys/constellation-fedora-coreos-config), create two new branches `release/v1.3`, `stream/v1.3` (new minor version) or use the existing ones (new patch version).
The release branch contains the squashed changeset and is branched from main while the stream branch contains the rebased changesets on top of the latest upstream changes.
[Consult this guide on rebasing forks (INTERNAL)](https://github.com/edgelesssys/wiki/blob/master/documentation/rebasing_forks.md#managing-release-branches) for guidance.
@ -19,6 +21,7 @@ This checklist will prepare `v1.3.0` from `v1.2.0`. Adjust your version numbers
* branch: `release/v1.3`
* Container image tag: `v1.3.0`
* Version of the image to build: `1.3.0`
```sh
# Alternative from CLI
gh workflow run build-micro-service-manual.yml --ref release/v1.3 -F microService=access-manager -F imageTag=v1.3.0 -F version=1.3.0
@ -26,41 +29,52 @@ This checklist will prepare `v1.3.0` from `v1.2.0`. Adjust your version numbers
gh workflow run build-micro-service-manual.yml --ref release/v1.3 -F microService=kmsserver -F imageTag=v1.3.0 -F version=1.3.0
gh workflow run build-micro-service-manual.yml --ref release/v1.3 -F microService=verification-service -F imageTag=v1.3.0 -F version=1.3.0
```
3. Use [Build operator manual](https://github.com/edgelesssys/constellation/actions/workflows/build-operator-manual.yml) and run the pipeline once with the following parameters:
* branch: `release/v1.3`
* Container image tag: `v1.3.0`
```sh
# Alternative from CLI
gh workflow run build-operator-manual.yml --ref release/v1.3 -F imageTag=v1.3.0
```
4. Review and update changelog with all changes since last release. [GitHub's diff view](https://github.com/edgelesssys/constellation/compare/v2.0.0...main) helps a lot!
1. Rename the "Unreleased" heading to "[v1.3.0] - YYYY-MM-DD" and link the version to the upcoming release tag.
2. Create a new block for unreleased changes
5. Update project version in [CMakeLists.txt](/CMakeLists.txt) to `1.3.0` (without v).
6. Update versions [versions.go](../../internal/versions/versions.go#L33-L39) to `v1.3.0` and **push your changes**.
7. Create a [production coreOS image](/.github/workflows/build-coreos.yml)
```sh
gh workflow run build-coreos.yml --ref release/v1.3 -F debug=false -F coreOSConfigBranch=release/v1.3 -F imageVersion=v1.3.0
```
8. Update [default images in config](/internal/config/images_enterprise.go)
9. Run manual E2E tests using [Linux](/.github/workflows/e2e-test-manual.yml) and [macOS](/.github/workflows/e2e-test-manual-macos.yml) to confirm functionality and stability.
```sh
gh workflow run e2e-test-manual.yml --ref release/v1.3 -F workerNodesCount=2 -F controlNodesCount=1 -F cloudProvider=azure -F machineType=Standard_DC4as_v5 -F sonobuoyTestSuiteCmd="--mode quick" -F kubernetesVersion=1.23 -F coreosImage=/CommunityGalleries/ConstellationCVM-b3782fa0-0df7-4f2f-963e-fc7fc42663df/Images/constellation/Versions/1.3.0 -F isDebugImage=false
gh workflow run e2e-test-manual-macos.yml --ref release/v1.3 -F workerNodesCount=2 -F controlNodesCount=1 -F cloudProvider=azure -F machineType=Standard_DC4as_v5 -F sonobuoyTestSuiteCmd="--mode quick" -F kubernetesVersion=1.23 -F coreosImage=/CommunityGalleries/ConstellationCVM-b3782fa0-0df7-4f2f-963e-fc7fc42663df/Images/constellation/Versions/1.3.0 -F isDebugImage=false
gh workflow run e2e-test-manual.yml --ref release/v1.3 -F workerNodesCount=2 -F controlNodesCount=1 -F cloudProvider=gcp -F machineType=n2d-standard-4 -F sonobuoyTestSuiteCmd="--mode quick" -F kubernetesVersion=1.23 -F coreosImage=projects/constellation-images/global/images/constellation-v1-3-0 -F isDebugImage=false
gh workflow run e2e-test-manual-macos.yml --ref release/v1.3 -F workerNodesCount=2 -F controlNodesCount=1 -F cloudProvider=gcp -F machineType=n2d-standard-4 -F sonobuoyTestSuiteCmd="--mode quick" -F kubernetesVersion=1.23 -F coreosImage=projects/constellation-images/global/images/constellation-v1-3-0 -F isDebugImage=false
```
10. [Generate measurements](/.github/workflows/generate-measurements.yml) for the images on each CSP.
```sh
gh workflow run generate-measurements.yml --ref release/v1.3 -F cloudProvider=azure -F coreosImage=/CommunityGalleries/ConstellationCVM-b3782fa0-0df7-4f2f-963e-fc7fc42663df/Images/constellation/Versions/1.3.0 -F isDebugImage=false
gh workflow run generate-measurements.yml --ref release/v1.3 -F cloudProvider=gcp -F coreosImage=projects/constellation-images/global/images/constellation-v1-3-0 -F isDebugImage=false
```
11. Create a new tag on this release branch
* `git tag v1.3.0`
* Run [Release CLI](https://github.com/edgelesssys/constellation/actions/workflows/release-cli.yml) action on the tag
```sh
gh workflow run release-cli.yml --ref v1.3.0
```
* The previous step will create a draft release. Check build output for link to draft release. Review & approve.
6. Follow [export flow (INTERNAL)](https://github.com/edgelesssys/wiki/blob/master/documentation/constellation/customer-onboarding.md#manual-export-and-import) to make image available in S3 for trusted launch users.
7. To bring updated version numbers and other changes (if any) to main, create a new branch `feat/release` from `release/v1.3`, rebase it onto main, and create a PR to main

View File

@ -6,7 +6,6 @@ Constellation is a Kubernetes distribution. As such, dependencies on Kubernetes
- Kubernetes resources (deployments made while initializing Kubernetes, including the `cloud-controller-manager`, `cluster-autoscaler` and more)
- Kubernetes go dependencies for the bootstrapper code
## Understand what has changed
Before adding support for a new Kubernetes version, it is a very good idea to [read the release notes](https://kubernetes.io/releases/notes/) and to identify breaking changes.
@ -17,7 +16,7 @@ Everything related to Kubernetes versions is tracked in [the versions file](/int
During cluster initialization, multiple Kubernetes resources are deployed. Some of these should be upgraded with Kubernetes.
You can check available version tags for container images using [the container registry tags API](https://docs.docker.com/registry/spec/api/#listing-image-tags):
```
```sh
curl -q https://k8s.gcr.io/v2/autoscaling/cluster-autoscaler/tags/list | jq .tags
curl -q https://k8s.gcr.io/v2/cloud-controller-manager/tags/list | jq .tags
curl -q https://us.gcr.io/v2/k8s-artifacts-prod/provider-aws/cloud-controller-manager/tags/list | jq .tags
@ -26,7 +25,6 @@ curl -q https://mcr.microsoft.com/v2/oss/kubernetes/azure-cloud-node-manager/tag
# [...]
```
## Upgrade go dependencies
The [`go.mod`](/go.mod) and [`go.sum`](/go.sum) files pin versions of the Kubernetes go packages. While these do not need to be on the exact versions used in the Kubernetes deployment, it is a good idea to keep them updated and on a similar version.
@ -37,7 +35,7 @@ See the diff of [this PR](https://github.com/edgelesssys/constellation/pull/110)
- Setup a Constellation cluster using the new image with the new bootstrapper binary and check if Kubernetes is deployed successfully.
```
```sh
# should print the new k8s version for every node
kubectl get nodes -o wide
# read the logs for pods deployed in the kube-system namespace and ensure they are healthy
@ -45,9 +43,10 @@ See the diff of [this PR](https://github.com/edgelesssys/constellation/pull/110)
kubectl -n kube-system logs [...]
kubectl -n kube-system describe pods
```
- Read the logs of the main Kubernetes components by getting a shell on the nodes and scan for errors / deprecation warnings:
```
```sh
journalctl -u kubelet
journalctl -u containerd
```

View File

@ -24,6 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Loadbalancer for control-plane recovery
- K8s conformance mode
- Local cluster creation based on QEMU
### Changed
<!-- For changes in existing functionality. -->

View File

@ -11,6 +11,7 @@ import (
azurecl "github.com/edgelesssys/constellation/v2/cli/internal/azure/client"
gcpcl "github.com/edgelesssys/constellation/v2/cli/internal/gcp/client"
"github.com/edgelesssys/constellation/v2/cli/internal/terraform"
"github.com/edgelesssys/constellation/v2/internal/state"
)
@ -38,3 +39,11 @@ type azureclient interface {
CreateInstances(ctx context.Context, input azurecl.CreateInstancesInput) error
TerminateResourceGroupResources(ctx context.Context) error
}
type qemuclient interface {
GetState() state.ConstellationState
CreateCluster(ctx context.Context, name string, input terraform.CreateClusterInput) error
DestroyCluster(ctx context.Context) error
CleanUpWorkspace() error
RemoveInstaller()
}

View File

@ -10,10 +10,12 @@ import (
"context"
"fmt"
"io"
"runtime"
azurecl "github.com/edgelesssys/constellation/v2/cli/internal/azure/client"
"github.com/edgelesssys/constellation/v2/cli/internal/gcp"
gcpcl "github.com/edgelesssys/constellation/v2/cli/internal/gcp/client"
"github.com/edgelesssys/constellation/v2/cli/internal/terraform"
"github.com/edgelesssys/constellation/v2/internal/cloud/cloudprovider"
"github.com/edgelesssys/constellation/v2/internal/cloud/cloudtypes"
"github.com/edgelesssys/constellation/v2/internal/config"
@ -26,6 +28,7 @@ type Creator struct {
out io.Writer
newGCPClient func(ctx context.Context, project, zone, region, name string) (gcpclient, error)
newAzureClient func(subscriptionID, tenantID, name, location, resourceGroup string) (azureclient, error)
newQEMUClient func(ctx context.Context) (qemuclient, error)
}
// NewCreator creates a new creator.
@ -38,6 +41,12 @@ func NewCreator(out io.Writer) *Creator {
newAzureClient: func(subscriptionID, tenantID, name, location, resourceGroup string) (azureclient, error) {
return azurecl.NewInitialized(subscriptionID, tenantID, name, location, resourceGroup)
},
newQEMUClient: func(ctx context.Context) (qemuclient, error) {
if runtime.GOARCH != "amd64" || runtime.GOOS != "linux" {
return nil, fmt.Errorf("creation of a QEMU based Constellation is not supported for %s/%s", runtime.GOOS, runtime.GOARCH)
}
return terraform.New(ctx, cloudprovider.QEMU)
},
}
}
@ -78,6 +87,13 @@ func (c *Creator) Create(ctx context.Context, provider cloudprovider.Provider, c
return state.ConstellationState{}, err
}
return c.createAzure(ctx, cl, config, insType, controlPlaneCount, workerCount, ingressRules)
case cloudprovider.QEMU:
cl, err := c.newQEMUClient(ctx)
if err != nil {
return state.ConstellationState{}, err
}
defer cl.RemoveInstaller()
return c.createQEMU(ctx, cl, name, config, controlPlaneCount, workerCount)
default:
return state.ConstellationState{}, fmt.Errorf("unsupported cloud provider: %s", provider)
}
@ -194,3 +210,28 @@ func (c *Creator) createAzure(ctx context.Context, cl azureclient, config *confi
return cl.GetState(), nil
}
func (c *Creator) createQEMU(ctx context.Context, cl qemuclient, name string, config *config.Config, controlPlaneCount, workerCount int,
) (stat state.ConstellationState, retErr error) {
defer rollbackOnError(context.Background(), c.out, &retErr, &rollbackerQEMU{client: cl})
input := terraform.CreateClusterInput{
CountControlPlanes: controlPlaneCount,
CountWorkers: workerCount,
QEMU: terraform.QEMUInput{
ImagePath: config.Provider.QEMU.Image,
ImageFormat: config.Provider.QEMU.ImageFormat,
CPUCount: config.Provider.QEMU.VCPUs,
MemorySizeMiB: config.Provider.QEMU.Memory,
StateDiskSizeGB: config.StateDiskSizeGB,
IPRangeStart: config.Provider.QEMU.IPRangeStart,
MetadataAPIImage: config.Provider.QEMU.MetadataAPIImage,
},
}
if err := cl.CreateCluster(ctx, name, input); err != nil {
return state.ConstellationState{}, err
}
return cl.GetState(), nil
}

View File

@ -28,7 +28,7 @@ func rollbackOnError(ctx context.Context, w io.Writer, onErr *error, roll rollba
fmt.Fprintf(w, "An error occurred: %s\n", *onErr)
fmt.Fprintln(w, "Attempting to roll back.")
if err := roll.rollback(ctx); err != nil {
*onErr = multierr.Append(*onErr, fmt.Errorf("on rollback: %w", err)) // TODO: print the error, or retrun it?
*onErr = multierr.Append(*onErr, fmt.Errorf("on rollback: %w", err)) // TODO: print the error, or return it?
return
}
fmt.Fprintln(w, "Rollback succeeded.")
@ -54,3 +54,14 @@ type rollbackerAzure struct {
func (r *rollbackerAzure) rollback(ctx context.Context) error {
return r.client.TerminateResourceGroupResources(ctx)
}
type rollbackerQEMU struct {
client qemuclient
}
func (r *rollbackerQEMU) rollback(ctx context.Context) error {
var err error
err = multierr.Append(err, r.client.DestroyCluster(ctx))
err = multierr.Append(err, r.client.CleanUpWorkspace())
return err
}

View File

@ -12,6 +12,7 @@ import (
azurecl "github.com/edgelesssys/constellation/v2/cli/internal/azure/client"
gcpcl "github.com/edgelesssys/constellation/v2/cli/internal/gcp/client"
"github.com/edgelesssys/constellation/v2/cli/internal/terraform"
"github.com/edgelesssys/constellation/v2/internal/cloud/cloudprovider"
"github.com/edgelesssys/constellation/v2/internal/state"
)
@ -20,6 +21,7 @@ import (
type Terminator struct {
newGCPClient func(ctx context.Context) (gcpclient, error)
newAzureClient func(subscriptionID, tenantID string) (azureclient, error)
newQEMUClient func(ctx context.Context) (qemuclient, error)
}
// NewTerminator create a new cloud terminator.
@ -31,6 +33,9 @@ func NewTerminator() *Terminator {
newAzureClient: func(subscriptionID, tenantID string) (azureclient, error) {
return azurecl.NewFromDefault(subscriptionID, tenantID)
},
newQEMUClient: func(ctx context.Context) (qemuclient, error) {
return terraform.New(ctx, cloudprovider.QEMU)
},
}
}
@ -51,6 +56,13 @@ func (t *Terminator) Terminate(ctx context.Context, state state.ConstellationSta
return err
}
return t.terminateAzure(ctx, cl, state)
case cloudprovider.QEMU:
cl, err := t.newQEMUClient(ctx)
if err != nil {
return err
}
defer cl.RemoveInstaller()
return t.terminateQEMU(ctx, cl)
default:
return fmt.Errorf("unsupported provider: %s", provider)
}
@ -80,3 +92,11 @@ func (t *Terminator) terminateAzure(ctx context.Context, cl azureclient, state s
return cl.TerminateResourceGroupResources(ctx)
}
func (t *Terminator) terminateQEMU(ctx context.Context, cl qemuclient) error {
if err := cl.DestroyCluster(ctx); err != nil {
return err
}
return cl.CleanUpWorkspace()
}

View File

@ -20,7 +20,7 @@ import (
func newConfigGenerateCmd() *cobra.Command {
cmd := &cobra.Command{
Use: "generate {aws|azure|gcp}",
Use: "generate {aws|azure|gcp|qemu}",
Short: "Generate a default configuration file",
Long: "Generate a default configuration file for your selected cloud provider.",
Args: cobra.MatchAll(
@ -55,6 +55,11 @@ func configGenerate(cmd *cobra.Command, fileHandler file.Handler, provider cloud
conf := config.Default()
conf.RemoveProviderExcept(provider)
// set a lower default for QEMU's state disk
if provider == cloudprovider.QEMU {
conf.StateDiskSizeGB = 10
}
if flags.file == "-" {
content, err := encoder.NewEncoder(conf).Encode()
if err != nil {
@ -90,7 +95,7 @@ func parseGenerateFlags(cmd *cobra.Command) (generateFlags, error) {
func generateCompletion(cmd *cobra.Command, args []string, toComplete string) ([]string, cobra.ShellCompDirective) {
switch len(args) {
case 0:
return []string{"aws", "gcp", "azure"}, cobra.ShellCompDirectiveNoFileComp
return []string{"aws", "gcp", "azure", "qemu"}, cobra.ShellCompDirectiveNoFileComp
default:
return []string{}, cobra.ShellCompDirectiveError
}

View File

@ -94,6 +94,9 @@ func create(cmd *cobra.Command, creator cloudCreator, fileHandler file.Handler)
instanceType = config.Provider.Azure.InstanceType
case cloudprovider.GCP:
instanceType = config.Provider.GCP.InstanceType
case cloudprovider.QEMU:
cpus := config.Provider.QEMU.VCPUs
instanceType = fmt.Sprintf("%d-vCPU", cpus)
}
if !flags.yes {

View File

@ -413,6 +413,7 @@ func TestAttestation(t *testing.T) {
cfg := config.Default()
cfg.RemoveProviderExcept(cloudprovider.QEMU)
cfg.Provider.QEMU.Image = "some/image/location"
cfg.Provider.QEMU.Measurements[0] = []byte("00000000000000000000000000000000")
cfg.Provider.QEMU.Measurements[1] = []byte("11111111111111111111111111111111")
cfg.Provider.QEMU.Measurements[2] = []byte("22222222222222222222222222222222")
@ -506,6 +507,7 @@ func defaultConfigWithExpectedMeasurements(t *testing.T, conf *config.Config, cs
conf.Provider.GCP.Measurements[8] = []byte("00000000000000000000000000000000")
conf.Provider.GCP.Measurements[9] = []byte("11111111111111111111111111111111")
case cloudprovider.QEMU:
conf.Provider.QEMU.Image = "some/image/location"
conf.Provider.QEMU.Measurements[8] = []byte("00000000000000000000000000000000")
conf.Provider.QEMU.Measurements[9] = []byte("11111111111111111111111111111111")
}

View File

@ -40,6 +40,10 @@ func TestValidateConfig(t *testing.T) {
wantOutput: true,
wantErr: true,
},
"Azure config with all required fields is valid": {
cnf: defaultConfigWithExpectedMeasurements(t, config.Default(), cloudprovider.Azure),
provider: cloudprovider.Azure,
},
"default GCP config is not valid": {
cnf: func() *config.Config {
cnf := config.Default()
@ -52,7 +56,11 @@ func TestValidateConfig(t *testing.T) {
wantOutput: true,
wantErr: true,
},
"default QEMU config is valid": {
"GCP config with all required fields is valid": {
cnf: defaultConfigWithExpectedMeasurements(t, config.Default(), cloudprovider.GCP),
provider: cloudprovider.GCP,
},
"default QEMU config is not valid": {
cnf: func() *config.Config {
cnf := config.Default()
qemu := cnf.Provider.QEMU
@ -61,6 +69,12 @@ func TestValidateConfig(t *testing.T) {
return cnf
}(),
provider: cloudprovider.QEMU,
wantOutput: true,
wantErr: true,
},
"QEMU config with all required fields is valid": {
cnf: defaultConfigWithExpectedMeasurements(t, config.Default(), cloudprovider.QEMU),
provider: cloudprovider.QEMU,
},
"config with an error": {
cnf: func() *config.Config {

View File

@ -0,0 +1,35 @@
/*
Copyright (c) Edgeless Systems GmbH
SPDX-License-Identifier: AGPL-3.0-only
*/
package terraform
// CreateClusterInput is user configuration for creating a cluster with Terraform.
type CreateClusterInput struct {
// CountControlPlanes is the number of control-plane nodes to create.
CountControlPlanes int
// CountWorkers is the number of worker nodes to create.
CountWorkers int
// QEMU is the configuration for QEMU clusters.
QEMU QEMUInput
}
// QEMUInput is user configuration for creating a QEMU cluster with Terraform.
type QEMUInput struct {
// CPUCount is the number of CPUs to allocate to each node.
CPUCount int
// MemorySizeMiB is the amount of memory to allocate to each node, in MiB.
MemorySizeMiB int
// StateDiskSizeGB is the size of the state disk to allocate to each node, in GB.
StateDiskSizeGB int
// IPRangeStart is the first IP address in the IP range to allocate to the cluster.
IPRangeStart int
// ImagePath is the path to the image to use for the nodes.
ImagePath string
// ImageFormat is the format of the image from ImagePath.
ImageFormat string
// MetadataAPIImage is the container image to use for the metadata API.
MetadataAPIImage string
}

View File

@ -0,0 +1,64 @@
/*
Copyright (c) Edgeless Systems GmbH
SPDX-License-Identifier: AGPL-3.0-only
*/
package terraform
import (
"embed"
"errors"
"io/fs"
"path"
"strings"
"github.com/edgelesssys/constellation/v2/internal/cloud/cloudprovider"
"github.com/edgelesssys/constellation/v2/internal/file"
"github.com/spf13/afero"
)
//go:embed terraform/*
var terraformFS embed.FS
// prepareWorkspace loads the embedded Terraform files,
// and writes them into the workspace.
func prepareWorkspace(fileHandler file.Handler, provider cloudprovider.Provider) error {
// use path.Join to ensure no forward slashes are used to read the embedded FS
rootDir := path.Join("terraform", strings.ToLower(provider.String()))
return fs.WalkDir(terraformFS, rootDir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
content, err := terraformFS.ReadFile(path)
if err != nil {
return err
}
fileName := strings.TrimPrefix(path, rootDir+"/")
return fileHandler.Write(fileName, content, file.OptMkdirAll)
})
}
// cleanUpWorkspace removes files that were loaded into the workspace.
func cleanUpWorkspace(fileHandler file.Handler, provider cloudprovider.Provider) error {
rootDir := path.Join("terraform", strings.ToLower(provider.String()))
return fs.WalkDir(terraformFS, rootDir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
fileName := strings.TrimPrefix(path, rootDir+"/")
return ignoreFileNotFoundErr(fileHandler.RemoveAll(fileName))
})
}
// ignoreFileNotFoundErr ignores the error if it is a file not found error.
func ignoreFileNotFoundErr(err error) error {
if errors.Is(err, afero.ErrFileNotFound) {
return nil
}
return err
}

View File

@ -0,0 +1,62 @@
/*
Copyright (c) Edgeless Systems GmbH
SPDX-License-Identifier: AGPL-3.0-only
*/
package terraform
import (
"io/fs"
"testing"
"github.com/edgelesssys/constellation/v2/internal/cloud/cloudprovider"
"github.com/edgelesssys/constellation/v2/internal/file"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestLoader(t *testing.T) {
testCases := map[string]struct {
provider cloudprovider.Provider
fileList []string
}{
"qemu": {
provider: cloudprovider.QEMU,
fileList: []string{
"main.tf",
"variables.tf",
"outputs.tf",
"modules",
},
},
}
for name, tc := range testCases {
t.Run(name, func(t *testing.T) {
assert := assert.New(t)
require := require.New(t)
file := file.NewHandler(afero.NewMemMapFs())
err := prepareWorkspace(file, tc.provider)
require.NoError(err)
checkFiles(t, file, func(err error) { assert.NoError(err) }, tc.fileList)
err = cleanUpWorkspace(file, tc.provider)
require.NoError(err)
checkFiles(t, file, func(err error) { assert.ErrorIs(err, fs.ErrNotExist) }, tc.fileList)
})
}
}
func checkFiles(t *testing.T, file file.Handler, assertion func(error), files []string) {
t.Helper()
for _, f := range files {
_, err := file.Stat(f)
assertion(err)
}
}

View File

@ -0,0 +1,203 @@
/*
Copyright (c) Edgeless Systems GmbH
SPDX-License-Identifier: AGPL-3.0-only
*/
package terraform
import (
"context"
"errors"
"fmt"
"github.com/edgelesssys/constellation/v2/internal/cloud/cloudprovider"
"github.com/edgelesssys/constellation/v2/internal/file"
"github.com/edgelesssys/constellation/v2/internal/state"
"github.com/hashicorp/go-version"
install "github.com/hashicorp/hc-install"
"github.com/hashicorp/hc-install/fs"
"github.com/hashicorp/hc-install/product"
"github.com/hashicorp/hc-install/releases"
"github.com/hashicorp/hc-install/src"
"github.com/hashicorp/terraform-exec/tfexec"
tfjson "github.com/hashicorp/terraform-json"
"github.com/spf13/afero"
)
const (
tfVersion = ">= 1.2.0"
terraformVarsFile = "terraform.tfvars"
)
// Client manages interaction with Terraform.
type Client struct {
tf tfInterface
provider cloudprovider.Provider
file file.Handler
state state.ConstellationState
remove func()
}
// New sets up a new Client for Terraform.
func New(ctx context.Context, provider cloudprovider.Provider) (*Client, error) {
tf, remove, err := GetExecutable(ctx, ".")
if err != nil {
return nil, err
}
file := file.NewHandler(afero.NewOsFs())
return &Client{
tf: tf,
provider: provider,
remove: remove,
file: file,
}, nil
}
// CreateCluster creates a Constellation cluster using Terraform.
func (c *Client) CreateCluster(ctx context.Context, name string, input CreateClusterInput) error {
if err := prepareWorkspace(c.file, c.provider); err != nil {
return err
}
if err := c.tf.Init(ctx); err != nil {
return err
}
if err := writeUserConfig(c.file, c.provider, name, input); err != nil {
return err
}
if err := c.tf.Apply(ctx); err != nil {
return err
}
tfState, err := c.tf.Show(ctx)
if err != nil {
return err
}
ipOutput, ok := tfState.Values.Outputs["ip"]
if !ok {
return errors.New("no IP output found")
}
ip, ok := ipOutput.Value.(string)
if !ok {
return errors.New("invalid type in IP output: not a string")
}
c.state = state.ConstellationState{
CloudProvider: c.provider.String(),
LoadBalancerIP: ip,
}
return nil
}
// DestroyInstances destroys a Constellation cluster using Terraform.
func (c *Client) DestroyCluster(ctx context.Context) error {
return c.tf.Destroy(ctx)
}
// RemoveInstaller removes the Terraform installer, if it was downloaded for this command.
func (c *Client) RemoveInstaller() {
c.remove()
}
// CleanUpWorkspace removes terraform files from the current directory.
func (c *Client) CleanUpWorkspace() error {
if err := cleanUpWorkspace(c.file, c.provider); err != nil {
return err
}
if err := ignoreFileNotFoundErr(c.file.Remove("terraform.tfvars")); err != nil {
return err
}
if err := ignoreFileNotFoundErr(c.file.Remove("terraform.tfstate")); err != nil {
return err
}
if err := ignoreFileNotFoundErr(c.file.Remove("terraform.tfstate.backup")); err != nil {
return err
}
if err := ignoreFileNotFoundErr(c.file.Remove(".terraform.lock.hcl")); err != nil {
return err
}
if err := ignoreFileNotFoundErr(c.file.RemoveAll(".terraform")); err != nil {
return err
}
return nil
}
// GetState returns the state of the cluster.
func (c *Client) GetState() state.ConstellationState {
return c.state
}
// writeUserConfig writes the user config file for Terraform.
func writeUserConfig(file file.Handler, provider cloudprovider.Provider, name string, input CreateClusterInput) error {
var userConfig string
switch provider {
case cloudprovider.QEMU:
userConfig = fmt.Sprintf(`
constellation_coreos_image = "%s"
image_format = "%s"
control_plane_count = %d
worker_count = %d
vcpus = %d
memory = %d
state_disk_size = %d
ip_range_start = %d
metadata_api_image = "%s"
name = "%s"
`,
input.QEMU.ImagePath, input.QEMU.ImageFormat,
input.CountControlPlanes, input.CountWorkers,
input.QEMU.CPUCount, input.QEMU.MemorySizeMiB, input.QEMU.StateDiskSizeGB,
input.QEMU.IPRangeStart,
input.QEMU.MetadataAPIImage,
name,
)
}
return file.Write(terraformVarsFile, []byte(userConfig))
}
// GetExecutable returns a Terraform executable either from the local filesystem,
// or downloads the latest version fulfilling the version constraint.
func GetExecutable(ctx context.Context, workingDir string) (terraform *tfexec.Terraform, remove func(), err error) {
inst := install.NewInstaller()
version, err := version.NewConstraint(tfVersion)
if err != nil {
return nil, nil, err
}
downloadVersion := &releases.LatestVersion{
Product: product.Terraform,
Constraints: version,
}
localVersion := &fs.Version{
Product: product.Terraform,
Constraints: version,
}
execPath, err := inst.Ensure(ctx, []src.Source{localVersion, downloadVersion})
if err != nil {
return nil, nil, err
}
tf, err := tfexec.NewTerraform(workingDir, execPath)
return tf, func() { _ = inst.Remove(context.Background()) }, err
}
type tfInterface interface {
Apply(context.Context, ...tfexec.ApplyOption) error
Destroy(context.Context, ...tfexec.DestroyOption) error
Init(context.Context, ...tfexec.InitOption) error
Show(context.Context, ...tfexec.ShowOption) (*tfjson.State, error)
}

View File

@ -25,15 +25,19 @@ provider "docker" {
}
resource "docker_image" "qemu-metadata" {
name = "ghcr.io/edgelesssys/constellation/qemu-metadata-api:v1.4.1-0.20220817163854-84e9f659542d"
name = "${var.metadata_api_image}"
keep_locally = true
}
resource "docker_container" "qemu-metadata" {
name = "qemu-metadata"
name = "${var.name}-qemu-metadata"
image = docker_image.qemu-metadata.latest
network_mode = "host"
rm = true
command = [
"--network",
"${var.name}-network",
]
mounts {
source = "/var/run/libvirt/libvirt-sock"
target = "/var/run/libvirt/libvirt-sock"
@ -54,6 +58,7 @@ module "control_plane" {
pool = libvirt_pool.cluster.name
boot_volume_id = libvirt_volume.constellation_coreos_image.id
machine = var.machine
name = var.name
}
module "worker" {
@ -69,23 +74,24 @@ module "worker" {
pool = libvirt_pool.cluster.name
boot_volume_id = libvirt_volume.constellation_coreos_image.id
machine = var.machine
name = var.name
}
resource "libvirt_pool" "cluster" {
name = "constellation"
name = "${var.name}-storage-pool"
type = "dir"
path = "/var/lib/libvirt/images"
}
resource "libvirt_volume" "constellation_coreos_image" {
name = "constellation-coreos-image"
name = "${var.name}-node-image"
pool = libvirt_pool.cluster.name
source = var.constellation_coreos_image
format = var.image_format
}
resource "libvirt_network" "constellation" {
name = "constellation"
name = "${var.name}-network"
mode = "nat"
addresses = ["10.42.0.0/16"]
dhcp {

View File

@ -12,7 +12,7 @@ locals {
}
resource "libvirt_domain" "instance_group" {
name = "${var.role}-${count.index}"
name = "${var.name}-${var.role}-${count.index}"
count = var.amount
memory = var.memory
vcpu = var.vcpus

View File

@ -52,3 +52,8 @@ variable "machine" {
type = string
description = "machine type. use 'q35' for secure boot and 'pc' for non secure boot. See 'qemu-system-x86_64 -machine help'"
}
variable "name" {
type = string
description = "name prefix of the cluster VMs"
}

View File

@ -0,0 +1,3 @@
output "ip" {
value = module.control_plane.instance_ips[0]
}

View File

@ -11,43 +11,47 @@ variable "image_format" {
variable "control_plane_count" {
type = number
default = 3
description = "amount of control plane nodes"
}
variable "worker_count" {
type = number
default = 2
description = "amount of worker nodes"
}
variable "vcpus" {
type = number
default = 2
description = "amount of vcpus per instance"
}
variable "memory" {
type = number
default = 2048
description = "amount of memory per instance (MiB)"
}
variable "state_disk_size" {
type = number
default = 10
description = "size of state disk (GiB)"
}
variable "ip_range_start" {
type = number
default = 100
description = "first ip address to use within subnet"
}
variable "machine" {
type = string
default = "q35"
description = "machine type. use 'q35' for secure boot and 'pc' for non secure boot. See 'qemu-system-x86_64 -machine help'"
}
variable "metadata_api_image" {
type = string
description = "container image of the QEMU metadata api server"
}
variable "name" {
type = string
default = "constellation"
description = "name prefix of the cluster VMs"
}

View File

@ -0,0 +1,235 @@
/*
Copyright (c) Edgeless Systems GmbH
SPDX-License-Identifier: AGPL-3.0-only
*/
package terraform
import (
"context"
"errors"
"io/fs"
"testing"
"github.com/edgelesssys/constellation/v2/internal/cloud/cloudprovider"
"github.com/edgelesssys/constellation/v2/internal/file"
"github.com/hashicorp/terraform-exec/tfexec"
tfjson "github.com/hashicorp/terraform-json"
"github.com/spf13/afero"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"go.uber.org/multierr"
)
func TestCreateInstances(t *testing.T) {
someErr := errors.New("error")
getState := func() *tfjson.State {
workingState := tfjson.State{
Values: &tfjson.StateValues{
Outputs: map[string]*tfjson.StateOutput{
"ip": {
Value: "192.0.2.100",
},
},
},
}
return &workingState
}
testCases := map[string]struct {
provider cloudprovider.Provider
input CreateClusterInput
tf *stubTerraform
fs afero.Fs
wantErr bool
}{
"works": {
provider: cloudprovider.QEMU,
tf: &stubTerraform{
showState: getState(),
},
fs: afero.NewMemMapFs(),
},
"init fails": {
provider: cloudprovider.QEMU,
tf: &stubTerraform{
initErr: someErr,
showState: getState(),
},
fs: afero.NewMemMapFs(),
wantErr: true,
},
"apply fails": {
provider: cloudprovider.QEMU,
tf: &stubTerraform{
applyErr: someErr,
showState: getState(),
},
fs: afero.NewMemMapFs(),
wantErr: true,
},
"show fails": {
provider: cloudprovider.QEMU,
tf: &stubTerraform{
showErr: someErr,
},
fs: afero.NewMemMapFs(),
wantErr: true,
},
"no ip": {
provider: cloudprovider.QEMU,
tf: &stubTerraform{
showState: &tfjson.State{
Values: &tfjson.StateValues{
Outputs: map[string]*tfjson.StateOutput{},
},
},
},
fs: afero.NewMemMapFs(),
wantErr: true,
},
"prepare workspace fails": {
provider: cloudprovider.QEMU,
tf: &stubTerraform{
showState: getState(),
},
fs: afero.NewReadOnlyFs(afero.NewMemMapFs()),
wantErr: true,
},
}
for name, tc := range testCases {
t.Run(name, func(t *testing.T) {
assert := assert.New(t)
c := &Client{
provider: tc.provider,
tf: tc.tf,
file: file.NewHandler(tc.fs),
}
err := c.CreateCluster(context.Background(), "test", tc.input)
if tc.wantErr {
assert.Error(err)
return
}
assert.NoError(err)
})
}
}
func TestDestroyInstances(t *testing.T) {
testCases := map[string]struct {
tf *stubTerraform
wantErr bool
}{
"works": {
tf: &stubTerraform{},
},
"destroy fails": {
tf: &stubTerraform{
destroyErr: errors.New("error"),
},
wantErr: true,
},
}
for name, tc := range testCases {
t.Run(name, func(t *testing.T) {
assert := assert.New(t)
c := &Client{
provider: cloudprovider.QEMU,
tf: tc.tf,
}
err := c.DestroyCluster(context.Background())
if tc.wantErr {
assert.Error(err)
return
}
assert.NoError(err)
})
}
}
func TestCleanupWorkspace(t *testing.T) {
someContent := []byte("some content")
testCases := map[string]struct {
provider cloudprovider.Provider
prepareFS func(file.Handler) error
wantErr bool
}{
"files are cleaned up": {
provider: cloudprovider.QEMU,
prepareFS: func(f file.Handler) error {
var err error
err = multierr.Append(err, f.Write("terraform.tfvars", someContent))
err = multierr.Append(err, f.Write("terraform.tfstate", someContent))
return multierr.Append(err, f.Write("terraform.tfstate.backup", someContent))
},
},
"no error if files do not exist": {
provider: cloudprovider.QEMU,
prepareFS: func(f file.Handler) error { return nil },
},
}
for name, tc := range testCases {
t.Run(name, func(t *testing.T) {
assert := assert.New(t)
require := require.New(t)
file := file.NewHandler(afero.NewMemMapFs())
require.NoError(tc.prepareFS(file))
c := &Client{
provider: tc.provider,
file: file,
tf: &stubTerraform{},
}
err := c.CleanUpWorkspace()
if tc.wantErr {
assert.Error(err)
return
}
assert.NoError(err)
_, err = file.Stat("terraform.tfvars")
assert.ErrorIs(err, fs.ErrNotExist)
_, err = file.Stat("terraform.tfstate")
assert.ErrorIs(err, fs.ErrNotExist)
_, err = file.Stat("terraform.tfstate.backup")
assert.ErrorIs(err, fs.ErrNotExist)
})
}
}
type stubTerraform struct {
applyErr error
destroyErr error
initErr error
showErr error
showState *tfjson.State
}
func (s *stubTerraform) Apply(context.Context, ...tfexec.ApplyOption) error {
return s.applyErr
}
func (s *stubTerraform) Destroy(context.Context, ...tfexec.DestroyOption) error {
return s.destroyErr
}
func (s *stubTerraform) Init(context.Context, ...tfexec.InitOption) error {
return s.initErr
}
func (s *stubTerraform) Show(context.Context, ...tfexec.ShowOption) (*tfjson.State, error) {
return s.showState, s.showErr
}

9
go.mod
View File

@ -71,6 +71,10 @@ require (
github.com/googleapis/gax-go/v2 v2.4.0
github.com/grpc-ecosystem/go-grpc-middleware v1.3.0
github.com/hashicorp/go-multierror v1.1.1
github.com/hashicorp/go-version v1.6.0
github.com/hashicorp/hc-install v0.4.0
github.com/hashicorp/terraform-exec v0.17.3
github.com/hashicorp/terraform-json v0.14.0
github.com/manifoldco/promptui v0.9.0
github.com/martinjungblut/go-cryptsetup v0.0.0-20220520180014-fd0874fd07a6
github.com/microsoft/ApplicationInsights-Go v0.4.4
@ -102,6 +106,11 @@ require (
k8s.io/utils v0.0.0-20220812165043-ad590609e2e5
)
require (
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/zclconf/go-cty v1.11.0 // indirect
)
require (
cloud.google.com/go v0.102.0 // indirect
code.cloudfoundry.org/clock v0.0.0-20180518195852-02e53af36e6c // indirect

53
go.sum
View File

@ -194,7 +194,9 @@ github.com/Masterminds/sprig/v3 v3.2.2 h1:17jRggJu518dr3QaafizSXOjKYp94wKfABxUmy
github.com/Masterminds/sprig/v3 v3.2.2/go.mod h1:UoaO7Yp8KlPnJIYWTFkMaqPUYKTfGFPhxNuwnnxkKlk=
github.com/Masterminds/squirrel v1.5.3 h1:YPpoceAcxuzIljlr5iWpNKaql7hLeG1KLSrhvdHpkZc=
github.com/Masterminds/squirrel v1.5.3/go.mod h1:NNaOrjSoIDfDA40n7sr2tPNZRfjzjA400rg+riTZj10=
github.com/Microsoft/go-winio v0.4.14/go.mod h1:qXqCSQ3Xa7+6tgxaGTIe4Kpcdsi+P8jBhyzoq1bpyYA=
github.com/Microsoft/go-winio v0.4.15/go.mod h1:tTuCMEN+UleMWgg9dVx4Hu52b1bJo+59jBh3ajtinzw=
github.com/Microsoft/go-winio v0.4.16/go.mod h1:XB6nPKklQyQ7GC9LdcBEcBl8PF76WugXOPRXwdLnMv0=
github.com/Microsoft/go-winio v0.4.17/go.mod h1:JPGBdM1cNvN/6ISo+n8V5iA4v8pBzdOpzfwIujj1a84=
github.com/Microsoft/go-winio v0.5.2 h1:a9IhgEQBCUEk6QCdml9CiJGhAws+YwffDHEMp1VMrpA=
github.com/Microsoft/go-winio v0.5.2/go.mod h1:WpS1mjBmmwHBEWmogvA2mj8546UReBk4v8QkMxJ6pZY=
@ -204,6 +206,8 @@ github.com/NYTimes/gziphandler v0.0.0-20170623195520-56545f4a5d46/go.mod h1:3wb0
github.com/NYTimes/gziphandler v1.1.1/go.mod h1:n/CVRwUEOgIxrgPvAQhUUr9oeUtvrhMomdKFjzJNB0c=
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
github.com/OpenPeeDeeP/depguard v1.0.1/go.mod h1:xsIw86fROiiwelg+jB2uM9PiKihMMmUx/1V+TNhjQvM=
github.com/ProtonMail/go-crypto v0.0.0-20210428141323-04723f9f07d7 h1:YoJbenK9C67SkzkDfmQuVln04ygHj3vjZfd9FL+GmQQ=
github.com/ProtonMail/go-crypto v0.0.0-20210428141323-04723f9f07d7/go.mod h1:z4/9nQmJSSwwds7ejkxaJwO37dru3geImFUdJlaLzQo=
github.com/PuerkitoBio/purell v1.1.1 h1:WEQqlqaGbrPkxLJWfBwQmfEAE1Z7ONdDLqrN38tNFfI=
github.com/PuerkitoBio/purell v1.1.1/go.mod h1:c11w/QuzBsJSee3cPx9rAFu61PvFxuPbtSwDGJws/X0=
github.com/PuerkitoBio/urlesc v0.0.0-20170810143723-de5bf2ad4578 h1:d+Bc7a5rLufV/sSk/8dngufqelfh6jnri85riMAaF/M=
@ -213,6 +217,8 @@ github.com/Shopify/sarama v1.19.0/go.mod h1:FVkBWblsNy7DGZRfXLU0O9RCGt5g3g3yEuWX
github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMxUHB2q5Ap20/P/eIdh4G0pI=
github.com/StackExchange/wmi v1.2.1/go.mod h1:rcmrprowKIVzvc+NUiLncP2uuArMWLCbu9SBzvHz7e8=
github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g=
github.com/acomagu/bufpipe v1.0.3 h1:fxAGrHZTgQ9w5QqVItgzwj235/uYZYgbXitB+dLupOk=
github.com/acomagu/bufpipe v1.0.3/go.mod h1:mxdxdup/WdsKVreO5GpW4+M/1CE2sMG4jeGJ2sYmHc4=
github.com/afex/hystrix-go v0.0.0-20180502004556-fa1af6a1f4f5/go.mod h1:SkGFH1ia65gfNATL8TAiHDNxPzPdmEL5uirI2Uyuz6c=
github.com/ajstarks/svgo v0.0.0-20180226025133-644b8db467af/go.mod h1:K08gAheRH3/J6wwsYMMT4xOr94bZjxIelGM0+d/wbFw=
github.com/alcortesm/tgz v0.0.0-20161220082320-9c5fe88206d7/go.mod h1:6zEj6s6u/ghQa61ZWa/C2Aw3RkjiTBOix7dkqa1VLIs=
@ -238,6 +244,8 @@ github.com/apex/log v1.1.4/go.mod h1:AlpoD9aScyQfJDVHmLMEcx4oU6LqzkWp4Mg9GdAcEvQ
github.com/apex/logs v0.0.4/go.mod h1:XzxuLZ5myVHDy9SAmYpamKKRNApGj54PfYLcFrXqDwo=
github.com/aphistic/golf v0.0.0-20180712155816-02c07f170c5a/go.mod h1:3NqKYiepwy8kCu4PNA+aP7WUV72eXWJeP9/r3/K9aLE=
github.com/aphistic/sweet v0.2.0/go.mod h1:fWDlIh/isSE9n6EPsRmC0det+whmX6dJid3stzu0Xys=
github.com/apparentlymart/go-textseg v1.0.0/go.mod h1:z96Txxhf3xSFMPmb5X/1W05FF/Nj9VFpLOpjS5yuumk=
github.com/apparentlymart/go-textseg/v13 v13.0.0/go.mod h1:ZK2fH7c4NqDTLtiYLvIkEghdlcqw7yxLeM89kiTRPUo=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
@ -484,6 +492,7 @@ github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb
github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
github.com/emicklei/go-restful/v3 v3.8.0 h1:eCZ8ulSerjdAiaNpF7GxXIE7ZCMo1moN1qX+S609eVw=
github.com/emicklei/go-restful/v3 v3.8.0/go.mod h1:6n3XBCmQQb25CM2LCACGz8ukIrRry+4bhvbpWn3mrbc=
github.com/emirpasic/gods v1.12.0 h1:QAUIPSaCu4G+POclxeqb3F+WPpdKqFGlw36+yOzGlrg=
github.com/emirpasic/gods v1.12.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o=
github.com/envoyproxy/go-control-plane v0.6.9/go.mod h1:SBwIajubJHhxtWwsL9s8ss4safvEdbitLhGGK48rN6g=
github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
@ -551,6 +560,14 @@ github.com/go-critic/go-critic v0.6.1/go.mod h1:SdNCfU0yF3UBjtaZGw6586/WocupMOJu
github.com/go-errors/errors v1.0.1/go.mod h1:f4zRHt4oKfwPJE5k8C9vpYG+aDHdBFUsgrm6/TyX73Q=
github.com/go-errors/errors v1.4.2 h1:J6MZopCL4uSllY1OfXM374weqZFFItUbrImctkmUxIA=
github.com/go-errors/errors v1.4.2/go.mod h1:sIVyrIiJhuEF+Pj9Ebtd6P/rEYROXFi3BopGUQ5a5Og=
github.com/go-git/gcfg v1.5.0 h1:Q5ViNfGF8zFgyJWPqYwA7qGFoMTEiBmdlkcfRmpIMa4=
github.com/go-git/gcfg v1.5.0/go.mod h1:5m20vg6GwYabIxaOonVkTdrILxQMpEShl1xiMF4ua+E=
github.com/go-git/go-billy/v5 v5.2.0/go.mod h1:pmpqyWchKfYfrkb/UVH4otLvyi/5gJlGI4Hb3ZqZ3W0=
github.com/go-git/go-billy/v5 v5.3.1 h1:CPiOUAzKtMRvolEKw+bG1PLRpT7D3LIs3/3ey4Aiu34=
github.com/go-git/go-billy/v5 v5.3.1/go.mod h1:pmpqyWchKfYfrkb/UVH4otLvyi/5gJlGI4Hb3ZqZ3W0=
github.com/go-git/go-git-fixtures/v4 v4.2.1/go.mod h1:K8zd3kDUAykwTdDCr+I0per6Y6vMiRR/nnVTBtavnB0=
github.com/go-git/go-git/v5 v5.4.2 h1:BXyZu9t0VkbiHtqrsvdq39UDhGJTl1h55VW6CSC4aY4=
github.com/go-git/go-git/v5 v5.4.2/go.mod h1:gQ1kArt6d+n+BGd+/B/I74HwRTLhth2+zti4ihgckDc=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
@ -872,8 +889,11 @@ github.com/hashicorp/consul/sdk v0.8.0/go.mod h1:GBvyrGALthsZObzUGsfgHZQDXjg4lOj
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-checkpoint v0.5.0/go.mod h1:7nfLNL10NsxqO4iWuW6tWW0HjZuDrwkBuEQsVcpCOgg=
github.com/hashicorp/go-cleanhttp v0.5.0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ=
github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48=
github.com/hashicorp/go-hclog v0.0.0-20180709165350-ff2cf002a8dd/go.mod h1:9bjs9uLqI8l75knNv3lV1kA55veR+WUPSiKIWcQHudI=
github.com/hashicorp/go-hclog v0.8.0/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
@ -898,10 +918,15 @@ github.com/hashicorp/go-uuid v1.0.1/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/b
github.com/hashicorp/go-version v1.1.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/go-version v1.2.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/go-version v1.2.1/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/go-version v1.5.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/go-version v1.6.0 h1:feTTfFNnjP967rlCxM/I9g701jU+RN74YKx2mOkIeek=
github.com/hashicorp/go-version v1.6.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/go.net v0.0.1/go.mod h1:hjKkEWcCURg++eb33jQU7oqQcI9XDCnUzHA0oac0k90=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
github.com/hashicorp/hc-install v0.4.0 h1:cZkRFr1WVa0Ty6x5fTvL1TuO1flul231rWkGH92oYYk=
github.com/hashicorp/hc-install v0.4.0/go.mod h1:5d155H8EC5ewegao9A4PUTMNPZaq+TbOzkJJZ4vrXeI=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64=
github.com/hashicorp/mdns v1.0.0/go.mod h1:tL+uN++7HEJ6SQLQ2/p+z2pH24WQKWjBPkE0mNTz8vQ=
@ -910,6 +935,10 @@ github.com/hashicorp/memberlist v0.1.3/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2p
github.com/hashicorp/memberlist v0.2.2/go.mod h1:MS2lj3INKhZjWNqd3N0m3J+Jxf3DAOnAH9VT3Sh9MUE=
github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc=
github.com/hashicorp/serf v0.9.5/go.mod h1:UWDWwZeL5cuWDJdl0C6wrvrUwEqtQ4ZKBKKENpqIUyk=
github.com/hashicorp/terraform-exec v0.17.3 h1:MX14Kvnka/oWGmIkyuyvL6POx25ZmKrjlaclkx3eErU=
github.com/hashicorp/terraform-exec v0.17.3/go.mod h1:+NELG0EqQekJzhvikkeQsOAZpsw0cv/03rbeQJqscAI=
github.com/hashicorp/terraform-json v0.14.0 h1:sh9iZ1Y8IFJLx+xQiKHGud6/TSUCM0N8e17dKDpqV7s=
github.com/hashicorp/terraform-json v0.14.0/go.mod h1:5A9HIWPkk4e5aeeXIBbkcOvaZbIYnAIkEyqP2pNSckM=
github.com/hashicorp/vault/api v1.0.4/go.mod h1:gDcqh3WGcR1cpF5AJz/B1UFheUEneMoIospckxBxk6Q=
github.com/hashicorp/vault/sdk v0.1.13/go.mod h1:B+hVj7TpuQY1Y/GPbCpffmgd+tSEwvhkWnjtSYCaS2M=
github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb/go.mod h1:+NfK9FKeTrX5uv1uIXGdwYDTeHna2qgaIlx54MXqjAM=
@ -941,8 +970,10 @@ github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANyt
github.com/influxdata/influxdb1-client v0.0.0-20191209144304-8bf82d3c094d/go.mod h1:qj24IKcXYK6Iy9ceXlo3Tc+vtHo9lIhSX5JddghvEPo=
github.com/ishidawataru/sctp v0.0.0-20190723014705-7c296d48a2b5/go.mod h1:DM4VvS+hD/kDi1U1QsX2fnZowwBhqD0Dk3bRPKF/Oc8=
github.com/jarcoal/httpmock v1.0.5/go.mod h1:ATjnClrvW/3tijVmpL/va5Z3aAyGvqU3gCT8nX0Txik=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 h1:BQSFePA1RWJOlocH6Fxy8MmwDt+yVQYULKfN0RoTN8A=
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99/go.mod h1:1lJo3i6rXxKeerYnT8Nvf0QmHCRC1n8sfWVwXF2Frvo=
github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
github.com/jessevdk/go-flags v1.5.0/go.mod h1:Fw0T6WPc1dYxT4mKEZRfG5kJhaTDP9pj1c2EWnYs/m4=
github.com/jgautheron/goconst v1.5.1/go.mod h1:aAosetZ5zaeC/2EfMeRswtxUFBpe2Hr7HzkgX4fanO4=
github.com/jhump/protoreflect v1.6.1/go.mod h1:RZQ/lnuN+zqeRVpQigTwO6o0AJUkxbnSnpuG7toUTG4=
github.com/jhump/protoreflect v1.8.2/go.mod h1:7GcYQDdMU/O/BBrl/cX6PNHpXh6cenjd8pneu5yW7Tg=
@ -989,6 +1020,8 @@ github.com/k0kubun/go-ansi v0.0.0-20180517002512-3bf9e2903213/go.mod h1:vNUNkEQ1
github.com/karrick/godirwalk v1.16.1 h1:DynhcF+bztK8gooS0+NDJFrdNZjJ3gzVzC545UNA9iw=
github.com/karrick/godirwalk v1.16.1/go.mod h1:j4mkqPuvaLI8mp1DroR3P6ad7cyYd4c1qeJ3RV7ULlk=
github.com/kevinburke/ssh_config v0.0.0-20190725054713-01f96b0aa0cd/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM=
github.com/kevinburke/ssh_config v0.0.0-20201106050909-4977a11b4351 h1:DowS9hvgyYSX4TO5NpyC606/Z4SxnNYbT+WX27or6Ck=
github.com/kevinburke/ssh_config v0.0.0-20201106050909-4977a11b4351/go.mod h1:CT57kijsi8u/K/BOFA39wgDQJ9CxiF4nAY/ojJ6r6mM=
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
github.com/kisielk/errcheck v1.2.0/go.mod h1:/BMXB+zMLi60iA8Vv6Ksmxu/1UDYcXs4uQLJ+jE2L00=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
@ -1016,6 +1049,7 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kulti/thelper v0.4.0/go.mod h1:vMu2Cizjy/grP+jmsvOFDx1kYP6+PD1lqg4Yu5exl2U=
github.com/kunwardeep/paralleltest v1.0.3/go.mod h1:vLydzomDFpk7yu5UX02RmP0H8QfRPOV/oFhWN85Mjb4=
github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348/go.mod h1:B69LEHPfb2qLo0BaaOLcbitczOKLWTsrBG9LczfCD4k=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/kyoh86/exportloopref v0.1.8/go.mod h1:1tUcJeiioIs7VWe5gcOObrux3lb66+sBqGZrRkMwPgg=
@ -1073,6 +1107,7 @@ github.com/markbates/safe v1.0.1/go.mod h1:nAqgmRi7cY2nqMc92/bSEeQA+R4OheNU2T1kN
github.com/martinjungblut/go-cryptsetup v0.0.0-20220520180014-fd0874fd07a6 h1:YDjLk3wsL5ZLhLC4TIwIvT2NkSCAdAV6pzzZaRfj4jk=
github.com/martinjungblut/go-cryptsetup v0.0.0-20220520180014-fd0874fd07a6/go.mod h1:gZoZ0+POlM1ge/VUxWpMmZVNPzzMJ7l436CgkQ5+qzU=
github.com/matoous/godox v0.0.0-20210227103229-6504466cf951/go.mod h1:1BELzlh859Sh1c6+90blK8lbYy0kwQf1bYlBhBysy1s=
github.com/matryer/is v1.2.0/go.mod h1:2fLPjFQM9rhQ15aVEtbuwhJinnOqrmgXPNdZsdwlWXA=
github.com/matryer/is v1.4.0/go.mod h1:8I/i5uYgLzgsgEloJE1U6xx5HkBQpAZvepWuujKwMRU=
github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
github.com/mattn/go-colorable v0.1.1/go.mod h1:FuOcm+DKB9mbwrcAfNl7/TZVBZ6rcnceauSikq3lYCQ=
@ -1386,6 +1421,7 @@ github.com/sassoftware/go-rpmutils v0.0.0-20190420191620-a8f1baeba37b/go.mod h1:
github.com/schollz/progressbar/v3 v3.8.6 h1:QruMUdzZ1TbEP++S1m73OqRJk20ON11m6Wqv4EoGg8c=
github.com/schollz/progressbar/v3 v3.8.6/go.mod h1:W5IEwbJecncFGBvuEh4A7HT1nZZ6WNIL2i3qbnI0WKY=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/sebdah/goldie v1.0.0/go.mod h1:jXP4hmWywNEwZzhMuv2ccnqTSFpuq8iyQhtQdkkZBH4=
github.com/seccomp/libseccomp-golang v0.9.2-0.20210429002308-3879420cc921/go.mod h1:JA8cRccbGaA1s33RQf7Y1+q9gHmZX1yB/z9WDN1C6fg=
github.com/securego/gosec/v2 v2.9.1/go.mod h1:oDcDLcatOJxkCGaCaq8lua1jTnYf6Sou4wdiJ1n4iHc=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
@ -1525,11 +1561,17 @@ github.com/viki-org/dnscache v0.0.0-20130720023526-c70c1f23c5d8/go.mod h1:dniwbG
github.com/vishvananda/netlink v1.1.0/go.mod h1:cTgwzPIzzgDAYoQrMm0EdrjRUBkTqKYppBueQtXaqoE=
github.com/vishvananda/netns v0.0.0-20191106174202-0a2b9b5464df/go.mod h1:JP3t17pCcGlemwknint6hfoeCVQrEMVwxRLRjXpq+BU=
github.com/vishvananda/netns v0.0.0-20200728191858-db3c7e526aae/go.mod h1:DD4vA1DwXk04H54A1oHXtwZmA0grkVMdPxx/VGLCah0=
github.com/vmihailenco/msgpack v3.3.3+incompatible h1:wapg9xDUZDzGCNFlwc5SqI1rvcciqcxEHac4CYj89xI=
github.com/vmihailenco/msgpack v3.3.3+incompatible/go.mod h1:fy3FlTQTDXWkZ7Bh6AcGMlsjHatGryHQYUTf1ShIgkk=
github.com/vmihailenco/msgpack/v4 v4.3.12 h1:07s4sz9IReOgdikxLTKNbBdqDMLsjPKXwvCazn8G65U=
github.com/vmihailenco/msgpack/v4 v4.3.12/go.mod h1:gborTTJjAo/GWTqqRjrLCn9pgNN+NXzzngzBKDPIqw4=
github.com/vmihailenco/tagparser v0.1.1 h1:quXMXlA39OCbd2wAdTsGDlK9RkOk6Wuw+x37wVyIuWY=
github.com/vmihailenco/tagparser v0.1.1/go.mod h1:OeAg3pn3UbLjkWt+rN9oFYB6u/cQgqMEUPoW2WPyhdI=
github.com/vmware/govmomi v0.20.3/go.mod h1:URlwyTFZX72RmxtxuaFL2Uj3fD1JTvZdx59bHWk6aFU=
github.com/xanzy/go-gitlab v0.31.0/go.mod h1:sPLojNBn68fMUWSxIJtdVVIP8uSBYqesTfDUseX11Ug=
github.com/xanzy/ssh-agent v0.2.1/go.mod h1:mLlQY/MoOhWBj+gOGMQkOeiEvkx+8pJSI+0Bx9h2kr4=
github.com/xanzy/ssh-agent v0.3.0 h1:wUMzuKtKilRgBAD1sUb8gOwwRr2FGoBVumcjoOACClI=
github.com/xanzy/ssh-agent v0.3.0/go.mod h1:3s9xbODqPuuhK9JV1R321M/FlMZSBvE5aY6eAcqrDh0=
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f h1:J9EGpcZtP0E/raorCMxlFGSTBrsSlaDGf3jU/qvAE2c=
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 h1:EzJWgHovont7NscjpAxXsDA8S8BMYve8Y5+7cuRE7R0=
@ -1557,6 +1599,11 @@ github.com/yuin/goldmark v1.4.1/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1
github.com/yvasiyarov/go-metrics v0.0.0-20140926110328-57bccd1ccd43 h1:+lm10QQTNSBd8DVTNGHx7o/IKu9HYDvLMffDhbyLccI=
github.com/yvasiyarov/gorelic v0.0.0-20141212073537-a9bba5b9ab50 h1:hlE8//ciYMztlGpl/VA+Zm1AcTPHYkHJPbHqE6WJUXE=
github.com/yvasiyarov/newrelic_platform_go v0.0.0-20140908184405-b21fdbd4370f h1:ERexzlUfuTvpE74urLSbIQW0Z/6hF9t8U4NsJLaioAY=
github.com/zclconf/go-cty v1.2.0/go.mod h1:hOPWgoHbaTUnI5k4D2ld+GRpFJSCe6bCM7m1q/N4PQ8=
github.com/zclconf/go-cty v1.10.0/go.mod h1:vVKLxnk3puL4qRAv72AO+W99LUD4da90g3uUAzyuvAk=
github.com/zclconf/go-cty v1.11.0 h1:726SxLdi2SDnjY+BStqB9J1hNp4+2WlzyXLuimibIe0=
github.com/zclconf/go-cty v1.11.0/go.mod h1:s9IfD1LK5ccNMSWCVFCE2rJfHiZgi7JijgeWIMfhLvA=
github.com/zclconf/go-cty-debug v0.0.0-20191215020915-b22d67c1ba0b/go.mod h1:ZRKQfBXbGkpdV6QMzT3rU1kSTAnfu1dO8dPKjYprgj8=
github.com/ziutek/mymysql v1.5.4 h1:GB0qdRGsTwQSBVYuVShFBKaXSnSnYYC2d9knnE1LHFs=
github.com/ziutek/mymysql v1.5.4/go.mod h1:LMSpPZ6DbqWFxNCHW77HeMg9I646SAhApZ/wKdgO/C0=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
@ -1670,8 +1717,10 @@ golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPh
golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
golang.org/x/crypto v0.0.0-20201221181555-eec23a3978ad/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I=
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8=
golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20210817164053-32db794688a5/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
@ -1731,6 +1780,7 @@ golang.org/x/mod v0.5.0/go.mod h1:5OXOZSfqPIIbmVBIIKWRFfZjPR0E5r58TLhUjH0a2Ro=
golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o=
golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3/go.mod h1:3p9vT2HGsQu2K1YbXdKPJLVgG5VJdoTa1poYQBtP1AY=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180811021610-c39426892332/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181023162649-9b4f9f5ad519/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
@ -1782,6 +1832,7 @@ golang.org/x/net v0.0.0-20201224014010-6772e930b67b/go.mod h1:m0MpNAwzfU5UDzcl9v
golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20210316092652-d523dce5a7f4/go.mod h1:RBQZq4jEuRlivfhVLdyRGr576XBO4/greRjx4P4O3yc=
golang.org/x/net v0.0.0-20210326060303-6b1517762897/go.mod h1:uSPa2vr4CLtc/ILN5odXGNXS6mhrKVzTaCXzk9m6W3k=
golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
golang.org/x/net v0.0.0-20210421230115-4e50805a0758/go.mod h1:72T/g9IO56b78aLF+1Kcs5dz7/ng1VjMUvfKvpfy+jM=
golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk=
@ -1932,6 +1983,7 @@ golang.org/x/sys v0.0.0-20210412220455-f1c623a9e750/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20210420072515-93ed5bcd2bfe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210423185535-09eb48e85fd7/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210502180810-71e4cd670f79/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210503080704-8803ae5d1324/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@ -2385,6 +2437,7 @@ gopkg.in/src-d/go-git.v4 v4.13.1/go.mod h1:nx5NYcxdKxq5fpltdHnPa2Exj4Sx0EclMWZQb
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
gopkg.in/warnings.v0 v0.1.1/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI=
gopkg.in/warnings.v0 v0.1.2 h1:wFXVbFY8DY5/xOe1ECiWdKCzZlxgshcYVNkBHstARME=
gopkg.in/warnings.v0 v0.1.2/go.mod h1:jksf8JmL6Qr/oQM2OXTHunEvvTAsrWBLb6OOjuVWRNI=
gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

View File

@ -27,6 +27,8 @@ while read line; do
;;
github.com/letsencrypt/boulder)
;;
github.com/hashicorp/*)
;;
*)
not_allowed
;;

View File

@ -56,6 +56,17 @@ require (
libvirt.org/go/libvirtxml v1.8007.0
)
require (
github.com/hashicorp/errwrap v1.1.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/hashicorp/go-multierror v1.1.1 // indirect
github.com/hashicorp/go-version v1.6.0 // indirect
github.com/hashicorp/hc-install v0.4.0 // indirect
github.com/hashicorp/terraform-exec v0.17.3 // indirect
github.com/hashicorp/terraform-json v0.14.0 // indirect
github.com/zclconf/go-cty v1.11.0 // indirect
)
require (
cloud.google.com/go v0.102.0 // indirect
cloud.google.com/go/iam v0.3.0 // indirect

View File

@ -204,6 +204,8 @@ github.com/apex/log v1.1.4/go.mod h1:AlpoD9aScyQfJDVHmLMEcx4oU6LqzkWp4Mg9GdAcEvQ
github.com/apex/logs v0.0.4/go.mod h1:XzxuLZ5myVHDy9SAmYpamKKRNApGj54PfYLcFrXqDwo=
github.com/aphistic/golf v0.0.0-20180712155816-02c07f170c5a/go.mod h1:3NqKYiepwy8kCu4PNA+aP7WUV72eXWJeP9/r3/K9aLE=
github.com/aphistic/sweet v0.2.0/go.mod h1:fWDlIh/isSE9n6EPsRmC0det+whmX6dJid3stzu0Xys=
github.com/apparentlymart/go-textseg v1.0.0/go.mod h1:z96Txxhf3xSFMPmb5X/1W05FF/Nj9VFpLOpjS5yuumk=
github.com/apparentlymart/go-textseg/v13 v13.0.0/go.mod h1:ZK2fH7c4NqDTLtiYLvIkEghdlcqw7yxLeM89kiTRPUo=
github.com/armon/circbuf v0.0.0-20150827004946-bbbad097214e/go.mod h1:3U/XgcO3hCbHZ8TKRvWD2dDTCfh9M9ya+I9JpbB7O8o=
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
github.com/armon/go-metrics v0.0.0-20180917152333-f0300d1749da/go.mod h1:Q73ZrmVTwzkszR9V5SSuryQ31EELlFMUz1kKyl939pY=
@ -673,14 +675,21 @@ github.com/hashicorp/consul/api v1.3.0/go.mod h1:MmDNSzIMUjNpY/mQ398R4bk2FnqQLoP
github.com/hashicorp/consul/sdk v0.1.1/go.mod h1:VKf9jXwCTEY1QZP2MOLRhb5i/I/ssyNV1vwHyQBF0x8=
github.com/hashicorp/consul/sdk v0.3.0/go.mod h1:VKf9jXwCTEY1QZP2MOLRhb5i/I/ssyNV1vwHyQBF0x8=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.1.0 h1:OxrOeh75EUXMY8TBjag2fzXGZ40LB6IKw45YeGUDY2I=
github.com/hashicorp/errwrap v1.1.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/go-checkpoint v0.5.0/go.mod h1:7nfLNL10NsxqO4iWuW6tWW0HjZuDrwkBuEQsVcpCOgg=
github.com/hashicorp/go-cleanhttp v0.5.0/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
github.com/hashicorp/go-cleanhttp v0.5.1/go.mod h1:JpRdi6/HCYpAwUzNwuwqhbovhLtngrth3wmdIIUrZ80=
github.com/hashicorp/go-cleanhttp v0.5.2 h1:035FKYIWjmULyFRBKPs8TBQoi0x6d9G4xc9neXJWAZQ=
github.com/hashicorp/go-cleanhttp v0.5.2/go.mod h1:kO/YDlP8L1346E6Sodw+PrpBSV4/SoxCXGY6BqNFT48=
github.com/hashicorp/go-hclog v0.0.0-20180709165350-ff2cf002a8dd/go.mod h1:9bjs9uLqI8l75knNv3lV1kA55veR+WUPSiKIWcQHudI=
github.com/hashicorp/go-hclog v0.8.0/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
github.com/hashicorp/go-hclog v0.9.2/go.mod h1:5CU+agLiy3J7N7QjHK5d05KxGsuXiQLrjA0H7acj2lQ=
github.com/hashicorp/go-immutable-radix v1.0.0/go.mod h1:0y9vanUI8NX6FsYoO3zeMjhV/C5i9g4Q3DwcSNZ4P60=
github.com/hashicorp/go-msgpack v0.5.3/go.mod h1:ahLV/dePpqEmjfWmKiqvPkv/twdG7iPBM1vqhUKIvfM=
github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk=
github.com/hashicorp/go-multierror v1.1.1 h1:H5DkEtf6CXdFp0N0Em5UCwQpXMWke8IA0+lD48awMYo=
github.com/hashicorp/go-multierror v1.1.1/go.mod h1:iw975J/qwKPdAO1clOe2L8331t/9/fmwbPZ6JB6eMoM=
github.com/hashicorp/go-plugin v1.0.1/go.mod h1:++UyYGoz3o5w9ZzAdZxtQKrWWP+iqPBn3cQptSMzBuY=
github.com/hashicorp/go-retryablehttp v0.5.4/go.mod h1:9B5zBasrRhHXnJnui7y6sL7es7NDiJgTc6Er0maI1Xs=
github.com/hashicorp/go-retryablehttp v0.6.4/go.mod h1:vAew36LZh98gCBJNLH42IQ1ER/9wtLZZ8meHqQvEYWY=
@ -693,14 +702,23 @@ github.com/hashicorp/go-uuid v1.0.0/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/b
github.com/hashicorp/go-uuid v1.0.1/go.mod h1:6SBZvOh/SIDV7/2o3Jml5SYk/TvGqwFJ/bN7x4byOro=
github.com/hashicorp/go-version v1.1.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/go-version v1.2.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/go-version v1.5.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/go-version v1.6.0 h1:feTTfFNnjP967rlCxM/I9g701jU+RN74YKx2mOkIeek=
github.com/hashicorp/go-version v1.6.0/go.mod h1:fltr4n8CU8Ke44wwGCBoEymUuxUHl09ZGVZPK5anwXA=
github.com/hashicorp/go.net v0.0.1/go.mod h1:hjKkEWcCURg++eb33jQU7oqQcI9XDCnUzHA0oac0k90=
github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
github.com/hashicorp/hc-install v0.4.0 h1:cZkRFr1WVa0Ty6x5fTvL1TuO1flul231rWkGH92oYYk=
github.com/hashicorp/hc-install v0.4.0/go.mod h1:5d155H8EC5ewegao9A4PUTMNPZaq+TbOzkJJZ4vrXeI=
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
github.com/hashicorp/logutils v1.0.0/go.mod h1:QIAnNjmIWmVIIkWDTG1z5v++HQmx9WQRO+LraFDTW64=
github.com/hashicorp/mdns v1.0.0/go.mod h1:tL+uN++7HEJ6SQLQ2/p+z2pH24WQKWjBPkE0mNTz8vQ=
github.com/hashicorp/memberlist v0.1.3/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2pPBoIllUwCN7I=
github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc=
github.com/hashicorp/terraform-exec v0.17.3 h1:MX14Kvnka/oWGmIkyuyvL6POx25ZmKrjlaclkx3eErU=
github.com/hashicorp/terraform-exec v0.17.3/go.mod h1:+NELG0EqQekJzhvikkeQsOAZpsw0cv/03rbeQJqscAI=
github.com/hashicorp/terraform-json v0.14.0 h1:sh9iZ1Y8IFJLx+xQiKHGud6/TSUCM0N8e17dKDpqV7s=
github.com/hashicorp/terraform-json v0.14.0/go.mod h1:5A9HIWPkk4e5aeeXIBbkcOvaZbIYnAIkEyqP2pNSckM=
github.com/hashicorp/vault/api v1.0.4/go.mod h1:gDcqh3WGcR1cpF5AJz/B1UFheUEneMoIospckxBxk6Q=
github.com/hashicorp/vault/sdk v0.1.13/go.mod h1:B+hVj7TpuQY1Y/GPbCpffmgd+tSEwvhkWnjtSYCaS2M=
github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb/go.mod h1:+NfK9FKeTrX5uv1uIXGdwYDTeHna2qgaIlx54MXqjAM=
@ -782,6 +800,7 @@ github.com/kr/pty v1.1.8/go.mod h1:O1sed60cT9XZ5uDucP5qwvh+TE3NnUj51EiZO/lmSfw=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348/go.mod h1:B69LEHPfb2qLo0BaaOLcbitczOKLWTsrBG9LczfCD4k=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/leodido/go-urn v1.1.0/go.mod h1:+cyI34gQWZcE1eQU7NVgKkkzdXDQHr1dBMtdAPozLkw=
@ -836,6 +855,7 @@ github.com/miekg/pkcs11 v1.0.2/go.mod h1:XsNlhZGX73bx86s2hdc/FuaLm2CPZJemRLMA+WT
github.com/miekg/pkcs11 v1.0.3/go.mod h1:XsNlhZGX73bx86s2hdc/FuaLm2CPZJemRLMA+WTFxgs=
github.com/mitchellh/cli v1.0.0/go.mod h1:hNIlj7HEI86fIcpObd7a0FcrxTWetlwJDGcceTlRvqc=
github.com/mitchellh/copystructure v1.0.0/go.mod h1:SNtv71yrdKgLRyLFxmLdkAbkKEFWgYaq1OVrnRcwhnw=
github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s=
github.com/mitchellh/go-homedir v1.0.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
@ -848,6 +868,7 @@ github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee/go.mod h1:F
github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/mitchellh/reflectwalk v1.0.0/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
github.com/mitchellh/reflectwalk v1.0.1/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
github.com/mitchellh/reflectwalk v1.0.2/go.mod h1:mSTlrgnPZtwu0c4WaC2kGObEpuNDbx0jmZXqmk4esnw=
github.com/moby/spdystream v0.2.0/go.mod h1:f7i0iNDQJ059oMTcWxx8MA/zKFIuD/lY+0GqbN2Wy8c=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
@ -1001,6 +1022,7 @@ github.com/ryanuber/go-glob v1.0.0/go.mod h1:807d1WSdnB0XRJzKNil9Om6lcp/3a0v4qIH
github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da/go.mod h1:gi+0XIa01GRL2eRQVjQkKGqKF3SF9vZR/HnPullcV2E=
github.com/sassoftware/go-rpmutils v0.0.0-20190420191620-a8f1baeba37b/go.mod h1:am+Fp8Bt506lA3Rk3QCmSqmYmLMnPDhdDUcosQCAx+I=
github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
github.com/sebdah/goldie v1.0.0/go.mod h1:jXP4hmWywNEwZzhMuv2ccnqTSFpuq8iyQhtQdkkZBH4=
github.com/sergi/go-diff v1.0.0/go.mod h1:0CfEIISq7TuYL3j771MWULgwwjU+GofnZX9QAmXWZgo=
github.com/sergi/go-diff v1.1.0/go.mod h1:STckp+ISIX8hZLjrqAeVduY0gWCT9IjLuqbuNXdaHfM=
github.com/sergi/go-diff v1.2.0 h1:XU+rvMAioB0UC3q1MFrIQy4Vo5/4VsRDQQXHsEya6xQ=
@ -1089,8 +1111,12 @@ github.com/ulikunitz/xz v0.5.7/go.mod h1:nbz6k7qbPmH4IRqmfOplQw/tblSgqTqBwxkY0oW
github.com/urfave/cli v1.20.0/go.mod h1:70zkFmudgCuE/ngEzBv17Jvp/497gISqfk5gWijbERA=
github.com/urfave/cli v1.22.1/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
github.com/urfave/cli v1.22.4/go.mod h1:Gos4lmkARVdJ6EkW0WaNv/tZAAMe9V7XWyB60NtXRu0=
github.com/vmihailenco/msgpack v3.3.3+incompatible h1:wapg9xDUZDzGCNFlwc5SqI1rvcciqcxEHac4CYj89xI=
github.com/vmihailenco/msgpack v3.3.3+incompatible/go.mod h1:fy3FlTQTDXWkZ7Bh6AcGMlsjHatGryHQYUTf1ShIgkk=
github.com/vmihailenco/msgpack/v4 v4.3.12 h1:07s4sz9IReOgdikxLTKNbBdqDMLsjPKXwvCazn8G65U=
github.com/vmihailenco/msgpack/v4 v4.3.12/go.mod h1:gborTTJjAo/GWTqqRjrLCn9pgNN+NXzzngzBKDPIqw4=
github.com/vmihailenco/tagparser v0.1.1 h1:quXMXlA39OCbd2wAdTsGDlK9RkOk6Wuw+x37wVyIuWY=
github.com/vmihailenco/tagparser v0.1.1/go.mod h1:OeAg3pn3UbLjkWt+rN9oFYB6u/cQgqMEUPoW2WPyhdI=
github.com/xanzy/go-gitlab v0.31.0/go.mod h1:sPLojNBn68fMUWSxIJtdVVIP8uSBYqesTfDUseX11Ug=
github.com/xanzy/ssh-agent v0.2.1/go.mod h1:mLlQY/MoOhWBj+gOGMQkOeiEvkx+8pJSI+0Bx9h2kr4=
github.com/xanzy/ssh-agent v0.3.0 h1:wUMzuKtKilRgBAD1sUb8gOwwRr2FGoBVumcjoOACClI=
@ -1103,6 +1129,11 @@ github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9de
github.com/yuin/goldmark v1.1.32/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
github.com/zclconf/go-cty v1.2.0/go.mod h1:hOPWgoHbaTUnI5k4D2ld+GRpFJSCe6bCM7m1q/N4PQ8=
github.com/zclconf/go-cty v1.10.0/go.mod h1:vVKLxnk3puL4qRAv72AO+W99LUD4da90g3uUAzyuvAk=
github.com/zclconf/go-cty v1.11.0 h1:726SxLdi2SDnjY+BStqB9J1hNp4+2WlzyXLuimibIe0=
github.com/zclconf/go-cty v1.11.0/go.mod h1:s9IfD1LK5ccNMSWCVFCE2rJfHiZgi7JijgeWIMfhLvA=
github.com/zclconf/go-cty-debug v0.0.0-20191215020915-b22d67c1ba0b/go.mod h1:ZRKQfBXbGkpdV6QMzT3rU1kSTAnfu1dO8dPKjYprgj8=
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.3/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
go.etcd.io/bbolt v1.3.4/go.mod h1:G5EMThwa9y8QZGBClrRx5EY+Yw9kAhnjy3bSjsnlVTQ=
@ -1203,6 +1234,7 @@ golang.org/x/crypto v0.0.0-20201002170205-7f63de1d35b0/go.mod h1:LzIPMQfyMNhhGPh
golang.org/x/crypto v0.0.0-20210314154223-e6e6c4f2bb5b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210421170649-83a5a9bb288b/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20211108221036-ceb1ce70b4fa/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20211215153901-e495a2d5b3d3/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
@ -1249,6 +1281,7 @@ golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 h1:kQgndtyPBW/JIYERgdxfwMYh3AVStj88WQTlNDi2a+o=
golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3/go.mod h1:3p9vT2HGsQu2K1YbXdKPJLVgG5VJdoTa1poYQBtP1AY=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180811021610-c39426892332/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20181023162649-9b4f9f5ad519/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=

View File

@ -19,6 +19,7 @@ import (
func main() {
bindPort := flag.String("port", "8080", "Port to bind to")
targetNetwork := flag.String("network", "constellation-network", "Name of the network in QEMU to use")
flag.Parse()
log := logger.New(logger.JSONLog, zapcore.InfoLevel)
@ -29,7 +30,7 @@ func main() {
}
defer conn.Close()
serv := server.New(log, &virtwrapper.Connect{Conn: conn})
serv := server.New(log, *targetNetwork, &virtwrapper.Connect{Conn: conn})
if err := serv.ListenAndServe(*bindPort); err != nil {
log.With(zap.Error(err)).Fatalf("Failed to serve")
}

View File

@ -24,12 +24,14 @@ import (
type Server struct {
log *logger.Logger
virt virConnect
network string
}
func New(log *logger.Logger, conn virConnect) *Server {
func New(log *logger.Logger, network string, conn virConnect) *Server {
return &Server{
log: log,
virt: conn,
network: network,
}
}
@ -186,7 +188,7 @@ func (s *Server) exportPCRs(w http.ResponseWriter, r *http.Request) {
// listAll returns a list of all active peers.
func (s *Server) listAll() ([]metadata.InstanceMetadata, error) {
net, err := s.virt.LookupNetworkByName("constellation")
net, err := s.virt.LookupNetworkByName(s.network)
if err != nil {
return nil, err
}

View File

@ -71,7 +71,7 @@ func TestListAll(t *testing.T) {
t.Run(name, func(t *testing.T) {
assert := assert.New(t)
server := New(logger.NewTest(t), tc.connect)
server := New(logger.NewTest(t), "test", tc.connect)
res, err := server.listAll()
@ -148,7 +148,7 @@ func TestListSelf(t *testing.T) {
assert := assert.New(t)
require := require.New(t)
server := New(logger.NewTest(t), tc.connect)
server := New(logger.NewTest(t), "test", tc.connect)
req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, "http://192.0.0.1/self", nil)
require.NoError(err)
@ -210,7 +210,7 @@ func TestListPeers(t *testing.T) {
assert := assert.New(t)
require := require.New(t)
server := New(logger.NewTest(t), tc.connect)
server := New(logger.NewTest(t), "test", tc.connect)
req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, "http://192.0.0.1/peers", nil)
require.NoError(err)
@ -265,7 +265,7 @@ func TestPostLog(t *testing.T) {
assert := assert.New(t)
require := require.New(t)
server := New(logger.NewTest(t), &stubConnect{})
server := New(logger.NewTest(t), "test", &stubConnect{})
req, err := http.NewRequestWithContext(context.Background(), tc.method, "http://192.0.0.1/logs", tc.message)
require.NoError(err)
@ -345,7 +345,7 @@ func TestExportPCRs(t *testing.T) {
assert := assert.New(t)
require := require.New(t)
server := New(logger.NewTest(t), tc.connect)
server := New(logger.NewTest(t), "test", tc.connect)
req, err := http.NewRequestWithContext(context.Background(), tc.method, "http://192.0.0.1/pcrs", strings.NewReader(tc.message))
require.NoError(err)

View File

@ -176,6 +176,24 @@ type GCPConfig struct {
}
type QEMUConfig struct {
// description: |
// Path to the image to use for the VMs.
Image string `yaml:"image" validate:"required"`
// description: |
// Format of the image to use for the VMs. Should be either qcow2 or raw.
ImageFormat string `yaml:"imageFormat" validate:"oneof=qcow2 raw"`
// description: |
// vCPU count for the VMs.
VCPUs int `yaml:"vcpus" validate:"required"`
// description: |
// Amount of memory per instance (MiB).
Memory int `yaml:"memory" validate:"required"`
// description: |
// First IP address to use within a node group's subnet.
IPRangeStart int `yaml:"ipRangeStart" validate:"required"`
// description: |
// Container image to use for the QEMU metadata server.
MetadataAPIImage string `yaml:"metadataAPIServer" validate:"required"`
// description: |
// Measurement used to enable measured boot.
Measurements Measurements `yaml:"measurements"`
@ -218,7 +236,12 @@ func Default() *Config {
EnforcedMeasurements: []uint32{0, 4, 8, 9, 11, 12},
},
QEMU: &QEMUConfig{
ImageFormat: "qcow2",
VCPUs: 2,
Memory: 2048,
IPRangeStart: 100,
Measurements: copyPCRMap(qemuPCRs),
MetadataAPIImage: "ghcr.io/edgelesssys/constellation/qemu-metadata-api:v2.1.0-pre.0.20220922072347-abb78344bc2a",
EnforcedMeasurements: []uint32{11, 12},
},
},

View File

@ -288,17 +288,47 @@ func init() {
FieldName: "qemu",
},
}
QEMUConfigDoc.Fields = make([]encoder.Doc, 2)
QEMUConfigDoc.Fields[0].Name = "measurements"
QEMUConfigDoc.Fields[0].Type = "Measurements"
QEMUConfigDoc.Fields = make([]encoder.Doc, 8)
QEMUConfigDoc.Fields[0].Name = "image"
QEMUConfigDoc.Fields[0].Type = "string"
QEMUConfigDoc.Fields[0].Note = ""
QEMUConfigDoc.Fields[0].Description = "Measurement used to enable measured boot."
QEMUConfigDoc.Fields[0].Comments[encoder.LineComment] = "Measurement used to enable measured boot."
QEMUConfigDoc.Fields[1].Name = "enforcedMeasurements"
QEMUConfigDoc.Fields[1].Type = "[]uint32"
QEMUConfigDoc.Fields[0].Description = "Path to the image to use for the VMs."
QEMUConfigDoc.Fields[0].Comments[encoder.LineComment] = "Path to the image to use for the VMs."
QEMUConfigDoc.Fields[1].Name = "imageFormat"
QEMUConfigDoc.Fields[1].Type = "string"
QEMUConfigDoc.Fields[1].Note = ""
QEMUConfigDoc.Fields[1].Description = "List of values that should be enforced to be equal to the ones from the measurement list. Any non-equal values not in this list will only result in a warning."
QEMUConfigDoc.Fields[1].Comments[encoder.LineComment] = "List of values that should be enforced to be equal to the ones from the measurement list. Any non-equal values not in this list will only result in a warning."
QEMUConfigDoc.Fields[1].Description = "Format of the image to use for the VMs. Should be either qcow2 or raw."
QEMUConfigDoc.Fields[1].Comments[encoder.LineComment] = "Format of the image to use for the VMs. Should be either qcow2 or raw."
QEMUConfigDoc.Fields[2].Name = "vcpus"
QEMUConfigDoc.Fields[2].Type = "int"
QEMUConfigDoc.Fields[2].Note = ""
QEMUConfigDoc.Fields[2].Description = "vCPU count for the VMs."
QEMUConfigDoc.Fields[2].Comments[encoder.LineComment] = "vCPU count for the VMs."
QEMUConfigDoc.Fields[3].Name = "memory"
QEMUConfigDoc.Fields[3].Type = "int"
QEMUConfigDoc.Fields[3].Note = ""
QEMUConfigDoc.Fields[3].Description = "Amount of memory per instance (MiB)."
QEMUConfigDoc.Fields[3].Comments[encoder.LineComment] = "Amount of memory per instance (MiB)."
QEMUConfigDoc.Fields[4].Name = "ipRangeStart"
QEMUConfigDoc.Fields[4].Type = "int"
QEMUConfigDoc.Fields[4].Note = ""
QEMUConfigDoc.Fields[4].Description = "First IP address to use within a node group's subnet."
QEMUConfigDoc.Fields[4].Comments[encoder.LineComment] = "First IP address to use within a node group's subnet."
QEMUConfigDoc.Fields[5].Name = "metadataAPIServer"
QEMUConfigDoc.Fields[5].Type = "string"
QEMUConfigDoc.Fields[5].Note = ""
QEMUConfigDoc.Fields[5].Description = "Container image to use for the QEMU metadata server."
QEMUConfigDoc.Fields[5].Comments[encoder.LineComment] = "Container image to use for the QEMU metadata server."
QEMUConfigDoc.Fields[6].Name = "measurements"
QEMUConfigDoc.Fields[6].Type = "Measurements"
QEMUConfigDoc.Fields[6].Note = ""
QEMUConfigDoc.Fields[6].Description = "Measurement used to enable measured boot."
QEMUConfigDoc.Fields[6].Comments[encoder.LineComment] = "Measurement used to enable measured boot."
QEMUConfigDoc.Fields[7].Name = "enforcedMeasurements"
QEMUConfigDoc.Fields[7].Type = "[]uint32"
QEMUConfigDoc.Fields[7].Note = ""
QEMUConfigDoc.Fields[7].Description = "List of values that should be enforced to be equal to the ones from the measurement list. Any non-equal values not in this list will only result in a warning."
QEMUConfigDoc.Fields[7].Comments[encoder.LineComment] = "List of values that should be enforced to be equal to the ones from the measurement list. Any non-equal values not in this list will only result in a warning."
}
func (_ Config) Doc() *encoder.Doc {

View File

@ -155,7 +155,7 @@ func TestFromFileStrictErrors(t *testing.T) {
}
func TestValidate(t *testing.T) {
const defaultMsgCount = 14 // expect this number of error messages by default because user-specific values are not set and multiple providers are defined by default
const defaultMsgCount = 15 // expect this number of error messages by default because user-specific values are not set and multiple providers are defined by default
testCases := map[string]struct {
cnf *Config

View File

@ -157,6 +157,11 @@ func (h *Handler) Remove(name string) error {
return h.fs.Remove(name)
}
// RemoveAll deletes the file or directory with the given name.
func (h *Handler) RemoveAll(name string) error {
return h.fs.RemoveAll(name)
}
// Stat returns a FileInfo describing the named file, or an error, if any
// happens.
func (h *Handler) Stat(name string) (fs.FileInfo, error) {

View File

@ -1,121 +0,0 @@
# Auotmated local image testing with QEMU / libvirt / terraform
## Usage
Prerequisite:
- [qcow2 constellation image](/image/README.md)
- [setup](#setup-libvirt--terraform)
- [qemu-metadata-api](/hack/qemu-metadata-api/README.md)
Optional: Write a `terraform.tfvars` file in the terraform workspace (`terraform/libvirt`), defining required variables and overriding optional variables.
See [variables.tf](./variables.tf) for a description of all available variables.
```tfvars
constellation_coreos_image="/path/to/image.qcow2"
# optional other vars, uncomment and change as needed
# control_plane_count=3
# worker_count=2
# vcpus=2
# memory=2048
# state_disk_size=10
# ip_range_start=100
# machine="q35"
```
Create terraform resources from within terraform workspace (`terraform/libvirt`):
```shell-session
cd terraform/libvirt
terraform init
terraform plan
terraform apply
# set CONST_DIR to your constellation workspace
export TF_DIR=$(pwd)
export CONST_DIR=$(pwd)
go run ../../hack/terraform-to-state/create-state.go "${TF_DIR}" "${CONST_DIR}"
# use constellation (everything after constellation create)
constellation config generate qemu
# run cdbg if using a debug image
cdbg deploy
constellation init
# cleanup
rm constellation-state.json constellation-mastersecret.base64 constellation-admin.conf wg0.conf
terraform destroy
```
## Setup libvirt & Terraform
<details>
<summary>Ubuntu</summary>
[General reference](https://ubuntu.com/server/docs/virtualization-libvirt)
```shell-session
# Install Terraform
curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
sudo apt-get update && sudo apt-get install terraform
# install libvirt, KVM and tools
sudo apt install qemu-kvm libvirt-daemon-system xsltproc
sudo systemctl enable libvirtd
sudo usermod -a -G libvirt $USER
# reboot
```
</details>
<details>
<summary>Fedora</summary>
```shell-session
sudo dnf install -y dnf-plugins-core
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/fedora/hashicorp.repo
sudo dnf -y install terraform qemu-kvm libvirt-daemon-config-network libvirt-daemon-kvm xsltproc
sudo usermod -a -G libvirt $USER
# reboot
```
</details>
## Change libvirt settings (on Ubuntu)
Open `/etc/libvirt/qemu.conf` and change the following settings:
```
security_driver = "none"
```
Then restart libvirt
```shell-session
sudo systemctl restart libvirtd
```
## Setup emulated TPM (on Ubuntu)
Only works if swtpm is version 0.7 or newer!
Ubuntu currently ships swtpm 0.6.3 so you need to install swtpm [from launchpad](https://launchpad.net/~stefanberger/+archive/ubuntu/swtpm-jammy/).
1. Uninstall current version of swtpm (if installed)
```
sudo apt remove swtpm swtpm-tools
```
2. Add ppa (this command shows the ppa for Ubuntu 22.04 jammy but others are available)
```
sudo add-apt-repository ppa:stefanberger/swtpm-jammy
sudo apt update
```
3. Install swtpm
```
sudo apt install swtpm swtpm-tools
```
4. Patch configuration under `/etc/swtpm_setup.conf`
```
# Program invoked for creating certificates
create_certs_tool = /usr/bin/swtpm_localca
```
5. Patch ownership of `/var/lib/swtpm-localca`
```shell-session
sudo chown -R swtpm:root /var/lib/swtpm-localca
```
## Misc
- List all domains: `virsh list --all`
- Destroy domain with nvram: `virsh undefine --nvram <name>`

View File

@ -1,7 +0,0 @@
output "control_plane_ips" {
value = module.control_plane.instance_ips
}
output "worker_ips" {
value = module.worker.instance_ips
}