Commit Graph

216 Commits

Author SHA1 Message Date
Daniel Weiße
93b70e5633
cli: revert libvirt terraform provider upgrade (#2454)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-10-16 10:43:30 +02:00
renovate[bot]
0f5039f2e9
deps: update Terraform libvirt to v0.7.4 (#2434)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-10-11 13:47:51 +02:00
Moritz Sanft
005e865a13
cli: use state file on init and upgrade (#2395)
* [wip] use state file in CLI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use state file in CLI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

take clusterConfig from IDFile for compat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

various fixes

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

wip

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add GCP-specific values in Helm loader test

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove unnecessary pointer

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* write ClusterValues in one step

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move stub to test file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove mention of id-file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move output to `migrateTerraform`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* unconditional assignments converting from idFile

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move require block in go modules file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fall back to id file on upgrade

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix linter check

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add notice to remove Terraform state check on manual migration

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add `name` field

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

fix name tests

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* return early if no Terraform diff

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* return infrastructure state even if no diff exists

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add TODO to remove comment

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use state-file in miniconstellation

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* cli: remove id-file (#2402)

* remove id-file from `constellation create`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add file renaming to handler

* rename id-file after upgrade

* use idFile on `constellation init`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation verify`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* linter fixes

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation mini`

* remove id-file from `constellation recover`

* linter fixes

* remove id-file from `constellation terminate`

* fix initSecret type

* fix recover argument precedence

* fix terminate test

* generate

* add TODO to remove id-file removal

* Update cli/internal/cmd/init.go

Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>

* fix verify arg parse logic

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add version test

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from docs

* add file not found log

* use state-file in miniconstellation

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation iam destroy`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `cdbg deploy`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>

* use state-file in CI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update orchestration docs

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>
2023-10-09 13:04:29 +02:00
3u13r
1452e64675
Refactor Terraform to have all ports in a list (#2409)
* terraform: aws refactoring

* terraform: gcp refactoring

* terraform: azure refactoring
2023-10-05 12:34:02 +02:00
Malte Poll
daa5b51904 terraform: disable secure boot for GCP 2023-09-27 17:58:19 +02:00
Malte Poll
4e2b9745bb terraform: disable secure boot for QEMU / MiniConstellation 2023-09-27 17:58:19 +02:00
renovate[bot]
6445c11110
deps: update Terraform aws to v5.17.0 (#2366)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-09-26 15:04:32 +02:00
renovate[bot]
5cfa91f3db
deps: update Terraform azurerm to v3.74.0 (#2368)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-09-26 13:17:17 +02:00
renovate[bot]
dffdc37004
deps: update Terraform azuread to v2.43.0 (#2367)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-09-26 11:48:55 +02:00
Adrian Stobbe
4680882708
cli: remove TF ApplyOutput dependency in CLI (#2323) 2023-09-25 17:10:23 +02:00
Adrian Stobbe
322c4aad10
cli: write infrastructure to new state file (#2321)
Co-authored-by: 3u13r <lc@edgeless.systems>
2023-09-25 16:19:43 +02:00
3u13r
2776e40df7
join: join over lb if available (#2348)
* join: join over lb if available
2023-09-25 10:23:35 +02:00
renovate[bot]
74fe8af8e6
deps: update Terraform google-beta to v4.83.0 new (#2345)
Co-authored-by: Leonard Cohnen <lc@edgeless.systems>
2023-09-22 10:37:19 +02:00
Moritz Sanft
dc837eddb5
cli: temporarily increase AWS ASG creation timeout (#2340)
* increase ASG timeout

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* make timeout dependent on SEV-SNP option

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
2023-09-20 11:25:02 +02:00
renovate[bot]
82e561f139
deps: update Terraform google to v4.83.0 (#2344) 2023-09-19 15:17:21 +02:00
Moritz Sanft
95cf4bdf21
cli: perform upgrades in-place in Terraform workspace (#2317)
* perform upgrades in-place in terraform workspace

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update buildfiles

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add iam upgrade apply test

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update buildfiles

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix linter

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* make config fetcher stubbable

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* change workspace restoring behaviour

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* allow overwriting existing Terraform files

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* allow overwrites of TF variables

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix iam upgrade apply

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix embed directive

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* make loader test less brittle

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* pass upgrade ID to user

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* naming nit

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use upgradeDir

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
2023-09-14 11:51:20 +02:00
3u13r
a25c90e9bb
remove deprecated constellation create flags (#2325)
* chore: clean-up TODOs

* cli: make OpenStack error explicit

* cli: remove deprecated flags

* config: require DeployCSIDriver field
2023-09-08 21:15:02 +02:00
Otto Bittner
6e5ba774d8 cli: disable nosmt via VMM temporarily.
AWS asked us to disable these options temporarily until they resolve
some internal issues that sometimes prevents these instances
from starting.
2023-09-05 08:23:18 +02:00
Otto Bittner
75ce11af14
cli: disable smt via cpu_options (#2291)
Disabling SMT dynamically inside the image creates problems on AWS.
The problem should be fixed by disabling smt through the VMM.
By recommendation from AWS: add idle=poll.
This should improve our launch success rate while they investigate some
upstream issues.
2023-09-01 11:26:21 +02:00
Daniel Weiße
0a911806d1
cli: remove/refactor upgrade package (#2266)
* Move IAM migration client to cloudcmd package

* Move Terraform Cluster upgrade client to cloudcmd package

* Use hcl for creating Terraform IAM variables files

* Unify terraform upgrade code

* Rename some cloudcmd files for better clarity

---------

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-23 10:35:42 +02:00
Daniel Weiße
9477999be2
cli: clean up terraform package (#2256)
* Clean up Terraform pkg

* Add note to Terraform migration functions expecting to be run on initialized workspace

---------

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-21 10:26:53 +02:00
renovate[bot]
ae7888a13f
deps: update Terraform azuread to v2.41.0 (#2254)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-08-17 10:29:49 +02:00
Thomas Tendyck
587ae6a575
deps: limit Terraform version to FOSS releases (#2241)
* deps: limit Terraform version to FOSS releases

* fix: enforce upper version constraint

---------

Co-authored-by: Leonard Cohnen <lc@edgeless.systems>
2023-08-16 23:25:53 +02:00
3u13r
8c321ec1ab
cli: add role to aws instance name (#2130) 2023-08-14 13:42:20 +02:00
Daniel Weiße
e30179a8aa Remove manual state migration steps for AWS
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-10 15:46:59 +02:00
Daniel Weiße
99c579b45a Add package design goals to CLI package documentation
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-09 15:42:24 +02:00
Daniel Weiße
23394ea2e2
cli: fix missing safety check in ShowIAM (#2165)
* Add missing safety check to ShowIAM

* someErr->assert.AnError

---------

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-09 15:25:59 +02:00
Daniel Weiße
d1ace13713
cli: add --workspace flag to set base directory for Constellation workspace (#2148)
* Remove `--config` and `--master-secret` falgs

* Add `--workspace` flag

* In CLI, only work on files with paths created from `cli/internal/cmd`

* Properly print values for GCP on IAM create when not directly updating the config

---------

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-04 13:53:51 +02:00
Malte Poll
0c20ccb477 terraform: create nodeGroups in tfvars from nodeGroups in config 2023-08-04 12:36:45 +02:00
Malte Poll
d0ec7a3e54 terraform: move OpenStack flavorID into nodeGroups 2023-08-04 12:36:45 +02:00
3u13r
720c48ea45
cli: refactor terraform output parsing (#2158) 2023-08-03 16:17:23 +02:00
Adrian Stobbe
70ce195a5f
cli: unify chart value setup (#2153) 2023-08-03 13:54:48 +02:00
3u13r
5119d843f1
terraform: fix uami parsing (#2155) 2023-08-03 13:22:26 +02:00
renovate[bot]
7e3123232e
deps: update Terraform azurerm to v3.67.0 (#2147)
Also update Azure terraform: 
ignore snp policy changes on resource

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: edgelessci <edgelessci@users.noreply.github.com>
Co-authored-by: Otto Bittner <cobittner@posteo.net>
2023-08-02 08:15:22 +02:00
Adrian Stobbe
26480016a9
azure: fix ccm config with correct uami client_id (#2144)
* fix azure ccm config with correct uami client_id

* fix tests
2023-08-01 08:40:44 +02:00
Adrian Stobbe
26305e8f80
cli: install helm charts in cli instead of bootstrapper (#2136)
* init

* fixup! init

* gcp working?

* fixup! fixup! init

* azure cfg for microService installation

* fixup! azure cfg for microService installation

* fixup! azure cfg for microService installation

* cleanup bootstrapper code

* cleanup helminstall code

* fixup! cleanup helminstall code

* Update internal/deploy/helm/install.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* daniel feedback

* TODO add provider (also to CreateCluster) so we can ensure that provider specific output

* fixup! daniel feedback

* use debugLog in helm installer

* placeholderHelmInstaller

* rename to stub

---------

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
2023-07-31 10:53:05 +02:00
Otto Bittner
7db058f946
cli: wait for public ip update before reading fqdn (#2135) 2023-07-26 15:23:37 +02:00
Adrian Stobbe
a87b7894db
aws: use new LB controller to fix SecurityGroup cleanup on K8s service deletion (#2090)
* add current chart

add current helm chart

* disable service controller for aws ccm

* add new iam roles

* doc AWS internet LB + add to LB test

* pass clusterName to helm for AWS LB

* fix update-aws-lb chart to also include .helmignore

* move chart outside services

* working state

* add subnet tags for AWS subnet discovery

* fix .helmignore load rule with file in subdirectory

* upgrade iam profile

* revert new loader impl since cilium is not correctly loaded

* install chart if not already present during `upgrade apply`

* cleanup PR + fix build + add todos

cleanup PR + add todos

* shared helm pkg for cli install and bootstrapper

* add link to eks docs

* refactor iamMigrationCmd

* delete unused helm.symwallk

* move iammigrate to upgrade pkg

* fixup! delete unused helm.symwallk

* add to upgradecheck

* remove nodeSelector from go code (Otto)

* update iam docs and sort permission + remove duplicate roles

* fix bug in `upgrade check`

* better upgrade check output when svc version upgrade not possible

* pr feedback

* remove force flag in upgrade_test

* use upgrader.GetUpgradeID instead of extra type

* remove todos + fix check

* update doc lb (leo)

* remove bootstrapper helm package

* Update cli/internal/cmd/upgradecheck.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* final nits

* add docs for e2e upgrade test setup

* Apply suggestions from code review

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update cli/internal/helm/loader.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update cli/internal/cmd/tfmigrationclient.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* fix daniel review

* link to the iam permissions instead of manually updating them (agreed with leo)

* disable iam upgrade in upgrade apply

---------

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
Co-authored-by: Malte Poll
2023-07-24 10:30:53 +02:00
Malte Poll
8da6a23aa5
bootstrapper: add fallback endpoint and custom endpoint to SAN field (#2108)
terraform: collect apiserver cert SANs and support custom endpoint

constants: add new constants for cluster configuration and custom endpoint

cloud: support apiserver cert sans and prepare for endpoint migration on AWS

config: add customEndpoint field

bootstrapper: use per-CSP apiserver cert SANs

cli: route customEndpoint to terraform and add migration for apiserver cert SANs

bootstrapper: change interface of GetLoadBalancerEndpoint to return host and port separately
2023-07-21 16:43:51 +02:00
Otto Bittner
cf822f7eee
cli: unify terraform variable creation (#2119)
Before we defined the variables twice.
Once for upgrades, once for create.
Also move default node group names into a constant
2023-07-21 10:04:29 +02:00
Otto Bittner
c2849f4bbe
cli: ignore name changes on lb public ip resource (#2117)
Changing the name forces a recreate, which would break existing clusters.
The name change seems to be "only" about having clearer names.
2023-07-19 10:15:23 +02:00
Malte Poll
5cbdb3a519
terraform: allows cluster name length of 10 characters on AWS (#2110) 2023-07-17 17:45:41 +02:00
renovate[bot]
f364bd6b9d
deps: update Terraform google-beta to v4.72.0 (#2027)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-07-06 11:53:40 +02:00
renovate[bot]
3f1faead94
deps: update Terraform google to v4.72.0 (#2026)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-07-06 11:53:15 +02:00
renovate[bot]
ff74afa00d
deps: update Terraform azurerm to v3.63.0 (#2025)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-07-06 11:52:44 +02:00
renovate[bot]
bf09313dcf
deps: update Terraform aws to v5.6.2 (#2024)
Co-authored-by: Leonard Cohnen <lc@edgeless.systems>
2023-07-06 10:21:35 +02:00
renovate[bot]
e6dbb13c6c
deps: update Terraform openstack to v1.52.1 (#2028)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-07-05 16:52:58 +02:00
Adrian Stobbe
c39df2f7da
terraform: openstack node groups (#1966)
* openstack

* rename to base_name

* fix openstack boot vtpm

* add docs for accessing bootstrapper logs

* rename to initial count
2023-07-03 16:33:00 +02:00
Malte Poll
66f1333c31
terraform: use single zone loadbalancer frontend on AWS (#1983)
This change is required to ensure we have not tls handshake errors when connecting to the kubernetes api.
Currently, the certificates used by kube-apiserver pods contain a SAN field with the (single) public ip of the loadbalancer.
If we would allow multiple loadbalancer frontend ips, we could encounter cases where the certificate is only valid for one public ip,
while we try to connect to a different ip.
To prevent this, we consciously disable support for the multi-zone loadbalancer frontend on AWS for now.
This will be re-enabled in the future.
2023-06-30 16:56:31 +02:00
Malte Poll
5f8ea1348a
terraform: instance_count => initial_count (#1989)
Normalize naming for the "instance_count" / "initial_count" int terraform to always use "initial_count".
This is required, since there is a naming confusion on AWS.
"initial_count" is more precise, since it reflects the fact that this value is ignored when applying the terraform template
after the scaling groups already exist.
2023-06-30 10:53:00 +02:00