Commit Graph

218 Commits

Author SHA1 Message Date
3u13r
0c89f57ac5
Support internal load balancers (#2388)
* arch: support internal lb on Azure

* arch: support internal lb on GCP

* helm: remove lb svc from verify deployment

* arch: support internal lb on AWS

* terraform: add jump hosts for internal lb

* cli: expose internalLoadBalancer in config

* ci: add e2e-manual-internal

* add in-cluster endpoint to terraform output
2023-10-17 15:46:15 +02:00
Moritz Sanft
a8605d7294
cli: use custom byte-slice marshalling for state file (#2460)
* custom byte slice marshalling

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* byte slice compatibility

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* other byte slice compat test

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add missing dep

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* export byte type alias

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* regenerate exported type

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* test marshal and unmarshal together

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
2023-10-17 10:35:54 +02:00
Daniel Weiße
93b70e5633
cli: revert libvirt terraform provider upgrade (#2454)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-10-16 10:43:30 +02:00
renovate[bot]
0f5039f2e9
deps: update Terraform libvirt to v0.7.4 (#2434)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-10-11 13:47:51 +02:00
Moritz Sanft
005e865a13
cli: use state file on init and upgrade (#2395)
* [wip] use state file in CLI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use state file in CLI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

take clusterConfig from IDFile for compat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

various fixes

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

wip

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add GCP-specific values in Helm loader test

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove unnecessary pointer

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* write ClusterValues in one step

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move stub to test file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove mention of id-file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move output to `migrateTerraform`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* unconditional assignments converting from idFile

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move require block in go modules file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fall back to id file on upgrade

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix linter check

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add notice to remove Terraform state check on manual migration

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add `name` field

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

fix name tests

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* return early if no Terraform diff

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* return infrastructure state even if no diff exists

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add TODO to remove comment

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use state-file in miniconstellation

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* cli: remove id-file (#2402)

* remove id-file from `constellation create`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add file renaming to handler

* rename id-file after upgrade

* use idFile on `constellation init`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation verify`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* linter fixes

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation mini`

* remove id-file from `constellation recover`

* linter fixes

* remove id-file from `constellation terminate`

* fix initSecret type

* fix recover argument precedence

* fix terminate test

* generate

* add TODO to remove id-file removal

* Update cli/internal/cmd/init.go

Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>

* fix verify arg parse logic

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add version test

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from docs

* add file not found log

* use state-file in miniconstellation

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation iam destroy`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `cdbg deploy`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>

* use state-file in CI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update orchestration docs

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>
2023-10-09 13:04:29 +02:00
3u13r
1452e64675
Refactor Terraform to have all ports in a list (#2409)
* terraform: aws refactoring

* terraform: gcp refactoring

* terraform: azure refactoring
2023-10-05 12:34:02 +02:00
Malte Poll
daa5b51904 terraform: disable secure boot for GCP 2023-09-27 17:58:19 +02:00
Malte Poll
4e2b9745bb terraform: disable secure boot for QEMU / MiniConstellation 2023-09-27 17:58:19 +02:00
renovate[bot]
6445c11110
deps: update Terraform aws to v5.17.0 (#2366)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-09-26 15:04:32 +02:00
renovate[bot]
5cfa91f3db
deps: update Terraform azurerm to v3.74.0 (#2368)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-09-26 13:17:17 +02:00
renovate[bot]
dffdc37004
deps: update Terraform azuread to v2.43.0 (#2367)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-09-26 11:48:55 +02:00
Adrian Stobbe
4680882708
cli: remove TF ApplyOutput dependency in CLI (#2323) 2023-09-25 17:10:23 +02:00
Adrian Stobbe
322c4aad10
cli: write infrastructure to new state file (#2321)
Co-authored-by: 3u13r <lc@edgeless.systems>
2023-09-25 16:19:43 +02:00
3u13r
2776e40df7
join: join over lb if available (#2348)
* join: join over lb if available
2023-09-25 10:23:35 +02:00
renovate[bot]
74fe8af8e6
deps: update Terraform google-beta to v4.83.0 new (#2345)
Co-authored-by: Leonard Cohnen <lc@edgeless.systems>
2023-09-22 10:37:19 +02:00
Moritz Sanft
dc837eddb5
cli: temporarily increase AWS ASG creation timeout (#2340)
* increase ASG timeout

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* make timeout dependent on SEV-SNP option

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
2023-09-20 11:25:02 +02:00
renovate[bot]
82e561f139
deps: update Terraform google to v4.83.0 (#2344) 2023-09-19 15:17:21 +02:00
Moritz Sanft
95cf4bdf21
cli: perform upgrades in-place in Terraform workspace (#2317)
* perform upgrades in-place in terraform workspace

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update buildfiles

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add iam upgrade apply test

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update buildfiles

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix linter

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* make config fetcher stubbable

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* change workspace restoring behaviour

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* allow overwriting existing Terraform files

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* allow overwrites of TF variables

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix iam upgrade apply

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix embed directive

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* make loader test less brittle

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* pass upgrade ID to user

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* naming nit

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use upgradeDir

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
2023-09-14 11:51:20 +02:00
3u13r
a25c90e9bb
remove deprecated constellation create flags (#2325)
* chore: clean-up TODOs

* cli: make OpenStack error explicit

* cli: remove deprecated flags

* config: require DeployCSIDriver field
2023-09-08 21:15:02 +02:00
Otto Bittner
6e5ba774d8 cli: disable nosmt via VMM temporarily.
AWS asked us to disable these options temporarily until they resolve
some internal issues that sometimes prevents these instances
from starting.
2023-09-05 08:23:18 +02:00
Otto Bittner
75ce11af14
cli: disable smt via cpu_options (#2291)
Disabling SMT dynamically inside the image creates problems on AWS.
The problem should be fixed by disabling smt through the VMM.
By recommendation from AWS: add idle=poll.
This should improve our launch success rate while they investigate some
upstream issues.
2023-09-01 11:26:21 +02:00
Daniel Weiße
0a911806d1
cli: remove/refactor upgrade package (#2266)
* Move IAM migration client to cloudcmd package

* Move Terraform Cluster upgrade client to cloudcmd package

* Use hcl for creating Terraform IAM variables files

* Unify terraform upgrade code

* Rename some cloudcmd files for better clarity

---------

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-23 10:35:42 +02:00
Daniel Weiße
9477999be2
cli: clean up terraform package (#2256)
* Clean up Terraform pkg

* Add note to Terraform migration functions expecting to be run on initialized workspace

---------

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-21 10:26:53 +02:00
renovate[bot]
ae7888a13f
deps: update Terraform azuread to v2.41.0 (#2254)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-08-17 10:29:49 +02:00
Thomas Tendyck
587ae6a575
deps: limit Terraform version to FOSS releases (#2241)
* deps: limit Terraform version to FOSS releases

* fix: enforce upper version constraint

---------

Co-authored-by: Leonard Cohnen <lc@edgeless.systems>
2023-08-16 23:25:53 +02:00
3u13r
8c321ec1ab
cli: add role to aws instance name (#2130) 2023-08-14 13:42:20 +02:00
Daniel Weiße
e30179a8aa Remove manual state migration steps for AWS
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-10 15:46:59 +02:00
Daniel Weiße
99c579b45a Add package design goals to CLI package documentation
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-09 15:42:24 +02:00
Daniel Weiße
23394ea2e2
cli: fix missing safety check in ShowIAM (#2165)
* Add missing safety check to ShowIAM

* someErr->assert.AnError

---------

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-09 15:25:59 +02:00
Daniel Weiße
d1ace13713
cli: add --workspace flag to set base directory for Constellation workspace (#2148)
* Remove `--config` and `--master-secret` falgs

* Add `--workspace` flag

* In CLI, only work on files with paths created from `cli/internal/cmd`

* Properly print values for GCP on IAM create when not directly updating the config

---------

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-04 13:53:51 +02:00
Malte Poll
0c20ccb477 terraform: create nodeGroups in tfvars from nodeGroups in config 2023-08-04 12:36:45 +02:00
Malte Poll
d0ec7a3e54 terraform: move OpenStack flavorID into nodeGroups 2023-08-04 12:36:45 +02:00
3u13r
720c48ea45
cli: refactor terraform output parsing (#2158) 2023-08-03 16:17:23 +02:00
Adrian Stobbe
70ce195a5f
cli: unify chart value setup (#2153) 2023-08-03 13:54:48 +02:00
3u13r
5119d843f1
terraform: fix uami parsing (#2155) 2023-08-03 13:22:26 +02:00
renovate[bot]
7e3123232e
deps: update Terraform azurerm to v3.67.0 (#2147)
Also update Azure terraform: 
ignore snp policy changes on resource

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: edgelessci <edgelessci@users.noreply.github.com>
Co-authored-by: Otto Bittner <cobittner@posteo.net>
2023-08-02 08:15:22 +02:00
Adrian Stobbe
26480016a9
azure: fix ccm config with correct uami client_id (#2144)
* fix azure ccm config with correct uami client_id

* fix tests
2023-08-01 08:40:44 +02:00
Adrian Stobbe
26305e8f80
cli: install helm charts in cli instead of bootstrapper (#2136)
* init

* fixup! init

* gcp working?

* fixup! fixup! init

* azure cfg for microService installation

* fixup! azure cfg for microService installation

* fixup! azure cfg for microService installation

* cleanup bootstrapper code

* cleanup helminstall code

* fixup! cleanup helminstall code

* Update internal/deploy/helm/install.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* daniel feedback

* TODO add provider (also to CreateCluster) so we can ensure that provider specific output

* fixup! daniel feedback

* use debugLog in helm installer

* placeholderHelmInstaller

* rename to stub

---------

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
2023-07-31 10:53:05 +02:00
Otto Bittner
7db058f946
cli: wait for public ip update before reading fqdn (#2135) 2023-07-26 15:23:37 +02:00
Adrian Stobbe
a87b7894db
aws: use new LB controller to fix SecurityGroup cleanup on K8s service deletion (#2090)
* add current chart

add current helm chart

* disable service controller for aws ccm

* add new iam roles

* doc AWS internet LB + add to LB test

* pass clusterName to helm for AWS LB

* fix update-aws-lb chart to also include .helmignore

* move chart outside services

* working state

* add subnet tags for AWS subnet discovery

* fix .helmignore load rule with file in subdirectory

* upgrade iam profile

* revert new loader impl since cilium is not correctly loaded

* install chart if not already present during `upgrade apply`

* cleanup PR + fix build + add todos

cleanup PR + add todos

* shared helm pkg for cli install and bootstrapper

* add link to eks docs

* refactor iamMigrationCmd

* delete unused helm.symwallk

* move iammigrate to upgrade pkg

* fixup! delete unused helm.symwallk

* add to upgradecheck

* remove nodeSelector from go code (Otto)

* update iam docs and sort permission + remove duplicate roles

* fix bug in `upgrade check`

* better upgrade check output when svc version upgrade not possible

* pr feedback

* remove force flag in upgrade_test

* use upgrader.GetUpgradeID instead of extra type

* remove todos + fix check

* update doc lb (leo)

* remove bootstrapper helm package

* Update cli/internal/cmd/upgradecheck.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* final nits

* add docs for e2e upgrade test setup

* Apply suggestions from code review

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update cli/internal/helm/loader.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update cli/internal/cmd/tfmigrationclient.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* fix daniel review

* link to the iam permissions instead of manually updating them (agreed with leo)

* disable iam upgrade in upgrade apply

---------

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
Co-authored-by: Malte Poll
2023-07-24 10:30:53 +02:00
Malte Poll
8da6a23aa5
bootstrapper: add fallback endpoint and custom endpoint to SAN field (#2108)
terraform: collect apiserver cert SANs and support custom endpoint

constants: add new constants for cluster configuration and custom endpoint

cloud: support apiserver cert sans and prepare for endpoint migration on AWS

config: add customEndpoint field

bootstrapper: use per-CSP apiserver cert SANs

cli: route customEndpoint to terraform and add migration for apiserver cert SANs

bootstrapper: change interface of GetLoadBalancerEndpoint to return host and port separately
2023-07-21 16:43:51 +02:00
Otto Bittner
cf822f7eee
cli: unify terraform variable creation (#2119)
Before we defined the variables twice.
Once for upgrades, once for create.
Also move default node group names into a constant
2023-07-21 10:04:29 +02:00
Otto Bittner
c2849f4bbe
cli: ignore name changes on lb public ip resource (#2117)
Changing the name forces a recreate, which would break existing clusters.
The name change seems to be "only" about having clearer names.
2023-07-19 10:15:23 +02:00
Malte Poll
5cbdb3a519
terraform: allows cluster name length of 10 characters on AWS (#2110) 2023-07-17 17:45:41 +02:00
renovate[bot]
f364bd6b9d
deps: update Terraform google-beta to v4.72.0 (#2027)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-07-06 11:53:40 +02:00
renovate[bot]
3f1faead94
deps: update Terraform google to v4.72.0 (#2026)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-07-06 11:53:15 +02:00
renovate[bot]
ff74afa00d
deps: update Terraform azurerm to v3.63.0 (#2025)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-07-06 11:52:44 +02:00
renovate[bot]
bf09313dcf
deps: update Terraform aws to v5.6.2 (#2024)
Co-authored-by: Leonard Cohnen <lc@edgeless.systems>
2023-07-06 10:21:35 +02:00
renovate[bot]
e6dbb13c6c
deps: update Terraform openstack to v1.52.1 (#2028)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-07-05 16:52:58 +02:00
Adrian Stobbe
c39df2f7da
terraform: openstack node groups (#1966)
* openstack

* rename to base_name

* fix openstack boot vtpm

* add docs for accessing bootstrapper logs

* rename to initial count
2023-07-03 16:33:00 +02:00