Commit Graph

70 Commits

Author SHA1 Message Date
Daniel Weiße
97aea98e77
ci: update GCP service accounts for CI (#2629)
* Update CI to use different GCP project for e2e tests
* Update GCP image project service accounts
* Update default GCP bucket name for image builds

---------

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-11-27 13:04:41 +01:00
Adrian Stobbe
98673b0983
ci: only generate lock files where provider is used (#2636) 2023-11-27 12:16:45 +01:00
Moritz Sanft
34bf3ad296
terraform-provider: add image datasource (#2642)
* terraform-provider: init

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* terraform-provider: add basic docgen

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* terraform-provider: fix build steps

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* terraform-provider: extend build process and docgen

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* dev-docs: document provider usage

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bazel: upload aspect lib mirror

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* terraform-provider: don't try to create lockfiles

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bazel: fix shellcheck issues

* bazel: separate paths to check

* terraform-provider: clean up old files

* terraform-provider: update provider resource

* terraform-provider: add image data source

* dev-docs: remove unnecessary init

* bazel: adhere to Terraform naming expectations

* terraform-provider: fix expected data type

* terraform-provider: generate docs

* terraform-provider: improve errors

* terraform-provider: add acceptance tests for data source

* terraform-provider: fix dependencies

* bazel: quote var reference

* terraform-provider: make region optional

* terraform-provider: bind imagefetcher to data source

* bazel: tidy

* terraform-provider: remove unused parameter

* terraform-provider: remove unused parameter

* terraform-provider: extend acceptance tests

* terraform-provider: allow tests to be ran without Bazel

* dev-docs: document testing

* terraform-provider: set binary path accordingly

* dev-docs: document docgen process for the provider

* bazel: run acceptance test in writable environment

* bazel: try to write to `$TMPDIR`

* terraform-provider: style nits

* terraform-provider: leave TODO

* bazel: tidy

* terraform-provider: regenerate docs

* terraform-provider: fix comment

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
2023-11-27 09:00:08 +01:00
Moritz Sanft
9a62657b80
terraform-provider: init provider scaffolding (#2632)
* terraform-provider: init

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* terraform-provider: add basic docgen

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* terraform-provider: fix build steps

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* terraform-provider: extend build process and docgen

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* dev-docs: document provider usage

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bazel: upload aspect lib mirror

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bazel: add docstring to fix linter

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* terraform-provider: don't try to create lockfiles

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bazel: fix shellcheck issues

* bazel: separate paths to check

* bazel: explain what updating lockfiles means

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* terraform-provider: fix linter checks

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
2023-11-24 15:58:21 +01:00
3u13r
0564e4ebb4
dev-docs: add on-prem terraform to vpn setup (#2619)
* vpn: add fake-on-prem infra

* dev-docs: move vpn helm
2023-11-23 16:13:37 +01:00
Markus Rudy
284c7e99d1
docs: add Helm chart for VPN connectivity (#2577)
Co-authored-by: 3u13r <lc@edgeless.systems>
2023-11-22 15:08:11 +01:00
Markus Rudy
69c64d6c69
docs: improve developer documentation for folks new to the codebase
* Correct example invocation of aws cli

* Add warning to Helm Intellisense recommendation

* Link code conventions in PR guidelines

* Tighten debugd README

* cmake is not used for building debugd anymore, remove references to it

* make the debug-cluster workflow the authoritative source for cdbg usage - don't replicate the same instructions in different places

* Document that Bazel eats a lot of RAM
2023-11-17 19:16:52 +01:00
Moritz Sanft
ce518486e5
dev-docs: fix instructions in release docs (#2611) 2023-11-17 10:46:45 +01:00
Moritz Sanft
8e4feb7e2a
terraform: add Terraform module for Azure (#2566)
* add Azure Terraform module

* add maa-patching command to cli

* refactor release process

* factor out image fetching to own action

* add CI

* generate

* fix some unnecessary changes

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use `constellation maa-patch` in ci

* insecure flag when using debug image

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* only update maa url if existing

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* make node group zone optional on aws and gcp

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* [remove] register updated workflow

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* Revert "[remove] register updated workflow"

This reverts commit e70b9515b7eabbcbe0d41fa1296c48750cd02ace.

* create MAA

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* make maa-patching only run on azure

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add comment

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* require node group zone for GCP and AWS

* remove unnecessary bazel action

* stamp version to correct file

* refer to `maa-patch` command in docs

* run Azure test in weekly e2e

* comment / naming improvements

* remove sa_account resource

* disable spellcheck ot use "URL"

* `create_maa` variable

* don't write maa url to config

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* default to nightly image

* use input ref and stream

* fix command check

* don't set region in weekly e2e call

* patch maa if url is not empty

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove `create_maa` variable

* remove binaries

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove undefined input

* replace invalid attestation URL error message

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* fix punctuation

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* skip hidden commands in clidocgen

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* enable spellcheck before code block

* move spellcheck trigger out of info block

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix workflow dependencies

* let image default to CLI version

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>
2023-11-13 18:46:20 +01:00
Adrian Stobbe
b765231175
deps: bump Go to 1.21.4 (#2569)
Co-authored-by: Malte Poll <1780588+malt3@users.noreply.github.com>
2023-11-09 20:17:14 +01:00
Adrian Stobbe
cea6204b37
terraform: Terraform module for AWS (#2503) 2023-11-08 19:10:01 +01:00
Otto Bittner
b0ee39a96d ci: publish s3proxy chart during release 2023-11-06 10:21:11 +01:00
Daniel Weiße
f4bfbe3564
docs: refer to apply command instead of init or upgrade apply (#2487)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-10-27 08:30:59 +02:00
Malte Poll
d22f53d7cc bazel: always use nix 2023-10-12 14:42:24 +02:00
Malte Poll
8e706d6de3 image: update README 2023-09-27 17:58:19 +02:00
Moritz Sanft
f4b2d02194
ci: collect cluster metrics to OpenSearch (#2347)
* add Metricbeat deployment to debugd

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* set metricbeat debugd image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix k8s deployment

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use 2 separate deployments

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* only deploy via k8s in non-debug-images

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add missing tilde

* remove k8s metrics

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* unify flag

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add cloud metadata processor to filebeat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* ci: fix debugd logcollection (#2355)

* add missing keyvault access role

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bump logstash image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bump filebeat / metricbeat image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* log used image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use debugging image versions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* increase wait timeout for image upload

* add cloud metadata processor to filebeat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix template locations in container

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix image version typo

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add filebeat / metricbeat users

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove user additions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update workflow step name

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* only mount config files

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* document potential rc

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix IAM permissions in workflow

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix AWS permissions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add missing workflow input

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* rename action

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* pin image versions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove unnecessary workflow inputs

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add refStream input

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove inputs.yml dep

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* increase system metric period

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix linkchecker

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
2023-09-27 16:17:31 +02:00
renovate[bot]
5cfa91f3db
deps: update Terraform azurerm to v3.74.0 (#2368)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-09-26 13:17:17 +02:00
Adrian Stobbe
118f789c2f
cli: fix Azure SEV-SNP latest version logic (#2343) 2023-09-25 11:53:02 +02:00
3u13r
a25c90e9bb
remove deprecated constellation create flags (#2325)
* chore: clean-up TODOs

* cli: make OpenStack error explicit

* cli: remove deprecated flags

* config: require DeployCSIDriver field
2023-09-08 21:15:02 +02:00
Daniel Weiße
442f904ceb
ci: don't automatically create git tag in release pipeline (#2316)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-09-07 08:47:01 +02:00
Adrian Stobbe
7c9a78fe51
make release idempotent (#2278) 2023-08-28 09:21:25 +02:00
Adrian Stobbe
f15c5444da
upgrade test from v2.10.1 (#2279) 2023-08-24 09:15:43 +02:00
Adrian Stobbe
c42e81bf23
docs: update release docs (#2226) 2023-08-22 14:30:20 +02:00
Otto Bittner
b2fa9988b9
docs: add note on how we handle deprecations (#2185) 2023-08-21 16:48:10 +02:00
Moritz Sanft
60bf770e62
ci: logcollection to OpenSearch in non-debug clusters (#2080)
* refactor `debugd` file structure

* create `hack`-tool to deploy logcollection to non-debug clusters

* integrate changes into CI

* update fields

* update workflow input names

* use `working-directory`

* add opensearch creds to upgrade workflow

* make template func generic

* make templating func generic

* linebreaks

* remove magic defaults

* move `os.Exit` to main package

* make logging index configurable

* make templating generic

* remove excess brace

* update fields

* copy fields

* fix flag name

* fix linter warnings

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* remove unused workflow inputs

* remove makefiles

* fix command

* bazel: fix output paths of container

This fixes the output paths of builds within the container by mounting
directories to paths that exist on the host. We also explicitly set the
output path in a .bazelrc to the user specific path. The rc file is
mounted into the container and overrides the host rc.
Also adding automatic stop in case start is called and a containers
is already running.
Sym links like bazel-out and paths bazel outputs should generally work
with this change.

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* tabs -> spaces

---------

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-08-21 08:01:33 +02:00
Daniel Weiße
946942ba68 Add package updating/creation tips to dev-docs
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-09 15:42:24 +02:00
Malte Poll
82de0b83bd docs: remove deprecated flags from docs 2023-08-04 12:36:45 +02:00
renovate[bot]
7e3123232e
deps: update Terraform azurerm to v3.67.0 (#2147)
Also update Azure terraform: 
ignore snp policy changes on resource

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: edgelessci <edgelessci@users.noreply.github.com>
Co-authored-by: Otto Bittner <cobittner@posteo.net>
2023-08-02 08:15:22 +02:00
Otto Bittner
002c3a9a32
ci: upgrade fromVersion for upgrade tests (#2145)
Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>
2023-08-01 10:34:11 +02:00
Adrian Stobbe
a3184af7a2
cli: add iam upgrade apply (#2132)
* add new iam upgrade apply

* remove iam tf plan from upgrade apply check

* add iam migration warning to upgrade apply

* update release process

* document migration

* Apply suggestions from code review

Co-authored-by: Otto Bittner <cobittner@posteo.net>

* add iam upgrade

* remove upgrade dir check in test

* ask only without --yes

* make iam upgrade provider specific

* test without seperate logins

* remove csi and only add conditionally

* Revert "test without seperate logins"

This reverts commit 05a12e59c9.

* fix msising cred

* support iam migration for all csps

* add iam upgrade label

---------

Co-authored-by: Otto Bittner <cobittner@posteo.net>
2023-07-26 17:29:03 +02:00
Adrian Stobbe
a87b7894db
aws: use new LB controller to fix SecurityGroup cleanup on K8s service deletion (#2090)
* add current chart

add current helm chart

* disable service controller for aws ccm

* add new iam roles

* doc AWS internet LB + add to LB test

* pass clusterName to helm for AWS LB

* fix update-aws-lb chart to also include .helmignore

* move chart outside services

* working state

* add subnet tags for AWS subnet discovery

* fix .helmignore load rule with file in subdirectory

* upgrade iam profile

* revert new loader impl since cilium is not correctly loaded

* install chart if not already present during `upgrade apply`

* cleanup PR + fix build + add todos

cleanup PR + add todos

* shared helm pkg for cli install and bootstrapper

* add link to eks docs

* refactor iamMigrationCmd

* delete unused helm.symwallk

* move iammigrate to upgrade pkg

* fixup! delete unused helm.symwallk

* add to upgradecheck

* remove nodeSelector from go code (Otto)

* update iam docs and sort permission + remove duplicate roles

* fix bug in `upgrade check`

* better upgrade check output when svc version upgrade not possible

* pr feedback

* remove force flag in upgrade_test

* use upgrader.GetUpgradeID instead of extra type

* remove todos + fix check

* update doc lb (leo)

* remove bootstrapper helm package

* Update cli/internal/cmd/upgradecheck.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* final nits

* add docs for e2e upgrade test setup

* Apply suggestions from code review

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update cli/internal/helm/loader.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update cli/internal/cmd/tfmigrationclient.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* fix daniel review

* link to the iam permissions instead of manually updating them (agreed with leo)

* disable iam upgrade in upgrade apply

---------

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
Co-authored-by: Malte Poll
2023-07-24 10:30:53 +02:00
Otto Bittner
08e790b8fd
docs: mention need to delete variants (#2096) 2023-07-17 10:29:59 +02:00
Adrian Stobbe
94b087197b
docs: how to set up MiniConstellation on Azure (#1999)
* init

* update doc

* move quick-setup to devdocs
2023-07-07 15:14:13 +02:00
Adrian Stobbe
c39df2f7da
terraform: openstack node groups (#1966)
* openstack

* rename to base_name

* fix openstack boot vtpm

* add docs for accessing bootstrapper logs

* rename to initial count
2023-07-03 16:33:00 +02:00
Adrian Stobbe
1edbe962c1
cli: fail fast when CLI and Constellation versions don't match (#1972)
* fail on version mismatch

* rename to validateCLIandConstellationVersionAreEqual

* fix test

* image version must only be major,minor patch equal (ignore suffix)

* add version support doc

* fix: do not check patch version equality for image and cli

* skip validate on force
2023-06-27 18:24:35 +02:00
Adrian Stobbe
7dcd8c3dab
dev-docs: refactor and add information for newbies (#1912)
* refactor dev-docs structure and add information

* improve doc

* Update dev-docs/workflows/create-debug-cluster.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update dev-docs/workflows/create-debug-cluster.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* pr feedback daniel

* Update dev-docs/README.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* move to howto again

* split up dev-setup and pull-request into sep files

* fix backticks

* add writing style convention + testing repo

* remove OSS cluster + reduce plugins vs code

* update bazel pre-pr doc

* ghcr img private hint

* add fetch measurement + provider sub-directory hint

* add label doc + pr title check in template

* add OSS build comment

* Update CONTRIBUTING.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update CONTRIBUTING.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update dev-docs/README.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update dev-docs/workflows/dev-setup.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* thomas feedback

* add go proverb mention

---------

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>
2023-06-19 17:39:43 +02:00
Adrian Stobbe
07de6482b2
config: drop support for deprecated Azure's service principal authentication (#1906)
* invalidate app client id field for azure and provide info

* remove TestNewWithDefaultOptions case

* fix test

* remove appClientID field

* remove client secret + rename err

* remove from docs

* otto feedback

* update docs

* delete env test in cfg since no envs set anymore

* Update dev-docs/workflows/github-actions.md

Co-authored-by: Otto Bittner <cobittner@posteo.net>

* WARNING to stderr

* fix check

---------

Co-authored-by: Otto Bittner <cobittner@posteo.net>
2023-06-14 17:50:57 +02:00
Adrian Stobbe
51d66b2609
doc: vs code linter issue (#1880) 2023-06-07 10:32:06 +02:00
Otto Bittner
1b92d96feb
dev-docs: add e2e tag to vsCode config (#1852) 2023-06-02 09:53:00 +02:00
Malte Poll
26bc653d0e bazel: build both cli variants as part of devbuild 2023-05-31 14:00:00 +02:00
Malte Poll
7cff47f30f
ci: run release workflow on temporary branch (#1628) 2023-05-25 10:14:42 +02:00
Leonard Cohnen
c98644df2b ci: use bazel for unittests 2023-05-23 15:11:10 +02:00
Moritz Sanft
339e750c18
bazel: add Bazel shell completion scripts (#1804)
* add bazel autocomplete script

* indentation

* shfmt

* shellcheck ignore completion file

* fix shellcheck ignore path
2023-05-22 12:54:38 +02:00
Adrian Stobbe
f99e06b63b
cli: new flag to set the attestation type for config generate (#1769)
* add attestation flag to specify type in config
2023-05-17 16:53:56 +02:00
miampf
e7b7a544f0
docs: add a qemu section (#1724) 2023-05-17 13:21:35 +00:00
Malte Poll
653bf3621d image: replicate AWS images to eu-west-1 and eu-west-3 2023-05-05 12:06:44 +02:00
Thomas Tendyck
17c45bc881 Update release.md 2023-05-05 10:57:32 +02:00
Leonard Cohnen
1825c11790 docs: remove second build documentation 2023-04-27 13:04:40 +02:00
Leonard Cohnen
f3dc37293f docs: align container_prefix suggestion 2023-04-27 13:04:40 +02:00
3u13r
1bdf410b52
bazel: allow custom container_prefix (#1693)
* build: allow custom container registry

* build: fix .bazeloverwriterc import
2023-04-27 11:52:02 +02:00