Commit Graph

105 Commits

Author SHA1 Message Date
Moritz Sanft
f4b2d02194
ci: collect cluster metrics to OpenSearch (#2347)
* add Metricbeat deployment to debugd

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* set metricbeat debugd image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix k8s deployment

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use 2 separate deployments

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* only deploy via k8s in non-debug-images

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add missing tilde

* remove k8s metrics

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* unify flag

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add cloud metadata processor to filebeat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* ci: fix debugd logcollection (#2355)

* add missing keyvault access role

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bump logstash image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bump filebeat / metricbeat image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* log used image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use debugging image versions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* increase wait timeout for image upload

* add cloud metadata processor to filebeat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix template locations in container

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix image version typo

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add filebeat / metricbeat users

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove user additions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update workflow step name

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* only mount config files

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* document potential rc

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix IAM permissions in workflow

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix AWS permissions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add missing workflow input

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* rename action

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* pin image versions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove unnecessary workflow inputs

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add refStream input

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove inputs.yml dep

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* increase system metric period

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix linkchecker

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
2023-09-27 16:17:31 +02:00
renovate[bot]
5cfa91f3db
deps: update Terraform azurerm to v3.74.0 (#2368)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-09-26 13:17:17 +02:00
Adrian Stobbe
118f789c2f
cli: fix Azure SEV-SNP latest version logic (#2343) 2023-09-25 11:53:02 +02:00
3u13r
a25c90e9bb
remove deprecated constellation create flags (#2325)
* chore: clean-up TODOs

* cli: make OpenStack error explicit

* cli: remove deprecated flags

* config: require DeployCSIDriver field
2023-09-08 21:15:02 +02:00
Daniel Weiße
442f904ceb
ci: don't automatically create git tag in release pipeline (#2316)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-09-07 08:47:01 +02:00
Adrian Stobbe
7c9a78fe51
make release idempotent (#2278) 2023-08-28 09:21:25 +02:00
Adrian Stobbe
f15c5444da
upgrade test from v2.10.1 (#2279) 2023-08-24 09:15:43 +02:00
Adrian Stobbe
c42e81bf23
docs: update release docs (#2226) 2023-08-22 14:30:20 +02:00
Otto Bittner
b2fa9988b9
docs: add note on how we handle deprecations (#2185) 2023-08-21 16:48:10 +02:00
Moritz Sanft
60bf770e62
ci: logcollection to OpenSearch in non-debug clusters (#2080)
* refactor `debugd` file structure

* create `hack`-tool to deploy logcollection to non-debug clusters

* integrate changes into CI

* update fields

* update workflow input names

* use `working-directory`

* add opensearch creds to upgrade workflow

* make template func generic

* make templating func generic

* linebreaks

* remove magic defaults

* move `os.Exit` to main package

* make logging index configurable

* make templating generic

* remove excess brace

* update fields

* copy fields

* fix flag name

* fix linter warnings

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* remove unused workflow inputs

* remove makefiles

* fix command

* bazel: fix output paths of container

This fixes the output paths of builds within the container by mounting
directories to paths that exist on the host. We also explicitly set the
output path in a .bazelrc to the user specific path. The rc file is
mounted into the container and overrides the host rc.
Also adding automatic stop in case start is called and a containers
is already running.
Sym links like bazel-out and paths bazel outputs should generally work
with this change.

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* tabs -> spaces

---------

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-08-21 08:01:33 +02:00
Daniel Weiße
946942ba68 Add package updating/creation tips to dev-docs
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-09 15:42:24 +02:00
Malte Poll
82de0b83bd docs: remove deprecated flags from docs 2023-08-04 12:36:45 +02:00
renovate[bot]
7e3123232e
deps: update Terraform azurerm to v3.67.0 (#2147)
Also update Azure terraform: 
ignore snp policy changes on resource

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: edgelessci <edgelessci@users.noreply.github.com>
Co-authored-by: Otto Bittner <cobittner@posteo.net>
2023-08-02 08:15:22 +02:00
Otto Bittner
002c3a9a32
ci: upgrade fromVersion for upgrade tests (#2145)
Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>
2023-08-01 10:34:11 +02:00
Adrian Stobbe
a3184af7a2
cli: add iam upgrade apply (#2132)
* add new iam upgrade apply

* remove iam tf plan from upgrade apply check

* add iam migration warning to upgrade apply

* update release process

* document migration

* Apply suggestions from code review

Co-authored-by: Otto Bittner <cobittner@posteo.net>

* add iam upgrade

* remove upgrade dir check in test

* ask only without --yes

* make iam upgrade provider specific

* test without seperate logins

* remove csi and only add conditionally

* Revert "test without seperate logins"

This reverts commit 05a12e59c9.

* fix msising cred

* support iam migration for all csps

* add iam upgrade label

---------

Co-authored-by: Otto Bittner <cobittner@posteo.net>
2023-07-26 17:29:03 +02:00
Adrian Stobbe
a87b7894db
aws: use new LB controller to fix SecurityGroup cleanup on K8s service deletion (#2090)
* add current chart

add current helm chart

* disable service controller for aws ccm

* add new iam roles

* doc AWS internet LB + add to LB test

* pass clusterName to helm for AWS LB

* fix update-aws-lb chart to also include .helmignore

* move chart outside services

* working state

* add subnet tags for AWS subnet discovery

* fix .helmignore load rule with file in subdirectory

* upgrade iam profile

* revert new loader impl since cilium is not correctly loaded

* install chart if not already present during `upgrade apply`

* cleanup PR + fix build + add todos

cleanup PR + add todos

* shared helm pkg for cli install and bootstrapper

* add link to eks docs

* refactor iamMigrationCmd

* delete unused helm.symwallk

* move iammigrate to upgrade pkg

* fixup! delete unused helm.symwallk

* add to upgradecheck

* remove nodeSelector from go code (Otto)

* update iam docs and sort permission + remove duplicate roles

* fix bug in `upgrade check`

* better upgrade check output when svc version upgrade not possible

* pr feedback

* remove force flag in upgrade_test

* use upgrader.GetUpgradeID instead of extra type

* remove todos + fix check

* update doc lb (leo)

* remove bootstrapper helm package

* Update cli/internal/cmd/upgradecheck.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* final nits

* add docs for e2e upgrade test setup

* Apply suggestions from code review

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update cli/internal/helm/loader.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update cli/internal/cmd/tfmigrationclient.go

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* fix daniel review

* link to the iam permissions instead of manually updating them (agreed with leo)

* disable iam upgrade in upgrade apply

---------

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
Co-authored-by: Malte Poll
2023-07-24 10:30:53 +02:00
Otto Bittner
08e790b8fd
docs: mention need to delete variants (#2096) 2023-07-17 10:29:59 +02:00
Adrian Stobbe
94b087197b
docs: how to set up MiniConstellation on Azure (#1999)
* init

* update doc

* move quick-setup to devdocs
2023-07-07 15:14:13 +02:00
Adrian Stobbe
c39df2f7da
terraform: openstack node groups (#1966)
* openstack

* rename to base_name

* fix openstack boot vtpm

* add docs for accessing bootstrapper logs

* rename to initial count
2023-07-03 16:33:00 +02:00
Adrian Stobbe
1edbe962c1
cli: fail fast when CLI and Constellation versions don't match (#1972)
* fail on version mismatch

* rename to validateCLIandConstellationVersionAreEqual

* fix test

* image version must only be major,minor patch equal (ignore suffix)

* add version support doc

* fix: do not check patch version equality for image and cli

* skip validate on force
2023-06-27 18:24:35 +02:00
Adrian Stobbe
7dcd8c3dab
dev-docs: refactor and add information for newbies (#1912)
* refactor dev-docs structure and add information

* improve doc

* Update dev-docs/workflows/create-debug-cluster.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update dev-docs/workflows/create-debug-cluster.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* pr feedback daniel

* Update dev-docs/README.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* move to howto again

* split up dev-setup and pull-request into sep files

* fix backticks

* add writing style convention + testing repo

* remove OSS cluster + reduce plugins vs code

* update bazel pre-pr doc

* ghcr img private hint

* add fetch measurement + provider sub-directory hint

* add label doc + pr title check in template

* add OSS build comment

* Update CONTRIBUTING.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update CONTRIBUTING.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update dev-docs/README.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update dev-docs/workflows/dev-setup.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* thomas feedback

* add go proverb mention

---------

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>
2023-06-19 17:39:43 +02:00
Adrian Stobbe
07de6482b2
config: drop support for deprecated Azure's service principal authentication (#1906)
* invalidate app client id field for azure and provide info

* remove TestNewWithDefaultOptions case

* fix test

* remove appClientID field

* remove client secret + rename err

* remove from docs

* otto feedback

* update docs

* delete env test in cfg since no envs set anymore

* Update dev-docs/workflows/github-actions.md

Co-authored-by: Otto Bittner <cobittner@posteo.net>

* WARNING to stderr

* fix check

---------

Co-authored-by: Otto Bittner <cobittner@posteo.net>
2023-06-14 17:50:57 +02:00
Adrian Stobbe
51d66b2609
doc: vs code linter issue (#1880) 2023-06-07 10:32:06 +02:00
Otto Bittner
1b92d96feb
dev-docs: add e2e tag to vsCode config (#1852) 2023-06-02 09:53:00 +02:00
Malte Poll
26bc653d0e bazel: build both cli variants as part of devbuild 2023-05-31 14:00:00 +02:00
Malte Poll
7cff47f30f
ci: run release workflow on temporary branch (#1628) 2023-05-25 10:14:42 +02:00
Leonard Cohnen
c98644df2b ci: use bazel for unittests 2023-05-23 15:11:10 +02:00
Moritz Sanft
339e750c18
bazel: add Bazel shell completion scripts (#1804)
* add bazel autocomplete script

* indentation

* shfmt

* shellcheck ignore completion file

* fix shellcheck ignore path
2023-05-22 12:54:38 +02:00
Adrian Stobbe
f99e06b63b
cli: new flag to set the attestation type for config generate (#1769)
* add attestation flag to specify type in config
2023-05-17 16:53:56 +02:00
miampf
e7b7a544f0
docs: add a qemu section (#1724) 2023-05-17 13:21:35 +00:00
Malte Poll
653bf3621d image: replicate AWS images to eu-west-1 and eu-west-3 2023-05-05 12:06:44 +02:00
Thomas Tendyck
17c45bc881 Update release.md 2023-05-05 10:57:32 +02:00
Leonard Cohnen
1825c11790 docs: remove second build documentation 2023-04-27 13:04:40 +02:00
Leonard Cohnen
f3dc37293f docs: align container_prefix suggestion 2023-04-27 13:04:40 +02:00
3u13r
1bdf410b52
bazel: allow custom container_prefix (#1693)
* build: allow custom container registry

* build: fix .bazeloverwriterc import
2023-04-27 11:52:02 +02:00
Leonard Cohnen
f49126bce4 docs: update release worklow 2023-04-06 18:20:50 +02:00
Paul Meyer
de5d8d796b
Update release.md (#1613) 2023-04-05 11:02:54 +02:00
Paul Meyer
399b052f9e
bazel: add protoc codegen to //:generate target (#1554)
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-30 14:47:29 +02:00
Otto Bittner
ccb31c9570
docs: add CDN cache invalidation to release docs (#1536) 2023-03-28 13:50:27 +02:00
Malte Poll
c3c0940adb
bazel: use remote caching (#1456)
* bazel: add configuration for remote caching
* ci: enable bazel remote caching for building binaries
* ci: use bazel directly when building go binaries
* ci: enable cache for most build steps
* dev-docs: document remote caching
2023-03-20 16:05:08 +01:00
Paul Meyer
e3f37e9a38 bazel: add shfmt to tidy target
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-16 04:39:45 -04:00
Malte Poll
a73cdb9b14
bazel: command to prepare development workspace (#1425)
This command symlinks all binaries into the current working directory (or the path specified by the first argument)

* bazel: command to prepare development workspace
* bazel: set malt3 as codeowner
2023-03-14 13:57:39 +01:00
Paul Meyer
02c97fac03 bazel: add tidy and check
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-14 03:43:51 -04:00
Otto Bittner
b356af5a39
docs: minor adjustements to the release docs (#1384) 2023-03-09 18:03:47 +01:00
Daniel Weiße
83d10b0e70
hack: remove unused tools (#1387)
* Remove unused pcr-compare tool
* Remove unused pcr-reader tool
* Remove obsolete image-measurement tool

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-03-09 16:59:33 +01:00
Malte Poll
bdba9d8ba6
bazel: add build files for go (#1186)
* build: correct toolchain order
* build: gazelle-update-repos
* build: use pregenerated proto for dependencies
* update bazeldnf
* deps: tpm simulator
* Update Google trillian module
* cli: add stamping as alternative build info source
* bazel: add go_test wrappers, mark special tests and select testing deps
* deps: add libvirt deps
* deps: go-libvirt patches
* deps: cloudflare circl patches
* bazel: add go_test wrappers, mark special tests and select testing deps
* bazel: keep gazelle overrides
* bazel: cleanup bazelrc
* bazel: switch CMakeLists.txt to use bazel
* bazel: fix injection of version information via stamping
* bazel: commit all build files
* dev-docs: document bazel usage
* deps: upgrade zig-cc for go 1.20
* bazel: update Perl for macOS arm64 & Linux arm64 support
* bazel: use static perl toolchain for OpenSSL
* bazel: use static protobuf (protoc) toolchain
* deps: add git and go to nix deps

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-09 15:23:42 +01:00
Otto Bittner
0e91e4aabf
docs: extend release docs with image deletion manual (#1371)
@malt3 wrote this guide on how to find and delete all image
resources. I found it very helpful.

Co-authored-by: Malte Poll <mp@edgeless.systems>
2023-03-08 16:02:29 +01:00
Fabian Kammel
b6b353c53e
docs: mention 'needs-backport' label in release process (#1216)
* Mention 'needs-backport' label in release docs.
* Add missing version prefix.
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2023-02-20 10:44:16 +01:00
Daniel Weiße
690b50b29d
dev-docs: Go package docs (#958)
* Remove unused package

* Add Go package docs to most packages

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
Co-authored-by: Fabian Kammel <fk@edgeless.systems>
2023-01-19 15:57:50 +01:00
Moritz Sanft
ae2db08f3a
ci: add e2e test for constellation recover (#845)
* AB#2256 Add recover e2e test

* AB#2256 move test & fix minor objections

* AB#2256 fix path

* AB#2256 rename hacky filename
2023-01-19 10:41:07 +01:00
Moritz Sanft
e844ceb2b1
cli: adopt Cobra cli reference style (#997)
* adapt to Cobra CLI ref style

* linting

* change multi-line reference style

* lowercase short descriptions

* Revert "lowercase short descriptions"

This reverts commit 499dc3577a.

* use 2 newlines on long description and add dots

* mark required flags

* Update cli/internal/cmd/iamcreateaws.go

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* Update cli/internal/cmd/upgradeexecute.go

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* Update cli/internal/cmd/upgradeexecute.go

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-01-17 14:01:56 +01:00
Otto Bittner
90b88e1cf9 kms: rename kms to keyservice
In the light of extending our eKMS support it will be helpful
to have a tighter use of the word "KMS".
KMS should refer to the actual component that manages keys.
The keyservice, also called KMS in the constellation code,
does not manage keys itself. It talks to a KMS backend,
which in turn does the actual key management.
2023-01-16 11:56:34 +01:00
Malte Poll
ba59d8302d dev-docs: describe automated release process 2023-01-12 13:24:07 +01:00
Daniel Weiße
41fca129f6
devdocs: documentation of logging conventions (#899)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-01-10 15:26:12 +01:00
Moritz Eckert
b2f8f72f1e
dev-docs: move into top-level dir (#924) 2023-01-10 14:18:41 +01:00