Commit Graph

161 Commits

Author SHA1 Message Date
Moritz Sanft
f4b2d02194
ci: collect cluster metrics to OpenSearch (#2347)
* add Metricbeat deployment to debugd

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* set metricbeat debugd image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix k8s deployment

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use 2 separate deployments

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* only deploy via k8s in non-debug-images

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add missing tilde

* remove k8s metrics

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* unify flag

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add cloud metadata processor to filebeat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* ci: fix debugd logcollection (#2355)

* add missing keyvault access role

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bump logstash image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bump filebeat / metricbeat image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* log used image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use debugging image versions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* increase wait timeout for image upload

* add cloud metadata processor to filebeat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix template locations in container

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix image version typo

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add filebeat / metricbeat users

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove user additions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update workflow step name

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* only mount config files

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* document potential rc

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix IAM permissions in workflow

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix AWS permissions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add missing workflow input

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* rename action

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* pin image versions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove unnecessary workflow inputs

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add refStream input

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove inputs.yml dep

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* increase system metric period

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix linkchecker

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
2023-09-27 16:17:31 +02:00
renovate[bot]
8f03213a9f
deps: update fedora:38 Docker digest to 6fc00f8 (#2371)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-09-26 13:20:43 +02:00
Daniel Weiße
7aba42baa5
ci: add more filters to e2e failure OpenSearch links (#2358)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-09-26 13:17:59 +02:00
3u13r
548bb2dfa6
debugd: send requests over lb (#2346) 2023-09-19 16:10:22 +02:00
Moritz Sanft
60bf770e62
ci: logcollection to OpenSearch in non-debug clusters (#2080)
* refactor `debugd` file structure

* create `hack`-tool to deploy logcollection to non-debug clusters

* integrate changes into CI

* update fields

* update workflow input names

* use `working-directory`

* add opensearch creds to upgrade workflow

* make template func generic

* make templating func generic

* linebreaks

* remove magic defaults

* move `os.Exit` to main package

* make logging index configurable

* make templating generic

* remove excess brace

* update fields

* copy fields

* fix flag name

* fix linter warnings

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* remove unused workflow inputs

* remove makefiles

* fix command

* bazel: fix output paths of container

This fixes the output paths of builds within the container by mounting
directories to paths that exist on the host. We also explicitly set the
output path in a .bazelrc to the user specific path. The rc file is
mounted into the container and overrides the host rc.
Also adding automatic stop in case start is called and a containers
is already running.
Sym links like bazel-out and paths bazel outputs should generally work
with this change.

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* tabs -> spaces

---------

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-08-21 08:01:33 +02:00
renovate[bot]
a5d6e80dd8
deps: update fedora:38 Docker digest to 61f921e (#2196)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2023-08-10 10:56:35 +02:00
Malte Poll
bd26e6bae7
cdbg: implement workspace / "-C" flag and "--bindir" (#2170) 2023-08-07 11:40:48 +02:00
Daniel Weiße
d1ace13713
cli: add --workspace flag to set base directory for Constellation workspace (#2148)
* Remove `--config` and `--master-secret` falgs

* Add `--workspace` flag

* In CLI, only work on files with paths created from `cli/internal/cmd`

* Properly print values for GCP on IAM create when not directly updating the config

---------

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-04 13:53:51 +02:00
Malte Poll
8da6a23aa5
bootstrapper: add fallback endpoint and custom endpoint to SAN field (#2108)
terraform: collect apiserver cert SANs and support custom endpoint

constants: add new constants for cluster configuration and custom endpoint

cloud: support apiserver cert sans and prepare for endpoint migration on AWS

config: add customEndpoint field

bootstrapper: use per-CSP apiserver cert SANs

cli: route customEndpoint to terraform and add migration for apiserver cert SANs

bootstrapper: change interface of GetLoadBalancerEndpoint to return host and port separately
2023-07-21 16:43:51 +02:00
renovate[bot]
49cff0aabb
deps: update module github.com/sigstore/rekor to v1.2.2 (#2033)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-07-06 15:41:14 +02:00
renovate[bot]
c58b97de76
deps: update fedora:38 Docker digest to a134743 (#2003)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-07-04 14:35:40 +02:00
Adrian Stobbe
7dcd8c3dab
dev-docs: refactor and add information for newbies (#1912)
* refactor dev-docs structure and add information

* improve doc

* Update dev-docs/workflows/create-debug-cluster.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update dev-docs/workflows/create-debug-cluster.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* pr feedback daniel

* Update dev-docs/README.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* move to howto again

* split up dev-setup and pull-request into sep files

* fix backticks

* add writing style convention + testing repo

* remove OSS cluster + reduce plugins vs code

* update bazel pre-pr doc

* ghcr img private hint

* add fetch measurement + provider sub-directory hint

* add label doc + pr title check in template

* add OSS build comment

* Update CONTRIBUTING.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update CONTRIBUTING.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update dev-docs/README.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update dev-docs/workflows/dev-setup.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* thomas feedback

* add go proverb mention

---------

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>
2023-06-19 17:39:43 +02:00
Malte Poll
537cdbcfad bazel: trim path to *.pb.go files embedded in go libraries
See https://github.com/bazelbuild/rules_go/issues/3581 for context.
2023-06-16 16:30:47 +02:00
Malte Poll
264b2df902
deps: upgrade to Fedora 38 (#1909)
* image: upgrade mkosi distro version to Fedora 38
* image: remove downgrade of GCP kernel
* ci: upgrade expected measurements for Fedora 38
* deps: upgrade bazeldnf packages to Fedora 38
* deps: upgrade container images to Fedora 38
2023-06-15 16:50:35 +02:00
3u13r
a2c98eb1d5
Correctly deploy the AWS CCM (#1853)
* aws: stop using the imds api for tags

* aws: disable tags in imds api

* aws: only tag instances with non-lecagy tag

* bootstrapper: always let coredns run before cilium

* debugd: make debugd less noisy

* fixup fix aws imds test

* fixup unsued context

* move getting instance id to readInstanceTag
2023-06-13 09:58:39 +02:00
Adrian Stobbe
e738f15f0f
cdbg: make endpoint deployment failure more transparent (#1883)
* add retry + timeout + intercept grpc logs

* LogStateChanges inside grplog pkg

* remove retry and tj/assert

* rename nit

* Update debugd/internal/cdbg/cmd/deploy.go

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* Update debugd/internal/cdbg/cmd/deploy.go

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* paul feedback

* return waitFn instead of WaitGroup

* Revert "return waitFn instead of WaitGroup"

This reverts commit 45700f30e341ce3af509b687febbc0125f7ddb38.

* log routine inside debugd constructor

* test doubles names

* Update debugd/internal/cdbg/cmd/deploy.go

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* fix newDebugClient closeFn

---------

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-06-12 13:45:34 +02:00
renovate[bot]
167052d443
deps: update dependency hermetic_cc_toolchain to v2.0.0 (#1860)
* deps: update dependency hermetic_cc_toolchain to v2.0.0
* deps: tidy all modules
* bazel: target glibc 2.23 to enable rbe

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: edgelessci <edgelessci@users.noreply.github.com>
Co-authored-by: Malte Poll <mp@edgeless.systems>
2023-06-09 17:39:30 +02:00
Adrian Stobbe
4284f892ce
api: rename /api/versions to versionsapi and /api/attestationcfig to attestationconfigapi (#1876)
* rename to attestationconfigapi + put client and fetcher inside pkg

* rename api/version to versionsapi and put fetcher + client inside pkg

* rename AttestationConfigAPIFetcher to Fetcher
2023-06-07 16:16:32 +02:00
Otto Bittner
30f2b332b3
api: restructure api pkg (#1851)
* api: rename AttestationVersionRepo to Client
* api: move client into separate subpkg for
clearer import paths.
* api: rename configapi -> attestationconfig
* api: rename versionsapi -> versions
* api: rename sut to client
* api: split versionsapi client and make it public
* api: split versionapi fetcher and make it public
* config: move attestationversion type to config
* api: fix attestationconfig client test

Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>
2023-06-02 09:19:23 +02:00
Adrian Stobbe
b51cc52945
config: sign Azure versions on upload & verify on fetch (#1836)
* add SignContent() + integrate into configAPI

* use static client for upload versions tool; fix staticupload calleeReference bug

* use version to get proper cosign pub key.

* mock fetcher in CLI tests

* only provide config.New constructor with fetcher

Co-authored-by: Otto Bittner <cobittner@posteo.net>
Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
2023-06-01 13:55:46 +02:00
3u13r
e0285c122e
todo responsibilities and cleanup (#1837)
* chore: add TODO responsibilities

* chore: remove not needed TODOs

* chore: remove outdated migrations

* chore: remove resolved goleak exception

* chore: remove not needed cosign env

* config: add link to our Azure snp docs
2023-06-01 12:33:06 +02:00
renovate[bot]
1fde3929a7
deps: update fedora:37 Docker digest to ab2fd8d (#1846)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: malt3 <mp@edgeless.systems>
2023-06-01 09:03:05 +02:00
Adrian Stobbe
0a6e5ec02e
config: dynamic attestation configuration through S3 backed API (#1808) 2023-05-25 17:43:44 +01:00
renovate[bot]
fe115bdb16
deps: update module github.com/sigstore/rekor to v1.1.1 [SECURITY] (#1729)
* deps: update module github.com/sigstore/rekor to v1.1.1 [SECURITY]

* deps: bump oras

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Leonard Cohnen <lc@edgeless.systems>
2023-05-11 17:23:17 +02:00
renovate[bot]
cad859153b
deps: update fedora:37 Docker digest to 67870e4 (#1743)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-05-05 14:38:36 +02:00
renovate[bot]
e09243bfb3
deps: update fedora:37 Docker digest to e734322 (#1689)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-04-27 17:51:14 +02:00
Malte Poll
69de06dd1f
image: OpenStack vTPM (#1616)
* cli: allow vpc traffic between nodes on OpenStack
* image: enable vTPM on OpenStack
* cli: add create tests for OpenStack
2023-04-05 16:49:03 +02:00
Paul Meyer
399b052f9e
bazel: add protoc codegen to //:generate target (#1554)
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-30 14:47:29 +02:00
Paul Meyer
e7fc541a57
bazel: add buf as protobuf formatter to //:tidy (#1511)
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-23 18:08:49 +01:00
Nils Hanke
cab6044f69 debugd: use nanosecond precision for logs 2023-03-21 17:50:12 +01:00
Malte Poll
33eddc74e1 debugd: add OpenStack support 2023-03-21 10:51:09 +01:00
Nils Hanke
3fceb2207d debugd: Use very basic JSON regex filter before JSON filter 2023-03-21 10:32:33 +01:00
Paul Meyer
6c4ebe12f3 go: remove superfluous else block
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-20 08:41:01 -04:00
Paul Meyer
0036b24266 go: remove unused parameters
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-20 08:41:01 -04:00
renovate[bot]
4d618a4b99
deps: update fedora:37 Docker digest (#1448)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-17 18:47:36 +01:00
Malte Poll
62e2e70699
bazel: use host platform by default (#1434) 2023-03-16 16:13:48 +01:00
Malte Poll
a73cdb9b14
bazel: command to prepare development workspace (#1425)
This command symlinks all binaries into the current working directory (or the path specified by the first argument)

* bazel: command to prepare development workspace
* bazel: set malt3 as codeowner
2023-03-14 13:57:39 +01:00
Nils Hanke
05e39d98c8 debugd: update Filebeat Dockerfile to 8.6.2 2023-03-13 18:39:33 +01:00
Nils Hanke
1d9d8af92a debugd: update Logstash Dockerfile to 8.6.1 2023-03-13 18:39:33 +01:00
Nils Hanke
2335d429a9 debugd: Enable ordered logs in Logstash 2023-03-13 18:39:33 +01:00
Malte Poll
bdba9d8ba6
bazel: add build files for go (#1186)
* build: correct toolchain order
* build: gazelle-update-repos
* build: use pregenerated proto for dependencies
* update bazeldnf
* deps: tpm simulator
* Update Google trillian module
* cli: add stamping as alternative build info source
* bazel: add go_test wrappers, mark special tests and select testing deps
* deps: add libvirt deps
* deps: go-libvirt patches
* deps: cloudflare circl patches
* bazel: add go_test wrappers, mark special tests and select testing deps
* bazel: keep gazelle overrides
* bazel: cleanup bazelrc
* bazel: switch CMakeLists.txt to use bazel
* bazel: fix injection of version information via stamping
* bazel: commit all build files
* dev-docs: document bazel usage
* deps: upgrade zig-cc for go 1.20
* bazel: update Perl for macOS arm64 & Linux arm64 support
* bazel: use static perl toolchain for OpenSSL
* bazel: use static protobuf (protoc) toolchain
* deps: add git and go to nix deps

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-09 15:23:42 +01:00
Malte Poll
b79f7d0c8c
cli: add basic support for constellation create on OpenStack (#1283)
* image: support OpenStack image build / upload

* cli: add OpenStack terraform template

* config: add OpenStack as CSP

* versionsapi: add OpenStack as CSP

* cli: add OpenStack as provider for `config generate` and `create`

* disk-mapper: add basic support for boot on OpenStack

* debugd: add placeholder for OpenStack

* image: fix config file sourcing for image upload
2023-02-27 18:19:52 +01:00
renovate[bot]
83bea18a4f
deps: update fedora:37 Docker digest (#1274)
Co-authored-by: katexochen <katexochen@users.noreply.github.com>
2023-02-24 17:46:17 +01:00
Paul Meyer
deea806d9c Improve code sequences with multiple errs
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-02-20 12:08:24 -05:00
Paul Meyer
12c866bcb9 deps: replace multierr with native errors.Join
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-02-20 12:08:24 -05:00
renovate[bot]
1732795345
deps: update fedora:37 Docker digest (#1192)
Co-authored-by: katexochen <katexochen@users.noreply.github.com>
2023-02-15 13:28:53 +01:00
Otto Bittner
f204c24174 cli: add version validation and force flag
Version validation checks that the configured versions
are not more than one minor version below the CLI's version.
The validation can be disabled using --force.
This is necessary for now during development as the CLI
does not have a prerelease version, as our images do.
2023-02-08 12:30:01 +01:00
Malte Poll
8c57995468
debugd: keep mutex locked while receiving files (#1070) 2023-01-27 17:57:32 +01:00
Malte Poll
fc8a43f4f1 debugd: add go package docs 2023-01-25 09:58:56 +01:00
Malte Poll
6f56ed69f8 debugd: implement upload of multiple binaries 2023-01-25 09:58:56 +01:00