Commit Graph

76 Commits

Author SHA1 Message Date
Moritz Sanft
9a282df846
ci: separate logs and metrics indices (#2544)
* separate logs and metrics indices

* tidy
2023-10-31 12:09:27 +01:00
Moritz Sanft
0c03076181
add missing field (#2542) 2023-10-31 10:12:49 +01:00
Moritz Sanft
005e865a13
cli: use state file on init and upgrade (#2395)
* [wip] use state file in CLI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use state file in CLI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

take clusterConfig from IDFile for compat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

various fixes

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

wip

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add GCP-specific values in Helm loader test

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove unnecessary pointer

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* write ClusterValues in one step

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move stub to test file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove mention of id-file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move output to `migrateTerraform`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* unconditional assignments converting from idFile

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* move require block in go modules file

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fall back to id file on upgrade

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix linter check

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add notice to remove Terraform state check on manual migration

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add `name` field

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

fix name tests

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* return early if no Terraform diff

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* return infrastructure state even if no diff exists

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add TODO to remove comment

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use state-file in miniconstellation

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* cli: remove id-file (#2402)

* remove id-file from `constellation create`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add file renaming to handler

* rename id-file after upgrade

* use idFile on `constellation init`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation verify`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* linter fixes

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation mini`

* remove id-file from `constellation recover`

* linter fixes

* remove id-file from `constellation terminate`

* fix initSecret type

* fix recover argument precedence

* fix terminate test

* generate

* add TODO to remove id-file removal

* Update cli/internal/cmd/init.go

Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>

* fix verify arg parse logic

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add version test

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from docs

* add file not found log

* use state-file in miniconstellation

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `constellation iam destroy`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove id-file from `cdbg deploy`

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>

* use state-file in CI

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update orchestration docs

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>
2023-10-09 13:04:29 +02:00
Moritz Sanft
f4b2d02194
ci: collect cluster metrics to OpenSearch (#2347)
* add Metricbeat deployment to debugd

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* set metricbeat debugd image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix k8s deployment

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use 2 separate deployments

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* only deploy via k8s in non-debug-images

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add missing tilde

* remove k8s metrics

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* unify flag

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add cloud metadata processor to filebeat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* ci: fix debugd logcollection (#2355)

* add missing keyvault access role

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bump logstash image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* bump filebeat / metricbeat image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* log used image version

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* use debugging image versions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* increase wait timeout for image upload

* add cloud metadata processor to filebeat

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix template locations in container

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix image version typo

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add filebeat / metricbeat users

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove user additions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* update workflow step name

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* only mount config files

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* document potential rc

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix IAM permissions in workflow

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix AWS permissions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* tidy

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add missing workflow input

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* rename action

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* pin image versions

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove unnecessary workflow inputs

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* add refStream input

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* remove inputs.yml dep

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* increase system metric period

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

* fix linkchecker

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>

---------

Signed-off-by: Moritz Sanft <58110325+msanft@users.noreply.github.com>
2023-09-27 16:17:31 +02:00
Daniel Weiße
7aba42baa5
ci: add more filters to e2e failure OpenSearch links (#2358)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-09-26 13:17:59 +02:00
3u13r
548bb2dfa6
debugd: send requests over lb (#2346) 2023-09-19 16:10:22 +02:00
Moritz Sanft
60bf770e62
ci: logcollection to OpenSearch in non-debug clusters (#2080)
* refactor `debugd` file structure

* create `hack`-tool to deploy logcollection to non-debug clusters

* integrate changes into CI

* update fields

* update workflow input names

* use `working-directory`

* add opensearch creds to upgrade workflow

* make template func generic

* make templating func generic

* linebreaks

* remove magic defaults

* move `os.Exit` to main package

* make logging index configurable

* make templating generic

* remove excess brace

* update fields

* copy fields

* fix flag name

* fix linter warnings

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* remove unused workflow inputs

* remove makefiles

* fix command

* bazel: fix output paths of container

This fixes the output paths of builds within the container by mounting
directories to paths that exist on the host. We also explicitly set the
output path in a .bazelrc to the user specific path. The rc file is
mounted into the container and overrides the host rc.
Also adding automatic stop in case start is called and a containers
is already running.
Sym links like bazel-out and paths bazel outputs should generally work
with this change.

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* tabs -> spaces

---------

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-08-21 08:01:33 +02:00
renovate[bot]
a5d6e80dd8
deps: update fedora:38 Docker digest to 61f921e (#2196)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2023-08-10 10:56:35 +02:00
Malte Poll
bd26e6bae7
cdbg: implement workspace / "-C" flag and "--bindir" (#2170) 2023-08-07 11:40:48 +02:00
Daniel Weiße
d1ace13713
cli: add --workspace flag to set base directory for Constellation workspace (#2148)
* Remove `--config` and `--master-secret` falgs

* Add `--workspace` flag

* In CLI, only work on files with paths created from `cli/internal/cmd`

* Properly print values for GCP on IAM create when not directly updating the config

---------

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2023-08-04 13:53:51 +02:00
Malte Poll
8da6a23aa5
bootstrapper: add fallback endpoint and custom endpoint to SAN field (#2108)
terraform: collect apiserver cert SANs and support custom endpoint

constants: add new constants for cluster configuration and custom endpoint

cloud: support apiserver cert sans and prepare for endpoint migration on AWS

config: add customEndpoint field

bootstrapper: use per-CSP apiserver cert SANs

cli: route customEndpoint to terraform and add migration for apiserver cert SANs

bootstrapper: change interface of GetLoadBalancerEndpoint to return host and port separately
2023-07-21 16:43:51 +02:00
renovate[bot]
c58b97de76
deps: update fedora:38 Docker digest to a134743 (#2003)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-07-04 14:35:40 +02:00
Malte Poll
264b2df902
deps: upgrade to Fedora 38 (#1909)
* image: upgrade mkosi distro version to Fedora 38
* image: remove downgrade of GCP kernel
* ci: upgrade expected measurements for Fedora 38
* deps: upgrade bazeldnf packages to Fedora 38
* deps: upgrade container images to Fedora 38
2023-06-15 16:50:35 +02:00
3u13r
a2c98eb1d5
Correctly deploy the AWS CCM (#1853)
* aws: stop using the imds api for tags

* aws: disable tags in imds api

* aws: only tag instances with non-lecagy tag

* bootstrapper: always let coredns run before cilium

* debugd: make debugd less noisy

* fixup fix aws imds test

* fixup unsued context

* move getting instance id to readInstanceTag
2023-06-13 09:58:39 +02:00
Adrian Stobbe
e738f15f0f
cdbg: make endpoint deployment failure more transparent (#1883)
* add retry + timeout + intercept grpc logs

* LogStateChanges inside grplog pkg

* remove retry and tj/assert

* rename nit

* Update debugd/internal/cdbg/cmd/deploy.go

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* Update debugd/internal/cdbg/cmd/deploy.go

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* paul feedback

* return waitFn instead of WaitGroup

* Revert "return waitFn instead of WaitGroup"

This reverts commit 45700f30e341ce3af509b687febbc0125f7ddb38.

* log routine inside debugd constructor

* test doubles names

* Update debugd/internal/cdbg/cmd/deploy.go

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>

* fix newDebugClient closeFn

---------

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-06-12 13:45:34 +02:00
Adrian Stobbe
4284f892ce
api: rename /api/versions to versionsapi and /api/attestationcfig to attestationconfigapi (#1876)
* rename to attestationconfigapi + put client and fetcher inside pkg

* rename api/version to versionsapi and put fetcher + client inside pkg

* rename AttestationConfigAPIFetcher to Fetcher
2023-06-07 16:16:32 +02:00
Otto Bittner
30f2b332b3
api: restructure api pkg (#1851)
* api: rename AttestationVersionRepo to Client
* api: move client into separate subpkg for
clearer import paths.
* api: rename configapi -> attestationconfig
* api: rename versionsapi -> versions
* api: rename sut to client
* api: split versionsapi client and make it public
* api: split versionapi fetcher and make it public
* config: move attestationversion type to config
* api: fix attestationconfig client test

Co-authored-by: Adrian Stobbe <stobbe.adrian@gmail.com>
2023-06-02 09:19:23 +02:00
Adrian Stobbe
b51cc52945
config: sign Azure versions on upload & verify on fetch (#1836)
* add SignContent() + integrate into configAPI

* use static client for upload versions tool; fix staticupload calleeReference bug

* use version to get proper cosign pub key.

* mock fetcher in CLI tests

* only provide config.New constructor with fetcher

Co-authored-by: Otto Bittner <cobittner@posteo.net>
Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
2023-06-01 13:55:46 +02:00
3u13r
e0285c122e
todo responsibilities and cleanup (#1837)
* chore: add TODO responsibilities

* chore: remove not needed TODOs

* chore: remove outdated migrations

* chore: remove resolved goleak exception

* chore: remove not needed cosign env

* config: add link to our Azure snp docs
2023-06-01 12:33:06 +02:00
renovate[bot]
1fde3929a7
deps: update fedora:37 Docker digest to ab2fd8d (#1846)
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: malt3 <mp@edgeless.systems>
2023-06-01 09:03:05 +02:00
renovate[bot]
cad859153b
deps: update fedora:37 Docker digest to 67870e4 (#1743)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-05-05 14:38:36 +02:00
renovate[bot]
e09243bfb3
deps: update fedora:37 Docker digest to e734322 (#1689)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-04-27 17:51:14 +02:00
Nils Hanke
cab6044f69 debugd: use nanosecond precision for logs 2023-03-21 17:50:12 +01:00
Nils Hanke
3fceb2207d debugd: Use very basic JSON regex filter before JSON filter 2023-03-21 10:32:33 +01:00
Paul Meyer
6c4ebe12f3 go: remove superfluous else block
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-20 08:41:01 -04:00
Paul Meyer
0036b24266 go: remove unused parameters
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-20 08:41:01 -04:00
renovate[bot]
4d618a4b99
deps: update fedora:37 Docker digest (#1448)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-17 18:47:36 +01:00
Nils Hanke
05e39d98c8 debugd: update Filebeat Dockerfile to 8.6.2 2023-03-13 18:39:33 +01:00
Nils Hanke
1d9d8af92a debugd: update Logstash Dockerfile to 8.6.1 2023-03-13 18:39:33 +01:00
Nils Hanke
2335d429a9 debugd: Enable ordered logs in Logstash 2023-03-13 18:39:33 +01:00
Malte Poll
bdba9d8ba6
bazel: add build files for go (#1186)
* build: correct toolchain order
* build: gazelle-update-repos
* build: use pregenerated proto for dependencies
* update bazeldnf
* deps: tpm simulator
* Update Google trillian module
* cli: add stamping as alternative build info source
* bazel: add go_test wrappers, mark special tests and select testing deps
* deps: add libvirt deps
* deps: go-libvirt patches
* deps: cloudflare circl patches
* bazel: add go_test wrappers, mark special tests and select testing deps
* bazel: keep gazelle overrides
* bazel: cleanup bazelrc
* bazel: switch CMakeLists.txt to use bazel
* bazel: fix injection of version information via stamping
* bazel: commit all build files
* dev-docs: document bazel usage
* deps: upgrade zig-cc for go 1.20
* bazel: update Perl for macOS arm64 & Linux arm64 support
* bazel: use static perl toolchain for OpenSSL
* bazel: use static protobuf (protoc) toolchain
* deps: add git and go to nix deps

Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-03-09 15:23:42 +01:00
renovate[bot]
83bea18a4f
deps: update fedora:37 Docker digest (#1274)
Co-authored-by: katexochen <katexochen@users.noreply.github.com>
2023-02-24 17:46:17 +01:00
Paul Meyer
deea806d9c Improve code sequences with multiple errs
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-02-20 12:08:24 -05:00
Paul Meyer
12c866bcb9 deps: replace multierr with native errors.Join
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-02-20 12:08:24 -05:00
renovate[bot]
1732795345
deps: update fedora:37 Docker digest (#1192)
Co-authored-by: katexochen <katexochen@users.noreply.github.com>
2023-02-15 13:28:53 +01:00
Otto Bittner
f204c24174 cli: add version validation and force flag
Version validation checks that the configured versions
are not more than one minor version below the CLI's version.
The validation can be disabled using --force.
This is necessary for now during development as the CLI
does not have a prerelease version, as our images do.
2023-02-08 12:30:01 +01:00
Malte Poll
8c57995468
debugd: keep mutex locked while receiving files (#1070) 2023-01-27 17:57:32 +01:00
Malte Poll
fc8a43f4f1 debugd: add go package docs 2023-01-25 09:58:56 +01:00
Malte Poll
6f56ed69f8 debugd: implement upload of multiple binaries 2023-01-25 09:58:56 +01:00
Paul Meyer
a31d79e9cb ci: curl flags
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2023-01-20 14:23:32 +01:00
Malte Poll
938f114086
ci: implement "console" stream for OS images (#969)
* image: add AUTOLOGIN environment variable to conditionally enable serial console login
* ci: implement "console" stream for OS images
* debugd: remove serial console login access code
2023-01-16 12:20:01 +01:00
renovate[bot]
868d911918
Update fedora:37 Docker digest to 99aa891 (#797)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-12-14 10:38:00 +01:00
Paul Meyer
e5e5d8eaae deubgd: add instance metadata to collected logs
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-12-13 18:24:48 +01:00
Paul Meyer
568f288f0d debugd: collect pod logs
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-12-13 18:24:48 +01:00
Paul Meyer
a9ed8c0191 e2e: enable systemd logcollection
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-12-06 18:05:32 +01:00
Paul Meyer
c52dfb09b9 debugd: run logcollector containers unprivileged
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-12-02 18:54:24 +01:00
Paul Meyer
67ce1ac3b2 debugd: update filebeat container
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-12-02 18:54:24 +01:00
Paul Meyer
de5c1f5fc8 debugd: build logstash image from fedora37
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-12-02 18:54:24 +01:00
Paul Meyer
474f7ad356 ci: build logcollector images
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-12-02 18:54:24 +01:00
Paul Meyer
4eeff3696d debugd: remove timeout for rpc calls
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-12-01 16:14:45 +01:00