Commit Graph

458 Commits

Author SHA1 Message Date
Otto Bittner
9b75d651fc Run cert-manager startupapicheck with verbose flag 2022-11-23 11:16:16 +01:00
Leonard Cohnen
1e98b686b6 kubernetes: verify Kubernetes components 2022-11-23 10:48:03 +01:00
Otto Bittner
2c9ddbc6e7 Remove unused LoadConfig type 2022-11-23 08:49:22 +01:00
Otto Bittner
6b2d9d16f8 Remove obsolote revive comments 2022-11-23 08:35:12 +01:00
renovate[bot]
d8c553207b
Update Terraform google to v4.44.0 (#622)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-22 14:30:40 +01:00
Daniel Weiße
b915d03487
AB#2615 Update docs to new CSI installation method (#606)
* Update docs to new CSI installation method

* Fix invalid volume expansion option

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
Co-authored-by: Moritz Eckert <m1gh7ym0@gmail.com>
2022-11-22 09:36:08 +01:00
Otto Bittner
1362e40f53
Surpress argument-limit errors and add TODO. (#603) 2022-11-21 17:31:01 +01:00
Otto Bittner
adc09a1ad1
AB#2593: Deploy verification service via Helm (#594) 2022-11-21 17:06:41 +01:00
Daniel Weiße
1f9b6ba90f
Add debug logging for verify command (#610)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-11-21 17:02:33 +01:00
Otto Bittner
bdd9dd922b
AB#2589: Deploy operators via Helm (#575)
* Only deploy operators on GCP/Azure.
* cert-manager is now deployed by default (GCP/Azure)
* remove OLM
2022-11-21 10:35:40 +01:00
Daniel Weiße
9aa9c1bb49
AB#2275 Add azuredisk CSI driver (#548)
* Add azuredisk CSI driver

* Update Changelog

* Update chart using go generate

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-11-18 15:47:01 +01:00
renovate[bot]
54ef6d21f4
Update Terraform aws to v4.40.0 (#586)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-18 15:41:02 +01:00
renovate[bot]
86b03bf08e
Update Terraform azurerm to v3.32.0 (#588)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-18 14:57:34 +01:00
Malte Poll
9d4172002c Upgrade container images to Fedora 37 2022-11-18 10:37:45 +01:00
Malte Poll
74aabe86fa Move PCR[8] -> PCR[12] 2022-11-18 10:37:45 +01:00
Fabian Kammel
56dccb77b4
Merge back changes from v2.2.2 release (#580)
* prepare v2.2.2 release and update release.md
* Updated QEMU measurements
* Terraform GCP: Always use the local account for resource creation (#571)
* CoreOS is no longer used, change docs to OS.
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
Co-authored-by: Malte Poll <mp@edgeless.systems>
2022-11-18 10:24:45 +01:00
Daniel Weiße
b966f57a2f
AB#2554 GCP CSI driver deployment (#532)
* Allow enabling/disabling of CSI driver through config

* Fix inconsistent namespace parsing

* Deploy GCP CSI driver on init

* Update invalid pod tolerations

* Add generate script for CSI charts

* Update generateCilium script

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-11-18 10:05:02 +01:00
Fabian Kammel
feae4a86bc
reserve enough time for stable tests (#564)
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-11-17 17:30:35 +01:00
renovate[bot]
b7852665f3
Update Terraform google to v4.43.1 (#576)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-17 16:44:33 +01:00
Nils Hanke
6e5895f200 User-friendlier errors 2022-11-17 13:49:34 +01:00
Nils Hanke
e1d8926395 Terraform: Only rollback after we fully created the workspace 2022-11-17 13:49:34 +01:00
Nils Hanke
19fb6f1233 Make AWS vars passing consistent with other CSPs 2022-11-17 13:49:34 +01:00
Nils Hanke
158dfe0e2b Remove unused name parameter in CreateCluster 2022-11-17 13:49:34 +01:00
Nils Hanke
b9b618a1f0 Terraform: Try to init before destroy 2022-11-17 13:49:34 +01:00
Nils Hanke
f27af5b588 Terraform: Make variables writing retryable 2022-11-17 13:49:34 +01:00
Nils Hanke
e93527144e Terraform: Try to use existing files on partially unpacked workspace 2022-11-17 13:49:34 +01:00
Nils Hanke
4a2cba988c Create separate Terraform workspace directory 2022-11-17 13:49:34 +01:00
Malte Poll
df0cd43f92
Terraform GCP: Always use local account for resource creation (#571)
* Terraform GCP: Always use local account for resource creation
* Update CHANGELOG
2022-11-17 10:33:36 +01:00
Fabian Kammel
ca4764c466
Merge v2.2.1 changes back to main (#563)
* Bump version to v2.2.0

Signed-off-by: Daniel Weiße <dw@edgeless.systems>

* Fix release detection in pipeline

Signed-off-by: Daniel Weiße <dw@edgeless.systems>

* Update CHANGELOG for 2.2.1

Signed-off-by: Fabian Kammel <fk@edgeless.systems>

* bump constellation versions to 2.2.1

Signed-off-by: Fabian Kammel <fk@edgeless.systems>

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
Co-authored-by: Daniel Weiße <dw@edgeless.systems>
2022-11-16 11:13:10 +01:00
Leonard Cohnen
d86d82d2d4 helm: go generate 2022-11-15 18:24:07 +01:00
Fabian Kammel
bb76a4e4c8
AB#2512 Config secrets via env var & config refactoring (#544)
* refactor measurements to use consistent types and less byte pushing
* refactor: only rely on a single multierr dependency
* extend config creation with envar support
* document changes
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-11-15 15:40:49 +01:00
renovate[bot]
5009de823f
Update Terraform aws to v4.39.0 (#538)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-14 10:35:26 +01:00
renovate[bot]
7bcd4b2f73
Update Terraform azurerm to v3.31.0 (#539)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-14 10:34:54 +01:00
Daniel Weiße
a07cab4b97
Update go-tpm dependency (#533)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-11-14 09:02:56 +01:00
Paul Meyer
7aa7492474 Fix shellcheck warnings
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-11 13:40:13 +01:00
Nils Hanke
db27a6a0dd Increase timeout for fetch-measurements 2022-11-11 11:38:50 +01:00
Fabian Kammel
b92b3772ca
Remove access manager (#470)
* remove access manager from code base
* document new node ssh workflow
* keep config backwards compatible
* slow down link checking to prevent http 429
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-11-11 08:44:36 +01:00
Nils Hanke
d41174659b Print "Initializing cluster..." on stderr 2022-11-10 17:51:14 +01:00
Nils Hanke
bc584d61fa Switch spinner TTY detection to stderr 2022-11-10 17:51:14 +01:00
Fabian Kammel
81a5907f26
consistently use stdout and stderr (#502)
* consistently use stdout and stderr
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-11-10 10:27:24 +01:00
Fabian Kammel
0d12e37c96
Document exported funcs,types,interfaces and enable check. (#475)
* Include EXC0014 and fix issues.
* Include EXC0012 and fix issues.
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
Co-authored-by: Otto Bittner <cobittner@posteo.net>
2022-11-09 15:57:54 +01:00
Daniel Weiße
c9873f2bfb
AB#2523 Refactor GCP metadata/cloud API (#387)
* Refactor GCP metadata/cloud API

* Remove cloud controller manager from metadata package

* Remove PublicIP

* Move shared cloud packages

* Remove dead code

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-11-09 14:43:48 +01:00
Malte Poll
97bb0f4a91
Update terraform lock files to include hashes for all platforms (#499)
- linux_arm64
- linux_amd64
- darwin_arm64
- darwin_amd64
- windows_amd64
2022-11-09 14:23:51 +01:00
renovate[bot]
9191f8ac61
Update Terraform docker to v2.23.0 (#495)
* Update Terraform docker to v2.23.0
* Readd removed terraform lock hashes

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: Malte Poll <mp@edgeless.systems>
2022-11-09 13:35:17 +01:00
renovate[bot]
0e34d35404
Update Terraform google to v4.43.0 (#484)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-09 10:30:02 +01:00
renovate[bot]
b8acb5e448
Update Terraform aws to v4.38.0 (#464)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-08 18:34:45 +01:00
Daniel Weiße
011f9c597d
Bring in changes from release branch (#479)
* Bump version to v2.2.0

* Update changelog

* Fix release detection in pipeline

* Fix PKI selection in pipeline

* Set enforced measurements for AWS

* Update default images

* Fix release docs

* Update mini-con defaults

* Fix measurements action

* Fix syft env variable naming

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-11-08 18:32:59 +01:00
Nils Hanke
ee55584b90 AWS: Apply security group to worker nodes 2022-11-08 11:22:06 +01:00
Malte Poll
41668d50c2 Add recovery loadbalancer on AWS 2022-11-08 00:07:04 +01:00
Nils Hanke
759c626e0f AWS: Don't expose SSH debugging ports on the LB 2022-11-07 13:57:22 +01:00
Malte Poll
fa6dfdff4f
Mark externally managed terraform resources to make infrastructure terraform appliable throughout its lifetime (#442)
* Mark externally managed terraform resources to make infrastructure terraform appliable throughout its lifetime
* Use correct field for nat gateway

Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-07 11:04:10 +01:00
Malte Poll
ed58fcccd3
CI: Add secure boot prod keys (#462)
* Add production secure boot keys
* Refactor OS build and upload settings
2022-11-04 16:48:52 +01:00
3u13r
309a4b5196
cli: remove debug env check for AWS (#460) 2022-11-04 15:31:51 +01:00
Fabian Kammel
04d0c770af
limit aws cluster name len (#454)
* limit aws cluster name len down to 10, 32-character name limit in AWS
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-11-04 13:35:32 +01:00
Nils Hanke
19fd3a351a Make azureCVMRxp in upgradeplan.go case-insensitive 2022-11-04 12:57:24 +01:00
Nils Hanke
4d9fbdb3d3 CI: Use lowercase image name for fetching measurements 2022-11-04 12:57:24 +01:00
renovate[bot]
b89fae8062
Update Terraform azurerm to v3.30.0 (#452)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-04 12:34:03 +01:00
renovate[bot]
44b1a92d6b
Update fedora Docker digest to 455fec9 (#447)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
Co-authored-by: Nirusu <Nirusu@users.noreply.github.com>
2022-11-04 11:49:41 +01:00
renovate[bot]
f71073a77f
Update Terraform google to v4.42.1 (#434)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-11-04 10:14:13 +01:00
Otto Bittner
f164af29cf
AB#2583: deploy autoscaler via helm (#438) 2022-11-03 16:42:19 +01:00
Leonard Cohnen
0d0191ba4d aws: make CCM work 2022-11-02 23:29:04 +01:00
Leonard Cohnen
58d083a433 cli: pass AWS state disk type to terraform 2022-11-02 23:29:04 +01:00
Leonard Cohnen
dd007f4772 metadata: move subnetCIDR to InstanceMetadata 2022-11-02 23:29:04 +01:00
Leonard Cohnen
0cdc7886ee metadata: don't use podCIDR for Azure CCM setup 2022-11-02 23:29:04 +01:00
Leonard Cohnen
be2b38f2ac terraform: use HTTPS health check for AWS 2022-11-02 23:29:04 +01:00
Leonard Cohnen
7e385c4c86 terraform: use AWS launch templates 2022-11-02 23:29:04 +01:00
Leonard Cohnen
3dce7de0f1 helm chart loader: increase error verbosity 2022-11-02 23:29:04 +01:00
Leonard Cohnen
cc38506ffa cli: AWS does not use a service account 2022-11-02 23:29:04 +01:00
Leonard Cohnen
015b12d8ff attestation: use AWS attestation 2022-11-02 23:29:04 +01:00
Leonard Cohnen
37e8f5fc28 cilium: AWS support 2022-11-02 23:29:04 +01:00
Nils Hanke
8d097424a1 Remove separate function for yesFlag in terminate 2022-11-02 18:18:30 +01:00
Nils Hanke
ad871d1993 Prompt before termination 2022-11-02 18:18:30 +01:00
Nils Hanke
c922136cd4 Fix typos 2022-11-02 18:18:30 +01:00
Otto Bittner
e363f03240
AB#2582: deploy CNM via Helm (#423) 2022-11-02 17:47:10 +01:00
Leonard Cohnen
741684843c terraform: fix azure password constraints 2022-11-02 09:57:54 +01:00
Otto Bittner
30bdbd9b85
Add helm unittests (#380) 2022-10-31 19:25:02 +01:00
renovate[bot]
c9e6b4c5b6
Update Terraform azurerm to v3.29.1 (#405)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-10-31 10:45:56 +01:00
Daniel Weiße
79f52e67cb
Update go-tpm-tools to fix AWS PCR selection (#390)
* Update go-tpm-tools to fix AWS PCR selection

Signed-off-by: Daniel Weiße <dw@edgeless.systems>

* Ignore leaking glog go routine

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-10-28 17:57:24 +02:00
Paul Meyer
86906ac536 Use atomic.Bool, added in Go 1.19
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-10-28 16:06:53 +02:00
Otto Bittner
091e3b2b2b AB#2538: deploy CCM via Helm
Also move helmloader interface/stubs
2022-10-27 18:12:47 +02:00
Otto Bittner
009b2e67e3 Use .Release.Namespace instead of namespace value 2022-10-27 18:12:47 +02:00
Nils Hanke
34f729ccd2 Case insensitive replace for every user input that could break azurerm 2022-10-27 11:35:14 +02:00
Daniel Weiße
e66cb84d6e
AB#2532 Dont clean up workspace if rollback fails (#360)
* Dont clean up workspace if rollback fails

* Remove dependency on CSP from terminate

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-10-26 15:57:00 +02:00
Paul Meyer
c05b22f1dc
Remove dead code (#373)
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-10-26 10:29:28 +02:00
Malte Poll
fa63e51370
Fix "enforceIdKeyDigest" capitalization (#369)
* Fix "enforceIdKeyDigest" capitalization
* Convert "enforceIdKeyDigest" to string for config map
2022-10-25 16:29:28 +02:00
Malte Poll
2d121d9243
Replace interface{} -> any (#370) 2022-10-25 15:51:23 +02:00
Malte Poll
7592143a69
Join-service helm chart: use correct casing for provider name (#368) 2022-10-25 13:21:27 +02:00
Malte Poll
52f140a968
Pin terraform provider hashes (#361) 2022-10-25 10:10:46 +02:00
Daniel Weiße
b35b74b772
Use tags for UID and role parsing (#242)
* Apply tags to all applicable GCP resources

* Move GCP UID and role from VM metadata to labels

* Adjust Azure tags to be in line with GCP and AWS

* Dont rely on resource name to find resources

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-10-24 16:58:21 +02:00
Otto Bittner
c2814aeddb
AB#2504: Deploy join-service via helm (#358) 2022-10-24 12:23:18 +02:00
Daniel Weiße
c82d5ccba9
Hide cursor and fix dots (#217)
* Hide cursor and fix dots spinner

* Allow restarting of spinner

* Don't spin on non TTY output

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-10-21 14:26:42 +02:00
Nils Hanke
04c4cff9f6
AB#2436: Initial support for create/terminate AWS NitroTPM instances
* Add .DS_Store to .gitignore

* Add AWS to config / supported instance types

* Move AWS terraform skeleton to cli/internal/terraform

* Move currently unused IAM to hack/terraform/aws

* Print supported AWS instance types when AWS dev flag is set

* Block everything aTLS related (e.g. init, verify) until AWS attestation is available

* Create/Terminate AWS dev cluster when dev flag is set

* Restrict Nitro instances to NitroTPM supported specifically

* Pin zone for subnets

This is not great for HA, but for now we need to avoid the two subnets
ending up in different zones, causing the load balancer to not be able
to connect to the targets.

Should be replaced later with a better implementation that just uses
multiple subnets within the same region dynamically
based on # of nodes or similar.

* Add AWS/GCP to Terraform TestLoader unit test

* Add uid tag and create log group

Co-authored-by: Daniel Weiße <dw@edgeless.systems>
Co-authored-by: Malte Poll <mp@edgeless.systems>
2022-10-21 12:24:18 +02:00
Otto Bittner
07f02a442c
Refactor Helm deployments (#341)
* Wrap KMS deployment in one main chart that
deploys all other services. Other services will follow.
* Use .tgz via helm-package as serialization format
* Change Release type to carry chart as byte slice
* Remove KMSConfig
* Use json-schema to validate values
* Extend release.md to mention updating helm charts
2022-10-21 12:01:28 +02:00
Malte Poll
f3d78a573f Disable Azure VM agent and report VM as ready 2022-10-21 11:04:25 +02:00
Malte Poll
ed9acef9d4 Upgrade terraform azure provider to 3.28.0 2022-10-21 11:04:25 +02:00
Malte Poll
743f5fa627 Remove all traces of CoreOS from the codebase 2022-10-21 11:04:25 +02:00
Malte Poll
3b6ee703f5 Move PCR indices for owner ID and cluster ID 2022-10-21 11:04:25 +02:00
Malte Poll
34367ea3cc Create mkosi image build pipeline 2022-10-21 11:04:25 +02:00
Daniel Weiße
085f7b1a2a Prompt user for confirmation before overwriting config
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-10-20 15:35:31 +02:00
Otto Bittner
c6ccee1250 AB#2490: deploy KMS via Helm
* Bundle helm-install related code in speparate package
* Move cilium installation to new helm package
2022-10-18 13:33:37 +02:00
Otto Bittner
62168bbf98 AB#2490: Add KMS helm chart
* Also run helm-lint in CI now
2022-10-18 13:33:37 +02:00
renovate[bot]
9af0640aad
Update Terraform azurerm to v3.27.0 (#301)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-10-18 11:54:29 +02:00
Paul Meyer
01df06e142
Use HTTPS for kube lb health check on Azure (#305)
Signed-off-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-10-18 11:46:22 +02:00
renovate[bot]
c85dc674ba
Update Terraform libvirt to v0.7.0 (#304)
* Update Terraform libvirt to v0.7.0

* Use disk block

* Remove nulled disk options

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
2022-10-18 11:24:43 +02:00
renovate[bot]
0c0a83550d
Update Terraform google to v4.41.0 (#302)
Co-authored-by: Paul Meyer <49727155+katexochen@users.noreply.github.com>
2022-10-18 08:30:24 +02:00
Daniel Weiße
f068e50dee
Attestation logging (#275)
* Add section for checking joinservice logs

* Add logging for attestation validation

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-10-14 16:29:21 +02:00
Malte Poll
0c65e41dae Use worker count to create workers on azure (instead of control plane count) 2022-10-14 14:44:08 +02:00
github-actions[bot]
74c3c93dec
Update CLI reference (#248)
Co-authored-by: katexochen <49727155+katexochen@users.noreply.github.com>
2022-10-14 10:48:20 +02:00
renovate[bot]
b8d8562a6f Update Terraform random to v3.4.3 2022-10-14 09:13:35 +02:00
Paul Meyer
282117666e
Fix Azure Terraform for non-CVMs (#251) 2022-10-13 16:35:55 +02:00
katexochen
4b2dd1317a Normalize URIs for azurerm Terraform provider 2022-10-13 15:29:29 +02:00
katexochen
1556e239ca Remove state file 2022-10-13 15:29:29 +02:00
katexochen
0d1fd8fb2a Remove Azure client from CLI 2022-10-13 15:29:29 +02:00
katexochen
f4af9c56f5 Use Terraform for create Azure 2022-10-13 15:29:29 +02:00
katexochen
98a16b2b47 Create Terraform module for Azure
Co-authored-by: Benedict Schlueter <bs@edgeless.systems>
2022-10-13 15:29:29 +02:00
katexochen
a4a61e98ee Fix Terraform validation errors 2022-10-13 14:54:19 +02:00
Fabian Kammel
57b8efd1ec
Improve measurements verification with Rekor (#206)
Fetched measurements are now verified using Rekor in addition to a signature check.
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-10-11 13:57:52 +02:00
Paul Meyer
1c29638421
Use env to find bash in shebang (#225) 2022-10-10 14:21:17 +02:00
katexochen
10004875f4 Add spinner interrrupt for rollback 2022-10-10 13:43:15 +02:00
Daniel Weiße
0edae36e43
AB#2426 Mini Constellation (#198)
* Mini Constellation commands to quickly deploy a local Constellation cluster

* Download libvirt container image if not present locally

* Fix libvirt KVM permission issues by creating kvm group using host GID inside container

* Remove QEMU specific values from state file

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
Co-authored-by: Nils Hanke <nils.hanke@outlook.com>
2022-10-07 09:38:43 +02:00
Leonard Cohnen
92618d5284 align load balancer timeout 2022-10-07 03:38:05 +02:00
Paul Meyer
b668b8ed2b
Reduce activation indication movement (#215) 2022-10-06 11:20:01 +02:00
Fabian Kammel
369480a50b
Feat/revive (#212)
* enable revive as linter
* fix var-naming revive issues
* fix blank-imports revive issues
* fix receiver-naming revive issues
* fix exported revive issues
* fix indent-error-flow revive issues
* fix unexported-return revive issues
* fix indent-error-flow revive issues
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-10-05 15:02:46 +02:00
Daniel Weiße
2ea695896f
AB#2439 Containerized libvirt (#191)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-10-05 09:11:30 +02:00
Valentyn Yukhymenko
abe40de3e5
Activity indicator for init command (#207)
* first version of spinner

- implemented class with basic method
- covered with dummy test
- integrated with init command

* Style and license remarks

* fixed review remarks

* fixed typo + integration of spinner with terminate command

* integration of spinner with create command
2022-10-04 18:17:05 +02:00
katexochen
f69db6f26e Enable serial port in debug mode 2022-09-30 16:50:52 +02:00
katexochen
9a96f2ffe1 No public IPs for GCP instances 2022-09-30 16:50:52 +02:00
katexochen
ccbc3d9123 Remove exposure of qemu ip_range_start value 2022-09-30 16:50:52 +02:00
katexochen
feffe40987 Remove GCP client from CLI 2022-09-30 16:50:52 +02:00
katexochen
d973740b03 Use Terraform for create on GCP 2022-09-30 16:50:52 +02:00
Daniel Weiße
804c173d52
Use terraform in CLI to create QEMU cluster (#172)
* Use terraform in CLI to create QEMU cluster

* Dont allow qemu creation on os/arch other than linux/amd64

* Allow usage of --name flag for QEMU resources

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-09-26 15:52:31 +02:00
Daniel Weiße
30f0554168
AB#2262 Automatic recovery (#158)
* Update `constellation recover` to be fully automated

* Update recovery docs

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-09-26 09:57:40 +02:00
katexochen
ba6e41ed5c Upgrade go module to v2 2022-09-22 09:10:19 +02:00
katexochen
88d200232a Remove autoscaling from CLI and bootstrapper 2022-09-20 13:41:23 +02:00
3u13r
774e300a32
Constellation conformance mode (#161)
* add conformance mode
2022-09-20 10:07:55 +02:00
Daniel Weiße
9c00f4efc2
Enable GCP serial console for debug mode (#162)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-09-19 15:21:33 +02:00
Thomas Tendyck
72d5aa7558 docs: fix command in trusted launch workflow and add fetch-measurements 2022-09-14 18:26:41 +02:00
Daniel Weiße
e367e1a68b
AB#2261 Add loadbalancer for control-plane recovery (#151)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-09-14 13:25:42 +02:00
Daniel Weiße
1f4fb3feda
Fix manifest url (#128)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-09-12 14:33:08 +02:00
Thomas Tendyck
0952435e25 fix some doc links 2022-09-12 13:09:55 +02:00
Thomas Tendyck
4b36d3a930 cli: minor improvements of output 2022-09-12 12:56:29 +02:00
Thomas Tendyck
d83a5f8693 cli verify: remove ownerid 2022-09-12 08:50:36 +02:00
Thomas Tendyck
53560ca6c5 cli verify: revert flow change to print correct errors again 2022-09-12 08:50:36 +02:00
Thomas Tendyck
ab45d5fbfe tidy config 2022-09-12 08:49:51 +02:00
Leonard Cohnen
7163c161b6 Deploy Konnectivity 2022-09-09 17:26:02 +02:00
Thomas Tendyck
a85777fd02 enforce pcr4 2022-09-08 17:34:12 +02:00
Daniel Weiße
8cb155d5c5
AB#2260 Refactor disk-mapper recovery (#82)
* Refactor disk-mapper recovery

* Adapt constellation recover command to use new disk-mapper recovery API

* Fix Cilium connectivity on rebooting nodes (#89)

* Lower CoreDNS reschedule timeout to 10 seconds (#93)

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-09-08 14:45:27 +02:00
Nils Hanke
ce0edc8c80 Purge provider argument from constellation create and verify 2022-09-08 13:38:24 +02:00
Moritz Eckert
fb5faa681c
Add provider to license check (#88) 2022-09-08 11:02:04 +02:00
Fabian Kammel
e3ede64ae6
Document trusted launch on Azure (#48)
* Document trusted launch usage for Azure

Signed-off-by: Fabian Kammel <fk@edgeless.systems>

* there is no valid link because there is no valid release yet

Signed-off-by: Fabian Kammel <fk@edgeless.systems>

* fix link

Signed-off-by: Fabian Kammel <fk@edgeless.systems>

* fix linter issues

Signed-off-by: Fabian Kammel <fk@edgeless.systems>

* improve

* importAzure.sh: print final image ID

Signed-off-by: Fabian Kammel <fk@edgeless.systems>
Co-authored-by: Thomas Tendyck <tt@edgeless.systems>
2022-09-07 15:05:24 +02:00
Nils Hanke
30725bb0c6 Warn when a debug cluster is created 2022-09-07 13:27:15 +02:00
Nils Hanke
fe70231f2a Rename IsImageDebug -> IsDebugImage for consistency 2022-09-07 13:27:15 +02:00
Nils Hanke
72d4456b3f GCP: Only create debugd loadbalancer when debugCluster is set 2022-09-07 13:27:15 +02:00
Nils Hanke
d74c7a3769 Azure: Only create debugd loadbalancer when debugCluster is set 2022-09-07 13:27:15 +02:00
Nils Hanke
1a4b4f564a Remove firewall configuration and make it static with a debug flag 2022-09-07 13:27:15 +02:00
Thomas Tendyck
9d264604c0 cli: remove GCP ADC project name check 2022-09-07 10:29:41 +02:00
Malte Poll
47b3195bac
cli: azure scale set poller: check for power state of every instance (#78) 2022-09-06 10:05:51 +02:00
Fabian Kammel
020cf51fc6
AB#2392 Store serial logs in actions (#39)
Co-authored-by: Fabian Kammel <fk@edgeless.systems>
2022-09-05 18:12:46 +02:00
Malte Poll
c38a142d64 Kubernetes 1.25 preview
Signed-off-by: Malte Poll <mp@edgeless.systems>
2022-09-05 16:57:28 +02:00
Leonard Cohnen
e80948a263 add tags to cluster id file struct 2022-09-05 16:35:59 +02:00
Leonard Cohnen
7b00005ed6 fix qemu initialization 2022-09-05 16:35:59 +02:00
Otto Bittner
405db3286e AB#2386: TrustedLaunch support for azure attestation
* There are now two attestation packages on azure.
The issuer on the server side is created base on successfully
querying the idkeydigest from the TPM. Fallback on err: Trusted Launch.
* The bootstrapper's issuer choice is validated by the CLI's validator,
which is created based on the local config.
* Add "azureCVM" field to new "internal-config" cm.
This field is populated by the bootstrapper.
* Group attestation OIDs by CSP (#42)
* Bootstrapper now uses IssuerWrapper type to pass
the issuer (and some context info) to the initserver.
* Introduce VMType package akin to cloudprovider. Used by
IssuerWrapper.
* Extend unittests.
* Remove CSP specific attestation integration tests

Co-authored-by: <dw@edgeless.systems>
Signed-off-by: Otto Bittner <cobittner@posteo.net>
2022-09-05 12:03:48 +02:00
Nils Hanke
71fb62fe31 Remove note to instance types specifically 2022-09-05 09:36:58 +02:00
Thomas Tendyck
bd63aa3c6b add license headers
sed -i '1i/*\nCopyright (c) Edgeless Systems GmbH\n\nSPDX-License-Identifier: AGPL-3.0-only\n*/\n' `grep -rL --include='*.go' 'DO NOT EDIT'`
gofumpt -w .
2022-09-05 09:17:25 +02:00
Fabian Kammel
106635a9ee
Restructure config docs (#44)
* more guided UX when generating and filling in config
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-09-02 17:11:06 +02:00
Nils Hanke
c0bfb9b61e Add 'constellation config instance-types' 2022-09-02 07:04:11 -07:00
Nils Hanke
0aefe2c0ba Move instanceType from CLI to config 2022-09-02 07:04:11 -07:00
Moritz Eckert
b95f3dbc91
Add docs to repo (#38) 2022-09-02 11:52:42 +02:00
Leonard Cohnen
cce2575d68 remove broken test: create azure service account 2022-09-01 17:06:01 +02:00
Leonard Cohnen
00e72db5d8 write master secret after config verification 2022-09-01 16:43:54 +02:00
Fabian Kammel
6440904865
Ref/update cosign key (#31)
* use new cosign keypair
* use community images for production image heuristic
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-09-01 12:58:31 +02:00
3u13r
f649219cbf
Feat/cilium strict mode2.0 (#25)
* bump cilium helm charts

* integrate cilium strict mode v2
2022-08-31 15:37:07 +02:00
Otto Bittner
4adc19b7f5 AB#2350: Configurably enforce idkeydigest on Azure
* Add join-config entry for "enforceIdKeyDigest" bool
* Add join-config entry for "idkeydigest"
* Initially filled with TPM value from bootstrapper
* Add config entries for idkeydigest and enforceIdKeyDigest
* Extend azure attestation validator to check idkeydigest,
if configured.
* Update unittests
* Add logger to NewValidator for all CSPs
* Add csp to Updateable type

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>
Co-authored-by: Daniel Weiße <dw@edgeless.systems>
2022-08-31 15:26:04 +02:00
katexochen
10e5249631 Manual client secrets on azure 2022-08-31 14:10:08 +02:00
katexochen
1861dc2744 Tag Azure resources with UID 2022-08-31 14:10:08 +02:00
katexochen
f15605cb45 Manually manage resource group on Azure 2022-08-31 14:10:08 +02:00
Daniel Weiße
ce02878019
AB#2308 / AB#2317 constellation upgrade plan (#3)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-31 11:59:07 +02:00
Daniel Weiße
b27e205399
Use 4 vCPU instances by default (#24)
* Use 4 vcpu instances by default

* Remove 2 vcpu instance type option

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-31 10:33:33 +02:00
Fabian Kammel
778952e07c
AB#2287 support community image IDs (#9)
* support community image IDs
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-08-30 15:15:51 +02:00
Daniel Weiße
7c832273fd
AB#2309 constellation upgrade execute (#2)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-29 16:49:44 +02:00
Fabian Kammel
22c912a56d move nodestate and role
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-08-29 16:07:55 +02:00
Nils Hanke
6da228758c
GCP: Add more N2D VMs to supported list (#6) 2022-08-29 09:50:40 +02:00
Malte Poll
708c6e057e Remove azure single instance support (#402) 2022-08-26 11:45:32 +02:00
Malte Poll
716ba52588 create on Azure: Allow toggling between CVMs / Trusted Launch VMs (#401) 2022-08-25 15:24:31 +02:00
Fabian Kammel
45beec15f5 AB#2360 enterprise build tag (#397)
* enterprise build switch to disable license checking in default (OSS) version
* remove community license quota
* empty image references on OSS build in config
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-08-25 14:06:29 +02:00
katexochen
6b1c20792a Use the correct context package 2022-08-24 14:56:30 +02:00
katexochen
e761c9bf97 Manually manage GCP service accounts 2022-08-24 11:44:05 +02:00
Malte Poll
f9c70d5c5a constellation create azure: use custom poller to check for scale set creation (#394)
Signed-off-by: Malte Poll <mp@edgeless.systems>
2022-08-24 11:31:43 +02:00
Daniel Weiße
d1495e9285 Fix helm csp selection (#362)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-24 09:45:02 +02:00
katexochen
df9db94079 Add method for building resource names 2022-08-23 18:11:20 +02:00
katexochen
14ef07aca9 Add method for building resource URIs 2022-08-23 18:11:20 +02:00
katexochen
a02a46e454 Use multiple loadbalancers on GCP 2022-08-23 18:11:20 +02:00
katexochen
c954ec089f Check for 404 errors in GCP termination 2022-08-23 18:11:20 +02:00
katexochen
9f599c3993 Remove checks for GetState/SetState 2022-08-23 18:11:20 +02:00
katexochen
f28e00659c Use uber/multierr for error composition 2022-08-23 18:11:20 +02:00
katexochen
a859accf1f Use id file for init ip 2022-08-23 18:11:20 +02:00
katexochen
7bbcc564bb Refactor id file interaction
* Use IP instead of endpoint in clusterIDsFile
* Move and rename validateEnpoint to addPortIfMissing
* Refactor clusterIDsFile handling in verify cmd
2022-08-23 18:11:20 +02:00
katexochen
c2faa20d6e Fix naming in state file 2022-08-23 18:11:20 +02:00
Malte Poll
e841d9201b Use Azure CVMs in e2e tests 2022-08-19 18:22:55 +02:00
Malte Poll
5883278d4a Enable secure boot on Azure CVMs 2022-08-19 14:39:36 +02:00
Otto Bittner
0892525915 Switch to Azure CVMs 2022-08-19 14:39:36 +02:00
Malte Poll
402fc7761b Disable l7 proxy on QEMU (#378) 2022-08-19 08:44:36 +02:00
Fabian Kammel
82eb9f4544 AB#2299 License check in CLI during init (#366)
* license server interaction
* logic to read from license file
* print license information during init
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
Co-authored-by: Moritz Eckert <m1gh7ym0@gmail.com>
2022-08-16 16:06:38 +02:00
Fabian Kammel
170a8bf5e0 AB#2306 Public image sharing in Google (#358)
* document how to publicly share images in gcloud
* Write disclamer in debugd
* Add disclamer about debug images to contributing file
* Print debug banner on startup
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-08-16 15:53:54 +02:00
Daniel Weiße
ba4471a228 AB#2316 Configurable enforced PCRs (#361)
* Add warnings for non enforced, untrusted PCRs

* Fix global state in Config PCR map

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-12 15:59:45 +02:00
3u13r
9478303f80 deploy cilium via helmchart (#321) 2022-08-12 10:20:19 +02:00
Daniel Weiße
8f5f84deb5 AB#2305 Fix missing atls verifier in init call (#352)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-09 14:04:40 +02:00
Daniel Weiße
60d5578475 AB#2215 Perform sanity check on GCP projectID (#349)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-09 10:26:29 +02:00
Daniel Weiße
ab536ae3c8 AB#2278 Remove hardcoded values from config (#346)
* Update file handler to avoid incorrect usage of file.Option

* Remove hardcoded values

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-08 11:04:17 +02:00
Malte Poll
bf5816cc00 linter cleanup (#344)
* go fmt
* static check
2022-08-05 15:30:23 +02:00
Daniel Weiße
8895693ae2 AB#2251 Parallel Azure scale set creation (#318)
* Parallel Azure scale set creation

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-05 10:35:38 +02:00
Malte Poll
081dfb5037 Upgrade Azure SDK
Signed-off-by: Malte Poll <mp@edgeless.systems>
2022-08-05 10:35:38 +02:00
Daniel Weiße
19871ee422 Enable integrity protection on boot (#300)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-02 12:35:23 +02:00
Daniel Weiße
aa7fcce8af Add configurable node disk type (#317)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-02 12:24:55 +02:00
Fabian Kammel
050e8fdc4a AB#2159 Feat/cli/fetch measurements (#301)
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-08-01 09:37:05 +02:00
Daniel Weiße
7baf98f014 Add test vectors for key derivation functions (#320)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-08-01 09:11:13 +02:00
Daniel Weiße
9a3bd38912 Generate random salt for key derivation on init (#309)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-07-29 09:52:47 +02:00
Daniel Weiße
a3a85b31cf Remove mentions of unique ID (#311)
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-07-27 16:10:50 +02:00
Moritz Eckert
ad02249b9a Add VerifyService port to GCP LB (#291)
* Add VerifyService port to GCP LB

* cli verify command: Use verify service port by default

Co-authored-by: Malte Poll <mp@edgeless.systems>
2022-07-26 16:35:14 +02:00
Daniel Weiße
db79784045 AB#2200 Merge Owner and Cluster ID (#282)
* Merge Owner and Cluster ID into single value

* Remove aTLS from KMS, as it is no longer used for cluster external communication

* Update verify command to use cluster-id instead of unique-id flag

* Remove owner ID from init output

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-07-26 10:58:39 +02:00
Otto Bittner
c743398a23 AB#2181: retry k8s downloads (#286)
Generalize retrier:
* Generalize Do to use a supplied 'retriable' function
* Make clock an optional argument in NewIntervalRetrier
* Move grpc/retrier to interal package
* Update existing unittests to not use retry feature

Add retryDownloadToTempDir:
* Wrap downloadToTempDir with retrier.
* Retry if TCP connection is reset.
* Abort by canceling the context.
* Use a mock server in the unit test that serves responses
depending on the state received through a state channel.

Co-authored-by: katexochen <49727155+katexochen@users.noreply.github.com>
2022-07-21 15:20:12 +02:00
Fabian Kammel
ba5a3aefe3 fix ci-lint issues (#287)
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-07-20 16:44:41 +02:00
Otto Bittner
a68ee817ff AB#2074: Choosable K8S Version (#277)
AB#2074: Add configurable k8s version

Configurable version flow:
* cli config holds/validates k8sVersion
* InitCluster receive a k8sVersion arg
* InitCluster creates CM "k8s-version"
* kubeadm's InitConfiguration receives k8sVersion
* joinservice spec mounts/reads k8s-version CM
* joinservice supplies k8sVersion via JoinTicketResponse
Other changes:
* Remove unused test code (FakeK8SClient)
* move VersionConfig map to /internal/versions
* installk8sComponents is now a function instead of a method
2022-07-18 12:28:02 +02:00
Fabian Kammel
a931f6692f Fix/bootstrapper regressions (#274)
* remove wireguard from e2e tests, conformance docs & config
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-07-15 11:53:14 +02:00
Malte Poll
260d2571c1 Only upload kubeadm certs if key is rotated
Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
Co-authored-by: 3u13r <lc@edgeless.systems>
2022-07-14 17:25:18 +02:00
Malte Poll
5d54ce689b Print kubeadm init/join output on success 2022-07-14 17:25:18 +02:00
katexochen
66b573ea5d Bootstrapper 2022-07-14 17:25:18 +02:00
katexochen
dea23604fb Bootstrapper 2022-07-14 17:25:18 +02:00
katexochen
916e5d6b55 Rename coordinator to bootstrapper and rename roles 2022-07-14 17:25:18 +02:00
Malte Poll
3280ed200c Test IntervalRetrier 2022-07-14 17:25:18 +02:00
katexochen
f79674cbb8 Bootstrapper 2022-07-14 17:25:18 +02:00
katexochen
32f1f5fd3e Delete Coordinator core and apis 2022-07-14 17:25:18 +02:00
Nils Hanke
14a15e131a Modify accepted list of Azure VM types (#250)
* Add more instances types for Azure (with commented out entries)

* Remove commented out entries

* Only AMD VMs

* Comment out CVMs (not supported yet)

* Adjust comments
2022-07-10 13:27:05 +02:00
Fabian Kammel
b4fd4fbacd Fix/add verify grpc port to lb (#262)
* Add verify port to lb
* Use correct health probe
Signed-off-by: Fabian Kammel <fk@edgeless.systems>
2022-07-08 18:18:23 +02:00
Nils Hanke
bc5471e9b3 Delete cluster IDs file on terminate 2022-07-05 14:41:58 +02:00
Nils Hanke
259c88fa1a IDsFilename -> ClusterIDsFilename 2022-07-05 14:41:58 +02:00
Thomas Tendyck
70efb92adc cli: fix vale lint errors in verify description 2022-07-04 12:19:38 +02:00
cm
3177b2fdb7 AB#2032 Write IDs to disk and read when verifying (#212)
* AB#2032 Write IDs to disk and read when verifying

* Update CHANGELOG.md

* update changelog

* update changelog

* cli verify: prefer flag values

* Rename fid file

Co-authored-by: Thomas Tendyck <tt@edgeless.systems>
2022-07-01 10:57:29 +02:00
Otto Bittner
7cada2c9e8 Add goleak to all tests (#227)
* Run goleak as part of all tests
We are already using goleak in various tests.
This commit adds a TestMain to all remaining tests
and calls goleak.VerifyTestMain in them.
* Add goleak to debugd/deploy package and fix bug.
* Run go mod tidy
* Fix integration tests
* Move goleak invocation for mount integration test
* Ignore leak in state integration tests

Co-authored-by: Fabian Kammel <fk@edgelss.systems>
2022-06-30 15:24:36 +02:00
Daniel Weiße
f9a581f329 Add aTLS endpoint to KMS (#236)
* Move file watcher and validator to internal

* Add aTLS endpoint to KMS for Kubernetes external requests

* Update Go version in Dockerfiles

* Move most KMS packages to internal

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-06-29 16:13:01 +02:00
Daniel Weiße
042f668d20 AB#2190 Verification service (#232)
* Add verification service

* Update verify command to use new Constellation verification service

* Deploy verification service on cluster init

* Update pcr-reader to use verification service

* Add verification service build workflow

Signed-off-by: Daniel Weiße <dw@edgeless.systems>

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-06-28 17:03:28 +02:00
Nils Hanke
e3f78a5bff Remove passing context seperately to initialize 2022-06-28 13:55:50 +02:00
Leonard Cohnen
e13f4d84c3 add gcp loadbalancer 2022-06-23 14:00:20 +02:00
Christoph Meyer
1e11188dac AB#2033 User-friendly wrap and reword errors
fix: readOrGenerated function signature
2022-06-22 12:02:10 +01:00
Christoph Meyer
9441e46e4b AB#2033 Remove redundant "failed" in error wrapping
Remove "failed" from wrapped errors
Where appropriate rephrase "unable to/could not" to "failed" in root
errors
Start error log messages with "Failed"
2022-06-22 12:02:10 +01:00
Fabian Kammel
0c9ca50be8 Feat/more version info (#224) 2022-06-21 15:12:27 +02:00
Fabian Kammel
392ad7fe45 Create Application Insights early so they are ready when VM needs them. (#213) 2022-06-15 12:19:41 +02:00
Daniel Weiße
1e19e64fbc Dynamic grpc client credentials (#204)
* Add an aTLS wrapper for grpc credentials

* Move grpc dialers to internal and use aTLS grpc credentials

Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-06-13 11:40:27 +02:00
Fabian Kammel
84552ca8f7 AB#2104 Feat/azure logging (#198)
implementation for azure early boot logging
2022-06-10 13:18:30 +02:00
Daniel Weiße
691ab84326 Update version variable
Signed-off-by: Daniel Weiße <dw@edgeless.systems>
2022-06-08 17:17:06 +02:00