Git-Mirrors/constellation
1
0
Fork 0
mirror of https://github.com/edgelesssys/constellation.git synced 2024-10-01 01:36:09 -04:00
Code Issues Packages Projects Releases Wiki Activity

dev-docs: refactor and add information for newbies (#1912)

Browse Source 
* refactor dev-docs structure and add information

* improve doc

* Update dev-docs/workflows/create-debug-cluster.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* Update dev-docs/workflows/create-debug-cluster.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* pr feedback daniel

* Update dev-docs/README.md

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>

* move to howto again

* split up dev-setup and pull-request into sep files

* fix backticks

* add writing style convention + testing repo

* remove OSS cluster + reduce plugins vs code

* update bazel pre-pr doc

* ghcr img private hint

* add fetch measurement + provider sub-directory hint

* add label doc + pr title check in template

* add OSS build comment

* Update CONTRIBUTING.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update CONTRIBUTING.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update dev-docs/README.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* Update dev-docs/workflows/dev-setup.md

Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>

* thomas feedback

* add go proverb mention

---------

Co-authored-by: Daniel Weiße <66256922+daniel-weisse@users.noreply.github.com>
Co-authored-by: Thomas Tendyck <51411342+thomasten@users.noreply.github.com>
This commit is contained in:
Adrian Stobbe 2023-06-19 17:39:43 +02:00 committed by GitHub
parent be4a636361
commit 7dcd8c3dab
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
15 changed files with 501 additions and 350 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/pull_request_template.md vendored
View File

@ -29,5 +29,7 @@ Feel free to edit, complete or extend this list while the PR is open.
<!-- Remove items that do not apply. For completed items, change [ ] to [x], or check after submitting. --> <!-- Remove items that do not apply. For completed items, change [ ] to [x], or check after submitting. -->
- [ ] Update [docs](https://github.com/edgelesssys/constellation/tree/main/docs) - [ ] Update [docs](https://github.com/edgelesssys/constellation/tree/main/docs)
<!-- more information in dev-docs/workflows/pull-request.md -->
- [ ] Add labels (e.g., for changelog category) - [ ] Add labels (e.g., for changelog category)
- [ ] Is PR title adequate for changelog?
- [ ] Link to Milestone - [ ] Link to Milestone

34
CONTRIBUTING.md
View File

@ -1,4 +1,6 @@
# First steps # Contributing
## First steps
Thank you for getting involved! Before you start, please familiarize yourself with the [documentation](https://docs.edgeless.systems/constellation). Thank you for getting involved! Before you start, please familiarize yourself with the [documentation](https://docs.edgeless.systems/constellation).
@ -12,12 +14,34 @@ If you want to support our development:
Constellation is licensed under the [AGPLv3](LICENSE). When contributing, you also need to agree to our [Contributor License Agreement](https://cla-assistant.io/edgelesssys/constellation). Constellation is licensed under the [AGPLv3](LICENSE). When contributing, you also need to agree to our [Contributor License Agreement](https://cla-assistant.io/edgelesssys/constellation).
# Developer docs ## Reporting issues and bugs, asking questions
This project uses the GitHub issue tracker. Check the existing issues before submitting to avoid duplicates.
Please report any security issue via a [private GitHub vulnerability report](https://github.com/edgelesssys/constellation/security/advisories/new) or write to <security@edgeless.systems>.
Your bug report should cover the following points:
* A quick summary and/or background of the issue
* Steps to reproduce (be specific, e.g., provide sample code)
* What you expected would happen
* What actually happens
* Further notes:
* Thoughts on possible causes
* Tested workarounds or fixes
## Contributing a pull request
See the [pull request workflow](./dev-docs/workflows/pull-request.md).
## Developer docs
We have some documentation, guidelines and conventions that you can go through to familiarize yourself with the project and get started hacking on it. We have some documentation, guidelines and conventions that you can go through to familiarize yourself with the project and get started hacking on it.
* Building, testing, deploying: [local development](/dev-docs/workflows/build-test-run.md) Here are some useful links inside `dev-docs`:
* [Code conventions](/dev-docs/conventions.md#code-conventions)
* [Process conventions](/dev-docs/conventions.md#process-conventions) * Building, testing, deploying: [local development](/dev-docs/workflows/build-develop-deploy.md)
* [General conventions](/dev-docs/conventions.md)
* [PR conventions](/dev-docs/workflows/pull-request.md#conventions)
* [Repository layout](/dev-docs/layout.md#repository-layout) * [Repository layout](/dev-docs/layout.md#repository-layout)
* [Upgrading Kubernetes](/dev-docs/workflows/upgrade-kubernetes.md) * [Upgrading Kubernetes](/dev-docs/workflows/upgrade-kubernetes.md)

3
debugd/README.md
View File

@ -16,7 +16,8 @@ make cdbg
``` ```
## debugd & cdbg usage ## debugd & cdbg usage
Before continuing, remeber to [set up](https://docs.edgeless.systems/constellation/getting-started/install#set-up-cloud-credentials) your cloud credentials for the CLI to work.
Before continuing, remember to [set up](https://docs.edgeless.systems/constellation/getting-started/install#set-up-cloud-credentials) your cloud credentials for the CLI to work.
With `cdbg` and `yq` installed in your path: With `cdbg` and `yq` installed in your path:

10
dev-docs/README.md Normal file
View File

@ -0,0 +1,10 @@
# Content structure
<!-- put most important links for quick reference here -->
To get started building and running Constellation: [local-development](./workflows/build-develop-deploy.md)
* `workflows`: What every developer will most likely need to build, develop, deploy.
* `howto`: Instructions for special deployments.
* `layout`: for a quick architectural overview of Constellation

110
dev-docs/conventions.md
View File

@ -1,39 +1,7 @@
# Process conventions # Writing to customers: style policy
## Pull request process Whenever you write text facing the customer (e.g, docs, warnings, errors), follow the [Microsoft Style Guide](https://learn.microsoft.com/en-us/style-guide/welcome/).
For quick reference, check [Top 10 tips for Microsoft style and voice](https://learn.microsoft.com/en-us/style-guide/top-10-tips-style-voice).
Submissions should remain focused in scope and avoid containing unrelated commits.
For pull requests, we employ the following workflow:
1. Fork the repository to your own GitHub account
2. Create a branch locally with a descriptive name
3. Commit changes to the branch
4. Write your code according to our development guidelines
5. Push changes to your fork
6. Clean up your commit history
7. Open a PR in our repository and summarize the changes in the description
## Reporting issues and bugs, asking questions
This project uses the GitHub issue tracker. Please check the existing issues before submitting to avoid duplicates.
To report a security issue, contact security@edgeless.systems.
Your bug report should cover the following points:
* A quick summary and/or background of the issue
* Steps to reproduce (be specific, e.g., provide sample code)
* What you expected would happen
* What actually happens
* Further notes:
* Thoughts on possible causes
* Tested workarounds or fixes
## Major changes and feature requests
You should discuss larger changes and feature requests with the maintainers. Please open an issue describing your plans.
[Run CI e2e tests](/.github/docs/README.md)
# Go code conventions # Go code conventions
@ -41,6 +9,8 @@ You should discuss larger changes and feature requests with the maintainers. Ple
Adhere to the style and best practices described in [Effective Go](https://golang.org/doc/effective_go.html). Read [Common Review Comments](https://github.com/golang/go/wiki/CodeReviewComments) for further information. Adhere to the style and best practices described in [Effective Go](https://golang.org/doc/effective_go.html). Read [Common Review Comments](https://github.com/golang/go/wiki/CodeReviewComments) for further information.
This project also aims to follow the [Go Proverbs](https://go-proverbs.github.io/).
## Linting ## Linting
This projects uses [golangci-lint](https://golangci-lint.run/) for linting. This projects uses [golangci-lint](https://golangci-lint.run/) for linting.
@ -51,7 +21,7 @@ It is also recommended to use golangci-lint (and [gofumpt](https://github.com/mv
## Logging ## Logging
We use a [custom subset](/internal/logger/) of [zap](https://pkg.go.dev/go.uber.org/zap) to provide logging for Constellation's services and components. We use a [custom subset](/internal/logger/) of [zap](https://pkg.go.dev/go.uber.org/zap) to provide logging for Constellation’s services and components.
Usage instructions can be found in the package documentation. Usage instructions can be found in the package documentation.
Certain components may further specify a subset of the logger for their use. For example, the CLI has a debug-only logger, restricting the use of the logger to only `Debugf()`. Certain components may further specify a subset of the logger for their use. For example, the CLI has a debug-only logger, restricting the use of the logger to only `Debugf()`.
@ -108,34 +78,6 @@ Further we try to adhere to the following guidelines:
As this project contains nested Go modules, we use a Go work file to ease integration with IDEs. You can find an introduction in the [Go workspace tutorial](https://go.dev/doc/tutorial/workspaces). As this project contains nested Go modules, we use a Go work file to ease integration with IDEs. You can find an introduction in the [Go workspace tutorial](https://go.dev/doc/tutorial/workspaces).
## Recommended VS Code Settings
The following can be added to your personal `settings.json`, but it is recommended to add it to
the `<REPOSITORY>/.vscode/settings.json` repo, so the settings will only affect this repository.
```jsonc
// Use gofumpt as formatter.
"gopls": {
"formatting.gofumpt": true,
},
// Use golangci-lint as linter. Make sure you've installed it.
"go.lintTool":"golangci-lint",
"go.lintFlags": ["--fast"],
// You can easily show Go test coverage by running a package test.
"go.coverageOptions": "showUncoveredCodeOnly",
// Executing unit tests with race detection.
// You can add preferences like "-v" or "-count=1"
"go.testFlags": ["-race"],
// Enable language features for files with build tags.
// Attention! This leads to integration/e2e tests being executed when
// running a package test within a package containing integration/e2e
// tests.
"go.buildTags": "integration e2e",
```
For some inexplicable reason, the `"go.lintTool":"golangci-lint",` might be overwritten. In case you don't get all linter suggestions, you might want to check the value of `go.lintTool` in the UI settings and make sure it is also set to `golangci-lint`.
Additionally, we use the [Redhat YAML formatter](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) to have uniform formatting in our `.yaml` files.
## Code documentation ## Code documentation
Documentation of the latest release are available on [pkg.go.dev](https://pkg.go.dev/github.com/edgelesssys/constellation/v2). Documentation of the latest release are available on [pkg.go.dev](https://pkg.go.dev/github.com/edgelesssys/constellation/v2).
@ -153,46 +95,18 @@ cd "${CONSTELLATION_DIR}
pkgsite pkgsite
``` ```
You can now view the documentation on http://localhost:8080/github.com/edgelesssys/constellation/v2 You can now view the documentation on <http://localhost:8080/github.com/edgelesssys/constellation/v2>
</details> </details>
## PR conventions
Our changelog is generated from PR titles. Which PR is listed in which category is determined by labels, see the [release.yml](/.github/release.yml).
The PR title should be structured in one of the following ways:
```
<module>: <title>
```
Where the `<module>` is
- the top level directory of the microservice or component, e.g., `joinservice`, `disk-mapper`, `upgrade-agent` but also `docs` and `rfc`
- in internal, the second level directory
- `deps` for dependency upgrades
- `ci` for things that are CI related
and `<title>` is all lower case (except proper names, including acronyms).
Ticket numbers shouldn't be part of the title.
In case the scope of your PR is too wide, use the alternative format.
```
<Title>
```
and `<Title>` starts with a capital letter.
## CLI reference ## CLI reference
The command reference within the CLI should follow the following conventions: The command reference within the CLI should follow the following conventions:
- Short description: Starts with a capital letter, beginnings of sentences, names and acronyms are capitalized, ends without a period. It should be a single sentence. * Short description: Starts with a capital letter, beginnings of sentences, names and acronyms are capitalized, ends without a period. It should be a single sentence.
- Long description: Starts with a capital letter, beginnings of sentences, names and acronyms are capitalized, ends with a period. * Long description: Starts with a capital letter, beginnings of sentences, names and acronyms are capitalized, ends with a period.
- If the long description contains multiple sentences, the first sentence is formatted as a long description, followed by 2 newlines and the rest of the sentences. The rest of the sentences start with a capital letter, beginnings of sentences, names and acronyms are capitalized and each sentence ends with a period. * If the long description contains multiple sentences, the first sentence is formatted as a long description, followed by 2 newlines and the rest of the sentences. The rest of the sentences start with a capital letter, beginnings of sentences, names and acronyms are capitalized and each sentence ends with a period.
- Flag: Starts with a lowercase letter, beginnings of sentences, names and acronyms are capitalized, ends without a period. * Flag: Starts with a lowercase letter, beginnings of sentences, names and acronyms are capitalized, ends without a period.
- If a flag contains multiple sentences, the first sentence is formatted as a normal flag, followed by a newline and the rest of the sentences. The rest of the sentences start with a capital letter, beginnings of sentences, names and acronyms are capitalized and each sentence ends with a period. * If a flag contains multiple sentences, the first sentence is formatted as a normal flag, followed by a newline and the rest of the sentences. The rest of the sentences start with a capital letter, beginnings of sentences, names and acronyms are capitalized and each sentence ends with a period.
## Naming convention ## Naming convention

106
dev-docs/workflows/bazel.md Normal file
View File

@ -0,0 +1,106 @@
# Bazel
Bazel is the primary build system for this project. It is used to build all Go code and will be used to build all artifacts in the future.
Still, we aim to keep the codebase compatible with `go build` and `go test` as well.
Whenever Go code is changed, you will have to run `bazel run //:tidy` to regenerate the Bazel build files for Go code.
## Bazel commands
### Build
#### Useful defaults
```sh
bazel build //bootstrapper/cmd/bootstrapper:bootstrapper # build bootstrapper
bazel build //cli:cli_oss # build CLI
bazel build //cli:cli_oss_linux_amd64 # cross compile CLI for linux amd64
bazel build //cli:cli_oss_linux_arm64 # cross compile CLI for linux arm64
bazel build //cli:cli_oss_darwin_amd64 # cross compile CLI for mac amd64
bazel build //cli:cli_oss_darwin_arm64 # cross compile CLI for mac arm64
```
#### General
* `bazel build //...` - build all targets (when `.bazeloverwriterc` is specified see [here](./build-develop-deploy.md#settings))
* `bazel build //subfolder/...` - build all targets in a subfolder (recursive)
* `bazel build //subfolder:all` - build all targets in a subfolder (non-recursive)
* `bazel build //subfolder:target` - build single target
### Run
* `bazel run --run_under="cd $PWD &&" //cli:cli_oss -- create -c 1 -w 1` - build + run a target with arguments in current working directory
### Pre-PR checks
* `bazel test //...` - run all tests
* `bazel run //:generate` - execute code generation
* `bazel run //:tidy` - tidy, format and generate
* `bazel run //:check` - execute checks and linters. To reduce verbosity of non-critical output, you can set `SILENT=1 bazel run //:check`
Note that its important to run `generate` before `check`. These checks are performed in the CI pipeline.
Also note that some errors shown in `check` (non-silent mode) by `golicenses_check` are ignored (for more see [golicenses.sh.in](../../bazel/ci/golicenses.sh.in)).
### Query
* `bazel query //...` - list all targets
* `bazel query //subfolder` - list all targets in a subfolder
* `bazel cquery --output=files //subfolder:target` - get location of a build artifact
### (Optional) Remote caching and execution
We use BuildBuddy for remote caching (and maybe remote execution in the future). To use it, you need to join the BuildBuddy organization and get an API key. Then, you can write it to `~/.bazelrc`:
```sh
build --remote_header=x-buildbuddy-api-key=<redacted>
```
To use the remote cache, build the project with `bazel build --config remote_cache //path/to:target`.
You can also copy the `remote_cache` config from `.bazelrc` to your `~/.bazelrc` and remove the `remote_cache` prefix to make it the default.
## Setup
### VS Code integration
You can continue to use the default Go language server and editor integration. This will show you different paths for external dependencies and not use the Bazel cache.
Alternatively, you can use [the go language server integration for Bazel](https://github.com/bazelbuild/rules_go/wiki/Editor-setup). This will use Bazel for dependency resolution and execute Bazel commands for building and testing.
### Command-line completion
[CLI completion for Bazel](https://bazel.build/install/completion) is available for Bash and zsh.
### Bash
When installing Bazel through the APT repository or Homebrew, completion scripts for bash should be installed automatically.
When building from source, you can install the completion script by adding the following line to your `~/.bashrc`:
```bash
source <path-to-constellation-repo>/bazel/bazel-complete.bash
```
### Zsh
When installing Bazel through the APT repository or Homebrew, completion scripts for zsh should be installed automatically. When using a heavily customized zsh config, you may need to follow [this workaround](https://bazel.build/install/completion).
When using Oh-My-Zsh, you can simply enable the [`zsh-autocomplete`](https://github.com/marlonrichert/zsh-autocomplete) plugin.
When building from source and not using Oh-My-Zsh, you can install the completion script as follows:
1. Locate the completion file, per default, it is located in `$HOME/.bazel/bin`
2. Add a file with the following to your `$fpath`
```zsh
fpath[1,0]=~/.zsh/completion/
mkdir -p ~/.zsh/completion/
cp /path/from/above/step/_bazel ~/.zsh/completion
```
3. When installing for the first time, you may need to run `rm -f ~/.zcompdump; compinit` to rebuild the completion cache.
4. (Optional) Add the following to your `.zshrc`
```zsh
# This way the completion script does not have to parse Bazel's options
# repeatedly. The directory in cache-path must be created manually.
zstyle ':completion:*' use-cache on
zstyle ':completion:*' cache-path ~/.zsh/cache
```

159
dev-docs/workflows/build-develop-deploy.md Normal file
View File

@ -0,0 +1,159 @@
# Getting started locally
The following are instructions for building all components in the constellation repository, except for images. A manual on how to build images locally can be found in the [image README](/image/README.md).
Prerequisites:
* 20GB (minimum), better 40 GB disk space (required if you want to cross compile for all platforms)
* [Latest version of Go](https://go.dev/doc/install).
* [Bazelisk installed as `bazel` in your path](https://github.com/bazelbuild/bazelisk/releases).
* [Docker](https://docs.docker.com/engine/install/). Can be installed with these commands on Ubuntu 22.04: `sudo apt update && sudo apt install docker.io`. As the build spawns docker containers your user account either needs to be in the `docker` group (Add with `sudo usermod -a -G docker $USER`) or you have to run builds with `sudo`. When using `sudo` remember that your root user might (depending on your distro and local config) not have the go binary in it's PATH. The current PATH can be forwarded to the root env with `sudo env PATH=$PATH <cmd>`.
## Prequisites
### Linux
* Packages on Ubuntu:
```sh
sudo apt install build-essential cmake libssl-dev pkg-config libcryptsetup12 libcryptsetup-dev
```
* Packages on Fedora:
```sh
sudo dnf install @development-tools pkg-config cmake openssl-devel cryptsetup-libs cryptsetup-devel
```
### Mac
* To fix unsupported shell options used in some build script:
```sh
brew install bash
```
* To troubleshoot potential problems with bazel on ARM architecture when running it for the first time, it might help to purge and retry:
```sh
bazel clean --expunge
```
## Settings
### Authenticate with the GitHub registry
Get a GitHub token with access to the registry and authenticate through `docker` (see [here](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry)).
Optionally, you can customize the default registry target:
```sh
echo "build --container_prefix=ghcr.io/USERNAME/constellation" >> .bazeloverwriterc
```
### Authenticate with your cloud provider
To provision the constellation cluster on the provider infrastructure, please authenticate with respective provider.
E.g. AWS:
```sh
aws login
```
For more details, see [here](https://docs.edgeless.systems/constellation/getting-started/install#set-up-cloud-credentials).
## Build
>IMPORTANT: The OSS version is not identical with the official release. Notably, when executing `constellation create` you will need to set the image version and measurements yourself.
Build the binaries always in a separate directory. Since you can only have one IAM + cluster configuration in one directory, it is recommended to create a sub-directory for each cloud provider. This way you can easily deploy your dev-build on each of the providers by switching to their sub-directory:
```sh
mkdir build && cd build
# ( mkdir (aws|gcp|azure) && cd (aws|gcp|azure) )
# build required binaries for a dev build
# and symlink them into the current directory
# also push the built container images
bazel run //:devbuild --cli_edition=oss --container_prefix=ghcr.io/USERNAME/constellation
./constellation ...
```
>IMPORTANT: New images are private by default. To set it to public see [here](https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility).
### Specialized Bazel build targets
See [bazel](./bazel.md#build).
## Develop & Contribute
### VS Code setup
We recommend to set up your IDE to conform with our conventions (see [here](./dev-setup.md)).
### Testing
See [here](./testing.md) on how to run tests and how we write tests.
### Contributing a PR
#### Pre-Opening a PR
Before opening a PR, please run [these Bazel targets](./bazel.md#pre-pr-checks)
#### PR-Guidelines
See [here](./pull-request.md).
## Deploy
> :warning: Debug images are not safe to use in production environments. :warning:
The debug images will open an additional **unsecured** port (4000) which accepts any binary to be run on the target machine. **Make sure that those machines are not exposed to the internet.**
### Cloud
To familiarize yourself with debugd and learn how to deploy a cluster using it, read [this](/debugd/README.md) manual.
If you want to deploy a cluster for production, please refer to our user documentation [here](https://docs.edgeless.systems/constellation/getting-started/first-steps#create-a-cluster).
### Locally
In case you want to have quicker iteration cycles during development you might want to setup a local cluster.
You can do this by utilizing our QEMU setup.
Instructions on how to set it up can be found in the [QEMU README](qemu.md).
## Image export
To download an image you will have to export it first.
Below you find general instructions on how to do this for GCP and Azure.
### GCP
In order to download an image you will have to export it to a bucket you have access to:
* "Owner" permissions on the project
* "Storage Admin" permissions on the bucket
* Export with:
```bash
gcloud compute images export --image=<image_path> --destination-uri=<bucket_uri> --export-format=qcow2 --project=<image_project>
```
* Click on "Download" on the created object
### Azure
To download an image from Azure you will have to create a disk from the image and generate a download link for that disk:
```bash
#!/usr/bin/env bash
VERSION=0.0.1
TARGET_DISK=export-${VERSION}
az disk create -g <resource_group> -l <target_region> -n $TARGET_DISK --hyper-v-generation V2 --os-type Linux --sku standard_lrs --security-type TrustedLaunch --gallery-image-reference <image_path>
az disk grant-access -n $TARGET_DISK -g constellation-images --access-level Read --duration-in-seconds 3600 | jq -r .accessSas
```
Depending on you internet connection you might have to modify the duration value.
The duration value specifies for how long the link is usable.

244
dev-docs/workflows/build-test-run.md
View File

@ -1,244 +0,0 @@
# Build
The following are instructions for building all components in the constellation repository, except for images. A manual on how to build images locally can be found in the [image README](/image/README.md).
Prerequisites:
* 20GB (minimum), better 40 GB disk space (required if you want to cross compile for all platforms)
* [Latest version of Go](https://go.dev/doc/install).
* [Bazelisk installed as `bazel` in your path](https://github.com/bazelbuild/bazelisk/releases).
* [Docker](https://docs.docker.com/engine/install/). Can be installed with these commands on Ubuntu 22.04: `sudo apt update && sudo apt install docker.io`. As the build spawns docker containers your user account either needs to be in the `docker` group (Add with `sudo usermod -a -G docker $USER`) or you have to run builds with `sudo`. When using `sudo` remember that your root user might (depending on your distro and local config) not have the go binary in it's PATH. The current PATH can be forwarded to the root env with `sudo env PATH=$PATH <cmd>`.
---
### On Linux
* Packages on Ubuntu:
```sh
sudo apt install build-essential cmake libssl-dev pkg-config libcryptsetup12 libcryptsetup-dev
```
* Packages on Fedora:
```sh
sudo dnf install @development-tools pkg-config cmake openssl-devel cryptsetup-libs cryptsetup-devel
```
### On Mac
```
brew install bash
```
to fix unsupported shell options used in some build script.
To troubleshoot potential problems with bazel on ARM architecture when running it for the first time, it might help to purge and retry:
```
bazel clean --expunge
```
---
Developer workspace:
```sh
mkdir build
cd build
# build required binaries for a dev build
# and symlink them into the current directory
# also push the built container images
# After the first run, set the pushed imaged to public.
bazel run //:devbuild --cli_edition=oss --container_prefix=ghcr.io/USERNAME/constellation
./constellation ...
```
Overwrite the default container_prefix in the `.bazeloverwriterc` in the root of the workspace:
```bazel
# cat .bazeloverwriterc
build --cli_edition=oss
build --container_prefix=ghcr.io/USERNAME/constellation
```
Bazel build:
```sh
bazel query //...
bazel build //path/to:target
bazel build //... # build everything (for every supported platform)
bazel build //bootstrapper/cmd/bootstrapper:bootstrapper # build bootstrapper
bazel build //cli:cli_oss # build CLI
bazel build //cli:cli_oss_linux_amd64 # cross compile CLI for linux amd64
bazel build //cli:cli_oss_linux_arm64 # cross compile CLI for linux arm64
bazel build //cli:cli_oss_darwin_amd64 # cross compile CLI for mac amd64
bazel build //cli:cli_oss_darwin_arm64 # cross compile CLI for mac arm64
```
## Remote caching and execution
We use BuildBuddy for remote caching (and maybe remote execution in the future). To use it, you need to join the BuildBuddy organization and get an API key. Then, you can write it to `~/.bazelrc`:
```
build --remote_header=x-buildbuddy-api-key=<redacted>
```
To use the remote cache, build the project with `bazel build --config remote_cache //path/to:target`.
You can also copy the `remote_cache` config from `.bazelrc` to your `~/.bazelrc` and remove the `remote_cache` prefix to make it the default.
# Test
You can run all integration like this:
```sh
ctest -j `nproc`
```
You can limit the execution of tests to specific targets with e.g. `ctest -R integration-node-operator`.
Some of the tests rely on libvirt and won't work if you don't have a virtualization capable CPU. You can find instructions on setting up libvirt in our [QEMU README](qemu.md).
Running unit tests with Bazel:
```sh
bazel test //...
```
# Opening a PR
Before opening a PR, please run the tests and
```
bazel run //:generate && bazel run //:tidy
bazel run //:check
```
The linter check doesn't work on Mac at the moment, but you can run the linter directly:
```
golangci-lint run
```
Furthermore, the PR titles are used for the changelog, so please stick to our [conventions](https://github.com/edgelesssys/constellation/blob/main/dev-docs/conventions.md#pr-conventions).
# Deploy
> :warning: Debug images are not safe to use in production environments. :warning:
The debug images will open an additional **unsecured** port (4000) which accepts any binary to be run on the target machine. **Make sure that those machines are not exposed to the internet.**
## Cloud
To familiarize yourself with debugd and learn how to deploy a cluster using it, read [this](/debugd/README.md) manual.
If you want to deploy a cluster for production, please refer to our user documentation [here](https://docs.edgeless.systems/constellation/getting-started/first-steps#create-a-cluster).
## Locally
In case you want to have quicker iteration cycles during development you might want to setup a local cluster.
You can do this by utilizing our QEMU setup.
Instructions on how to set it up can be found in the [QEMU README](qemu.md).
# Verification
In order to verify your cluster we describe a [verification workflow](https://docs.edgeless.systems/constellation/workflows/verify-cluster) in our official docs.
Apart from that you can also reproduce some of the measurements described in the [docs](https://docs.edgeless.systems/constellation/architecture/attestation#runtime-measurements) locally.
Use the provided scripts in `/image/measured-boot` to generated measurements for a built image. Measurements for release images are also available in our image API.
# Dependency management
## Go
Go dependencies are managed with [Go modules](https://blog.golang.org/using-go-modules) that are all linked from the main [go.work](/go.work) file.
[Follow the go documentation](https://go.dev/doc/modules/managing-dependencies) on how to use Go modules.
# Bazel
Bazel is the primary build system for this project. It is used to build all Go code and will be used to build all artifacts in the future.
Still, we aim to keep the codebase compatible with `go build` and `go test` as well.
Whenever Go code is changed, you will have to run `bazel run //:tidy` to regenerate the Bazel build files for Go code.
## Bazel commands
* `bazel query //...` - list all targets
* `bazel query //subfolder` - list all targets in a subfolder
* `bazel build //...` - build all targets
* `bazel build //subfolder/...` - build all targets in a subfolder (recursive)
* `bazel build //subfolder:all` - build all targets in a subfolder (non-recursive)
* `bazel build //subfolder:target` - build single target
* `bazel run --run_under="cd $PWD &&" //cli:cli_oss -- create -c 1 -w 1` - build + run a target with arguments in current working directory
* `bazel cquery --output=files //subfolder:target` - get location of a build artifact
* `bazel test //...` - run all tests
* `bazel run //:tidy` - tidy, format and generate
* `bazel run //:check` - execute checks and linters
* `bazel run //:generate` - execute code generation
## Editor integration
You can continue to use the default Go language server and editor integration. This will show you different paths for external dependencies and not use the Bazel cache.
Alternatively, you can use [the go language server integration for Bazel](https://github.com/bazelbuild/rules_go/wiki/Editor-setup). This will use Bazel for dependency resolution and execute Bazel commands for building and testing.
## Command-line completion
[CLI completion for Bazel](https://bazel.build/install/completion) is available for Bash and zsh.
### Bash
When installing Bazel through the APT repository or Homebrew, completion scripts for bash should be installed automatically.
When building from source, you can install the completion script by adding the following line to your `~/.bashrc`:
```bash
source <path-to-constellation-repo>/bazel/bazel-complete.bash
```
### Zsh
When installing Bazel through the APT repository or Homebrew, completion scripts for zsh should be installed automatically. When using a heavily customized zsh config, you may need to follow [this workaround](https://bazel.build/install/completion).
When using Oh-My-Zsh, you can simply enable the [`zsh-autocomplete`](https://github.com/marlonrichert/zsh-autocomplete) plugin.
When building from source and not using Oh-My-Zsh, you can install the completion script as follows:
1. Locate the completion file, per default, it is located in `$HOME/.bazel/bin`
2. Add a file with the following to your `$fpath`
```zsh
fpath[1,0]=~/.zsh/completion/
mkdir -p ~/.zsh/completion/
cp /path/from/above/step/_bazel ~/.zsh/completion
```
3. When installing for the first time, you may need to run `rm -f ~/.zcompdump; compinit` to rebuild the completion cache.
4. (Optional) Add the following to your `.zshrc`
```zsh
# This way the completion script does not have to parse Bazel's options
# repeatedly. The directory in cache-path must be created manually.
zstyle ':completion:*' use-cache on
zstyle ':completion:*' cache-path ~/.zsh/cache
```
# Image export
To download an image you will have to export it first.
Below you find general instructions on how to do this for GCP and Azure.
## GCP
In order to download an image you will have to export it to a bucket you have access to:
* "Owner" permissions on the project
* "Storage Admin" permissions on the bucket
* Export with:
```bash
gcloud compute images export --image=<image_path> --destination-uri=<bucket_uri> --export-format=qcow2 --project=<image_project>
```
* Click on "Download" on the created object
## Azure
To download an image from Azure you will have to create a disk from the image and generate a download link for that disk:
```bash
#!/usr/bin/env bash
VERSION=0.0.1
TARGET_DISK=export-${VERSION}
az disk create -g <resource_group> -l <target_region> -n $TARGET_DISK --hyper-v-generation V2 --os-type Linux --sku standard_lrs --security-type TrustedLaunch --gallery-image-reference <image_path>
az disk grant-access -n $TARGET_DISK -g constellation-images --access-level Read --duration-in-seconds 3600 | jq -r .accessSas
```
Depending on you internet connection you might have to modify the duration value.
The duration value specifies for how long the link is usable.

46
dev-docs/workflows/create-debug-cluster.md Normal file
View File

@ -0,0 +1,46 @@
# Creating a Debug cluster
A debug cluster allows quicker iteration cycles during development by being able to upload new bootstrapper binaries through the `cdbg` tool.
After building (see [here](./build-develop-deploy.md#build)), you can find all CLIs and binaries in the `build` directory.
The cluster creation mostly follows the [official docs instructions](https://docs.edgeless.systems/constellation/getting-started/first-steps), but varies slightly in the following steps:
`./constellation config generate <CSP>`
by default uses the referenced nightly image.
To replace them with the latest debug image, run
```sh
bazel run //internal/api/versionsapi/cli -- latest --ref main --stream debug
```
to fetch the latest version and insert in the `image` field of the config file.
Before cluster creation you need to configure the cluster as debug.
Set `debugCluster: true` in the config:
```sh
yq eval -i '.debugCluster=true' constellation-conf.yaml
```
Fetch measurements for the debug image:
```sh
./constellation config fetch-measurements --insecure
```
Create the cluster and deploy the debug images:
```sh
./constellation create ...
```
```sh
./cdbg deploy
```
Finally run:
```sh
./constellation init
```

46
dev-docs/workflows/dev-setup.md Normal file
View File

@ -0,0 +1,46 @@
# VS Code
## Recommended Settings
The following can be added to your personal `settings.json`, but it is recommended to add it to
the `<REPOSITORY>/.vscode/settings.json` repo, so the settings will only affect this repository.
```jsonc
// Use gofumpt as formatter.
"gopls": {
"formatting.gofumpt": true,
},
// Use golangci-lint as linter. Make sure you've installed it.
"go.lintTool":"golangci-lint",
"go.lintFlags": ["--fast"],
// You can easily show Go test coverage by running a package test.
"go.coverageOptions": "showUncoveredCodeOnly",
// Executing unit tests with race detection.
// You can add preferences like "-v" or "-count=1"
"go.testFlags": ["-race"],
// Enable language features for files with build tags.
// Attention! This leads to integration/e2e tests being executed when
// running a package test within a package containing integration/e2e
// tests.
"go.buildTags": "integration e2e",
```
For some inexplicable reason, the `"go.lintTool":"golangci-lint",` might be overwritten. In case you don't get all linter suggestions, you might want to check the value of `go.lintTool` in the UI settings and make sure it is also set to `golangci-lint`.
Additionally, we use the [Redhat YAML formatter](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) to have uniform formatting in our `.yaml` files.
## Recommended extensions
* Bazel (BazelBuild.vscode-bazel): Bazel syntax highlighting and more
* Go (golang.go): Go language support for VS Code
* HashiCorp Terraform (hashicorp.terraform): Syntax highlighting for Terraform files
* ShellCheck (timonwong.shellcheck): Shell script linter
* vscode-proto3 (zxh404.vscode-proto3): Protobuf language support
* Code Spell Checker (streetsidesoftware.code-spell-checker): Highlights potential spelling mistakes
* Helm Intellisense: (Tim-Koehler.helm-intellisense): Syntax highlighting and more for Helm charts
* YAML (redhat.vscode-yaml): YAML language support. (Does not work with Helm charts)
* markdownlint (DavidAnson.vscode-markdownlint): Markdown linter
## Bazel support
You might also consider to set up Bazel in the IDE (see [here](./bazel.md#vs-code-integration)).

59
dev-docs/workflows/pull-request.md Normal file
View File

@ -0,0 +1,59 @@
# Pull Request (PR)
## Process
Submissions should remain focused in scope and avoid containing unrelated commits.
For pull requests, we employ the following workflow:
1. Fork the repository to your own GitHub account
2. Create a branch locally with a descriptive name
3. Commit changes to the branch
4. Write your code according to our development guidelines
5. Push changes to your fork
6. Clean up your commit history
7. Open a PR in our repository and summarize the changes in the description
### Major changes and feature requests
You should discuss larger changes and feature requests with the maintainers. Please open an issue describing your plans.
[Run CI e2e tests](github-actions.md)
## Conventions
### Title
Our changelog is generated from PR titles, so please stick to the naming convention.
The PR title should be structured in one of the following ways:
```
<module>: <title>
```
Where the `<module>` is
* the top level directory of the microservice or component, e.g., `joinservice`, `disk-mapper`, `upgrade-agent` but also `docs` and `rfc`
* in internal, the second level directory
* `deps` for dependency upgrades
* `ci` for things that are CI related
and `<title>` is all lower case (except proper names, including acronyms).
Ticket numbers shouldn’t be part of the title.
In case the scope of your PR is too wide, use the alternative format.
```
<Title>
```
and `<Title>` starts with a capital letter.
### Labels
The labels are used for changelog generation (targeted at constellation users), so select the label with this purpose in mind.
To exclude the PR from changelog only use these labels:
* `no changelog` / `dependencies`
The changelog generation is described [here](https://docs.github.com/en/repositories/releasing-projects-on-github/automatically-generated-release-notes). This is our configuration [release.yml](/.github/release.yml).

23
dev-docs/workflows/testing.md Normal file
View File

@ -0,0 +1,23 @@
# Test
## Running tests
### Unit tests
Running unit tests with Bazel:
```sh
bazel test //...
```
### Integration tests
You can run all integration like this:
```sh
ctest -j `nproc`
```
You can limit the execution of tests to specific targets with e.g. `ctest -R integration-node-operator`.
Some of the tests rely on libvirt and won’t work if you don’t have a virtualization capable CPU. You can find instructions on setting up libvirt in our [QEMU README](qemu.md).

2
dev-docs/workflows/upgrade-kubernetes.md
View File

@ -49,4 +49,4 @@ Normally renovate will handle the upgrading of Kubernetes dependencies.
- Conduct e2e tests - Conduct e2e tests
- [Run the sonobuoy test suite against your branch](https://sonobuoy.io/) - [Run the sonobuoy test suite against your branch](https://sonobuoy.io/)
- [Run CI e2e tests](/.github/docs/README.md) - [Run CI e2e tests](github-actions.md)

5
dev-docs/workflows/verify-cluster.md Normal file
View File

@ -0,0 +1,5 @@
# Verification
In order to verify your cluster we describe a [verification workflow](https://docs.edgeless.systems/constellation/workflows/verify-cluster) in our official docs.
Apart from that you can also reproduce some of the measurements described in the [docs](https://docs.edgeless.systems/constellation/architecture/attestation#runtime-measurements) locally.
Use the provided scripts in `/image/measured-boot` to generated measurements for a built image. Measurements for release images are also available in our image API.