Git-Mirrors/veilid
1
0
Fork 0
mirror of https://gitlab.com/veilid/veilid.git synced 2024-10-01 01:26:08 -04:00
Code Issues Packages Projects Releases Wiki Activity
2a73946246
veilid/.gitlab-ci.yml

181 lines
5.5 KiB
YAML
Raw Normal View History

cleanup and stats accounting organization
2021-11-26 09:54:38 -05:00
variables:
use new runner
2022-11-22 17:16:53 -05:00
NO_DOCKER: 1
FORCE_COLOR: 1
EARTHLY_EXEC_CMD: "/bin/sh"
Add submodule strategyt to build
2022-11-27 00:18:08 -05:00
GIT_SUBMODULE_STRATEGY: normal
cleanup and stats accounting organization
2021-11-26 09:54:38 -05:00
Add merge_requests as build target, split into two architectures
2023-02-20 14:14:05 -05:00
stages:
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
- prepare
Update .gitlab-ci.yml file to the intermediary package distribution pipeline.
2023-06-20 22:46:05 -04:00
- test
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
- build_packages
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 20:59:23 -04:00
- release
Update .gitlab-ci.yml file to the intermediary package distribution pipeline.
2023-06-20 22:46:05 -04:00
- distribute
initial CI attempt
2021-11-24 16:49:13 -05:00
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
# base earthly setup for jobs
.base:
tags: [ saas-linux-medium-amd64 ]
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
image: docker
services:
- docker:dind
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
.earthly: &earthly_setup
- apk update && apk add git
- wget https://github.com/earthly/earthly/releases/download/v0.7.15/earthly-linux-amd64 -O /usr/local/bin/earthly
- chmod +x /usr/local/bin/earthly
- earthly bootstrap
- echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
Use $CI_REGISTRY_IMAGE for the registry path GitLab will return an error if you have an upper case letter at the start of your username. ``` invalid reference format: repository name must be lowercase ``` The built-in `$CI_REGISTRY_IMAGE` variable does the right thing. Closes https://gitlab.com/veilid/veilid/-/issues/352
2023-12-25 16:14:23 -05:00
- test "$CI_PROJECT_PATH" != "veilid/veilid" && project_args="--CI_REGISTRY_IMAGE=$CI_REGISTRY_IMAGE"
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
# Create the build container if:
# - no container in the registry
# - run as part of a schedule
# - run manually from the pipelines web page
build_cache:
extends: .base
stage: prepare
script:
Use $CI_REGISTRY_IMAGE for the registry path GitLab will return an error if you have an upper case letter at the start of your username. ``` invalid reference format: repository name must be lowercase ``` The built-in `$CI_REGISTRY_IMAGE` variable does the right thing. Closes https://gitlab.com/veilid/veilid/-/issues/352
2023-12-25 16:14:23 -05:00
- if ! docker manifest inspect $CI_REGISTRY_IMAGE/build-cache:latest > /dev/null || [[ "$CI_PIPELINE_SOURCE" == "schedule" ]] || [[ "$CI_PIPELINE_SOURCE" == "web" ]] ; then
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
- *earthly_setup
- earthly --use-inline-cache --save-inline-cache --strict --push -P +build-linux-cache $project_args
- else
- echo "No need to rebuild"
- fi
# Runs a basic unit test build, this task will use the `build-cache:latest` as set up in the projects Container Registry
test_build:
extends: .base
stage: test
Tests passed. This commit is a clean up of .gitlab-ci.yml
2023-08-22 17:49:55 -04:00
only:
CI config and Changelog tweaks Fixed a few typos before releasing v0.2.1
2023-09-03 20:43:46 -04:00
- main
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
- pushes
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
- merge_requests
script:
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
- *earthly_setup
- earthly --use-inline-cache +unit-tests-linux --BASE=container $project_args
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
resource_group: test
Removed the manual trigger from tests -- YOLO
2023-11-02 21:48:07 -04:00
# when: manual
Added ephemeral create/delete to test stage.
2023-07-22 13:07:00 -04:00
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
release_job:
stage: release
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
image: registry.gitlab.com/gitlab-org/release-cli:latest
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
script:
- echo "running release_job"
release: # See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
tag_name: '$CI_COMMIT_TAG'
description: '$CI_COMMIT_TAG'
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Added jobs to create and delete the ephemeral build machines.
2023-07-21 15:30:33 -04:00
create_build_machines:
stage: build_packages
tags:
- build-orchestration
script:
Modified path to CICD scripts in CI config I thought CICD's working directory was in the project root but the release failed to find the scripts. I've changed the script executions to absolute paths. There's a directory is named for the runner's ID, is different on each machine, and changes if the runner is replaced. There's a variable that should overcome this, CI_RUNNER_ID, which I've used in the asbolute paths. Fingers crossed, let's try it again.
2024-04-28 19:21:39 -04:00
- /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/build-orchestration/build-machine-ctl.sh create amd64-deb
- /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/build-orchestration/build-machine-ctl.sh create arm64-deb
- /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/build-orchestration/build-machine-ctl.sh create amd64-rpm
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
package_amd64_deb:
stage: build_packages
Added jobs to create and delete the ephemeral build machines.
2023-07-21 15:30:33 -04:00
needs:
- create_build_machines
Update .gitlab-ci.yml file to the intermediary package distribution pipeline.
2023-06-20 22:46:05 -04:00
tags:
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
- build-amd64-deb
Update .gitlab-ci.yml file to the intermediary package distribution pipeline.
2023-06-20 22:46:05 -04:00
script:
Update .gitlab-ci.yml file
2023-07-04 14:31:02 -04:00
- earthly bootstrap
Update .gitlab-ci.yml file
2023-07-03 22:50:50 -04:00
- earthly +package-linux-amd64-deb
Modified path to CICD scripts in CI config I thought CICD's working directory was in the project root but the release failed to find the scripts. I've changed the script executions to absolute paths. There's a directory is named for the runner's ID, is different on each machine, and changes if the runner is replaced. There's a variable that should overcome this, CI_RUNNER_ID, which I've used in the asbolute paths. Fingers crossed, let's try it again.
2024-04-28 19:21:39 -04:00
- /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/build-machine/scp-to-orchestrator.sh
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
package_arm64_deb:
stage: build_packages
Added jobs to create and delete the ephemeral build machines.
2023-07-21 15:30:33 -04:00
needs:
- create_build_machines
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
tags:
- build-arm64-deb
script:
- earthly bootstrap
Removed amd64 from package job name, added arm64.deb to package job scripts
2023-07-15 00:37:30 -04:00
- earthly +package-linux-arm64-deb
Modified path to CICD scripts in CI config I thought CICD's working directory was in the project root but the release failed to find the scripts. I've changed the script executions to absolute paths. There's a directory is named for the runner's ID, is different on each machine, and changes if the runner is replaced. There's a variable that should overcome this, CI_RUNNER_ID, which I've used in the asbolute paths. Fingers crossed, let's try it again.
2024-04-28 19:21:39 -04:00
- /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/build-machine/scp-to-orchestrator.sh
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Update .gitlab-ci.yml file
2023-07-04 12:34:48 -04:00
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
package_amd64_rpm:
stage: build_packages
Added jobs to create and delete the ephemeral build machines.
2023-07-21 15:30:33 -04:00
needs:
- create_build_machines
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
tags:
- build-amd64-rpm
script:
- earthly bootstrap
- earthly +package-linux-amd64-rpm
Modified path to CICD scripts in CI config I thought CICD's working directory was in the project root but the release failed to find the scripts. I've changed the script executions to absolute paths. There's a directory is named for the runner's ID, is different on each machine, and changes if the runner is replaced. There's a variable that should overcome this, CI_RUNNER_ID, which I've used in the asbolute paths. Fingers crossed, let's try it again.
2024-04-28 19:21:39 -04:00
- /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/build-machine/scp-to-orchestrator.sh
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 20:59:23 -04:00
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
publish_crates:
moved crates and pypi publishing to build_packages stage
2024-04-04 20:53:28 -04:00
stage: build_packages
Update .gitlab-ci.yml file
2023-07-16 15:18:35 -04:00
tags:
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
- build-amd64-deb
Update .gitlab-ci.yml file
2023-07-16 15:18:35 -04:00
script:
Modified path to CICD scripts in CI config I thought CICD's working directory was in the project root but the release failed to find the scripts. I've changed the script executions to absolute paths. There's a directory is named for the runner's ID, is different on each machine, and changes if the runner is replaced. There's a variable that should overcome this, CI_RUNNER_ID, which I've used in the asbolute paths. Fingers crossed, let's try it again.
2024-04-28 19:21:39 -04:00
- cd /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/
Added crates.io and Pypi release steps to build_repositories job. Maybe it works, maybe it doesn't. We'll find out at next release.
2024-03-03 16:19:21 -05:00
- vlt login
- vlt run --command="cargo publish -p veilid-tools --dry-run"
- vlt run --command="cargo publish -p veilid-tools"
- vlt run --command="cargo publish -p veilid-core --dry-run"
- vlt run --command="cargo publish -p veilid-core"
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
publish_python:
moved crates and pypi publishing to build_packages stage
2024-04-04 20:53:28 -04:00
stage: build_packages
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
tags:
- build-amd64-deb
script:
- vlt login
Modified path to CICD scripts in CI config I thought CICD's working directory was in the project root but the release failed to find the scripts. I've changed the script executions to absolute paths. There's a directory is named for the runner's ID, is different on each machine, and changes if the runner is replaced. There's a variable that should overcome this, CI_RUNNER_ID, which I've used in the asbolute paths. Fingers crossed, let's try it again.
2024-04-28 19:21:39 -04:00
- cd /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/veilid-python && /home/gitlab-runner/.local/bin/poetry build
Added crates.io and Pypi release steps to build_repositories job. Maybe it works, maybe it doesn't. We'll find out at next release.
2024-03-03 16:19:21 -05:00
- vlt run --command="/home/gitlab-runner/.local/bin/poetry publish"
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Update .gitlab-ci.yml file
2023-07-16 15:18:35 -04:00
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
build_repositories:
stage: distribute
Correcting distribute stage order of execution
2024-04-04 20:46:47 -04:00
needs:
- publish_python
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
tags:
- build-orchestration
Migrating CICD scripts to Veilid repo Copied CICD scripts into the repository so that the community can make contributions to the build system. Wrote a brief description of the build and distribute process. Modified the CICD config to use the repo hosted scripts. [ci skip]
2024-04-21 14:37:45 -04:00
variables:
SECURE_FILES_DOWNLOAD_PATH: './'
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
script:
Modified path to CICD scripts in CI config I thought CICD's working directory was in the project root but the release failed to find the scripts. I've changed the script executions to absolute paths. There's a directory is named for the runner's ID, is different on each machine, and changes if the runner is replaced. There's a variable that should overcome this, CI_RUNNER_ID, which I've used in the asbolute paths. Fingers crossed, let's try it again.
2024-04-28 19:21:39 -04:00
- cp /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/build-orchestration/generate-release.sh ~
- /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/build-orchestration/distribute-packages.sh
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Update .gitlab-ci.yml file
2023-07-16 15:18:35 -04:00
deploy_repos:
stage: distribute
needs:
- build_repositories
tags:
- repo-server
script:
Modified path to CICD scripts in CI config I thought CICD's working directory was in the project root but the release failed to find the scripts. I've changed the script executions to absolute paths. There's a directory is named for the runner's ID, is different on each machine, and changes if the runner is replaced. There's a variable that should overcome this, CI_RUNNER_ID, which I've used in the asbolute paths. Fingers crossed, let's try it again.
2024-04-28 19:21:39 -04:00
- /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/repo-server/deploy-repo.sh
Update .gitlab-ci.yml file
2023-08-02 10:14:18 -04:00
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
delete_build_machines:
stage: distribute
needs:
- deploy_repos
tags:
- build-orchestration
script:
Modified path to CICD scripts in CI config I thought CICD's working directory was in the project root but the release failed to find the scripts. I've changed the script executions to absolute paths. There's a directory is named for the runner's ID, is different on each machine, and changes if the runner is replaced. There's a variable that should overcome this, CI_RUNNER_ID, which I've used in the asbolute paths. Fingers crossed, let's try it again.
2024-04-28 19:21:39 -04:00
- /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/build-orchestration/build-machine-ctl.sh delete amd64-deb
- /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/build-orchestration/build-machine-ctl.sh delete arm64-deb
- /home/gitlab-runner/builds/$CI_RUNNER_ID/0/veilid/veilid/scripts/cicd/build-orchestration/build-machine-ctl.sh delete amd64-rpm
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Reference in New Issue Copy Permalink