Git-Mirrors/veilid
1
0
Fork 0
mirror of https://gitlab.com/veilid/veilid.git synced 2024-10-01 01:26:08 -04:00
Code Issues Packages Projects Releases Wiki Activity
627bbf209e
veilid/.gitlab-ci.yml

399 lines
12 KiB
YAML
Raw Normal View History

cleanup and stats accounting organization
2021-11-26 09:54:38 -05:00
variables:
use new runner
2022-11-22 17:16:53 -05:00
NO_DOCKER: 1
FORCE_COLOR: 1
EARTHLY_EXEC_CMD: "/bin/sh"
Add submodule strategyt to build
2022-11-27 00:18:08 -05:00
GIT_SUBMODULE_STRATEGY: normal
cleanup and stats accounting organization
2021-11-26 09:54:38 -05:00
Add merge_requests as build target, split into two architectures
2023-02-20 14:14:05 -05:00
stages:
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
- prepare
Update .gitlab-ci.yml file to the intermediary package distribution pipeline.
2023-06-20 22:46:05 -04:00
- test
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
- build_packages
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 20:59:23 -04:00
- release
Update .gitlab-ci.yml file to the intermediary package distribution pipeline.
2023-06-20 22:46:05 -04:00
- distribute
initial CI attempt
2021-11-24 16:49:13 -05:00
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
# base earthly setup for jobs
.base:
tags: [ saas-linux-medium-amd64 ]
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
image: docker
services:
- docker:dind
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
.earthly: &earthly_setup
- apk update && apk add git
- wget https://github.com/earthly/earthly/releases/download/v0.7.15/earthly-linux-amd64 -O /usr/local/bin/earthly
- chmod +x /usr/local/bin/earthly
- earthly bootstrap
- echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY
Use $CI_REGISTRY_IMAGE for the registry path GitLab will return an error if you have an upper case letter at the start of your username. ``` invalid reference format: repository name must be lowercase ``` The built-in `$CI_REGISTRY_IMAGE` variable does the right thing. Closes https://gitlab.com/veilid/veilid/-/issues/352
2023-12-25 16:14:23 -05:00
- test "$CI_PROJECT_PATH" != "veilid/veilid" && project_args="--CI_REGISTRY_IMAGE=$CI_REGISTRY_IMAGE"
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
# Create the build container if:
# - no container in the registry
# - run as part of a schedule
# - run manually from the pipelines web page
build_cache:
extends: .base
stage: prepare
script:
Rebuild container cache if Earthfile is newer than cache
2024-05-03 05:20:32 -04:00
- apk update && apk add jq && apk add curl
Add support for no container repository (i.e. a new fork)
2024-05-22 17:48:17 -04:00
- if ! docker manifest inspect $CI_REGISTRY_IMAGE/build-cache:latest > /dev/null; then
- CACHE_EPOCH=0
- else
- 'CONT_REPO_ID=$(curl "https://gitlab.com/api/graphql" --header "Content-Type: application/json" --request POST --data "{\"query\": \"query { project(fullPath: \\\"$CI_PROJECT_PATH\\\" ) { containerRepositories( name: \\\"build-cache\\\" ) { nodes { id }}}} \"}" | jq -r ".data.project.containerRepositories.nodes[0].id")'
- echo "CONTAINER REPO ID = $CONT_REPO_ID"
- 'CACHE_TS=$(curl "https://gitlab.com/api/graphql" --header "Content-Type: application/json" --request POST --data "{\"query\": \"query { containerRepository(id: \\\"$CONT_REPO_ID\\\") { tags( first: 1 name: \\\"latest\\\" ) { nodes { location publishedAt }}}} \"}" | jq -r ".data.containerRepository.tags.nodes[0].publishedAt" | cut -d "+" -f1 | sed "s/T/ /g" )'
- CACHE_EPOCH=$(date -d "$CACHE_TS" +%s)
- fi
Rebuild container cache if Earthfile is newer than cache
2024-05-03 05:20:32 -04:00
- EARTHLY_EPOCH=$(git log -1 --format=%ct Earthfile)
- echo "CACHE EPOCH = $CACHE_EPOCH, EARTHLY EPOCH = $EARTHLY_EPOCH"
Add support for no container repository (i.e. a new fork)
2024-05-22 17:48:17 -04:00
- if [[ $EARTHLY_EPOCH -gt $CACHE_EPOCH ]] || [[ "$CI_PIPELINE_SOURCE" == "schedule" ]] || [[ "$CI_PIPELINE_SOURCE" == "web" ]] ; then
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
- *earthly_setup
- earthly --use-inline-cache --save-inline-cache --strict --push -P +build-linux-cache $project_args
- else
- echo "No need to rebuild"
- fi
Fixes scp from build machines to orchestrator Each machine needed a script this specified that machine's arch and OS type. Also added a rule on the cache and test jobs to not run if the dry run trigger is present. [ci dryrun]
2024-04-29 18:07:40 -04:00
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
[ci dryrun]
2024-04-29 18:16:38 -04:00
when: never
Update .gitlab-ci.yml file
2024-07-14 19:19:53 -04:00
- if: $IS_NIGHTLY == "true"
when: never
[ci dryrun]
2024-04-29 20:44:21 -04:00
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_PIPELINE_SOURCE == "push"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
Fixes scp from build machines to orchestrator Each machine needed a script this specified that machine's arch and OS type. Also added a rule on the cache and test jobs to not run if the dry run trigger is present. [ci dryrun]
2024-04-29 18:07:40 -04:00
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
# Runs a basic unit test build, this task will use the `build-cache:latest` as set up in the projects Container Registry
test_build:
extends: .base
stage: test
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
script:
Refactored Earthfile to improve caching Targets have been parallelized so that initial push of the container cache should build the whole build a little quicker, plus the container should now use the cache for more of the build and so speed up normal builds to just the compilation and test of the code that has changed The `build_cache` target now builds a `build-cache:latest` container that is stored in the GitLab project Container Registry, and then used (if it exists) by the `test_build` target. The `build_cache` task runs under 3 conditions, 1. the container does not exist, 2. if scheduled, 3. if run manually from the Pipelines page in the GitLab interface. It is recommended that the build is set up to run on a weekly schedule via the `Pipeline schedules` page in GitLab with the schedule of `0 2 * * 6`.
2023-10-31 07:17:13 -04:00
- *earthly_setup
- earthly --use-inline-cache +unit-tests-linux --BASE=container $project_args
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
resource_group: test
Fixes scp from build machines to orchestrator Each machine needed a script this specified that machine's arch and OS type. Also added a rule on the cache and test jobs to not run if the dry run trigger is present. [ci dryrun]
2024-04-29 18:07:40 -04:00
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
[ci dryrun]
2024-04-29 18:15:19 -04:00
when: never
Update .gitlab-ci.yml file
2024-07-14 19:19:53 -04:00
- if: $IS_NIGHTLY == "true"
when: never
[ci dryrun]
2024-04-29 18:15:19 -04:00
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_PIPELINE_SOURCE == "push"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
Added ephemeral create/delete to test stage.
2023-07-22 13:07:00 -04:00
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
# Actual release -- triggered by pushing a new version tag
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
release_job:
stage: release
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
image: registry.gitlab.com/gitlab-org/release-cli:latest
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
script:
- echo "running release_job"
release: # See https://docs.gitlab.com/ee/ci/yaml/#release for available properties
tag_name: '$CI_COMMIT_TAG'
description: '$CI_COMMIT_TAG'
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Added jobs to create and delete the ephemeral build machines.
2023-07-21 15:30:33 -04:00
create_build_machines:
stage: build_packages
tags:
- build-orchestration
script:
Calling bash explicitly The paths work now, but the scripts are not set with execute permission. I don't know if that permission will survive the transfer from my machine across the git and runner stuff so I added an explicit call for bash instead.
2024-04-28 21:39:18 -04:00
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-rpm
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
package_amd64_deb:
stage: build_packages
Added jobs to create and delete the ephemeral build machines.
2023-07-21 15:30:33 -04:00
needs:
- create_build_machines
Update .gitlab-ci.yml file to the intermediary package distribution pipeline.
2023-06-20 22:46:05 -04:00
tags:
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
- build-amd64-deb
Update .gitlab-ci.yml file to the intermediary package distribution pipeline.
2023-06-20 22:46:05 -04:00
script:
Update .gitlab-ci.yml file
2023-07-04 14:31:02 -04:00
- earthly bootstrap
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
- earthly +package-linux-amd64-deb
Fixes scp from build machines to orchestrator Each machine needed a script this specified that machine's arch and OS type. Also added a rule on the cache and test jobs to not run if the dry run trigger is present. [ci dryrun]
2024-04-29 18:07:40 -04:00
- bash scripts/cicd/build-machine/scp-amd64-debs-to-orchestrator.sh
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
package_arm64_deb:
stage: build_packages
Added jobs to create and delete the ephemeral build machines.
2023-07-21 15:30:33 -04:00
needs:
- create_build_machines
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
tags:
- build-arm64-deb
script:
- earthly bootstrap
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
- earthly +package-linux-arm64-deb
Fixes scp from build machines to orchestrator Each machine needed a script this specified that machine's arch and OS type. Also added a rule on the cache and test jobs to not run if the dry run trigger is present. [ci dryrun]
2024-04-29 18:07:40 -04:00
- bash scripts/cicd/build-machine/scp-arm64-debs-to-orchestrator.sh
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Update .gitlab-ci.yml file
2023-07-04 12:34:48 -04:00
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
package_amd64_rpm:
stage: build_packages
Added jobs to create and delete the ephemeral build machines.
2023-07-21 15:30:33 -04:00
needs:
- create_build_machines
Update .gitlab-ci.yml file
2023-07-16 13:04:23 -04:00
tags:
- build-amd64-rpm
script:
- earthly bootstrap
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
- earthly +package-linux-amd64-rpm
[ci dryrun]
2024-04-29 18:34:52 -04:00
- bash scripts/cicd/build-machine/scp-amd64-rpms-to-orchestrator.sh
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 20:59:23 -04:00
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
publish_crates:
moved crates and pypi publishing to build_packages stage
2024-04-04 20:53:28 -04:00
stage: build_packages
Update .gitlab-ci.yml file
2023-07-16 15:18:35 -04:00
tags:
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
- build-amd64-deb
Update .gitlab-ci.yml file
2023-07-16 15:18:35 -04:00
script:
Added crates.io and Pypi release steps to build_repositories job. Maybe it works, maybe it doesn't. We'll find out at next release.
2024-03-03 16:19:21 -05:00
- vlt login
- vlt run --command="cargo publish -p veilid-tools --dry-run"
- vlt run --command="cargo publish -p veilid-tools"
- vlt run --command="cargo publish -p veilid-core --dry-run"
- vlt run --command="cargo publish -p veilid-core"
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
publish_python:
moved crates and pypi publishing to build_packages stage
2024-04-04 20:53:28 -04:00
stage: build_packages
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
tags:
- build-amd64-deb
script:
- vlt login
It was a typo in the script name Was missing the r in ctrl then spread via copy paste. Fixed the typo and reverted the paths to relative because that should work just fine.
2024-04-28 20:24:45 -04:00
- cd veilid-python && /home/gitlab-runner/.local/bin/poetry build
Added crates.io and Pypi release steps to build_repositories job. Maybe it works, maybe it doesn't. We'll find out at next release.
2024-03-03 16:19:21 -05:00
- vlt run --command="/home/gitlab-runner/.local/bin/poetry publish"
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Update .gitlab-ci.yml file
2023-07-31 21:23:36 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Update .gitlab-ci.yml file
2023-07-16 15:18:35 -04:00
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
build_repositories:
stage: distribute
Correcting distribute stage order of execution
2024-04-04 20:46:47 -04:00
needs:
- publish_python
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
tags:
- build-orchestration
Migrating CICD scripts to Veilid repo Copied CICD scripts into the repository so that the community can make contributions to the build system. Wrote a brief description of the build and distribute process. Modified the CICD config to use the repo hosted scripts. [ci skip]
2024-04-21 14:37:45 -04:00
variables:
Another tweak to GitLab secure storage
2024-07-19 16:49:29 -04:00
SECURE_FILES_DOWNLOAD_PATH: './'
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
script:
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
- curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
- cp scripts/cicd/build-orchestration/rpm-repo-building/Dockerfile ~/rpm-build-container
- cp scripts/cicd/build-orchestration/rpm-repo-building/repobuild.sh ~/rpm-build-container
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
- cp scripts/cicd/build-orchestration/generate-stable-release.sh ~
Fixed CICD missing params
2024-07-14 23:58:26 -04:00
- bash scripts/cicd/build-orchestration/distribute-stable-packages.sh
Moved crate and python publishing to a dev machine. The orchestration machine doesn't have the resources for the cargo build steps.
2024-04-04 19:22:10 -04:00
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Update .gitlab-ci.yml file
2023-07-16 15:18:35 -04:00
deploy_repos:
stage: distribute
needs:
- build_repositories
tags:
- repo-server
script:
Calling bash explicitly The paths work now, but the scripts are not set with execute permission. I don't know if that permission will survive the transfer from my machine across the git and runner stuff so I added an explicit call for bash instead.
2024-04-28 21:39:18 -04:00
- bash scripts/cicd/repo-server/deploy-repo.sh
Update .gitlab-ci.yml file
2023-08-02 10:14:18 -04:00
rules:
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
delete_build_machines:
stage: distribute
needs:
- deploy_repos
tags:
- build-orchestration
script:
Calling bash explicitly The paths work now, but the scripts are not set with execute permission. I don't know if that permission will survive the transfer from my machine across the git and runner stuff so I added an explicit call for bash instead.
2024-04-28 21:39:18 -04:00
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-rpm
Adding release stage triggered by existence of a tag that matches the v+semver format
2023-07-31 21:08:41 -04:00
rules:
Earthly Build Caching in GitLab Container Registry
2023-08-27 13:17:29 -04:00
- if: '$CI_COMMIT_TAG =~ /v\d.+/'
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
Added Earthfile condition and RPM spec fix Dry run will now trigger for Earthfile changes. I also reordered the changelog in veilid-server.spec to descending to correct that error. Commented out the crates.io publishing dry run. This branch does not have access to the protected variables.
2024-04-29 16:24:04 -04:00
# Dryrun release -- triggered by changes in .gitlab-ci.yml, CICD scripts, or Earthfile
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
dryrun_create_build_machines:
stage: build_packages
tags:
- build-orchestration
script:
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-rpm
rules:
Switched to regex trigger The previous trigger condition wasn't liked by Gitlab's yaml parser. I'm switching to trying a commit message based regex trigger. This message does not contain the trigger and so should not fire the dry run.
2024-04-29 17:22:58 -04:00
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
dryrun_package_amd64_deb:
stage: build_packages
needs:
- dryrun_create_build_machines
tags:
- build-amd64-deb
script:
- earthly bootstrap
- earthly +package-linux-amd64-deb
Fixes scp from build machines to orchestrator Each machine needed a script this specified that machine's arch and OS type. Also added a rule on the cache and test jobs to not run if the dry run trigger is present. [ci dryrun]
2024-04-29 18:07:40 -04:00
- bash scripts/cicd/build-machine/scp-amd64-debs-to-orchestrator.sh
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
rules:
Switched to regex trigger The previous trigger condition wasn't liked by Gitlab's yaml parser. I'm switching to trying a commit message based regex trigger. This message does not contain the trigger and so should not fire the dry run.
2024-04-29 17:22:58 -04:00
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
dryrun_package_arm64_deb:
stage: build_packages
needs:
- dryrun_create_build_machines
tags:
- build-arm64-deb
script:
- earthly bootstrap
- earthly +package-linux-arm64-deb
Fixes scp from build machines to orchestrator Each machine needed a script this specified that machine's arch and OS type. Also added a rule on the cache and test jobs to not run if the dry run trigger is present. [ci dryrun]
2024-04-29 18:07:40 -04:00
- bash scripts/cicd/build-machine/scp-arm64-debs-to-orchestrator.sh
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
rules:
Switched to regex trigger The previous trigger condition wasn't liked by Gitlab's yaml parser. I'm switching to trying a commit message based regex trigger. This message does not contain the trigger and so should not fire the dry run.
2024-04-29 17:22:58 -04:00
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
Further dry run trigger refinements The dry run should now be configured if changes to .gitlab-ci.yml, Earthfile, contents of scripts/cicd, or contents of package arrive as the result of a push OR merge but NOT when a tag is present. I think that will prevent the dry run executing alongside the actual release process when a new release tag is pushed but that also contains changes to the above mentioned files. <fingers crossed emoji>
2024-04-29 17:03:38 -04:00
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
dryrun_package_amd64_rpm:
stage: build_packages
needs:
- dryrun_create_build_machines
tags:
- build-amd64-rpm
script:
- earthly bootstrap
- earthly +package-linux-amd64-rpm
[ci dryrun]
2024-04-29 18:34:52 -04:00
- bash scripts/cicd/build-machine/scp-amd64-rpms-to-orchestrator.sh
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
rules:
Switched to regex trigger The previous trigger condition wasn't liked by Gitlab's yaml parser. I'm switching to trying a commit message based regex trigger. This message does not contain the trigger and so should not fire the dry run.
2024-04-29 17:22:58 -04:00
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
Uncommented crates.io section.
2024-04-29 20:04:30 -04:00
dryrun_publish_crates:
stage: build_packages
needs:
- dryrun_create_build_machines
tags:
- build-amd64-deb
script:
- vlt login
- vlt run --command="cargo publish -p veilid-tools --dry-run"
- vlt run --command="cargo publish -p veilid-core --dry-run"
rules:
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
dryrun_publish_python:
stage: build_packages
needs:
- dryrun_create_build_machines
tags:
- build-amd64-deb
script:
- cd veilid-python && /home/gitlab-runner/.local/bin/poetry build
rules:
Switched to regex trigger The previous trigger condition wasn't liked by Gitlab's yaml parser. I'm switching to trying a commit message based regex trigger. This message does not contain the trigger and so should not fire the dry run.
2024-04-29 17:22:58 -04:00
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
dryrun_build_repositories:
stage: distribute
tags:
- build-orchestration
variables:
Another tweak to GitLab secure storage
2024-07-19 16:49:29 -04:00
SECURE_FILES_DOWNLOAD_PATH: './'
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
script:
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
- curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
- cp scripts/cicd/build-orchestration/generate-release.sh ~
- bash scripts/cicd/build-orchestration/distribute-packages.sh
rules:
Switched to regex trigger The previous trigger condition wasn't liked by Gitlab's yaml parser. I'm switching to trying a commit message based regex trigger. This message does not contain the trigger and so should not fire the dry run.
2024-04-29 17:22:58 -04:00
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
dryrun_deploy_repos:
stage: distribute
needs:
- dryrun_build_repositories
tags:
- repo-server
script:
[ci dryrun]
2024-04-29 20:44:21 -04:00
- ls -al $HOME/repo.tar
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
rules:
Switched to regex trigger The previous trigger condition wasn't liked by Gitlab's yaml parser. I'm switching to trying a commit message based regex trigger. This message does not contain the trigger and so should not fire the dry run.
2024-04-29 17:22:58 -04:00
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
Added CICD dry run This is the first attempt at setting up a dry run pipeline to test changes to the CICD config without actually publishing the compiled binaries and packages built by the process. The dry run should be triggered by any changes to .gitlab-ci.yml or changes to any of the scripts under scripts/cicd/.
2024-04-29 15:40:07 -04:00
dryrun_delete_build_machines:
stage: distribute
needs:
- dryrun_deploy_repos
tags:
- build-orchestration
script:
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-rpm
rules:
Testing dry run trigger The trigger is [ci dryrun]
2024-04-29 17:26:13 -04:00
- if: $CI_COMMIT_MESSAGE =~ /\[ci dryrun]/
[ci dryrun]
2024-04-29 20:22:44 -04:00
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
# Nightly build pipeline
nightly_create_build_machines:
stage: build_packages
tags:
- build-orchestration
script:
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh create amd64-rpm
rules:
The CI yaml doesn't accept a bool as a custom variable. I've switched it to a string to pass linting but I think this is going to break stuff.
2024-07-14 18:43:24 -04:00
- if: $IS_NIGHTLY == "true"
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
nightly_package_amd64_deb:
stage: build_packages
needs:
- nightly_create_build_machines
tags:
- build-amd64-deb
script:
- earthly bootstrap
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
- earthly +package-linux-amd64-deb --IS_NIGHTLY="$IS_NIGHTLY"
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
- bash scripts/cicd/build-machine/scp-amd64-debs-to-orchestrator.sh
rules:
The CI yaml doesn't accept a bool as a custom variable. I've switched it to a string to pass linting but I think this is going to break stuff.
2024-07-14 18:43:24 -04:00
- if: $IS_NIGHTLY == "true"
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
nightly_package_arm64_deb:
stage: build_packages
needs:
- nightly_create_build_machines
tags:
- build-arm64-deb
script:
- earthly bootstrap
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
- earthly +package-linux-arm64-deb --IS_NIGHTLY="$IS_NIGHTLY"
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
- bash scripts/cicd/build-machine/scp-arm64-debs-to-orchestrator.sh
rules:
The CI yaml doesn't accept a bool as a custom variable. I've switched it to a string to pass linting but I think this is going to break stuff.
2024-07-14 18:43:24 -04:00
- if: $IS_NIGHTLY == "true"
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
nightly_package_amd64_rpm:
stage: build_packages
needs:
- nightly_create_build_machines
tags:
- build-amd64-rpm
script:
- earthly bootstrap
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
- earthly +package-linux-amd64-rpm --IS_NIGHTLY="$IS_NIGHTLY"
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
- bash scripts/cicd/build-machine/scp-amd64-rpms-to-orchestrator.sh
rules:
The CI yaml doesn't accept a bool as a custom variable. I've switched it to a string to pass linting but I think this is going to break stuff.
2024-07-14 18:43:24 -04:00
- if: $IS_NIGHTLY == "true"
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
nightly_build_repositories:
stage: distribute
tags:
- build-orchestration
variables:
Another tweak to GitLab secure storage
2024-07-19 16:49:29 -04:00
SECURE_FILES_DOWNLOAD_PATH: './'
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
script:
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
- curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
- cp scripts/cicd/build-orchestration/generate-nightly-release.sh ~
Fixed CICD missing params
2024-07-14 23:58:26 -04:00
- bash scripts/cicd/build-orchestration/distribute-nightly-packages.sh
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
rules:
The CI yaml doesn't accept a bool as a custom variable. I've switched it to a string to pass linting but I think this is going to break stuff.
2024-07-14 18:43:24 -04:00
- if: $IS_NIGHTLY == "true"
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
nightly_delete_build_machines:
stage: distribute
needs:
Update .gitlab-ci.yml file
2024-07-14 19:17:29 -04:00
- nightly_build_repositories
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
tags:
- build-orchestration
script:
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete arm64-deb
- bash scripts/cicd/build-orchestration/build-machine-ctrl.sh delete amd64-rpm
rules:
The CI yaml doesn't accept a bool as a custom variable. I've switched it to a string to pass linting but I think this is going to break stuff.
2024-07-14 18:43:24 -04:00
- if: $IS_NIGHTLY == "true"
Reference in New Issue Copy Permalink