Git-Mirrors/veilid
1
0
Fork 0
mirror of https://gitlab.com/veilid/veilid.git synced 2024-10-01 01:26:08 -04:00
Code Issues Packages Projects Releases Wiki Activity
859d2214f7
veilid/scripts/cicd/build-orchestration/distribute-nightly-packages.sh

71 lines
3.1 KiB
Bash
Raw Normal View History

Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
#!/bin/bash
# Clean and reset the workspaces
echo "Setting up the workspace"
# Rsync active repo to local workspace
Saving my work so I can switch back to main for a few minutes of troubleshooting
2024-07-17 12:17:54 -04:00
rsync --archive gitlab-runner@10.116.0.3:/srv/ $HOME/srv/
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
# Delete previous versions of packages
rm -rf $HOME/srv/apt/pool/nightly/main/*.deb
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
rm -rf $HOME/srv/rpm/nightly/x86_64/*
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
# Setup crypto
export GNUPGHOME="$(mktemp -d ~/pgpkeys-XXXXXX)"
Another tweak to GitLab secure storage
2024-07-19 16:49:29 -04:00
cat veilid-packages-key.private | gpg --import
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
gpg --armor --export admin@veilid.org > $HOME/srv/gpg/veilid-packages-key.public
# Copy .deb files into the workspace and generate repo files
echo "Starting deb process"
Saving my work so I can switch back to main for a few minutes of troubleshooting
2024-07-17 12:17:54 -04:00
cd $HOME
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
tar -xf amd64-debs.tar
tar -xf arm64-debs.tar
cp *.deb /home/gitlab-runner/srv/apt/pool/nightly/main
cd /home/gitlab-runner/srv/apt
echo "Creating Packages file"
dpkg-scanpackages --arch amd64 pool/nightly > dists/nightly/main/binary-amd64/Packages
dpkg-scanpackages --arch arm64 pool/nightly > dists/nightly/main/binary-arm64/Packages
cat dists/nightly/main/binary-amd64/Packages | gzip -9 > dists/nightly/main/binary-amd64/Packages.gz
cat dists/nightly/main/binary-arm64/Packages | gzip -9 > dists/nightly/main/binary-arm64/Packages.gz
echo "Creating Release file"
Saving my work so I can switch back to main for a few minutes of troubleshooting
2024-07-17 12:17:54 -04:00
cd $HOME/srv/apt/dists/nightly
bash $HOME/generate-nightly-release.sh > Release
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
echo "Signing Release file and creating InRelease"
Saving my work so I can switch back to main for a few minutes of troubleshooting
2024-07-17 12:17:54 -04:00
cat $HOME/srv/apt/dists/nightly/Release | gpg --default-key admin@veilid.org -abs > /home/gitlab-runner/srv/apt/dists/nightly/Release.gpg
cat $HOME/srv/apt/dists/nightly/Release | gpg --default-key admin@veilid.org -abs --clearsign > /home/gitlab-runner/srv/apt/dists/nightly/InRelease
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
# Copy .rpm files into the workspace and generate repo files
echo "Starting rpm process"
cd $HOME
tar -xf amd64-rpms.tar
echo "Copying rpms to container workspace"
cp *x86_64.rpm $HOME/rpm-build-container/mount/repo/nightly/x86_64
echo "Copying signing material to container workspace"
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
cp -R $GNUPGHOME/* $HOME/rpm-build-container/mount/keystore
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
echo "Executing container actions"
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
docker run --rm -d -it -e IS_NIGHTLY=$IS_NIGHTLY --name rpm-repo-builder --mount type=bind,source=$HOME/rpm-build-container/mount,target=/mount rpm-repo-builder-img:v12
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
sleep 2
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
cp -R $HOME/rpm-build-container/mount/repo/nightly/x86_64/* $HOME/srv/rpm/nightly/x86_64
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
cd $HOME/srv/rpm/nightly/x86_64
echo "Signing the rpm repository"
gpg --default-key admin@veilid.org --detach-sign --armor $HOME/srv/rpm/nightly/x86_64/repodata/repomd.xml
echo "[veilid-nightly-x86_64-rpm-repo]
name=Veilid Nightly x86_64 RPM Repo
baseurl=https://packages.veilid.net/rpm/nightly/x86_64
enabled=1
gpgcheck=1
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
gpgkey=https://packages.veilid.net/gpg/veilid-packages-key.public" > $HOME/srv/rpm/nightly/x86_64/veilid-nightly-x86_64-rpm.repo
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
# Tar the repo data and transfer to the repo server
echo "Moving the repo scaffold to the repo server"
cd $HOME
Added --delete option to rsync push to repo server The rsync push successfully put the new repo files on the repo server, but also left the original files in place which caused old packages and repo errata to build up over time. Adding the --delete switch results removal of files on the repo server that were deleted as part of the the distribution scripts "Clean and reset the workspaces" section.
2024-07-19 15:40:31 -04:00
rsync --archive --delete $HOME/srv/* gitlab-runner@10.116.0.3:/srv
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
# Cleanup
echo "Cleaning up the workspace"
rm -rf $GNUPGHOME
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
rm $HOME/*.tar
Adding automatic nightly release builds - Restructured linux repo directory structure to include Stable and Nightly releases - Reworked Earthfile to pass IS_NIGHTLY boolean to package build scripts for use in file names - Added nightly pipeline to CICD config yaml. Pipeline is triggered by IS_NIGHTLY variable - Two distribution scripts now exist, one for Stable and one for Nightly - Reworked RPM build spec files to create appropriate file names - Reworked debian release generation scripts - RPM repo directory structure is prepped for arm64 builds - Switched from scp to rsync for moving the completed repo from ochestration to repo server - Created schedule in Gitlab UI to trigger nightly release at 2300 Central Time
2024-07-14 18:22:50 -04:00
rm $HOME/*.deb
rm $HOME/*.rpm
Further adventures in nightly release automation Pushing this up in order to end-to-end test automated nightly releases. Please enter the commit message for your changes. Lines starting
2024-07-19 15:08:25 -04:00
rm -rf $HOME/rpm-build-container/mount/keystore/*
rm -rf $HOME/rpm-build-container/mount/repo/nightly/x86_64/*
echo "Nightly packages distribution process complete"
Reference in New Issue Copy Permalink