Commit Graph

56 Commits

Author SHA1 Message Date
David Robertson
1fbe0316a9
Add suffices to scripts in scripts-dev (#12137)
* Rename scripts-dev to have suffices

* Update references to `scripts-dev`

* Changelog

* These scripts don't pass mypy
2022-03-02 18:00:26 +00:00
David Robertson
5f62a094de
Detox, part 1 of N (#12119)
* Don't use `tox` for `check-sampleconfig`
* Don't use `tox` for check-newsfragment
2022-03-01 19:47:02 +00:00
David Robertson
af2c1e3d2a
Tidy the building of sdists and wheels (#12051)
* Don't build distribution pkgs in tests.yml
* Run `release-artifacts` on release branches
* Use backend-meta workflow for packaging
2022-02-22 11:33:37 +00:00
Richard van der Hoff
bb98c593a5
Prepare for rename of default complement branch (#11971)
use `HEAD` rather than hardcoding `master`
2022-02-11 14:43:34 +00:00
kegsay
a3865ed525
Run Complement tests sequentially (#11910)
Since #11811 there has been general Complement flakiness around networking.
It seems like tests are hitting the wrong containers. In an effort to diagnose
the cause of this, as well as reduce its impact on this project, set the
parallelsim to 1 (no parallelism) when running tests.

If this fixes the flakiness then this indicates the cause and I can diagnose
this further. If this doesn't fix the flakiness then that implies some kind
of test pollution which also helps to diagnose this further.
2022-02-04 13:04:57 +00:00
kegsay
0938f32e93
CI: run Complement on the VM, not inside Docker (#11811)
* CI: run Complement on the VM, not inside Docker

This requires https://github.com/matrix-org/complement/pull/289

We now run Complement on the VM instead of inside a Docker container.
This is to allow Complement to bind to any high-numbered port when it
starts up its own federation servers. We want to do this to allow for
more concurrency when running complement tests. Previously, Complement
only ever bound to `:8448` when running its own federation server. This
prevented multiple federation tests running at the same time as they would
fight each other on the port. This did however allow Complement to run
in Docker, as the host could just port forward `:8448` to allow homeserver
containers to communicate to Complement. Now that we are using random
ports however, we cannot use Docker to run Complement. This ends up
being a good thing because:
 - Running Complement tests locally is closer to how they run in CI.
 - Allows the `CI` env var to be removed in Complement.
 - Slightly speeds up runs as we don't need to pull down the Complement
   image prior to running tests. This assumes GHA caches actions sensibly.

* Changelog

* Full stop

* Update .github/workflows/tests.yml

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

* Review comments

* Update .github/workflows/tests.yml

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-01-25 15:05:22 +00:00
Richard van der Hoff
2d327d25bf
Skip the initial amd64-only Docker build (#11810)
PyNaCl's recent 1.5.0 release on PyPi includes arm64 wheels, which means our
arm64 docker images now build in a sensible amount of time, so we can skip the
amd64-only build.
2022-01-24 18:31:23 +00:00
Shay
9006ee36d1
Drop support for and remove references to EOL Python 3.6 (#11683)
* remove reference in comments to python3.6

* upgrade tox python env in script

* bump python version in example for completeness

* upgrade python version requirement in setup doc

* upgrade necessary python version in __init__.py

* upgrade python version in setup.py

* newsfragment

* drops refs to bionic and replace with focal

* bump refs to postgres 9.6 to 10

* fix hanging ci

* try installing tzdata first

* revert change made in b979f336

* ignore new random mypy error while debugging other error

* fix lint error for temporary workaround

* revert change to install list

* try passing env var

* export debian frontend var?

* move line and add comment

* bump pillow dependency

* bump lxml depenency

* install libjpeg-dev for pillow

* bump automat version to one compatible with py3.8

* add libwebp for pillow

* bump twisted trunk python version

* change suffix of newsfragment

* remove redundant python 3.7 checks

* lint
2022-01-21 14:23:26 -08:00
Richard van der Hoff
d41c4654db
Use buildkit's cache feature to speed up docker builds (#11691)
Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
2022-01-12 10:37:57 +00:00
kegsay
3be63654e4
Prettier complement logs (#11707)
* Prettier complement logs

* Changelog
2022-01-10 11:46:40 +00:00
Shay
87da37374a
Drop EOL python 3.6 from CI (#11595)
* remove python 3.6 and postgres 9.6 from github workflow

* remove python 3.6 env from tox

* newsfragment

* correct postgres version

* add py310 to tox env list
2021-12-21 11:37:04 -08:00
Patrick Cloke
a4521ce0a8
Support the stable /hierarchy endpoint from MSC2946 (#11329)
This also makes additional updates where the implementation
had drifted from the approved MSC.

Unstable endpoints will be removed at a later data.
2021-11-29 14:32:20 -05:00
Patrick Cloke
9d1971a5c4
Return the stable event field from /send_join per MSC3083. (#11413)
This does not remove the unstable field and still parses both.
Handling of the unstable field will need to be removed in the
future.
2021-11-29 15:43:20 +00:00
Richard van der Hoff
e2e9bea1ce
Publish a develop docker image (#11380)
I'd find it helpful to have a docker image corresponding to current develop,
without having to build my own.
2021-11-19 10:56:59 +00:00
Hillery Shay
7d70582eb0
Fix broken export-data admin command and add a test for it to CI (#11078)
Fix broken export-data admin command and add a test for it to CI
2021-10-18 08:14:12 -07:00
Patrick Cloke
8711e15734
Always dump logs from trial during CI. (#11068)
Instead of only dumping them if trial passes.
2021-10-12 20:09:49 +00:00
Erik Johnston
0b4d5ce5e3
Fix CI to run the unit tests without optional deps (#11017)
This also turns off calculating code coverage, as we didn't use it and it was a lot of noise
2021-10-08 10:05:48 +01:00
David Robertson
6f6e956338
Run CI with Python 3.10 and Postgres 14 (#10992) 2021-10-05 12:43:04 +01:00
Patrick Cloke
ebd8baf61f
Clear our destination directories before copying files to GitHub pages. (#10869)
This should fix stale deleted files being still accessible.
2021-09-21 16:32:46 +00:00
David Robertson
f455b0e420
GHA: reintroduce an env var for $GITHUB_HEAD_REF (#10659)
This should ensure GHA runs synapse against the same-named sytest branch
2021-09-20 17:35:16 +01:00
Dan Callahan
2efc838f05
Avoid duplicate issues from Twisted trunk failures (#10672)
Setting `update_existing: true` in the `create-an-issue` GitHub Action
will avoid opening duplicate issues if an open issue already exists with
an identical title.

If no open issues match the title, then a new issue will be created.

This helps avoid spamming our issue tracker should there be a failure
when testing against Twisted's trunk.

This PR also pins the SHA of the `create-an-issue` action to mitigate
the risk of a malicious actor gaining access to JasonEtco's account.

See GitHub's page on security hardening third party actions for more:
https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions

Signed-off-by: Dan Callahan <danc@element.io>
2021-08-23 13:06:49 +00:00
Richard van der Hoff
4db65f911a
Run a nightly CI build against Twisted trunk. (#10651)
This creates a GHA workflow which runs at 8am every day, and runs mypy, trial and sytest against Twisted's current trunk. If any of the jobs fail, it opens an issue.
2021-08-23 11:12:45 +01:00
David Robertson
878528913d Remove buildkite-era comment 2021-08-12 11:48:36 +01:00
David Robertson
74fcd5aab9 portdb also uses coverage, so provide $TOP there 2021-08-12 10:41:01 +01:00
David Robertson
cb5976ebd7 set TOP in sytest containers 2021-08-11 20:08:48 +01:00
David Robertson
33ef86aa25 Rename ci to .ci 2021-08-11 19:59:57 +01:00
David Robertson
58e5da5aa0 Remove buildkite from portdb CI tests 2021-08-10 16:38:13 +01:00
David Robertson
c5988a8eb7 Remove unused BUILDKITE_BRANCH env var 2021-08-10 16:38:13 +01:00
David Robertson
3d67b8c82b Move sytest worker-blacklist to ci directory 2021-08-10 16:38:13 +01:00
David Robertson
03fb99a5c8 check-newsfragment: pass pr number explicitly
use PULL_REQUEST_NUMBER instead of BUILDKITE_PULL_REQUEST
remove the other user of BUILDKITE_PULL_REQUEST, namely merge_base_branch.sh
2021-08-10 16:38:13 +01:00
David Robertson
8da9e3cb69 Move test_old_deps.sh to new ci dir 2021-08-10 16:38:13 +01:00
Erik Johnston
e33f14e8d5
Don't fail CI when lint-newfile job was skipped (#10529) 2021-08-05 11:22:27 +01:00
Richard van der Hoff
a6ea32a798
Fix the tests-done github actions step, again (#10512) 2021-08-02 21:06:34 +01:00
Patrick Cloke
13944678c3
Use new go test running syntax for complement. (#10488)
Updates CI and the helper script t ensures all tests are run (in parallel).
2021-07-27 12:08:51 +00:00
Richard van der Hoff
f22252d4f9
Enable docker image caching for the deb build (#10431) 2021-07-26 11:36:01 +01:00
Erik Johnston
ab82fd6ed1 Merge branch 'release-v1.39' into develop 2021-07-23 09:19:24 +01:00
Richard van der Hoff
4c3fdfc808
Fix an error in the docker workflow (#10461) 2021-07-22 21:50:30 +01:00
Richard van der Hoff
5e2df47f72
Cancel redundant GHA workflows (#10451) 2021-07-22 11:35:06 +01:00
Richard van der Hoff
f1347bcfdc
Fix the tests-done Github Actions job (#10444) 2021-07-22 11:10:30 +01:00
Richard van der Hoff
c6509991f3
Move the docker image build to Github Actions (#10416)
it's flaky on circleCI, and having to manage multiple CI providers is painful.
2021-07-21 12:33:35 +01:00
Richard van der Hoff
97c8ae90f7
Add a github actions job recording success of other jobs. (#10430) 2021-07-20 11:41:19 +01:00
Richard van der Hoff
07e0992a76
Make GHA config more efficient (#10383)
A few things here:

* Build the debs for single distro for each PR, so that we can see if it breaks. Do the same for develop. Building all the debs ties up the GHA workers for ages.
* Stop building the debs for release branches. Again, it takes ages, and I don't think anyone is actually going to stop and look at them. We'll know they are working when we make an RC.
* Change the configs so that if we manually cancel a workflow, it actually does something.
2021-07-14 14:41:23 +01:00
Richard van der Hoff
2d8b60e0f2
Github Actions workflow to attach release artifacts to release (#10379) 2021-07-13 11:50:14 +01:00
Richard van der Hoff
ae81ec428d Build the python release artifacts in GHA too 2021-07-13 00:20:11 +01:00
Richard van der Hoff
5f2848f379
build debs in GHA (#10247)
GHA workflow to build the debs
2021-07-12 19:03:14 +01:00
Richard van der Hoff
26196df575 Merge branch 'master' into release-v1.38 2021-07-07 11:08:08 +01:00
Richard van der Hoff
7c82378992
build the docs for master (#10323) 2021-07-07 10:43:54 +01:00
Andrew Morgan
f0e02f5df2
Create an index.html file when generating a docs build (#10242)
Currently when a new build of the docs is created, an `index.html` file does not exist. Typically this would be generated from a`docs/README.md` file - which we have - however we're currently using [docs/README.md](394673055d/docs/README.md) to explain the docs and point to the website. It is not part of the content of the website. So we end up not having an `index.html` file, which will result in a 404 page if one tries to navigate to `https://matrix-org.github.io/synapse/<docs_version>/index.html`.

This isn't a really problem for the default version of the documentation (currently `develop`), as [navigating to the top-level root](https://matrix-org.github.io/synapse/) of the website (without specifying a version) will [redirect](a77e6925f2/index.html (L2)) you to the Welcome and Overview page of the `develop` docs version.

However, ideally once we add a GUI for switching between versions, we'll want to send the user to `matrix-org.github.io/synapse/<version>/index.html`, which currently isn't generated.

This PR modifies the CI that builds the docs to simply copy the rendered [Welcome & Overview page](https://matrix-org.github.io/synapse/develop/welcome_and_overview.html) to `index.html`.
2021-06-24 18:00:56 +01:00
Andrew Morgan
7c536d0fef
Deploy a documentation version for each new Synapse release (#10198)
This PR will run a new "Deploy release-specific documentation" job whenever a push to a branch name matching `release-v*` occurs. Doing so will create/add to a folder named `vX.Y` on the `gh-pages` branch. Doing so will allow us to build up `major.minor` releases of the docs as we release Synapse.

This is especially useful for having a mechanism for keeping around documentation of old/removed features (for those running older versions of Synapse), without needing to clutter the latest copy of the docs.

After a [discussion](https://matrix.to/#/!XaqDhxuTIlvldquJaV:matrix.org/$rKmkBmQle8OwTlGcoyu0BkcWXdnHW3_oap8BMgclwIY?via=matrix.org&via=vector.modular.im&via=envs.net) in #synapse-dev, we wanted to use tags to trigger the documentation deployments, which I agreed with. However, I soon realised that the bash-foo required to turn a tag of `v1.2.3rc1` into `1.2` was a lot more complex than the branch's `release-v1.2`. So, I've gone with the latter for simplicity.

In the future we'll have some UI on the website to switch between versions, but for now you can simply just change 'develop' to 'v1.2' in the URL.
2021-06-18 19:26:25 +01:00
Patrick Cloke
a14884fbb0
Use the matching complement branch when running tests in CI. (#10160)
This implements similar behavior to sytest where a matching branch is used,
if one exists. This is useful when needing to modify both application code
and tests at the same time. The following rules are used to find a matching
complement branch:

1. Search for the branch name of the pull request. (E.g. feature/foo.)
2. Search for the base branch of the pull request. (E.g. develop or release-vX.Y.)
3. Search for the reference branch of the commit. (E.g. master or release-vX.Y.)
4. Fallback to 'master', the default complement branch name.
2021-06-11 08:17:17 -04:00