Commit Graph

186 Commits

Author SHA1 Message Date
David Robertson
5e84461653
Minor speedups to CI linting (#13827) 2022-09-16 16:18:32 +01:00
David Robertson
540afb0bfc
Simplify CI tests DAG (#13784)
* Simplify CI tests DAG

* Changelog
2022-09-13 10:17:23 +01:00
Mathieu Velten
fa2f3d8d0c
Fix GHA skippable syntax (#13778)
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
2022-09-12 17:31:23 +00:00
Erik Johnston
dd7484b562
Fix CI on non-PR builds (#13769)
Mark cargo-test as skippable since it only runs on Rust code change.
2022-09-12 13:26:33 +01:00
Erik Johnston
c85c5ace52
Add rust to CI (#13763) 2022-09-09 11:29:04 +01:00
Erik Johnston
1cc729c177
Fix latest deps (#13743) 2022-09-08 13:58:31 +01:00
Erik Johnston
8d7fcf9b76
Fix latest deps CI (#13734) 2022-09-07 14:07:06 +00:00
Erik Johnston
dc0e896b68
Add some rust caching to CI (#13735) 2022-09-07 13:56:59 +00:00
Erik Johnston
c9b7e97355
Add a stub Rust crate (#12595) 2022-09-06 19:01:37 +01:00
Erik Johnston
b5effc7201
Update trial old deps CI to use poetry 1.2.0 (#13707) 2022-09-06 11:43:04 +00:00
Patrick Cloke
32fc3b7ba4
Remove configuration options for direct TCP replication. (#13647)
Removes the ability to configure legacy direct TCP replication. Workers now require Redis to run.
2022-09-06 07:50:02 +00:00
Erik Johnston
8edf3f66d5
Reduce number of CI jobs run on PRs (#13713)
* Reduce number of CI jobs run on PRs

* Newsfile

* Also limit sytest jobs

* Fix typo

* Fix up

* Fixup
2022-09-05 13:31:42 +01:00
Erik Johnston
0fdb685c2b
Only run trial CI on all python versions on non-PRs (#13698) 2022-09-02 13:12:47 +01:00
Erik Johnston
18e4092801
Bump docker image to use stable poetry version (#13688) 2022-09-01 13:46:47 +01:00
Richard van der Hoff
4f6de33f41
Print complement failure results last (#13639)
Since github always scrolls to the bottom of any test output, let's put the
failed tests last and hide any successful packages.
2022-08-28 20:05:30 +00:00
Kat Gerasimova
a282446502
Update automation for incoming issues (#13629)
GitHub appears to be deprecating addProjectNextItem by not allowing it to be used alongside projectV2 to get the project ID, so switching to using addProjectV2ItemById instead.
2022-08-25 12:09:23 +01:00
Andy Balaam
371db86a86
First draft of triage_labelled action (#13612) 2022-08-24 13:59:33 +01:00
Kat Gerasimova
2e2040c93e
Add GitHub automation for new issues (#13610)
Set up automation to move newly opened issues in GitHub to the issue triage board.
2022-08-24 12:10:32 +01:00
David Robertson
ba8938b090
Reject non-strict types in Pydantic models (#13502) 2022-08-17 11:17:04 +01:00
reivilibre
05aeeb3a80
Enable Complement CI tests in the 'latest deps' test run. (#13213)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2022-08-01 10:55:31 +00:00
reivilibre
b817574be7
Re-enable running Complement tests against Synapse with workers. (#13420) 2022-08-01 11:51:44 +01:00
David Robertson
e9ce4d089b
Use and recommend poetry 1.1.14, up from 1.1.12 (#13285) 2022-07-15 16:18:47 +01:00
Patrick Cloke
90e9b4fa1e
Do not fail build if complement with workers fails. (#13266) 2022-07-13 08:30:42 -04:00
reivilibre
68695d8007
Factor out some common Complement CI setup commands to a script. (#13157) 2022-07-05 14:24:42 +01:00
reivilibre
c04e25789e
Enable Complement testing in the 'Twisted Trunk' CI runs. (#13079) 2022-07-01 15:42:49 +00:00
reivilibre
9667bad55d
Improve startup times in Complement test runs against workers, particularly in CPU-constrained environments. (#13127)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-30 11:58:12 +00:00
reivilibre
e12ff697a4
Sort failing jobs in Complement CI to the top of the logs to make them easier to read. (#13057) 2022-06-15 16:13:36 +00:00
Erik Johnston
de334ac183
Add a CI job to check that schema deltas are in the correct folder. (#13063) 2022-06-15 16:27:18 +01:00
Erik Johnston
9ad2197fa7
Rename complement-developonly (#13046) 2022-06-15 15:11:42 +00:00
reivilibre
538044ac01
Collapse Docker build commands in Complement CI runs to make the logs easier to read. (#13058) 2022-06-15 14:42:27 +00:00
reivilibre
09a3c5ce0b
Fix Complement runs always being Postgres (#13034)
* Fix Complement runs always being Postgres

* Newsfile

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2022-06-14 13:13:35 +01:00
reivilibre
3d1d510fa9
Enable testing against PostgreSQL databases in Complement CI. (#12965) 2022-06-09 14:16:34 +01:00
reivilibre
3c8f1290b8
Use a GitHub form for issues rather than a hard-to-read, easy-to-ignore template. (#12982)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-06-08 11:11:02 +00:00
Michael Telatynski
782cb7420a
Fix complement tests using the wrong path (#12933) 2022-06-01 15:57:09 +01:00
reivilibre
bf01e51554
Test Synapse against Complement with workers. (#12810)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-05-31 13:02:00 +00:00
Henry
b8fa24b022
Use docker/metadata-action to generate docker image tags (#12573)
Update the "Build docker images" GitHub Actions workflow to use
`docker/metadata-action` to generate docker image tags, instead of a
custom shell script.

Signed-off-by: Henry <97804910+henryclw@users.noreply.github.com>
2022-05-05 12:36:42 +00:00
David Robertson
57fac2a234
Allow unused ignores in "bleeding edge" CI (#12576)
* Allow unused ignores in "bleeding edge" CI

Where "bleeding edge" means the Twisted Trunk and Latest Deps jobs.

Follow up from #12531.
Resolves #12574.

* Use `--extras all` in latest deps mypy CI

Twisted trunk job already does this.

Missed in #12531.

* changelog
2022-04-29 17:57:23 +01:00
David Robertson
5d3509dfda
Revert accidental direct-to-develop commits.
This reverts commit 5a320baa45.
This reverts commit f282d5fc11.
This reverts commit ce6ecdd4b4.
2022-04-28 11:33:05 +01:00
David Robertson
f282d5fc11
Use --extras all in latest deps mypy CI
Twisted trunk job already does this.

Missed in #12531.
2022-04-28 11:29:13 +01:00
David Robertson
ce6ecdd4b4
Allow unused ignores in "bleeding edge" CI
Where "bleeding edge" means the Twisted Trunk and Latest Deps jobs.

Follow up from #12531.
Resolves #12574.
2022-04-28 11:28:22 +01:00
David Robertson
6463244375
Remove unused # type: ignores (#12531)
Over time we've begun to use newer versions of mypy, typeshed, stub
packages---and of course we've improved our own annotations. This makes
some type ignore comments no longer necessary. I have removed them.

There was one exception: a module that imports `select.epoll`. The
ignore is redundant on Linux, but I've kept it ignored for those of us
who work on the source tree using not-Linux. (#11771)

I'm more interested in the config line which enforces this. I want
unused ignores to be reported, because I think it's useful feedback when
annotating to know when you've fixed a problem you had to previously
ignore.

* Installing extras before typechecking

Lacking an easy way to install all extras generically, let's bite the bullet and
make install the hand-maintained `all` extra before typechecking.

Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to
the release/v1 branch.
2022-04-27 14:03:44 +01:00
David Robertson
e5a76ec00b
Dump setuptools; correct pyproject version number (#12478) 2022-04-20 17:33:20 +01:00
David Robertson
eed38c5027
Add CI job to act as a canary for testing against latest dependencies (#12472)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-04-20 12:48:44 +01:00
David Robertson
d24cd17820
Use poetry lockfile in twisted trunk CI job (#12425)
Fixes #12458

Co-authored-by: Sean Quah <seanq@element.io>
Co-authored-by: Dan Callahan <danc@element.io>
2022-04-13 11:26:53 +01:00
David Robertson
5a275a2377
Run "main" trial tests under poetry (#12438)
* Run "main" trial tests under poetry

Olddeps and twisted trunk tests are handled in separate PRs.

The PyPy config is a best-effort only; it's completely untested.

Pulled out from #12337.

* Changelog
2022-04-12 17:41:21 +01:00
David Robertson
58c657322a
Run lints under poetry in CI; remove lint tox jobs (#12434)
Co-authored-by: Dan Callahan <danc@element.io>
2022-04-12 17:35:48 +01:00
David Robertson
4e13743738
Poetry: select olddeps using poetry (#12407)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-04-12 10:50:11 +00:00
Richard van der Hoff
5f72ea1bde
Move complement setup stuff into the Synapse repo (#12404)
Fixes matrix-org/complement#330 (or it will, once we remove the old files).

It's not quite a lift-and-shift: I've also taken the opportunity to get rid of the custom CA that we used to use to sign the TLS certs, which has been superceded by the CA exposed by Complement.
2022-04-11 11:39:28 +01:00
David Robertson
98ec375b26
CI: Fix the export-data job to run under poetry (#12418)
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-04-08 19:18:54 +00:00
David Robertson
7851a2c62f
CI: Get the portdb job to run under poetry (#12412) 2022-04-08 12:00:05 +01:00
Andrew Morgan
163fd686b5
Update mdbook to 0.4.17 (#12339)
Update mdbook (the tool used to render the documentation website) to version 0.4.17.
2022-04-05 17:21:58 +01:00
David Robertson
9a3f1f5383
Nuke the tox packaging job (#12334) 2022-03-31 11:01:26 +01:00
Andrew Morgan
19a1d6a42a
Remove references to "msc2403" (#12165) 2022-03-30 11:15:47 +00:00
kegsay
8b7b371ff6
Re-enable test concurrency on Complement (#12283) 2022-03-24 15:59:06 -04:00
reivilibre
bfa7d6b035
Fix CI not attaching source distributions and wheels to the GitHub releases. (#12131) 2022-03-08 15:11:50 +00:00
Richard van der Hoff
0211f18d65
Switch the tests-done job to an Action (#12161)
I've factored it out for easier use in other workflows.
2022-03-07 12:24:06 +00:00
David Robertson
36071d39f7
Changelog (#12153) 2022-03-04 12:01:51 +00:00
Eric Eastwood
a511a890d7
Enable MSC2716 Complement tests in Synapse (#12145)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-03-03 11:19:20 +00:00
Eric Eastwood
31b125ccec
Enable MSC3030 Complement tests in Synapse (#12144)
The Complement tests for MSC3030 are now merged, https://github.com/matrix-org/complement/pull/178

Synapse implmentation: https://github.com/matrix-org/synapse/pull/9445
2022-03-03 11:45:23 +01:00
David Robertson
1fbe0316a9
Add suffices to scripts in scripts-dev (#12137)
* Rename scripts-dev to have suffices

* Update references to `scripts-dev`

* Changelog

* These scripts don't pass mypy
2022-03-02 18:00:26 +00:00
David Robertson
5f62a094de
Detox, part 1 of N (#12119)
* Don't use `tox` for `check-sampleconfig`
* Don't use `tox` for check-newsfragment
2022-03-01 19:47:02 +00:00
David Robertson
af2c1e3d2a
Tidy the building of sdists and wheels (#12051)
* Don't build distribution pkgs in tests.yml
* Run `release-artifacts` on release branches
* Use backend-meta workflow for packaging
2022-02-22 11:33:37 +00:00
Richard van der Hoff
bb98c593a5
Prepare for rename of default complement branch (#11971)
use `HEAD` rather than hardcoding `master`
2022-02-11 14:43:34 +00:00
kegsay
a3865ed525
Run Complement tests sequentially (#11910)
Since #11811 there has been general Complement flakiness around networking.
It seems like tests are hitting the wrong containers. In an effort to diagnose
the cause of this, as well as reduce its impact on this project, set the
parallelsim to 1 (no parallelism) when running tests.

If this fixes the flakiness then this indicates the cause and I can diagnose
this further. If this doesn't fix the flakiness then that implies some kind
of test pollution which also helps to diagnose this further.
2022-02-04 13:04:57 +00:00
kegsay
0938f32e93
CI: run Complement on the VM, not inside Docker (#11811)
* CI: run Complement on the VM, not inside Docker

This requires https://github.com/matrix-org/complement/pull/289

We now run Complement on the VM instead of inside a Docker container.
This is to allow Complement to bind to any high-numbered port when it
starts up its own federation servers. We want to do this to allow for
more concurrency when running complement tests. Previously, Complement
only ever bound to `:8448` when running its own federation server. This
prevented multiple federation tests running at the same time as they would
fight each other on the port. This did however allow Complement to run
in Docker, as the host could just port forward `:8448` to allow homeserver
containers to communicate to Complement. Now that we are using random
ports however, we cannot use Docker to run Complement. This ends up
being a good thing because:
 - Running Complement tests locally is closer to how they run in CI.
 - Allows the `CI` env var to be removed in Complement.
 - Slightly speeds up runs as we don't need to pull down the Complement
   image prior to running tests. This assumes GHA caches actions sensibly.

* Changelog

* Full stop

* Update .github/workflows/tests.yml

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

* Review comments

* Update .github/workflows/tests.yml

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-01-25 15:05:22 +00:00
Richard van der Hoff
2d327d25bf
Skip the initial amd64-only Docker build (#11810)
PyNaCl's recent 1.5.0 release on PyPi includes arm64 wheels, which means our
arm64 docker images now build in a sensible amount of time, so we can skip the
amd64-only build.
2022-01-24 18:31:23 +00:00
Shay
9006ee36d1
Drop support for and remove references to EOL Python 3.6 (#11683)
* remove reference in comments to python3.6

* upgrade tox python env in script

* bump python version in example for completeness

* upgrade python version requirement in setup doc

* upgrade necessary python version in __init__.py

* upgrade python version in setup.py

* newsfragment

* drops refs to bionic and replace with focal

* bump refs to postgres 9.6 to 10

* fix hanging ci

* try installing tzdata first

* revert change made in b979f336

* ignore new random mypy error while debugging other error

* fix lint error for temporary workaround

* revert change to install list

* try passing env var

* export debian frontend var?

* move line and add comment

* bump pillow dependency

* bump lxml depenency

* install libjpeg-dev for pillow

* bump automat version to one compatible with py3.8

* add libwebp for pillow

* bump twisted trunk python version

* change suffix of newsfragment

* remove redundant python 3.7 checks

* lint
2022-01-21 14:23:26 -08:00
David Robertson
4ca8fcdd5a
Invite PR submitters to credit themselves (#11744)
Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
2022-01-13 18:12:59 +00:00
Richard van der Hoff
d41c4654db
Use buildkit's cache feature to speed up docker builds (#11691)
Having spent much of the last week attempting to run complement tests from somewhere with damp string instead of internet... something had to be done.
2022-01-12 10:37:57 +00:00
kegsay
3be63654e4
Prettier complement logs (#11707)
* Prettier complement logs

* Changelog
2022-01-10 11:46:40 +00:00
Shay
87da37374a
Drop EOL python 3.6 from CI (#11595)
* remove python 3.6 and postgres 9.6 from github workflow

* remove python 3.6 env from tox

* newsfragment

* correct postgres version

* add py310 to tox env list
2021-12-21 11:37:04 -08:00
Patrick Cloke
a4521ce0a8
Support the stable /hierarchy endpoint from MSC2946 (#11329)
This also makes additional updates where the implementation
had drifted from the approved MSC.

Unstable endpoints will be removed at a later data.
2021-11-29 14:32:20 -05:00
Patrick Cloke
9d1971a5c4
Return the stable event field from /send_join per MSC3083. (#11413)
This does not remove the unstable field and still parses both.
Handling of the unstable field will need to be removed in the
future.
2021-11-29 15:43:20 +00:00
Richard van der Hoff
e2e9bea1ce
Publish a develop docker image (#11380)
I'd find it helpful to have a docker image corresponding to current develop,
without having to build my own.
2021-11-19 10:56:59 +00:00
Dirk Klimpel
93aa670642
Update outdated links in PULL_REQUEST_TEMPLATE.md (#11225) 2021-11-01 17:29:51 +00:00
Hillery Shay
7d70582eb0
Fix broken export-data admin command and add a test for it to CI (#11078)
Fix broken export-data admin command and add a test for it to CI
2021-10-18 08:14:12 -07:00
Patrick Cloke
8711e15734
Always dump logs from trial during CI. (#11068)
Instead of only dumping them if trial passes.
2021-10-12 20:09:49 +00:00
Erik Johnston
0b4d5ce5e3
Fix CI to run the unit tests without optional deps (#11017)
This also turns off calculating code coverage, as we didn't use it and it was a lot of noise
2021-10-08 10:05:48 +01:00
Brendan Abolivier
b0460936c8
Add the synapse-core team as code owners (#10994)
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
2021-10-06 16:03:17 +02:00
David Robertson
6f6e956338
Run CI with Python 3.10 and Postgres 14 (#10992) 2021-10-05 12:43:04 +01:00
Patrick Cloke
ebd8baf61f
Clear our destination directories before copying files to GitHub pages. (#10869)
This should fix stale deleted files being still accessible.
2021-09-21 16:32:46 +00:00
David Robertson
f455b0e420
GHA: reintroduce an env var for $GITHUB_HEAD_REF (#10659)
This should ensure GHA runs synapse against the same-named sytest branch
2021-09-20 17:35:16 +01:00
Dan Callahan
2efc838f05
Avoid duplicate issues from Twisted trunk failures (#10672)
Setting `update_existing: true` in the `create-an-issue` GitHub Action
will avoid opening duplicate issues if an open issue already exists with
an identical title.

If no open issues match the title, then a new issue will be created.

This helps avoid spamming our issue tracker should there be a failure
when testing against Twisted's trunk.

This PR also pins the SHA of the `create-an-issue` action to mitigate
the risk of a malicious actor gaining access to JasonEtco's account.

See GitHub's page on security hardening third party actions for more:
https://docs.github.com/en/actions/learn-github-actions/security-hardening-for-github-actions#using-third-party-actions

Signed-off-by: Dan Callahan <danc@element.io>
2021-08-23 13:06:49 +00:00
Richard van der Hoff
4db65f911a
Run a nightly CI build against Twisted trunk. (#10651)
This creates a GHA workflow which runs at 8am every day, and runs mypy, trial and sytest against Twisted's current trunk. If any of the jobs fail, it opens an issue.
2021-08-23 11:12:45 +01:00
David Robertson
878528913d Remove buildkite-era comment 2021-08-12 11:48:36 +01:00
David Robertson
74fcd5aab9 portdb also uses coverage, so provide $TOP there 2021-08-12 10:41:01 +01:00
David Robertson
cb5976ebd7 set TOP in sytest containers 2021-08-11 20:08:48 +01:00
David Robertson
33ef86aa25 Rename ci to .ci 2021-08-11 19:59:57 +01:00
David Robertson
58e5da5aa0 Remove buildkite from portdb CI tests 2021-08-10 16:38:13 +01:00
David Robertson
c5988a8eb7 Remove unused BUILDKITE_BRANCH env var 2021-08-10 16:38:13 +01:00
David Robertson
3d67b8c82b Move sytest worker-blacklist to ci directory 2021-08-10 16:38:13 +01:00
David Robertson
03fb99a5c8 check-newsfragment: pass pr number explicitly
use PULL_REQUEST_NUMBER instead of BUILDKITE_PULL_REQUEST
remove the other user of BUILDKITE_PULL_REQUEST, namely merge_base_branch.sh
2021-08-10 16:38:13 +01:00
David Robertson
8da9e3cb69 Move test_old_deps.sh to new ci dir 2021-08-10 16:38:13 +01:00
Erik Johnston
e33f14e8d5
Don't fail CI when lint-newfile job was skipped (#10529) 2021-08-05 11:22:27 +01:00
Richard van der Hoff
a6ea32a798
Fix the tests-done github actions step, again (#10512) 2021-08-02 21:06:34 +01:00
Patrick Cloke
13944678c3
Use new go test running syntax for complement. (#10488)
Updates CI and the helper script t ensures all tests are run (in parallel).
2021-07-27 12:08:51 +00:00
Richard van der Hoff
f22252d4f9
Enable docker image caching for the deb build (#10431) 2021-07-26 11:36:01 +01:00
Erik Johnston
ab82fd6ed1 Merge branch 'release-v1.39' into develop 2021-07-23 09:19:24 +01:00
Richard van der Hoff
4c3fdfc808
Fix an error in the docker workflow (#10461) 2021-07-22 21:50:30 +01:00
Richard van der Hoff
5e2df47f72
Cancel redundant GHA workflows (#10451) 2021-07-22 11:35:06 +01:00