This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.
Spawned from missing messages we were seeing on `matrix.org` from a
federated Gtiter bridged room, https://gitlab.com/gitterHQ/webapp/-/issues/2770.
The underlying issue in Synapse is tracked by https://github.com/matrix-org/synapse/issues/10066
where the message and join event race and the message is `soft_failed` before the
`join` event reaches the remote federated server.
Less soft_failed events = better and usually this should only trigger for events
where people are doing bad things and trying to fuzz and fake everything.
Fixes#1834.
`get_new_events_for_appservice` internally calls `get_events_as_list`, which will filter out any rejected events. If all returned events are filtered out, `_notify_interested_services` will return without updating the last handled stream position. If there are 100 consecutive such events, processing will halt altogether.
Breaking the loop is now done by checking whether we're up-to-date with `current_max` in the loop condition, instead of relying on an empty `events` list.
Signed-off-by: Willem Mulder <14mRh4X0r@gmail.com>
If backfilling is slow then the client may time out and retry, causing
Synapse to start a new `/backfill` before the existing backfill has
finished, duplicating work.
This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels:
* emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions.
* emit a span while we hold a database connection open
* emit a span for each database transaction - actual actual transaction.
* emit a span for each database query.
I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).
The existing tracing reports an error each time there is a timeout, which isn't
really representative.
Additionally, we log things about the way `wait_for_events` works
(eg, the result of the callback) to the *parent* span, which is confusing.
Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
==============================
Bugfixes
--------
- Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](https://github.com/matrix-org/synapse/issues/10079))
-----BEGIN PGP SIGNATURE-----
iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmCvpZMQHGVyaWtAbWF0
cml4Lm9yZwAKCRClQuTtGw+sCdPwCACQIlIWd6eIoXLUc+wDcrd+k5xL376EdYah
x7ABswiYSm+9C4xr58gJD3xc6eiD2PCIWdZN0rsQDLIOfSXW6x1lyKD+Ds0HySok
MaVpsoxbb9o/Zf9qtXF2bLSArZUQwfoNaA45NLgNzfUIijf1e+bd2wNEgHlRSoMz
m10GggOFU0Ds/CCYZpxZw/eXDbLWL7eaHR30vw/jQ1cEsV+S4ucnUmLHFCF7YCyI
Np80pnywH5cYKerecldFWenL4YZJswVbx+AW9e3lBzq5jOrRZkmLkaHg10mCR2f6
CV03ie65Ce+7x5UU6v6nHA0DYUTQGIjlJBtyCN3tFglUduQ8Gpu0
=Mfsf
-----END PGP SIGNATURE-----
Merge tag 'v1.35.0rc2' into develop
Synapse 1.35.0rc2 (2021-05-27)
==============================
Bugfixes
--------
- Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](https://github.com/matrix-org/synapse/issues/10079))
