anonymousland-synapse/synapse/handlers
Eric Eastwood 1ca70fd312
Allow room creator to send MSC2716 related events in existing room versions (#10566)
* Allow room creator to send MSC2716 related events in existing room versions

Discussed at https://github.com/matrix-org/matrix-doc/pull/2716/#discussion_r682474869

Restoring `get_create_event_for_room_txn` from,
44bb3f0cf5

* Add changelog

* Stop people from trying to redact MSC2716 events in unsupported room versions

* Populate rooms.creator column for easy lookup

> From some [out of band discussion](https://matrix.to/#/!UytJQHLQYfvYWsGrGY:jki.re/$p2fKESoFst038x6pOOmsY0C49S2gLKMr0jhNMz_JJz0?via=jki.re&via=matrix.org), my plan is to use `rooms.creator`. But currently, we don't fill in `creator` for remote rooms when a user is invited to a room for example. So we need to add some code to fill in `creator` wherever we add to the `rooms` table. And also add a background update to fill in the rows missing `creator` (we can use the same logic that `get_create_event_for_room_txn` is doing by looking in the state events to get the `creator`).
>
> https://github.com/matrix-org/synapse/pull/10566#issuecomment-901616642

* Remove and switch away from get_create_event_for_room_txn

* Fix no create event being found because no state events persisted yet

* Fix and add tests for rooms creator bg update

* Populate rooms.creator field for easy lookup

Part of https://github.com/matrix-org/synapse/pull/10566

 - Fill in creator whenever we insert into the rooms table
 - Add background update to backfill any missing creator values

* Add changelog

* Fix usage

* Remove extra delta already included in #10697

* Don't worry about setting creator for invite

* Only iterate over rows missing the creator

See https://github.com/matrix-org/synapse/pull/10697#discussion_r695940898

* Use constant to fetch room creator field

See https://github.com/matrix-org/synapse/pull/10697#discussion_r696803029

* More protection from other random types

See https://github.com/matrix-org/synapse/pull/10697#discussion_r696806853

* Move new background update to end of list

See https://github.com/matrix-org/synapse/pull/10697#discussion_r696814181

* Fix query casing

* Fix ambiguity iterating over cursor instead of list

Fix `psycopg2.ProgrammingError: no results to fetch` error
when tests run with Postgres.

```
SYNAPSE_POSTGRES=1 SYNAPSE_TEST_LOG_LEVEL=INFO python -m twisted.trial tests.storage.databases.main.test_room
```

---

We use `txn.fetchall` because it will return the results as a
list or an empty list when there are no results.

Docs:

> `cursor` objects are iterable, so, instead of calling explicitly fetchone() in a loop, the object itself can be used:
>
> https://www.psycopg.org/docs/cursor.html#cursor-iterable

And I'm guessing iterating over a raw cursor does something weird when there are no results.

---

Test CI failure: https://github.com/matrix-org/synapse/pull/10697/checks?check_run_id=3468916530
```
tests.test_visibility.FilterEventsForServerTestCase.test_large_room
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/runner/work/synapse/synapse/tests/storage/databases/main/test_room.py", line 85, in test_background_populate_rooms_creator_column
    self.get_success(
  File "/home/runner/work/synapse/synapse/tests/unittest.py", line 500, in get_success
    return self.successResultOf(d)
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/trial/_synctest.py", line 700, in successResultOf
    self.fail(
twisted.trial.unittest.FailTest: Success result expected on <Deferred at 0x7f4022f3eb50 current result: None>, found failure result instead:
Traceback (most recent call last):
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 701, in errback
    self._startRunCallbacks(fail)
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 764, in _startRunCallbacks
    self._runCallbacks()
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1751, in gotResult
    current_context.run(_inlineCallbacks, r, gen, status)
--- <exception caught here> ---
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 1657, in _inlineCallbacks
    result = current_context.run(
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/failure.py", line 500, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 224, in do_next_background_update
    await self._do_background_update(desired_duration_ms)
  File "/home/runner/work/synapse/synapse/synapse/storage/background_updates.py", line 261, in _do_background_update
    items_updated = await update_handler(progress, batch_size)
  File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1399, in _background_populate_rooms_creator_column
    end = await self.db_pool.runInteraction(
  File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 686, in runInteraction
    result = await self.runWithConnection(
  File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 791, in runWithConnection
    return await make_deferred_yieldable(
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/internet/defer.py", line 858, in _runCallbacks
    current.result = callback(  # type: ignore[misc]
  File "/home/runner/work/synapse/synapse/tests/server.py", line 425, in <lambda>
    d.addCallback(lambda x: function(*args, **kwargs))
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 293, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/deprecate.py", line 298, in deprecatedFunction
    return function(*args, **kwargs)
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/python/compat.py", line 404, in reraise
    raise exception.with_traceback(traceback)
  File "/home/runner/work/synapse/synapse/.tox/py/lib/python3.9/site-packages/twisted/enterprise/adbapi.py", line 284, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 786, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/home/runner/work/synapse/synapse/synapse/storage/database.py", line 554, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/home/runner/work/synapse/synapse/synapse/storage/databases/main/room.py", line 1375, in _background_populate_rooms_creator_column_txn
    for room_id, event_json in txn:
psycopg2.ProgrammingError: no results to fetch
```

* Move code not under the MSC2716 room version underneath an experimental config option

See https://github.com/matrix-org/synapse/pull/10566#issuecomment-906437909

* Add ordering to rooms creator background update

See https://github.com/matrix-org/synapse/pull/10697#discussion_r696815277

* Add comment to better document constant

See https://github.com/matrix-org/synapse/pull/10697#discussion_r699674458

* Use constant field
2021-09-04 00:58:49 -05:00
..
ui_auth Implement MSC3231: Token authenticated registration (#10142) 2021-08-21 22:14:43 +01:00
__init__.py Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
_base.py Fix import of the default SAML mapping provider. (#10477) 2021-07-27 11:34:15 -04:00
account_data.py Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
account_validity.py Add a module type for account validity (#9884) 2021-07-16 18:11:53 +02:00
admin.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
appservice.py Only return an appservice protocol if it has a service providing it. (#10532) 2021-08-05 08:22:14 -04:00
auth.py Remove pushers when deleting 3pid from account (#10581) 2021-08-26 13:53:57 +01:00
cas.py [pyupgrade] synapse/ (#10348) 2021-07-19 15:28:05 +01:00
deactivate_account.py Port "Allow users to click account renewal links multiple times without hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832) 2021-04-19 19:16:34 +01:00
device.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
devicemessage.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
directory.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
e2e_keys.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
e2e_room_keys.py Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
event_auth.py Clarify error message when joining a restricted room. (#10572) 2021-08-11 14:31:39 -04:00
events.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
federation_event.py Allow room creator to send MSC2716 related events in existing room versions (#10566) 2021-09-04 00:58:49 -05:00
federation.py Populate rooms.creator field for easy lookup (#10697) 2021-09-01 16:27:58 +01:00
groups_local.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
identity.py Send unstable-prefixed room_type in store-invite IS API requests (#10435) 2021-08-04 13:39:57 -04:00
initial_sync.py Convert room member storage tuples to attrs. (#10629) 2021-08-18 09:22:07 -04:00
message.py Allow room creator to send MSC2716 related events in existing room versions (#10566) 2021-09-04 00:58:49 -05:00
oidc.py [pyupgrade] synapse/ (#10348) 2021-07-19 15:28:05 +01:00
pagination.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
password_policy.py Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
presence.py Additional type hints for REST servlets (part 2). (#10674) 2021-08-26 11:53:52 +00:00
profile.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
read_marker.py Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
receipts.py Handle string read receipt data (#10606) 2021-08-16 12:22:38 +01:00
register.py Correctly initialise the synapse_user_logins metric. (#10677) 2021-08-24 09:17:51 +00:00
room_list.py Fix exceptions in logs when failing to get remote room list (#10541) 2021-08-06 14:05:41 +01:00
room_member_worker.py Implement knock feature (#6739) 2021-06-09 19:39:51 +01:00
room_member.py Enforce the max length for per-room display names / avatar URLs. (#10654) 2021-08-23 11:25:33 -04:00
room_summary.py Ignore rooms with unknown room versions in the spaces summary. (#10727) 2021-09-01 17:01:08 +00:00
room.py Raise an error if an unknown preset is used to create a room. (#10738) 2021-09-03 13:46:18 +00:00
saml.py [pyupgrade] synapse/ (#10348) 2021-07-19 15:28:05 +01:00
search.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
send_email.py Fix incompatibility with Twisted < 21. (#10713) 2021-08-27 16:33:41 +01:00
set_password.py Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
sso.py Correctly initialise the synapse_user_logins metric. (#10677) 2021-08-24 09:17:51 +00:00
state_deltas.py Remove redundant "coding: utf-8" lines (#9786) 2021-04-14 15:34:27 +01:00
stats.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
sync.py Additional type hints for the sync REST servlet. (#10666) 2021-08-23 08:14:42 -04:00
typing.py Include room ID in ignored EDU log messages (#10507) 2021-08-03 13:35:49 +00:00
user_directory.py Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00