Commit Graph

51 Commits

Author SHA1 Message Date
Shay
96bcc5d902
Revert "check sqlite database file exists before porting/#14692" (#15301) 2023-03-21 10:49:25 -07:00
Patrick Cloke
4fc8875876
Refactor media modules. (#15146)
* Removes the `v1` directory from `test.rest.media.v1`.
* Moves the non-REST code from `synapse.rest.media.v1` to `synapse.media`.
* Flatten the `v1` directory from `synapse.rest.media`,  but leave compatiblity
  with 3rd party media repositories and spam checkers.
2023-02-27 08:26:05 -05:00
dependabot[bot]
9bb2eac719
Bump black from 22.12.0 to 23.1.0 (#15103) 2023-02-22 15:29:09 -05:00
David Robertson
e26d7d5ae7
Teach portdb about un_partial_stated_event_stream (#15108)
* Sort BOOLEAN_COLUMNS and APPEND_ONLY_TABLES

So I can see if a given table is present in logarithmic time, rather
than linear.

* Teach portdb about `un_partial_stated_event_streams`

* Comments comments comments

* Changelog
2023-02-20 13:35:24 +00:00
Erik Johnston
65d0386693
Always notify replication when a stream advances (#14877)
This ensures that all other workers are told about stream updates in a timely manner, without having to remember to manually poke replication.
2023-01-20 18:02:18 +00:00
Jeyachandran Rathnam
5c9be9c760
Check sqlite database file exists before porting. (#14692)
To avoid creating an empty SQLite file if the given path
is incorrect.
2022-12-22 13:26:37 -05:00
reivilibre
96251af50d
Fix a bug introduced in v1.67.0 where not specifying a config file or a server URL would lead to the register_new_matrix_user script failing. (#14637) 2022-12-07 13:39:27 +00:00
Finn
fe50738e59
let update_synapse_database run on a multi-database configurations (#13422)
* Allow sharded database in db migrate script

Signed-off-by: Finn Herzfeld <finn@beeper.com>

* Update changelog.d/13422.bugfix

Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>

* Remove check entirely

* remove unused import

Signed-off-by: Finn Herzfeld <finn@beeper.com>
Co-authored-by: finn <finn@beeper.com>
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2022-10-19 19:08:40 +01:00
Patrick Cloke
3bbe532abb
Add an API for listing threads in a room. (#13394)
Implement the /threads endpoint from MSC3856.

This is currently unstable and behind an experimental configuration
flag.

It includes a background update to backfill data, results from
the /threads endpoint will be partial until that finishes.
2022-10-13 08:02:11 -04:00
Brendan Abolivier
be76cd8200
Allow admins to require a manual approval process before new accounts can be used (using MSC3866) (#13556) 2022-09-29 15:23:24 +02:00
Brendan Abolivier
8ae42ab8fa
Support enabling/disabling pushers (from MSC3881) (#13799)
Partial implementation of MSC3881
2022-09-21 14:39:01 +00:00
David Robertson
fff9b955fa
Generate separate snapshots for logical databases (#13792)
* Generate separate snapshots for sqlite, postgres and common
* Cleanup postgres dbs in the TRAP
* Say which logical DB we're applying updates to
* Run background updates on the state DB
* Add new option for accepting a SCHEMA_NUMBER
2022-09-20 14:14:12 +01:00
Nick Mills-Barrett
cdbb641232
Add receipts event stream ordering (#13703) 2022-09-13 08:16:37 +01:00
Richard van der Hoff
d092e6f32a
Support registration_shared_secret in a file (#13614)
A new `registration_shared_secret_path` option. This is kinda handy for k8s deployments and things.
2022-08-25 16:27:46 +00:00
Richard van der Hoff
a2ce614447
register_new_matrix_user: read server url from config (#13616)
Fixes https://github.com/matrix-org/synapse/issues/3672:
`https://localhost:8448` is virtually never right.
2022-08-25 15:29:08 +01:00
Brendan Abolivier
47822fd2e8
Merge branch 'master' into develop 2022-07-19 16:14:02 +02:00
Andrew Morgan
6faaf76a32
Remove 'anonymised' from the phone home stats documentation (#13321) 2022-07-19 12:38:29 +00:00
Patrick Cloke
4db7862e0f
Drop unused tables from groups/communities. (#12967)
These tables have been unused since Synapse v1.61.0, although schema version 72
was added in Synapse v1.62.0.
2022-07-13 09:55:14 -04:00
David Robertson
28d96cb2b4
Ensure portdb selects _all_ rows with negative rowids (#13226) 2022-07-11 10:36:18 +01:00
reivilibre
fb7d24ab6d
Check that auto_vacuum is disabled when porting a SQLite database to Postgres, as VACUUMs must not be performed between runs of the script. (#13195) 2022-07-07 10:08:04 +00:00
Patrick Cloke
b0366853ca Merge remote-tracking branch 'origin/release-v1.62' into develop 2022-06-30 13:27:24 -04:00
Erik Johnston
a3a05c812d
Add index to help delete old push actions (#13141) 2022-06-30 14:05:49 +00:00
Erik Johnston
92a0c18ef0
Improve performance of getting unread counts in rooms (#13119) 2022-06-29 10:32:38 +00:00
Erik Johnston
0d1d3e0708
Speed up get_unread_event_push_actions_by_room (#13005)
Fixes #11887 hopefully.

The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller.

This needs two major changes:
1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it
2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`.

In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
2022-06-15 15:17:14 +00:00
Patrick Cloke
53b77b203a
Replace noop background updates with DELETE. (#12954)
Removes the `register_noop_background_update` and deletes the background
updates directly in a delta file.
2022-06-13 14:06:27 -04:00
David Robertson
f30bcbd84a
Fix Synapse git info missing in version strings (#12973) 2022-06-07 15:24:11 +01:00
Patrick Cloke
f7baffd8ec
Remove remaining pieces of groups code. (#12966)
* Remove an unused stream ID generator.
* Remove the now unused remote profile cache.
2022-06-06 13:20:05 -04:00
Patrick Cloke
6b46c3eb3d
Remove groups code from synapse_port_db. (#12899) 2022-06-03 12:13:35 -04:00
Patrick Cloke
2e8763ec96
Remove most groups datastore code. (#12895)
The remaining piece is a background update that is needed
for backwards compatibility.
2022-05-31 20:28:17 -04:00
Jae Lo Presti
f16ec055cc
hash_password: raise an error if no config file is specified (#12789) 2022-05-19 13:03:13 +00:00
Patrick Cloke
4586119f0b
Add missing type hints to config classes. (#12402) 2022-04-11 12:07:23 -04:00
David Robertson
961ee75a9b
Disallow untyped defs in synapse._scripts (#12422)
Of note: 

* No untyped defs in `register_new_matrix_user`

This one might be contraversial. `request_registration` has three
dependency-injection arguments used for testing. I'm removing the
injection of the `requests` module and using `unitest.mock.patch` in the
test cases instead.

Doing `reveal_type(requests)` and `reveal_type(requests.get)` before the
change:

```
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "Any"
synapse/_scripts/register_new_matrix_user.py:46: note: Revealed type is "Any"
```

And after:

```
synapse/_scripts/register_new_matrix_user.py:44: note: Revealed type is "types.ModuleType"
synapse/_scripts/register_new_matrix_user.py:45: note: Revealed type is "def (url: Union[builtins.str, builtins.bytes], params: Union[Union[_typeshed.SupportsItems[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]], typing.Iterable[Tuple[Union[builtins.str, builtins.bytes, builtins.int, builtins.float], Union[builtins.str, builtins.bytes, builtins.int, builtins.float, typing.Iterable[Union[builtins.str, builtins.bytes, builtins.int, builtins.float]], None]]], builtins.str, builtins.bytes], None] =, data: Union[Any, None] =, headers: Union[Any, None] =, cookies: Union[Any, None] =, files: Union[Any, None] =, auth: Union[Any, None] =, timeout: Union[Any, None] =, allow_redirects: builtins.bool =, proxies: Union[Any, None] =, hooks: Union[Any, None] =, stream: Union[Any, None] =, verify: Union[Any, None] =, cert: Union[Any, None] =, json: Union[Any, None] =) -> requests.models.Response"
```

* Drive-by comment in `synapse.storage.types`

* No untyped defs in `synapse_port_db`

This was by far the most painful. I'm happy to break this up into
smaller pieces for review if it's not managable as-is.
2022-04-11 12:41:55 +01:00
David Robertson
0cd182f296
Make synapse._scripts pass typechecks (#12421) 2022-04-08 14:00:12 +00:00
Erik Johnston
5c9e39e619
Track device list updates per room. (#12321)
This is a first step in dealing with #7721.

The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things:

1. we can defer calculating the set of remote servers that need to be poked about the change; and
2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed.

However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled.

There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly).

Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
2022-04-04 15:25:20 +01:00
David Robertson
4aeb00ca20
Move synctl into synapse._scripts and expose as an entrypoint (#12140) 2022-03-04 11:58:49 +00:00
David Robertson
f3f0ab10fe
Move scripts directory inside synapse, exposing as setuptools entry_points (#12118)
* Two scripts are basically entry_points already
* Move and rename scripts/* to synapse/_scripts/*.py
* Delete sync_room_to_group.pl
* Expose entry points in setup.py
* Update linter script and config
* Fixup scripts & docs mentioning scripts that moved

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2022-03-02 13:00:16 +00:00
lukasdenk
d8be9924ef
Add a flag to the synapse_review_recent_signups script to ignore and filter appservice users. (#11675) 2022-01-17 16:43:25 +00:00
Patrick Cloke
66c4b774fd
Add type hints to synapse._scripts (#11297) 2021-11-10 17:55:32 +00:00
Samuel Philipp
4e393af52f
Fixed config parse bug in review_recent_signups (#11191) 2021-10-27 17:25:18 +01:00
V02460
b7f7ca24b1
Remove shebang line from module files (#10415)
Signed-off-by: Kai A. Hiller <V02460@gmail.com>
2021-07-29 21:34:14 +01:00
Erik Johnston
6655ea5587
Add script for getting info about recently registered users (#10290) 2021-07-06 13:03:16 +01:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Dmitry Borodaenko
7c43447477
Strip trailing / from server_url in register_new_matrix_user (#8823)
When server URL provided to register_new_matrix_user includes path
component (e.g. "http://localhost:8008/"), the command fails with
"ERROR! Received 400 Bad Request". Stripping trailing slash from the
server_url command argument makes sure combined endpoint URL remains
valid.

Signed-off-by: Dmitry Borodaenko angdraug@debian.org
2020-11-26 10:57:26 +00:00
Dirk Klimpel
3f0ff53158
Remove deprecated /_matrix/client/*/admin endpoints (#8785)
These are now only available via `/_synapse/admin/v1`.
2020-11-25 16:26:11 -05:00
Jonathan de Jong
837293c314
Remove obsolete __future__ imports (#8337) 2020-09-17 08:37:01 -04:00
Dagfinn Ilmari Mannsåker
a3f11567d9
Replace all remaining six usage with native Python 3 equivalents (#7704) 2020-06-16 08:51:47 -04:00
Andrew Morgan
bc29a19731 Replace instance variations of homeserver with correct case/spacing 2019-11-12 13:08:12 +00:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Amber Brown
3f189c902e
Fix flake8 (#4519) 2019-01-30 10:53:17 +00:00
Neil Johnson
d2f7c4e6b1
create support user (#4141)
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
2018-12-14 18:20:59 +00:00