synapse-product/synapse/storage/databases/main
James Salter d902181de9
Unified search query syntax using the full-text search capabilities of the underlying DB. (#11635)
Support a unified search query syntax which leverages more of the full-text
search of each database supported by Synapse.

Supports, with the same syntax across Postgresql 11+ and Sqlite:

- quoted "search terms"
- `AND`, `OR`, `-` (negation) operators
- Matching words based on their stem, e.g. searches for "dog" matches
  documents containing "dogs". 

This is achieved by 

- If on postgresql 11+, pass the user input to `websearch_to_tsquery`
- If on sqlite, manually parse the query and transform it into the sqlite-specific
  query syntax.

Note that postgresql 10, which is close to end-of-life, falls back to using
`phraseto_tsquery`, which only supports a subset of the features.

Multiple terms separated by a space are implicitly ANDed.

Note that:

1. There is no escaping of full-text syntax that might be supported by the database;
  e.g. `NOT`, `NEAR`, `*` in sqlite. This runs the risk that people might discover this
  as accidental functionality and depend on something we don't guarantee.
2. English text is assumed for stemming. To support other languages, either the target
  language needs to be known at the time of indexing the message (via room metadata,
  or otherwise), or a separate index for each language supported could be created.

Sqlite docs: https://www.sqlite.org/fts3.html#full_text_index_queries
Postgres docs: https://www.postgresql.org/docs/11/textsearch-controls.html
2022-10-25 14:05:22 -04:00
..
__init__.py Show erasure status when listing users in the Admin API (#14205) 2022-10-21 13:52:44 +01:00
account_data.py Speed up fetching large numbers of push rules (#13592) 2022-08-23 13:15:43 +01:00
appservice.py Federation Sender & Appservice Pusher Stream Optimisations (#13251) 2022-07-15 09:36:56 +01:00
cache.py Invalidate rooms for user caches when receiving membership events (#14155) 2022-10-17 13:27:51 +01:00
censor_events.py Safe async event cache (#13308) 2022-07-19 11:25:29 +00:00
client_ips.py
deviceinbox.py Add missing types to opentracing. (#13345) 2022-07-21 12:01:52 +00:00
devices.py Add debugging to help diagnose lost device-list-update (#14268) 2022-10-24 10:45:10 +01:00
directory.py
e2e_room_keys.py Add StreamKeyType class and replace string literals with constants (#12567) 2022-05-16 15:35:31 +00:00
end_to_end_keys.py Refactor _get_e2e_device_keys_txn to split large queries (#13956) 2022-10-03 13:46:36 +01:00
event_federation.py Stop getting missing prev_events after we already know their signature is invalid (#13816) 2022-10-15 00:36:49 -05:00
event_push_actions.py Update the thread_id right before use (in case the bg update hasn't finished) (#14222) 2022-10-18 14:55:41 +00:00
events_bg_updates.py Bg update to populate new events table columns (#13215) 2022-07-15 12:47:26 +01:00
events_forward_extremities.py
events_worker.py Fix MSC3030 /timestamp_to_event returning outliers that it has no idea whether are near a gap or not (#14215) 2022-10-18 19:46:25 -05:00
events.py Properly update the threads table when thread events are redacted. (#14248) 2022-10-21 09:11:19 -04:00
filtering.py
keys.py Add some type hints to datastore (#12485) 2022-04-27 13:05:00 +01:00
lock.py Require SQLite >= 3.27.0 (#13760) 2022-09-09 11:14:10 +01:00
media_repository.py Replace noop background updates with DELETE. (#12954) 2022-06-13 14:06:27 -04:00
metrics.py Add some type hints to datastore (#12717) 2022-05-17 15:29:06 +01:00
monthly_active_users.py Safe async event cache (#13308) 2022-07-19 11:25:29 +00:00
openid.py
presence.py Revert "Make all process_replication_rows methods async (#13304)" (#13312) 2022-07-18 14:28:14 +01:00
profile.py Remove remaining pieces of groups code. (#12966) 2022-06-06 13:20:05 -04:00
purge_events.py Add support to purge rows from MSC2716 and other tables when purging a room (#13825) 2022-09-16 10:56:56 -05:00
push_rule.py Implementation for MSC3664: Pushrules for relations (#11804) 2022-10-25 14:38:01 +01:00
pusher.py Track device IDs for pushers (#13831) 2022-09-21 15:31:53 +00:00
receipts.py Properly return the thread ID down sync. (#14159) 2022-10-13 12:15:41 -04:00
registration.py Allow admins to require a manual approval process before new accounts can be used (using MSC3866) (#13556) 2022-09-29 15:23:24 +02:00
rejections.py
relations.py Revert behavior change for bundling edits of non-message events (#14283) 2022-10-24 17:07:16 +01:00
room_batch.py
room.py When restarting a partial join resync, prioritise the server which actioned a partial join (#14126) 2022-10-18 12:33:18 +01:00
roommember.py Bump flake8-bugbear from 21.3.2 to 22.9.23 (#14042) 2022-10-19 19:38:24 +00:00
search.py Unified search query syntax using the full-text search capabilities of the underlying DB. (#11635) 2022-10-25 14:05:22 -04:00
session.py
signatures.py remove constantly lib use and switch to enums. (#12624) 2022-05-04 11:26:11 +00:00
state_deltas.py Wait for lazy join to complete when getting current state (#12872) 2022-06-01 16:02:53 +01:00
state.py Instrument get_metadata_for_events for tracing (#13730) 2022-09-07 11:41:52 -05:00
stats.py Require SQLite >= 3.27.0 (#13760) 2022-09-09 11:14:10 +01:00
stream.py Support filtering the /messages API by relation type (MSC3874). (#14148) 2022-10-17 11:32:11 -04:00
tags.py Revert "Make all process_replication_rows methods async (#13304)" (#13312) 2022-07-18 14:28:14 +01:00
transactions.py Require SQLite >= 3.27.0 (#13760) 2022-09-09 11:14:10 +01:00
ui_auth.py Add some type hints to datastore (#12485) 2022-04-27 13:05:00 +01:00
user_directory.py Wait for lazy join to complete when getting current state (#12872) 2022-06-01 16:02:53 +01:00
user_erasure_store.py