This patch disables the message indexer an message store if
tantivy is not installed.
The search API endpoint forwards all search requests to the homeserver.
The device list in the UI thread is replicated so we can show UI clients
the list without the need for a lock.
The previous implementation relied on loading and reloading of the device
list from the store every time an event changed either the devices or their
trust state.
This leads to a couple of ineficiencies leading to timeouts while
waiting on the database lock if a user has a large number of devices.
The new implementation never loads devices in the UI thread from the
database, they get passed through the thread queue by the main thread
which already holds them in memory.
Since some public rooms can be quite large downloading the room history
for such rooms can be quite resource intensive.
Replicating the whole room history locally for such rooms might be
undesirable. This patch adds the ability to only download room history
and index messages for encrypted rooms.
Do note that the search API supports searching in multiple rooms, thus
if the indexing is restricted to encrypted rooms a search across
multiple rooms will need to make a search request to the server as well
as a local search. This mode is currently unsupported.
Tantivy's query parser can throw errors if the syntax of the query is
invalid.
There is a more lenient query parser[1] in the works but until then
catch query parser errors and return an error response to the user.
[1] https://github.com/tantivy-search/tantivy/issues/5
This patch moves all the indexing, event storing and searching into a
separate class.
The index and message store are now represented as a single class and
messages are indexed and stored atomically now which should minimize the
chance of store/index inconsistencies.
Messages are now loaded from the store as a single SQL query and the
context for the messages is as well loaded from the store instead of
fetched from the server.
The room state and start/end tokens for the context aren't currently
loaded.
This is not ideal. We only support a single room or all rooms, which is
good enough for now since riot does that.
The tantivy boolean query logic isn't ideal and didn't work out with
multiple rooms in the query.
These headers are important for riot to be able to use pan's search
endpoint instead of the default Homeserver endpoint. It fails to search
using pan with CORS errors.
Fetching the full state with the initial sync is important because we
don't store any room state locally.
Since we don't store any sync token we are getting the full state
anyways. But once we do this change will be crucial for encryption to
work correctly.
This patch adds the ability to fetch the whole room history. Note that
this is still incomplete as it not fetch the whole history if it gets
interrupted.
Another deficiency is that it always tries to start fetching room
history after a restart even though we know that it doesn't need to do
so.
Tantivy allows search results to be ordered by arbitrary index fields as
long as they are a fast field and have the FastValue trait implemented.
This trait is only for u64 and i64 fields implemented.
To order the the search results by recency we need to add a unsigned
field that stores the server timestamp of the message.
This patch adds support for the Matrix search API endpoint.
Events are stored in the pan sqlite database while the indexing is
handled by tanvity.
An tantivy index is created for each pan user. Currently only ordering
by ranking is supported, and all the search options are ignored for now.
