Commit Graph

445 Commits

Author SHA1 Message Date
Damir Jelić
42e363fafc tests: Test pan client syncs. 2019-06-19 11:57:01 +02:00
Damir Jelić
773b708f14 daemon: Remove unnecessary pass. 2019-06-19 11:56:32 +02:00
Damir Jelić
92267f9fc9 tests: Add initial proxy_tests. 2019-06-19 11:22:47 +02:00
Damir Jelić
315c4bb6f7 tests: Add fixtures for proxydaemon tests. 2019-06-19 11:22:11 +02:00
Damir Jelić
2c831de532 config: Set some more default values for the server config. 2019-06-19 11:21:43 +02:00
Damir Jelić
c8d81b9cf4 client: Catch keyboard exceptions for the client tasks. 2019-06-18 16:31:17 +02:00
Damir Jelić
e766b1ae4b config: Make the delay between room history fetching configurable. 2019-06-18 16:30:38 +02:00
Damir Jelić
0ad0da2151 daemon: Add missing await. 2019-06-18 15:35:50 +02:00
Damir Jelić
aafc56b44d daemon: Add CORS headers to the responses.
This add CORS headers to all the responses, they are required for web
based clients (e.g. Riot) to work.
2019-06-18 10:10:25 +02:00
Damir Jelić
f5ffd69a79 config: Make the batch size for room history fetching configurable. 2019-06-17 16:59:54 +02:00
Damir Jelić
0940b67eb8 pantalaimon: Add the ability to index only encrypted rooms.
Since some public rooms can be quite large downloading the room history
for such rooms can be quite resource intensive.

Replicating the whole room history locally for such rooms might be
undesirable. This patch adds the ability to only download room history
and index messages for encrypted rooms.

Do note that the search API supports searching in multiple rooms, thus
if the indexing is restricted to encrypted rooms a search across
multiple rooms will need to make a search request to the server as well
as a local search. This mode is currently unsupported.
2019-06-17 15:23:06 +02:00
Damir Jelić
83f62b0378 config: Make additional Homeserver requests for the search endpoint optional. 2019-06-17 12:26:38 +02:00
Damir Jelić
a6a62434e0 client: Add the context token fetching back for the search. 2019-06-17 10:29:23 +02:00
Damir Jelić
1850ead698 daemon: Sort the imports. 2019-06-14 15:23:41 +02:00
Damir Jelić
fa3e11a1ec index: Catch query parsing errors.
Tantivy's query parser can throw errors if the syntax of the query is
invalid.

There is a more lenient query parser[1] in the works but until then
catch query parser errors and return an error response to the user.

[1] https://github.com/tantivy-search/tantivy/issues/5
2019-06-14 15:15:58 +02:00
Damir Jelić
3a1b001244 index: Rewrite the search logic.
This patch moves all the indexing, event storing and searching into a
separate class.

The index and message store are now represented as a single class and
messages are indexed and stored atomically now which should minimize the
chance of store/index inconsistencies.

Messages are now loaded from the store as a single SQL query and the
context for the messages is as well loaded from the store instead of
fetched from the server.

The room state and start/end tokens for the context aren't currently
loaded.
2019-06-14 14:53:25 +02:00
Damir Jelić
a489031962 client: Fix a style issue. 2019-06-13 13:01:58 +02:00
Damir Jelić
1f5afd75c6 client: Allow per room search.
This is not ideal. We only support a single room or all rooms, which is
good enough for now since riot does that.

The tantivy boolean query logic isn't ideal and didn't work out with
multiple rooms in the query.
2019-06-13 12:32:52 +02:00
Damir Jelić
fe10f54ea9 client: Error out gracefully if we get an invalid limit. 2019-06-13 12:32:21 +02:00
Damir Jelić
ba09f77cb6 client: Simplify the sync logic. 2019-06-12 16:24:03 +02:00
Damir Jelić
6092e27748 client: Pass the sync token to the sync method. 2019-06-12 15:39:43 +02:00
Damir Jelić
b4e60b603a client: Store the history fetcher tasks to the db. 2019-06-12 15:39:08 +02:00
Damir Jelić
1ad2a3af28 daemon: Add access control headers to the search endpoint.
These headers are important for riot to be able to use pan's search
endpoint instead of the default Homeserver endpoint. It fails to search
using pan with CORS errors.
2019-06-12 15:34:08 +02:00
Damir Jelić
a2d6f3cc84 client: Fetch room members before the first encrytped send. 2019-06-12 15:33:27 +02:00
Damir Jelić
383f197e58 client: Store and load sync tokens. 2019-06-11 16:11:53 +02:00
Damir Jelić
2fc738dc20 client: Change the initial sync to contain full state.
Fetching the full state with the initial sync is important because we
don't store any room state locally.

Since we don't store any sync token we are getting the full state
anyways. But once we do this change will be crucial for encryption to
work correctly.
2019-06-11 15:00:35 +02:00
Damir Jelić
8c3cfbc0dd client: Fetch old room messages if there was a limited room timeline.
This patch adds the ability to fetch the whole room history. Note that
this is still incomplete as it not fetch the whole history if it gets
interrupted.

Another deficiency is that it always tries to start fetching room
history after a restart even though we know that it doesn't need to do
so.
2019-06-11 13:59:34 +02:00
Damir Jelić
c7ca5d9851 pantalaimon: Mypy fixes. 2019-06-11 11:24:37 +02:00
Damir Jelić
5b58294a95 pantalaimon: Isort the tree. 2019-06-11 11:24:18 +02:00
Damir Jelić
a164924b28 tests: Fix the event searching tests. 2019-06-10 16:48:40 +02:00
Damir Jelić
1e3d891a57 client: Add the ability to order the search by recency. 2019-06-10 16:12:45 +02:00
Damir Jelić
88dbd2000a index: Change the index format.
Tantivy allows search results to be ordered by arbitrary index fields as
long as they are a fast field and have the FastValue trait implemented.
This trait is only for u64 and i64 fields implemented.

To order the the search results by recency we need to add a unsigned
field that stores the server timestamp of the message.
2019-06-10 15:56:05 +02:00
Damir Jelić
818db73a48 daemon: Handle invalid order by fields for the search. 2019-06-10 15:55:37 +02:00
Damir Jelić
8927bb0c90 client: Fix the search response format. 2019-06-10 15:53:19 +02:00
Damir Jelić
71e672a431 client: Make the room messages callback more specific. 2019-06-10 15:52:22 +02:00
Damir Jelić
1db93a731f client: Add the ability to add the context and state to search results. 2019-06-07 11:59:53 +02:00
Damir Jelić
725c043e87 pantalaimon: Initial search support.
This patch adds support for the Matrix search API endpoint.

Events are stored in the pan sqlite database while the indexing is
handled by tanvity.

An tantivy index is created for each pan user. Currently only ordering
by ranking is supported, and all the search options are ignored for now.
2019-06-06 14:25:14 +02:00
Damir Jelić
9444e540df man: Move the man folder under the docs folder. 2019-05-25 12:47:53 +02:00
Damir Jelić
5718596f77 Merge branch 'matthew/macos' 2019-05-24 22:37:24 +02:00
Matthew Hodgson
d185875d91 typo 2019-05-24 21:34:05 +01:00
Matthew Hodgson
4d268f639a add pan GIF 2019-05-24 21:28:43 +01:00
Matthew Hodgson
043b2fa3df yet more voodoo 2019-05-24 17:41:37 +01:00
Matthew Hodgson
71ecd229ff dark magic runes 2019-05-24 16:53:57 +01:00
Matthew Hodgson
b3b2a35bfd macOS runes 2019-05-24 16:09:12 +01:00
Damir Jelić
6c91466537 man: Add the man pages in the markdown format. 2019-05-24 16:00:26 +02:00
Damir Jelić
6a578c95b6 README: Update the readme to the current pantalaimon version. 2019-05-24 14:00:33 +02:00
Damir Jelić
160fbbaf98 panctl: Add a version option. 2019-05-24 13:48:39 +02:00
Damir Jelić
f8820c0e73 pantalaimon: Add a version option. 2019-05-24 13:48:21 +02:00
Damir Jelić
2a2babee7b man: Add a panctl man page. 2019-05-24 13:36:04 +02:00
Damir Jelić
7ed29e8691 man: Add a daemon man page. 2019-05-24 13:35:49 +02:00