Erik Johnston
ad53fc3cf4
Short circuit when we have delta ids
2017-05-17 09:57:34 +01:00
Erik Johnston
6fa8148ccb
Merge pull request #2228 from matrix-org/erikj/speed_up_get_hosts
...
Speed up get_joined_hosts
2017-05-16 17:40:55 +01:00
Erik Johnston
7c69849a0d
Merge pull request #2227 from matrix-org/erikj/presence_caches
...
Make presence use cached users/hosts in room
2017-05-16 17:40:47 +01:00
Erik Johnston
13f540ef1b
Speed up get_joined_hosts
2017-05-16 16:05:22 +01:00
Erik Johnston
ec5c4499f4
Make presence use cached users/hosts in room
2017-05-16 16:01:43 +01:00
Erik Johnston
f2a5b6dbfd
Speed up get_domain_from_id
2017-05-16 15:59:37 +01:00
Erik Johnston
b8492b6c2f
Merge pull request #2224 from matrix-org/erikj/prefill_state
...
Prefill state caches
2017-05-16 15:50:11 +01:00
Erik Johnston
331570ea6f
Remove spurious merge artifacts
2017-05-16 15:33:07 +01:00
Richard van der Hoff
d648f65aaf
Merge pull request #2218 from matrix-org/rav/event_search_index
...
Add an index to event_search
2017-05-16 13:26:07 +01:00
Erik Johnston
608b5a6317
Take a copy before prefilling, as it may be a frozendict
2017-05-16 12:55:29 +01:00
Erik Johnston
f451b64c8f
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state
2017-05-15 16:09:32 +01:00
Erik Johnston
2c9475b58e
Merge pull request #2221 from psaavedra/sync_timeline_limit_filter_by_name
...
Configurable maximum number of events requested by /sync and /messages
2017-05-15 16:08:46 +01:00
Erik Johnston
d12ae7fd1c
Don't log exceptions for NotRetryingDestination
2017-05-15 15:42:18 +01:00
Pablo Saavedra
224137fcf9
Fixed syntax nits
2017-05-15 16:21:02 +02:00
Erik Johnston
e4435b014e
Update comment
2017-05-15 15:11:30 +01:00
Erik Johnston
871605f4e2
Comments
2017-05-15 15:11:30 +01:00
Erik Johnston
e0d2f6d5b0
Add more granular event send metrics
2017-05-15 15:11:30 +01:00
Erik Johnston
bfbc907cec
Prefill state caches
2017-05-15 15:11:13 +01:00
Pablo Saavedra
627e6ea2b0
Fixed implementation errors
...
* Added HS as property in SyncRestServlet
* Fixed set_timeline_upper_limit function implementat¡ion
2017-05-15 14:51:43 +02:00
Pablo Saavedra
9da4316ca5
Configurable maximum number of events requested by /sync and /messages ( #2220 )
...
Set the limit on the returned events in the timeline in the get and sync
operations. The default value is -1, means no upper limit.
For example, using `filter_timeline_limit: 5000`:
POST /_matrix/client/r0/user/user:id/filter
{
room: {
timeline: {
limit: 1000000000000000000
}
}
}
GET /_matrix/client/r0/user/user:id/filter/filter:id
{
room: {
timeline: {
limit: 5000
}
}
}
The server cuts down the room.timeline.limit.
2017-05-13 18:17:54 +02:00
Richard van der Hoff
ff3d810ea8
Add a comment to old delta
2017-05-11 12:48:50 +01:00
Richard van der Hoff
34194aaff7
Don't create event_search index on sqlite
...
... because the table is virtual
2017-05-11 12:46:55 +01:00
Richard van der Hoff
114f290947
Add more logging for purging
...
Log the number of events we will be deleting at info.
2017-05-11 12:08:47 +01:00
Richard van der Hoff
baafb85ba4
Add an index to event_search
...
- to make the purge API quicker
2017-05-11 12:05:22 +01:00
Richard van der Hoff
29ded770b1
Merge pull request #2214 from matrix-org/rav/hurry_up_purge
...
When purging, don't de-delta state groups we're about to delete
2017-05-11 12:04:25 +01:00
Richard van der Hoff
dc026bb16f
Tidy purge code and add some comments
...
Try to make this clearer with more comments and some variable renames
2017-05-11 10:56:12 +01:00
Slipeer
328378f9cb
Fix users claimed non-exclusively by an app service don't get notifications #2211
2017-05-11 11:42:08 +03:00
Richard van der Hoff
8e345ce465
Don't de-delta state groups we're about to delete
2017-05-10 18:44:22 +01:00
Richard van der Hoff
b64d312421
add some logging to purge_history
2017-05-10 18:44:22 +01:00
Luke Barnard
ccad2ed824
Modify condition on empty localpart
2017-05-10 17:34:30 +01:00
Luke Barnard
369195caa5
Modify register/available to be GET with query param
...
- GET is now the method for register/available
- a query parameter "username" is now used
Also, empty usernames are now handled with an error message on registration or via register/available: `User ID cannot be empty`
2017-05-10 17:23:55 +01:00
Erik Johnston
a3648f84b2
Merge pull request #2208 from matrix-org/erikj/ratelimit_overrid
...
Add per user ratelimiting overrides
2017-05-10 15:54:48 +01:00
Richard van der Hoff
5331cd150a
Merge pull request #2206 from matrix-org/rav/one_time_key_upload_change_sig
...
Allow clients to upload one-time-keys with new sigs
2017-05-10 14:18:40 +01:00
Luke Barnard
f7278e612e
Change register/available to POST (from GET)
2017-05-10 11:40:18 +01:00
Erik Johnston
b990b2fce5
Add per user ratelimiting overrides
2017-05-10 11:05:43 +01:00
Richard van der Hoff
aedaba018f
Replace some instances of preserve_context_over_deferred
2017-05-09 19:04:56 +01:00
Richard van der Hoff
de042b3b88
Do some logging when one-time-keys get claimed
...
might help us figure out if https://github.com/vector-im/riot-web/issues/3868
has happened.
2017-05-09 19:04:56 +01:00
Richard van der Hoff
a7e9d8762d
Allow clients to upload one-time-keys with new sigs
...
When a client retries a key upload, don't give an error if the signature has
changed (but the key is the same).
Fixes https://github.com/vector-im/riot-android/issues/1208 , hopefully.
2017-05-09 19:04:56 +01:00
Erik Johnston
ca238bc023
Merge branch 'release-v0.21.0' of github.com:matrix-org/synapse into develop
2017-05-08 17:35:11 +01:00
Erik Johnston
40dcf0d856
Merge pull request #2203 from matrix-org/erikj/event_cache_hit_ratio
...
Don't update event cache hit ratio from get_joined_users
2017-05-08 16:52:39 +01:00
Erik Johnston
d3c3026496
Merge pull request #2201 from matrix-org/erikj/store_device_cache
...
Cache check to see if device exists
2017-05-08 16:23:04 +01:00
Erik Johnston
093f7e47cc
Expand docstring a bit
2017-05-08 16:14:46 +01:00
Erik Johnston
6a12998a83
Add missing yields
2017-05-08 16:10:51 +01:00
Erik Johnston
ffad4fe35b
Don't update event cache hit ratio from get_joined_users
...
Otherwise the hit ration of plain get_events gets completely skewed by
calls to get_joined_users* functions.
2017-05-08 16:06:17 +01:00
Erik Johnston
94e6ad71f5
Invalidate cache on device deletion
2017-05-08 15:55:59 +01:00
Erik Johnston
8571f864d2
Cache one time key counts
2017-05-08 15:34:27 +01:00
Erik Johnston
fc6d4974a6
Comment
2017-05-08 15:33:57 +01:00
Erik Johnston
738ccf61c0
Cache check to see if device exists
2017-05-08 15:32:18 +01:00
Erik Johnston
dcabef952c
Increase client_ip cache size
2017-05-08 15:09:19 +01:00
Erik Johnston
771c8a83c7
Bump version and changelog
2017-05-08 13:23:46 +01:00
Erik Johnston
6631985990
Merge pull request #2200 from matrix-org/erikj/revert_push
...
Revert speed up push
2017-05-08 13:20:52 +01:00
Erik Johnston
e0f20e9425
Revert "Remove unused import"
...
This reverts commit ab37bef83b
2017-05-08 13:07:43 +01:00
Erik Johnston
fe7c1b969c
Revert "We don't care about forgotten rooms"
...
This reverts commit ad8b316939
2017-05-08 13:07:43 +01:00
Erik Johnston
78f306a6f7
Revert "Speed up filtering of a single event in push"
...
This reverts commit 421fdf7460
2017-05-08 13:07:41 +01:00
Erik Johnston
9ac98197bb
Bump version and changelog
2017-05-08 11:07:54 +01:00
Erik Johnston
27c28eaa27
Merge pull request #2190 from matrix-org/erikj/mark_remote_as_back_more
...
Always mark remotes as up if we receive a signed request from them
2017-05-05 14:08:12 +01:00
Erik Johnston
be2672716d
Merge pull request #2189 from matrix-org/erikj/handle_remote_device_list
...
Handle exceptions thrown in handling remote device list updates
2017-05-05 14:01:27 +01:00
Erik Johnston
653d90c1a5
Comment
2017-05-05 14:01:17 +01:00
Erik Johnston
310b1ccdc1
Use preserve_fn and add logs
2017-05-05 13:41:19 +01:00
Kegsay
a59b0ad1a1
Merge pull request #2192 from matrix-org/kegan/simple-http-client-timeouts
...
Rewrite SimpleHttpClient.request to include timeouts
2017-05-05 11:52:43 +01:00
Erik Johnston
7b222fc56e
Remove redundant reset of destination timers
2017-05-05 11:14:09 +01:00
Kegan Dougal
d0debb2116
Remember how twisted works
2017-05-05 11:00:21 +01:00
Erik Johnston
66f371e8b8
Merge pull request #2176 from matrix-org/erikj/faster_get_joined
...
Make get_joined_users faster
2017-05-05 10:59:55 +01:00
Erik Johnston
b843631d71
Add comment and TODO
2017-05-05 10:59:32 +01:00
Kegan Dougal
c2ddd773bc
Include the clock
2017-05-05 10:52:46 +01:00
Kegan Dougal
7dd3bf5e24
Rewrite SimpleHttpClient.request to include timeouts
...
Fixes #2191
2017-05-05 10:49:19 +01:00
Erik Johnston
db7d0c3127
Always mark remotes as up if we receive a signed request from them
2017-05-05 10:34:53 +01:00
Erik Johnston
f346048a6e
Handle exceptions thrown in handling remote device list updates
2017-05-05 10:34:10 +01:00
Erik Johnston
e3aa8a7aa8
Merge pull request #2185 from matrix-org/erikj/smaller_caches
...
Optimise caches for single key
2017-05-05 10:19:05 +01:00
Erik Johnston
cf589f2c1e
Fixes
2017-05-05 10:17:56 +01:00
Erik Johnston
8af4569583
Merge pull request #2174 from matrix-org/erikj/current_cache_hosts
...
Add cache for get_current_hosts_in_room
2017-05-05 10:15:24 +01:00
Erik Johnston
587f07543f
Revert "Prefill state caches"
2017-05-04 15:07:27 +01:00
Erik Johnston
aa93cb9f44
Add comment
2017-05-04 14:59:28 +01:00
Erik Johnston
537dbadea0
Intern host strings
2017-05-04 14:55:28 +01:00
Erik Johnston
07a07588a0
Make caches bigger
2017-05-04 14:52:28 +01:00
Erik Johnston
dfaa58f72d
Fix comment and num args
2017-05-04 14:50:24 +01:00
Erik Johnston
9ac263ed1b
Add new storage functions to slave store
2017-05-04 14:29:03 +01:00
Erik Johnston
d2d8ed4884
Optimise caches with single key
2017-05-04 14:18:46 +01:00
Erik Johnston
5d8290429c
Reduce size of get_users_in_room
2017-05-04 13:43:19 +01:00
Luke Barnard
6aa423a1a8
Merge pull request #2183 from matrix-org/luke/username-availability
...
Implement username availability checker
2017-05-04 09:58:40 +01:00
Luke Barnard
3669065466
Appease the flake8 gods
2017-05-03 18:05:49 +01:00
Erik Johnston
7ebf518c02
Make get_joined_users faster
2017-05-03 15:55:54 +01:00
Luke Barnard
34ed4f4206
Implement username availability checker
...
Outlined here: https://github.com/vector-im/riot-web/issues/3605#issuecomment-298679388
```HTTP
GET /_matrix/.../register/available
{
"username": "desiredlocalpart123"
}
```
If available, the response looks like
```HTTP
HTTP/1.1 200 OK
{
"available": true
}
```
Otherwise,
```HTTP
HTTP/1.1 429
{
"errcode": "M_LIMIT_EXCEEDED",
"error": "Too Many Requests",
"retry_after_ms": 2000
}
```
or
```HTTP
HTTP/1.1 400
{
"errcode": "M_USER_IN_USE",
"error": "User ID already taken."
}
```
or
```HTTP
HTTP/1.1 400
{
"errcode": "M_INVALID_USERNAME",
"error": "Some reason for username being invalid"
}
```
2017-05-03 12:04:12 +01:00
David Baker
60833c8978
Merge pull request #2147 from matrix-org/dbkr/http_request_propagate_error
...
Propagate errors sensibly from proxied IS requests
2017-05-03 11:23:25 +01:00
David Baker
482a2ad122
No need for the exception variable
2017-05-03 11:02:59 +01:00
David Baker
c0380402bc
List caught expection types
2017-05-03 10:56:22 +01:00
Erik Johnston
cdbf38728d
Merge pull request #2175 from matrix-org/erikj/prefill_state
...
Prefill state caches
2017-05-03 10:54:11 +01:00
Erik Johnston
0c27383dd7
Merge pull request #2170 from matrix-org/erikj/fed_hole_state
...
Don't fetch state for missing events that we fetched
2017-05-03 10:49:37 +01:00
Erik Johnston
ef862186dd
Merge together redundant calculations/logging
2017-05-03 10:06:43 +01:00
Erik Johnston
2c2dcf81d0
Update comment
2017-05-03 10:00:29 +01:00
Erik Johnston
1827057acc
Comments
2017-05-03 09:56:05 +01:00
Erik Johnston
8346e6e696
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/prefill_state
2017-05-03 09:46:40 +01:00
Erik Johnston
e4c15fcb5c
Merge pull request #2178 from matrix-org/erikj/message_metrics
...
Add more granular event send metrics
2017-05-02 17:57:34 +01:00
Erik Johnston
3e5a62ecd8
Add more granular event send metrics
2017-05-02 14:23:26 +01:00
Richard van der Hoff
82475a18d9
Merge pull request #2180 from matrix-org/rav/fix_timeout_on_timeout
...
Instantiate DeferredTimedOutError correctly
2017-05-02 13:32:58 +01:00
Richard van der Hoff
2e996271fe
Instantiate DeferredTimedOutError correctly
...
Call `super` correctly, so that we correctly initialise the `errcode` field.
Fixes https://github.com/matrix-org/synapse/issues/2179 .
2017-05-02 13:26:17 +01:00
Erik Johnston
a2c89a225c
Prefill state caches
2017-05-02 10:40:31 +01:00
Erik Johnston
7166854f41
Add cache for get_current_hosts_in_room
2017-05-02 10:36:35 +01:00
Erik Johnston
3033261891
Merge pull request #2080 from matrix-org/erikj/filter_speed
...
Speed up filtering of a single event in push
2017-04-28 14:17:13 +01:00
Erik Johnston
2347efc065
Fixup
2017-04-28 12:46:53 +01:00
Erik Johnston
9b147cd730
Remove unncessary call in _get_missing_events_for_pdu
2017-04-28 11:55:25 +01:00
Erik Johnston
3a9f5bf6dd
Don't fetch state for missing events that we fetched
2017-04-28 11:26:46 +01:00
Erik Johnston
ab37bef83b
Remove unused import
2017-04-28 09:57:23 +01:00
Erik Johnston
ad8b316939
We don't care about forgotten rooms
2017-04-28 09:52:36 +01:00
Erik Johnston
421fdf7460
Speed up filtering of a single event in push
2017-04-28 09:52:36 +01:00
Erik Johnston
25a96e0c63
Merge pull request #2163 from matrix-org/erikj/fix_invite_state
...
Fix invite state to always include all events
2017-04-27 17:36:30 +01:00
Erik Johnston
46826bb078
Comment and remove spurious logging
2017-04-27 17:25:44 +01:00
Erik Johnston
f87b287291
Merge pull request #2127 from APwhitehat/alreadystarted
...
print something legible if synapse already running
2017-04-27 15:46:53 +01:00
Erik Johnston
bb9246e525
Merge pull request #2131 from matthewjwolff/develop
...
web_client_location documentation fix
2017-04-27 15:46:40 +01:00
Richard van der Hoff
c84770b877
Fix bgupdate error if index already exists ( #2167 )
...
When creating a new table index in the background, guard against it existing already. Fixes
https://github.com/matrix-org/synapse/issues/2135 .
Also, make sure we restore the autocommit flag when we're done, otherwise we
get more failures from other operations later on. Fixes
https://github.com/matrix-org/synapse/issues/1890 (hopefully).
2017-04-27 15:27:48 +01:00
Erik Johnston
380fb87ecc
Merge pull request #2168 from matrix-org/erikj/federation_logging
...
Add some extra logging for edge cases of federation
2017-04-27 15:19:12 +01:00
Erik Johnston
87ae59f5e9
Typo
2017-04-27 15:16:21 +01:00
Erik Johnston
e42b4ebf0f
Add some extra logging for edge cases of federation
2017-04-27 14:38:21 +01:00
Erik Johnston
d3c150411c
Merge pull request #2130 from APwhitehat/roomexists
...
Check that requested room_id exists
2017-04-27 09:20:26 +01:00
Erik Johnston
1e166470ab
Fix tests
2017-04-26 16:23:30 +01:00
Erik Johnston
34e682d385
Fix invite state to always include all events
2017-04-26 16:18:08 +01:00
David Baker
5fd12dce01
Remove debugging
2017-04-26 12:36:26 +01:00
David Baker
82ae0238f9
Revert accidental commit
2017-04-26 11:43:16 +01:00
David Baker
81804909d3
Merge remote-tracking branch 'origin/develop' into dbkr/http_request_propagate_error
2017-04-26 11:31:55 +01:00
David Baker
c366276056
Fix get_json
2017-04-26 10:07:01 +01:00
David Baker
1a9255c12e
Use CodeMessageException subclass instead
...
Parse json errors from get_json client methods and throw special
errors.
2017-04-25 19:30:55 +01:00
Erik Johnston
f053a1409e
Make state caches cache in ascii
2017-04-25 17:22:55 +01:00
Erik Johnston
acb58bfb6a
fix up
2017-04-25 15:39:19 +01:00
Erik Johnston
f7181615f2
Don't specify default as dict
2017-04-25 15:22:59 +01:00
Erik Johnston
f144365281
Comment
2017-04-25 15:18:26 +01:00
Erik Johnston
d9aa645f86
Reduce size of joined_user cache
...
The _get_joined_users_from_context cache stores a mapping from user_id
to avatar_url and display_name. Instead of storing those in a dict,
store them in a namedtuple as that uses much less memory.
We also try converting the string to ascii to further reduce the size.
2017-04-25 14:38:51 +01:00
Erik Johnston
22f3d3ae76
Reduce _get_state_group_for_event cache size
2017-04-25 11:43:03 +01:00
Erik Johnston
b4da08cad8
Merge pull request #2158 from matrix-org/erikj/reduce_cache_size
...
Reduce cache size by not storing deferreds
2017-04-25 11:18:07 +01:00
Erik Johnston
efab1dadde
Remove DEBUG_CACHES
2017-04-25 10:54:09 +01:00
Erik Johnston
119cb9bbcf
Reduce cache size by not storing deferreds
...
Currently the cache descriptors store deferreds rather than raw values,
this is a simple way of triggering only one database hit and sharing the
result if two callers attempt to get the same value.
However, there are a few caches that simply store a mapping from string
to string (or int). These caches can have a large number of entries,
under the assumption that each entry is small. However, the size of a
deferred (specifically the size of ObservableDeferred) is signigicantly
larger than that of the raw value, 2kb vs 32b.
This PR therefore changes the cache descriptors to store the raw values
rather than the deferreds.
As a side effect cached storage function now either return a deferred or
the actual value, as the cached list decriptor already does. This is
fine as we always end up just yield'ing on the returned value
eventually, which handles that case correctly.
2017-04-25 10:23:11 +01:00
Mark Haines
e6e2627636
Fix code for reporting old verify keys in synapse
2017-04-24 18:51:25 +01:00
Richard van der Hoff
30f7bfa121
Merge pull request #2145 from matrix-org/rav/reject_invite_to_unreachable_server
...
Fix rejection of invites to unreachable servers
2017-04-24 15:20:52 +01:00
Erik Johnston
7af825bae4
Merge pull request #2155 from matrix-org/erikj/string_intern
...
Only intern ascii strings
2017-04-24 14:28:41 +01:00
Erik Johnston
d134d0935e
Only intern ascii strings
2017-04-24 14:07:48 +01:00
Erik Johnston
e4f3431116
Remove unused cache
2017-04-24 13:27:38 +01:00
David Baker
a46982cee9
Need the HTTP status code
2017-04-21 16:20:12 +01:00
David Baker
70caf49914
Do the same for get_json
2017-04-21 16:09:03 +01:00
David Baker
a1595cec78
Don't error for 3xx responses
2017-04-21 11:51:17 +01:00
David Baker
2e165295b7
Merge remote-tracking branch 'origin/develop' into dbkr/http_request_propagate_error
2017-04-21 11:35:52 +01:00
David Baker
a90a0f5c8a
Propagate errors sensibly from proxied IS requests
...
When we're proxying Matrix endpoints, parse out Matrix error
responses and turn them into SynapseErrors so they can be
propagated sensibly upstream.
2017-04-21 11:32:48 +01:00
Richard van der Hoff
91b3981800
Try harder when sending leave events
...
When we're rejecting invites, ignore the backoff data, so that we have a better
chance of not getting the room out of sync.
2017-04-21 01:50:36 +01:00
Richard van der Hoff
0cdb32fc43
Remove redundant try/except clauses
...
The `except SynapseError` clauses were pointless because the wrapped functions
would never throw a `SynapseError` (they either throw a `CodeMessageException`
or a `RuntimeError`).
The `except CodeMessageException` is now also pointless because the caller
treats all exceptions equally, so we may as well just throw the
`CodeMessageException`.
2017-04-21 01:32:01 +01:00
Richard van der Hoff
838810b76a
Broaden the conditions for locally_rejecting invites
...
The logic for marking invites as locally rejected was all well and good, but
didn't happen when the remote server returned a 500, or wasn't reachable, or
had no DNS, or whatever.
Just expand the except clause to catch everything.
Fixes https://github.com/matrix-org/synapse/issues/761 .
2017-04-21 01:31:37 +01:00
Richard van der Hoff
736b9a4784
Remove redundant function
...
inline `reject_remote_invite`, which only existed to make tracing the callflow
more difficult.
2017-04-21 01:31:09 +01:00
Richard van der Hoff
4903ccf159
Fix some lies, and other clarifications, in docstrings
...
The documentation on get_json has been wrong ever since the very first commit
to synapse...
2017-04-21 01:31:09 +01:00
Matthew Wolff
d4040e9e28
Queried CONDITIONAL_REQUIREMENTS
2017-04-18 16:19:48 -05:00
Luke Barnard
3fb8784c92
m.read_marker -> m.fully_read ( #2128 )
...
Also:
- change the REST endpoint to have a "S" on the end (so it's now /read_markers)
- change the content of the m.read_up_to event to have the key "event_id" instead of "marker".
2017-04-18 17:46:15 +01:00
Matthew Wolff
54f9a4cb59
Fixed travis build failure
...
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
2017-04-17 01:38:27 -05:00
Matthew Wolff
8e780b113d
web_server_root documentation fix
...
Signed-off-by: Matthew Wolff <matthewjwolff@gmail.com>
2017-04-17 00:49:11 -05:00
Anant Prakash
574d573ac2
Check that requested room_id exists
2017-04-14 23:50:59 +05:30
