Erik Johnston
2284eb3a53
Add database config class ( #6513 )
...
This encapsulates config for a given database and is the way to get new
connections.
2019-12-18 10:45:12 +00:00
Erik Johnston
5859a5c569
Fix presence timeouts when synchrotron restarts. ( #6212 )
...
* Fix presence timeouts when synchrotron restarts.
Handling timeouts would fail if there was an external process that had
timed out, e.g. a synchrotron restarting. This was due to a couple of
variable name typoes.
Fixes #3715 .
2019-10-18 06:42:26 +01:00
Richard van der Hoff
a139420a3c
Fix races in room stats (and other) updates. ( #6187 )
...
Hopefully this will fix the occasional failures we were seeing in the room directory.
The problem was that events are not necessarily persisted (and `current_state_delta_stream` updated) in the same order as their stream_id. So for instance current_state_delta 9 might be persisted *before* current_state_delta 8. Then, when the room stats saw stream_id 9, it assumed it had done everything up to 9, and never came back to do stream_id 8.
We can solve this easily by only processing up to the stream_id where we know all events have been persisted.
2019-10-10 11:29:01 +01:00
Matthew Hodgson
8401bcd206
fix typo
2019-09-03 12:44:14 +01:00
Andrew Morgan
4548d1f87e
Remove unnecessary parentheses around return statements ( #5931 )
...
Python will return a tuple whether there are parentheses around the returned values or not.
I'm just sick of my editor complaining about this all over the place :)
2019-08-30 16:28:26 +01:00
Amber Brown
97a8b4caf7
Move some timeout checking logs to DEBUG #5785
2019-07-30 02:02:18 +10:00
Amber Brown
4806651744
Replace returnValue with return ( #5736 )
2019-07-23 23:00:55 +10:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ ( #5606 )
2019-07-04 00:07:04 +10:00
Erik Johnston
915280f1ed
Fixup comment
2019-07-01 10:22:42 +01:00
Erik Johnston
8181e290a9
Fix sync tightloop bug.
...
If, for some reason, presence updates take a while to persist then it
can trigger clients to tightloop calling `/sync` due to the presence
handler returning updates but not advancing the stream token.
Fixes #5503 .
2019-06-21 11:10:27 +01:00
Amber Brown
32e7c9e7f2
Run Black. ( #5482 )
2019-06-20 19:32:02 +10:00
Richard van der Hoff
71063a69b8
Fix missing logcontext for PresenceHandler.on_shutdown. ( #5369 )
...
Fixes some warnings, and a scary-looking stacktrace when sytest kills the
process.
2019-06-06 14:45:17 +01:00
Erik Johnston
75538813fc
Fix background updates to handle redactions/rejections ( #5352 )
...
* Fix background updates to handle redactions/rejections
In background updates based on current state delta stream we need to
handle that we may not have all the events (or at least that
`get_events` may raise an exception).
2019-06-06 00:45:46 +10:00
Richard van der Hoff
5726378ece
Fix "db txn 'update_presence' from sentinel context" log messages ( #5275 )
...
Fixes #4414 .
2019-05-28 21:20:11 +01:00
Richard van der Hoff
bd0d45ca69
Fix infinite loop in presence handler
...
Fixes #5102
2019-04-26 11:14:49 +01:00
Andrew Morgan
4a4d5c4fd6
Fix grammar and document get_current_users_in_room ( #4998 )
2019-04-03 14:32:20 +01:00
Neil Johnson
e8419554ff
Remove presence lists ( #4989 )
...
Remove presence list support as per MSC 1819
2019-04-03 11:11:15 +01:00
Erik Johnston
4e5f0f7ca0
Use an assert
2019-03-28 14:05:05 +00:00
Erik Johnston
40e56997bc
Review comments
2019-03-28 13:48:41 +00:00
Erik Johnston
197fae1639
Use event streams to calculate presence
...
Primarily this fixes a bug in the handling of remote users joining a
room where the server sent out the presence for all local users in the
room to all servers in the room.
We also change to using the state delta stream, rather than the
distributor, as it will make it easier to split processing out of the
master process (as well as being more flexible).
Finally, when sending presence states to newly joined servers we filter
out old presence states to reduce the number sent. Initially we filter
out states that are offline and have a last active more than a week ago,
though this can be changed down the line.
Fixes #3962
2019-03-27 13:41:36 +00:00
Richard van der Hoff
856c83f5f8
Avoid rebuilding Edu objects in worker mode ( #4770 )
...
In worker mode, on the federation sender, when we receive an edu for sending
over the replication socket, it is parsed into an Edu object. There is no point
extracting the contents of it so that we can then immediately build another Edu.
2019-03-04 12:57:44 +00:00
Amber Brown
c334ca67bb
Integrate presence from hotfixes ( #3694 )
2018-08-18 01:08:45 +10:00
Amber Brown
99dd975dae
Run tests under PostgreSQL ( #3423 )
2018-08-13 16:47:46 +10:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 ( #3678 )
2018-08-10 23:50:21 +10:00
Amber Brown
49af402019
run isort
2018-07-09 16:09:20 +10:00
Amber Brown
07cad26d65
Remove all global reactor imports & pass it around explicitly ( #3424 )
2018-06-25 14:08:28 +01:00
Amber Brown
c936a52a9e
Consistently use six's iteritems and wrap lazy keys/values in list() if they're not meant to be lazy ( #3307 )
2018-05-31 19:03:47 +10:00
Amber Brown
754826a830
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-28 18:57:23 +10:00
Amber Brown
1f69693347
Merge pull request #3244 from NotAFile/py3-six-4
...
replace some iteritems with six
2018-05-24 13:04:07 -05:00
Amber Brown
53cc2cde1f
cleanup
2018-05-22 17:32:57 -05:00
Amber Brown
8f5a688d42
cleanups, self-registration
2018-05-22 10:56:03 -05:00
Amber Brown
a8990fa2ec
Merge remote-tracking branch 'origin/develop' into 3218-official-prom
2018-05-22 10:50:26 -05:00
Richard van der Hoff
8810685df9
Stub out ServerNoticesSender on the workers
...
... and have the sync endpoints call it directly rather than obsure indirection
via PresenceHandler
2018-05-22 11:54:51 +01:00
Richard van der Hoff
9ea219c514
Send users a server notice about consent
...
When a user first syncs, we will send them a server notice asking them to
consent to the privacy policy if they have not already done so.
2018-05-22 11:54:51 +01:00
Amber Brown
df9f72d9e5
replacing portions
2018-05-21 19:47:37 -05:00
Adrian Tschira
933bf2dd35
replace some iteritems with six
...
Signed-off-by: Adrian Tschira <nota@notafile.com>
2018-05-19 17:59:26 +02:00
Richard van der Hoff
d43d480d86
Remove unused update_external_syncs
...
This method isn't used anywhere. Burninate it.
2018-05-17 18:22:19 +01:00
Richard van der Hoff
9255a6cb17
Improve exception handling for background processes
...
There were a bunch of places where we fire off a process to happen in the
background, but don't have any exception handling on it - instead relying on
the unhandled error being logged when the relevent deferred gets
garbage-collected.
This is unsatisfactory for a number of reasons:
- logging on garbage collection is best-effort and may happen some time after
the error, if at all
- it can be hard to figure out where the error actually happened.
- it is logged as a scary CRITICAL error which (a) I always forget to grep for
and (b) it's not really CRITICAL if a background process we don't care about
fails.
So this is an attempt to add exception handling to everything we fire off into
the background.
2018-04-27 11:07:40 +01:00
Erik Johnston
265b993b8a
Split replication layer into two
2018-03-13 10:55:47 +00:00
Erik Johnston
c3f79c9da5
Split out edu/query registration to a separate class
2018-03-13 10:24:27 +00:00
Matthew Hodgson
34c3d0a386
typo
2017-11-17 01:54:02 +00:00
Richard van der Hoff
eaaabc6c4f
replace 'except:' with 'except Exception:'
...
what could possibly go wrong
2017-10-23 15:52:32 +01:00
Erik Johnston
ec5c4499f4
Make presence use cached users/hosts in room
2017-05-16 16:01:43 +01:00
Erik Johnston
c7ddb5ef7a
Reuse get_interested_parties
2017-04-12 10:16:26 +01:00
Erik Johnston
414522aed5
Move get_interested_parties
2017-04-11 15:33:26 +01:00
Erik Johnston
2be8a281d2
Comments
2017-04-11 15:28:24 +01:00
Erik Johnston
6308ac45b0
Move get_interested_remotes back to presence handler
2017-04-11 15:19:26 +01:00
Erik Johnston
29574fd5b3
Reduce federation presence replication traffic
...
This is mainly done by moving the calculation of where to send presence
updates from the presence handler to the transaction queue, so we only
need to send the presence event (and not the destinations) across the
replication connection. Before we were duplicating by sending the full
state across once per destination.
2017-04-10 16:48:30 +01:00
Erik Johnston
877c029c16
Use iteritems
2017-04-06 15:51:22 +01:00
Erik Johnston
36d2b66f90
Add a timestamp to USER_SYNC command
...
This timestamp is used to indicate when the user last sync'd
2017-03-31 15:42:22 +01:00
Erik Johnston
9d0170ac6c
Fix up presence
2017-03-31 11:36:32 +01:00
Erik Johnston
63fcc42990
Remove user from process_presence when stops syncing
2017-03-30 14:26:08 +01:00
Erik Johnston
e9dd8370b0
Add functions to presence to support remote syncs
...
The TCP replication protocol streams deltas of who has started or
stopped syncing. This is different from the HTTP API which periodically
sends the full list of users who are syncing. This commit adds support
for the new TCP style of sending deltas.
2017-03-30 13:25:14 +01:00
Erik Johnston
7fc1f1e2b6
Cache hosts in room
2017-03-24 11:46:24 +00:00
Erik Johnston
6957bfdca6
Don't recreate so many sets
2017-03-16 11:54:26 +00:00
Erik Johnston
54d2b7e596
Merge pull request #2014 from Half-Shot/hs/fix-appservice-presence
...
Add fallback to last_active_ts if it beats the last sync time on a presence timeout.
2017-03-15 17:37:15 +00:00
Erik Johnston
f83ac78201
Cache set of users whose presence the other user should see
2017-03-15 15:29:19 +00:00
Will Hunt
e6032054bf
Add a great comment to handle_timeout for active vs sync times.
2017-03-15 15:24:48 +00:00
Will Hunt
ebf5a6b14c
Add fallback to last_active_ts if it beats the last sync time.
2017-03-15 15:17:16 +00:00
Erik Johnston
e892457a03
Comment
2017-03-15 15:01:39 +00:00
Erik Johnston
6c82de5100
Format presence events on the edges instead of reformatting them multiple times
2017-03-15 14:27:34 +00:00
Erik Johnston
9e617cd4c2
Cache get_presence storage
2017-02-13 13:50:03 +00:00
Erik Johnston
54a79c1d37
Make presence.get_new_events a bit faster
...
We do this by caching the set of users a user shares rooms with.
2017-02-02 13:07:18 +00:00
Erik Johnston
fd1c18c088
Use DB cache of joined users for presence
2017-01-30 17:00:24 +00:00
Erik Johnston
59ef517e6b
Use new federation_sender DI
2016-11-16 14:47:52 +00:00
Paul "LeoNerd" Evans
ca35e54d6b
Fix typo "persiting"
2016-09-13 13:26:33 +01:00
Erik Johnston
52b2318777
Clobber EDUs in send queue
2016-09-09 15:59:08 +01:00
Erik Johnston
6c4d582144
Deduplicate presence in _update_states
2016-09-09 14:28:22 +01:00
Erik Johnston
8b93af662d
Check the user_id for presence/typing matches origin
2016-09-08 15:07:38 +01:00
Erik Johnston
3c4208a057
Record counts of state changes
2016-09-06 11:31:01 +01:00
Erik Johnston
438ef47637
Short circuit if presence is the same
2016-09-06 10:28:35 +01:00
Erik Johnston
74a3b4a650
Fiddle should_notify to better report stats
2016-09-06 10:23:38 +01:00
Erik Johnston
69054e3d4c
Record why we have chosen to notify
2016-09-05 14:12:11 +01:00
Erik Johnston
265d847ffd
Fix typo in log line
2016-09-01 14:50:06 +01:00
Erik Johnston
c7f665d700
Merge pull request #1055 from matrix-org/erikj/occaisonally_persist
...
Occaisonally persist unpersisted presence updates
2016-08-30 15:59:00 +01:00
Erik Johnston
097330bae8
Check correct variable
2016-08-30 15:50:20 +01:00
Erik Johnston
21b977ccfe
Occaisonally persist unpersisted presence updates
2016-08-30 15:39:50 +01:00
Erik Johnston
bc1a8b1f7a
Don't notify for online -> online transitions.
...
Specifically, if currently_active remains true then we should not notify
if only the last active time changes.
2016-08-30 15:05:32 +01:00
Erik Johnston
bed10f9880
Use state handler instead of get_users_in_room/get_joined_hosts
2016-08-26 14:54:30 +01:00
Erik Johnston
4e1cebd56f
Make synchrotron accept /events
2016-08-12 15:31:44 +01:00
Will Hunt
2510db3e76
Don't change status_msg on /sync
2016-08-10 12:59:59 +01:00
Erik Johnston
216a05b3e3
.values() returns list of sets
2016-06-06 16:00:09 +01:00
Erik Johnston
96dc600579
Fix typos
2016-06-06 15:44:41 +01:00
Erik Johnston
377eb480ca
Fire after 30s not 8h
2016-06-06 15:14:21 +01:00
Erik Johnston
4ce84a1acd
Change metric style
2016-06-03 13:49:16 +01:00
Erik Johnston
a7ff5a1770
Presence metrics. Change def of small delta
2016-06-03 13:40:55 +01:00
Mark Haines
70599ce925
Allow external processes to mark a user as syncing. ( #812 )
...
* Add infrastructure to the presence handler to track sync requests in external processes
* Expire stale entries for dead external processes
* Add an http endpoint for making users as syncing
Add some docstrings and comments.
* Fixes
2016-06-02 15:20:15 +01:00
Mark Haines
b153f5b150
Merge pull request #787 from matrix-org/markjh/liberate_presence_handler
...
Move the presence handler out of the Handlers object
2016-05-17 10:09:43 +01:00
Mark Haines
1a3a2002ff
Spell "domain" correctly
...
s/domian/domain/g
2016-05-16 19:17:23 +01:00
Mark Haines
e168abbcff
Don't inherit PresenceHandler from BaseHandler, remove references to self.hs from presence handler
2016-05-16 19:08:40 +01:00
Mark Haines
3b86ecfa79
Move the presence handler out of the Handlers object
2016-05-16 18:56:37 +01:00
Erik Johnston
08dfa8eee2
Add and use get_domian_from_id
2016-05-09 10:36:03 +01:00
Erik Johnston
56b5e83e36
Reduce database inserts when sending transactions
2016-05-06 11:20:18 +01:00
Mark Haines
7a38612620
Remove another unused function from presence
2016-03-17 11:54:19 +00:00
Mark Haines
673c96ce97
Remove dead code left over from presence changes
2016-03-17 11:03:47 +00:00
Erik Johnston
2a78dac60d
Merge pull request #612 from matrix-org/erikj/cache_size
...
Add environment variable SYNAPSE_CACHE_FACTOR, default it to 0.1
2016-03-02 14:36:10 +00:00
Mark Haines
60a0f81c7a
Add a /replication API for extracting the updates that happened on
...
synapse
This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.
This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.
Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.
Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.
The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
Erik Johnston
910fc0f28f
Add enviroment variable SYNAPSE_CACHE_FACTOR, default it to 0.1
2016-03-01 12:56:39 +00:00
Erik Johnston
b5f77eb12a
Check presence token interval is less than 100, rather than the token itself
2016-02-23 15:47:37 +00:00
Erik Johnston
e3e0ac6ec7
Merge pull request #602 from matrix-org/erikj/presence
...
Change the way we figure out presence updates for small deltas
2016-02-23 15:18:41 +00:00
Erik Johnston
13f86c3489
Handle get_all_entities_changed returning None
2016-02-23 15:05:37 +00:00
Erik Johnston
6e0209112b
Add comments
2016-02-23 14:57:45 +00:00
Erik Johnston
c77dae7a1a
Change the way we figure out presence updates for small deltas
2016-02-23 14:54:40 +00:00
Erik Johnston
a7b2ce32f7
Merge pull request #600 from matrix-org/erikj/presence
...
Measure PresenceEventSource.get_new_events
2016-02-23 14:54:26 +00:00
Erik Johnston
02e928cf9b
Don't include defer.returnValue in Measure block
2016-02-23 14:52:16 +00:00
Erik Johnston
56a94ccd9e
Measure PresenceEventSource.get_new_events
2016-02-23 14:52:16 +00:00
Erik Johnston
138c405974
Pick up currently_active across federation
2016-02-23 10:40:11 +00:00
Erik Johnston
3dbaeef58c
Correctly filter states
2016-02-19 12:27:35 +00:00
Erik Johnston
42ac5f0c1a
Only send presence updates to remote hosts if user is ours
2016-02-19 12:19:56 +00:00
Erik Johnston
24d9f2c140
Add Measures to presence
2016-02-19 11:50:48 +00:00
Erik Johnston
4a95eb0a12
Add presence metric
2016-02-19 11:32:04 +00:00
Erik Johnston
be799453aa
Remove spurious extra metrics
2016-02-19 11:29:33 +00:00
Erik Johnston
929cb0ed7d
Don't set currently_active for remote presence
2016-02-19 10:58:27 +00:00
Erik Johnston
5f4eca3816
More metrics
2016-02-19 10:21:41 +00:00
Erik Johnston
5614b4dafb
Add presence metrics
2016-02-19 09:50:54 +00:00
Erik Johnston
48b652bcbe
Remove invalid arg.
2016-02-18 14:57:09 +00:00
Erik Johnston
b4796a62ee
Add unit test
2016-02-18 11:52:33 +00:00
Erik Johnston
fe95f2217c
Add stuff pulled from the DB to the cache
2016-02-18 10:26:24 +00:00
Erik Johnston
8351538873
PEP8
2016-02-18 10:12:12 +00:00
Erik Johnston
112283e230
Prefix TS fields with _ts
2016-02-18 10:11:43 +00:00
Erik Johnston
b31ec214a5
Remove status_msg when going offline. Don't offline -> online if you send a message
2016-02-18 09:54:08 +00:00
Erik Johnston
114b929f8b
Check presence state is a valid one
2016-02-18 09:16:32 +00:00
Erik Johnston
ddca9c56fc
Move if statement
2016-02-18 09:11:53 +00:00
Erik Johnston
58371fa263
Comment
2016-02-18 09:09:50 +00:00
Erik Johnston
c229c87398
Remove spurious comment
2016-02-17 15:48:29 +00:00
Erik Johnston
e5999bfb1a
Initial cut
2016-02-17 15:40:50 +00:00
Erik Johnston
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
Matthew Hodgson
6c28ac260c
copyrights
2016-01-07 04:26:29 +00:00
Mark Haines
a9526831a4
Wrap calls to distributor.fire in appropriately named functions so that static analysis can work out want is calling what
2015-12-01 20:53:04 +00:00
Mark Haines
31069ecf6a
Rename presence_handler.send_invite to presence_handler.send_presence_invite to distinguish it from normal invites
2015-12-01 15:59:45 +00:00
Daniel Wagner-Hall
38d82edf0e
Allow guest users to join and message rooms
2015-11-10 16:57:13 +00:00
Daniel Wagner-Hall
ca2f90742d
Open up /events to anonymous users for room events only
...
Squash-merge of PR #345 from daniel/anonymousevents
2015-11-05 14:32:26 +00:00
Mark Haines
987803781e
Fix some races in the synapse presence handler caused by not yielding on deferreds
2015-10-09 19:59:50 +01:00
Erik Johnston
0d4abf7777
Typo
2015-08-18 11:19:08 +01:00
Erik Johnston
e55291ce5e
None check
2015-08-18 11:17:37 +01:00
Erik Johnston
83eb627b5a
More helpful variable names
2015-08-18 10:33:11 +01:00
Erik Johnston
776ee6d92b
Doc strings
2015-08-18 10:30:07 +01:00
Erik Johnston
f72ed6c6a3
Remove debug try/catch
2015-08-18 10:29:49 +01:00
Erik Johnston
1a9510bb84
Implement a batched presence_handler.get_state and use it
2015-08-17 10:40:23 +01:00
Erik Johnston
bd1236c0ee
Consolidate duplicate code in notifier
2015-07-02 11:46:05 +01:00
Erik Johnston
0460406298
Don't do unecessary db ops in presence.get_state
2015-06-16 16:59:38 +01:00
Mark Haines
1a9a9abcc7
Add a cache for getting the presence list for a user
2015-05-22 16:11:17 +01:00
Mark Haines
106a3051b8
Remove spurious TODO comment
2015-05-22 15:53:03 +01:00
Mark Haines
7ae8afb7ef
Removed unused 'is_visible' method
2015-05-20 14:48:11 +01:00
Mark Haines
e01b825cc9
Clean up the presence_list checking logic a bit
2015-05-20 13:21:59 +01:00
Mark Haines
e4c65b338d
Speed up the get_pagination_rows as well
2015-05-18 18:21:06 +01:00
Mark Haines
591c4bf223
Cache the most recent serial for each room
2015-05-18 16:21:51 +01:00
Mark Haines
e1150cac4b
Move updating the serial and state of the presence cache into a single function
2015-05-18 15:46:37 +01:00
Mark Haines
880fb46de0
Merge branch 'notifier_performance' into markjh/presence_performance
2015-05-18 14:33:58 +01:00
