Erik Johnston
7c9cdb2245
Store federation stream positions in the database
2016-11-21 11:33:08 +00:00
Erik Johnston
f8ee66250a
Handle sending events and device messages over federation
2016-11-17 15:48:04 +00:00
Erik Johnston
ed787cf09e
Hook up the send queue and create a federation sender worker
2016-11-16 17:34:44 +00:00
Erik Johnston
24772ba56e
Respect use_frozen_dicts option in workers
2016-11-08 11:07:18 +00:00
Paul "LeoNerd" Evans
1cc22da600
Set up the process collector during metrics __init__; that way all split-process workers have it
2016-10-27 18:09:34 +01:00
Paul "LeoNerd" Evans
aac13b1f9a
Pass the Metrics group into the process collector instead of having it find its own one; this avoids it needing to import from synapse.metrics
2016-10-27 18:08:15 +01:00
Paul "LeoNerd" Evans
3ae48a1f99
Move the process metrics collector code into its own file
2016-10-19 18:10:24 +01:00
Erik Johnston
a68ade6ed3
Merge pull request #1162 from larroy/master
...
Use sys.executable instead of hardcoded python. fixes #1161
2016-10-14 21:42:55 +01:00
Pedro Larroy
c57f871184
Use sys.executable instead of hardcoded python. fixes #1161
2016-10-08 23:55:20 +02:00
Erik Johnston
748d8fdc7b
Reduce DB hits for replication
...
Some streams will occaisonally advance their positions without actually
having any new rows to send over federation. Currently this means that
the token will not advance on the workers, leading to them repeatedly
sending a slightly out of date token. This in turns requires the master
to hit the DB to check if there are any new rows, rather than hitting
the no op logic where we check if the given token matches the current
token.
This commit changes the API to always return an entry if the position
for a stream has changed, allowing workers to advance their tokens
correctly.
2016-09-23 16:49:21 +01:00
Erik Johnston
87528f0756
Support /initialSync in synchrotron worker
2016-09-21 11:46:28 +01:00
Matthew Hodgson
883df2e983
fix logger for client_reader worker
2016-09-17 14:12:04 +01:00
Erik Johnston
4be85281f9
Enable state caches on workers
2016-09-15 14:31:22 +01:00
Erik Johnston
e0b6db29ed
Split out public room list into a worker process
2016-09-14 14:42:51 +01:00
Erik Johnston
44330a21e9
Comment
2016-09-09 17:22:07 +01:00
Mark Haines
7ed5acacf4
Fix up the calls to the notifier for device messages
2016-09-01 18:08:40 +01:00
Erik Johnston
873f870e5a
Add new direct message storage functions to slave store
2016-08-30 09:40:32 +01:00
Mark Haines
689f4cb914
Update comment
2016-08-22 16:17:31 +01:00
Mark Haines
c8f9b45bc2
Add usage stats to prometheus monitoring
2016-08-22 15:34:38 +01:00
Erik Johnston
bcbd74dc5b
Remove log lines
2016-08-18 15:52:10 +01:00
Erik Johnston
07229bbdae
Add appservice worker
2016-08-18 14:59:55 +01:00
Mark Haines
403ecd8a2c
Missed a s/federation reader/media repository/ in a log message
2016-08-18 10:26:15 +01:00
Mark Haines
396624864a
Add a media repository worker
2016-08-18 09:38:42 +01:00
Erik Johnston
784a2d4f2c
Remove broken cache stuff
2016-08-15 11:25:48 +01:00
Erik Johnston
64e7e11853
Implement cache replication stream
2016-08-15 11:16:45 +01:00
Erik Johnston
4e1cebd56f
Make synchrotron accept /events
2016-08-12 15:31:44 +01:00
Will Hunt
5b5148b7ec
Synced up synchrotron set_state with PresenceHandler set_state
2016-08-11 11:48:30 +01:00
Erik Johnston
24f36469bc
Add federation /version API
2016-08-05 16:36:07 +01:00
Erik Johnston
5aa024e501
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/fed_reader
2016-07-29 11:24:56 +01:00
Erik Johnston
ec8b217722
Add destination retry to slave store
2016-07-28 17:35:53 +01:00
Erik Johnston
76b89d0edb
Add slace storage functions for public room list
2016-07-28 17:03:40 +01:00
Richard van der Hoff
05e7e5e972
Fix flake8 violation
...
Apparently flake8 v3 puts the error on a different line to v2. Easiest way to
make sure that happens is by putting the whole statement on one line :)
2016-07-26 11:59:08 +01:00
Erik Johnston
aede7248ab
Split out a FederationReader process
2016-07-21 17:37:44 +01:00
Erik Johnston
66868119dc
Add metrics for psutil derived memory usage
2016-07-20 16:00:21 +01:00
Mark Haines
05f1a4596a
Merge branch 'master' into develop
2016-06-23 11:17:48 +01:00
Mark Haines
5cc7564c5c
Optionally start or stop workers in synctl.
...
Optionally start or stop an individual worker by passing -w with
the path to the worker config.
Optionally start or stop every worker and the main synapse by
passing -a with a path to a directory containing worker configs.
The "-w" is intended to be used to bounce individual workers proceses.
THe "-a" is intended for when you want to restart all the workers
simultaneuously, for example when performing database upgrades.
2016-06-21 16:38:05 +01:00
Mark Haines
13e334506c
Remove the legacy v0 content upload API.
...
The existing content can still be downloaded. The last upload to the
matrix.org server was in January 2015, so it is probably safe to remove
the upload API.
2016-06-21 11:47:39 +01:00
Mark Haines
8c75040c25
Fix setting gc thresholds in the workers
2016-06-17 11:48:12 +01:00
Mark Haines
a352b68acf
Use worker_ prefixes for worker config, use existing support for multiple config files
2016-06-16 17:29:50 +01:00
Mark Haines
364d616792
Access the event_cache_size directly from the server object.
...
This means that the workers can override the event_cache_size
directly without clobbering the value in the main synapse config.
2016-06-16 12:53:15 +01:00
Mark Haines
bde13833cb
Access replication_url from the worker config directly
2016-06-16 12:44:40 +01:00
Mark Haines
885ee861f7
Inline the synchrotron and pusher configs into the main config
2016-06-16 11:06:12 +01:00
Mark Haines
7dbb473339
Add function to load config without generating it
...
Renames ``load_config`` to ``load_or_generate_config``
Adds a method called ``load_config`` that just loads the
config.
The main synapse.app.homeserver will continue to use
``load_or_generate_config`` to retain backwards compat.
However new worker processes can use ``load_config`` to
load the config avoiding some of the cruft needed to generate
the config.
As the new ``load_config`` method is expected to be used by new
configs it removes support for the legacy commandline overrides
that ``load_or_generate_config`` supports
2016-06-09 18:50:38 +01:00
Erik Johnston
2d1d1025fa
Add gc_threshold to pusher and synchrotron
2016-06-07 16:26:25 +01:00
Erik Johnston
dded389ac1
Allow setting of gc.set_thresholds
2016-06-07 15:45:56 +01:00
Mark Haines
b0932b34cb
Merge pull request #845 from matrix-org/markjh/synchrotron_presence
...
Fix a KeyError in the synchrotron presence
2016-06-06 16:52:27 +01:00
Mark Haines
4a5bbb1941
Fix a KeyError in the synchrotron presence
2016-06-06 16:37:12 +01:00
Mark Haines
5ef84da4f1
Yield on the sleeps intended to backoff replication
2016-06-06 16:05:28 +01:00
Mark Haines
ac9716f154
Fix spelling
2016-06-03 18:10:00 +01:00
Mark Haines
8f79084bd4
Add get_presence_list_accepted to the broken caches in synchrotron
2016-06-03 18:03:40 +01:00
Mark Haines
48340e4f13
Clear the list of ongoing syncs on shutdown
2016-06-03 15:02:27 +01:00
Mark Haines
da491e75b2
Appease flake8
2016-06-03 14:56:36 +01:00
Mark Haines
0b3c80a234
Use ClientIpStore to record client ips
2016-06-03 14:55:01 +01:00
Mark Haines
80aade3805
Send updates to the syncing users every ten seconds or immediately if they've just come online
2016-06-03 14:24:19 +01:00
Mark Haines
abb151f3c9
Add a separate process that can handle /sync requests
2016-06-03 11:57:26 +01:00
David Baker
1f31cc37f8
Working unsubscribe links going straight to the HS
...
and authed by macaroons that let you delete pushers and nothing else
2016-06-02 17:21:31 +01:00
Erik Johnston
c39f305067
os.environ requires a string
2016-05-16 17:21:30 +01:00
Erik Johnston
c5c5a7403b
Make synctl read a cache factor from config file
2016-05-16 17:01:57 +01:00
Mark Haines
b3f29dc1e5
Manually expire broken caches like the who_forgot_in_room
2016-05-13 17:16:27 +01:00
Mark Haines
f03ddc98ec
Use the SlavedAccountDataStore
2016-05-13 17:01:28 +01:00
Mark Haines
206eb9fd94
Shift some of the state_group methods into the SlavedEventStore
2016-05-13 16:58:14 +01:00
David Baker
b5e646a18c
Make email notifs work on the pusher synapse
...
Plus general bugfix to email notif code
2016-05-13 11:36:50 +01:00
David Baker
ae1af262f6
Pass through _get_state_group_for_events
2016-05-10 19:18:03 +02:00
David Baker
90afc07f39
StateStore, not EventsStore
2016-05-10 19:10:46 +02:00
David Baker
89b5ef7c4b
Cached functions must be accessed through the dict
2016-05-10 19:05:22 +02:00
David Baker
35b6e6d2a8
Pass though _get_state_group_for_events
2016-05-10 18:56:40 +02:00
David Baker
3367e65476
Pass through get_state_groups
2016-05-10 18:53:15 +02:00
David Baker
0c4ccdcb83
Also pass through get_profile_displayname
2016-05-10 18:51:14 +02:00
David Baker
9ef05a12c3
Add date header & message id
2016-05-04 14:52:10 +01:00
David Baker
8cc82aad87
Add db functions used for email to the pusher app
2016-05-04 11:47:59 +01:00
Mark Haines
71df327190
Actually start the pusher daemon
2016-04-26 17:07:09 +01:00
Mark Haines
c9eab73f2a
Fix typo in default pusher config
2016-04-26 17:06:18 +01:00
Mark Haines
b80b93ea0f
Add a log context to the daemonized pusher
2016-04-26 15:57:28 +01:00
Mark Haines
6df5a6a833
Optionally daemonize the pusher
2016-04-26 15:37:41 +01:00
Mark Haines
9c417c54d4
Add a couple of update methods to the PusherSlaveStore
2016-04-26 10:45:02 +01:00
Mark Haines
f15e9e8de4
Remove the uncomments from the comments
2016-04-25 17:56:24 +01:00
Mark Haines
72e2fafa20
Add a metrics listener and a ssh listener to the pusher
2016-04-25 17:34:25 +01:00
Mark Haines
f22f46f4f9
Move the listenTCP call outside the manhole function
2016-04-25 14:59:21 +01:00
Mark Haines
5905f36f05
Split out setting up the manhole to a separate file
2016-04-22 17:09:15 +01:00
Mark Haines
62607d5452
Merge branch 'develop' into markjh/split_out_site.py
...
Conflicts:
synapse/app/homeserver.py
2016-04-22 16:26:57 +01:00
Mark Haines
e856036f4c
Move SynapseSite to its own file
2016-04-22 16:09:55 +01:00
Mark Haines
9e7aa98c22
Split out create_resource_tree to a separate file
2016-04-22 15:40:51 +01:00
Mark Haines
a3ac837599
Optionally split out the pushers into a separate process
2016-04-21 17:22:37 +01:00
Mark Haines
82d7eea7e3
Move the versionstring code out of app.homeserver into util
2016-04-11 14:57:09 +01:00
Erik Johnston
8aab9d87fa
Don't require config to create database
2016-04-06 14:15:45 +01:00
Mark Haines
239badea9b
Use syntax that works on both py2.7 and py3
2016-03-07 20:13:10 +00:00
Daniel Wagner-Hall
95481e7ba7
Merge pull request #571 from matrix-org/daniel/asids
...
Mark AS users with their AS's ID
2016-03-03 13:56:28 +00:00
Mark Haines
60a0f81c7a
Add a /replication API for extracting the updates that happened on
...
synapse
This is necessary for replicating the data in synapse to be visible to a
separate service because presence and typing notifications aren't stored
in a database so won't be visible to another process.
This API can be used to either get the raw data by requesting the tables
themselves or to just receive notifications for updates by following the
streams meta-stream.
Returns updates for each table requested a JSON array of arrays with a
row for each row in the table.
Each table is prefixed by a header row with the: name of the table,
current stream_id position for the table, number of rows, number of
columns and the names of the columns.
This is followed by the rows that have been added to the server since
the requester last asked.
The API has a timeout and is hooked up to the notifier so that a slave
can long poll for updates.
2016-03-01 14:49:41 +00:00
Daniel Wagner-Hall
763360594d
Mark AS users with their AS's ID
2016-02-11 17:26:42 +00:00
Erik Johnston
2c1fbea531
Fix up logcontexts
2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall
737c4223ef
Host /media/r0 as well as /media/v1
2016-02-05 10:47:46 +00:00
Daniel Wagner-Hall
6a9f1209df
Error if macaroon key is missing from config
...
Currently we store all access tokens in the DB, and fall back to that
check if we can't validate the macaroon, so our fallback works here, but
for guests, their macaroons don't get persisted, so we don't get to
find them in the database. Each restart, we generate a new ephemeral
key, so guests lose access after each server restart.
I tried to fix up the config stuff to be less insane, but gave up, so
instead I bolt on yet another piece of custom one-off insanity.
Also, add some basic tests for config generation and loading.
2016-02-05 01:58:23 +00:00
Erik Johnston
f8aae79a72
Simplify get_rooms
2016-02-03 13:24:35 +00:00
Daniel Wagner-Hall
d83d004ccd
Fix flake8 warnings for new flake8
2016-02-02 17:18:50 +00:00
Erik Johnston
167d1df699
Merge pull request #534 from matrix-org/erikj/setup
...
Add a Homeserver.setup method
2016-01-28 14:45:36 +00:00
Erik Johnston
7ed2bbeb11
Clean up a bit. Add comment
2016-01-28 14:32:05 +00:00
Erik Johnston
a6477d5933
Remove chdir
2016-01-28 09:19:55 +00:00
Erik Johnston
fa90c180ee
Merge branch 'develop' of github.com:matrix-org/synapse into erikj/setup
2016-01-27 17:37:33 +00:00
Erik Johnston
9fda8b5193
Don't turn on profiling
2016-01-26 18:27:23 +00:00
Erik Johnston
87f9477b10
Add a Homeserver.setup method.
...
This is for setting up dependencies that require work on startup. This
is useful for the DataStore that wants to read a bunch from the database
before initiliazing.
2016-01-26 15:51:06 +00:00
