Matrix-Mirrors/forked-synapse
1
0
Fork 0
mirror of https://mau.dev/maunium/synapse.git synced 2024-10-01 01:36:05 -04:00
Code Issues Packages Projects Releases Wiki Activity
10,484 Commits 5 Branches 723 Tags 124 MiB
d4c14e1438
Commit Graph

7613 Commits

Author SHA1 Message Date
Richard van der Hoff
a18828c129 Fix docstring for StateHandler.resolve_state_groups 
The return type was a complete lie, so fix it
 2018-02-01 12:41:34 +00:00
Richard van der Hoff
6da4c4d3bd Factor out resolve_state_groups to a separate handler 
We extract the storage-independent bits of the state group resolution out to a
separate functiom, and stick it in a new handler, in preparation for its use
from the storage layer.
 2018-02-01 12:40:04 +00:00
Richard van der Hoff
0cbda53819 Rename resolve_state_groups -> resolve_state_groups_for_events 
(to make way for a method that actually just does the state group resolution)
 2018-02-01 12:40:00 +00:00
Richard van der Hoff
77c0629ebc
Merge pull request #2837 from matrix-org/rav/fix_quarantine_media 
Fix sql error in quarantine_media
 2018-02-01 11:45:58 +01:00
Travis Ralston
e16e45b1b4 pep8 
Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-31 15:30:38 -07:00
Richard van der Hoff
e1e4ec9f9d factor _get_new_state_after_events out of _calculate_state_delta 
This reduces the scope of a bunch of variables
 2018-01-31 21:32:09 +00:00
Richard van der Hoff
78e7e05188
Merge pull request #2838 from matrix-org/rav/fix_logging_on_dns_fail 
Remove spurious log argument
 2018-01-31 22:18:46 +01:00
Richard van der Hoff
ad48dfe73d
Merge pull request #2835 from matrix-org/rav/remove_event_type_param 
Remove unused "event_type" param on state.get_current_state_ids
 2018-01-31 22:17:57 +01:00
Richard van der Hoff
518a74586c
Merge pull request #2834 from matrix-org/rav/better_persist_event_exception_handling 
Improve exception handling in persist_event
 2018-01-31 22:17:38 +01:00
Richard van der Hoff
d1fe4db882
Merge pull request #2833 from matrix-org/rav/pusher_hacks 
Logging and metrics for the http pusher
 2018-01-31 22:16:59 +01:00
Richard van der Hoff
421d68ca8c
Merge pull request #2817 from matrix-org/rav/http_conn_pool 
Use a connection pool for the SimpleHttpClient
 2018-01-31 22:14:22 +01:00
Travis Ralston
63c4383927 Documentation and naming 
Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-31 08:07:52 -07:00
Richard van der Hoff
af19f5e9aa Remove spurious log argument 
... which would cause scary-looking and unhelpful errors in the log on dns fail
 2018-01-30 17:52:03 +00:00
Richard van der Hoff
773f0eed1e Fix sql error in quarantine_media 2018-01-30 15:02:51 +00:00
Richard van der Hoff
adfc0c9539 docstring for get_current_state_ids 2018-01-29 17:39:55 +00:00
Richard van der Hoff
d413a2ba98 Remove unused "event_type" param on state.get_current_state_ids 
this param doesn't seem to be used, and is a bit pointless anyway because it
can easily be replicated by the caller. It is also horrible, because it changes
the return type of the method.
 2018-01-29 17:06:57 +00:00
Richard van der Hoff
b387ee17b6 Improve exception handling in persist_event 
1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that
a Failure object will be constructed using the current exception state,
*including* its stack trace - so the stack trace is saved in the Failure,
leading to better exception reports.

2. Set `consumeErrors=True` on the ObservableDeferred, because we know that
there will always be at least one observer - which avoids a spurious "CRITICAL:
unhandled exception in Deferred" error in the logs
 2018-01-29 17:05:33 +00:00
Richard van der Hoff
03dd745fe2 Better logging when pushes fail 2018-01-29 15:49:06 +00:00
Richard van der Hoff
e051abd20b add appid/device_display_name to to pusher logging 2018-01-29 15:04:16 +00:00
Richard van der Hoff
02ba118f81 Increase http conn pool size 2018-01-29 14:30:15 +00:00
Richard van der Hoff
46022025ea Fix SQL for user search 
fix some syntax errors for user search when search_all_users is enabled

fixes #2801, hopefully
 2018-01-27 17:21:57 +00:00
Richard van der Hoff
4b0f06e99c
Merge pull request #2830 from matrix-org/rav/factor_out_get_conn 
Factor out get_db_conn to HomeServer base class
 2018-01-27 17:56:25 +01:00
Neil Johnson
e98f0f9112
Merge pull request #2827 from matrix-org/fix_server_500_on_public_rooms_call_when_no_rooms_exist 
Fix server 500 on public rooms call when no rooms exist
 2018-01-26 20:22:40 +00:00
Richard van der Hoff
25adde9a04 Factor out get_db_conn to HomeServer base class 
This function is identical to all subclasses, so we may as well push it up to
the base class to reduce duplication (and make use of it in the tests)
 2018-01-26 00:56:49 +00:00
Neil Johnson
73560237d6 add white space line 2018-01-26 00:15:10 +00:00
Neil Johnson
86c4f49a31 rather than try reconstruct the results object, better to guard against the xrange step argument being 0 2018-01-26 00:12:02 +00:00
Neil Johnson
f632083576 fix return type, should be a dict 2018-01-25 23:52:17 +00:00
Neil Johnson
6c6e197b0a fix PEP8 violation 2018-01-25 23:47:46 +00:00
Neil Johnson
d02e43b15f remove white space 2018-01-25 23:29:46 +00:00
Neil Johnson
349c739966 synapse 500s on a call to publicRooms in the case where the number of public rooms is zero, the specific cause is due to xrange trying to use a step value of zero, but if the total room number really is zero then it makes sense to just bail and save the extra processing 2018-01-25 23:28:44 +00:00
Matthew Hodgson
9a72b70630 fix thinko on 3pid whitelisting 2018-01-24 11:07:47 +01:00
Richard van der Hoff
25e2456ee7
Merge pull request #2816 from matrix-org/rav/metrics_comments 
Add some comments about the reactor tick time metric
 2018-01-23 19:39:26 +00:00
Matthew Hodgson
d32385336f
add ?ts massaging for ASes (#2754) 
blindly implement ?ts for AS. untested
 2018-01-23 09:59:06 +01:00
Richard van der Hoff
4528dd2443 Fix logging and add user_id 2018-01-22 20:15:42 +00:00
Richard van der Hoff
93efd7eb04 logging and debug for http pusher 2018-01-22 18:14:10 +00:00
Matthew Hodgson
ab9f844aaf
Add federation_domain_whitelist option (#2820) 
Add federation_domain_whitelist

gives a way to restrict which domains your HS is allowed to federate with.
useful mainly for gracefully preventing a private but internet-connected HS from trying to federate to the wider public Matrix network
 2018-01-22 19:11:18 +01:00
Matthew Hodgson
d84f65255e
Merge pull request #2813 from matrix-org/matthew/registrations_require_3pid 
add registrations_require_3pid and allow_local_3pids
 2018-01-22 13:57:22 +00:00
Travis Ralston
a94d9b6b82 Appease the linter 
These are ids anyways, not mxc uris.

Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-20 22:49:46 -07:00
Travis Ralston
5552ed9a7f Add an admin route to get all the media in a room 
This is intended to be used by administrators to monitor the media that is passing through their server, if they wish.

Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-20 22:37:53 -07:00
Richard van der Hoff
2c8526cac7 Use a connection pool for the SimpleHttpClient 
In particular I hope this will help the pusher, which makes many requests to
sygnal, and is currently negotiating SSL for each one.
 2018-01-20 00:55:44 +00:00
Richard van der Hoff
87b7d72760 Add some comments about the reactor tick time metric 2018-01-19 23:51:04 +00:00
Matthew Hodgson
49fce04624 fix typo (thanks sytest) 2018-01-19 19:55:38 +00:00
Richard van der Hoff
b0d9e633ee
Merge pull request #2814 from matrix-org/rav/fix_urlcache_thumbs 
Use the right path for url_preview thumbnails
 2018-01-19 18:57:15 +00:00
Richard van der Hoff
ad7ec63d08 Use the right path for url_preview thumbnails 
This was introduced by #2627: we were overwriting the original media for url
previews with the thumbnails :/

(fixes https://github.com/vector-im/riot-web/issues/6012, hopefully)
 2018-01-19 18:29:39 +00:00
Matthew Hodgson
62d7d66ae5 oops, check all login types 2018-01-19 18:23:56 +00:00
Matthew Hodgson
8fe253f19b fix PR nitpicking 2018-01-19 18:23:45 +00:00
Matthew Hodgson
293380bef7 trailing commas 2018-01-19 15:38:53 +00:00
Matthew Hodgson
447f4f0d5f rewrite based on PR feedback: 
* [ ] split config options into allowed_local_3pids and registrations_require_3pid
 * [ ] simplify and comment logic for picking registration flows
 * [ ] fix docstring and move check_3pid_allowed into a new util module
 * [ ] use check_3pid_allowed everywhere

@erikjohnston PTAL
 2018-01-19 15:33:55 +00:00
Matthew Hodgson
9d332e0f79 fix up v1, and improve errors 2018-01-19 00:53:58 +00:00
Matthew Hodgson
0af58f14ee fix pep8 2018-01-19 00:33:51 +00:00
Matthew Hodgson
28a6ccb49c add registrations_require_3pid 
lets homeservers specify a whitelist for 3PIDs that users are allowed to associate with.
Typically useful for stopping people from registering with non-work emails
 2018-01-19 00:19:58 +00:00
Erik Johnston
cd871a3057 Fix storage provider bug introduced when renamed to store_local 2018-01-18 18:37:59 +00:00
Erik Johnston
8ff6726c0d
Merge pull request #2812 from matrix-org/erikj/media_storage_provider_config 
Make storage providers configurable
 2018-01-18 18:33:57 +00:00
Erik Johnston
d69768348f Fix passing wrong config to provider constructor 2018-01-18 17:14:05 +00:00
Erik Johnston
8e85220373 Remove duplicate directory test 2018-01-18 17:12:35 +00:00
Erik Johnston
3fe2bae857 Missing staticmethod 2018-01-18 17:11:45 +00:00
Erik Johnston
aae77da73f Fixup comments 2018-01-18 17:11:29 +00:00
Erik Johnston
b6dc7044a9
Merge pull request #2804 from matrix-org/erikj/file_consumer 
Add decent impl of a FileConsumer
 2018-01-18 16:31:33 +00:00
Erik Johnston
9a89dae8c5 Fix typo in thumbnail resource causing access times to be incorrect 2018-01-18 15:06:24 +00:00
Erik Johnston
0af5dc63a8 Make storage providers more configurable 2018-01-18 14:07:21 +00:00
Richard van der Hoff
d57765fc8a Fix bugs in block metrics 
... which I introduced in #2785
 2018-01-18 12:24:42 +00:00
Erik Johnston
2cf6a7bc20 Use better file consumer 2018-01-18 12:00:46 +00:00
Erik Johnston
4a53f3a3e8 Ensure media is in local cache before thumbnailing 2018-01-18 12:00:46 +00:00
Erik Johnston
be0dfcd4a2 Do logcontexts correctly 2018-01-18 11:57:57 +00:00
Erik Johnston
1432f7ccd5 Move test stuff to tests 2018-01-18 11:57:57 +00:00
Erik Johnston
2f18a2647b Make all fields private 2018-01-18 11:57:54 +00:00
Richard van der Hoff
d6af5512bb
Merge pull request #2809 from matrix-org/rav/metrics_errors 
better exception logging in callbackmetrics
 2018-01-18 11:46:37 +00:00
Richard van der Hoff
ce236f8ac8 better exception logging in callbackmetrics 
when we fail to render a metric, give a clue as to which metric it was
 2018-01-18 11:30:49 +00:00
Erik Johnston
dc519602ac Ensure we registerProducer isn't called twice 2018-01-18 11:07:17 +00:00
Erik Johnston
17b54389fe Fix _notify_empty typo 2018-01-18 11:05:34 +00:00
Erik Johnston
28b338ed9b Move definition of paused_producer to __init__ 2018-01-18 11:04:41 +00:00
Erik Johnston
a177325b49 Fix comments 2018-01-18 11:02:43 +00:00
Richard van der Hoff
36da256cc6
Merge pull request #2805 from matrix-org/rav/log_state_res 
Log room when doing state resolution
 2018-01-17 18:05:04 +00:00
Richard van der Hoff
1224612a79 Log room when doing state resolution 
Mostly because it helps figure out what is prompting the resolution
 2018-01-17 17:11:59 +00:00
Erik Johnston
bc67e7d260 Add decent impl of a FileConsumer 
Twisted core doesn't have a general purpose one, so we need to write one
ourselves.

Features:
- All writing happens in background thread
- Supports both push and pull producers
- Push producers get paused if the consumer falls behind
 2018-01-17 16:43:03 +00:00
Erik Johnston
a87006f9c7
Merge pull request #2783 from matrix-org/erikj/media_last_accessed 
Keep track of last access time for local media
 2018-01-17 16:39:02 +00:00
Matthew Hodgson
06db5c4b76
Merge pull request #2803 from matrix-org/matthew/fix-userdir-sql 
fix SQL when searching all users
 2018-01-17 16:27:54 +00:00
Matthew Hodgson
2d9ab533f9 fix SQL when searching all users 2018-01-17 15:58:52 +00:00
Richard van der Hoff
390093d45e Split resolve_events into two functions 
... so that the return type doesn't depend on the arg types
 2018-01-17 15:44:31 +00:00
Erik Johnston
2fb3a28c98
Remove lost comment 2018-01-17 14:59:44 +00:00
Richard van der Hoff
a7e4ff9cca
Merge pull request #2795 from matrix-org/rav/track_db_scheduling 
Track DB scheduling delay per-request
 2018-01-17 14:29:37 +00:00
Richard van der Hoff
f884cfffb9
Merge pull request #2797 from matrix-org/rav/user_id_checking 
Sanity checking for user ids
 2018-01-17 14:29:12 +00:00
Richard van der Hoff
a5213df1f7 Sanity checking for user ids 
Check the user_id passed to a couple of APIs for validity, to avoid
"IndexError: list index out of range" exception which looks scary and results
in a 500 rather than a more useful error.

Fixes #1432, among other things
 2018-01-17 14:28:54 +00:00
Richard van der Hoff
e8f7541d3f Merge remote-tracking branch 'origin/develop' into rav/track_db_scheduling 2018-01-17 14:01:57 +00:00
Richard van der Hoff
fb6563b4be
Merge pull request #2793 from matrix-org/rav/db_txn_time_in_millis 
Track db txn time in millisecs
 2018-01-17 13:52:42 +00:00
Richard van der Hoff
1954e867b4
Merge pull request #2796 from matrix-org/rav/fix_closed_connection_errors 
Fix 'NoneType' object has no attribute 'writeHeaders'
 2018-01-17 13:52:24 +00:00
Richard van der Hoff
11ab2f56f5 Merge branch 'develop' into rav/fix_closed_connection_errors 2018-01-17 11:25:11 +00:00
Richard van der Hoff
0486a7814a Merge branch 'develop' into rav/rework_run_interaction 2018-01-17 11:24:38 +00:00
Richard van der Hoff
90c14da992 Merge branch 'develop' into rav/db_txn_time_in_millis 2018-01-17 11:19:55 +00:00
Richard van der Hoff
38506773eb Merge remote-tracking branch 'origin/develop' into rav/optimise_logging_context 2018-01-17 10:57:13 +00:00
Erik Johnston
300edc2348 Update last access time when thumbnails are viewed 2018-01-17 10:24:43 +00:00
Erik Johnston
05f98a2224 Keep track of last access time for local media 2018-01-17 10:24:43 +00:00
Erik Johnston
3cb2dabaad
Merge pull request #2789 from matrix-org/erikj/fix_thumbnails 
Fix thumbnailing remote files
 2018-01-17 10:22:20 +00:00
Erik Johnston
d728c47142 Add docstring 2018-01-17 10:06:14 +00:00
Richard van der Hoff
936482d507 Fix 'NoneType' object has no attribute 'writeHeaders' 
Avoid throwing a (harmless) exception when we try to write an error response to
an http request where the client has disconnected.

This comes up as a CRITICAL error in the logs which tends to mislead people
into thinking there's an actual problem
 2018-01-16 17:58:16 +00:00
Richard van der Hoff
3d12d97415 Track DB scheduling delay per-request 
For each request, track the amount of time spent waiting for a db
connection. This entails adding it to the LoggingContext and we may as well add
metrics for it while we are passing.
 2018-01-16 17:23:32 +00:00
Richard van der Hoff
0f5d2cc37c Merge branch 'rav/db_txn_time_in_millis' into rav/track_db_scheduling 2018-01-16 17:09:15 +00:00
Richard van der Hoff
8615f19d20 rework runInteraction in terms of runConnection 
... so that we can share the code
 2018-01-16 17:08:29 +00:00
Matthew Hodgson
5e97ca7ee6 fix typo 2018-01-16 16:52:35 +00:00
Erik Johnston
d863f68cab Use local vars 2018-01-16 16:24:15 +00:00
Erik Johnston
6368e5c0ab Change _generate_thumbnails to take media_type 2018-01-16 16:17:38 +00:00
Erik Johnston
0a90d9ede4 Move setting of file_id up to caller 2018-01-16 16:03:05 +00:00
Richard van der Hoff
6324b65f08 Track db txn time in millisecs 
... to reduce the amount of floating-point foo we do.
 2018-01-16 15:53:18 +00:00
Richard van der Hoff
44a498418c Optimise LoggingContext creation and copying 
It turns out that the only thing we use the __dict__ of LoggingContext for is
`request`, and given we create lots of LoggingContexts and then copy them every
time we do a db transaction or log line, using the __dict__ seems a bit
redundant. Let's try to optimise things by making the request attribute
explicit.
 2018-01-16 15:49:42 +00:00
Erik Johnston
5dfc83704b Fix typo 2018-01-16 14:32:56 +00:00
Richard van der Hoff
febdca4b37
Merge pull request #2785 from matrix-org/rav/reorganise_metrics_again 
Reorganise request and block metrics
 2018-01-16 14:10:21 +00:00
Richard van der Hoff
f5f89fda21
Merge pull request #2790 from matrix-org/rav/preserve_event_logcontext_leak 
Fix a logcontext leak in persist_events
 2018-01-16 14:09:13 +00:00
Erik Johnston
307f88dfb6 Fix up log lines 2018-01-16 13:53:52 +00:00
Richard van der Hoff
807e848f0f
Merge pull request #2787 from matrix-org/rav/worker_event_counts 
Metrics for events processed in appservice and fed sender
 2018-01-16 13:14:09 +00:00
Richard van der Hoff
4a31a61ef9
Merge pull request #2786 from matrix-org/rav/rdata_metrics 
Metrics for number of RDATA commands received
 2018-01-16 13:13:53 +00:00
Erik Johnston
9795b9ebb1 Correctly use server_name/file_id when generating/fetching remote thumbnails 2018-01-16 12:02:06 +00:00
Erik Johnston
c5b589f2e8 Log when we respond with 404 2018-01-16 12:01:40 +00:00
Richard van der Hoff
64ddec1bc0 Fix a logcontext leak in persist_events 
ObserveableDeferred expects its callbacks to be called without any
logcontexts, whereas it turns out we were calling them with the logcontext of
the request which initiated the persistence loop.

It seems wrong that we are attributing work done in the persistence loop to the
request that happened to initiate it, so let's solve this by dropping the
logcontext for it.

(I'm not sure this actually causes any real problems other than messages in the
debug log, but let's clean it up anyway)
 2018-01-16 11:47:36 +00:00
Erik Johnston
a4c5e4a645 Fix thumbnailing remote files 2018-01-16 11:37:50 +00:00
Erik Johnston
1159abbdd2
Merge pull request #2767 from matrix-org/erikj/media_storage_refactor 
Refactor MediaRepository to separate out storage
 2018-01-16 10:23:50 +00:00
Richard van der Hoff
a027c2af8d Metrics for events processed in appservice and fed sender 
More metrics I wished I'd had
 2018-01-15 18:23:24 +00:00
Richard van der Hoff
5c3c32f16f Metrics for number of RDATA commands received 
I found myself wishing we had this.
 2018-01-15 17:45:55 +00:00
Richard van der Hoff
39f4e29d01 Reorganise request and block metrics 
In order to circumvent the number of duplicate foo:count metrics increasing
without bounds, it's time for a rearrangement.

The following are all deprecated, and replaced with synapse_util_metrics_block_count:
  synapse_util_metrics_block_timer:count
  synapse_util_metrics_block_ru_utime:count
  synapse_util_metrics_block_ru_stime:count
  synapse_util_metrics_block_db_txn_count:count
  synapse_util_metrics_block_db_txn_duration:count

The following are all deprecated, and replaced with synapse_http_server_response_count:
   synapse_http_server_requests
   synapse_http_server_response_time:count
   synapse_http_server_response_ru_utime:count
   synapse_http_server_response_ru_stime:count
   synapse_http_server_response_db_txn_count:count
   synapse_http_server_response_db_txn_duration:count

The following are renamed (the old metrics are kept for now, but deprecated):

  synapse_util_metrics_block_timer:total ->
     synapse_util_metrics_block_time_seconds

  synapse_util_metrics_block_ru_utime:total ->
     synapse_util_metrics_block_ru_utime_seconds

  synapse_util_metrics_block_ru_stime:total ->
     synapse_util_metrics_block_ru_stime_seconds

  synapse_util_metrics_block_db_txn_count:total ->
     synapse_util_metrics_block_db_txn_count

  synapse_util_metrics_block_db_txn_duration:total ->
     synapse_util_metrics_block_db_txn_duration_seconds

  synapse_http_server_response_time:total ->
     synapse_http_server_response_time_seconds

  synapse_http_server_response_ru_utime:total ->
     synapse_http_server_response_ru_utime_seconds

  synapse_http_server_response_ru_stime:total ->
     synapse_http_server_response_ru_stime_seconds

   synapse_http_server_response_db_txn_count:total ->
      synapse_http_server_response_db_txn_count

   synapse_http_server_response_db_txn_duration:total
      synapse_http_server_response_db_txn_duration_seconds
 2018-01-15 17:09:44 +00:00
Richard van der Hoff
992018d1c0 mechanism to render metrics with alternative names 2018-01-15 17:04:39 +00:00
Richard van der Hoff
80fa610f9c Add some comments to metrics classes 2018-01-15 16:52:52 +00:00
Richard van der Hoff
19d274085f Make Counter render floats 
Prometheus handles all metrics as floats, and sometimes we store non-integer
values in them (notably, durations in seconds), so let's render them as floats
too.

(Note that the standard client libraries also treat Counters as floats.)
 2018-01-12 23:49:44 +00:00
Richard van der Hoff
21bf87a146 Reinstate media download on thumbnail request 
We need to actually download the remote media when we get a request for a
thumbnail.
 2018-01-12 15:38:06 +00:00
Erik Johnston
694f1c1b18 Fix up comments 2018-01-12 15:02:46 +00:00
Erik Johnston
e21370ba54 Correctly reraise exception 2018-01-12 14:44:02 +00:00
Erik Johnston
85a4d78213 Make Responder a context manager 2018-01-12 13:32:03 +00:00
Erik Johnston
dcc8eded41 Add missing class var 2018-01-12 13:16:27 +00:00
Erik Johnston
fefeb0ab0e
Merge pull request #2774 from matrix-org/erikj/synctl 
When using synctl with workers, don't start the main synapse automatically
 2018-01-12 12:00:07 +00:00
Erik Johnston
81391fa162 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/media_storage_refactor 2018-01-12 11:28:49 +00:00
Erik Johnston
1e4edd1717 Remove unnecessary condition 2018-01-12 11:28:32 +00:00
Erik Johnston
c6c009603c Remove unused variables 2018-01-12 11:24:05 +00:00
Erik Johnston
4d88958cf6 Make class var local 2018-01-12 11:23:54 +00:00
Erik Johnston
227c491510 Comments 2018-01-12 11:22:41 +00:00
Erik Johnston
f4d93ae424 Actually make it work 2018-01-12 10:39:27 +00:00
Erik Johnston
f68e4cf690 Refactor 2018-01-12 10:11:12 +00:00
Richard van der Hoff
5f23b6d5ea
Merge pull request #2766 from matrix-org/rav/room_event 
Add /room/{id}/event/{id} to synapse
 2018-01-11 14:47:18 +00:00
Erik Johnston
7cd34512d8 When using synctl with workers, don't start the main synapse automatically 2018-01-11 11:37:39 +00:00
Erik Johnston
07ab948c38 Merge branch 'master' of github.com:matrix-org/synapse into develop 2018-01-11 11:23:40 +00:00
Erik Johnston
825a07a974
Merge pull request #2773 from matrix-org/erikj/hash_bg 
Do bcrypt hashing in a background thread
 2018-01-10 18:11:41 +00:00
Erik Johnston
f8e1ab5fee Do bcrypt hashing in a background thread 2018-01-10 18:01:28 +00:00
Michael Telatynski
5f07f5694c fix order of operations derp and also use .get to default to {} 
Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
 2018-01-10 15:11:35 +00:00
Michael Telatynski
8c9d5b4873 Fix publicised groups API (singular) over federation 
which was missing its fed client API, since there is no other API
it might as well reuse the bulk one and unwrap it

Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
 2018-01-10 15:04:51 +00:00
Richard van der Hoff
c175a5f0f2
Merge pull request #2770 from matrix-org/rav/fix_request_metrics 
Update http request metrics before calling servlet
 2018-01-10 14:53:40 +00:00
Richard van der Hoff
d90e8ea444 Update http request metrics before calling servlet 
Make sure that we set the servlet name in the metrics object *before* calling
the servlet, in case the servlet throws an exception.
 2018-01-09 18:27:35 +00:00
hera
174eacc8ba oops 2018-01-09 18:14:32 +00:00
Matthew Hodgson
a66f489678 fix GIST->GIN switch 2018-01-09 16:55:51 +00:00
Matthew Hodgson
e79db0a673 switch back from GIST to GIN indexes 2018-01-09 16:37:48 +00:00
Matthew Hodgson
e365ad329f oops, tweak work_mem when actually storing 2018-01-09 16:30:30 +00:00
Matthew Hodgson
19f9227643 avoid 80s GIN inserts by tweaking work_mem 
see https://github.com/matrix-org/synapse/issues/2753 for details
 2018-01-09 16:25:04 +00:00
Erik Johnston
8f03aa9f61 Add StorageProvider concept 2018-01-09 16:16:12 +00:00
Erik Johnston
2442e9876c Make PreviewUrlResource use MediaStorage 2018-01-09 16:15:07 +00:00
Erik Johnston
9d30a7691c Make ThumbnailResource use MediaStorage 2018-01-09 16:15:07 +00:00
Erik Johnston
9e20840e02 Use MediaStorage for remote media 2018-01-09 16:15:07 +00:00
Erik Johnston
dd3092c3a3 Use MediaStorage for local files 2018-01-09 16:15:07 +00:00
Erik Johnston
ada470bccb Add MediaStorage class 2018-01-09 16:15:07 +00:00
Erik Johnston
1ee787912b Add some helper classes 2018-01-09 16:15:07 +00:00
Erik Johnston
47ca5eb882 Split out add_file_headers 2018-01-09 16:15:07 +00:00
Erik Johnston
b6c9deffda Remove dead TODO 2018-01-09 15:53:23 +00:00
Richard van der Hoff
51c9d9ed65 Add /room/{id}/event/{id} to synapse 
Turns out that there is a valid usecase for retrieving event by id (notably
having received a push), but event ids should be scoped to room, so /event/{id}
is wrong.
 2018-01-09 14:39:12 +00:00
Erik Johnston
b30cd5b107 Remove dead code related to default thumbnails 2018-01-09 14:38:33 +00:00
Richard van der Hoff
cb66a2d387
Merge pull request #2763 from matrix-org/rav/fix_config_uts 
Fix broken config UTs
 2018-01-09 12:08:08 +00:00
Richard van der Hoff
0211464ba2 Fix broken config UTs 
https://github.com/matrix-org/synapse/pull/2755 broke log-config generation,
which in turn broke the unit tests.
 2018-01-09 11:28:33 +00:00
Richard van der Hoff
3a556f1ea0 Make indentation of generated log config consistent 
(we had a mix of 2- and 4-space indents)
 2018-01-09 11:27:19 +00:00
Travis Ralston
eccfc8e928
Fix templating error with unban permission message 
Fixes https://github.com/matrix-org/synapse/issues/2759

Signed-off-by: Travis Ralston <travpc@gmail.com>
 2018-01-07 19:52:58 -07:00
Richard van der Hoff
840f72356e
Remove 'verbosity'/'log_file' from generated cfg 
... because these only really exist to confuse people nowadays.

Also bring log config more into line with the generated log config, by making `level_for_storage`
apply to the `synapse.storage.SQL` logger rather than `synapse.storage`.
 2018-01-05 12:30:28 +00:00
Erik Johnston
864a6d2977 Bump version and changelog 2018-01-05 10:54:01 +00:00
Richard van der Hoff
6e375f4597
Merge pull request #2744 from matrix-org/rav/login_logging 
Better logging when login can't find a 3pid
 2018-01-05 09:54:39 +00:00
Richard van der Hoff
bd91857028 Check missing fields in event_from_pdu_json 
Return a 400 rather than a 500 when somebody messes up their send_join
 2017-12-30 18:40:19 +00:00
Richard van der Hoff
3079f80d4a Factor out event_from_pdu_json 
turns out we have two copies of this, and neither needs to be an instance
method
 2017-12-30 18:40:19 +00:00
Richard van der Hoff
65abc90fb6 federation_server: clean up imports 2017-12-30 18:40:19 +00:00
Richard van der Hoff
a7b726ad18 federation_client: clean up imports 2017-12-30 18:40:19 +00:00
Richard van der Hoff
75c1b8df01 Better logging when login can't find a 3pid 2017-12-20 19:31:00 +00:00
Richard van der Hoff
3f9f1c50f3
Merge pull request #2683 from seckrv/fix_pwd_auth_prov_typo 
synapse/config/password_auth_providers: Fixed bracket typo
 2017-12-18 22:37:21 +00:00
Silke
df0f602796 Implement listen_tcp method in remaining workers 
Signed-off-by: Silke <silke@slxh.eu>
 2017-12-18 20:00:42 +01:00
Silke
26cd3f5690 Remove logger argument and do not catch replication listener 
Signed-off-by: Silke <silke@slxh.eu>
 2017-12-18 20:00:42 +01:00
Silke Hofstra
ed48ecc58c Add methods for listening on multiple addresses 
Add listen_tcp and listen_ssl which implement Twisted's reactor.listenTCP
and reactor.listenSSL for multiple addresses.

Signed-off-by: Silke Hofstra <silke@slxh.eu>
 2017-12-17 13:15:48 +01:00
Silke Hofstra
37d1a90025 Allow binds to both :: and 0.0.0.0 
Binding on 0.0.0.0 when :: is specified in the bind_addresses is now allowed.
This causes a warning explaining the behaviour.
Configuration changed to match.

See #2232

Signed-off-by: Silke Hofstra <silke@slxh.eu>
 2017-12-17 13:10:31 +01:00
Willem Mulder
3e59143ba8 Adapt the default config to bind on IPv6. 
Most deployments are on Linux (or Mac OS), so this would actually bind
on both IPv4 and IPv6.

Resolves #1886.

Signed-off-by: Willem Mulder <willemmaster@hotmail.com>
 2017-12-17 13:07:37 +01:00
Erik Johnston
069ae2a5d6 Bump changelog and version 2017-12-13 15:11:22 +00:00
Erik Johnston
d8a6c734fa Merge branch 'develop' of github.com:matrix-org/synapse into erikj/createroom_content 2017-12-07 14:24:01 +00:00
Erik Johnston
ef045dcd71 Copy dict in update_membership too 2017-12-07 14:17:15 +00:00
Matthew Hodgson
33cb7ef0b7
Merge pull request #2723 from matrix-org/matthew/search-all-local-users 
Add all local users to the user_directory and optionally search them
 2017-12-05 11:09:47 +00:00
Matthew Hodgson
cdc2cb5d11 fix StoreError syntax 2017-12-05 11:09:31 +00:00
Richard van der Hoff
16ec3805e5 Fix error when deleting devices 
This was introduced in d7ea8c4 / PR #2728
 2017-12-05 09:49:22 +00:00
Richard van der Hoff
da1010c83a support custom login types for validating users 
Wire the custom login type support from password providers into the UI-auth
user-validation flows.
 2017-12-05 09:43:30 +00:00
Richard van der Hoff
d7ea8c4800 Factor out a validate_user_via_ui_auth method 
Collect together all the places that validate a logged-in user via UI auth.
 2017-12-05 09:42:30 +00:00
Richard van der Hoff
d5f9fb06b0 Refactor UI auth implementation 
Instead of returning False when auth is incomplete, throw an exception which
can be caught with a wrapper.
 2017-12-05 09:40:05 +00:00
Matthew Hodgson
c22e73293a speed up the rate of initial spam for users 2017-12-04 18:05:28 +00:00
Matthew Hodgson
b11dca2025 better doc 2017-12-04 17:51:33 +00:00
Matthew Hodgson
74e0cc74ce fix pep8 and tests 2017-12-04 15:11:38 +00:00
Matthew Hodgson
1bd40ca73e switch to a simpler 'search_all_users' button as per review feedback 2017-12-04 14:58:39 +00:00
Matthew Hodgson
f397153dfc Merge branch 'develop' into matthew/search-all-local-users 2017-11-30 01:51:38 +00:00
Matthew Hodgson
5406392f8b specify default user_directory_include_pattern 2017-11-30 01:45:34 +00:00
Matthew Hodgson
f61e107f63 remove null constraint on user_dir.room_id 2017-11-30 01:43:50 +00:00
Matthew Hodgson
4b1fceb913 fix alternation operator for FTS4 - how did this ever work!? 2017-11-30 01:34:03 +00:00
Matthew Hodgson
a4bb133b68 fix thinkos galore 2017-11-30 01:17:15 +00:00
Matthew Hodgson
cd3697e8b7 kick the user_directory index when new users register 2017-11-29 18:33:34 +00:00
Matthew Hodgson
3241c7aac3 untested WIP but might actually work 2017-11-29 18:27:05 +00:00
Richard van der Hoff
624c46eb06
Merge pull request #2721 from matrix-org/rav/get_user_by_access_token_comments 
Improve comments on get_user_by_access_token
 2017-11-29 17:57:06 +00:00
Matthew Hodgson
47d99a20d5 Add user_directory_include_pattern config param to expand search results to additional users 
Initial commit; this doesn't work yet - the LIKE filtering seems too aggressive.
It also needs _do_initial_spam to be aware of prepopulating the whole user_directory_search table with all users...
...and it needs a handle_user_signup() or something to be added so that new signups get incrementally added to the table too.

Committing it here as a WIP
 2017-11-29 16:46:45 +00:00
Richard van der Hoff
ad7e570d07 Delete devices in various logout situations 
Make sure that we delete devices whenever a user is logged out due to any of
the following situations:

 * /logout
 * /logout_all
 * change password
 * deactivate account (by the user or by an admin)
 * invalidate access token from a dynamic module

Fixes #2672.
 2017-11-29 16:44:35 +00:00
Richard van der Hoff
ae31f8ce45 Move set_password into its own handler 
Non-functional refactoring to move set_password. This means that we'll be able
to properly deactivate devices and access tokens without introducing a
dependency loop.
 2017-11-29 16:44:35 +00:00
Richard van der Hoff
7ca5c68233 Move deactivate_account into its own handler 
Non-functional refactoring to move deactivate_account. This means that we'll be
able to properly deactivate devices and access tokens without introducing a
dependency loop.
 2017-11-29 16:44:35 +00:00
Richard van der Hoff
2c6d63922a Remove pushers when deleting access tokens 
Whenever an access token is invalidated, we should remove the associated
pushers.
 2017-11-29 16:44:35 +00:00
Richard van der Hoff
97d1a1dc01
Merge pull request #2718 from matrix-org/rav/notify_logcontexts 
Clear logcontext before starting fed txn queue runner
 2017-11-29 16:01:46 +00:00
Richard van der Hoff
8b45de90a4
Merge pull request #2719 from matrix-org/rav/handle_missing_hashes 
Fix 500 when joining matrix-dev
 2017-11-29 16:01:33 +00:00
Richard van der Hoff
7303ed65e1 Fix 500 when joining matrix-dev 
matrix-dev has an event (`$/6ANj/9QWQyd71N6DpRQPf+SDUu11+HVMeKSpMzBCwM:zemos.net`)
which has no `hashes` member.

Check for missing `hashes` element in events.
 2017-11-29 16:00:46 +00:00
Richard van der Hoff
da562bd6a1 Improve comments on get_user_by_access_token 
because I have to reverse-engineer this every time.
 2017-11-29 15:52:41 +00:00
Richard van der Hoff
d4fb4f7c52 Clear logcontext before starting fed txn queue runner 
These processes take a long time compared to the request, so there is lots of
"Entering|Restoring dead context" in the logs. Let's try to shut it up a bit.
 2017-11-28 15:26:14 +00:00
Erik Johnston
dfbc45302e
PEP8 2017-11-28 15:23:26 +00:00
Erik Johnston
c4c1d170af
Fix wrong avatars when inviting multiple users when creating room 
We reused the `content` dictionary between invite requests, which meant they could end up reusing the profile info for a previous user
 2017-11-28 15:19:15 +00:00
Luke Barnard
ab1b2d0ff2 Allow guest access to group APIs for reading 2017-11-28 11:23:00 +00:00
Richard van der Hoff
5a4da5bf78
Merge pull request #2697 from matrix-org/rav/fix_urlcache_index_error 
Fix error on sqlite 3.7
 2017-11-27 12:25:48 +00:00
Richard van der Hoff
84b31a3e7a
Merge pull request #2713 from matrix-org/rav/no_upsert_forever 
Avoid retrying forever on IntegrityError
 2017-11-27 12:19:35 +00:00
Richard van der Hoff
df6c72ede3
Merge pull request #2711 from matrix-org/rav/fix_dns_errhandler 
Fix error handling on dns lookup
 2017-11-27 12:19:18 +00:00
Richard van der Hoff
04bb79f139
Merge pull request #2710 from matrix-org/rav/remove_dead_code 
Tiny code cleanups
 2017-11-27 12:15:44 +00:00
Richard van der Hoff
e828a7380a
Merge pull request #2708 from matrix-org/rav/replication_logcontext_leaks 
Fix some logcontext leaks in replication resource
 2017-11-27 12:15:33 +00:00
Richard van der Hoff
7ef22a41a3
Merge pull request #2707 from matrix-org/rav/fix_urlpreview 
Fix OPTIONS on preview_url
 2017-11-27 12:15:14 +00:00
Richard van der Hoff
6be01f599b Improve tracebacks on exceptions 
Use failure.Failure to recover our failure, which will give us a useful
stacktrace, unlike the rethrown exception.
 2017-11-27 12:05:58 +00:00
Richard van der Hoff
63ccaa5873 Avoid retrying forever on IntegrityError 2017-11-27 12:00:07 +00:00
Richard van der Hoff
8b38096a89 Fix error handling on dns lookup 
pass the right arguments to the errback handler

Fixes "TypeError('eb() takes exactly 2 arguments (1 given)',)"
 2017-11-24 16:47:48 +00:00
Richard van der Hoff
795b0849f3 Add a comment which might save some confusion 2017-11-24 00:34:56 +00:00
Richard van der Hoff
7f14f0ae38 Remove dead sync_callback 
This is never used; let's remove it to stop confusing things.
 2017-11-24 00:32:04 +00:00
Richard van der Hoff
0edf085b68 Fix some logcontext leaks in replication resource 
The @measure_func annotations rely on the wrapped function respecting the
logcontext rules. Add the necessary yields to make this work.
 2017-11-23 23:19:43 +00:00
Richard van der Hoff
8132a6b7ac Fix OPTIONS on preview_url 
Fixes #2706
 2017-11-23 17:52:31 +00:00
Richard van der Hoff
6b48b3e277 fix sql fails 2017-11-22 18:06:24 +00:00
Richard van der Hoff
2908f955d1 Check database in has_completed_background_updates 
so that the right thing happens on workers.
 2017-11-22 18:02:15 +00:00
Richard van der Hoff
68ca864141 Add config option to disable media_repo on main synapse 
... to stop us doing the cache cleanup jobs on the master.
 2017-11-22 16:20:27 +00:00
Richard van der Hoff
e1fd4751de Build MediaRepositoryResource as a homeserver dependency 
This avoids the scenario where we have four different PreviewUrlResources
configured on a single app, each of which have their own caches and cache
clearing jobs.
 2017-11-22 16:19:49 +00:00
Richard van der Hoff
d5a7c56ef9
Merge pull request #2698 from matrix-org/rav/remove_dead_dependencies 
Clean up dependency list
 2017-11-21 17:38:42 +00:00
Richard van der Hoff
0b4aa2dc21
Merge pull request #2689 from matrix-org/rav/unlock_account_data_upsert 
Avoid locking account_data tables for upserts
 2017-11-21 13:39:14 +00:00
Richard van der Hoff
7298ed7c51 Clean up dependency list 
remove those that aren't used at all, and replace the ones that don't have
builders with simple getters rather than dynamically-generated methods.
 2017-11-21 11:15:41 +00:00
Richard van der Hoff
7098b65cb8 Fix error on sqlite 3.7 
Create the url_cache index on local_media_repository as a background update, so
that we can detect whether we are on sqlite or not and create a partial or
complete index accordingly.

To avoid running the cleanup job before we have built the index, add a bailout
which will defer the cleanup if the bg updates are still running.

Fixes https://github.com/matrix-org/synapse/issues/2572.
 2017-11-21 11:14:17 +00:00
Oliver Kurz
83d8d4d8cd Allow use of higher versions of saml2 
The package was pinned to <4.0 with 07cf96eb because "from saml2 import
config" did not work. This seems to have been fixed in the mean time in the
saml2 package and therefore should not stop to use a more recent version.

Signed-off-by: Oliver Kurz <okurz@suse.de>
 2017-11-20 11:14:39 +01:00
Richard van der Hoff
59a7275258
Merge pull request #2688 from matrix-org/rav/unlock_more_upsert 
Avoid locking for upsert on pushers tables
 2017-11-17 13:09:31 +00:00
Richard van der Hoff
d8a05418f9 Merge branch 'master' into develop 2017-11-17 12:13:14 +00:00
Luke Barnard
cdf6fc15b0
Merge pull request #2686 from matrix-org/luke/as-flair 
Add automagical AS Publicised Group(s)
 2017-11-17 10:13:46 +00:00
Richard van der Hoff
74bbeb4373 Bump version in __init__.py 2017-11-17 10:10:53 +00:00
Jurek
eded7084d2 Fix auth handler #2678 2017-11-17 10:07:27 +00:00
Matthew Hodgson
34c3d0a386 typo 2017-11-17 01:54:02 +00:00
Luke Barnard
7fb0e98b03 Extract group_id from the dict for multiple use 2017-11-16 18:18:30 +00:00
Richard van der Hoff
c46139a17e Avoid locking account_data tables for upserts 2017-11-16 18:08:01 +00:00
Luke Barnard
d8391f0541 Remove unused GROUP_ID_REGEX 2017-11-16 18:05:57 +00:00
Luke Barnard
4e8374856d Document get_groups_for_user 2017-11-16 18:03:46 +00:00
Luke Barnard
270f9cd23a Flake8 2017-11-16 18:03:31 +00:00
Luke Barnard
9d83d52027 Use a generator instead of a list 2017-11-16 17:57:34 +00:00
Luke Barnard
5b48eec4a1 Make sure we check AS groups for lookup on bulk 2017-11-16 17:55:15 +00:00
Luke Barnard
b1edf26051 Check group_id belongs to this domain 2017-11-16 17:54:27 +00:00
Richard van der Hoff
06e5bcfc83 Avoid locking for upsert on pushers tables 
* replace the upsert into deleted_pushers with an insert
* no need to lock for upsert on pusher_throttle
 2017-11-16 17:52:23 +00:00
Jurek
624a8bbd67 Fix auth handler #2678 2017-11-16 17:19:02 +00:00
Luke Barnard
97bd18af4e Add automagical AS Publicised Group(s) 
via registration file "users" namespace:

```YAML
...
namespaces:
  users:
    - exclusive: true
      regex: '.*luke.*'
      group_id: '+all_the_lukes:hsdomain'
...
```

This is part of giving App Services their own groups for matching users. With this, ghost users will be given the appeareance that they are in a group and that they have publicised the fact, but _only_ from the perspective of the `get_publicised_groups_for_user` API.
 2017-11-16 16:44:55 +00:00
Richard van der Hoff
77a1227870 Fix broken ref to IntegrityError 2017-11-16 16:03:38 +00:00
Richard van der Hoff
7ab2b69e18 Avoid locking pushers table on upsert 
Now that _simple_upsert will retry on IntegrityError, we don't need to lock the
table.
 2017-11-16 15:32:01 +00:00
Richard van der Hoff
10aaa1bc15 _simple_upsert: retry on IntegrityError 
wrap the call to _simple_upsert_txn in a loop so that we retry on an
integrityerror: this means we can avoid locking the table provided there is an
unique index.
 2017-11-16 15:30:15 +00:00
Richard van der Hoff
cdc9e50a5d Cleanup in _simple_upsert_txn 
Bail out early to reduce indentation
 2017-11-16 15:29:10 +00:00
Richard von Seck
6f05de0e5e synapse/config/password_auth_providers: Fixed bracket typo 
Signed-off-by: Richard von Seck <richard.von-seck@gmx.net>
 2017-11-16 15:59:38 +01:00
Richard van der Hoff
f959c01600
Merge pull request #2661 from matrix-org/rav/statereadstore 
Pull out bits of StateStore to a mixin
 2017-11-15 17:23:01 +00:00
Richard van der Hoff
30d2730ee2 Declare support for r0.3.0 2017-11-15 16:24:22 +00:00
Richard van der Hoff
1b83c09c03
Merge pull request #2675 from matrix-org/rav/remove_broken_logcontext_funcs 
Remove preserve_context_over_{fn, deferred}
 2017-11-15 11:13:53 +00:00
David Baker
7190a550dc
Merge pull request #2650 from matrix-org/dbkr/push_include_content_option 
Rename redact_content option to include_content
 2017-11-15 10:47:38 +00:00
Richard van der Hoff
b2cd6accf5 Remove __PreservingContextDeferred too 2017-11-14 23:00:10 +00:00
Richard van der Hoff
008aa2fc6d
Merge pull request #2671 from matrix-org/rav/room_list_fixes 
Reshuffle room list request code
 2017-11-14 18:11:02 +00:00
Erik Johnston
03feb7a34d Bump version and changelog 2017-11-14 14:51:25 +00:00
Richard van der Hoff
35a4b63240 Pull out bits of StateStore to a mixin 
... so that we don't need to secretly gut-wrench it for use in the slaved
stores. I haven't done the other stores yet, but we should. I'm tired of the
workers breaking every time we tweak the stores because I forgot to gut-wrench
the right method.

fixes https://github.com/matrix-org/synapse/issues/2655.
 2017-11-14 11:43:58 +00:00
Richard van der Hoff
4dd1bfa8c1 Revert "Revert "move _state_group_cache to statestore"" 
We're going to fix this properly on this branch, so that the _state_group_cache
can end up in StateGroupReadStore.

This reverts commit ab335edb02.
 2017-11-14 11:43:58 +00:00
Richard van der Hoff
6caa379ba1
Merge pull request #2658 from matrix-org/rav/store_heirarchy_init 
Make __init__ consistent across Store hierarchy
 2017-11-14 11:43:12 +00:00
Richard van der Hoff
7e6fa29cb5 Remove preserve_context_over_{fn, deferred} 
Both of these functions ae known to leak logcontexts. Replace the remaining
calls to them and kill them off.
 2017-11-14 11:22:42 +00:00
Richard van der Hoff
44a1bfd6a6 Reshuffle room list request code 
I'm not entirely sure if this will actually help anything, but it simplifies
the code and might give further clues about why room list search requests are
blowing out the get_current_state_ids caches.
 2017-11-14 10:29:58 +00:00
Richard van der Hoff
1fc66c7460 Add a load of logging to the room_list handler 
So we can see what it gets up to.
 2017-11-14 10:23:47 +00:00
Richard van der Hoff
7bd6c87eca
Merge pull request #2668 from turt2live/travis/whoami 
Add a route for determining who you are
 2017-11-14 09:54:21 +00:00
Travis Ralston
812c191939 Remove redundent call 
Signed-off-by: Travis Ralston <travpc@gmail.com>
 2017-11-13 12:44:21 -07:00
Richard van der Hoff
c741ba59c9
Merge pull request #2669 from matrix-org/rav/cache_urlpreview_failure 
Cache failures in url_preview handler
 2017-11-13 18:36:24 +00:00
David Baker
45ab288e07 Print instead of logging 
because we had to wait until the logger was set up
 2017-11-13 18:32:08 +00:00
Richard van der Hoff
8b33ac8f6c Merge branch 'develop' into rav/invalid_request_utf8 2017-11-13 11:56:22 +00:00
Richard van der Hoff
6cfee09be9 Make __init__ consitstent across Store heirarchy 
Add db_conn parameters to the `__init__` methods of the *Store classes, so that
they are all consistent, which makes the multiple inheritance work correctly
(and so that we can later extract mixins which can be used in the slavedstores)
 2017-11-13 10:46:07 +00:00
Erik Johnston
ab335edb02 Revert "move _state_group_cache to statestore" 
This reverts commit f5cf3638e9.
 2017-11-13 10:05:33 +00:00
Erik Johnston
bfbf1e1f1a Up cache size of get_global_account_data_by_type_for_user 2017-11-13 09:52:11 +00:00
Travis Ralston
2d314b771f Add a route for determining who you are 
Useful for applications which may have an access token, but no idea as to who owns it.

Signed-off-by: Travis Ralston <travpc@gmail.com>
 2017-11-12 23:39:38 -07:00
Richard van der Hoff
5d15abb120 Bit more logging 2017-11-10 16:58:04 +00:00
Richard van der Hoff
46790f50cf Cache failures in url_preview handler 
Reshuffle the caching logic in the url_preview handler so that failures are
cached (and to generally simplify things and fix the logcontext leaks).
 2017-11-10 16:50:50 +00:00
Richard van der Hoff
e508145c9b Add some more comments appservice user registration 
Explain why we don't validate userids registered via app services
 2017-11-10 12:39:45 +00:00
Richard van der Hoff
e0ebd1e4bd Downcase userids for shared-secret registration 2017-11-10 12:39:05 +00:00
Richard van der Hoff
f90649eb2b Fix 500 on invalid utf-8 in request 
If somebody sends us a request where the the body is invalid utf-8, we should
return a 400 rather than a 500. (json.loads throws a UnicodeError in this
situation)

We might as well catch all Exceptions here: it seems very unlikely that we
would get a request that *isn't caused by invalid json.
 2017-11-10 09:15:39 +00:00
Richard van der Hoff
9b599bc18d Downcase userid on registration 
Force username to lowercase before attempting to register

https://github.com/matrix-org/synapse/issues/2660
 2017-11-09 22:20:01 +00:00
Richard van der Hoff
9b803ccc98 Revert "Allow upper-case characters in mxids" 
This reverts commit b70b646903.
 2017-11-09 21:57:24 +00:00
Richard van der Hoff
b70b646903 Allow upper-case characters in mxids 
Because we're never going to be able to fix this :'(
 2017-11-09 19:36:13 +00:00
Erik Johnston
2dce6b15c3 Fix typo 2017-11-09 15:56:16 +00:00
Erik Johnston
4e2b2508af Register group servlet 2017-11-09 15:49:42 +00:00
Erik Johnston
13735843c7 Namespace visibility options for groups 2017-11-09 15:27:18 +00:00
Richard van der Hoff
618c7b816a
Merge pull request #2656 from matrix-org/rav/fix_deactivate 
Fix 'NoneType' not iterable in /deactivate
 2017-11-09 15:20:35 +00:00
Richard van der Hoff
889102315e Fix 'NoneType' not iterable in /deactivate 
make sure we actually return a value from user_delete_access_tokens
 2017-11-09 15:15:33 +00:00
David Baker
b2a788e902 Make the commented config have the default 2017-11-09 10:11:42 +00:00
Erik Johnston
82e4bfb53d Add brackets 2017-11-09 10:06:42 +00:00
Erik Johnston
e8814410ef Have an explicit API to update room config 2017-11-08 16:13:27 +00:00
Erik Johnston
94ff2cda73
Revert "Modify group room association API to allow modification of is_public" 2017-11-08 15:43:34 +00:00
Erik Johnston
d305987b40
Merge pull request #2631 from xyzz/fix_appservice_event_backlog 
Fix appservices being backlogged and not receiving new events due to a bug in notify_interested_services
 2017-11-08 11:54:10 +00:00
Erik Johnston
167eb01d83
Merge pull request #2637 from spantaleev/avoid-noop-media-deletes 
Avoid no-op media deletes
 2017-11-08 11:53:27 +00:00
David Baker
ad408beb66 better comments 2017-11-08 11:50:08 +00:00
David Baker
1b870937ae Log if any of the old config flags are set 2017-11-08 11:46:24 +00:00
David Baker
2a98ba0ed3 Rename redact_content option to include_content 
The redact_content option never worked because it read the wrong config
section. The PR introducing it
(https://github.com/matrix-org/synapse/pull/2301) had feedback suggesting the
name be changed to not re-use the term 'redact' but this wasn't
incorporated.

This reanmes the option to give it a less confusing name, and also
means that people who've set the redact_content option won't suddenly
see a behaviour change when upgrading synapse, but instead can set
include_content if they want to.

This PR also updates the wording of the config comment to clarify
that this has no effect on event_id_only push.

Includes https://github.com/matrix-org/synapse/pull/2422
 2017-11-08 10:35:30 +00:00
Richard van der Hoff
02a9a93bde
Merge pull request #2649 from matrix-org/rav/fix_delta_on_state_res 
Fix bug in state group storage
 2017-11-08 09:22:13 +00:00
Richard van der Hoff
e148438e97 s/items/iteritems/ 2017-11-08 09:21:41 +00:00
Ilya Zhuravlev
d46386d57e Remove useless assignment in notify_interested_services 2017-11-07 22:23:22 +03:00
Matthew Hodgson
228ccf1fe3
Merge pull request #2643 from matrix-org/matthew/user_dir_typos 
Fix various embarrassing typos around user_directory and add some doc.
 2017-11-07 17:31:11 +00:00
Richard van der Hoff
780dbb378f Update deltas when doing auth resolution 
Fixes a bug where the persisted state groups were different to those actually
being used after auth resolution.
 2017-11-07 16:43:00 +00:00
Richard van der Hoff
1ca4288135 factor out _update_context_for_auth_events 
This is duplicated, so let's factor it out before fixing it
 2017-11-07 16:43:00 +00:00
Richard van der Hoff
f5cf3638e9 move _state_group_cache to statestore 
this is internal to statestore, so let's keep it there.
 2017-11-07 16:43:00 +00:00
Erik Johnston
5ef5e14ecc
Merge pull request #2636 from farialima/me-master 
Fix for #2635: correctly update rooms avatar/display name when modified by admin
 2017-11-07 13:49:27 +00:00
Erik Johnston
76c9af193c Revert "Merge branch 'master' of github.com:matrix-org/synapse into develop" 
This reverts commit f9b255cd62, reversing
changes made to 1bd654dabd.
 2017-11-07 13:32:35 +00:00
Erik Johnston
f9b255cd62 Merge branch 'master' of github.com:matrix-org/synapse into develop 2017-11-07 13:31:03 +00:00
Luke Barnard
38b265cb51 Remember to pick is_admin out of the db 2017-11-07 11:24:04 +00:00
Luke Barnard
5561c09091 Return whether a user is an admin within a group 2017-11-07 11:18:45 +00:00
Matthew Hodgson
3db5ff69b2
Merge pull request #2576 from maximevaillancourt/exclude-noscript-url-preview 
Ignore <noscript> tags when generating URL preview descriptions
 2017-11-07 11:09:22 +00:00
Matthew Hodgson
631fa4a1b7 create new indexes before dropping old ones to keep safetynet in place 2017-11-07 10:41:55 +00:00
Richard van der Hoff
bf993db11c Logging and logcontext fixes for Limiter 
Add some logging to the Limiter in a similar spirit to the Linearizer, to help
debug issues.

Also fix a logcontext leak.

Also refactor slightly to avoid throwing exceptions.
 2017-11-07 00:48:57 +00:00
Matthew Hodgson
4ad883398f s/users_in_pubic_room/users_in_public_rooms/g 2017-11-04 19:39:40 +00:00
Matthew Hodgson
d802e8ca6a s/users_in_pubic_room/users_in_public_rooms/g 2017-11-04 19:38:13 +00:00
Matthew Hodgson
a100700630 fix copyright.... 2017-11-04 19:35:49 +00:00
Matthew Hodgson
b6b075fd49 s/popualte/populate/ 2017-11-04 19:35:33 +00:00
Matthew Hodgson
d1622e080f s/intial/initial/ 2017-11-04 19:35:14 +00:00
Slavi Pantaleev
805196fbeb Avoid no-op media deletes 
If there are no media entries to delete,
avoid creating transactions, prepared statements
and unnecessary log entries.

Signed-off-by: Slavi Pantaleev <slavi@devture.com>
 2017-11-04 09:50:15 +02:00
Francois Granade
f103b91ffa removed unused import flagged by flake8a 2017-11-03 18:45:49 +01:00
Francois Granade
fa4f337b49 Fix for issue 2635: correctly update rooms avatar/display name when modified by admin 2017-11-03 18:25:04 +01:00
Ilya Zhuravlev
45fbe4ff67 Fix appservices being backlogged and not receiving new events due to a bug in notify_interested_services 2017-11-02 22:49:43 +03:00
David Baker
f851bc8182
Merge pull request #2630 from matrix-org/luke/fix-rooms-in-group 
Make the get_rooms_in_group API more sane
 2017-11-02 17:23:17 +00:00
David Baker
9e09a1800b
Merge pull request #2629 from matrix-org/rav/register_inhibit_login 
support inhibit_login in /register
 2017-11-02 16:51:35 +00:00
Luke Barnard
a34c586a89 Make the get_rooms_in_group API more sane 
Return entries with is_public = True when they're public and is_public = False otherwise.
 2017-11-02 16:42:30 +00:00
Richard van der Hoff
6c3a02072b support inhibit_login in /register 
Allow things to pass inhibit_login when registering to ... inhibit logins.
 2017-11-02 16:31:07 +00:00
David Baker
4b36897cd9 Merge remote-tracking branch 'origin/develop' into rav/module_api_hooks 2017-11-02 15:19:17 +00:00
David Baker
d4553818a0
Merge pull request #2627 from matrix-org/rav/custom_rest_endpoints 
Add a hook for custom rest endpoints
 2017-11-02 15:18:37 +00:00
David Baker
6b6f03ae05
Merge pull request #2626 from matrix-org/rav/refactor_module_api 
Factor _AccountHandler proxy out to ModuleApi
 2017-11-02 15:15:30 +00:00
Richard van der Hoff
6b60f7dca0 Add more hooks to ModuleApi 
add `get_user_by_req` and `invalidate_access_token`
 2017-11-02 14:37:39 +00:00
Richard van der Hoff
fcdfc911ee Add a hook for custom rest endpoints 
Let the user specify custom modules which can be used for implementing extra
endpoints.
 2017-11-02 14:36:55 +00:00
Richard van der Hoff
1189be43a2 Factor _AccountHandler proxy out to ModuleApi 
We're going to need to use this from places that aren't password auth, so let's
move it to a proper class.
 2017-11-02 14:36:11 +00:00
Richard van der Hoff
6650a07ede Factor out _configure_named_resource 
This was a bit of a code vomit, so let's factor it out to preserve some sanity
 2017-11-02 14:33:37 +00:00
David Baker
b19d9e2174
Merge pull request #2624 from matrix-org/rav/password_provider_notify_logout 
Notify auth providers on logout
 2017-11-02 10:55:17 +00:00
David Baker
1f080a6c97
Merge pull request #2623 from matrix-org/rav/callbacks_for_auth_providers 
Allow password_auth_providers to return a callback
 2017-11-02 10:49:03 +00:00
David Baker
04897c9dc1
Merge pull request #2622 from matrix-org/rav/db_access_for_auth_providers 
Let auth providers get to the database
 2017-11-02 10:41:25 +00:00
Richard van der Hoff
979eed4362 Fix user-interactive password auth 
this got broken in the previous commit
 2017-11-01 17:03:20 +00:00
Richard van der Hoff
bc8a5c0330 Notify auth providers on logout 
Provide a hook by which auth providers can be notified of logouts.
 2017-11-01 16:51:51 +00:00
Richard van der Hoff
4c8f94ac94 Allow password_auth_providers to return a callback 
... so that they have a way to record access tokens.
 2017-11-01 16:51:03 +00:00
Richard van der Hoff
846a94fbc9
Merge pull request #2620 from matrix-org/rav/auth_non_password 
Let password auth providers handle arbitrary login types
 2017-11-01 16:45:33 +00:00
Richard van der Hoff
3cd6b22c7b Let password auth providers handle arbitrary login types 
Provide a hook where password auth providers can say they know about other
login types, and get passed the relevant parameters
 2017-11-01 16:43:57 +00:00
David Baker
c9b9ef575b
Merge pull request #2621 from matrix-org/rav/refactor_accesstoken_delete 
Move access token deletion into auth handler
 2017-11-01 16:26:06 +00:00
Matthew Hodgson
275826f234
Merge pull request #2617 from matrix-org/matthew/auto-displayname 
automatically set default displayname on register
 2017-11-01 16:21:16 +00:00
David Baker
4f0488b307 Merge remote-tracking branch 'origin/develop' into rav/refactor_accesstoken_delete 2017-11-01 16:20:19 +00:00
David Baker
e5e930aec3
Merge pull request #2615 from matrix-org/rav/break_auth_device_dep 
Break dependency of auth_handler on device_handler
 2017-11-01 16:06:31 +00:00
David Baker
fbbacb284e
Merge pull request #2613 from matrix-org/rav/kill_refresh_tokens 
Remove the last vestiges of refresh_tokens
 2017-11-01 15:57:35 +00:00
Matthew Hodgson
9f7a555b4e switch to setting default displayname in the storage layer 
to avoid clobbering guest user displaynames on registration
 2017-11-01 15:51:30 +00:00
Richard van der Hoff
dd13310fb8 Move access token deletion into auth handler 
Also move duplicated deactivation code into the auth handler.

I want to add some hooks when we deactivate an access token, so let's bring it
all in here so that there's somewhere to put it.
 2017-11-01 15:46:22 +00:00
David Baker
0bb253f37b Apparently this is python 2017-11-01 14:02:52 +00:00
David Baker
59e7e62c4b Log login requests 
Carefully though, to avoid logging passwords
 2017-11-01 13:58:01 +00:00
Matthew Hodgson
f8420d6279 automatically set default displayname on register 
to avoid leaking ugly MXIDs and cluttering up the timeline with
displayname changes as well as membership joins for autojoin rooms
(e.g. the status autojoin rooms), automatically set the displayname
to match the localpart of the mxid upon registration.
 2017-11-01 13:15:41 +00:00
Luke Barnard
99354b430e
Merge pull request #2612 from matrix-org/luke/groups-room-relationship-is-public 
Modify group room association API to allow modification of is_public
 2017-11-01 11:08:36 +00:00
Richard van der Hoff
74c56f794c Break dependency of auth_handler on device_handler 
I'm going to need to make the device_handler depend on the auth_handler, so I
need to break this dependency to avoid a cycle.

It turns out that the auth_handler was only using the device_handler in one
place which was an edge case which we can more elegantly handle by throwing an
error rather than fixing it up.
 2017-11-01 10:27:06 +00:00
Luke Barnard
318a249c8b Leave is_public as required argument of update_room_group_association 2017-11-01 09:36:01 +00:00
Luke Barnard
207fabbc6a Update docs for updating room group association 2017-11-01 09:35:15 +00:00
Richard van der Hoff
356bcafc44 Remove the last vestiges of refresh_tokens 2017-10-31 20:35:58 +00:00
Richard van der Hoff
3e0aaad190 Let auth providers get to the database 
Somewhat open to abuse, but also somewhat unavoidable :/
 2017-10-31 17:22:29 +00:00
Luke Barnard
13b3d7b4a0 Flake8 2017-10-31 17:20:11 +00:00
Luke Barnard
20fe347906 Modify group room association API to allow modification of is_public 
also includes renamings to make things more consistent.
 2017-10-31 17:04:28 +00:00
Richard van der Hoff
1650eb5847 DB schema interface for password auth providers 
Provide an interface by which password auth providers can register db schema
files to be run at startup
 2017-10-31 14:01:53 +00:00
David Baker
c31a7c3ff6
Merge pull request #2609 from matrix-org/rav/refactor_login 
Refactor some logic from LoginRestServlet into AuthHandler
 2017-10-31 13:51:36 +00:00
Richard van der Hoff
1b65ae00ac Refactor some logic from LoginRestServlet into AuthHandler 
I'm going to need some more flexibility in handling login types in password
auth providers, so as a first step, move some stuff from LoginRestServlet into
AuthHandler.

In particular, we pass everything other than SAML, JWT and token logins down to
the AuthHandler, which now has responsibility for checking the login type and
fishing the password out of the login dictionary, as well as qualifying the
user_id if need be. Ideally SAML, JWT and token would go that way too, but
there's no real need for it right now and I'm trying to minimise impact.

This commit *should* be non-functional.
 2017-10-31 10:48:41 +00:00
Richard van der Hoff
ffc574a6f9 Clean up backwards-compat hacks for ldap 
try to make the backwards-compat flows follow the same code paths as the modern
impl.

This commit should be non-functional.
 2017-10-31 10:47:02 +00:00
Luke Barnard
9bc17fc5fb Fix wording on group creation error 2017-10-30 15:17:23 +00:00
Matthew Hodgson
208a6647f1 fix typo 2017-10-29 20:54:20 +00:00
Erik Johnston
d0abb4e8e6 Fix typo when checking if user is invited to group 2017-10-27 16:57:19 +01:00
Erik Johnston
977078f06d Fix bad merge 2017-10-27 15:10:50 +01:00
Erik Johnston
6980c4557e Merge branch 'erikj/attestation_jitter' of github.com:matrix-org/synapse into develop 2017-10-27 15:09:05 +01:00
Erik Johnston
af92f5b00f Revert "Add jitter to validity period of attestations" 2017-10-27 15:07:21 +01:00
Erik Johnston
4ab8abbc2b Merge branch 'erikj/attestation_local_fix' of github.com:matrix-org/synapse into develop 2017-10-27 15:07:08 +01:00
Erik Johnston
6af3656deb Merge pull request #2595 from matrix-org/erikj/attestation_commnet 
Add comment about attestations
 2017-10-27 14:20:19 +01:00
Richard van der Hoff
4d83632009 Merge pull request #2591 from matrix-org/rav/device_delete_auth 
Device deletion: check UI auth matches access token
 2017-10-27 12:30:10 +01:00
Richard van der Hoff
110b373e9c Merge pull request #2589 from matrix-org/rav/as_deactivate_account 
Allow ASes to deactivate their own users
 2017-10-27 12:29:32 +01:00
Erik Johnston
ca571b0ec3 Add jitter to validity period of attestations 
This helps ensure that the renewals of attestations are spread out more
evenly.
 2017-10-27 11:57:27 +01:00
Luke Barnard
d8c26162a1 Merge pull request #2582 from matrix-org/luke/group-is-public 
Add is_public to groups table to allow for private groups
 2017-10-27 11:41:13 +01:00
Erik Johnston
c067088747 Add comment about attestations 2017-10-27 11:35:41 +01:00
Luke Barnard
5451cc7792 Request is_public from database 2017-10-27 11:27:43 +01:00
Luke Barnard
124314672f group is dict 2017-10-27 11:08:19 +01:00
Luke Barnard
6362298fa5 Create groups with is_public = True 2017-10-27 11:04:20 +01:00
Richard van der Hoff
8b56977b6f Merge pull request #2586 from matrix-org/rav/frontend_proxy_auth_header 
Front-end proxy: pass through auth header
 2017-10-27 11:01:50 +01:00
Richard van der Hoff
173567a7f2 Docstring for post_urlencoded_get_json 2017-10-27 10:59:50 +01:00
Luke Barnard
c7d9f25d22 Fix create_group to pass requester_user_id 2017-10-27 10:57:20 +01:00
Erik Johnston
e27b76d117 Import logger 2017-10-27 10:54:02 +01:00
Richard van der Hoff
8854c039f2 Merge pull request #2585 from matrix-org/rav/unstable_to_r0 
Support /keys/upload on /r0 as well as /unstable
 2017-10-27 10:53:48 +01:00
Richard van der Hoff
14f581abc2 Merge pull request #2584 from matrix-org/rav/fix_httpclient_logcontexts 
Fix logcontext leaks in httpclient
 2017-10-27 10:53:29 +01:00
Luke Barnard
2ca46c7afc Correct logic for checking private group membership 2017-10-27 10:48:01 +01:00
Erik Johnston
82d8c1bacb Fixup 2017-10-27 10:30:21 +01:00
Erik Johnston
2fd9831f7c Merge pull request #2574 from matrix-org/erikj/room_list_fixes 
Add logging and fix log contexts for publicRooms
 2017-10-27 10:01:23 +01:00
Erik Johnston
195abfe7a5 Remove incorrect attestations 2017-10-27 09:58:13 +01:00
Erik Johnston
d8dde19f04 Log if we try to do attestations for our own user and group 2017-10-27 09:55:01 +01:00
Erik Johnston
585972b51a Don't generate group attestations for local users 2017-10-27 09:46:56 +01:00
Richard van der Hoff
7a6546228b Device deletion: check UI auth matches access token 
(otherwise there's no point in the UI auth)
 2017-10-27 00:04:31 +01:00
Richard van der Hoff
785bd7fd75 Allow ASes to deactivate their own users 2017-10-27 00:01:00 +01:00
Richard van der Hoff
c89e6aadff Merge pull request #2581 from matrix-org/rav/fix_init_with_no_logfile 
Fix error when running synapse with no logfile
 2017-10-26 22:16:57 +01:00
Richard van der Hoff
54a2525133 Front-end proxy: pass through auth header 
So that access-token-in-an-auth-header works.
 2017-10-26 18:19:01 +01:00
Richard van der Hoff
0a5866bec9 Support /keys/upload on /r0 as well as /unstable 
(So that we can stop riot relying on it in /unstable)
 2017-10-26 18:18:23 +01:00
Richard van der Hoff
0d8e3ad48b Fix logcontext leaks in httpclient 
`preserve_context_over_fn` is borked
 2017-10-26 18:17:10 +01:00
Richard van der Hoff
12ef02dc3d SimpleHTTPClient: add support for headers 
Sometimes we need to pass headers into these methods
 2017-10-26 17:59:50 +01:00
Luke Barnard
69e8a05f35 Make it work 2017-10-26 17:55:58 +01:00