Commit Graph

790 Commits

Author SHA1 Message Date
Erik Johnston
0c93df89b6 Move MediaRepository to media_repository module 2016-04-19 11:31:43 +01:00
Erik Johnston
43f0941e8f Split out BaseMediaResource into MediaRepository
This is so that a single MediaRepository can be shared across all
resources, rather than having a "copy" per resource.

In particular this allows us to guard against both the thumbnail and
download resource triggering a download of remote content at the same
time.
2016-04-19 11:24:59 +01:00
Matthew Hodgson
aaabbd3e9e explicitly pass in the charset from Content-Type to lxml to fix cyrillic woes better 2016-04-15 14:32:25 +01:00
Matthew Hodgson
84f9cac4d0 fix cyrillic URL previews by hardcoding all page decoding to UTF-8 for now, rather than relying on lxml's heuristics which seem to get it wrong 2016-04-15 13:20:08 +01:00
Matthew Hodgson
f78b479118 fix urlparse import thinko breaking tiny URLs 2016-04-14 15:23:55 +01:00
Kegan Dougal
83776d6219 Make v2_alpha reg follow the AS API specification
The spec is clear the key should be 'user' not 'username' and this is indeed
the case for v1. This is not true for v2_alpha though, which is what this
commit is fixing.
2016-04-14 14:52:26 +01:00
Matthew Hodgson
bd77216d06 comment out 2c838f6459 due to risk of https://en.wikipedia.org/wiki/Billion_laughs attacks - thanks @torhve 2016-04-14 14:39:24 +01:00
Erik Johnston
d0633e6dbe Sanitize the optional dependencies for spider API 2016-04-13 13:38:09 +01:00
David Baker
a04c076b7f Make the /set part mandatory 2016-04-12 13:54:41 +01:00
David Baker
7b39bcdaae Mis-named function 2016-04-12 13:35:08 +01:00
David Baker
d937f342bb Split into separate servlet classes 2016-04-12 13:33:30 +01:00
David Baker
8a76094965 Add get endpoint for pushers
As per https://github.com/matrix-org/matrix-doc/pull/308
2016-04-11 18:00:03 +01:00
Erik Johnston
17515bae14 PEP8 2016-04-11 11:02:50 +01:00
Matthew Hodgson
5ffacc5e84 fix typos and needless try/except from PR review 2016-04-11 10:39:16 +01:00
Matthew Hodgson
83b2f83da0 actually throw meaningful errors 2016-04-08 21:36:59 +01:00
Mark Haines
b36270b5e1 Fix pep8 warning 2016-04-08 19:52:23 +01:00
Matthew Hodgson
1ccabe2965 more PR feedback 2016-04-08 18:58:08 +01:00
Matthew Hodgson
dafef5a688 Add url_preview_enabled config option to turn on/off preview_url endpoint. defaults to off.
Add url_preview_ip_range_blacklist to let admins specify internal IP ranges that must not be spidered.
Add url_preview_url_blacklist to let admins specify URL patterns that must not be spidered.
Implement a custom SpiderEndpoint and associated support classes to implement url_preview_ip_range_blacklist
Add commentary and generally address PR feedback
2016-04-08 18:37:15 +01:00
Matthew Hodgson
9f7dc2bef7 Merge branch 'develop' into matthew/preview_urls 2016-04-04 00:38:21 +01:00
Matthew Hodgson
cf51c4120e report image size (bytewise) in OG meta 2016-04-03 23:57:05 +01:00
Matthew Hodgson
0834b152fb char encoding 2016-04-03 12:59:27 +01:00
Matthew Hodgson
8b98a7e8c3 pep8 2016-04-03 12:56:29 +01:00
Matthew Hodgson
eab4d462f8 fix etag typing error. fix timestamp typing error 2016-04-03 02:02:46 +01:00
Matthew Hodgson
c3916462f6 rebase all image URLs 2016-04-03 01:33:12 +01:00
Matthew Hodgson
110780b18b remove stale todo 2016-04-03 00:48:31 +01:00
Matthew Hodgson
b09e29a03c Ensure only one download for a given URL is active at a time 2016-04-03 00:47:40 +01:00
Matthew Hodgson
7426c86eb8 add a persistent cache of URL lookups, and fix up the in-memory one to work 2016-04-03 00:31:57 +01:00
Matthew Hodgson
d1b154a10f support gzip compression, and don't pass through error msgs 2016-04-02 03:06:39 +01:00
Matthew Hodgson
9377157961 how was _respond_default_thumbnail ever meant to work? 2016-04-02 02:31:45 +01:00
Matthew Hodgson
2c838f6459 pass back SVGs as their own thumbnails 2016-04-02 02:30:07 +01:00
Matthew Hodgson
5037ee0d37 handle missing dimensions without crashing 2016-04-02 02:29:57 +01:00
Matthew Hodgson
b26e8604f1 make meta comparisons case insensitive 2016-04-02 01:35:44 +01:00
Matthew Hodgson
5fd07da764 refactor calc_og; spider image URLs; fix xpath; add a (broken) expiringcache; loads of other fixes 2016-04-02 00:35:49 +01:00
Mark Haines
2a37467fa1 Use google style doc strings.
pycharm supports them so there is no need to use the other format.

Might as well convert the existing strings to reduce the risk of
people accidentally cargo culting the wrong doc string format.
2016-04-01 16:12:07 +01:00
Matthew Hodgson
c60b751694 fix assorted redirect, unicode and screenscraping bugs 2016-04-01 02:17:48 +01:00
Matthew Hodgson
683e564815 handle spidered relative images correctly 2016-03-31 23:52:58 +01:00
Matthew Hodgson
72550c3803 prevent choking on invalid utf-8, and handle image thumbnailing smarter 2016-03-31 15:14:14 +01:00
Matthew Hodgson
bb9a2ca87c synthesise basig OG metadata from pages lacking it 2016-03-31 14:15:09 +01:00
Matthew Hodgson
a8a5dd3b44 handle requests with missing content-length headers (e.g. YouTube) 2016-03-31 01:55:21 +01:00
Erik Johnston
08a8514b7a Remove spurious comment 2016-03-30 15:05:33 +01:00
Erik Johnston
fddb6fddc1 Require user to have left room to forget room
This dramatically simplifies the forget API code - in particular it no
longer generates a leave event.
2016-03-30 11:03:00 +01:00
Niklas Riekenbrauck
3f9948a069 Add JWT support 2016-03-29 14:36:36 +02:00
Matthew Hodgson
ae5831d303 fix bugs 2016-03-29 03:32:55 +01:00
Matthew Hodgson
19038582d3 debug 2016-03-29 03:14:16 +01:00
Matthew Hodgson
64b4aead15 make it work 2016-03-29 03:13:25 +01:00
Matthew Hodgson
dd4287ca5d make it build 2016-03-29 02:07:57 +01:00
Matthew Hodgson
e0c2490a14 Merge branch 'develop' into matthew/preview_urls 2016-03-29 01:20:25 +01:00
Matthew Hodgson
d9d48aad2d Merge branch 'develop' into matthew/preview_urls 2016-03-27 22:54:42 +01:00
Mark Haines
191c7bef6b Deduplicate identical /sync requests 2016-03-24 17:47:31 +00:00
Erik Johnston
3e7fac0d56 Add published room list edit API 2016-03-21 15:06:07 +00:00
Erik Johnston
916227b4df Merge pull request #652 from matrix-org/erikj/delete_alias
Update aliases event after deletion
2016-03-18 14:02:46 +00:00
Erik Johnston
2cd9260500 Update aliases event after deletion
Attempt to update the appropriate `m.room.aliases` event after deleting
an alias. This may fail due to the deleter not being in the room.

Will also check if the canonical alias of the event is set to the
deleted alias, and if so will attempt to delete it.
2016-03-17 11:42:00 +00:00
David Baker
5670205e2a remove debug logging 2016-03-16 19:49:42 +00:00
David Baker
f984decd66 Unused import 2016-03-16 19:40:48 +00:00
David Baker
a7daa5ae13 Make registration idempotent, part 2: be idempotent if the client specifies a username. 2016-03-16 19:36:57 +00:00
David Baker
f5e90422f5 take extra return val from check_auth in account too 2016-03-16 14:33:19 +00:00
David Baker
99797947aa pep8 & remove debug logging 2016-03-16 12:51:34 +00:00
David Baker
c12b9d719a Make registration idempotent: if you specify the same session, make it give you an access token for the user that was registered on previous uses of that session. Tweak the UI auth layer to not delete sessions when their auth has completed and hence expire themn so they don't hang around until server restart. Allow server-side data to be associated with UI auth sessions. 2016-03-16 11:56:24 +00:00
Mark Haines
12904932c4 Hook up adding a pusher to the notifier for replication. 2016-03-15 17:42:03 +00:00
Mark Haines
398cd1edfb Fix regression where synapse checked whether push rules were valid JSON before the compatibility hack that handled clients sending invalid JSON 2016-03-14 14:16:41 +00:00
Erik Johnston
494d0c8e02 Merge pull request #642 from matrix-org/erikj/logout
Implement logout
2016-03-11 20:16:25 +00:00
Mark Haines
e9c1cabac2 Use parse_json_object_from_request to parse JSON out of request bodies 2016-03-11 16:41:03 +00:00
Erik Johnston
b13035cc91 Implement logout 2016-03-11 16:27:50 +00:00
David Baker
aa11db5f11 Fix cache invalidation so deleting access tokens (which we did when changing password) actually takes effect without HS restart. Reinstate the code to avoid logging out the session that changed the password, removed in 415c2f0549 2016-03-11 13:14:18 +00:00
blide
40160e24ab Register endpoint returns refresh_token
Guest registration still doesn't return refresh_token
2016-03-10 10:29:19 +03:00
Mark Haines
b7dbe5147a Add a parse_json_object function
to deduplicate all the copy+pasted _parse_json functions. Also document
the parse_.* functions.
2016-03-09 11:26:26 +00:00
Mark Haines
7076082ae6 Fix relative imports so they work in both py3 and py27 2016-03-08 11:45:50 +00:00
Mark Haines
239badea9b Use syntax that works on both py2.7 and py3 2016-03-07 20:13:10 +00:00
Mark Haines
ec7460b4f2 Merge branch 'develop' into markjh/pushrule_stream 2016-03-04 14:44:34 +00:00
Mark Haines
1b4f4a936f Hook up the push rules stream to account_data in /sync 2016-03-04 14:44:01 +00:00
Erik Johnston
dd463e246d Merge pull request #614 from matrix-org/erikj/alias_delete
Allow alias creators to delete aliases
2016-03-04 09:02:33 +00:00
Daniel Wagner-Hall
b4022cc487 Pass whole requester to ratelimiting
This will enable more detailed decisions
2016-03-03 16:43:42 +00:00
Mark Haines
3406eba4ef Move the code for formatting push rules into a separate function 2016-03-03 16:11:59 +00:00
Mark Haines
ddf9e7b302 Hook up the push rules to the notifier 2016-03-03 14:57:45 +00:00
Erik Johnston
f9af8962f8 Allow alias creators to delete aliases 2016-03-01 14:46:31 +00:00
Mark Haines
de27f7fc79 Add support for changing the actions for default rules
See matrix-org/matrix-doc#283

Works by adding dummy rules to the push rules table with a negative priority class and then using those rules to clobber the default rule actions when adding the default rules in ``list_with_base_rules``
2016-02-26 14:28:19 +00:00
Gergely Polonkai
87acd8fb07 Fix to appease the PEP8 dragon 2016-02-26 12:05:38 +01:00
Gergely Polonkai
a53774721a Add error codes for malformed/bad JSON in /login
Signed-off-by: Gergely Polonkai <gergely@polonkaieu>
2016-02-26 10:22:35 +01:00
Mark Haines
15c2ac2cac Make sure we return a JSON object when returning the values of specif…
…ic keys from a push rule
2016-02-25 15:13:07 +00:00
Mark Haines
9892d017b2 Remove unused get_rule_attr method 2016-02-24 16:31:07 +00:00
Daniel Wagner-Hall
869580206d Ignore invalid POST bodies when joining rooms 2016-02-24 08:50:28 +00:00
Daniel Wagner-Hall
577951b032 Allow third_party_signed to be specified on /join 2016-02-23 15:11:25 +00:00
Erik Johnston
e5ad2e5267 Merge pull request #582 from matrix-org/erikj/presence
Rewrite presence for performance.
2016-02-19 09:37:50 +00:00
Erik Johnston
e12ec335a5 "You are not..." 2016-02-18 17:01:53 +00:00
Mark Haines
b9977ea667 Remove dead code for setting device specific rules.
It wasn't possible to hit the code from the API because of a typo
in parsing the request path. Since no-one was using the feature
we might as well remove the dead code.
2016-02-18 16:05:13 +00:00
Daniel Wagner-Hall
7e90fb6a57 Merge branch 'develop' into daniel/roomcleanupincremental
Conflicts:
	synapse/rest/client/v1/room.py
2016-02-17 15:53:59 +00:00
Daniel Wagner-Hall
591af2d074 Some cleanup
I'm not particularly happy with the "action" switching, but there's no
convenient way to defer the work that needs to happen after it, so... :(
2016-02-17 15:50:13 +00:00
Erik Johnston
e5999bfb1a Initial cut 2016-02-17 15:40:50 +00:00
Patrik Oldsberg
71d5d2c669 client/v1/room: include event_id in response to state event PUT, in accordance with the spec
Signed-off-by: Patrik Oldsberg <patrik.oldsberg@ericsson.com>
2016-02-17 11:53:43 +01:00
Mark Haines
458782bf67 Fix typo in request validation for adding push rules. 2016-02-16 18:00:30 +00:00
Daniel Wagner-Hall
4bfb32f685 Branch off member and non member sends
Unclean, needs tidy-up, but works
2016-02-15 18:21:30 +00:00
Daniel Wagner-Hall
1a2197d7bf Simplify room creation code 2016-02-15 18:19:01 +00:00
Daniel Wagner-Hall
e560045cfd Simplify room creation code 2016-02-15 18:18:39 +00:00
Daniel Wagner-Hall
150fcde0dc Reuse update_membership from /join 2016-02-15 16:16:03 +00:00
Daniel Wagner-Hall
e71095801f Merge implementation of /join by alias or ID
This code is kind of rough (passing the remote servers down a long
chain), but is a step towards improvement.
2016-02-15 15:39:16 +00:00
Daniel Wagner-Hall
dbeed36dec Merge some room joining codepaths
Force joining by alias to go through the send_membership_event checks,
rather than bypassing them straight into _do_join. This is the first of
many stages of cleanup.
2016-02-15 14:38:27 +00:00
Daniel Wagner-Hall
4de08a4672 Revert "Merge two of the room join codepaths"
This reverts commit cf81375b94.

It subtly violates a guest joining auth check
2016-02-12 16:17:24 +00:00
Daniel Wagner-Hall
cf81375b94 Merge two of the room join codepaths
There's at least one more to merge in.

Side-effects:
 * Stop reporting None as displayname and avatar_url in some cases
 * Joining a room by alias populates guest-ness in join event
 * Remove unspec'd PUT version of /join/<room_id_or_alias> which has not
   been called on matrix.org according to logs
 * Stop recording access_token_id on /join/room_id - currently we don't
   record it on /join/room_alias; I can try to thread it through at some
   point.
2016-02-12 15:11:49 +00:00
Mark Haines
58c9f20692 Catch the exceptions thrown by twisted when you write to a closed connection 2016-02-12 13:46:59 +00:00
Erik Johnston
a1b7902944 Add some paranoia logging 2016-02-11 09:22:37 +00:00
Daniel Wagner-Hall
ebaa999f92 Revert "Reject additional path segments"
This reverts commit 1d19a5ec0f.

iOS Console is apparently relying on these paths.
2016-02-09 12:46:52 +00:00
Erik Johnston
2bb5f035af Merge pull request #564 from matrix-org/erikj/logcontext
Fix up logcontexts
2016-02-08 15:16:16 +00:00
Erik Johnston
2c1fbea531 Fix up logcontexts 2016-02-08 14:26:45 +00:00
Daniel Wagner-Hall
1d19a5ec0f Reject additional path segments 2016-02-08 10:50:55 +00:00
David Baker
34dda7cc7f Merge pull request #557 from matrix-org/dbkr/profile_dont_return_null
Omit keys rather then returning null in profile API
2016-02-04 15:39:12 +00:00
Erik Johnston
709e09e1c3 Remove old log line 2016-02-03 16:32:20 +00:00
Erik Johnston
33c71c3a4b Preserve log context over when deferring to thread pool in media repo 2016-02-03 16:17:18 +00:00
David Baker
156cea5b45 No braces here 2016-02-03 15:04:51 +00:00
David Baker
24277fbb97 Don't return null if profile display name / avatar url isn't set: omit them instead 2016-02-03 14:59:19 +00:00
Daniel Wagner-Hall
5054806ec1 Rename config field to reflect yaml name 2016-02-03 14:42:01 +00:00
Daniel Wagner-Hall
d83d004ccd Fix flake8 warnings for new flake8 2016-02-02 17:18:50 +00:00
Erik Johnston
04ad93e6fd Merge pull request #545 from matrix-org/erikj/sync
Move /sync state calculations from rest to handler
2016-02-02 15:28:43 +00:00
David Baker
d7ac861d3b Pull guest access token out of the auth session params, otherwise it will break if you open the email on a different device. 2016-02-01 16:33:19 +00:00
Erik Johnston
fa48020a52 Move state calculations from rest to handler 2016-02-01 15:59:40 +00:00
Mark Haines
5687a00e4e Allow three_pid_creds as well as threePidCreds in /account/3pid 2016-01-29 13:26:15 +00:00
Mark Haines
8c94833b72 Fix adding push rules relative to other rules 2016-01-27 10:24:20 +00:00
Mark Haines
7179fdd550 Merge pull request #528 from matrix-org/markjh/missing_yield
Add missing yield in push_rules set enabled
2016-01-25 21:26:30 +01:00
Mark Haines
e18257f0e5 Add missing yield in push_rules set enabled 2016-01-25 16:51:56 +00:00
Erik Johnston
4021f95261 Move logic from rest/ to handlers/ 2016-01-25 10:10:44 +00:00
Matthew Hodgson
7dd0c1730a initial WIP of a tentative preview_url endpoint - incomplete, untested, experimental, etc. just putting it here for safekeeping for now 2016-01-24 18:47:27 -05:00
Erik Johnston
975903ae17 Sanitize filters 2016-01-22 10:41:30 +00:00
Erik Johnston
c43b6dcc75 Fix change_password 2016-01-20 16:14:48 +00:00
Erik Johnston
73ca8e5834 Whine if we give a from param to /sync 2016-01-20 15:42:57 +00:00
Daniel Wagner-Hall
da417aa56d Allow non-guests to peek on rooms using /events 2016-01-20 15:34:07 +00:00
Erik Johnston
ac2842ff1e Only compute unread notifications for rooms we send down stream 2016-01-19 17:19:53 +00:00
Erik Johnston
3adcc4c86a Return highlight_count in /sync 2016-01-19 11:35:50 +00:00
Erik Johnston
0d241e1114 Take a deepcopy of push rules before mutating them 2016-01-19 10:15:12 +00:00
Erik Johnston
d1f56f732e Use static for const dicts 2016-01-18 15:17:56 +00:00
Erik Johnston
0e39dcd135 Remove internal ids 2016-01-18 14:50:17 +00:00
Daniel Wagner-Hall
ac5a4477ad Require unbanning before other membership changes 2016-01-15 16:27:26 +00:00
Richard van der Hoff
2978053d16 Merge branch 'release-v0.12.1' into develop 2016-01-14 15:04:08 +00:00
David Baker
a7927c13fd Fix enabling & disabling push rules 2016-01-14 10:53:44 +00:00
Richard van der Hoff
49f33f6438 Add 'event' result to 'context' endpoint
... because the context isn't much use without the event.
2016-01-13 16:42:14 +00:00
Mark Haines
9c1f853d58 Rename 'user_name' to 'user_id' in push to make it consistent with the rest of the code 2016-01-13 13:32:59 +00:00
Daniel Wagner-Hall
42aa1f3f33 Merge pull request #478 from matrix-org/daniel/userobject
Introduce a User object

I'm sick of passing around more and more things as tuple items around
the whole world, and needing to edit every call site every time there is
more information about a user. So pass them around together as an
object.

This object has incredibly poorly named fields because we have a
convention that `user` indicates a UserID object, and `user_id`
indicates a string. I tried to clean up the whole repo to fix this, but
gave up. So instead, I introduce a second convention. A user_object is a
User, and a user_id_object is a UserId. I may have cried a little bit.
2016-01-11 17:50:22 +00:00
Daniel Wagner-Hall
2110e35fd6 Introduce a Requester object
This tracks data about the entity which made the request. This is
instead of passing around a tuple, which requires call-site
modifications every time a new piece of optional context is passed
around.

I tried to introduce a User object. I gave up.
2016-01-11 17:48:45 +00:00
David Baker
c232780081 Merge pull request #456 from matrix-org/store_event_actions
Send unread notification counts
2016-01-08 14:47:15 +00:00
Mark Haines
8677b7d698 Only use cropped thumbnails when asked for a cropped thumbnail.
Even though ones cropped with scale might be technically valid.
2016-01-07 18:57:15 +00:00
Matthew Hodgson
6c28ac260c copyrights 2016-01-07 04:26:29 +00:00
Daniel Wagner-Hall
11a974da21 Add /_matrix/versions to report supported versions 2016-01-06 18:08:52 +00:00
David Baker
442fcc02f7 Merge remote-tracking branch 'origin/develop' into store_event_actions 2016-01-06 17:28:55 +00:00
Mark Haines
392773ccb2 Guest users must be joined to a room to see it in /sync 2016-01-06 16:44:13 +00:00
David Baker
c79f221192 Add is_guest flag to users db to track whether a user is a guest user or not. Use this so we can run _filter_events_for_client when calculating event_push_actions. 2016-01-06 11:38:09 +00:00
David Baker
eb03625626 Merge remote-tracking branch 'origin/develop' into store_event_actions 2016-01-05 18:39:50 +00:00
Daniel Wagner-Hall
cfd07aafff Allow guests to upgrade their accounts 2016-01-05 18:01:18 +00:00
Erik Johnston
8737ead008 Use larger thumbnail rather than smaller. 2016-01-05 17:30:30 +00:00
David Baker
c77e7e60fc Only joined rooms have unread_notif_count 2016-01-04 15:49:06 +00:00
David Baker
928c575c6f Merge remote-tracking branch 'origin/develop' into store_event_actions 2016-01-04 13:39:51 +00:00
Richard van der Hoff
32d9fd0b26 Expose /login under r0
The spec says /login should be available at r0 and 'unstable', so make it so.
2016-01-02 17:24:28 +00:00
David Baker
140a50f641 Merge remote-tracking branch 'origin/develop' into store_event_actions 2015-12-22 17:23:35 +00:00