Commit graph

179 commits

Author SHA1 Message Date
Tulir Asokan
cf45cfd314 Merge remote-tracking branch 'upstream/release-v1.46' 2021-10-27 15:42:34 +03:00
Patrick Cloke
efd0074ab7
Ensure each charset is attempted only once during media preview. (#11089)
There's no point in trying more than once since it is guaranteed to
continually fail.
2021-10-14 18:51:44 +00:00
Patrick Cloke
e2f0b49b3f
Attempt different character encodings when previewing a URL. (#11077)
This follows similar logic to BeautifulSoup where we attempt different
character encodings until we find one which works.
2021-10-14 10:17:20 -04:00
Patrick Cloke
732bbf6737
Be more lenient when parsing the version for oEmbed responses. (#11065) 2021-10-13 07:00:07 -04:00
Tulir Asokan
80adb0a6ca Merge remote-tracking branch 'upstream/release-v1.45' 2021-10-12 13:54:46 +03:00
Patrick Cloke
1b112840d2
Autodiscover oEmbed endpoint from returned HTML (#10822)
Searches the returned HTML for an oEmbed endpoint using the
autodiscovery mechanism (`<link rel=...>`), and will request it
to generate the preview.
2021-10-08 14:14:42 -04:00
Tulir Asokan
8631aaeb5a Merge remote-tracking branch 'upstream/release-v1.44' 2021-09-29 16:08:42 +03:00
Sean Quah
2be0fde3d6
Fix empty url_cache_thumbnails/yyyy-mm-dd/ directories being left behind (#10924) 2021-09-29 10:24:37 +01:00
Sean Quah
f7768f62cb
Avoid storing URL cache files in storage providers (#10911)
URL cache files are short-lived and it does not make sense to offload
them (eg. to the cloud) or back them up.
2021-09-27 12:55:27 +01:00
Patrick Cloke
bb7fdd821b
Use direct references for configuration variables (part 5). (#10897) 2021-09-24 07:25:21 -04:00
Erik Johnston
50022cff96
Add reactor to SynapseRequest and fix up types. (#10868) 2021-09-24 11:01:25 +01:00
Patrick Cloke
6fc8be9a1b
Include more information in oEmbed previews. (#10819)
* Improved titles (fall back to the author name if there's not title) and include the site name.
* Handle photo/video payloads.
* Include the original URL in the Open Graph response.
* Fix the expiration time (by properly converting from seconds to milliseconds).
2021-09-22 09:45:20 -04:00
Patrick Cloke
ba7a91aea5
Refactor oEmbed previews (#10814)
The major change is moving the decision of whether to use oEmbed
further up the call-stack. This reverts the _download_url method to
being a "dumb" functionwhich takes a single URL and downloads it
(as it was before #7920).

This also makes more minor refactorings:

* Renames internal variables for clarity.
* Factors out shared code between the HTML and rich oEmbed
  previews.
* Fixes tests to preview an oEmbed image.
2021-09-21 16:09:57 +00:00
Patrick Cloke
b93259082c
Add missing type hints to non-client REST servlets. (#10817)
Including admin, consent, key, synapse, and media. All REST servlets
(the synapse.rest module) now require typed method definitions.
2021-09-15 08:45:32 -04:00
Tulir Asokan
1e11b73441 Merge remote-tracking branch 'upstream/release-v1.43' 2021-09-14 10:00:40 -04:00
Patrick Cloke
89ba834818
Use attrs internally for the URL preview code & add documentation. (#10753) 2021-09-07 13:10:34 +00:00
Patrick Cloke
e2481dbe93
Allow configuration of the oEmbed URLs. (#10714)
This adds configuration options (under an `oembed` section) to
configure which URLs are matched to use oEmbed for URL
previews.
2021-08-31 18:37:07 -04:00
Tulir Asokan
7359964d9f Merge remote-tracking branch 'upstream/release-v1.40' 2021-08-03 14:38:58 +03:00
sri-vidyut
8e1febc6a1
Support underscores (in addition to hyphens) for charset detection. (#10410) 2021-07-27 17:29:42 +00:00
Patrick Cloke
5db118626b
Add a return type to parse_string. (#10438)
And set the required attribute in a few places which will error if
a parameter is not provided.
2021-07-21 09:47:56 -04:00
Tulir Asokan
9754df5623 Merge remote-tracking branch 'upstream/release-v1.39' 2021-07-20 16:33:01 +03:00
Jonathan de Jong
98aec1cc9d
Use inline type hints in handlers/ and rest/. (#10382) 2021-07-16 18:22:36 +01:00
Tulir Asokan
c9cee6c534 Merge remote-tracking branch 'upstream/release-v1.33.0' 2021-04-28 14:15:25 +03:00
Jonathan de Jong
4b965c862d
Remove redundant "coding: utf-8" lines (#9786)
Part of #9744

Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now.

`Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14 15:34:27 +01:00
Tulir Asokan
dbe12b3eb1 Merge remote-tracking branch 'upstream/release-v1.32.0' 2021-04-13 16:44:48 +03:00
Patrick Cloke
44bb881096
Add type hints to expiring cache. (#9730) 2021-04-06 08:58:18 -04:00
Tulir Asokan
7def4428cf Merge remote-tracking branch 'upstream/release-v1.31.0' 2021-03-30 14:03:23 +03:00
Erik Johnston
b5efcb577e
Make it possible to use dmypy (#9692)
Running `dmypy run` will do a `mypy` check while spinning up a daemon
that makes rerunning `dmypy run` a lot faster.

`dmypy` doesn't support `follow_imports = silent` and has
`local_partial_types` enabled, so this PR enables those options and
fixes the issues that were newly raised. Note that `local_partial_types`
will be enabled by default in upcoming mypy releases.
2021-03-26 16:49:46 +00:00
Patrick Cloke
b7748d3c00
Import HomeServer from the proper module. (#9665) 2021-03-23 07:12:48 -04:00
Tulir Asokan
8b753230af Merge remote-tracking branch 'upstream/release-v1.30.0' 2021-03-16 16:58:38 +02:00
Patrick Cloke
55da8df078
Fix additional type hints from Twisted 21.2.0. (#9591) 2021-03-12 11:37:57 -05:00
Tulir Asokan
cb663b97e3 Add missing comma 2021-03-04 12:53:44 +02:00
Tulir Asokan
adb990d8ba Merge remote-tracking branch 'upstream/release-v1.29.0' 2021-03-04 12:49:37 +02:00
Patrick Cloke
a0bc9d387e
Use the proper Request in type hints. (#9515)
This also pins the Twisted version in the mypy job for CI until
proper type hints are fixed throughout Synapse.
2021-03-01 12:23:46 -05:00
Tim Leung
ddb240293a
Add support for no_proxy and case insensitive env variables (#9372)
### Changes proposed in this PR

- Add support for the `no_proxy` and `NO_PROXY` environment variables
  - Internally rely on urllib's [`proxy_bypass_environment`](bdb941be42/Lib/urllib/request.py (L2519))
- Extract env variables using urllib's `getproxies`/[`getproxies_environment`](bdb941be42/Lib/urllib/request.py (L2488)) which supports lowercase + uppercase, preferring lowercase, except for `HTTP_PROXY` in a CGI environment

This does contain behaviour changes for consumers so making sure these are called out:
- `no_proxy`/`NO_PROXY` is now respected
- lowercase `https_proxy` is now allowed and taken over `HTTPS_PROXY`

Related to #9306 which also uses `ProxyAgent`

Signed-off-by: Timothy Leung tim95@hotmail.co.uk
2021-02-26 17:37:57 +00:00
Tulir Asokan
f6684e4e55 Merge remote-tracking branch 'upstream/release-v1.28.0' 2021-02-19 15:17:45 +02:00
Eric Eastwood
0a00b7ff14
Update black, and run auto formatting over the codebase (#9381)
- Update black version to the latest
 - Run black auto formatting over the codebase
    - Run autoformatting according to [`docs/code_style.md
`](80d6dc9783/docs/code_style.md)
 - Update `code_style.md` docs around installing black to use the correct version
2021-02-16 22:32:34 +00:00
Patrick Cloke
0963d39ea6
Handle additional errors when previewing URLs. (#9333)
* Handle the case of lxml not finding a document tree.
* Parse the document encoding from the XML tag.
2021-02-08 12:33:30 -05:00
Tulir Asokan
7f7fb9b566 Merge remote-tracking branch 'upstream/release-v1.27.0' 2021-02-02 16:06:33 +02:00
Patrick Cloke
4937fe3d6b
Try to recover from unknown encodings when previewing media. (#9164)
Treat unknown encodings (according to lxml) as UTF-8
when generating a preview for HTML documents. This
isn't fully accurate, but will hopefully give a reasonable
title and summary.
2021-01-26 07:32:17 -05:00
Tulir Asokan
d3547df958 Merge remote-tracking branch 'upstream/release-v1.26.0' 2021-01-20 18:06:27 +02:00
Patrick Cloke
d34c6e1279
Add type hints to media rest resources. (#9093) 2021-01-15 10:57:37 -05:00
Tulir Asokan
f461e13192 Merge remote-tracking branch 'upstream/release-v1.25.0' 2021-01-06 14:44:59 +02:00
Patrick Cloke
1f3748f033
Do not raise a 500 exception when previewing empty media. (#8883) 2020-12-07 10:00:08 -05:00
Tulir Asokan
55d926999c Fix URL previewer user agent to fix Twitter previews 2020-11-18 14:13:47 +02:00
Richard van der Hoff
11c9e17738
Add type annotations to SimpleHttpClient (#8372) 2020-09-24 15:47:20 +01:00
Patrick Cloke
aec294ee0d
Use slots in attrs classes where possible (#8296)
slots use less memory (and attribute access is faster) while slightly
limiting the flexibility of the class attributes. This focuses on objects
which are instantiated "often" and for short periods of time.
2020-09-14 12:50:06 -04:00
Patrick Cloke
4e874ed593
Remove unnecessary maybeDeferred calls (#8044) 2020-08-07 09:44:48 -04:00
David Vo
4dd27e6d11
Reduce unnecessary whitespace in JSON. (#7372) 2020-08-07 08:02:55 -04:00
Erik Johnston
a7bdf98d01
Rename database classes to make some sense (#8033) 2020-08-05 21:38:57 +01:00