Commit graph

19094 commits

Author SHA1 Message Date
Andrew Morgan
182147195b
Check third party rules before persisting knocks over federation (#10212)
An accidental mis-ordering of operations during #6739 technically allowed an incoming knock event over federation in before checking it against any configured Third Party Access Rules modules.

This PR corrects that by performing the TPAR check *before* persisting the event.
2021-06-21 11:57:09 +01:00
Richard van der Hoff
107c06081f
Ensure that errors during startup are written to the logs and the console. (#10191)
* Defer stdio redirection until we are about to start the reactor

* Catch and handle exceptions during startup
2021-06-21 11:41:25 +01:00
Andrew Morgan
7c536d0fef
Deploy a documentation version for each new Synapse release (#10198)
This PR will run a new "Deploy release-specific documentation" job whenever a push to a branch name matching `release-v*` occurs. Doing so will create/add to a folder named `vX.Y` on the `gh-pages` branch. Doing so will allow us to build up `major.minor` releases of the docs as we release Synapse.

This is especially useful for having a mechanism for keeping around documentation of old/removed features (for those running older versions of Synapse), without needing to clutter the latest copy of the docs.

After a [discussion](https://matrix.to/#/!XaqDhxuTIlvldquJaV:matrix.org/$rKmkBmQle8OwTlGcoyu0BkcWXdnHW3_oap8BMgclwIY?via=matrix.org&via=vector.modular.im&via=envs.net) in #synapse-dev, we wanted to use tags to trigger the documentation deployments, which I agreed with. However, I soon realised that the bash-foo required to turn a tag of `v1.2.3rc1` into `1.2` was a lot more complex than the branch's `release-v1.2`. So, I've gone with the latter for simplicity.

In the future we'll have some UI on the website to switch between versions, but for now you can simply just change 'develop' to 'v1.2' in the URL.
2021-06-18 19:26:25 +01:00
Patrick Cloke
0bd968921c
Fix a missing await when in the spaces summary. (#10208)
This could cause a minor data leak if someone defined a non-restricted join rule
with an allow key or used a restricted join rule in an older room version, but this is
unlikely.

Additionally this starts adding unit tests to the spaces summary handler.
2021-06-18 18:41:33 +01:00
Brendan Abolivier
e9f2ad8603
Describe callbacks signatures as async in new modules doc (#10206) 2021-06-18 16:55:53 +02:00
Brendan Abolivier
1b3e398bea
Standardise the module interface (#10062)
This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
2021-06-18 12:15:52 +01:00
Richard van der Hoff
91fa9cca99
Expose opentracing trace id in response headers (#10199)
Fixes: #9480
2021-06-18 11:43:22 +01:00
Brendan Abolivier
08c8469322
Remove support for ACME v1 (#10194)
Fixes #9778

ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
2021-06-17 18:56:48 +01:00
Patrick Cloke
8c97d5863f
Update MSC3083 support per changes in the MSC. (#10189)
Adds a "type" field and generalize "space" to "room_id".
2021-06-17 12:53:27 -04:00
Richard van der Hoff
fcf3c7032b
Ensure that we do not cache empty sync responses after a timeout (#10158)
Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.
2021-06-17 16:23:11 +01:00
Richard van der Hoff
9cf6e0eae7
Rip out the DNS lookup limiter (#10190)
As I've written in various places in the past (#7113, #9865) I'm pretty sure this is doing nothing useful at all.
2021-06-17 16:22:41 +01:00
Marcus
8070b893db
update black to 21.6b0 (#10197)
Reformat all files with the new version.

Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
2021-06-17 15:20:06 +01:00
Andrew Morgan
6f1a28de19
Fix incorrect time magnitude on delayed call (#10195)
Fixes https://github.com/matrix-org/synapse/issues/10030.

We were expecting milliseconds where we should have provided a value in seconds.

The impact of this bug isn't too bad. The code is intended to count the number of remote servers that the homeserver can see and report that as a metric. This metric is supposed to run initially 1 second after server startup, and every 60s as well. Instead, it ran 1,000 seconds after server startup, and every 60s after startup.

This fix allows for the correct metrics to be collected immediately, as well as preventing a random collection 1,000s in the future after startup.
2021-06-17 15:04:26 +01:00
Eric Eastwood
a911dd768b
Add fields to better debug where events are being soft_failed (#10168)
Follow-up to https://github.com/matrix-org/synapse/pull/10156#discussion_r650292223
2021-06-17 14:59:45 +01:00
Richard van der Hoff
52c60bd0a9
Fix persist_events to stop leaking opentracing contexts (#10193) 2021-06-17 11:21:53 +01:00
Patrick Cloke
18edc9ab06
Improve comments in the structured logging code. (#10188) 2021-06-16 19:18:02 +01:00
Patrick Cloke
76f9c701c3
Always require users to re-authenticate for dangerous operations. (#10184)
Dangerous actions means deactivating an account, modifying an account
password, or adding a 3PID.

Other actions (deleting devices, uploading keys) can re-use the same UI
auth session if ui_auth.session_timeout is configured.
2021-06-16 11:07:28 -04:00
Michael Kaye
b8b282aa32
A guide to the request log lines format. (#8436)
This doc is short but a useful guide to what the request log lines mean.

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
Co-authored-by: Erik Johnston <erik@matrix.org>
Co-authored-by: Daniele Sluijters <daenney@users.noreply.github.com>
2021-06-16 13:31:55 +01:00
Erik Johnston
36c426e294
Add debug logging when we enter/exit Measure block (#10183)
It can be helpful to know when trying to track down slow requests.
2021-06-16 13:29:54 +01:00
Lukas Lihotzki
2c240213f4
Fix requestOpenIdToken response: integer expires_in (#10175)
`expires_in` must be an integer according to the OpenAPI spec:
https://github.com/matrix-org/matrix-doc/blob/master/data/api/client-server/definitions/openid_token.yaml#L32

True division (`/`) returns a float instead (`"expires_in": 3600.0`).
Floor division (`//`) returns an integer, so the response is spec compliant.

Signed-off-by: Lukas Lihotzki <lukas@lihotzki.de>
2021-06-16 13:16:35 +01:00
Dirk Klimpel
0adc2882c1
Fix broken links in documentation (#10180)
* Fix broken links in documentation

* newsfile
2021-06-16 13:15:52 +01:00
Richard van der Hoff
9e405034e5
Make opentracing trace into event persistence (#10134)
* Trace event persistence

When we persist a batch of events, set the parent opentracing span to the that
from the request, so that we can trace all the way in.

* changelog

* When we force tracing, set a baggage item

... so that we can check again later.

* Link in both directions between persist_events spans
2021-06-16 11:41:15 +01:00
Erik Johnston
d09e24a52d Merge branch 'master' into develop 2021-06-15 15:52:24 +01:00
Erik Johnston
1c8045f674 1.36.0 2021-06-15 15:42:02 +01:00
Patrick Cloke
4911f7931d
Remove support for unstable MSC1772 prefixes. (#10161)
The stable prefixes have been supported since v1.34.0. The unstable
prefixes are not supported by any known clients.
2021-06-15 08:03:17 -04:00
Patrick Cloke
9e5ab6dd58
Remove the experimental flag for knocking and use stable prefixes / endpoints. (#10167)
* Room version 7 for knocking.
* Stable prefixes and endpoints (both client and federation) for knocking.
* Removes the experimental configuration flag.
2021-06-15 07:45:14 -04:00
Michael Kutzner
aac2c49b9b
Fix 'ip_range_whitelist' not working for federation servers (#10115)
Add 'federation_ip_range_whitelist'. This allows backwards-compatibility, If 'federation_ip_range_blacklist' is set. Otherwise 'ip_range_whitelist' will be used for federation servers.

Signed-off-by: Michael Kutzner 1mikure@gmail.com
2021-06-15 08:53:55 +01:00
Richard van der Hoff
1dfdc87b9b
Refactor EventPersistenceQueue (#10145)
some cleanup, pulled out of #10134.
2021-06-14 11:59:27 +01:00
Richard van der Hoff
d7808a2dde
Extend ResponseCache to pass a context object into the callback (#10157)
This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug.

The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
2021-06-14 10:26:09 +01:00
Richard van der Hoff
13577aa55e
Notes on boolean columns in database schemas (#10164) 2021-06-11 17:13:56 +01:00
Erik Johnston
29966a285d Synapse 1.36.0rc2 (2021-06-11)
==============================
 
 Bugfixes
 --------
 
 - Fix a bug which caused  presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](https://github.com/matrix-org/synapse/issues/10149))
 - Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](https://github.com/matrix-org/synapse/issues/10163))
 - Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](https://github.com/matrix-org/synapse/issues/10165))
 -----BEGIN PGP SIGNATURE-----
 
 iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmDDdrcQHGVyaWtAbWF0
 cml4Lm9yZwAKCRClQuTtGw+sCTzfB/4qaTqW2mBiwjf52SmOu6HNyd8uQd6nLIAZ
 mJC218Wakh2tT0W4iVkKwUgpuHFbtcy0rSNTlXtW4kG8XzhpTvT56RH9qls99aD3
 SGKqYpOv6NkWZibN6NdVvLDW85ixficDTXco3BljRCIMlORhY0swy+LWLwksdjWj
 6kQ+Gi/QAtKP3Pt5epYs0Ix5o1T94DfZOWE//mqBhG5cMDAw/K/G/c8tRfEjclt9
 wACBjmt2fw/Lbn9j3b0feNVp+xnFcFNuAK2bSEd8Y3yph1mhjdsIszULnM7IFNsR
 Q8zg+i7PJKNq8pQjei8j8T/aKscPTPH5XGqOSLlizj15snsiwlkz
 =C6lV
 -----END PGP SIGNATURE-----

Merge tag 'v1.36.0rc2' into develop

Synapse 1.36.0rc2 (2021-06-11)
==============================

Bugfixes
--------

- Fix a bug which caused  presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](https://github.com/matrix-org/synapse/issues/10149))
- Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](https://github.com/matrix-org/synapse/issues/10163))
- Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](https://github.com/matrix-org/synapse/issues/10165))
2021-06-11 15:46:38 +01:00
Erik Johnston
cbf350db63 Fixup changelog 2021-06-11 15:30:42 +01:00
Erik Johnston
fb10a73e85 1.36.0rc2 2021-06-11 15:21:34 +01:00
Erik Johnston
cdd985c64f Only send a presence state to a destination once (#10165)
It turns out that we were sending the same presence state to a remote
potentially multiple times.
2021-06-11 15:21:08 +01:00
Erik Johnston
5e0b4719ea Fix sending presence over federation when using workers (#10163)
When using a federation sender we'd send out all local presence updates over
federation even when they shouldn't be.

Fixes #10153.
2021-06-11 15:20:54 +01:00
Erik Johnston
c955f22e2c Fix bug when running presence off master (#10149)
Hopefully fixes #10027.
2021-06-11 15:20:45 +01:00
Erik Johnston
968f8283b4
Only send a presence state to a destination once (#10165)
It turns out that we were sending the same presence state to a remote
potentially multiple times.
2021-06-11 15:19:42 +01:00
Richard van der Hoff
c1b9922498
Support for database schema version ranges (#9933)
This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.
2021-06-11 14:45:53 +01:00
Patrick Cloke
a14884fbb0
Use the matching complement branch when running tests in CI. (#10160)
This implements similar behavior to sytest where a matching branch is used,
if one exists. This is useful when needing to modify both application code
and tests at the same time. The following rules are used to find a matching
complement branch:

1. Search for the branch name of the pull request. (E.g. feature/foo.)
2. Search for the base branch of the pull request. (E.g. develop or release-vX.Y.)
3. Search for the reference branch of the commit. (E.g. master or release-vX.Y.)
4. Fallback to 'master', the default complement branch name.
2021-06-11 08:17:17 -04:00
Erik Johnston
c8dd4db9eb
Fix sending presence over federation when using workers (#10163)
When using a federation sender we'd send out all local presence updates over
federation even when they shouldn't be.

Fixes #10153.
2021-06-11 13:08:30 +01:00
Andrew Morgan
a15a046c93
Clean up a broken import in admin_cmd.py (#10154) 2021-06-11 11:34:40 +01:00
Erik Johnston
d26d15ba3d
Fix bug when running presence off master (#10149)
Hopefully fixes #10027.
2021-06-11 10:27:12 +01:00
Eric Eastwood
b31daac01c
Add metrics to track how often events are soft_failed (#10156)
Spawned from missing messages we were seeing on `matrix.org` from a
federated Gtiter bridged room, https://gitlab.com/gitterHQ/webapp/-/issues/2770.
The underlying issue in Synapse is tracked by https://github.com/matrix-org/synapse/issues/10066
where the message and join event race and the message is `soft_failed` before the
`join` event reaches the remote federated server.

Less soft_failed events = better and usually this should only trigger for events
where people are doing bad things and trying to fuzz and fake everything.
2021-06-11 10:12:35 +01:00
Eric Eastwood
e21c347332
Document how to see logger output when running the twisted tests (#10148) 2021-06-11 09:57:34 +01:00
Aaron Raimist
e6245e6d48
Mention that you need to configure max upload size in reverse proxy as well (#10122)
Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-06-10 11:40:24 +01:00
Andrew Morgan
aec2cf1c98
Update Complement run with Synapse-supported MSC-related build tags (#10155)
This PR updates the build tags that we perform Complement runs with to match our [buildkite pipeline](618b3e90bc/synapse/pipeline.yml (L570)), as well as adding `msc2403` (as it will be required once #9359 is merged). Build tags are what we use to determine which tests to run in Complement (really it determines which test files are compiled into the final binary).

I haven't put in a comment about updating the buildkite side here, as we've decided to migrate fully to GitHub Actions anyhow.
2021-06-09 20:59:40 +01:00
Andrew Morgan
a7a37437bc
Integrate knock rooms with the public rooms directory (#9359)
This PR implements the ["Changes regarding the Public Rooms Directory"](https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#changes-regarding-the-public-rooms-directory) section of knocking MSC2403.

Specifically, it:

* Allows rooms with `join_rule` "knock" to be returned by the query behind the public rooms directory
* Adds the field `join_rule` to each room entry returned by a public rooms directory query, so clients can know whether to attempt a join or knock on a room

Based on https://github.com/matrix-org/synapse/issues/6739. Complement tests for this change: https://github.com/matrix-org/complement/pull/72
2021-06-09 20:31:31 +01:00
Sorunome
d936371b69
Implement knock feature (#6739)
This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403

Signed-off-by: Sorunome mail@sorunome.de
Signed-off-by: Andrew Morgan andrewm@element.io
2021-06-09 19:39:51 +01:00
Patrick Cloke
11846dff8c
Limit the number of in-flight /keys/query requests from a single device. (#10144) 2021-06-09 07:05:32 -04:00
Richard van der Hoff
1bf83a191b
Clean up the interface for injecting opentracing over HTTP (#10143)
* Remove unused helper functions

* Clean up the interface for injecting opentracing over HTTP

* changelog
2021-06-09 11:33:00 +01:00