3666 commits

Author	SHA1	Message	Date
Shay	429ecb7564	Handle remote download responses with UNKNOWN_LENGTH more gracefully (#17439) ... Prior to this PR, remote downloads which did not provide a `content-length` were decremented from the remote download ratelimiter at the max allowable size, leading to excessive ratelimiting - see https://github.com/element-hq/synapse/issues/17394. This PR adds a linearizer to limit concurrent remote downloads to 6 per IP address, and decrements remote downloads without a `content-length` from the ratelimiter *after* the download is complete and the response length is known. Also adds logic to ensure that responses with a known length respect the `max_download_size`.	2024-07-16 11:13:55 +01:00
Shay	899d33f2ba	Remove unnecessary call to resume producing in fake channel (#17449) ... This fell out of the authenticated media work - this bit of code masked a bug but does not break anything when removed, so probably should be removed.	2024-07-16 09:52:39 +01:00
Erik Johnston	df11af14db	Fix bug where sync could get stuck when using workers (#17438) ... This is because we serialized the token wrong if the instance map contained entries from before the minimum token.	2024-07-15 16:13:04 +01:00
Eric Eastwood	ab62aa09da	Add room subscriptions to Sliding Sync /sync (#17432) ... Add room subscriptions to Sliding Sync `/sync` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Currently, you can only subscribe to rooms you have had *any* membership in before. In the future, we will allow `world_readable` rooms to be subscribed to without joining.	2024-07-15 10:37:10 +01:00
Eric Eastwood	fb66e938b2	Add is_dm room field to Sliding Sync /sync (#17429) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-11 18:19:26 -05:00
Eric Eastwood	5a97bbd895	Add heroes and room summary fields to Sliding Sync /sync (#17419) ... Additional room summary fields: `joined_count`, `invited_count` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-11 14:05:38 -05:00
Erik Johnston	606da398fc	Fix filtering room types on remote rooms (#17434) ... We can only fetch room types for rooms the server is in, so we need to only filter rooms that we're joined to. Also includes a perf fix to bulk fetch room types.	2024-07-11 16:00:44 +01:00
Erik Johnston	4ca13ce0dd	Handle to-device extensions to Sliding Sync (#17416) ... Implements MSC3885 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-10 11:58:42 +01:00
Eric Eastwood	1cf3ff6b40	Add rooms name and avatar to Sliding Sync /sync (#17418) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-09 12:26:45 -05:00
Erik Johnston	8cdd2d214e	Fix bug in sliding sync when using old DB. (#17398) ... We don't necessarily have `instance_name` for old events (before we support multiple event persisters). We treat those as if the `instance_name` was "master". --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-08 20:30:23 +01:00
Eric Eastwood	3fef535ff2	Add rooms.bump_stamp to Sliding Sync /sync for easier client-side sorting (#17395) ... `bump_stamp` corresponds to the `stream_ordering` of the latest `DEFAULT_BUMP_EVENT_TYPES` in the room. This helps clients sort more readily without them needing to pull in a bunch of the timeline to determine the last activity. `bump_event_types` is a thing because for example, we don't want display name changes to mark the room as unread and bump it to the top. For encrypted rooms, we just have to consider any activity as a bump because we can't see the content and the client has to figure it out for themselves. Outside of Synapse, `bump_stamp` is just a free-form counter so other implementations could use `received_ts`or `origin_server_ts` (see the [*Security considerations* section in MSC3575 about the potential pitfalls of using `origin_server_ts`](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md#security-considerations)). It doesn't have any guarantee about always going up. In the Synapse case, it could go down if an event was redacted/removed (or purged in cases of retention policies). In the future, we could add `bump_event_types` as [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) mentions if people need to customize the event types. --- In the Sliding Sync proxy, a similar [`timestamp` field was added](https://github.com/matrix-org/sliding-sync/pull/247) for the same purpose but the name is not obvious what it pertains to or what it's for. The `timestamp` field was also added to Ruma in https://github.com/ruma/ruma/pull/1622	2024-07-08 13:17:08 -05:00
Shay	cf69f8d59b	Support MSC3916 by adding a federation /thumbnail endpoint and authenticated _matrix/client/v1/media/thumbnail endpoint (#17388) ... [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916) added the endpoints `_matrix/federation/v1/media/thumbnail` and the authenticated `_matrix/client/v1/media/thumbnail`. This PR implements those endpoints, along with stabilizing `_matrix/client/v1/media/config` and `_matrix/client/v1/media/preview_url`. Complement tests are at https://github.com/matrix-org/complement/pull/728	2024-07-08 10:11:20 +01:00
dependabot[bot]	20de685a4b	Bump ruff from 0.3.7 to 0.5.0 (#17381)	2024-07-05 12:35:57 +00:00
Erik Johnston	57538eb4d9	Finish up work to allow per-user feature flags (#17392) ... Follows on from @H-Shay's great work at https://github.com/matrix-org/synapse/pull/15344 and MSC4026. Also enables its use for MSC3881, mainly as an easy but concrete example of how to use it.	2024-07-05 13:02:35 +01:00
Travis Ralston	2ec257d608	Upon deactivation, forget all of the user's rooms (#17400) ... This can help ensure that the rooms are eventually purged if the other local users also forget them. Synapse already clears some of the room information as part of the `_background_remove_left_rooms` background task, but this doesn't catch `events`, `event_json`, etc.	2024-07-05 11:56:52 +01:00
Eric Eastwood	22aeb78b77	Add rooms.required_state to Sliding Sync /sync (#17342) ... Also handles excluding rooms with partial state when people are asking for room membership events unless it's `$LAZY` room membership.	2024-07-04 12:25:36 -05:00
Eric Eastwood	7be03d854b	Add room_types/not_room_types filtering to Sliding Sync /sync (#17337) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-02 12:46:27 -05:00
Eric Eastwood	fa91655805	Return some room data in Sliding Sync /sync (#17320) ... - Timeline events - Stripped `invite_state` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-02 11:07:05 -05:00
Erik Johnston	99c107920d	Merge remote-tracking branch 'origin/release-v1.110' into develop	2024-07-02 15:12:36 +01:00
Erik Johnston	1609855ff8	Limit size of presence EDUs (#17371) ... Otherwise they are unbounded. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2024-07-02 14:07:11 +01:00
Shay	8f890447b0	Support MSC3916 by adding _matrix/client/v1/media/download endpoint (#17365)	2024-07-02 14:07:04 +01:00
Erik Johnston	b905ae27ca	Fix regression when bounding future tokens (#17391) ... Fix bug added in #17386, where we accidentally used `room_key` for the receipts stream. See first commit. Reviewable commit-by-commit	2024-07-02 14:06:36 +01:00
Erik Johnston	1ce59d7ba0	Fix sync waiting for an invalid token from the "future" (#17386) ... Fixes https://github.com/element-hq/synapse/issues/17274, hopefully. Basically, old versions of Synapse could advance streams without persisting anything in the DB (fixed in #17229). On restart those updates would get lost, and so the position of the stream would revert to an older position. If this happened across an upgrade to a later Synapse version which included #17215, then sync could get blocked indefinitely (until the stream advanced to the position in the token). We fix this by bounding the stream positions we'll wait for to the maximum position of the underlying stream ID generator.	2024-07-02 12:40:03 +01:00
Erik Johnston	b3b793786c	Fix sync waiting for an invalid token from the "future" (#17386) ... Fixes https://github.com/element-hq/synapse/issues/17274, hopefully. Basically, old versions of Synapse could advance streams without persisting anything in the DB (fixed in #17229). On restart those updates would get lost, and so the position of the stream would revert to an older position. If this happened across an upgrade to a later Synapse version which included #17215, then sync could get blocked indefinitely (until the stream advanced to the position in the token). We fix this by bounding the stream positions we'll wait for to the maximum position of the underlying stream ID generator.	2024-07-02 12:39:49 +01:00
Erik Johnston	2f5a77efae	Limit size of presence EDUs (#17371) ... Otherwise they are unbounded. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2024-07-01 12:48:36 +01:00
Shay	a023538822	Re-introduce federation /download endpoint (#17350)	2024-06-25 14:35:37 +00:00
Erik Johnston	554a92601a	Reintroduce "Reduce device lists replication traffic."" (#17361) ... Reintroduces https://github.com/element-hq/synapse/pull/17333 Turns out the reason for revert was down two master instances running	2024-06-25 10:34:34 +01:00
Erik Johnston	a98cb87bee	Revert "Reduce device lists replication traffic." (#17360) ... Reverts element-hq/synapse#17333 It looks like master was still sending out replication RDATA with the old format... somehow	2024-06-25 09:57:34 +01:00
Eric Eastwood	6e8af83193	Add is_invite filtering to Sliding Sync /sync (#17335) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-06-24 19:07:56 -05:00
Erik Johnston	930a64b6c1	Reintroduce #17291. (#17338) ... This is #17291 (which got reverted), with some added fixups, and change so that tests actually pick up the error. The problem was that we were not calculating any new chain IDs due to a missing `not` in a condition.	2024-06-24 14:40:28 +00:00
Shay	7a11c0ac4f	Add support for MSC3823 - Account Suspension Part 2 (#17255)	2024-06-24 14:21:22 +01:00
Erik Johnston	cf711ac03c	Reduce device lists replication traffic. (#17333) ... Reduce the replication traffic of device lists, by not sending every destination that needs to be sent the device list update over replication. Instead a "hosts to send to have been calculated" notification over replication, and then federation senders read the destinations from the DB. For non federation senders this should heavily reduce the impact of a user in many large rooms changing a device.	2024-06-24 14:15:13 +01:00
Erik Johnston	4243c1f074	Revert "Handle large chain calc better (#17291)" (#17334) ... This reverts commit bdf82efea5 (#17291) This seems to have stopped persisting auth chains for new events, and so is causing state res to fall back to the slow methods	2024-06-19 17:39:33 +01:00
Erik Johnston	3239b7459c	Register sliding sync under a different path (#17331) ... As the API is slightly incompatible.	2024-06-19 16:18:45 +00:00
Alexander Fechler	9104a9f0d0	Filter added to Admin-API GET /rooms (#17276)	2024-06-19 11:45:48 +01:00
Erik Johnston	bdf82efea5	Handle large chain calc better (#17291) ... We calculate the auth chain links outside of the main persist event transaction to ensure that we do not block other event sending during the calculation.	2024-06-19 10:33:53 +01:00
Andrew Morgan	afaf2d9388	Require the 'from' parameter for /notifications be an integer (#17283) ... Co-authored-by: Erik Johnston <erikj@element.io>	2024-06-19 10:05:39 +01:00
Andrew Morgan	199223062a	Revert "Support MSC3916 by adding a federation /download endpoint" (#17325)	2024-06-18 16:54:19 +01:00
Eric Eastwood	a5485437cf	Add is_encrypted filtering to Sliding Sync /sync (#17281) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-06-17 12:06:18 -05:00
Eric Eastwood	e5b8a3e37f	Add stream_ordering sort to Sliding Sync /sync (#17293) ... Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.	2024-06-17 11:27:14 -05:00
Quentin Gliech	e88332b5f4	Merge branch 'release-v1.109' into develop	2024-06-17 15:51:16 +02:00
Quentin Gliech	f983a77ab0	Set our own stream position from the current sequence value on startup (#17309)	2024-06-17 11:50:00 +00:00
Erik Johnston	a3cb244755	Automatically apply SQL for inconsistent sequence (#17305) ... Rather than forcing the server operator to apply the SQL manually. This should be safe, as there should be only one writer for these sequences.	2024-06-14 16:40:29 +01:00
Richard van der Hoff	3aae60f17b	Enable cross-signing key upload without UIA (#17284) ... Per MSC3967, which is now stable, we should not require UIA when uploading cross-signing keys for the first time. Fixes: #17227	2024-06-14 11:14:56 +01:00
Richard van der Hoff	2c36a679ae	Include user membership on events (#17282) ... MSC4115 has now completed FCP, so we can enable it by default and switch to the stable identifier.	2024-06-13 21:45:54 +00:00
Eric Eastwood	c12ee0d5ba	Add is_dm filtering to Sliding Sync /sync (#17277) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-06-13 13:56:58 -05:00
Eric Eastwood	8aaff851b1	Fix newly_left rooms not appearing if we returned early (Sliding Sync) (#17301) ... Fix `newly_left` rooms not appearing if we returned early when `membership_snapshot_token.is_before_or_eq(to_token.room_key)`. Introduced in https://github.com/element-hq/synapse/pull/17187 (part of Sliding Sync) The tests didn't catch it because they had a small typo in it `room_id1` vs `room_id2`. Found while working on https://github.com/element-hq/synapse/pull/17293	2024-06-13 11:36:57 -05:00
Eric Eastwood	8c58eb7f17	Add event.internal_metadata.instance_name (#17300) ... Add `event.internal_metadata.instance_name` (the worker instance that persisted the event) to go alongside the existing `event.internal_metadata.stream_ordering`. `instance_name` is useful to properly compare and query for events with a token since you need to compare both the `stream_ordering` and `instance_name` against the vector clock/`instance_map` in the `RoomStreamToken`. This is pre-requisite work and may be used in https://github.com/element-hq/synapse/pull/17293 Adding `event.internal_metadata.instance_name` was first mentioned in the initial Sliding Sync PR while pairing with @erikjohnston, see 09609cb0db (diff-5cd773fb307aa754bd3948871ba118b1ef0303f4d72d42a2d21e38242bf4e096R405-R410)	2024-06-13 11:32:50 -05:00
Eric Eastwood	ebdce69f6a	Fix get_last_event_in_room_before_stream_ordering(...) finding the wrong last event (#17295) ... PR where this was introduced: https://github.com/matrix-org/synapse/pull/14817 ### What does this affect? `get_last_event_in_room_before_stream_ordering(...)` is used in Sync v2 in a lot of different state calculations. `get_last_event_in_room_before_stream_ordering(...)` is also used in `/rooms/{roomId}/members`	2024-06-13 11:00:52 -05:00
Andrew Morgan	c6eb99c878	Bump mypy from 1.8.0 to 1.9.0 (#17297) ... Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-06-13 15:50:10 +01:00