1177 commits

Author	SHA1	Message	Date
Eric Eastwood	a90f3d4ae2	Merge branch 'develop' into madlittlemods/sliding-sync-pre-populate-room-meta-data	2024-08-13 12:28:36 -05:00
Eric Eastwood	a308d99f30	Sliding Sync: Exclude partially stated rooms if we must await full state (#17538) ... Previously, we just had very basic partial room exclusion based on whether we were lazy-loading room members. Now with this PR, we added `must_await_full_state(...)` with rules to check if we have a we're only requesting `required_state` which is completely satisfied even with partial state. Partially-stated rooms should have all state events except for remote membership events so if we require a remote membership event anywhere, then we need to return `True`.	2024-08-13 12:27:42 -05:00
Eric Eastwood	eb3a185cfc	Fix federating backfill test	2024-08-13 12:24:53 -05:00
Erik Johnston	44ac2aa3b6	SSS: Implement PREVIOUSLY room tracking (#17535) ... Implement tracking of rooms that have had updates that have not been sent down to clients. Simplified Sliding Sync (SSS)	2024-08-08 10:44:17 +01:00
Erik Johnston	c270355349	SS: Reset connection if token is unrecognized (#17529) ... This triggers the client to start a new sliding sync connection. If we don't do this and the client asks for the full range of rooms, we end up sending down all rooms and their state from scratch (which can be very slow) This causes things like https://github.com/element-hq/element-x-ios/issues/3115 after we restart the server --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-08-06 10:39:11 +01:00
Eric Eastwood	e3db7b2d81	Sliding Sync: Easier to understand timeline assertions in tests (#17511) ... Added `_assertTimelineEqual(...)` because I got fed up trying to understand the crazy diffs from the standard `self.assertEqual(...)`/`self.assertListEqual(...)` Before: ``` [FAIL] Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 103, in test_rooms_limited_initial_sync self.assertListEqual( File "/usr/lib/python3.12/unittest/case.py", line 1091, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/usr/lib/python3.12/unittest/case.py", line 1073, in assertSequenceEqual self.fail(msg) twisted.trial.unittest.FailTest: Lists differ: ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0w[95 chars]isM'] != ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4[95 chars]nnU'] First differing element 0: '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA' '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E' - ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA', - '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E', ? ^ + ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E', ? ^ - '$q4PRxQ_pBZkQI1keYuZPTtExQ23DqpUI3-Lxwfj_isM'] + '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA', + '$j3Xj-t2F1wH9kUHsI8X5yqS7hkdSyN2owaArfvk8nnU'] ``` After: ``` [FAIL] Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 178, in test_rooms_limited_initial_sync self._assertTimelineEqual( File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 110, in _assertTimelineEqual self._assertListEqual( File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 79, in _assertListEqual self.fail(f"{diff_message}\n{message}") twisted.trial.unittest.FailTest: Items must Expected items to be in actual ('?' = missing expected items): [ (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4 (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5 ? (12, master) $bgOcc3D-2QSkbk4aBxKVyOOQJGs7ZuncRJwG3cEANZg (m.room.member, @user1:test) join ] Actual ('+' = found expected items): [ + (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5 + (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4 (9, master) $FmCNyc11YeFwiJ4an7_q6H0LCCjQOKd6UCr5VKeXXUw (m.room.message, None) activity3 ] ```	2024-08-05 13:20:15 -05:00
Eric Eastwood	2b620e0a15	Sliding Sync: Add typing notification extension (MSC3961) (#17505) ... [MSC3961](https://github.com/matrix-org/matrix-spec-proposals/pull/3961): Sliding Sync Extension: Typing Notifications Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-31 13:20:23 -05:00
Eric Eastwood	39731bb205	Sliding Sync: Split and move tests (#17504) ... Split and move Sliding Sync tests so we have some more sane test file sizes	2024-07-31 12:20:46 -05:00
Eric Eastwood	46de0ee16b	Sliding Sync: Update filters to be robust against remote invite rooms (#17450) ... Update `filters.is_encrypted` and `filters.types`/`filters.not_types` to be robust when dealing with remote invite rooms in Sliding Sync. Part of [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Follow-up to https://github.com/element-hq/synapse/pull/17434 We now take into account current state, fallback to stripped state for invite/knock rooms, then historical state. If we can't determine the info needed to filter a room (either from state or stripped state), it is filtered out.	2024-07-30 13:20:29 -05:00
Eric Eastwood	b221f0b84b	Sliding Sync: Add receipts extension (MSC3960) (#17489) ... [MSC3960](https://github.com/matrix-org/matrix-spec-proposals/pull/3960): Receipts extension Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-30 12:49:55 -05:00
Olivier 'reivilibre	53db8a914e	Merge branch 'master' into develop	2024-07-30 17:10:46 +01:00
reivilibre	c56b070e6f	Upgrade locked dependency on Twisted to 24.7.0rc1. (#17502) ... I also update the tests and HTTP Proxy code to fix it for this new Twisted release. Pulls in fix for https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7 Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>	2024-07-30 15:23:23 +01:00
Erik Johnston	34306be5aa	Only send rooms with updates down sliding sync (#17479) ... Rather than always including all rooms in range. Also adds a pre-filter to rooms that checks the stream change cache to see if anything might have happened. Based on #17447 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-30 09:30:44 +01:00
Erik Johnston	be4a16ff44	Sliding Sync: Track whether we have sent rooms down to clients (#17447) ... The basic idea is that we introduce a new token for a sliding sync connection, which stores the mapping of room to room "status" (i.e. have we sent the room down?). This token allows us to handle duplicate requests properly. In future it can be used to store more "per-connection" information safely. In future this should be migrated into the DB, so its important that we try to reduce the number of syncs where we need to update the per-connection information. In this PoC this only happens when we: a) send down a set of room for the first time, or b) we have previously sent down a room and there are updates but we are not sending the room down the sync (due to not falling in a list range) Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-29 22:45:48 +01:00
Eric Eastwood	568051c0f0	Refactor Sliding Sync tests to better utilize the SlidingSyncBase.do_sync(...) (pt. 2) (#17482) ... `SlidingSyncBase.do_sync()` for tests was first introduced in https://github.com/element-hq/synapse/pull/17452 Part 1: https://github.com/element-hq/synapse/pull/17481	2024-07-25 11:01:47 -05:00
Eric Eastwood	ebbabfe782	Refactor Sliding Sync tests to better utilize the SlidingSyncBase (pt. 1) (#17481) ... `SlidingSyncBase` for tests was first introduced in https://github.com/element-hq/synapse/pull/17452 Part 2: https://github.com/element-hq/synapse/pull/17482	2024-07-25 10:43:35 -05:00
Eric Eastwood	729026e604	Sliding Sync: Add Account Data extension (MSC3959) (#17477) ... Extensions based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-24 17:10:38 -05:00
Erik Johnston	bdf37ad4c4	Sliding Sync: ensure bump stamp ignores backfilled events (#17478) ... Backfill events have a negative stream ordering, and so its not useful to use to compare with other (positive) stream orderings. Plus, the Rust SDK currently assumes `bump_stamp` is positive.	2024-07-24 15:21:56 +01:00
Erik Johnston	8bbc98e66d	Use a new token format for sliding sync (#17452) ... This is in preparation for adding per-connection state. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-24 11:47:25 +01:00
Eric Eastwood	de05a64246	Sliding Sync: Add E2EE extension (MSC3884) (#17454) ... Spec: [MSC3884](https://github.com/matrix-org/matrix-spec-proposals/pull/3884) Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-22 15:40:06 -05:00
Erik Johnston	d221512498	SS: Implement $ME support (#17469) ... `$ME` can be used as a substitute for the requester's user ID.	2024-07-22 17:48:09 +01:00
Shay	dc8ddc6472	Prepare for authenticated media freeze (#17433) ... As part of the rollout of [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3916-authentication-for-media.md) this PR adds support for designating authenticated media and ensuring that authenticated media is not served over unauthenticated endpoints.	2024-07-22 10:33:17 +01:00
Eric Eastwood	3fee32ed6b	Order heroes by stream_ordering (as spec'ed) (#17435) ... The spec specifically mentions `stream_ordering` but that's a Synapse specific concept. In any case, the essence of the spec is basically the first 5 members of the room which `stream_ordering` accomplishes. Split off from https://github.com/element-hq/synapse/pull/17419#discussion_r1671342794 ## Spec compliance > This should be the first 5 members of the room, **ordered by stream ordering**, which are joined or invited. The list must never include the client’s own user ID. When no joined or invited members are available, this should consist of the banned and left users. > > *-- https://spec.matrix.org/v1.10/client-server-api/#_matrixclientv3sync_roomsummary* Related to https://github.com/matrix-org/matrix-spec/issues/1334	2024-07-17 13:10:15 -05:00
Shay	429ecb7564	Handle remote download responses with UNKNOWN_LENGTH more gracefully (#17439) ... Prior to this PR, remote downloads which did not provide a `content-length` were decremented from the remote download ratelimiter at the max allowable size, leading to excessive ratelimiting - see https://github.com/element-hq/synapse/issues/17394. This PR adds a linearizer to limit concurrent remote downloads to 6 per IP address, and decrements remote downloads without a `content-length` from the ratelimiter *after* the download is complete and the response length is known. Also adds logic to ensure that responses with a known length respect the `max_download_size`.	2024-07-16 11:13:55 +01:00
Eric Eastwood	ab62aa09da	Add room subscriptions to Sliding Sync /sync (#17432) ... Add room subscriptions to Sliding Sync `/sync` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Currently, you can only subscribe to rooms you have had *any* membership in before. In the future, we will allow `world_readable` rooms to be subscribed to without joining.	2024-07-15 10:37:10 +01:00
Eric Eastwood	fb66e938b2	Add is_dm room field to Sliding Sync /sync (#17429) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-11 18:19:26 -05:00
Eric Eastwood	5a97bbd895	Add heroes and room summary fields to Sliding Sync /sync (#17419) ... Additional room summary fields: `joined_count`, `invited_count` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-11 14:05:38 -05:00
Erik Johnston	4ca13ce0dd	Handle to-device extensions to Sliding Sync (#17416) ... Implements MSC3885 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-10 11:58:42 +01:00
Eric Eastwood	1cf3ff6b40	Add rooms name and avatar to Sliding Sync /sync (#17418) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-09 12:26:45 -05:00
Eric Eastwood	3fef535ff2	Add rooms.bump_stamp to Sliding Sync /sync for easier client-side sorting (#17395) ... `bump_stamp` corresponds to the `stream_ordering` of the latest `DEFAULT_BUMP_EVENT_TYPES` in the room. This helps clients sort more readily without them needing to pull in a bunch of the timeline to determine the last activity. `bump_event_types` is a thing because for example, we don't want display name changes to mark the room as unread and bump it to the top. For encrypted rooms, we just have to consider any activity as a bump because we can't see the content and the client has to figure it out for themselves. Outside of Synapse, `bump_stamp` is just a free-form counter so other implementations could use `received_ts`or `origin_server_ts` (see the [*Security considerations* section in MSC3575 about the potential pitfalls of using `origin_server_ts`](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md#security-considerations)). It doesn't have any guarantee about always going up. In the Synapse case, it could go down if an event was redacted/removed (or purged in cases of retention policies). In the future, we could add `bump_event_types` as [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) mentions if people need to customize the event types. --- In the Sliding Sync proxy, a similar [`timestamp` field was added](https://github.com/matrix-org/sliding-sync/pull/247) for the same purpose but the name is not obvious what it pertains to or what it's for. The `timestamp` field was also added to Ruma in https://github.com/ruma/ruma/pull/1622	2024-07-08 13:17:08 -05:00
Shay	cf69f8d59b	Support MSC3916 by adding a federation /thumbnail endpoint and authenticated _matrix/client/v1/media/thumbnail endpoint (#17388) ... [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916) added the endpoints `_matrix/federation/v1/media/thumbnail` and the authenticated `_matrix/client/v1/media/thumbnail`. This PR implements those endpoints, along with stabilizing `_matrix/client/v1/media/config` and `_matrix/client/v1/media/preview_url`. Complement tests are at https://github.com/matrix-org/complement/pull/728	2024-07-08 10:11:20 +01:00
Erik Johnston	57538eb4d9	Finish up work to allow per-user feature flags (#17392) ... Follows on from @H-Shay's great work at https://github.com/matrix-org/synapse/pull/15344 and MSC4026. Also enables its use for MSC3881, mainly as an easy but concrete example of how to use it.	2024-07-05 13:02:35 +01:00
Eric Eastwood	22aeb78b77	Add rooms.required_state to Sliding Sync /sync (#17342) ... Also handles excluding rooms with partial state when people are asking for room membership events unless it's `$LAZY` room membership.	2024-07-04 12:25:36 -05:00
Eric Eastwood	fa91655805	Return some room data in Sliding Sync /sync (#17320) ... - Timeline events - Stripped `invite_state` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-02 11:07:05 -05:00
Shay	8f890447b0	Support MSC3916 by adding _matrix/client/v1/media/download endpoint (#17365)	2024-07-02 14:07:04 +01:00
Erik Johnston	b3b793786c	Fix sync waiting for an invalid token from the "future" (#17386) ... Fixes https://github.com/element-hq/synapse/issues/17274, hopefully. Basically, old versions of Synapse could advance streams without persisting anything in the DB (fixed in #17229). On restart those updates would get lost, and so the position of the stream would revert to an older position. If this happened across an upgrade to a later Synapse version which included #17215, then sync could get blocked indefinitely (until the stream advanced to the position in the token). We fix this by bounding the stream positions we'll wait for to the maximum position of the underlying stream ID generator.	2024-07-02 12:39:49 +01:00
Eric Eastwood	6e8af83193	Add is_invite filtering to Sliding Sync /sync (#17335) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-06-24 19:07:56 -05:00
Shay	7a11c0ac4f	Add support for MSC3823 - Account Suspension Part 2 (#17255)	2024-06-24 14:21:22 +01:00
Erik Johnston	3239b7459c	Register sliding sync under a different path (#17331) ... As the API is slightly incompatible.	2024-06-19 16:18:45 +00:00
Alexander Fechler	9104a9f0d0	Filter added to Admin-API GET /rooms (#17276)	2024-06-19 11:45:48 +01:00
Andrew Morgan	afaf2d9388	Require the 'from' parameter for /notifications be an integer (#17283) ... Co-authored-by: Erik Johnston <erikj@element.io>	2024-06-19 10:05:39 +01:00
Eric Eastwood	e5b8a3e37f	Add stream_ordering sort to Sliding Sync /sync (#17293) ... Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.	2024-06-17 11:27:14 -05:00
Richard van der Hoff	3aae60f17b	Enable cross-signing key upload without UIA (#17284) ... Per MSC3967, which is now stable, we should not require UIA when uploading cross-signing keys for the first time. Fixes: #17227	2024-06-14 11:14:56 +01:00
Richard van der Hoff	2c36a679ae	Include user membership on events (#17282) ... MSC4115 has now completed FCP, so we can enable it by default and switch to the stable identifier.	2024-06-13 21:45:54 +00:00
Eric Eastwood	c12ee0d5ba	Add is_dm filtering to Sliding Sync /sync (#17277) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-06-13 13:56:58 -05:00
Andrew Morgan	c6eb99c878	Bump mypy from 1.8.0 to 1.9.0 (#17297) ... Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-06-13 15:50:10 +01:00
Travis Ralston	f1c4dfb08b	Add report room API (MSC4151) (#17270) ... https://github.com/matrix-org/matrix-spec-proposals/pull/4151 This is intended to be enabled by default for immediate use. When FCP is complete, the unstable endpoint will be dropped and stable endpoint supported instead - no backwards compatibility is expected for the unstable endpoint.	2024-06-12 12:27:46 +02:00
Eric Eastwood	dad1559721	Reorganize Pydantic models and types used in handlers (#17279) ... Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues. - Moved Pydantic parsing models to `synapse/types/rest` - Moved handler types to `synapse/types/handlers`	2024-06-10 15:03:50 -05:00
Eric Eastwood	4a7c58642c	Add Sliding Sync /sync endpoint (initial implementation) (#17187) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering). Rooms appear in the Sliding sync response based on: - `invite`, `join`, `knock`, `ban` membership events - Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`) - `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`) - In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range. ### Example request `POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync` ```json { "lists": { "foo-list": { "ranges": [ [0, 99] ], "sort": [ "by_notification_level", "by_recency", "by_name" ], "required_state": [ ["m.room.join_rules", ""], ["m.room.history_visibility", ""], ["m.space.child", "*"] ], "timeline_limit": 100 } } } ``` Response: ```json { "next_pos": "s58_224_0_13_10_1_1_16_0_1", "lists": { "foo-list": { "count": 1, "ops": [ { "op": "SYNC", "range": [0, 99], "room_ids": [ "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server" ] } ] } }, "rooms": {}, "extensions": {} } ```	2024-06-06 14:44:32 -05:00
Erik Johnston	bb5a692946	Fix slipped logging context when media rejected (#17239) ... When a module rejects a piece of media we end up trying to close the same logging context twice. Instead of fixing the existing code we refactor to use an async context manager, which is easier to write correctly.	2024-05-29 11:14:42 +01:00