3708 commits

Author	SHA1	Message	Date
Eric Eastwood	eb3a185cfc	Fix federating backfill test	2024-08-13 12:24:53 -05:00
Eric Eastwood	517946d940	Fix lints	2024-08-12 20:31:25 -05:00
Eric Eastwood	f600eacd0d	Adjust test description	2024-08-12 20:30:48 -05:00
Eric Eastwood	3423eb72d5	Add test to make sure snapshot evolves with membership	2024-08-12 20:29:58 -05:00
Eric Eastwood	5589ae48ca	Add test for remote invite rejected/retracted	2024-08-12 20:14:14 -05:00
Eric Eastwood	83a5858083	Add tests for remote invites	2024-08-12 19:57:28 -05:00
Eric Eastwood	ab074f5335	Fix events from rooms we're not joined to affecting the joined room stream ordering	2024-08-12 19:40:53 -05:00
Eric Eastwood	53232e6df5	Fill in for remote invites (out of band, outlier membership)	2024-08-12 18:14:02 -05:00
Eric Eastwood	f069659343	Fix lints	2024-08-12 15:49:40 -05:00
Eric Eastwood	0af3b4822c	Refactor to sliding_sync_membership_snapshots	2024-08-12 15:10:44 -05:00
Eric Eastwood	ca909013c8	Fill in stream_ordering/bump_stamp for any event being persisted	2024-08-08 17:49:15 -05:00
Eric Eastwood	cc2d2b6b9f	Fill in stream_ordering/bump_stamp when we add current state to the joined rooms table	2024-08-08 15:41:55 -05:00
Eric Eastwood	5cf3ad3d7f	Handle server left room	2024-08-07 20:47:13 -05:00
Eric Eastwood	bf78692ba0	Handle to_delete	2024-08-07 20:09:53 -05:00
Eric Eastwood	a1aaa47dad	Add more tests	2024-08-07 19:58:51 -05:00
Eric Eastwood	c590474757	Test non-joins	2024-08-07 19:24:58 -05:00
Eric Eastwood	5b1053f23e	Better test assertions	2024-08-07 19:07:43 -05:00
Eric Eastwood	61cea4e9b7	Closer to right	2024-08-07 18:07:53 -05:00
Eric Eastwood	cb335805d4	Server left room test	2024-08-07 10:46:34 -05:00
Eric Eastwood	2f3bd27284	Test is running	2024-08-06 16:50:14 -05:00
Eric Eastwood	e3db7b2d81	Sliding Sync: Easier to understand timeline assertions in tests (#17511) ... Added `_assertTimelineEqual(...)` because I got fed up trying to understand the crazy diffs from the standard `self.assertEqual(...)`/`self.assertListEqual(...)` Before: ``` [FAIL] Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 103, in test_rooms_limited_initial_sync self.assertListEqual( File "/usr/lib/python3.12/unittest/case.py", line 1091, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/usr/lib/python3.12/unittest/case.py", line 1073, in assertSequenceEqual self.fail(msg) twisted.trial.unittest.FailTest: Lists differ: ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0w[95 chars]isM'] != ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4[95 chars]nnU'] First differing element 0: '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA' '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E' - ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA', - '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E', ? ^ + ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E', ? ^ - '$q4PRxQ_pBZkQI1keYuZPTtExQ23DqpUI3-Lxwfj_isM'] + '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA', + '$j3Xj-t2F1wH9kUHsI8X5yqS7hkdSyN2owaArfvk8nnU'] ``` After: ``` [FAIL] Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 178, in test_rooms_limited_initial_sync self._assertTimelineEqual( File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 110, in _assertTimelineEqual self._assertListEqual( File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 79, in _assertListEqual self.fail(f"{diff_message}\n{message}") twisted.trial.unittest.FailTest: Items must Expected items to be in actual ('?' = missing expected items): [ (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4 (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5 ? (12, master) $bgOcc3D-2QSkbk4aBxKVyOOQJGs7ZuncRJwG3cEANZg (m.room.member, @user1:test) join ] Actual ('+' = found expected items): [ + (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5 + (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4 (9, master) $FmCNyc11YeFwiJ4an7_q6H0LCCjQOKd6UCr5VKeXXUw (m.room.message, None) activity3 ] ```	2024-08-05 13:20:15 -05:00
Eric Eastwood	2b620e0a15	Sliding Sync: Add typing notification extension (MSC3961) (#17505) ... [MSC3961](https://github.com/matrix-org/matrix-spec-proposals/pull/3961): Sliding Sync Extension: Typing Notifications Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-31 13:20:23 -05:00
Eric Eastwood	39731bb205	Sliding Sync: Split and move tests (#17504) ... Split and move Sliding Sync tests so we have some more sane test file sizes	2024-07-31 12:20:46 -05:00
Eric Eastwood	46de0ee16b	Sliding Sync: Update filters to be robust against remote invite rooms (#17450) ... Update `filters.is_encrypted` and `filters.types`/`filters.not_types` to be robust when dealing with remote invite rooms in Sliding Sync. Part of [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Follow-up to https://github.com/element-hq/synapse/pull/17434 We now take into account current state, fallback to stripped state for invite/knock rooms, then historical state. If we can't determine the info needed to filter a room (either from state or stripped state), it is filtered out.	2024-07-30 13:20:29 -05:00
Eric Eastwood	b221f0b84b	Sliding Sync: Add receipts extension (MSC3960) (#17489) ... [MSC3960](https://github.com/matrix-org/matrix-spec-proposals/pull/3960): Receipts extension Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-30 12:49:55 -05:00
Olivier 'reivilibre	53db8a914e	Merge branch 'master' into develop	2024-07-30 17:10:46 +01:00
reivilibre	c56b070e6f	Upgrade locked dependency on Twisted to 24.7.0rc1. (#17502) ... I also update the tests and HTTP Proxy code to fix it for this new Twisted release. Pulls in fix for https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7 Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>	2024-07-30 15:23:23 +01:00
Richard van der Hoff	808dab0699	Fix failures property in /keys/query (#17499) ... Fixes: https://github.com/element-hq/synapse/issues/17498 Fixes: https://github.com/element-hq/element-web/issues/27867	2024-07-30 09:51:24 +01:00
Erik Johnston	34306be5aa	Only send rooms with updates down sliding sync (#17479) ... Rather than always including all rooms in range. Also adds a pre-filter to rooms that checks the stream change cache to see if anything might have happened. Based on #17447 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-30 09:30:44 +01:00
Erik Johnston	be4a16ff44	Sliding Sync: Track whether we have sent rooms down to clients (#17447) ... The basic idea is that we introduce a new token for a sliding sync connection, which stores the mapping of room to room "status" (i.e. have we sent the room down?). This token allows us to handle duplicate requests properly. In future it can be used to store more "per-connection" information safely. In future this should be migrated into the DB, so its important that we try to reduce the number of syncs where we need to update the per-connection information. In this PoC this only happens when we: a) send down a set of room for the first time, or b) we have previously sent down a room and there are updates but we are not sending the room down the sync (due to not falling in a list range) Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-29 22:45:48 +01:00
Eric Eastwood	568051c0f0	Refactor Sliding Sync tests to better utilize the SlidingSyncBase.do_sync(...) (pt. 2) (#17482) ... `SlidingSyncBase.do_sync()` for tests was first introduced in https://github.com/element-hq/synapse/pull/17452 Part 1: https://github.com/element-hq/synapse/pull/17481	2024-07-25 11:01:47 -05:00
Eric Eastwood	ebbabfe782	Refactor Sliding Sync tests to better utilize the SlidingSyncBase (pt. 1) (#17481) ... `SlidingSyncBase` for tests was first introduced in https://github.com/element-hq/synapse/pull/17452 Part 2: https://github.com/element-hq/synapse/pull/17482	2024-07-25 10:43:35 -05:00
Eric Eastwood	729026e604	Sliding Sync: Add Account Data extension (MSC3959) (#17477) ... Extensions based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-24 17:10:38 -05:00
Erik Johnston	bdf37ad4c4	Sliding Sync: ensure bump stamp ignores backfilled events (#17478) ... Backfill events have a negative stream ordering, and so its not useful to use to compare with other (positive) stream orderings. Plus, the Rust SDK currently assumes `bump_stamp` is positive.	2024-07-24 15:21:56 +01:00
Erik Johnston	8bbc98e66d	Use a new token format for sliding sync (#17452) ... This is in preparation for adding per-connection state. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-24 11:47:25 +01:00
Erik Johnston	d225b6b3eb	Speed up SS room sorting (#17468) ... We do this by bulk fetching the latest stream ordering. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2024-07-23 14:03:14 +01:00
Eric Eastwood	de05a64246	Sliding Sync: Add E2EE extension (MSC3884) (#17454) ... Spec: [MSC3884](https://github.com/matrix-org/matrix-spec-proposals/pull/3884) Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-22 15:40:06 -05:00
Erik Johnston	d221512498	SS: Implement $ME support (#17469) ... `$ME` can be used as a substitute for the requester's user ID.	2024-07-22 17:48:09 +01:00
Shay	dc8ddc6472	Prepare for authenticated media freeze (#17433) ... As part of the rollout of [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3916-authentication-for-media.md) this PR adds support for designating authenticated media and ensuring that authenticated media is not served over unauthenticated endpoints.	2024-07-22 10:33:17 +01:00
Erik Johnston	d3f9afd8d9	Add a cache on get_rooms_for_local_user_where_membership_is (#17460) ... As it gets used in sliding sync. We basically invalidate it in all the same places as `get_rooms_for_user`. Most of the changes are due to needing the arguments you pass in to be hashable (which lists aren't)	2024-07-19 16:19:15 +01:00
Eric Eastwood	3fee32ed6b	Order heroes by stream_ordering (as spec'ed) (#17435) ... The spec specifically mentions `stream_ordering` but that's a Synapse specific concept. In any case, the essence of the spec is basically the first 5 members of the room which `stream_ordering` accomplishes. Split off from https://github.com/element-hq/synapse/pull/17419#discussion_r1671342794 ## Spec compliance > This should be the first 5 members of the room, **ordered by stream ordering**, which are joined or invited. The list must never include the client’s own user ID. When no joined or invited members are available, this should consist of the banned and left users. > > *-- https://spec.matrix.org/v1.10/client-server-api/#_matrixclientv3sync_roomsummary* Related to https://github.com/matrix-org/matrix-spec/issues/1334	2024-07-17 13:10:15 -05:00
dependabot[bot]	79924aebef	Bump mypy from 1.9.0 to 1.10.1 (#17445)	2024-07-16 12:08:06 +01:00
Shay	429ecb7564	Handle remote download responses with UNKNOWN_LENGTH more gracefully (#17439) ... Prior to this PR, remote downloads which did not provide a `content-length` were decremented from the remote download ratelimiter at the max allowable size, leading to excessive ratelimiting - see https://github.com/element-hq/synapse/issues/17394. This PR adds a linearizer to limit concurrent remote downloads to 6 per IP address, and decrements remote downloads without a `content-length` from the ratelimiter *after* the download is complete and the response length is known. Also adds logic to ensure that responses with a known length respect the `max_download_size`.	2024-07-16 11:13:55 +01:00
Shay	899d33f2ba	Remove unnecessary call to resume producing in fake channel (#17449) ... This fell out of the authenticated media work - this bit of code masked a bug but does not break anything when removed, so probably should be removed.	2024-07-16 09:52:39 +01:00
Erik Johnston	df11af14db	Fix bug where sync could get stuck when using workers (#17438) ... This is because we serialized the token wrong if the instance map contained entries from before the minimum token.	2024-07-15 16:13:04 +01:00
Eric Eastwood	ab62aa09da	Add room subscriptions to Sliding Sync /sync (#17432) ... Add room subscriptions to Sliding Sync `/sync` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Currently, you can only subscribe to rooms you have had *any* membership in before. In the future, we will allow `world_readable` rooms to be subscribed to without joining.	2024-07-15 10:37:10 +01:00
Eric Eastwood	fb66e938b2	Add is_dm room field to Sliding Sync /sync (#17429) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-11 18:19:26 -05:00
Eric Eastwood	5a97bbd895	Add heroes and room summary fields to Sliding Sync /sync (#17419) ... Additional room summary fields: `joined_count`, `invited_count` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-11 14:05:38 -05:00
Erik Johnston	606da398fc	Fix filtering room types on remote rooms (#17434) ... We can only fetch room types for rooms the server is in, so we need to only filter rooms that we're joined to. Also includes a perf fix to bulk fetch room types.	2024-07-11 16:00:44 +01:00
Erik Johnston	4ca13ce0dd	Handle to-device extensions to Sliding Sync (#17416) ... Implements MSC3885 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-10 11:58:42 +01:00