3737 commits

Author	SHA1	Message	Date
Eric Eastwood	a5e06c6a8d	Move back to the main store	2024-08-21 11:14:15 -05:00
Eric Eastwood	cc200ee9f5	Merge branch 'develop' into madlittlemods/sliding-sync-pre-populate-room-meta-data	2024-08-20 17:25:48 -05:00
Eric Eastwood	45c89ec625	Move pre-processing completely outside transaction	2024-08-20 15:41:53 -05:00
Erik Johnston	6eb98a4f1c	Sliding Sync: Handle timeline limit changes (take 2) (#17579) ... This supersedes #17503, given the per-connection state is being heavily rewritten it felt easier to recreate the PR on top of that work. This correctly handles the case of timeline limits going up and down. This does not handle changes in `required_state`, but that can be done as a separate PR. Based on #17575. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-08-20 10:31:25 +01:00
Eric Eastwood	574a04a40f	Test state reset on membership	2024-08-19 23:30:25 -05:00
Eric Eastwood	8ee2e114dd	Add test to handle state reset in the meta data	2024-08-19 23:22:24 -05:00
Eric Eastwood	df0c57d383	Merge branch 'develop' into madlittlemods/sliding-sync-pre-populate-room-meta-data	2024-08-19 16:34:41 -05:00
Erik Johnston	8b8d74d12f	Sliding sync: Correctly track which read receipts we have or have not sent down. (#17575) ... Add connection tracking to the receipts extension. Based on #17574 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-08-19 21:16:07 +01:00
Eric Eastwood	fa63c02648	Fix lints	2024-08-15 23:30:16 -05:00
Eric Eastwood	419be7c6b2	Finish off background update tests	2024-08-15 23:29:29 -05:00
Eric Eastwood	ef5f0fca3a	Add more tests	2024-08-15 23:18:50 -05:00
Eric Eastwood	fb5af8f5fa	Add background update test for sliding_sync_membership_snapshots	2024-08-15 22:13:32 -05:00
Eric Eastwood	6c2fc1d20f	Move background updates to StateBackgroundUpdateStore ... So we can access `_get_state_groups_from_groups_txn(...)`	2024-08-15 20:51:43 -05:00
Eric Eastwood	d113e743ae	Fix lints	2024-08-14 19:30:52 -05:00
Eric Eastwood	23e0d34a2d	Add more tests	2024-08-14 19:30:22 -05:00
Eric Eastwood	1c931cb3e7	Add background update for sliding_sync_joined_rooms	2024-08-14 19:19:15 -05:00
Eric Eastwood	9f551f0e97	Fix lints	2024-08-14 11:32:33 -05:00
Shay	b05b2e14bb	Handle lower-case http headers in _Mulitpart_Parser_Protocol (#17545)	2024-08-14 09:49:01 +01:00
Eric Eastwood	f49003c35c	No invites needed	2024-08-13 18:55:59 -05:00
Eric Eastwood	8b0e1692f9	More realistic remote room forgotten test	2024-08-13 18:51:11 -05:00
Eric Eastwood	96a4614f92	Update fixme comment	2024-08-13 14:50:11 -05:00
Eric Eastwood	32ae162278	Fix rejecting invite when no_longer_in_room (and other non-join transitions)	2024-08-13 14:35:24 -05:00
Eric Eastwood	a90f3d4ae2	Merge branch 'develop' into madlittlemods/sliding-sync-pre-populate-room-meta-data	2024-08-13 12:28:36 -05:00
Eric Eastwood	a308d99f30	Sliding Sync: Exclude partially stated rooms if we must await full state (#17538) ... Previously, we just had very basic partial room exclusion based on whether we were lazy-loading room members. Now with this PR, we added `must_await_full_state(...)` with rules to check if we have a we're only requesting `required_state` which is completely satisfied even with partial state. Partially-stated rooms should have all state events except for remote membership events so if we require a remote membership event anywhere, then we need to return `True`.	2024-08-13 12:27:42 -05:00
Eric Eastwood	eb3a185cfc	Fix federating backfill test	2024-08-13 12:24:53 -05:00
Erik Johnston	a9fc1fd112	Use a larger, dedicated threadpool for media sending (#17564)	2024-08-13 17:59:47 +01:00
Andrew Morgan	6a11bdf01d	Add a utility function for generating fake event IDs (#17557)	2024-08-13 16:55:05 +00:00
Eric Eastwood	517946d940	Fix lints	2024-08-12 20:31:25 -05:00
Eric Eastwood	f600eacd0d	Adjust test description	2024-08-12 20:30:48 -05:00
Eric Eastwood	3423eb72d5	Add test to make sure snapshot evolves with membership	2024-08-12 20:29:58 -05:00
Eric Eastwood	5589ae48ca	Add test for remote invite rejected/retracted	2024-08-12 20:14:14 -05:00
Eric Eastwood	83a5858083	Add tests for remote invites	2024-08-12 19:57:28 -05:00
Eric Eastwood	ab074f5335	Fix events from rooms we're not joined to affecting the joined room stream ordering	2024-08-12 19:40:53 -05:00
Eric Eastwood	53232e6df5	Fill in for remote invites (out of band, outlier membership)	2024-08-12 18:14:02 -05:00
Eric Eastwood	f069659343	Fix lints	2024-08-12 15:49:40 -05:00
Eric Eastwood	0af3b4822c	Refactor to sliding_sync_membership_snapshots	2024-08-12 15:10:44 -05:00
Eric Eastwood	ca909013c8	Fill in stream_ordering/bump_stamp for any event being persisted	2024-08-08 17:49:15 -05:00
Eric Eastwood	cc2d2b6b9f	Fill in stream_ordering/bump_stamp when we add current state to the joined rooms table	2024-08-08 15:41:55 -05:00
Erik Johnston	44ac2aa3b6	SSS: Implement PREVIOUSLY room tracking (#17535) ... Implement tracking of rooms that have had updates that have not been sent down to clients. Simplified Sliding Sync (SSS)	2024-08-08 10:44:17 +01:00
Eric Eastwood	5cf3ad3d7f	Handle server left room	2024-08-07 20:47:13 -05:00
Eric Eastwood	bf78692ba0	Handle to_delete	2024-08-07 20:09:53 -05:00
Eric Eastwood	a1aaa47dad	Add more tests	2024-08-07 19:58:51 -05:00
Eric Eastwood	c590474757	Test non-joins	2024-08-07 19:24:58 -05:00
Eric Eastwood	5b1053f23e	Better test assertions	2024-08-07 19:07:43 -05:00
Eric Eastwood	61cea4e9b7	Closer to right	2024-08-07 18:07:53 -05:00
Eric Eastwood	11db575218	Sliding Sync: Use stream_ordering based timeline pagination for incremental sync (#17510) ... Use `stream_ordering` based `timeline` pagination for incremental `/sync` in Sliding Sync. Previously, we were always using a `topological_ordering` but we should only be using that for historical scenarios (initial `/sync`, newly joined, or haven't sent the room down the connection before). This is slightly different than what the [spec suggests](https://spec.matrix.org/v1.10/client-server-api/#syncing) > Events are ordered in this API according to the arrival time of the event on the homeserver. This can conflict with other APIs which order events based on their partial ordering in the event graph. This can result in duplicate events being received (once per distinct API called). Clients SHOULD de-duplicate events based on the event ID when this happens. But we've had a [discussion below in this PR](https://github.com/element-hq/synapse/pull/17510#discussion_r1699105569) and this matches what Sync v2 already does and seems like it makes sense. Created a spec issue https://github.com/matrix-org/matrix-spec/issues/1917 to clarify this. Related issues: - https://github.com/matrix-org/matrix-spec/issues/1917 - https://github.com/matrix-org/matrix-spec/issues/852 - https://github.com/matrix-org/matrix-spec-proposals/pull/4033	2024-08-07 11:27:50 -05:00
Eric Eastwood	cb335805d4	Server left room test	2024-08-07 10:46:34 -05:00
Eric Eastwood	2f3bd27284	Test is running	2024-08-06 16:50:14 -05:00
Erik Johnston	c270355349	SS: Reset connection if token is unrecognized (#17529) ... This triggers the client to start a new sliding sync connection. If we don't do this and the client asks for the full range of rooms, we end up sending down all rooms and their state from scratch (which can be very slow) This causes things like https://github.com/element-hq/element-x-ios/issues/3115 after we restart the server --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-08-06 10:39:11 +01:00
Eric Eastwood	e3db7b2d81	Sliding Sync: Easier to understand timeline assertions in tests (#17511) ... Added `_assertTimelineEqual(...)` because I got fed up trying to understand the crazy diffs from the standard `self.assertEqual(...)`/`self.assertListEqual(...)` Before: ``` [FAIL] Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 103, in test_rooms_limited_initial_sync self.assertListEqual( File "/usr/lib/python3.12/unittest/case.py", line 1091, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/usr/lib/python3.12/unittest/case.py", line 1073, in assertSequenceEqual self.fail(msg) twisted.trial.unittest.FailTest: Lists differ: ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0w[95 chars]isM'] != ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4[95 chars]nnU'] First differing element 0: '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA' '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E' - ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA', - '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E', ? ^ + ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E', ? ^ - '$q4PRxQ_pBZkQI1keYuZPTtExQ23DqpUI3-Lxwfj_isM'] + '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA', + '$j3Xj-t2F1wH9kUHsI8X5yqS7hkdSyN2owaArfvk8nnU'] ``` After: ``` [FAIL] Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 178, in test_rooms_limited_initial_sync self._assertTimelineEqual( File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 110, in _assertTimelineEqual self._assertListEqual( File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 79, in _assertListEqual self.fail(f"{diff_message}\n{message}") twisted.trial.unittest.FailTest: Items must Expected items to be in actual ('?' = missing expected items): [ (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4 (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5 ? (12, master) $bgOcc3D-2QSkbk4aBxKVyOOQJGs7ZuncRJwG3cEANZg (m.room.member, @user1:test) join ] Actual ('+' = found expected items): [ + (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5 + (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4 (9, master) $FmCNyc11YeFwiJ4an7_q6H0LCCjQOKd6UCr5VKeXXUw (m.room.message, None) activity3 ] ```	2024-08-05 13:20:15 -05:00