24067 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Erik Johnston	f7ad92e37d	Add tests for notification counts	2024-08-16 10:45:15 +01:00
Erik Johnston	ba7969b570	Newsfile	2024-08-09 10:32:58 +01:00
Erik Johnston	98b3f56411	SS: Hook up notification counts	2024-08-09 10:31:56 +01:00
Erik Johnston	44ac2aa3b6	SSS: Implement PREVIOUSLY room tracking (#17535) ... Implement tracking of rooms that have had updates that have not been sent down to clients. Simplified Sliding Sync (SSS)	2024-08-08 10:44:17 +01:00
Eric Eastwood	11db575218	Sliding Sync: Use stream_ordering based timeline pagination for incremental sync (#17510) ... Use `stream_ordering` based `timeline` pagination for incremental `/sync` in Sliding Sync. Previously, we were always using a `topological_ordering` but we should only be using that for historical scenarios (initial `/sync`, newly joined, or haven't sent the room down the connection before). This is slightly different than what the [spec suggests](https://spec.matrix.org/v1.10/client-server-api/#syncing) > Events are ordered in this API according to the arrival time of the event on the homeserver. This can conflict with other APIs which order events based on their partial ordering in the event graph. This can result in duplicate events being received (once per distinct API called). Clients SHOULD de-duplicate events based on the event ID when this happens. But we've had a [discussion below in this PR](https://github.com/element-hq/synapse/pull/17510#discussion_r1699105569) and this matches what Sync v2 already does and seems like it makes sense. Created a spec issue https://github.com/matrix-org/matrix-spec/issues/1917 to clarify this. Related issues: - https://github.com/matrix-org/matrix-spec/issues/1917 - https://github.com/matrix-org/matrix-spec/issues/852 - https://github.com/matrix-org/matrix-spec-proposals/pull/4033	2024-08-07 11:27:50 -05:00
dependabot[bot]	30e9f6e469	Bump bytes from 1.6.1 to 1.7.1 (#17526)	2024-08-07 10:37:54 +01:00
dependabot[bot]	eb62d12063	Bump regex from 1.10.5 to 1.10.6 (#17527)	2024-08-07 10:37:13 +01:00
Erik Johnston	ceb3686dcd	Fixup sliding sync comment (#17531) ... c.f. https://github.com/element-hq/synapse/pull/17529#discussion_r1705780925	2024-08-07 10:32:36 +01:00
Eric Eastwood	1dfa59b238	Sliding Sync: Add more tracing (#17514) ... Spawning from looking at a couple traces and wanting a little more info. Follow-up to github.com/element-hq/synapse/pull/17501 The changes in this PR allow you to find slow Sliding Sync traces ignoring the `wait_for_events` time. In Jaeger, you can now filter for the `current_sync_for_user` operation with `RESULT.result=true` indicating that it actually returned non-empty results. If you want to find traces for your own user, you can use `RESULT.result=true ARG.sync_config.user="@madlittlemods:matrix.org"`	2024-08-06 11:43:43 -05:00
Andrew Morgan	bef6568537	Merge branch 'release-v1.113' into develop	2024-08-06 14:19:12 +01:00
Andrew Morgan	244a255065	Clarify auto_accept_invites.worker_to_run_on config docs (#17515)	2024-08-06 13:26:51 +01:00
Andrew Morgan	932cb0a928	1.113.0rc1	2024-08-06 12:24:47 +01:00
dependabot[bot]	2dad718265	Bump phonenumbers from 8.13.39 to 8.13.42 (#17521)	2024-08-06 11:47:19 +01:00
dependabot[bot]	5d8446298c	Bump towncrier from 23.11.0 to 24.7.1 (#17523)	2024-08-06 11:47:06 +01:00
dependabot[bot]	d845e939a9	Bump black from 24.4.2 to 24.8.0 (#17522)	2024-08-06 11:46:48 +01:00
dependabot[bot]	23727869c7	Bump serde_json from 1.0.121 to 1.0.122 (#17525)	2024-08-06 11:45:44 +01:00
Erik Johnston	c270355349	SS: Reset connection if token is unrecognized (#17529) ... This triggers the client to start a new sliding sync connection. If we don't do this and the client asks for the full range of rooms, we end up sending down all rooms and their state from scratch (which can be very slow) This causes things like https://github.com/element-hq/element-x-ios/issues/3115 after we restart the server --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-08-06 10:39:11 +01:00
Eric Eastwood	e3db7b2d81	Sliding Sync: Easier to understand timeline assertions in tests (#17511) ... Added `_assertTimelineEqual(...)` because I got fed up trying to understand the crazy diffs from the standard `self.assertEqual(...)`/`self.assertListEqual(...)` Before: ``` [FAIL] Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 103, in test_rooms_limited_initial_sync self.assertListEqual( File "/usr/lib/python3.12/unittest/case.py", line 1091, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/usr/lib/python3.12/unittest/case.py", line 1073, in assertSequenceEqual self.fail(msg) twisted.trial.unittest.FailTest: Lists differ: ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0w[95 chars]isM'] != ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4[95 chars]nnU'] First differing element 0: '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA' '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E' - ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA', - '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E', ? ^ + ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E', ? ^ - '$q4PRxQ_pBZkQI1keYuZPTtExQ23DqpUI3-Lxwfj_isM'] + '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA', + '$j3Xj-t2F1wH9kUHsI8X5yqS7hkdSyN2owaArfvk8nnU'] ``` After: ``` [FAIL] Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 178, in test_rooms_limited_initial_sync self._assertTimelineEqual( File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 110, in _assertTimelineEqual self._assertListEqual( File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 79, in _assertListEqual self.fail(f"{diff_message}\n{message}") twisted.trial.unittest.FailTest: Items must Expected items to be in actual ('?' = missing expected items): [ (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4 (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5 ? (12, master) $bgOcc3D-2QSkbk4aBxKVyOOQJGs7ZuncRJwG3cEANZg (m.room.member, @user1:test) join ] Actual ('+' = found expected items): [ + (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5 + (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4 (9, master) $FmCNyc11YeFwiJ4an7_q6H0LCCjQOKd6UCr5VKeXXUw (m.room.message, None) activity3 ] ```	2024-08-05 13:20:15 -05:00
Eric Eastwood	2b620e0a15	Sliding Sync: Add typing notification extension (MSC3961) (#17505) ... [MSC3961](https://github.com/matrix-org/matrix-spec-proposals/pull/3961): Sliding Sync Extension: Typing Notifications Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-31 13:20:23 -05:00
Eric Eastwood	39731bb205	Sliding Sync: Split and move tests (#17504) ... Split and move Sliding Sync tests so we have some more sane test file sizes	2024-07-31 12:20:46 -05:00
Eric Eastwood	1d6186265a	Sliding Sync: Fix limited response description (make accurate) (#17507)	2024-07-31 11:47:26 -05:00
Eric Eastwood	46de0ee16b	Sliding Sync: Update filters to be robust against remote invite rooms (#17450) ... Update `filters.is_encrypted` and `filters.types`/`filters.not_types` to be robust when dealing with remote invite rooms in Sliding Sync. Part of [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Follow-up to https://github.com/element-hq/synapse/pull/17434 We now take into account current state, fallback to stripped state for invite/knock rooms, then historical state. If we can't determine the info needed to filter a room (either from state or stripped state), it is filtered out.	2024-07-30 13:20:29 -05:00
Eric Eastwood	b221f0b84b	Sliding Sync: Add receipts extension (MSC3960) (#17489) ... [MSC3960](https://github.com/matrix-org/matrix-spec-proposals/pull/3960): Receipts extension Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-30 12:49:55 -05:00
Olivier 'reivilibre	b2c55bd049	Merge branch 'master' into develop	2024-07-30 18:09:05 +01:00
Olivier 'reivilibre	ed583d9c81	Merge branch 'release-v1.112'	2024-07-30 18:07:35 +01:00
dependabot[bot]	f76dc9923c	Bump types-setuptools from 70.1.0.20240627 to 71.1.0.20240726 (#17497) ... Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-07-30 17:33:43 +01:00
dependabot[bot]	7e997fb8b1	Bump types-pyopenssl from 24.1.0.20240425 to 24.1.0.20240722 (#17496) ... Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-07-30 17:33:18 +01:00
dependabot[bot]	dbc2290cbe	Bump bcrypt from 4.1.3 to 4.2.0 (#17495) ... Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-07-30 17:32:49 +01:00
dependabot[bot]	2f6b86e79a	Bump serde_json from 1.0.120 to 1.0.121 (#17493) ... Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-07-30 17:32:16 +01:00
Olivier 'reivilibre	37f9876ccf	1.112.0	2024-07-30 17:24:09 +01:00
reivilibre	8b449a8ce6	Upgrade locked dependency on Twisted to 24.7.0rc1. (#17502) ... I also update the tests and HTTP Proxy code to fix it for this new Twisted release. Pulls in fix for https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7 Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>	2024-07-30 17:14:14 +01:00
Olivier 'reivilibre	53db8a914e	Merge branch 'master' into develop	2024-07-30 17:10:46 +01:00
Olivier 'reivilibre	e4868f8a1e	Add bold emphasis to some parts of the changelog	2024-07-30 16:23:58 +01:00
Olivier 'reivilibre	dcad81082c	1.111.1	2024-07-30 16:16:35 +01:00
reivilibre	c56b070e6f	Upgrade locked dependency on Twisted to 24.7.0rc1. (#17502) ... I also update the tests and HTTP Proxy code to fix it for this new Twisted release. Pulls in fix for https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7 Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>	2024-07-30 15:23:23 +01:00
dependabot[bot]	be726724a8	Bump ruff from 0.5.4 to 0.5.5 (#17494) ... Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-07-30 11:44:54 +01:00
Erik Johnston	62ae56a4ac	Add some more opentracing to sliding sync (#17501) ... This will make it easier to see what it is doing in jaeger.	2024-07-30 10:54:11 +01:00
Richard van der Hoff	808dab0699	Fix failures property in /keys/query (#17499) ... Fixes: https://github.com/element-hq/synapse/issues/17498 Fixes: https://github.com/element-hq/element-web/issues/27867	2024-07-30 09:51:24 +01:00
Erik Johnston	34306be5aa	Only send rooms with updates down sliding sync (#17479) ... Rather than always including all rooms in range. Also adds a pre-filter to rooms that checks the stream change cache to see if anything might have happened. Based on #17447 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-30 09:30:44 +01:00
Erik Johnston	be4a16ff44	Sliding Sync: Track whether we have sent rooms down to clients (#17447) ... The basic idea is that we introduce a new token for a sliding sync connection, which stores the mapping of room to room "status" (i.e. have we sent the room down?). This token allows us to handle duplicate requests properly. In future it can be used to store more "per-connection" information safely. In future this should be migrated into the DB, so its important that we try to reduce the number of syncs where we need to update the per-connection information. In this PoC this only happens when we: a) send down a set of room for the first time, or b) we have previously sent down a room and there are updates but we are not sending the room down the sync (due to not falling in a list range) Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-29 22:45:48 +01:00
Eric Eastwood	568051c0f0	Refactor Sliding Sync tests to better utilize the SlidingSyncBase.do_sync(...) (pt. 2) (#17482) ... `SlidingSyncBase.do_sync()` for tests was first introduced in https://github.com/element-hq/synapse/pull/17452 Part 1: https://github.com/element-hq/synapse/pull/17481	2024-07-25 11:01:47 -05:00
Eric Eastwood	ebbabfe782	Refactor Sliding Sync tests to better utilize the SlidingSyncBase (pt. 1) (#17481) ... `SlidingSyncBase` for tests was first introduced in https://github.com/element-hq/synapse/pull/17452 Part 2: https://github.com/element-hq/synapse/pull/17482	2024-07-25 10:43:35 -05:00
YLong Shi	69ac4b6a6e	Update config_documentation - Change example of msisdn in allowed_local_3pids (#17476) ... Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2024-07-25 11:07:44 +00:00
Eric Eastwood	729026e604	Sliding Sync: Add Account Data extension (MSC3959) (#17477) ... Extensions based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-24 17:10:38 -05:00
Erik Johnston	bdf37ad4c4	Sliding Sync: ensure bump stamp ignores backfilled events (#17478) ... Backfill events have a negative stream ordering, and so its not useful to use to compare with other (positive) stream orderings. Plus, the Rust SDK currently assumes `bump_stamp` is positive.	2024-07-24 15:21:56 +01:00
Erik Johnston	8bbc98e66d	Use a new token format for sliding sync (#17452) ... This is in preparation for adding per-connection state. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>	2024-07-24 11:47:25 +01:00
Maciej Laskowski	4b9f4c2abf	Update debian template - new link to the delegation docs (#17475) ... Update debian template - new link to the delegation docs	2024-07-24 10:32:56 +01:00
Devon Hudson	e8ee784c75	Address changelog review comments	2024-07-23 09:14:45 -06:00
Devon Hudson	48c1307911	1.112.0rc1	2024-07-23 09:01:43 -06:00
Erik Johnston	d225b6b3eb	Speed up SS room sorting (#17468) ... We do this by bulk fetching the latest stream ordering. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2024-07-23 14:03:14 +01:00