761 commits

Author	SHA1	Message	Date
Eric Eastwood	a5e06c6a8d	Move back to the main store	2024-08-21 11:14:15 -05:00
Eric Eastwood	45c89ec625	Move pre-processing completely outside transaction	2024-08-20 15:41:53 -05:00
Eric Eastwood	574a04a40f	Test state reset on membership	2024-08-19 23:30:25 -05:00
Eric Eastwood	8ee2e114dd	Add test to handle state reset in the meta data	2024-08-19 23:22:24 -05:00
Eric Eastwood	fa63c02648	Fix lints	2024-08-15 23:30:16 -05:00
Eric Eastwood	419be7c6b2	Finish off background update tests	2024-08-15 23:29:29 -05:00
Eric Eastwood	ef5f0fca3a	Add more tests	2024-08-15 23:18:50 -05:00
Eric Eastwood	fb5af8f5fa	Add background update test for sliding_sync_membership_snapshots	2024-08-15 22:13:32 -05:00
Eric Eastwood	6c2fc1d20f	Move background updates to StateBackgroundUpdateStore ... So we can access `_get_state_groups_from_groups_txn(...)`	2024-08-15 20:51:43 -05:00
Eric Eastwood	d113e743ae	Fix lints	2024-08-14 19:30:52 -05:00
Eric Eastwood	23e0d34a2d	Add more tests	2024-08-14 19:30:22 -05:00
Eric Eastwood	1c931cb3e7	Add background update for sliding_sync_joined_rooms	2024-08-14 19:19:15 -05:00
Eric Eastwood	9f551f0e97	Fix lints	2024-08-14 11:32:33 -05:00
Eric Eastwood	f49003c35c	No invites needed	2024-08-13 18:55:59 -05:00
Eric Eastwood	8b0e1692f9	More realistic remote room forgotten test	2024-08-13 18:51:11 -05:00
Eric Eastwood	32ae162278	Fix rejecting invite when no_longer_in_room (and other non-join transitions)	2024-08-13 14:35:24 -05:00
Eric Eastwood	a90f3d4ae2	Merge branch 'develop' into madlittlemods/sliding-sync-pre-populate-room-meta-data	2024-08-13 12:28:36 -05:00
Eric Eastwood	517946d940	Fix lints	2024-08-12 20:31:25 -05:00
Eric Eastwood	f600eacd0d	Adjust test description	2024-08-12 20:30:48 -05:00
Eric Eastwood	3423eb72d5	Add test to make sure snapshot evolves with membership	2024-08-12 20:29:58 -05:00
Eric Eastwood	5589ae48ca	Add test for remote invite rejected/retracted	2024-08-12 20:14:14 -05:00
Eric Eastwood	83a5858083	Add tests for remote invites	2024-08-12 19:57:28 -05:00
Eric Eastwood	ab074f5335	Fix events from rooms we're not joined to affecting the joined room stream ordering	2024-08-12 19:40:53 -05:00
Eric Eastwood	53232e6df5	Fill in for remote invites (out of band, outlier membership)	2024-08-12 18:14:02 -05:00
Eric Eastwood	f069659343	Fix lints	2024-08-12 15:49:40 -05:00
Eric Eastwood	0af3b4822c	Refactor to sliding_sync_membership_snapshots	2024-08-12 15:10:44 -05:00
Eric Eastwood	ca909013c8	Fill in stream_ordering/bump_stamp for any event being persisted	2024-08-08 17:49:15 -05:00
Eric Eastwood	cc2d2b6b9f	Fill in stream_ordering/bump_stamp when we add current state to the joined rooms table	2024-08-08 15:41:55 -05:00
Eric Eastwood	5cf3ad3d7f	Handle server left room	2024-08-07 20:47:13 -05:00
Eric Eastwood	bf78692ba0	Handle to_delete	2024-08-07 20:09:53 -05:00
Eric Eastwood	a1aaa47dad	Add more tests	2024-08-07 19:58:51 -05:00
Eric Eastwood	c590474757	Test non-joins	2024-08-07 19:24:58 -05:00
Eric Eastwood	5b1053f23e	Better test assertions	2024-08-07 19:07:43 -05:00
Eric Eastwood	61cea4e9b7	Closer to right	2024-08-07 18:07:53 -05:00
Eric Eastwood	11db575218	Sliding Sync: Use stream_ordering based timeline pagination for incremental sync (#17510) ... Use `stream_ordering` based `timeline` pagination for incremental `/sync` in Sliding Sync. Previously, we were always using a `topological_ordering` but we should only be using that for historical scenarios (initial `/sync`, newly joined, or haven't sent the room down the connection before). This is slightly different than what the [spec suggests](https://spec.matrix.org/v1.10/client-server-api/#syncing) > Events are ordered in this API according to the arrival time of the event on the homeserver. This can conflict with other APIs which order events based on their partial ordering in the event graph. This can result in duplicate events being received (once per distinct API called). Clients SHOULD de-duplicate events based on the event ID when this happens. But we've had a [discussion below in this PR](https://github.com/element-hq/synapse/pull/17510#discussion_r1699105569) and this matches what Sync v2 already does and seems like it makes sense. Created a spec issue https://github.com/matrix-org/matrix-spec/issues/1917 to clarify this. Related issues: - https://github.com/matrix-org/matrix-spec/issues/1917 - https://github.com/matrix-org/matrix-spec/issues/852 - https://github.com/matrix-org/matrix-spec-proposals/pull/4033	2024-08-07 11:27:50 -05:00
Eric Eastwood	cb335805d4	Server left room test	2024-08-07 10:46:34 -05:00
Eric Eastwood	2f3bd27284	Test is running	2024-08-06 16:50:14 -05:00
Eric Eastwood	3fee32ed6b	Order heroes by stream_ordering (as spec'ed) (#17435) ... The spec specifically mentions `stream_ordering` but that's a Synapse specific concept. In any case, the essence of the spec is basically the first 5 members of the room which `stream_ordering` accomplishes. Split off from https://github.com/element-hq/synapse/pull/17419#discussion_r1671342794 ## Spec compliance > This should be the first 5 members of the room, **ordered by stream ordering**, which are joined or invited. The list must never include the client’s own user ID. When no joined or invited members are available, this should consist of the banned and left users. > > *-- https://spec.matrix.org/v1.10/client-server-api/#_matrixclientv3sync_roomsummary* Related to https://github.com/matrix-org/matrix-spec/issues/1334	2024-07-17 13:10:15 -05:00
Eric Eastwood	3fef535ff2	Add rooms.bump_stamp to Sliding Sync /sync for easier client-side sorting (#17395) ... `bump_stamp` corresponds to the `stream_ordering` of the latest `DEFAULT_BUMP_EVENT_TYPES` in the room. This helps clients sort more readily without them needing to pull in a bunch of the timeline to determine the last activity. `bump_event_types` is a thing because for example, we don't want display name changes to mark the room as unread and bump it to the top. For encrypted rooms, we just have to consider any activity as a bump because we can't see the content and the client has to figure it out for themselves. Outside of Synapse, `bump_stamp` is just a free-form counter so other implementations could use `received_ts`or `origin_server_ts` (see the [*Security considerations* section in MSC3575 about the potential pitfalls of using `origin_server_ts`](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md#security-considerations)). It doesn't have any guarantee about always going up. In the Synapse case, it could go down if an event was redacted/removed (or purged in cases of retention policies). In the future, we could add `bump_event_types` as [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) mentions if people need to customize the event types. --- In the Sliding Sync proxy, a similar [`timestamp` field was added](https://github.com/matrix-org/sliding-sync/pull/247) for the same purpose but the name is not obvious what it pertains to or what it's for. The `timestamp` field was also added to Ruma in https://github.com/ruma/ruma/pull/1622	2024-07-08 13:17:08 -05:00
Eric Eastwood	fa91655805	Return some room data in Sliding Sync /sync (#17320) ... - Timeline events - Stripped `invite_state` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-07-02 11:07:05 -05:00
Erik Johnston	554a92601a	Reintroduce "Reduce device lists replication traffic."" (#17361) ... Reintroduces https://github.com/element-hq/synapse/pull/17333 Turns out the reason for revert was down two master instances running	2024-06-25 10:34:34 +01:00
Erik Johnston	a98cb87bee	Revert "Reduce device lists replication traffic." (#17360) ... Reverts element-hq/synapse#17333 It looks like master was still sending out replication RDATA with the old format... somehow	2024-06-25 09:57:34 +01:00
Erik Johnston	930a64b6c1	Reintroduce #17291. (#17338) ... This is #17291 (which got reverted), with some added fixups, and change so that tests actually pick up the error. The problem was that we were not calculating any new chain IDs due to a missing `not` in a condition.	2024-06-24 14:40:28 +00:00
Erik Johnston	cf711ac03c	Reduce device lists replication traffic. (#17333) ... Reduce the replication traffic of device lists, by not sending every destination that needs to be sent the device list update over replication. Instead a "hosts to send to have been calculated" notification over replication, and then federation senders read the destinations from the DB. For non federation senders this should heavily reduce the impact of a user in many large rooms changing a device.	2024-06-24 14:15:13 +01:00
Erik Johnston	4243c1f074	Revert "Handle large chain calc better (#17291)" (#17334) ... This reverts commit bdf82efea5 (#17291) This seems to have stopped persisting auth chains for new events, and so is causing state res to fall back to the slow methods	2024-06-19 17:39:33 +01:00
Erik Johnston	bdf82efea5	Handle large chain calc better (#17291) ... We calculate the auth chain links outside of the main persist event transaction to ensure that we do not block other event sending during the calculation.	2024-06-19 10:33:53 +01:00
Eric Eastwood	e5b8a3e37f	Add stream_ordering sort to Sliding Sync /sync (#17293) ... Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.	2024-06-17 11:27:14 -05:00
Quentin Gliech	e88332b5f4	Merge branch 'release-v1.109' into develop	2024-06-17 15:51:16 +02:00
Quentin Gliech	f983a77ab0	Set our own stream position from the current sequence value on startup (#17309)	2024-06-17 11:50:00 +00:00
Erik Johnston	a3cb244755	Automatically apply SQL for inconsistent sequence (#17305) ... Rather than forcing the server operator to apply the SQL manually. This should be safe, as there should be only one writer for these sequences.	2024-06-14 16:40:29 +01:00