Commit graph

24224 commits

Author SHA1 Message Date
Eric Eastwood
4dc9e268e6 Add test for catch-up background update 2024-08-27 18:08:17 -05:00
Eric Eastwood
9a7d8c2be4 Start catch-up if nothing written yet 2024-08-27 17:28:07 -05:00
Eric Eastwood
c51a309da5 Maybe: always start background update 2024-08-27 17:11:51 -05:00
Eric Eastwood
9764f626ea Fix query in Postgres 2024-08-27 12:09:00 -05:00
Eric Eastwood
7a0c281028 Add placeholder tests 2024-08-26 19:43:52 -05:00
Eric Eastwood
7fe5d31e20 Note down caveat about forgotten 2024-08-26 18:52:10 -05:00
Eric Eastwood
53473a0eb4 Adapt sliding_sync_joined_rooms background update to use event_stream_ordering for progress
This way we can re-use it for the catch-up background process
2024-08-26 18:35:49 -05:00
Eric Eastwood
eb3c84cf45 Kick-off background update for out-of-date snapshots 2024-08-26 17:31:26 -05:00
Eric Eastwood
6a44686dc3 Why it matters 2024-08-26 16:32:59 -05:00
Eric Eastwood
a94c1dd62c Add more context for why 2024-08-26 16:27:20 -05:00
Eric Eastwood
8bddbe23bd Clear out-of-date rows 2024-08-26 16:19:47 -05:00
Eric Eastwood
addb91485f Split test cases 2024-08-26 16:11:56 -05:00
Eric Eastwood
9795556052 Update comment 2024-08-26 14:42:55 -05:00
Eric Eastwood
a57d47b778 Use simple_upsert_txn for sliding_sync_membership_snapshots in background update 2024-08-22 23:59:06 -05:00
Eric Eastwood
b6a7d2bf6c Use simple_upsert_txn for sliding_sync_joined_rooms in background update 2024-08-22 23:30:00 -05:00
Eric Eastwood
f8926d07df Fix partial-stated room re-syncing state but nothing has changed
Fixes failing test in CI: `tests.handlers.test_federation.PartialJoinTestCase.test_failed_partial_join_is_clean`
```
2024-08-22 18:57:22-0500 [-] synapse.metrics.background_process_metrics - 253 - ERROR - sync_partial_state_room-0 - Background process 'sync_partial_state_room' threw an exception
	Traceback (most recent call last):
	  File "synapse/synapse/metrics/background_process_metrics.py", line 251, in run
	    return await func(*args, **kwargs)
	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "synapse/synapse/handlers/federation.py", line 1842, in _sync_partial_state_room_wrapper
	    await self._sync_partial_state_room(
	  File "synapse/synapse/handlers/federation.py", line 1933, in _sync_partial_state_room
	    await self.state_handler.update_current_state(room_id)
	  File "synapse/synapse/state/__init__.py", line 554, in update_current_state
	    await self._storage_controllers.persistence.update_current_state(room_id)
	  File "synapse/synapse/storage/controllers/persist_events.py", line 491, in update_current_state
	    await self._event_persist_queue.add_to_queue(
	  File "synapse/synapse/storage/controllers/persist_events.py", line 245, in add_to_queue
	    res = await make_deferred_yieldable(end_item.deferred.observe())
	          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "synapse/synapse/storage/controllers/persist_events.py", line 288, in handle_queue_loop
	    ret = await self._per_item_callback(room_id, item.task)
	          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	  File "synapse/synapse/storage/controllers/persist_events.py", line 370, in _process_event_persist_queue_task
	    await self._update_current_state(room_id, task)
	  File "synapse/synapse/storage/controllers/persist_events.py", line 507, in _update_current_state
	    await self.persist_events_store._calculate_sliding_sync_table_changes(
	  File "synapse/synapse/storage/databases/main/events.py", line 624, in _calculate_sliding_sync_table_changes
	    assert most_recent_event_pos_results, (
	           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
	AssertionError: We should not be seeing `None` here because we are still in the room (!room:example.com) and it should at-least have a join membership event that's keeping us here.
```
2024-08-22 19:06:04 -05:00
Eric Eastwood
21cc97ba9d Use simple_upsert_many_txn for sliding_sync_membership_snapshots
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726820006
2024-08-22 18:52:05 -05:00
Eric Eastwood
fdb8b5931f Correct comment 2024-08-22 18:43:51 -05:00
Eric Eastwood
4b866c4fca Simplify what we need to think about to grab the best effort value 2024-08-22 18:36:43 -05:00
Eric Eastwood
088a4c7cf0 Use simple_upsert_txn to update sliding_sync_joined_rooms
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726817206
2024-08-22 18:26:37 -05:00
Eric Eastwood
0726a6d58b Derive best effort stream_ordering outside of the transaction
See https://github.com/element-hq/synapse/pull/17512#discussion_r1727995882
2024-08-22 18:13:00 -05:00
Eric Eastwood
6edc4c78ce Allow for no bump_stamp (fix portdb CI job)
See https://github.com/element-hq/synapse/pull/17512#discussion_r1725998219
2024-08-22 17:09:43 -05:00
Eric Eastwood
44432e2118 Move tests to dedicated file
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726849798
2024-08-22 16:56:09 -05:00
Eric Eastwood
bcba8cccfe No need for transaction
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726844107
2024-08-22 16:52:31 -05:00
Eric Eastwood
693c06b2f1 Move away from backfill language 2024-08-22 16:48:02 -05:00
Eric Eastwood
4d87fa61c6 "backfill" -> "bg_update"
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726837698
2024-08-22 16:44:02 -05:00
Eric Eastwood
d61aada8ba Simplify _update_sliding_sync_tables_with_new_persisted_events_txn()
See
https://github.com/element-hq/synapse/pull/17512#discussion_r1719997640
https://github.com/element-hq/synapse/pull/17512#discussion_r1726828894
https://github.com/element-hq/synapse/pull/17512#discussion_r1726836440
2024-08-22 16:35:59 -05:00
Eric Eastwood
6723824c4a Prefer simple_delete_many_txn
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726819380
2024-08-22 15:47:44 -05:00
Eric Eastwood
980ee9aad6 Prefer simple_update_txn
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726808112
2024-08-22 15:40:08 -05:00
Eric Eastwood
fc73b6ffc9 Rename insert_key/insert_value
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726805894
2024-08-22 11:44:57 -05:00
Eric Eastwood
9b8d2017af Check events_and_context for state events
See https://github.com/element-hq/synapse/pull/17512#discussion_r1726798803
2024-08-22 11:34:40 -05:00
Eric Eastwood
339500d067 Fix sub-query selecting multiple rows 2024-08-21 22:56:25 -05:00
Eric Eastwood
31300f4ce5 More docstring 2024-08-21 22:15:19 -05:00
Eric Eastwood
b45b1896aa Fill out docstring 2024-08-21 22:09:57 -05:00
Eric Eastwood
ee2ef0b4d9 Add forgotten column 2024-08-21 21:54:22 -05:00
Eric Eastwood
0a938b137a Add missing boolean column to portdb script 2024-08-21 19:59:35 -05:00
Eric Eastwood
97248362d0 Log which room is strange 2024-08-21 19:26:14 -05:00
Eric Eastwood
02711552cf Better handle none case 2024-08-21 19:11:08 -05:00
Eric Eastwood
8ddf5c7235 Add tombstone to tests 2024-08-21 19:05:59 -05:00
Eric Eastwood
513ec8e906 Update tests 2024-08-21 18:51:04 -05:00
Eric Eastwood
cda2311520 Add tombstone_successor_room_id column 2024-08-21 18:21:44 -05:00
Eric Eastwood
c612572d12 Move away from stream_id
See https://github.com/element-hq/synapse/pull/17512#discussion_r1725806543
2024-08-21 17:14:34 -05:00
Eric Eastwood
5b1db39bb7 Add sender column so we can tell leaves from kicks 2024-08-21 16:42:12 -05:00
Eric Eastwood
e7a3328228 Pre-populate membership and membership_event_stream_ordering
See https://github.com/element-hq/synapse/pull/17512#discussion_r1725311745
2024-08-21 16:20:00 -05:00
Eric Eastwood
f6d7ffd9c5 Move _calculate_sliding_sync_table_changes(...) after we assign stream_ordering to events
See https://github.com/element-hq/synapse/pull/17512#discussion_r1725728637
2024-08-21 16:10:14 -05:00
Eric Eastwood
772c501bb6 Use available stream_id
See https://github.com/element-hq/synapse/pull/17512#discussion_r1725310035
2024-08-21 15:04:51 -05:00
Eric Eastwood
cda92af4a6 No need to update event_stream_ordering/bump_stamp ON CONFLICT 2024-08-21 14:46:50 -05:00
Eric Eastwood
d3f90e4bd8 Get full events for _sliding_sync_joined_rooms_backfill 2024-08-21 14:39:54 -05:00
Eric Eastwood
a5e06c6a8d Move back to the main store 2024-08-21 11:14:15 -05:00
Eric Eastwood
0233e20aa3 Use full event version after solving the circular import issues 2024-08-21 10:44:49 -05:00