Commit graph

24158 commits

Author SHA1 Message Date
Eric Eastwood
574a04a40f Test state reset on membership 2024-08-19 23:30:25 -05:00
Eric Eastwood
8ee2e114dd Add test to handle state reset in the meta data 2024-08-19 23:22:24 -05:00
Eric Eastwood
98fb56e5fe Prefer _update_sliding_sync_tables_with_new_persisted_events_txn(...) to do the right thing
See https://github.com/element-hq/synapse/pull/17512#discussion_r1719992152
2024-08-19 22:30:50 -05:00
Eric Eastwood
df0c57d383 Merge branch 'develop' into madlittlemods/sliding-sync-pre-populate-room-meta-data 2024-08-19 16:34:41 -05:00
Erik Johnston
8b8d74d12f
Sliding sync: Correctly track which read receipts we have or have not sent down. (#17575)
Add connection tracking to the receipts extension.

Based on #17574

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-08-19 21:16:07 +01:00
Erik Johnston
261e746281
Sliding sync: Add classes for per-connection state (#17574)
This is some prep work ahead of correctly tracking receipts, where we
will also want to track the room status in terms of last receipt we had
sent down.

Essentially, we add two classes `PerConnectionState` and a mutable
version, and then operate on those.

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-08-19 20:09:41 +01:00
Erik Johnston
993644ded0
Fix zero length media handling (#17570)
Results in:

```
AssertionError: null
  File "synapse/http/server.py", line 332, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "synapse/http/server.py", line 544, in _async_render
    callback_return = await raw_callback_return
  File "synapse/federation/transport/server/_base.py", line 369, in new_func
    response = await func(
  File "synapse/federation/transport/server/federation.py", line 826, in on_GET
    await self.media_repo.get_local_media(
  File "synapse/media/media_repository.py", line 473, in get_local_media
    await respond_with_multipart_responder(
  File "synapse/media/_base.py", line 353, in respond_with_multipart_responder
    assert content_length is not None
```
2024-08-19 15:06:44 +01:00
Erik Johnston
a5d25bb623
Test github token before running release script (#17562)
This stops people from getting half way through a step and it failing
due to the github token having expired (this happens to me every damn
time).
2024-08-19 14:15:36 +01:00
Erik Johnston
f162c92f2a
Speed up /keys/changes (#17548)
Follow on from #17537.

This is just adding a batched lookup function (you might want to hide
whitespace in the diff).
2024-08-16 16:04:02 +01:00
Erik Johnston
9ce489be5e
Add a flag to /versions about SSS support (#17571)
So that clients can check for support. Note that if the feature is only
enabled for some users, the `/versions` request must be authenticated to
pick up that SSS is enabled for the user
2024-08-16 08:54:57 +01:00
Eric Eastwood
d2f5247e77 Update comment 2024-08-16 00:15:03 -05:00
Eric Eastwood
c89d859c7c Fill in docstrings 2024-08-15 23:52:01 -05:00
Eric Eastwood
2ec93e3f0d Move function next to other helpers 2024-08-15 23:39:39 -05:00
Eric Eastwood
fa63c02648 Fix lints 2024-08-15 23:30:16 -05:00
Eric Eastwood
419be7c6b2 Finish off background update tests 2024-08-15 23:29:29 -05:00
Eric Eastwood
ef5f0fca3a Add more tests 2024-08-15 23:18:50 -05:00
Eric Eastwood
fb5af8f5fa Add background update test for sliding_sync_membership_snapshots 2024-08-15 22:13:32 -05:00
Eric Eastwood
8461faf384 Add historical case to background update 2024-08-15 21:56:12 -05:00
Eric Eastwood
6c2fc1d20f Move background updates to StateBackgroundUpdateStore
So we can access `_get_state_groups_from_groups_txn(...)`
2024-08-15 20:51:43 -05:00
Eric Eastwood
cbeff57402 Use helper 2024-08-15 20:31:57 -05:00
Eric Eastwood
4b42e44ef9 Work on background update for sliding_sync_membership_snapshots 2024-08-15 00:21:35 -05:00
Eric Eastwood
d113e743ae Fix lints 2024-08-14 19:30:52 -05:00
Eric Eastwood
23e0d34a2d Add more tests 2024-08-14 19:30:22 -05:00
Eric Eastwood
1c931cb3e7 Add background update for sliding_sync_joined_rooms 2024-08-14 19:19:15 -05:00
Eric Eastwood
9f551f0e97 Fix lints 2024-08-14 11:32:33 -05:00
Eric Eastwood
c8508f113a Clean up tables when a room is purged/deleted 2024-08-14 11:27:57 -05:00
Andrew Morgan
fae75b0376
Register the media threadpool with our metrics (#17566)
Some checks failed
Tests / check-lockfile (push) Has been cancelled
Tests / lint-crlf (push) Has been cancelled
Tests / lint-newsfile (push) Has been cancelled
Deploy the documentation / GitHub Pages (push) Has been cancelled
Build release artifacts / Build .deb packages (push) Has been cancelled
Build release artifacts / Attach assets to release (push) Has been cancelled
Tests / check-sampleconfig (push) Has been cancelled
Tests / check-schema-delta (push) Has been cancelled
Tests / lint (push) Has been cancelled
Tests / Typechecking (push) Has been cancelled
Tests / lint-pydantic (push) Has been cancelled
Tests / lint-clippy (push) Has been cancelled
Tests / lint-clippy-nightly (push) Has been cancelled
Tests / lint-rustfmt (push) Has been cancelled
Tests / lint-readme (push) Has been cancelled
Tests / linting-done (push) Has been cancelled
Tests / calculate-test-jobs (push) Has been cancelled
Tests / trial (push) Has been cancelled
Tests / trial-olddeps (push) Has been cancelled
Tests / trial-pypy (all, pypy-3.8) (push) Has been cancelled
Tests / sytest (push) Has been cancelled
Tests / export-data (push) Has been cancelled
Tests / portdb (11, 3.8) (push) Has been cancelled
Tests / portdb (15, 3.11) (push) Has been cancelled
Tests / complement (monolith, Postgres) (push) Has been cancelled
Tests / complement (monolith, SQLite) (push) Has been cancelled
Tests / complement (workers, Postgres) (push) Has been cancelled
Tests / cargo-test (push) Has been cancelled
Tests / cargo-bench (push) Has been cancelled
Tests / tests-done (push) Has been cancelled
2024-08-14 15:11:22 +01:00
Tulir Asokan
f77bfbfa30
Fix fetching signing keys when old_verify_keys is omitted (#17568)
Some checks are pending
Build release artifacts / Build sdist (push) Waiting to run
Build release artifacts / Attach assets to release (push) Blocked by required conditions
Tests / changes (push) Waiting to run
Tests / check-sampleconfig (push) Blocked by required conditions
Tests / check-schema-delta (push) Blocked by required conditions
Tests / check-lockfile (push) Waiting to run
Tests / lint (push) Blocked by required conditions
Tests / Typechecking (push) Blocked by required conditions
Tests / lint-crlf (push) Waiting to run
Tests / lint-newsfile (push) Waiting to run
Tests / lint-pydantic (push) Blocked by required conditions
Tests / lint-clippy (push) Blocked by required conditions
Tests / trial-olddeps (push) Blocked by required conditions
Tests / trial-pypy (all, pypy-3.8) (push) Blocked by required conditions
Tests / portdb (11, 3.8) (push) Blocked by required conditions
Tests / complement (monolith, SQLite) (push) Blocked by required conditions
Tests / complement (workers, Postgres) (push) Blocked by required conditions
Tests / cargo-test (push) Blocked by required conditions
Tests / cargo-bench (push) Blocked by required conditions
Tests / tests-done (push) Blocked by required conditions
Tests / lint-clippy-nightly (push) Blocked by required conditions
Tests / lint-rustfmt (push) Blocked by required conditions
Tests / lint-readme (push) Blocked by required conditions
Tests / linting-done (push) Blocked by required conditions
Tests / calculate-test-jobs (push) Blocked by required conditions
Tests / trial (push) Blocked by required conditions
Tests / sytest (push) Blocked by required conditions
Tests / export-data (push) Blocked by required conditions
Tests / portdb (15, 3.11) (push) Blocked by required conditions
Tests / complement (monolith, Postgres) (push) Blocked by required conditions
`old_verify_keys` isn't marked as required in
https://spec.matrix.org/v1.11/server-server-api/#get_matrixkeyv2server
and there's no functional difference between an empty object and
omitting the object, so I don't think there's any reason synapse should
explode when the field is omitted.
2024-08-14 14:13:56 +01:00
Erik Johnston
1892ba5f67
Fix 'Producer was not unregistered' error (#17569)
Follows on from #17567
2024-08-14 13:46:22 +01:00
Erik Johnston
a51daffba5
Reduce concurrent thread usage in media (#17567)
Follow on from #17558

Basically, we want to reduce the number of threads we want to use at a
time, i.e. reduce the number of threads that are paused/blocked. We do
this by returning from the thread when the consumer pauses the producer,
rather than pausing in the thread.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-08-14 12:41:53 +01:00
Shay
b05b2e14bb
Handle lower-case http headers in _Mulitpart_Parser_Protocol (#17545) 2024-08-14 09:49:01 +01:00
Eric Eastwood
f49003c35c No invites needed 2024-08-13 18:55:59 -05:00
Eric Eastwood
8b0e1692f9 More realistic remote room forgotten test 2024-08-13 18:51:11 -05:00
Eric Eastwood
5df94f47b5 Fix running into StopIteration
More context about how/why `StopIteration` was being ignored silently
which made this problem harder to debug. See
https://github.com/element-hq/synapse/pull/17512#discussion_r1715954505
2024-08-13 17:15:09 -05:00
Eric Eastwood
3566abd9bc Fix boolean schema for Postgres 2024-08-13 15:14:48 -05:00
Eric Eastwood
96a4614f92 Update fixme comment 2024-08-13 14:50:11 -05:00
Eric Eastwood
dc447a673f Clarify when/why we upsert 2024-08-13 14:47:17 -05:00
Eric Eastwood
32ae162278 Fix rejecting invite when no_longer_in_room (and other non-join transitions) 2024-08-13 14:35:24 -05:00
Eric Eastwood
a90f3d4ae2 Merge branch 'develop' into madlittlemods/sliding-sync-pre-populate-room-meta-data 2024-08-13 12:28:36 -05:00
Eric Eastwood
a308d99f30
Sliding Sync: Exclude partially stated rooms if we must await full state (#17538)
Some checks are pending
Build release artifacts / Build sdist (push) Waiting to run
Build release artifacts / Attach assets to release (push) Blocked by required conditions
Tests / lint (push) Blocked by required conditions
Tests / changes (push) Waiting to run
Tests / check-sampleconfig (push) Blocked by required conditions
Tests / check-schema-delta (push) Blocked by required conditions
Tests / check-lockfile (push) Waiting to run
Tests / Typechecking (push) Blocked by required conditions
Tests / lint-crlf (push) Waiting to run
Tests / lint-newsfile (push) Waiting to run
Tests / lint-pydantic (push) Blocked by required conditions
Tests / lint-clippy (push) Blocked by required conditions
Tests / lint-clippy-nightly (push) Blocked by required conditions
Tests / lint-rustfmt (push) Blocked by required conditions
Tests / lint-readme (push) Blocked by required conditions
Tests / linting-done (push) Blocked by required conditions
Tests / calculate-test-jobs (push) Blocked by required conditions
Tests / trial (push) Blocked by required conditions
Tests / trial-pypy (all, pypy-3.8) (push) Blocked by required conditions
Tests / sytest (push) Blocked by required conditions
Tests / export-data (push) Blocked by required conditions
Tests / trial-olddeps (push) Blocked by required conditions
Tests / portdb (11, 3.8) (push) Blocked by required conditions
Tests / portdb (15, 3.11) (push) Blocked by required conditions
Tests / complement (monolith, Postgres) (push) Blocked by required conditions
Tests / complement (monolith, SQLite) (push) Blocked by required conditions
Tests / complement (workers, Postgres) (push) Blocked by required conditions
Tests / cargo-test (push) Blocked by required conditions
Tests / cargo-bench (push) Blocked by required conditions
Tests / tests-done (push) Blocked by required conditions
Previously, we just had very basic partial room exclusion based on
whether we were lazy-loading room members. Now with this PR, we added
`must_await_full_state(...)` with rules to check if we have a we're only
requesting `required_state` which is completely satisfied even with
partial state.

Partially-stated rooms should have all state events except for remote
membership events so if we require a remote membership event anywhere,
then we need to return `True`.
2024-08-13 12:27:42 -05:00
Eric Eastwood
eb3a185cfc Fix federating backfill test 2024-08-13 12:24:53 -05:00
Erik Johnston
a9fc1fd112
Use a larger, dedicated threadpool for media sending (#17564) 2024-08-13 17:59:47 +01:00
Andrew Morgan
6a11bdf01d
Add a utility function for generating fake event IDs (#17557) 2024-08-13 16:55:05 +00:00
Patrick Cloke
8fea190a1f
Add missing docstrings related to profile methods. (#17559) 2024-08-13 17:04:35 +01:00
Erik Johnston
81c19c4cd2 Merge branch 'master' into develop 2024-08-13 15:58:11 +01:00
Erik Johnston
aaa3c36420
Remove logging in multipart (#17563)
This is really spurious and causes a lot of spam. I don't think there is
a use for it even at DEBUG level.
2024-08-13 15:56:18 +01:00
Erik Johnston
3e7eb45eb1
Fixup media logcontexts (#17561)
Regression from #17558
2024-08-13 15:00:57 +01:00
Erik Johnston
bab37dfc6f 1.113.0
Some checks failed
Tests / changes (push) Has been cancelled
Tests / check-lockfile (push) Has been cancelled
Tests / lint-crlf (push) Has been cancelled
Tests / cargo-bench (push) Has been cancelled
Deploy the documentation / GitHub Pages (push) Has been cancelled
Build release artifacts / Build .deb packages (push) Has been cancelled
Build release artifacts / Attach assets to release (push) Has been cancelled
Tests / check-sampleconfig (push) Has been cancelled
Tests / check-schema-delta (push) Has been cancelled
Tests / lint (push) Has been cancelled
Tests / Typechecking (push) Has been cancelled
Tests / lint-pydantic (push) Has been cancelled
Tests / lint-clippy (push) Has been cancelled
Tests / lint-clippy-nightly (push) Has been cancelled
Tests / lint-rustfmt (push) Has been cancelled
Tests / lint-readme (push) Has been cancelled
Tests / linting-done (push) Has been cancelled
Tests / calculate-test-jobs (push) Has been cancelled
Tests / trial (push) Has been cancelled
Tests / trial-olddeps (push) Has been cancelled
Tests / trial-pypy (all, pypy-3.8) (push) Has been cancelled
Tests / sytest (push) Has been cancelled
Tests / export-data (push) Has been cancelled
Tests / portdb (11, 3.8) (push) Has been cancelled
Tests / portdb (15, 3.11) (push) Has been cancelled
Tests / complement (monolith, Postgres) (push) Has been cancelled
Tests / complement (monolith, SQLite) (push) Has been cancelled
Tests / complement (workers, Postgres) (push) Has been cancelled
Tests / cargo-test (push) Has been cancelled
Tests / tests-done (push) Has been cancelled
2024-08-13 14:37:01 +01:00
Erik Johnston
9f9ec92526
Speed up responding to media requests (#17558)
Some checks are pending
Build release artifacts / Build sdist (push) Waiting to run
Build release artifacts / Attach assets to release (push) Blocked by required conditions
Tests / lint-crlf (push) Waiting to run
Tests / Typechecking (push) Blocked by required conditions
Tests / check-schema-delta (push) Blocked by required conditions
Tests / lint-newsfile (push) Waiting to run
Tests / lint-pydantic (push) Blocked by required conditions
Tests / lint-clippy (push) Blocked by required conditions
Tests / changes (push) Waiting to run
Tests / check-sampleconfig (push) Blocked by required conditions
Tests / check-lockfile (push) Waiting to run
Tests / lint (push) Blocked by required conditions
Tests / lint-clippy-nightly (push) Blocked by required conditions
Tests / lint-rustfmt (push) Blocked by required conditions
Tests / lint-readme (push) Blocked by required conditions
Tests / linting-done (push) Blocked by required conditions
Tests / calculate-test-jobs (push) Blocked by required conditions
Tests / trial (push) Blocked by required conditions
Tests / trial-olddeps (push) Blocked by required conditions
Tests / trial-pypy (all, pypy-3.8) (push) Blocked by required conditions
Tests / sytest (push) Blocked by required conditions
Tests / export-data (push) Blocked by required conditions
Tests / portdb (11, 3.8) (push) Blocked by required conditions
Tests / portdb (15, 3.11) (push) Blocked by required conditions
Tests / complement (monolith, Postgres) (push) Blocked by required conditions
Tests / complement (monolith, SQLite) (push) Blocked by required conditions
Tests / complement (workers, Postgres) (push) Blocked by required conditions
Tests / cargo-test (push) Blocked by required conditions
Tests / cargo-bench (push) Blocked by required conditions
Tests / tests-done (push) Blocked by required conditions
We do this by reading from a threadpool, rather than blocking the main
thread.

This is broadly what we do in the [S3 storage
provider](https://github.com/matrix-org/synapse-s3-storage-provider/blob/main/s3_storage_provider.py#L234)
2024-08-13 14:06:17 +01:00
dependabot[bot]
ff7b27013e
Bump serde from 1.0.204 to 1.0.206 (#17556) 2024-08-13 08:59:35 +01:00