Commit graph

25307 commits

Author SHA1 Message Date
Eric Eastwood
faf5b40520
Sliding Sync: Fix _bulk_get_max_event_pos(...) being inefficient (#17728)
Fix `_bulk_get_max_event_pos(...)` being inefficient. It kept adding all
of the `batch_results` to the `results` over and over every time we
checked a single room in the batch.

I think we still ended up with the right answer before because we
accumulate `recheck_rooms` and actually recheck them to overwrite the
bad data we wrote to the `results` before.

Introduced in
https://github.com/element-hq/synapse/pull/17606/files#diff-cbd54e4b5a2a1646299d659a2d5884d6cb14e608efd2e1658e72b465bb66e31bR1481
2024-09-19 09:32:16 +01:00
Eric Eastwood
af998e6c66
Sliding sync: Ignore invites from ignored users (#17729)
`m.ignored_user_list` in account data
2024-09-18 18:09:23 -05:00
Eric Eastwood
61b7c31772
Sliding Sync: Shortcut for checking if certain background updates have completed (#17724)
Shortcut for checking if certain background updates have completed

Pulling this change out from one of @erikjohnston's branches
(https://github.com/element-hq/synapse/compare/develop...erikj/ss_perf)

---------

Co-authored-by: Erik Johnston <erikj@element.io>
2024-09-18 13:12:14 -05:00
Devon Hudson
4dfaa7f772
Merge branch 'origin/develop' into matrix-org-hotfixes 2024-09-18 10:39:11 -06:00
Kegan Dougal
3c8a116e1a
Sliding Sync: bugfix: ensure we can sync with SSS even with missing rooms (#17727)
Fixes https://github.com/element-hq/element-x-ios/issues/3300

Some rooms are missing from `sliding_sync_joined_rooms`. When this
happens, the first call will succeed, but any subsequent calls for this
room ID will cause the cache to return `None` for the room ID, rather
than not having the key at all. This then causes the `<=` check to
throw.

Root cause: https://github.com/element-hq/synapse/issues/17726

### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [ ] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [ ] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
2024-09-18 16:25:50 +00:00
Shay
51dd4df0a3
Add an Admin API endpoint to redact all a user's events (#17506) 2024-09-18 10:08:01 +00:00
Eric Eastwood
8881ad6d4b
Sliding Sync: Short-circuit have_finished_sliding_sync_background_jobs (#17723)
We only need to check it if returned bump stamp is `None`, which is rare.

Pulling this change out from one of @erikjohnston's branches
(https://github.com/element-hq/synapse/compare/develop...erikj/ss_perf)
2024-09-17 17:36:59 -05:00
Olivier 'reivilibre
d40bc279ed Merge branch 'master' into develop 2024-09-17 15:47:32 +01:00
Olivier 'reivilibre
521532a3dc Merge branch 'develop' into matrix-org-hotfixes 2024-09-17 14:33:32 +01:00
Olivier 'reivilibre
d10872ee75 1.115.0 2024-09-17 14:32:29 +01:00
Eric Eastwood
03937a1cae
Sliding Sync: Return room tags in account data extension (#17707)
The account data extension was also updated to avoid copies when we pull
the data out of the cache.

Fix https://github.com/element-hq/synapse/issues/17694
2024-09-16 13:47:35 -05:00
dependabot[bot]
285de43e48
Bump anyhow from 1.0.87 to 1.0.89 (#17716)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-16 18:52:48 +01:00
dependabot[bot]
4900438712
Bump pyasn1 from 0.6.0 to 0.6.1 (#17714)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-16 18:52:10 +01:00
dependabot[bot]
cf982d2e32
Bump ruff from 0.6.4 to 0.6.5 (#17715)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-16 18:51:33 +01:00
dependabot[bot]
7589565edd
Bump types-requests from 2.32.0.20240712 to 2.32.0.20240914 (#17713)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-16 18:32:39 +01:00
dependabot[bot]
7ed23e072e
Bump sentry-sdk from 2.13.0 to 2.14.0 (#17712) 2024-09-16 18:32:01 +01:00
Andrew Morgan
3db6a83d35 Merge branch 'develop' of github.com:element-hq/synapse into matrix-org-hotfixes 2024-09-16 10:32:01 +01:00
David Baker
4ac783549c
Sliding Sync: Support filtering by 'tags' / 'not_tags' in SSS (#17662)
This appears to be enough to make Element Web work (or at least move it
on to the next hurdle)

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-09-12 20:18:19 -05:00
Erik Johnston
1cb84aaab5
Sliding Sync: Increase concurrency of sliding sync a bit (#17696)
For initial requests a typical page size is 20 rooms, so we may as well
do the batching as 20.

This should speed up bigger syncs a little bit.
2024-09-12 16:36:16 -05:00
Eric Eastwood
9b83fb7c16
Sliding Sync: Move filters tests to rest layer (#17703)
Move filters tests to rest layer in order to test the new (with sliding
sync tables) and fallback paths that Sliding Sync can use.

Also found a bug in the new path because it's not being tested which is
also fixed in this PR. We now take into account `has_known_state` when
filtering.

Spawning from
https://github.com/element-hq/synapse/pull/17662#discussion_r1755574791.
This should have been done when we started using the new sliding sync
tables in https://github.com/element-hq/synapse/pull/17630
2024-09-12 15:27:03 -05:00
Andrew Morgan
c5b4be6d07 Merge branch 'release-v1.115' into develop 2024-09-12 13:05:43 +01:00
Andrew Morgan
a9e191b28e Merge branch 'release-v1.115' of github.com:element-hq/synapse into matrix-org-hotfixes 2024-09-12 11:20:45 +01:00
Andrew Morgan
4c66a7cbed 1.115.0rc2 2024-09-12 11:10:31 +01:00
Éloi Rivard
ebad618bf0
import pydantic objects from the _pydantic_compat module (#17667)
This PR changes `from pydantic import BaseModel` to `from
synapse._pydantic_compat import BaseModel` (as well as `constr`,
`conbytes`, `conint`, `confloat`).

It allows `check_pydantic_models.py` to mock those pydantic objects only
in the synapse module, and not interfere with pydantic objects in
external dependencies.

This should solve the CI problems for #17144, which breaks because
`check_pydantic_models.py` patches pydantic models from
[scim2-models](https://scim2-models.readthedocs.io/).

/cc @DMRobertson @gotmax23
fixes #17659 


### Pull Request Checklist

<!-- Please read
https://element-hq.github.io/synapse/latest/development/contributing_guide.html
before submitting your pull request -->

* [x] Pull request is based on the develop branch
* [x] Pull request includes a [changelog
file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog).
The entry should:
- Be a short description of your change which makes sense to users.
"Fixed a bug that prevented receiving messages from other servers."
instead of "Moved X method from `EventStore` to `EventWorkerStore`.".
  - Use markdown where necessary, mostly for `code blocks`.
  - End with either a period (.) or an exclamation mark (!).
  - Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by
@github_username." or "Contributed by [Your Name]." to the end of the
entry.
* [x] [Code
style](https://element-hq.github.io/synapse/latest/code_style.html) is
correct
(run the
[linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
2024-09-11 21:01:43 +00:00
Eric Eastwood
16af80b8fb
Sliding Sync: Use Sliding Sync tables for sorting (#17693)
Use Sliding Sync tables for sorting
(`bulk_get_last_event_pos_in_room_before_stream_ordering(...)` ->
`_bulk_get_max_event_pos(...)`)
2024-09-11 12:16:24 -05:00
Eric Eastwood
e4a1f271b9
Sliding Sync: Make sure we get up-to-date information from get_sliding_sync_rooms_for_user(...) (#17692)
We need to bust the `get_sliding_sync_rooms_for_user`
cache when the room encryption is updated and any
other field that is used in the query.

Follow-up to https://github.com/element-hq/synapse/pull/17630

- Bust cache for membership change (cross-reference
`get_rooms_for_user`)
- Bust cache for room `encryption` (cross-reference
`get_room_encryption`)
- Bust cache for `forgotten` (cross-reference
`did_forget`/`get_forgotten_rooms_for_user`)
2024-09-11 12:13:54 -05:00
Erik Johnston
6b131a99fe Merge remote-tracking branch 'origin/release-v1.115' into develop 2024-09-11 16:43:07 +01:00
Erik Johnston
76f7c91e44 Sliding sync: don't fetch room summary for named rooms. (#17683)
For rooms with a name we can skip fetching a full room summary, as we
don't need to calculate heroes, and instead just fetch the room counts
directly.

This also changes things to not return counts and heroes for non-joined
rooms. For left/banned rooms we were returning zero values anyway, and
for invite/knock rooms we don't really want to leak such information
(even if some of is included in the stripped state).
2024-09-11 16:42:50 +01:00
Erik Johnston
b732d13d4c Sliding sync: various fixups to the background update (#17652) 2024-09-11 16:42:15 +01:00
Erik Johnston
596b96411b
Sliding sync: various fixups to the background update (#17652) 2024-09-11 15:38:46 +01:00
Erik Johnston
7bb499154a Merge remote-tracking branch 'origin/develop' into matrix-org-hotfixes 2024-09-11 14:16:51 +01:00
Erik Johnston
f6c2b0ec2e
Sliding sync: don't fetch room summary for named rooms. (#17683)
For rooms with a name we can skip fetching a full room summary, as we
don't need to calculate heroes, and instead just fetch the room counts
directly.

This also changes things to not return counts and heroes for non-joined
rooms. For left/banned rooms we were returning zero values anyway, and
for invite/knock rooms we don't really want to leak such information
(even if some of is included in the stripped state).
2024-09-11 13:16:57 +01:00
Travis Ralston
a7fcac5648
Enable guest access on new media endpoints, per MSC4189 (#17675) 2024-09-10 18:29:24 +01:00
V02460
e06e3c4004
Add config option turn_shared_secret_path (#17690)
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-09-10 17:27:46 +00:00
dependabot[bot]
60441059a3
Bump anyhow from 1.0.86 to 1.0.87 (#17685)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-10 18:05:31 +01:00
Jeremy Wright
1b197752b6
Fix minor misspelling in README.rst. (#17664) 2024-09-10 17:33:25 +01:00
dependabot[bot]
598a83d005
Bump cryptography from 43.0.0 to 43.0.1 (#17689)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-10 17:32:17 +01:00
dependabot[bot]
be603de2cb
Bump serde_json from 1.0.127 to 1.0.128 (#17687)
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-10 17:31:34 +01:00
dependabot[bot]
62523571ae
Bump serde from 1.0.209 to 1.0.210 (#17686) 2024-09-10 17:30:37 +01:00
Devon Hudson
5562a89168
Update changelog 2024-09-10 08:48:41 -06:00
Devon Hudson
59bcbcec0a
1.115.0rc1 2024-09-10 08:42:01 -06:00
Erik Johnston
e7234d0ce1 Merge remote-tracking branch 'origin/develop' into matrix-org-hotfixes 2024-09-10 10:35:30 +01:00
dependabot[bot]
d8b926d323
Bump idna from 3.7 to 3.8 (#17682) 2024-09-10 10:34:54 +01:00
dependabot[bot]
2efed1d4fb
Bump types-setuptools from 71.1.0.20240818 to 74.1.0.20240907 (#17681) 2024-09-10 10:34:45 +01:00
dependabot[bot]
cd24bc2f36
Bump ruff from 0.6.2 to 0.6.4 (#17680) 2024-09-10 10:34:34 +01:00
dependabot[bot]
a193d4a1b5
Bump authlib from 1.3.1 to 1.3.2 (#17679) 2024-09-10 10:34:26 +01:00
Erik Johnston
0c512abce1 Merge remote-tracking branch 'origin/develop' into matrix-org-hotfixes 2024-09-10 10:32:01 +01:00
Erik Johnston
b3047f3f17
Sliding sync: various fixups to the sliding sync joined room background job (#17673)
Follow-up to #17652, https://github.com/element-hq/synapse/pull/17641,
https://github.com/element-hq/synapse/pull/17634,
https://github.com/element-hq/synapse/pull/17631 and
https://github.com/element-hq/synapse/pull/17632 to fix-up
https://github.com/element-hq/synapse/pull/17512
2024-09-10 10:22:46 +01:00
Erik Johnston
9689ac3294
Sliding Sync: Look for bump _stamp in the room timeline (#17684)
This allows us to skip checking the database a lot of the time.

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-09-10 10:20:30 +01:00
Erik Johnston
588e5b521d
Sliding Sync: Retrieve fewer events from DB in sync (#17688)
When using timeline limit of 1 we end up fetching 2 events from the DB
purely to tell if the response was "limited" or not. Lets not do that.
2024-09-10 09:52:42 +01:00