8254 commits

Author	SHA1	Message	Date
Erik Johnston	2f5a77efae	Limit size of presence EDUs (#17371) ... Otherwise they are unbounded. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2024-07-01 12:48:36 +01:00
davidegirardi	b11f5c984b	Clarify url_preview_url_blacklist is a usability feature (#17356)	2024-06-28 14:39:54 +01:00
Till	27756c9fdf	Use rstcheck to "lint" the README (#17367) ... Follow up to https://github.com/element-hq/synapse/pull/17363, so we can detect issues with the RST file early on. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))	2024-06-27 13:36:08 +02:00
Erik Johnston	cc5e5893fe	Handle multiple rows device inbox (#17362) ... Fix bug where we don't get new to-device from remote if they resent a message we've already persisted and have recorded in the DB twice. `device_federation_inbox` table doesn't have a unique index, and so we can race and store an entry in there twice. If we do so then `simple_select_one_txn` will throw an error due to the query returning more than one row. We should add an unique index, but it doesn't really matter so lets just handle the case of multiple rows correctly for now.	2024-06-27 11:04:31 +01:00
Till Faelligen	7c169f4d2c	Merge branch 'release-v1.110' into develop	2024-06-26 19:00:51 +02:00
Till Faelligen	f75da9cc53	1.110.0rc2	2024-06-26 18:14:57 +02:00
Till	75c19bf57a	Fix uploading packages to PyPi (#17363) ... As per https://github.com/sphinx-doc/sphinx/issues/3921#issuecomment-315581557, we need double underscores. Running `rst2html README.rst > /dev/null` found some more warnings.	2024-06-26 18:14:29 +02:00
Till	b924a8e1a9	Fix uploading packages to PyPi (#17363) ... As per https://github.com/sphinx-doc/sphinx/issues/3921#issuecomment-315581557, we need double underscores. Running `rst2html README.rst > /dev/null` found some more warnings.	2024-06-26 18:01:39 +02:00
Till Faelligen	9f47513458	1.110.0rc1	2024-06-26 14:09:13 +02:00
Andrew Morgan	ef7fbdfebd	Fixes to the table of contents in the README (#17329)	2024-06-25 15:20:59 +00:00
Denis Kasak	9cf0ef9c70	Fix outdated Security Disclosure Policy references (#17341)	2024-06-25 15:58:30 +01:00
Shay	a023538822	Re-introduce federation /download endpoint (#17350)	2024-06-25 14:35:37 +00:00
Erik Johnston	c89fea3fd1	Limit amount of replication we send (#17358) ... Fixes up #17333, where we failed to actually send less data (the `DISTINCT` didn't work due to `stream_id` being different). We fix this by making it so that every device list outbound poke for a given user ID has the same stream ID. We can't change the query to only return e.g. max stream ID as the receivers look up the destinations to send to by doing `SELECT WHERE stream_id = ?`	2024-06-25 11:17:39 +01:00
Erik Johnston	554a92601a	Reintroduce "Reduce device lists replication traffic."" (#17361) ... Reintroduces https://github.com/element-hq/synapse/pull/17333 Turns out the reason for revert was down two master instances running	2024-06-25 10:34:34 +01:00
Erik Johnston	a98cb87bee	Revert "Reduce device lists replication traffic." (#17360) ... Reverts element-hq/synapse#17333 It looks like master was still sending out replication RDATA with the old format... somehow	2024-06-25 09:57:34 +01:00
Eric Eastwood	6e8af83193	Add is_invite filtering to Sliding Sync /sync (#17335) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-06-24 19:07:56 -05:00
devonh	805e6c9a8f	Correct error in user_directory docs (#17348) ... ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: reivilibre <oliverw@matrix.org>	2024-06-24 17:18:58 +00:00
devonh	3c61ddbbc9	Add default values for rc_invites per_issuer to docs (#17347) ... A simple change to update the docs where default values were missing. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Kim Brose <2803622+HarHarLinks@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>	2024-06-24 17:16:09 +00:00
Eric Eastwood	ae4c236a6d	Fix room type typo in mailer (#17336) ... Correct event content field is `EventContentFields.ROOM_TYPE` (`type`) ✅ , not `room_type` ❌ Spec: https://spec.matrix.org/v1.10/client-server-api/#mroomcreate	2024-06-24 10:02:16 -05:00
Erik Johnston	930a64b6c1	Reintroduce #17291. (#17338) ... This is #17291 (which got reverted), with some added fixups, and change so that tests actually pick up the error. The problem was that we were not calculating any new chain IDs due to a missing `not` in a condition.	2024-06-24 14:40:28 +00:00
Shay	7a11c0ac4f	Add support for MSC3823 - Account Suspension Part 2 (#17255)	2024-06-24 14:21:22 +01:00
Erik Johnston	cf711ac03c	Reduce device lists replication traffic. (#17333) ... Reduce the replication traffic of device lists, by not sending every destination that needs to be sent the device list update over replication. Instead a "hosts to send to have been calculated" notification over replication, and then federation senders read the destinations from the DB. For non federation senders this should heavily reduce the impact of a user in many large rooms changing a device.	2024-06-24 14:15:13 +01:00
Denis Kasak	700d2cc4a0	Tidy up integer parsing (#17339) ... The parse_integer function was previously made to reject negative values by default in https://github.com/element-hq/synapse/pull/16920, but the documentation stated otherwise. This fixes the documentation and also: - Removes explicit negative=False parameters from call sites. - Brings the negative default of parse_integer_from_args in alignment with parse_integer.	2024-06-24 15:12:14 +02:00
Erik Johnston	4243c1f074	Revert "Handle large chain calc better (#17291)" (#17334) ... This reverts commit bdf82efea5 (#17291) This seems to have stopped persisting auth chains for new events, and so is causing state res to fall back to the slow methods	2024-06-19 17:39:33 +01:00
Erik Johnston	3239b7459c	Register sliding sync under a different path (#17331) ... As the API is slightly incompatible.	2024-06-19 16:18:45 +00:00
Jörg Thalheim	c99203d98c	register-new-matrix-user: add a flag to ignore already existing users (#17304) ... Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>	2024-06-19 12:03:08 +01:00
Alexander Fechler	9104a9f0d0	Filter added to Admin-API GET /rooms (#17276)	2024-06-19 11:45:48 +01:00
Andrew Morgan	a412a5829d	Update the README with Element branding and a few fixes (#17324) ... Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>	2024-06-19 10:58:22 +01:00
Aaron Dewes	7ef89b985d	Remove expire_access_token from Docker configuration (#17198) ... Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>	2024-06-19 10:58:06 +01:00
Erik Johnston	bdf82efea5	Handle large chain calc better (#17291) ... We calculate the auth chain links outside of the main persist event transaction to ensure that we do not block other event sending during the calculation.	2024-06-19 10:33:53 +01:00
Andrew Morgan	afaf2d9388	Require the 'from' parameter for /notifications be an integer (#17283) ... Co-authored-by: Erik Johnston <erikj@element.io>	2024-06-19 10:05:39 +01:00
Andrew Morgan	199223062a	Revert "Support MSC3916 by adding a federation /download endpoint" (#17325)	2024-06-18 16:54:19 +01:00
Jörg Thalheim	97c3d98816	register_new_matrix_user: add password-file flag (#17294) ... Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>	2024-06-18 16:21:51 +01:00
Johannes Marbach	79767a1108	Add support for via query parameter from MSC4156 (#17322) ... This adds support for the `via` query parameter from https://github.com/matrix-org/matrix-spec-proposals/pull/4156.	2024-06-18 10:03:39 +00:00
Dirk Klimpel	1c7d85fdfe	fix missing quotes for exclude_rooms_from_sync (#17308) ... We tried to configure rooms `exclude_rooms_from_sync`. If we do not quote we get an error. The example should be valid.	2024-06-18 10:37:02 +01:00
Eric Eastwood	a5485437cf	Add is_encrypted filtering to Sliding Sync /sync (#17281) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-06-17 12:06:18 -05:00
Eric Eastwood	e5b8a3e37f	Add stream_ordering sort to Sliding Sync /sync (#17293) ... Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.	2024-06-17 11:27:14 -05:00
Richard van der Hoff	3aae60f17b	Enable cross-signing key upload without UIA (#17284) ... Per MSC3967, which is now stable, we should not require UIA when uploading cross-signing keys for the first time. Fixes: #17227	2024-06-14 11:14:56 +01:00
Richard van der Hoff	2c36a679ae	Include user membership on events (#17282) ... MSC4115 has now completed FCP, so we can enable it by default and switch to the stable identifier.	2024-06-13 21:45:54 +00:00
Eric Eastwood	c12ee0d5ba	Add is_dm filtering to Sliding Sync /sync (#17277) ... Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync	2024-06-13 13:56:58 -05:00
Eric Eastwood	8aaff851b1	Fix newly_left rooms not appearing if we returned early (Sliding Sync) (#17301) ... Fix `newly_left` rooms not appearing if we returned early when `membership_snapshot_token.is_before_or_eq(to_token.room_key)`. Introduced in https://github.com/element-hq/synapse/pull/17187 (part of Sliding Sync) The tests didn't catch it because they had a small typo in it `room_id1` vs `room_id2`. Found while working on https://github.com/element-hq/synapse/pull/17293	2024-06-13 11:36:57 -05:00
Eric Eastwood	8c58eb7f17	Add event.internal_metadata.instance_name (#17300) ... Add `event.internal_metadata.instance_name` (the worker instance that persisted the event) to go alongside the existing `event.internal_metadata.stream_ordering`. `instance_name` is useful to properly compare and query for events with a token since you need to compare both the `stream_ordering` and `instance_name` against the vector clock/`instance_map` in the `RoomStreamToken`. This is pre-requisite work and may be used in https://github.com/element-hq/synapse/pull/17293 Adding `event.internal_metadata.instance_name` was first mentioned in the initial Sliding Sync PR while pairing with @erikjohnston, see 09609cb0db (diff-5cd773fb307aa754bd3948871ba118b1ef0303f4d72d42a2d21e38242bf4e096R405-R410)	2024-06-13 11:32:50 -05:00
Eric Eastwood	ebdce69f6a	Fix get_last_event_in_room_before_stream_ordering(...) finding the wrong last event (#17295) ... PR where this was introduced: https://github.com/matrix-org/synapse/pull/14817 ### What does this affect? `get_last_event_in_room_before_stream_ordering(...)` is used in Sync v2 in a lot of different state calculations. `get_last_event_in_room_before_stream_ordering(...)` is also used in `/rooms/{roomId}/members`	2024-06-13 11:00:52 -05:00
Andrew Morgan	c6eb99c878	Bump mypy from 1.8.0 to 1.9.0 (#17297) ... Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>	2024-06-13 15:50:10 +01:00
Quentin Gliech	5db3eec5bc	Clarify that MSC4151 is enabled on matrix.org (#17296) ... This clarifies in the comments that the MSC is being used in matrix.org See #17270	2024-06-13 13:49:00 +00:00
Travis Ralston	f1c4dfb08b	Add report room API (MSC4151) (#17270) ... https://github.com/matrix-org/matrix-spec-proposals/pull/4151 This is intended to be enabled by default for immediate use. When FCP is complete, the unstable endpoint will be dropped and stable endpoint supported instead - no backwards compatibility is expected for the unstable endpoint.	2024-06-12 12:27:46 +02:00
Eric Eastwood	dad1559721	Reorganize Pydantic models and types used in handlers (#17279) ... Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues. - Moved Pydantic parsing models to `synapse/types/rest` - Moved handler types to `synapse/types/handlers`	2024-06-10 15:03:50 -05:00
Eric Eastwood	265ee88f34	Wrong retention policy being used when filtering events (lint ControlVarUsedAfterBlockViolation WPS441) (#17272) ... Fix loop var being used outside block. Before this change, we were always using the last room_id's retention policy for all events being filtered. I found this bug with the [new lint rule, `ControlVarUsedAfterBlockViolation` `WPS441`](https://github.com/astral-sh/ruff/pull/11769), that I re-implemented in `ruff`. Shout-out to @reivilibre for all the help in the beginning! ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))	2024-06-10 09:48:38 -05:00
Shay	ab94bce02c	Support MSC3916 by adding a federation /download endpoint (#17172)	2024-06-07 13:54:28 +01:00
reivilibre	17d6c28285	Add debug logging for when room keys are uploaded, including whether they are replacing other room keys. (#17266) ... Fixes: #17013 Add logging for whether room keys are replaced This is motivated by the Crypto team who need to diagnose crypto issues. The existing opentracing logging is not enough because it is not enabled for all users.	2024-06-07 12:01:21 +01:00