Merge branch 'develop' into rav/pass_site_to_make_request

CHANGES.md

@@ -1,3 +1,89 @@
+Synapse 1.23.0rc1 (2020-11-13)
+==============================
+
+Features
+--------
+
+- Add a push rule that highlights when a jitsi conference is created in a room. ([\#8286](https://github.com/matrix-org/synapse/issues/8286))
+- Add an admin api to delete a single file or files that were not used for a defined time from server. Contributed by @dklimpel. ([\#8519](https://github.com/matrix-org/synapse/issues/8519))
+- Split admin API for reported events (`GET /_synapse/admin/v1/event_reports`) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel. ([\#8539](https://github.com/matrix-org/synapse/issues/8539))
+- Support generating structured logs via the standard logging configuration. ([\#8607](https://github.com/matrix-org/synapse/issues/8607), [\#8685](https://github.com/matrix-org/synapse/issues/8685))
+- Add an admin API to allow server admins to list users' pushers. Contributed by @dklimpel. ([\#8610](https://github.com/matrix-org/synapse/issues/8610), [\#8689](https://github.com/matrix-org/synapse/issues/8689))
+- Add an admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information about uploaded media. Contributed by @dklimpel. ([\#8647](https://github.com/matrix-org/synapse/issues/8647))
+- Add an admin API for local user media statistics. Contributed by @dklimpel. ([\#8700](https://github.com/matrix-org/synapse/issues/8700))
+- Add `displayname` to Shared-Secret Registration for admins. ([\#8722](https://github.com/matrix-org/synapse/issues/8722))
+
+
+Bugfixes
+--------
+
+- Fix fetching of E2E cross signing keys over federation when only one of the master key and device signing key is cached already. ([\#8455](https://github.com/matrix-org/synapse/issues/8455))
+- Fix a bug where Synapse would blindly forward bad responses from federation to clients when retrieving profile information. ([\#8580](https://github.com/matrix-org/synapse/issues/8580))
+- Fix a bug where the account validity endpoint would silently fail if the user ID did not have an expiration time. It now returns a 400 error. ([\#8620](https://github.com/matrix-org/synapse/issues/8620))
+- Fix email notifications for invites without local state. ([\#8627](https://github.com/matrix-org/synapse/issues/8627))
+- Fix handling of invalid group IDs to return a 400 rather than log an exception and return a 500. ([\#8628](https://github.com/matrix-org/synapse/issues/8628))
+- Fix handling of User-Agent headers that are invalid UTF-8, which caused user agents of users to not get correctly recorded. ([\#8632](https://github.com/matrix-org/synapse/issues/8632))
+- Fix a bug in the `joined_rooms` admin API if the user has never joined any rooms. The bug was introduced, along with the API, in v1.21.0. ([\#8643](https://github.com/matrix-org/synapse/issues/8643))
+- Fix exception during handling multiple concurrent requests for remote media when using multiple media repositories. ([\#8682](https://github.com/matrix-org/synapse/issues/8682))
+- Fix bug that prevented Synapse from recovering after losing connection to the database. ([\#8726](https://github.com/matrix-org/synapse/issues/8726))
+- Fix bug where the `/_synapse/admin/v1/send_server_notice` API could send notices to non-notice rooms. ([\#8728](https://github.com/matrix-org/synapse/issues/8728))
+- Fix PostgreSQL port script fails when DB has no backfilled events. Broke in v1.21.0. ([\#8729](https://github.com/matrix-org/synapse/issues/8729))
+- Fix PostgreSQL port script to correctly handle foreign key constraints. Broke in v1.21.0. ([\#8730](https://github.com/matrix-org/synapse/issues/8730))
+- Fix PostgreSQL port script so that it can be run again after a failure. Broke in v1.21.0. ([\#8755](https://github.com/matrix-org/synapse/issues/8755))
+
+
+Improved Documentation
+----------------------
+
+- Instructions for Azure AD in the OpenID Connect documentation. Contributed by peterk. ([\#8582](https://github.com/matrix-org/synapse/issues/8582))
+- Improve the sample configuration for single sign-on providers. ([\#8635](https://github.com/matrix-org/synapse/issues/8635))
+- Fix the filepath of Dex's example config and the link to Dex's Getting Started guide in the OpenID Connect docs. ([\#8657](https://github.com/matrix-org/synapse/issues/8657))
+- Note support for Python 3.9. ([\#8665](https://github.com/matrix-org/synapse/issues/8665))
+- Minor updates to docs on running tests. ([\#8666](https://github.com/matrix-org/synapse/issues/8666))
+- Interlink prometheus/grafana documentation. ([\#8667](https://github.com/matrix-org/synapse/issues/8667))
+- Notes on SSO logins and media_repository worker. ([\#8701](https://github.com/matrix-org/synapse/issues/8701))
+- Document experimental support for running multiple event persisters. ([\#8706](https://github.com/matrix-org/synapse/issues/8706))
+- Add information regarding the various sources of, and expected contributions to, Synapse's documentation to `CONTRIBUTING.md`. ([\#8714](https://github.com/matrix-org/synapse/issues/8714))
+- Migrate documentation `docs/admin_api/event_reports` to markdown. ([\#8742](https://github.com/matrix-org/synapse/issues/8742))
+- Add some helpful hints to the README for new Synapse developers. Contributed by @chagai95. ([\#8746](https://github.com/matrix-org/synapse/issues/8746))
+
+
+Internal Changes
+----------------
+
+- Optimise `/createRoom` with multiple invited users. ([\#8559](https://github.com/matrix-org/synapse/issues/8559))
+- Implement and use an @lru_cache decorator. ([\#8595](https://github.com/matrix-org/synapse/issues/8595))
+- Don't instansiate Requester directly. ([\#8614](https://github.com/matrix-org/synapse/issues/8614))
+- Type hints for `RegistrationStore`. ([\#8615](https://github.com/matrix-org/synapse/issues/8615))
+- Change schema to support access tokens belonging to one user but granting access to another. ([\#8616](https://github.com/matrix-org/synapse/issues/8616))
+- Remove unused OPTIONS handlers. ([\#8621](https://github.com/matrix-org/synapse/issues/8621))
+- Run `mypy` as part of the lint.sh script. ([\#8633](https://github.com/matrix-org/synapse/issues/8633))
+- Correct Synapse's PyPI package name in the OpenID Connect installation instructions. ([\#8634](https://github.com/matrix-org/synapse/issues/8634))
+- Catch exceptions during initialization of `password_providers`. Contributed by Nicolai Søborg. ([\#8636](https://github.com/matrix-org/synapse/issues/8636))
+- Fix typos and spelling errors in the code. ([\#8639](https://github.com/matrix-org/synapse/issues/8639))
+- Reduce number of OpenTracing spans started. ([\#8640](https://github.com/matrix-org/synapse/issues/8640), [\#8668](https://github.com/matrix-org/synapse/issues/8668), [\#8670](https://github.com/matrix-org/synapse/issues/8670))
+- Add field `total` to device list in admin API. ([\#8644](https://github.com/matrix-org/synapse/issues/8644))
+- Add more type hints to the application services code. ([\#8655](https://github.com/matrix-org/synapse/issues/8655), [\#8693](https://github.com/matrix-org/synapse/issues/8693))
+- Tell Black to format code for Python 3.5. ([\#8664](https://github.com/matrix-org/synapse/issues/8664))
+- Don't pull event from DB when handling replication traffic. ([\#8669](https://github.com/matrix-org/synapse/issues/8669))
+- Abstract some invite-related code in preparation for landing knocking. ([\#8671](https://github.com/matrix-org/synapse/issues/8671), [\#8688](https://github.com/matrix-org/synapse/issues/8688))
+- Clarify representation of events in logfiles. ([\#8679](https://github.com/matrix-org/synapse/issues/8679))
+- Don't require `hiredis` package to be installed to run unit tests. ([\#8680](https://github.com/matrix-org/synapse/issues/8680))
+- Fix typing info on cache call signature to accept `on_invalidate`. ([\#8684](https://github.com/matrix-org/synapse/issues/8684))
+- Fail tests if they do not await coroutines. ([\#8690](https://github.com/matrix-org/synapse/issues/8690))
+- Improve start time by adding an index to `e2e_cross_signing_keys.stream_id`. ([\#8694](https://github.com/matrix-org/synapse/issues/8694))
+- Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8697](https://github.com/matrix-org/synapse/issues/8697))
+- Use Python 3.8 in Docker images by default. ([\#8698](https://github.com/matrix-org/synapse/issues/8698))
+- Remove the "draft" status of the Room Details Admin API. ([\#8702](https://github.com/matrix-org/synapse/issues/8702))
+- Improve the error returned when a non-string displayname or avatar_url is used when updating a user's profile. ([\#8705](https://github.com/matrix-org/synapse/issues/8705))
+- Block attempts by clients to send server ACLs, or redactions of server ACLs, that would result in the local server being blocked from the room. ([\#8708](https://github.com/matrix-org/synapse/issues/8708))
+- Add metrics the allow the local sysadmin to track 3PID `/requestToken` requests. ([\#8712](https://github.com/matrix-org/synapse/issues/8712))
+- Consolidate duplicated lists of purged tables that are checked in tests. ([\#8713](https://github.com/matrix-org/synapse/issues/8713))
+- Add some `mdui:UIInfo` element examples for `saml2_config` in the homeserver config. ([\#8718](https://github.com/matrix-org/synapse/issues/8718))
+- Improve the error message returned when a remote server incorrectly sets the `Content-Type` header in response to a JSON request. ([\#8719](https://github.com/matrix-org/synapse/issues/8719))
+- Speed up repeated state resolutions on the same room by caching event ID to auth event ID lookups. ([\#8752](https://github.com/matrix-org/synapse/issues/8752))
+
+
 Synapse 1.22.1 (2020-10-30)
 ===========================
 

changelog.d/8286.feature

@@ -1 +0,0 @@
-Add a push rule that highlights when a jitsi conference is created in a room.

changelog.d/8455.bugfix

@@ -1 +0,0 @@
-Fix fetching of E2E cross signing keys over federation when only one of the master key and device signing key is cached already.

changelog.d/8519.feature

@@ -1 +0,0 @@
-Add an admin api to delete a single file or files were not used for a defined time from server. Contributed by @dklimpel.

changelog.d/8539.feature

@@ -1 +0,0 @@
-Split admin API for reported events (`GET /_synapse/admin/v1/event_reports`) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel.

changelog.d/8559.misc

@@ -1 +0,0 @@
-Optimise `/createRoom` with multiple invited users.

changelog.d/8580.bugfix

@@ -1 +0,0 @@
-Fix a bug where Synapse would blindly forward bad responses from federation to clients when retrieving profile information.

changelog.d/8582.doc

@@ -1 +0,0 @@
-Instructions for Azure AD in the OpenID Connect documentation. Contributed by peterk.

changelog.d/8595.misc

@@ -1 +0,0 @@
-Implement and use an @lru_cache decorator.

changelog.d/8607.feature

@@ -1 +0,0 @@
-Support generating structured logs via the standard logging configuration.

changelog.d/8610.feature

@@ -1 +0,0 @@
-Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel.

changelog.d/8614.misc

@@ -1 +0,0 @@
-Don't instansiate Requester directly.

changelog.d/8615.misc

@@ -1 +0,0 @@
-Type hints for `RegistrationStore`.

changelog.d/8616.misc

@@ -1 +0,0 @@
-Change schema to support access tokens belonging to one user but granting access to another.

changelog.d/8620.bugfix

@@ -1 +0,0 @@
-Fix a bug where the account validity endpoint would silently fail if the user ID did not have an expiration time. It now returns a 400 error.

changelog.d/8621.misc

@@ -1 +0,0 @@
-Remove unused OPTIONS handlers.

changelog.d/8627.bugfix

@@ -1 +0,0 @@
-Fix email notifications for invites without local state.

changelog.d/8628.bugfix

@@ -1 +0,0 @@
-Fix handling of invalid group IDs to return a 400 rather than log an exception and return a 500.

changelog.d/8632.bugfix

@@ -1 +0,0 @@
-Fix handling of User-Agent headers that are invalid UTF-8, which caused user agents of users to not get correctly recorded.

changelog.d/8633.misc

@@ -1 +0,0 @@
-Run `mypy` as part of the lint.sh script.

changelog.d/8634.misc

@@ -1 +0,0 @@
-Correct Synapse's PyPI package name in the OpenID Connect installation instructions.

changelog.d/8635.doc

@@ -1 +0,0 @@
-Improve the sample configuration for single sign-on providers.

changelog.d/8636.misc

@@ -1 +0,0 @@
-Catch exceptions during initialization of `password_providers`. Contributed by Nicolai Søborg.

changelog.d/8639.misc

@@ -1 +0,0 @@
-Fix typos and spelling errors in the code.

changelog.d/8640.misc

@@ -1 +0,0 @@
-Reduce number of OpenTracing spans started.

changelog.d/8643.bugfix

@@ -1 +0,0 @@
-Fix a bug in the `joined_rooms` admin API if the user has never joined any rooms. The bug was introduced, along with the API, in v1.21.0.

changelog.d/8644.misc

@@ -1 +0,0 @@
-Add field `total` to device list in admin API.

changelog.d/8647.feature

@@ -1 +0,0 @@
-Add an admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information about uploaded media. Contributed by @dklimpel.

changelog.d/8655.misc

@@ -1 +0,0 @@
-Add more type hints to the application services code.

changelog.d/8657.doc

@@ -1 +0,0 @@
-Fix the filepath of Dex's example config and the link to Dex's Getting Started guide in the OpenID Connect docs.

changelog.d/8664.misc

@@ -1 +0,0 @@
-Tell Black to format code for Python 3.5.

changelog.d/8665.doc

@@ -1 +0,0 @@
-Note support for Python 3.9.

changelog.d/8666.doc

@@ -1 +0,0 @@
-Minor updates to docs on running tests.

changelog.d/8667.doc

@@ -1 +0,0 @@
-Interlink prometheus/grafana documentation.

changelog.d/8668.misc

@@ -1 +0,0 @@
-Reduce number of OpenTracing spans started.

changelog.d/8669.misc

@@ -1 +0,0 @@
-Don't pull event from DB when handling replication traffic.

changelog.d/8670.misc

@@ -1 +0,0 @@
-Reduce number of OpenTracing spans started.

changelog.d/8671.misc

@@ -1 +0,0 @@
-Abstract some invite-related code in preparation for landing knocking.

changelog.d/8679.misc

@@ -1 +0,0 @@
-Clarify representation of events in logfiles.

changelog.d/8680.misc

@@ -1 +0,0 @@
-Don't require `hiredis` package to be installed to run unit tests.

changelog.d/8682.bugfix

@@ -1 +0,0 @@
-Fix exception during handling multiple concurrent requests for remote media when using multiple media repositories.

changelog.d/8684.misc

@@ -1 +0,0 @@
-Fix typing info on cache call signature to accept `on_invalidate`.

changelog.d/8685.feature

@@ -1 +0,0 @@
-Support generating structured logs via the standard logging configuration.

changelog.d/8688.misc

@@ -1 +0,0 @@
-Abstract some invite-related code in preparation for landing knocking.

changelog.d/8689.feature

@@ -1 +0,0 @@
-Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel.

changelog.d/8690.misc

@@ -1 +0,0 @@
-Fail tests if they do not await coroutines.

changelog.d/8693.misc

@@ -1 +0,0 @@
-Add more type hints to the application services code.

changelog.d/8694.misc

@@ -1 +0,0 @@
-Improve start time by adding an index to `e2e_cross_signing_keys.stream_id`.

changelog.d/8697.misc

@@ -1 +0,0 @@
- Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting.

changelog.d/8698.misc

@@ -1 +0,0 @@
-Use Python 3.8 in Docker images by default.

changelog.d/8700.feature

@@ -1 +0,0 @@
-Add an admin API for local user media statistics. Contributed by @dklimpel.

changelog.d/8701.doc

@@ -1 +0,0 @@
-Notes on SSO logins and media_repository worker.

changelog.d/8702.misc

@@ -1 +0,0 @@
-Remove the "draft" status of the Room Details Admin API.

changelog.d/8705.misc

@@ -1 +0,0 @@
-Improve the error returned when a non-string displayname or avatar_url is used when updating a user's profile.

changelog.d/8706.doc

@@ -1 +0,0 @@
-Document experimental support for running multiple event persisters.

changelog.d/8708.misc

@@ -1 +0,0 @@
-Block attempts by clients to send server ACLs, or redactions of server ACLs, that would result in the local server being blocked from the room.

changelog.d/8712.misc

@@ -1 +0,0 @@
-Add metrics the allow the local sysadmin to track 3PID `/requestToken` requests.

changelog.d/8713.misc

@@ -1 +0,0 @@
-Consolidate duplicated lists of purged tables that are checked in tests.

changelog.d/8714.doc

@@ -1 +0,0 @@
-Add information regarding the various sources of, and expected contributions to, Synapse's documentation to `CONTRIBUTING.md`.

changelog.d/8718.misc

@@ -1 +0,0 @@
-Add some `mdui:UIInfo` element examples for `saml2_config` in the homeserver config.

changelog.d/8719.misc

@@ -1 +0,0 @@
-Improve the error message returned when a remote server incorrectly sets the `Content-Type` header in response to a JSON request.

changelog.d/8722.feature

@@ -1 +0,0 @@
-Add `displayname` to Shared-Secret Registration for admins.

changelog.d/8726.bugfix

@@ -1 +0,0 @@
-Fix bug where Synapse would not recover after losing connection to the database.

changelog.d/8728.bugfix

@@ -1 +0,0 @@
-Fix bug where the `/_synapse/admin/v1/send_server_notice` API could send notices to non-notice rooms.

changelog.d/8729.bugfix

@@ -1 +0,0 @@
-Fix port script fails when DB has no backfilled events. Broke in v1.21.0.

changelog.d/8730.bugfix

@@ -1 +0,0 @@
-Fix port script to correctly handle foreign key constraints. Broke in v1.21.0.

changelog.d/8734.doc

@@ -0,0 +1 @@
+Clarify the usecase for an msisdn delegate. Contributed by Adrian Wannenmacher.

changelog.d/8742.doc

@@ -1 +0,0 @@
-Migrate documentation `docs/admin_api/event_reports` to markdown.

changelog.d/8746.doc

@@ -1 +0,0 @@
-Add some helpful hints to the README for new Synapse developers. Contributed by @chagai95.

changelog.d/8752.misc

@@ -1 +0,0 @@
-Speed up repeated state resolutions on the same room by caching event ID to auth event ID lookups.

changelog.d/8754.misc

@@ -0,0 +1 @@
+Generalise `RoomStore.maybe_store_room_on_invite` to handle other, non-invite membership events.

changelog.d/8755.bugfix

@@ -1 +0,0 @@
-Fix port script so that it can be run again after a failure. Broke in v1.21.0.

changelog.d/8759.misc

@@ -0,0 +1 @@
+Refactor test utilities for injecting HTTP requests.

changelog.d/8760.misc

@@ -0,0 +1 @@
+Refactor test utilities for injecting HTTP requests.

docs/sample_config.yaml

@@ -1230,8 +1230,9 @@ account_validity:
 # email will be globally disabled.
 #
 # Additionally, if `msisdn` is not set, registration and password resets via msisdn
-# will be disabled regardless. This is due to Synapse currently not supporting any
+# will be disabled regardless, and users will not be able to associate an msisdn
-# method of sending SMS messages on its own.
+# identifier to their account. This is due to Synapse currently not supporting
+# any method of sending SMS messages on its own.
 #
 # To enable using an identity server for operations regarding a particular third-party
 # identifier type, set the value to the URL of that identity server as shown in the

synapse/__init__.py

@@ -48,7 +48,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.22.1"
+__version__ = "1.23.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/config/registration.py

@@ -347,8 +347,9 @@ class RegistrationConfig(Config):
         # email will be globally disabled.
         #
         # Additionally, if `msisdn` is not set, registration and password resets via msisdn
-        # will be disabled regardless. This is due to Synapse currently not supporting any
+        # will be disabled regardless, and users will not be able to associate an msisdn
-        # method of sending SMS messages on its own.
+        # identifier to their account. This is due to Synapse currently not supporting
+        # any method of sending SMS messages on its own.
         #
         # To enable using an identity server for operations regarding a particular third-party
         # identifier type, set the value to the URL of that identity server as shown in the

synapse/handlers/federation.py

@@ -67,7 +67,7 @@ from synapse.replication.http.devices import ReplicationUserDevicesResyncRestSer
 from synapse.replication.http.federation import (
     ReplicationCleanRoomRestServlet,
     ReplicationFederationSendEventsRestServlet,
-    ReplicationStoreRoomOnInviteRestServlet,
+    ReplicationStoreRoomOnOutlierMembershipRestServlet,
 )
 from synapse.state import StateResolutionStore
 from synapse.storage.databases.main.events_worker import EventRedactBehaviour
@@ -152,12 +152,14 @@ class FederationHandler(BaseHandler):
             self._user_device_resync = ReplicationUserDevicesResyncRestServlet.make_client(
                 hs
             )
-            self._maybe_store_room_on_invite = ReplicationStoreRoomOnInviteRestServlet.make_client(
+            self._maybe_store_room_on_outlier_membership = ReplicationStoreRoomOnOutlierMembershipRestServlet.make_client(
                 hs
             )
         else:
             self._device_list_updater = hs.get_device_handler().device_list_updater
-            self._maybe_store_room_on_invite = self.store.maybe_store_room_on_invite
+            self._maybe_store_room_on_outlier_membership = (
+                self.store.maybe_store_room_on_outlier_membership
+            )
 
         # When joining a room we need to queue any events for that room up.
         # For each room, a list of (pdu, origin) tuples.
@@ -1617,7 +1619,7 @@ class FederationHandler(BaseHandler):
         # keep a record of the room version, if we don't yet know it.
         # (this may get overwritten if we later get a different room version in a
         # join dance).
-        await self._maybe_store_room_on_invite(
+        await self._maybe_store_room_on_outlier_membership(
             room_id=event.room_id, room_version=room_version
         )
 

synapse/replication/http/federation.py

@@ -254,20 +254,20 @@ class ReplicationCleanRoomRestServlet(ReplicationEndpoint):
         return 200, {}
 
 
-class ReplicationStoreRoomOnInviteRestServlet(ReplicationEndpoint):
+class ReplicationStoreRoomOnOutlierMembershipRestServlet(ReplicationEndpoint):
     """Called to clean up any data in DB for a given room, ready for the
     server to join the room.
 
     Request format:
 
-        POST /_synapse/replication/store_room_on_invite/:room_id/:txn_id
+        POST /_synapse/replication/store_room_on_outlier_membership/:room_id/:txn_id
 
         {
             "room_version": "1",
         }
     """
 
-    NAME = "store_room_on_invite"
+    NAME = "store_room_on_outlier_membership"
     PATH_ARGS = ("room_id",)
 
     def __init__(self, hs):
@@ -282,7 +282,7 @@ class ReplicationStoreRoomOnInviteRestServlet(ReplicationEndpoint):
     async def _handle_request(self, request, room_id):
         content = parse_json_object_from_request(request)
         room_version = KNOWN_ROOM_VERSIONS[content["room_version"]]
-        await self.store.maybe_store_room_on_invite(room_id, room_version)
+        await self.store.maybe_store_room_on_outlier_membership(room_id, room_version)
         return 200, {}
 
 
@@ -291,4 +291,4 @@ def register_servlets(hs, http_server):
     ReplicationFederationSendEduRestServlet(hs).register(http_server)
     ReplicationGetQueryRestServlet(hs).register(http_server)
     ReplicationCleanRoomRestServlet(hs).register(http_server)
-    ReplicationStoreRoomOnInviteRestServlet(hs).register(http_server)
+    ReplicationStoreRoomOnOutlierMembershipRestServlet(hs).register(http_server)

synapse/storage/databases/main/room.py

@@ -1240,13 +1240,15 @@ class RoomStore(RoomBackgroundUpdateStore, RoomWorkerStore, SearchStore):
             logger.error("store_room with room_id=%s failed: %s", room_id, e)
             raise StoreError(500, "Problem creating room.")
 
-    async def maybe_store_room_on_invite(self, room_id: str, room_version: RoomVersion):
+    async def maybe_store_room_on_outlier_membership(
+        self, room_id: str, room_version: RoomVersion
+    ):
         """
-        When we receive an invite over federation, store the version of the room if we
+        When we receive an invite or any other event over federation that may relate to a room
-        don't already know the room version.
+        we are not in, store the version of the room if we don't already know the room version.
         """
         await self.db_pool.simple_upsert(
-            desc="maybe_store_room_on_invite",
+            desc="maybe_store_room_on_outlier_membership",
             table="rooms",
             keyvalues={"room_id": room_id},
             values={},

tests/replication/_base.py

@@ -240,8 +240,8 @@ class BaseMultiWorkerStreamTestCase(unittest.HomeserverTestCase):
             lambda: self._handle_http_replication_attempt(self.hs, 8765),
         )
 
-    def create_test_json_resource(self):
+    def create_test_resource(self):
-        """Overrides `HomeserverTestCase.create_test_json_resource`.
+        """Overrides `HomeserverTestCase.create_test_resource`.
         """
         # We override this so that it automatically registers all the HTTP
         # replication servlets, without having to explicitly do that in all

tests/rest/admin/test_admin.py

@@ -36,7 +36,7 @@ from tests.server import FakeSite, make_request
 class VersionTestCase(unittest.HomeserverTestCase):
     url = "/_synapse/admin/v1/server_version"
 
-    def create_test_json_resource(self):
+    def create_test_resource(self):
         resource = JsonResource(self.hs)
         VersionServlet(self.hs).register(resource)
         return resource

tests/rest/client/v1/utils.py

@@ -317,9 +317,7 @@ class RestHelper:
             path,
             content=image_data,
             access_token=tok,
-        )
+            custom_headers=[(b"Content-Length", str(image_length))],
-        request.requestHeaders.addRawHeader(
-            b"Content-Length", str(image_length).encode("UTF-8")
         )
         request.render(resource)
         self.hs.get_reactor().pump([100])

tests/rest/key/v2/test_remote_key_resource.py

@@ -41,7 +41,7 @@ class BaseRemoteKeyResourceTestCase(unittest.HomeserverTestCase):
         self.http_client = Mock()
         return self.setup_test_homeserver(http_client=self.http_client)
 
-    def create_test_json_resource(self):
+    def create_test_resource(self):
         return create_resource_tree(
             {"/_matrix/key/v2": KeyApiV2Resource(self.hs)}, root_resource=NoResource()
         )

tests/rest/test_health.py

@@ -20,11 +20,9 @@ from tests import unittest
 
 
 class HealthCheckTests(unittest.HomeserverTestCase):
-    def setUp(self):
+    def create_test_resource(self):
-        super().setUp()
-
         # replace the JsonResource with a HealthResource.
-        self.resource = HealthResource()
+        return HealthResource()
 
     def test_health(self):
         request, channel = self.make_request("GET", "/health", shorthand=False)

tests/rest/test_well_known.py

@@ -20,11 +20,9 @@ from tests import unittest
 
 
 class WellKnownTests(unittest.HomeserverTestCase):
-    def setUp(self):
+    def create_test_resource(self):
-        super().setUp()
-
         # replace the JsonResource with a WellKnownResource
-        self.resource = WellKnownResource(self.hs)
+        return WellKnownResource(self.hs)
 
     def test_well_known(self):
         self.hs.config.public_baseurl = "https://tesths"

tests/server.py

@@ -2,7 +2,7 @@ import json
 import logging
 from collections import deque
 from io import SEEK_END, BytesIO
-from typing import Callable
+from typing import Callable, Iterable, Optional, Tuple, Union
 
 import attr
 from typing_extensions import Deque
@@ -152,6 +152,9 @@ def make_request(
     shorthand=True,
     federation_auth_origin=None,
     content_is_form=False,
+    custom_headers: Optional[
+        Iterable[Tuple[Union[bytes, str], Union[bytes, str]]]
+    ] = None,
 ):
     """
     Make a web request using the given method and path, feed it the
@@ -172,6 +175,8 @@ def make_request(
         content_is_form: Whether the content is URL encoded form data. Adds the
             'Content-Type': 'application/x-www-form-urlencoded' header.
 
+        custom_headers: (name, value) pairs to add as request headers
+
     Returns:
         Tuple[synapse.http.site.SynapseRequest, channel]
     """
@@ -227,6 +232,10 @@ def make_request(
             # Assume the body is JSON
             req.requestHeaders.addRawHeader(b"Content-Type", b"application/json")
 
+    if custom_headers:
+        for k, v in custom_headers:
+            req.requestHeaders.addRawHeader(k, v)
+
     req.requestReceived(method, path, b"1.1")
 
     return req, channel

tests/storage/test_client_ips.py

@@ -21,6 +21,7 @@ from synapse.http.site import XForwardedForRequest
 from synapse.rest.client.v1 import login
 
 from tests import unittest
+from tests.server import make_request
 from tests.test_utils import make_awaitable
 from tests.unittest import override_config
 
@@ -408,17 +409,17 @@ class ClientIpAuthTestCase(unittest.HomeserverTestCase):
         # Advance to a known time
         self.reactor.advance(123456 - self.reactor.seconds())
 
-        request, channel = self.make_request(
+        headers1 = {b"User-Agent": b"Mozzila pizza"}
+        headers1.update(headers)
+
+        request, channel = make_request(
+            self.reactor,
             "GET",
             "/_matrix/client/r0/admin/users/" + self.user_id,
             access_token=access_token,
+            custom_headers=headers1.items(),
             **make_request_args,
         )
-        request.requestHeaders.addRawHeader(b"User-Agent", b"Mozzila pizza")
-
-        # Add the optional headers
-        for h, v in headers.items():
-            request.requestHeaders.addRawHeader(h, v)
         self.render(request)
 
         # Advance so the save loop occurs

tests/unittest.py

@@ -30,6 +30,7 @@ from twisted.internet.defer import Deferred, ensureDeferred, succeed
 from twisted.python.failure import Failure
 from twisted.python.threadpool import ThreadPool
 from twisted.trial import unittest
+from twisted.web.resource import Resource
 
 from synapse.api.constants import EventTypes, Membership
 from synapse.config.homeserver import HomeServerConfig
@@ -239,10 +240,8 @@ class HomeserverTestCase(TestCase):
         if not isinstance(self.hs, HomeServer):
             raise Exception("A homeserver wasn't returned, but %r" % (self.hs,))
 
-        # Register the resources
+        # create the root resource, and a site to wrap it.
-        self.resource = self.create_test_json_resource()
+        self.resource = self.create_test_resource()
-
-        # create a site to wrap the resource.
         self.site = SynapseSite(
             logger_name="synapse.access.http.fake",
             site_tag=self.hs.config.server.server_name,
@@ -323,15 +322,12 @@ class HomeserverTestCase(TestCase):
         hs = self.setup_test_homeserver()
         return hs
 
-    def create_test_json_resource(self):
+    def create_test_resource(self) -> Resource:
         """
-        Create a test JsonResource, with the relevant servlets registerd to it
+        Create a the root resource for the test server.
 
-        The default implementation calls each function in `servlets` to do the
+        The default implementation creates a JsonResource and calls each function in
-        registration.
+        `servlets` to register servletes against it
-
-        Returns:
-            JsonResource:
         """
         resource = JsonResource(self.hs)