Commit graph

24281 commits

Author SHA1 Message Date
dependabot[bot]
f4032d3e71
Bump serde from 1.0.208 to 1.0.209 (#17613) 2024-08-28 10:09:26 +01:00
eyJhb
8da16e55fe
hash_password accepts stdin now (#17608)
`hash_password` now actually accepts password from stdin. The `getpass`
reads from TTY, and does NOT accept stdin in any way.

The manpage has been updated to reflect that.
2024-08-27 18:51:43 +01:00
dependabot[bot]
d9cc0faf4b
Bump pyyaml from 6.0.1 to 6.0.2 (#17611) 2024-08-27 14:55:56 +01:00
dependabot[bot]
cca77af68f
Bump phonenumbers from 8.13.43 to 8.13.44 (#17610) 2024-08-27 14:55:47 +01:00
dependabot[bot]
48742da536
Bump attrs from 23.2.0 to 24.2.0 (#17609) 2024-08-27 14:55:38 +01:00
dependabot[bot]
940b932405
Bump pygithub from 2.3.0 to 2.4.0 (#17612) 2024-08-27 14:55:27 +01:00
dependabot[bot]
a2b2f6d09b
Bump serde_json from 1.0.125 to 1.0.127 (#17614) 2024-08-27 14:55:03 +01:00
Erik Johnston
defd4aca67
Speed up fetching latest stream positions via cache (#17606)
The idea is to engineer it so that the vast majority of the rooms can
stay in the cache, so we can just ignore them.
2024-08-27 11:03:56 +00:00
Erik Johnston
b4d95409fb
Fix @tag_args for non-methods (#17604)
The decorator assumed we were always wrapping function methods
2024-08-27 11:47:28 +01:00
dependabot[bot]
f1a1c7fc53
Bump types-setuptools from 71.1.0.20240726 to 71.1.0.20240818 (#17586) 2024-08-23 09:53:14 +01:00
dependabot[bot]
cb9fa062b7
Bump sentry-sdk from 2.12.0 to 2.13.0 (#17585) 2024-08-23 09:53:06 +01:00
dependabot[bot]
74b75cfd54
Bump cryptography from 42.0.8 to 43.0.0 (#17584) 2024-08-23 09:52:53 +01:00
dependabot[bot]
87d13fd143
Bump types-jsonschema from 4.23.0.20240712 to 4.23.0.20240813 (#17583) 2024-08-23 09:52:34 +01:00
dependabot[bot]
ad2cd9aefd
Bump serde_json from 1.0.124 to 1.0.125 (#17582) 2024-08-23 09:52:26 +01:00
dependabot[bot]
ad0ee53993
Bump serde from 1.0.206 to 1.0.208 (#17581) 2024-08-23 09:52:16 +01:00
Till Faelligen
84880cbe94
Route createRoom to event creators, add sliding_sync worker type 2024-08-21 11:56:08 +02:00
Erik Johnston
92b38c1afd
Sliding sync: Split up handler into its own module (#17595)
That file was getting long.

The changes are non functional, and simply split things up into:
- the main class
- the connection store
- the extensions
- the types
2024-08-20 18:30:23 +00:00
Quentin Gliech
a8e313836d
changelog: move SSSS some changes in the features section 2024-08-20 15:18:13 +02:00
Quentin Gliech
7c9684b5dc
1.114.0rc1 2024-08-20 14:57:22 +02:00
Erik Johnston
f1e8d2d15a
Sliding Sync: Speed up getting receipts for initial rooms (#17592)
Let's only pull out the events we care about. Note that the index isn't
necessary here, as postgres is happy to scan the set of rooms for the
events.
2024-08-20 12:57:34 +01:00
Erik Johnston
10428046e4
Add metrics for sliding sync processing time (#17593)
This should let us see how quickly we actually process things in
practice.
2024-08-20 11:36:49 +00:00
Erik Johnston
6eb98a4f1c
Sliding Sync: Handle timeline limit changes (take 2) (#17579)
This supersedes #17503, given the per-connection state is being heavily
rewritten it felt easier to recreate the PR on top of that work.

This correctly handles the case of timeline limits going up and down.

This does not handle changes in `required_state`, but that can be done
as a separate PR.

Based on #17575.

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-08-20 10:31:25 +01:00
Erik Johnston
950ba844f7
Sliding Sync: Batch up fetching receipts (#17589)
This is to make initial sliding sync a bit faster
2024-08-20 10:13:26 +01:00
Erik Johnston
8b8d74d12f
Sliding sync: Correctly track which read receipts we have or have not sent down. (#17575)
Add connection tracking to the receipts extension.

Based on #17574

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-08-19 21:16:07 +01:00
Erik Johnston
261e746281
Sliding sync: Add classes for per-connection state (#17574)
This is some prep work ahead of correctly tracking receipts, where we
will also want to track the room status in terms of last receipt we had
sent down.

Essentially, we add two classes `PerConnectionState` and a mutable
version, and then operate on those.

---------

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
2024-08-19 20:09:41 +01:00
Erik Johnston
993644ded0
Fix zero length media handling (#17570)
Results in:

```
AssertionError: null
  File "synapse/http/server.py", line 332, in _async_render_wrapper
    callback_return = await self._async_render(request)
  File "synapse/http/server.py", line 544, in _async_render
    callback_return = await raw_callback_return
  File "synapse/federation/transport/server/_base.py", line 369, in new_func
    response = await func(
  File "synapse/federation/transport/server/federation.py", line 826, in on_GET
    await self.media_repo.get_local_media(
  File "synapse/media/media_repository.py", line 473, in get_local_media
    await respond_with_multipart_responder(
  File "synapse/media/_base.py", line 353, in respond_with_multipart_responder
    assert content_length is not None
```
2024-08-19 15:06:44 +01:00
Erik Johnston
a5d25bb623
Test github token before running release script (#17562)
This stops people from getting half way through a step and it failing
due to the github token having expired (this happens to me every damn
time).
2024-08-19 14:15:36 +01:00
Till Faelligen
50bb114032
Add metrics 2024-08-19 14:53:24 +02:00
Erik Johnston
f162c92f2a
Speed up /keys/changes (#17548)
Follow on from #17537.

This is just adding a batched lookup function (you might want to hide
whitespace in the diff).
2024-08-16 16:04:02 +01:00
Erik Johnston
9ce489be5e
Add a flag to /versions about SSS support (#17571)
So that clients can check for support. Note that if the feature is only
enabled for some users, the `/versions` request must be authenticated to
pick up that SSS is enabled for the user
2024-08-16 08:54:57 +01:00
Till Faelligen
e466404b4f
Fix paths, remove logging (for now), disable passwords 2024-08-15 14:22:50 +02:00
Till Faelligen
4dd02db657
Fix parameter ordering 2024-08-15 11:33:10 +02:00
Andrew Morgan
333f806d06 fixup: Add SYNAPSE_CONFIG_TEMPLATE_DIR env var 2024-08-14 19:00:57 +01:00
Andrew Morgan
550d2a1e77 start.py: Support SYNAPSE_CONFIG_TEMPLATE_DIR 2024-08-14 18:57:23 +01:00
Andrew Morgan
61349bd860 create data directory if it doesn't exist
ideally this would be done in `main`
2024-08-14 18:52:47 +01:00
Andrew Morgan
20bd26ecad Allow configuring data dir in start.py 2024-08-14 18:47:47 +01:00
Andrew Morgan
e91cd9c722 Allow running python from a venv 2024-08-14 17:27:36 +01:00
Andrew Morgan
18a331a18c Document additional environment variables 2024-08-14 17:27:34 +01:00
Andrew Morgan
d7e3925a14 Allow specifying where generated config files are stored
* Store generated worker config files in SYNAPSE_CONFIG_DIR, rather than hardcoding
  `/conf`.
* Move the `workers_have_been_configured` filepath to the config dir.
2024-08-14 17:27:30 +01:00
Andrew Morgan
9df46343b9 Remove unused config_path dict entry 2024-08-14 17:27:27 +01:00
Andrew Morgan
a035d84ce0 Add SYNAPSE_CONFIG_TEMPLATE_DIR env var to worker config script
This allows specifying a directory other than `/conf` where config templates should live. This allows this
script to be used outside of the context of a docker container where /conf is bound to a dir on the host.
2024-08-14 17:27:24 +01:00
Andrew Morgan
fae75b0376
Register the media threadpool with our metrics (#17566)
Some checks failed
Tests / check-lockfile (push) Has been cancelled
Tests / lint-crlf (push) Has been cancelled
Tests / lint-newsfile (push) Has been cancelled
Deploy the documentation / GitHub Pages (push) Has been cancelled
Build release artifacts / Build .deb packages (push) Has been cancelled
Build release artifacts / Attach assets to release (push) Has been cancelled
Tests / check-sampleconfig (push) Has been cancelled
Tests / check-schema-delta (push) Has been cancelled
Tests / lint (push) Has been cancelled
Tests / Typechecking (push) Has been cancelled
Tests / lint-pydantic (push) Has been cancelled
Tests / lint-clippy (push) Has been cancelled
Tests / lint-clippy-nightly (push) Has been cancelled
Tests / lint-rustfmt (push) Has been cancelled
Tests / lint-readme (push) Has been cancelled
Tests / linting-done (push) Has been cancelled
Tests / calculate-test-jobs (push) Has been cancelled
Tests / trial (push) Has been cancelled
Tests / trial-olddeps (push) Has been cancelled
Tests / trial-pypy (all, pypy-3.8) (push) Has been cancelled
Tests / sytest (push) Has been cancelled
Tests / export-data (push) Has been cancelled
Tests / portdb (11, 3.8) (push) Has been cancelled
Tests / portdb (15, 3.11) (push) Has been cancelled
Tests / complement (monolith, Postgres) (push) Has been cancelled
Tests / complement (monolith, SQLite) (push) Has been cancelled
Tests / complement (workers, Postgres) (push) Has been cancelled
Tests / cargo-test (push) Has been cancelled
Tests / cargo-bench (push) Has been cancelled
Tests / tests-done (push) Has been cancelled
2024-08-14 15:11:22 +01:00
Tulir Asokan
f77bfbfa30
Fix fetching signing keys when old_verify_keys is omitted (#17568)
Some checks are pending
Build release artifacts / Build sdist (push) Waiting to run
Build release artifacts / Attach assets to release (push) Blocked by required conditions
Tests / changes (push) Waiting to run
Tests / check-sampleconfig (push) Blocked by required conditions
Tests / check-schema-delta (push) Blocked by required conditions
Tests / check-lockfile (push) Waiting to run
Tests / lint (push) Blocked by required conditions
Tests / Typechecking (push) Blocked by required conditions
Tests / lint-crlf (push) Waiting to run
Tests / lint-newsfile (push) Waiting to run
Tests / lint-pydantic (push) Blocked by required conditions
Tests / lint-clippy (push) Blocked by required conditions
Tests / trial-olddeps (push) Blocked by required conditions
Tests / trial-pypy (all, pypy-3.8) (push) Blocked by required conditions
Tests / portdb (11, 3.8) (push) Blocked by required conditions
Tests / complement (monolith, SQLite) (push) Blocked by required conditions
Tests / complement (workers, Postgres) (push) Blocked by required conditions
Tests / cargo-test (push) Blocked by required conditions
Tests / cargo-bench (push) Blocked by required conditions
Tests / tests-done (push) Blocked by required conditions
Tests / lint-clippy-nightly (push) Blocked by required conditions
Tests / lint-rustfmt (push) Blocked by required conditions
Tests / lint-readme (push) Blocked by required conditions
Tests / linting-done (push) Blocked by required conditions
Tests / calculate-test-jobs (push) Blocked by required conditions
Tests / trial (push) Blocked by required conditions
Tests / sytest (push) Blocked by required conditions
Tests / export-data (push) Blocked by required conditions
Tests / portdb (15, 3.11) (push) Blocked by required conditions
Tests / complement (monolith, Postgres) (push) Blocked by required conditions
`old_verify_keys` isn't marked as required in
https://spec.matrix.org/v1.11/server-server-api/#get_matrixkeyv2server
and there's no functional difference between an empty object and
omitting the object, so I don't think there's any reason synapse should
explode when the field is omitted.
2024-08-14 14:13:56 +01:00
Erik Johnston
1892ba5f67
Fix 'Producer was not unregistered' error (#17569)
Follows on from #17567
2024-08-14 13:46:22 +01:00
Erik Johnston
a51daffba5
Reduce concurrent thread usage in media (#17567)
Follow on from #17558

Basically, we want to reduce the number of threads we want to use at a
time, i.e. reduce the number of threads that are paused/blocked. We do
this by returning from the thread when the consumer pauses the producer,
rather than pausing in the thread.

---------

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2024-08-14 12:41:53 +01:00
Shay
b05b2e14bb
Handle lower-case http headers in _Mulitpart_Parser_Protocol (#17545) 2024-08-14 09:49:01 +01:00
Eric Eastwood
a308d99f30
Sliding Sync: Exclude partially stated rooms if we must await full state (#17538)
Some checks are pending
Build release artifacts / Build sdist (push) Waiting to run
Build release artifacts / Attach assets to release (push) Blocked by required conditions
Tests / lint (push) Blocked by required conditions
Tests / changes (push) Waiting to run
Tests / check-sampleconfig (push) Blocked by required conditions
Tests / check-schema-delta (push) Blocked by required conditions
Tests / check-lockfile (push) Waiting to run
Tests / Typechecking (push) Blocked by required conditions
Tests / lint-crlf (push) Waiting to run
Tests / lint-newsfile (push) Waiting to run
Tests / lint-pydantic (push) Blocked by required conditions
Tests / lint-clippy (push) Blocked by required conditions
Tests / lint-clippy-nightly (push) Blocked by required conditions
Tests / lint-rustfmt (push) Blocked by required conditions
Tests / lint-readme (push) Blocked by required conditions
Tests / linting-done (push) Blocked by required conditions
Tests / calculate-test-jobs (push) Blocked by required conditions
Tests / trial (push) Blocked by required conditions
Tests / trial-pypy (all, pypy-3.8) (push) Blocked by required conditions
Tests / sytest (push) Blocked by required conditions
Tests / export-data (push) Blocked by required conditions
Tests / trial-olddeps (push) Blocked by required conditions
Tests / portdb (11, 3.8) (push) Blocked by required conditions
Tests / portdb (15, 3.11) (push) Blocked by required conditions
Tests / complement (monolith, Postgres) (push) Blocked by required conditions
Tests / complement (monolith, SQLite) (push) Blocked by required conditions
Tests / complement (workers, Postgres) (push) Blocked by required conditions
Tests / cargo-test (push) Blocked by required conditions
Tests / cargo-bench (push) Blocked by required conditions
Tests / tests-done (push) Blocked by required conditions
Previously, we just had very basic partial room exclusion based on
whether we were lazy-loading room members. Now with this PR, we added
`must_await_full_state(...)` with rules to check if we have a we're only
requesting `required_state` which is completely satisfied even with
partial state.

Partially-stated rooms should have all state events except for remote
membership events so if we require a remote membership event anywhere,
then we need to return `True`.
2024-08-13 12:27:42 -05:00
Erik Johnston
a9fc1fd112
Use a larger, dedicated threadpool for media sending (#17564) 2024-08-13 17:59:47 +01:00
Andrew Morgan
6a11bdf01d
Add a utility function for generating fake event IDs (#17557) 2024-08-13 16:55:05 +00:00
Patrick Cloke
8fea190a1f
Add missing docstrings related to profile methods. (#17559) 2024-08-13 17:04:35 +01:00