Commit graph

66 commits

Author SHA1 Message Date
Olivier 'reivilibre
d5f42f4655 Use sticky routing for synchrotrons 2024-09-24 13:21:06 +01:00
Olivier 'reivilibre
b21487300c Add application_names to the postgres connections based on the worker names 2024-09-09 16:15:52 +01:00
Olivier 'reivilibre
5d2241e372 Configure sliding-sync workers with sticky load balancing 2024-09-02 16:27:02 +01:00
Till Faelligen
e64a35cb00
Route key queries to the client_reader 2024-08-30 12:01:31 +02:00
Olivier 'reivilibre
56bcba2b16 Add metrics listener config for main process 2024-08-29 11:48:22 +01:00
Olivier 'reivilibre
40a684e169 Gen metrics scraping configurations, for use with file_sd_configs 2024-08-28 18:58:02 +01:00
Till Faelligen
84880cbe94
Route createRoom to event creators, add sliding_sync worker type 2024-08-21 11:56:08 +02:00
Till Faelligen
50bb114032
Add metrics 2024-08-19 14:53:24 +02:00
Till Faelligen
4dd02db657
Fix parameter ordering 2024-08-15 11:33:10 +02:00
Andrew Morgan
333f806d06 fixup: Add SYNAPSE_CONFIG_TEMPLATE_DIR env var 2024-08-14 19:00:57 +01:00
Andrew Morgan
e91cd9c722 Allow running python from a venv 2024-08-14 17:27:36 +01:00
Andrew Morgan
18a331a18c Document additional environment variables 2024-08-14 17:27:34 +01:00
Andrew Morgan
d7e3925a14 Allow specifying where generated config files are stored
* Store generated worker config files in SYNAPSE_CONFIG_DIR, rather than hardcoding
  `/conf`.
* Move the `workers_have_been_configured` filepath to the config dir.
2024-08-14 17:27:30 +01:00
Andrew Morgan
9df46343b9 Remove unused config_path dict entry 2024-08-14 17:27:27 +01:00
Andrew Morgan
a035d84ce0 Add SYNAPSE_CONFIG_TEMPLATE_DIR env var to worker config script
This allows specifying a directory other than `/conf` where config templates should live. This allows this
script to be used outside of the context of a docker container where /conf is bound to a dir on the host.
2024-08-14 17:27:24 +01:00
Andrew Morgan
4d6f7c0fc9
Route auth'd fed media requests to media repo in Complement tests (#17422) 2024-07-09 15:12:11 +00:00
Shay
8f890447b0
Support MSC3916 by adding _matrix/client/v1/media/download endpoint (#17365) 2024-07-02 14:07:04 +01:00
Andrew Morgan
7cb3f8a979
Route /make_knock and /send_knock to workers in Complement docker image (#17195) 2024-05-16 16:53:26 +00:00
Erik Johnston
fd48fc4585
Fixups to new push stream (#17038)
Follow on from #17037
2024-03-28 16:29:23 +00:00
Erik Johnston
23740eaa3d
Correctly mention previous copyright (#16820)
During the migration the automated script to update the copyright
headers accidentally got rid of some of the existing copyright lines.
Reinstate them.
2024-01-23 11:26:48 +00:00
reivilibre
fa2700f001
Add a --generate-only option to the Complement launcher. (#16828)
Pulled out of #16803 since the drive-by cleanup was maybe not as
drive-by as I had hoped.

<!--
Fixes: # <!-- -->
<!--
Supersedes: # <!-- -->
<!--
Follows: # <!-- -->
<!--
Part of: # <!-- -->
Base: `develop` <!-- git-stack-base-branch:develop -->

<!--
This pull request is commit-by-commit review friendly. <!-- -->
<!--
This pull request is intended for commit-by-commit review. <!-- -->

Original commit schedule, with full messages:

<ol>
<li>

Add a --generate-only option 

</li>
</ol>

---------

Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2024-01-22 16:00:04 +00:00
Patrick Cloke
8e1e62c9e0 Update license headers 2023-11-21 15:29:58 -05:00
Erik Johnston
8940d1b28e
Add /notifications endpoint to workers (#16265) 2023-09-07 09:26:07 +00:00
Patrick Cloke
ad3f43be9a
Run pyupgrade for python 3.7 & 3.8. (#16110) 2023-08-15 08:11:20 -04:00
Jason Little
224ef0b669
Unix Sockets for HTTP Replication (#15708)
Unix socket support for `federation` and `client` Listeners has existed now for a little while(since [1.81.0](https://github.com/matrix-org/synapse/pull/15353)), but there was one last hold out before it could be complete: HTTP Replication communication. This should finish it up. The Listeners would have always worked, but would have had no way to be talked to/at.

---------

Co-authored-by: Eric Eastwood <madlittlemods@gmail.com>
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
Co-authored-by: Eric Eastwood <erice@element.io>
2023-07-11 13:08:06 -05:00
Eric Eastwood
0f02f0b4da
Remove experimental MSC2716 implementation to incrementally import history into existing rooms (#15748)
Context for why we're removing the implementation:

 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010
 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734

Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737

Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
2023-06-16 14:12:24 -05:00
Eric Eastwood
30a5076da8
Log when events are (unexpectedly) filtered out of responses in tests (#14213)
See https://github.com/matrix-org/synapse/pull/14095#discussion_r990335492

This is useful because when see that a relevant event is an `outlier` or `soft-failed`, then that's a good unexpected indicator explaining why it's not showing up. `filter_events_for_client` is used in `/sync`, `/messages`, `/context` which are all common end-to-end assertion touch points (also notifications, relations).
2023-06-01 21:27:18 -05:00
Jason Little
e4f545c452
Remove worker_replication_* settings (#15491)
* Add master to the instance_map as part of Complement, have ReplicationEndpoint look at instance_map for master.

* Fix typo in drive by.

* Remove unnecessary worker_replication_* bits from unit tests and add master to instance_map(hopefully in the right place)

* Several updates:

1. Switch from master to main for naming the main process in the instance_map. Add useful constants for easier adjustment of names in the future.
2. Add backwards compatibility for worker_replication_* to allow time to transition to new style. Make sure to prioritize declaring main directly on the instance_map.
3. Clean up old comments/commented out code.
4. Adjust unit tests to match with new code.
5. Adjust Complement setup infrastructure to only add main to the instance_map if workers are used and remove now unused options from the worker.yaml template.

* Initial Docs upload

* Changelog

* Missed some commented out code that can go now

* Remove TODO comment that no longer holds true.

* Fix links in docs

* More docs

* Remove debug logging

* Apply suggestions from code review

Co-authored-by: reivilibre <olivier@librepush.net>

* Apply suggestions from code review

Co-authored-by: reivilibre <olivier@librepush.net>

* Update version to latest, include completeish before/after examples in upgrade notes.

* Fix up and docs too

---------

Co-authored-by: reivilibre <olivier@librepush.net>
2023-05-11 11:30:56 +01:00
Dirk Klimpel
e4a25d022c
Load /capabilities endpoint on workers (#15436) 2023-04-14 12:26:07 -04:00
Dirk Klimpel
4af0aec54d
Load /directory/room/{roomAlias} endpoint on workers (#15333)
* Enable `directory`

* move to worker store

* newsfile

* disable `ClientDirectoryListServer` and `ClientAppserviceDirectoryListServer` for workers
2023-04-14 10:24:06 +01:00
Dirk Klimpel
4fc85e5a92
Load /password_policy endpoint on workers. (#15331) 2023-03-27 07:37:17 -04:00
Jason Little
3d70cc393f
Load /register/available endpoint on workers (#15268) 2023-03-17 09:50:31 -04:00
reivilibre
66fc166b96
Make configure_workers_and_start script used in Complement tests compatible with older versions of Python. (#15275) 2023-03-17 09:02:30 -04:00
Jason Little
003a25ae5c
Additional functionality for declaring worker types in Complement (#14921)
Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
2023-03-14 16:29:33 +00:00
Dirk Klimpel
ecbe0ddbe7
Add support for knocking to workers. (#15133) 2023-03-02 12:59:53 -05:00
David Robertson
c4f4dc35cd
Dockerfile-workers: spell out when config isn't generated (#15186)
* Complement: Spell out when config isn't generated

* Changelog
2023-03-02 15:55:26 +00:00
reivilibre
682d31c702
Allow use of the /filter Client-Server APIs on workers. (#15134) 2023-02-28 16:37:19 +00:00
Eric Eastwood
8f10c8b054
Move MSC3030 /timestamp_to_event endpoint to stable v1 location (#14471)
Fix https://github.com/matrix-org/synapse/issues/14390

 - Client API: `/_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` -> `/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>`
 - Federation API: `/_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` -> `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>`

Complement test changes: https://github.com/matrix-org/complement/pull/559
2022-11-28 15:54:18 -06:00
realtyem
c15e9a0edb
Remove need for worker_main_http_uri setting to use /keys/upload. (#14400) 2022-11-16 22:16:25 +00:00
realtyem
e9cbddc8e7
Modernize configure_workers_and_start.py bootstrapping script for Dockerfile-workers. (#14294) 2022-11-09 12:02:15 +00:00
realtyem
d85cba1aa0
Add all Stream Writer worker types to configure_workers_and_start.py (#14197)
Co-authored-by: reivilibre <oliverw@matrix.org>
2022-11-08 13:14:00 +00:00
realtyem
6c5082f3e0
Flush stdout/err in Dockerfile-workers before replacing the current process (#14195)
Also update `subprocess.check_output` to the slightly newer `subprocess.run`.

Signed-off-by: Jason Little <realtyem@gmail.com>
2022-10-18 11:56:20 +00:00
realtyem
c75836fe77
Strip whitespace from worker types in Dockerfile-workers (#14165) 2022-10-14 19:38:04 +00:00
realtyem
c7446906bd
Set LD_PRELOAD to load jemalloc in Dockerfile-workers. (#14182) 2022-10-14 13:29:49 +01:00
Patrick Cloke
c3e4edb4d6
Stabilize the threads API. (#14175)
Stabilize the threads API (MSC3856) by supporting (only) the v1
path for the endpoint.

This also marks the API as safe for workers since it is a read-only
API.
2022-10-14 07:16:50 -04:00
Patrick Cloke
c604d2c218
Mark /relations endpoint as usable on workers. (#14028)
Co-authored-by: Eric Eastwood <erice@element.io>
2022-10-12 06:46:13 -04:00
reivilibre
65e675504f
Add the ability to set the log level using the SYNAPSE_TEST_LOG_LEVEL environment when using complement.sh. (#13152) 2022-07-05 09:46:20 +00:00
reivilibre
9667bad55d
Improve startup times in Complement test runs against workers, particularly in CPU-constrained environments. (#13127)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-30 11:58:12 +00:00
reivilibre
3c5549e74a
Refactor the Dockerfile-workers configuration script to use Jinja2 templates in Synapse workers' Supervisord blocks. (#13054)
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2022-06-27 11:43:20 +01:00
reivilibre
212be2edc1
Use updated update_user_directory_from_worker options in Complement tests. (#13069) 2022-06-15 15:54:32 +01:00