Commit graph

1443 commits

Author SHA1 Message Date
Richard van der Hoff
9481707a52
Fixes to the federation rate limiter (#5621)
- Put the default window_size back to 1000ms (broken by #5181)
- Make the `rc_federation` config actually do something
- fix an off-by-one error in the 'concurrent' limit
- Avoid creating an unused `_PerHostRatelimiter` object for every single
  incoming request
2019-07-05 11:10:19 +01:00
Erik Johnston
c061d4f237 Fixup from review comments. 2019-07-04 11:41:06 +01:00
Amber Brown
463b072b12
Move logging utilities out of the side drawer of util/ and into logging/ (#5606) 2019-07-04 00:07:04 +10:00
Amber Brown
0ee9076ffe Fix media repo breaking (#5593) 2019-07-02 19:01:28 +01:00
Erik Johnston
8ee69f299c Add basic function to get all data for a user out of synapse 2019-07-02 12:09:04 +01:00
Amber Brown
f40a7dc41f
Make the http server handle coroutine-making REST servlets (#5475) 2019-06-29 17:06:55 +10:00
Amber Brown
be3b901ccd
Update the TLS cipher string and provide configurability for TLS on outgoing federation (#5550) 2019-06-28 18:19:09 +10:00
Brendan Abolivier
bfe84e051e Split public rooms directory auth config in two 2019-06-24 15:42:31 +01:00
Richard van der Hoff
c3c6b00d95
Pass config_dir_path and data_dir_path into Config.read_config. (#5522)
* Pull config_dir_path and data_dir_path calculation out of read_config_files

* Pass config_dir_path and data_dir_path into read_config
2019-06-24 11:34:45 +01:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston
554609288b Run as background process and fix comments 2019-06-19 11:33:03 +01:00
Erik Johnston
b42f90470f Add experimental option to reduce extremities.
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.

THIS IS REALLY EXPERIMENTAL.
2019-06-18 15:02:18 +01:00
Erik Johnston
dd927b29e1
Merge pull request #5388 from matrix-org/erikj/fix_email_push
Fix email notifications for unnamed rooms with multiple people
2019-06-17 13:54:35 +01:00
Brendan Abolivier
f12e1f029c
Merge pull request #5440 from matrix-org/babolivier/third_party_event_rules
Allow server admins to define implementations of extra rules for allowing or denying incoming events
2019-06-14 19:37:59 +01:00
Erik Johnston
9ca4ae7131
Merge pull request #5461 from matrix-org/erikj/histograms_are_cumalitive
Prometheus histograms are cumalative
2019-06-14 18:21:42 +01:00
Brendan Abolivier
f874b16b2e Add plugin APIs for implementations of custom event rules. 2019-06-14 18:16:03 +01:00
Brendan Abolivier
6d56a694f4 Don't send renewal emails to deactivated users 2019-06-14 15:05:56 +01:00
Erik Johnston
3ed595e327 Prometheus histograms are cumalative 2019-06-14 14:07:32 +01:00
Brendan Abolivier
d0530382ee
Track deactivated accounts in the database (#5378) 2019-06-14 13:18:24 +01:00
Amber H. Brown
a10c8dae85 fix prometheus rendering error 2019-06-14 21:09:33 +10:00
Amber Brown
6312d6cc7c
Expose statistics on extrems to prometheus (#5384) 2019-06-13 22:40:52 +10:00
Erik Johnston
7e68691ce9 Merge branch 'master' of github.com:matrix-org/synapse into develop 2019-06-11 17:25:16 +01:00
Erik Johnston
a766c41d25 Bump bleach version so that tests can run on old deps. 2019-06-11 12:34:18 +01:00
Erik Johnston
10383e6e6f Change password reset links to /_matrix. 2019-06-11 11:34:33 +01:00
Erik Johnston
abce00fc6a Merge branch 'release-v1.0.0' of github.com:matrix-org/synapse into develop 2019-06-10 18:33:55 +01:00
Erik Johnston
a6b1817940
Merge pull request #5417 from matrix-org/rav/shared_ssl_context
Share an SSL context object between SSL connections
2019-06-10 18:16:12 +01:00
Richard van der Hoff
db74c4fc6c fix ci on py2, again 2019-06-10 17:55:01 +01:00
Richard van der Hoff
19780a521e fix CI on python 2.7 2019-06-10 17:41:10 +01:00
Richard van der Hoff
e016681221 Tests for SSL certs for federation connections
Add some tests for bad certificates for federation and .well-known connections
2019-06-10 16:14:20 +01:00
Erik Johnston
0382b0ffee
Merge pull request #5415 from matrix-org/erikj/fix_null_valid_until_ms
Fix key verification when key stored with null valid_until_ms
2019-06-10 15:56:36 +01:00
Erik Johnston
43badd2cd4 Fix key verification when key stored with null valid_until_ms
Some keys are stored in the synapse database with a null valid_until_ms
which caused an exception to be thrown when using that key. We fix this
by treating nulls as zeroes, i.e. they keys will match verification
requests with a minimum_valid_until_ms of zero (i.e. don't validate ts)
but will not match requests with a non-zero minimum_valid_until_ms.

Fixes #5391.
2019-06-10 15:37:10 +01:00
Brendan Abolivier
26b62796c2
Merge pull request #5363 from matrix-org/babolivier/account_validity_send_mail_auth
Don't check whether the user's account is expired on /send_mail requests
2019-06-10 11:57:02 +01:00
Erik Johnston
2ebeda48b2 Add test 2019-06-07 12:15:33 +01:00
Erik Johnston
8182a1cfb5 Refactor email tests 2019-06-07 12:15:33 +01:00
Andrew Morgan
2d1d7b7e6f Prevent multiple device list updates from breaking a batch send (#5156)
fixes #5153
2019-06-06 23:54:00 +01:00
Neil Johnson
a11865016e
Set default room version to v4. (#5379)
Set default room version to v4.
2019-06-06 20:13:47 +01:00
Andrew Morgan
3719680ee4
Add ability to perform password reset via email without trusting the identity server (#5377)
Sends password reset emails from the homeserver instead of proxying to the identity server. This is now the default behaviour for security reasons. If you wish to continue proxying password reset requests to the identity server you must now enable the email.trust_identity_server_for_password_resets option.

This PR is a culmination of 3 smaller PRs which have each been separately reviewed:

* #5308
* #5345
* #5368
2019-06-06 17:34:07 +01:00
Richard van der Hoff
9fbb20a531
Stop hardcoding trust of old matrix.org key (#5374)
There are a few changes going on here:

* We make checking the signature on a key server response optional: if no
  verify_keys are specified, we trust to TLS to validate the connection.

* We change the default config so that it does not require responses to be
  signed by the old key.

* We replace the old 'perspectives' config with 'trusted_key_servers', which
  is also formatted slightly differently.

* We emit a warning to the logs every time we trust a key server response
  signed by the old key.
2019-06-06 17:33:11 +01:00
Richard van der Hoff
cb3b381fcb
Merge pull request #5359 from matrix-org/rav/enable_tls_verification
Validate federation server TLS certificates by default.
2019-06-06 10:50:42 +01:00
Richard van der Hoff
2eb47e5ee7
Merge pull request #5353 from matrix-org/rav/verify_key_logging
Associate a request_name with each verify request, for logging
2019-06-06 09:33:09 +01:00
Brendan Abolivier
fe13bd52ac
Don't check whether the user's account is expired on /send_mail requests 2019-06-05 16:35:05 +01:00
Richard van der Hoff
7603a706eb Merge branch 'rav/fix_custom_ca' into rav/enable_tls_verification 2019-06-05 16:32:35 +01:00
Erik Johnston
75538813fc Fix background updates to handle redactions/rejections (#5352)
* Fix background updates to handle redactions/rejections

In background updates based on current state delta stream we need to
handle that we may not have all the events (or at least that
`get_events` may raise an exception).
2019-06-06 00:45:46 +10:00
Richard van der Hoff
e2dfb922e1 Validate federation server TLS certificates by default. 2019-06-05 14:17:50 +01:00
Richard van der Hoff
cea9750d11 Associate a request_name with each verify request, for logging
Also:
* rename VerifyKeyRequest->VerifyJsonRequest
* calculate key_ids on VerifyJsonRequest construction
* refactor things to pass around VerifyJsonRequests instead of 4-tuples
2019-06-05 10:46:26 +01:00
Richard van der Hoff
def5ea4062 Don't bomb out on direct key fetches as soon as one fails 2019-06-04 00:16:56 +01:00
Richard van der Hoff
fec2dcb1a5
Enforce validity period on server_keys for fed requests. (#5321)
When handling incoming federation requests, make sure that we have an
up-to-date copy of the signing key.

We do not yet enforce the validity period for event signatures.
2019-06-03 22:59:51 +01:00
Amber Brown
2889b05554
Unify v1 and v2 REST client APIs (#5226) 2019-06-03 21:28:59 +10:00
Richard van der Hoff
d828d1dc57
Merge pull request #5309 from matrix-org/rav/limit_displayname_length
Limit displaynames and avatar URLs
2019-06-01 11:34:50 +01:00
Richard van der Hoff
93003aa172 add some tests 2019-06-01 11:14:37 +01:00
Erik Johnston
58cce39f3a
Merge pull request #5276 from matrix-org/babolivier/account_validity_job_delta
Allow configuring a range for the account validity startup job
2019-05-31 12:11:56 +01:00
Brendan Abolivier
4d794dae21 Move delta from +10% to -10% 2019-05-31 11:09:39 +01:00
Brendan Abolivier
847b9dcd1c Make max_delta equal to period * 10% 2019-05-31 09:54:46 +01:00
Erik Johnston
e9e5d3392d
Merge pull request #5278 from matrix-org/erikj/cleanup_bad_extremities
Add DB bg update to cleanup extremities.
2019-05-30 14:28:26 +01:00
Amber Brown
46c8f7a517
Implement the SHHS complexity API (#5216) 2019-05-30 01:47:16 +10:00
Erik Johnston
d7add713a8 Add test 2019-05-29 15:59:06 +01:00
Richard van der Hoff
540f40f0cd
Merge pull request #5251 from matrix-org/rav/server_keys/01-check_sig
Ensure that server_keys fetched via a notary server are correctly signed.
2019-05-28 21:32:17 +01:00
Brendan Abolivier
7e1c7cc274
Typo 2019-05-28 17:13:26 +01:00
Brendan Abolivier
52839886d6
Allow configuring a range for the account validity startup job
When enabling the account validity feature, Synapse will look at startup for registered account without an expiration date, and will set one equals to 'now + validity_period' for them. On large servers, it can mean that a large number of users will have the same expiration date, which means that they will all be sent a renewal email at the same time, which isn't ideal.
In order to mitigate this, this PR allows server admins to define a 'max_delta' so that the expiration date is a random value in the [now + validity_period ; now + validity_period + max_delta] range. This allows renewal emails to be progressively sent over a configured period instead of being sent all in one big batch.
2019-05-28 16:52:45 +01:00
Richard van der Hoff
fa1b293da2
Simplification to Keyring.wait_for_previous_lookups. (#5250)
The list of server names was redundant, since it was equivalent to the keys on
the server_to_deferred map. This reduces the number of large lists being passed
around, and has the benefit of deduplicating the entries in `wait_on`.
2019-05-24 22:17:18 +01:00
Erik Johnston
8c41c04ee4
Merge pull request #5244 from matrix-org/rav/server_keys/00-factor-out-fetchers
Factor out KeyFetchers from KeyRing
2019-05-23 16:28:51 +01:00
Richard van der Hoff
753b1270da Require sig from origin server on perspectives responses 2019-05-23 15:01:09 +01:00
Andrew Morgan
6368150a74
Add config option for setting homeserver's default room version (#5223)
Replaces DEFAULT_ROOM_VERSION constant with a method that first checks the config, then returns a hardcoded value if the option is not present.

That hardcoded value is now located in the server.py config file.
2019-05-23 15:00:20 +01:00
Richard van der Hoff
895b79ac2e Factor out KeyFetchers from KeyRing
Rather than have three methods which have to have the same interface,
factor out a separate interface which is provided by three implementations.

I find it easier to grok the code this way.
2019-05-23 13:46:47 +01:00
Richard van der Hoff
b75537beaf Store key validity time in the storage layer
This is a first step to checking that the key is valid at the required moment.

The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
2019-05-23 11:52:22 +01:00
Richard van der Hoff
2e052110ee
Rewrite store_server_verify_key to store several keys at once (#5234)
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
2019-05-23 11:45:39 +01:00
Amber Brown
4a30e4acb4
Room Statistics (#4338) 2019-05-21 11:36:50 -05:00
Erik Johnston
9526aa96a6
Merge pull request #5212 from matrix-org/erikj/deny_multiple_reactions
Block attempts to annotate the same event twice
2019-05-21 17:08:14 +01:00
Brendan Abolivier
6a5a70edf0
Merge pull request #5204 from matrix-org/babolivier/account_validity_expiration_date
Add startup background job for account validity
2019-05-21 14:55:15 +01:00
Erik Johnston
c7ec06e8a6 Block attempts to annotate the same event twice 2019-05-20 17:39:05 +01:00
Erik Johnston
1dff859d6a Rename relation types to match MSC 2019-05-20 14:31:19 +01:00
Erik Johnston
57ba3451b6
Merge pull request #5209 from matrix-org/erikj/reactions_base
Land basic reaction and edit support.
2019-05-20 14:06:40 +01:00
Erik Johnston
2ac9c965dd Fixup comments 2019-05-20 12:32:26 +01:00
Erik Johnston
935af0da38 Correctly update aggregation counts after redaction 2019-05-20 12:09:27 +01:00
ReidAnderson
3787133c9e Limit UserIds to a length that fits in a state key (#5198) 2019-05-20 11:20:08 +01:00
Brendan Abolivier
ad5b4074e1
Add startup background job for account validity
If account validity is enabled in the server's configuration, this job will run at startup as a background job and will stick an expiration date to any registered account missing one.
2019-05-17 19:37:31 +01:00
Erik Johnston
d4ca533d70 Make tests use different user for each reaction it sends
As users aren't allowed to react with the same emoji more than once.
2019-05-17 18:03:05 +01:00
Brendan Abolivier
fdeac1e984
Merge pull request #5196 from matrix-org/babolivier/per_room_profiles
Add an option to disable per-room profiles
2019-05-17 12:10:49 +01:00
David Baker
07cff7b121
Merge pull request #5174 from matrix-org/dbkr/add_dummy_flow_to_recaptcha_only
Re-order registration stages to do msisdn & email auth last
2019-05-16 17:27:39 +01:00
Erik Johnston
d46aab3fa8 Add basic editing support 2019-05-16 16:54:45 +01:00
Brendan Abolivier
8f9ce1a8a2
Lint 2019-05-16 15:25:54 +01:00
Brendan Abolivier
cc8c139a39
Lint 2019-05-16 15:20:59 +01:00
Brendan Abolivier
efdc55db75
Forgot copyright 2019-05-16 15:10:24 +01:00
Brendan Abolivier
54a582ed44
Add test case 2019-05-16 15:09:16 +01:00
Erik Johnston
95f3fcda3c Check that event is visible in new APIs 2019-05-16 14:19:06 +01:00
Erik Johnston
a0603523d2 Add aggregations API 2019-05-16 09:37:20 +01:00
Amber Brown
f1e5b41388
Make all the rate limiting options more consistent (#5181) 2019-05-15 12:06:04 -05:00
Erik Johnston
b50641e357 Add simple pagination API 2019-05-15 13:36:51 +01:00
Erik Johnston
efe3c7977a Add simple send_relation API and track in DB 2019-05-15 13:36:51 +01:00
Erik Johnston
d94544051b
Merge pull request #5184 from matrix-org/erikj/expose_get_events_as_array
Expose DataStore._get_events as get_events_as_list
2019-05-15 10:17:38 +01:00
Erik Johnston
495e859e58 Merge branch 'erikj/fix_config_ratelimiting' into erikj/test 2019-05-14 14:42:47 +01:00
Erik Johnston
dc4f6d1b01 Use correct config option for ratelimiting in tests 2019-05-14 14:37:40 +01:00
Erik Johnston
4fb44fb5b9 Expose DataStore._get_events as get_events_as_list
This is in preparation for reaction work which requires it.
2019-05-14 13:37:44 +01:00
Amber Brown
df2ebd75d3
Migrate all tests to use the dict-based config format instead of hanging items off HomeserverConfig (#5171) 2019-05-13 15:01:14 -05:00
Andrew Morgan
5a4b328f52 Add ability to blacklist ip ranges for federation traffic (#5043) 2019-05-13 19:05:06 +01:00
David Baker
822072b1bb Terms might not be the last stage 2019-05-13 16:10:26 +01:00
David Baker
9e99143c47 Merge remote-tracking branch 'origin/develop' into dbkr/add_dummy_flow_to_recaptcha_only 2019-05-13 15:37:03 +01:00
David Baker
8782bfb783 And now I realise why the test is failing... 2019-05-13 15:34:11 +01:00
Andrew Morgan
2f48c4e1ae
URL preview blacklisting fixes (#5155)
Prevents a SynapseError being raised inside of a IResolutionReceiver and instead opts to just return 0 results. This thus means that we have to lump a failed lookup and a blacklisted lookup together with the same error message, but the substitute should be generic enough to cover both cases.
2019-05-10 10:32:44 -07:00
Amber Brown
b36c82576e
Run Black on the tests again (#5170) 2019-05-10 00:12:11 -05:00