mirrors/synapse
1
0
Fork 0
mirror of https://github.com/element-hq/synapse.git synced 2024-11-23 01:55:53 +03:00
Code Issues Projects Releases Packages Wiki Activity
18452 commits 935 branches 728 tags 488 MiB
Commit graph

427 commits

Author SHA1 Message Date
Erik Johnston
a319cb1dd1
Change device list streams to have one row per ID (#7010) 
* Add 'device_lists_outbound_pokes' as extra table.

This makes sure we check all the relevant tables to get the current max
stream ID.

Currently not doing so isn't problematic as the max stream ID in
`device_lists_outbound_pokes` is the same as in `device_lists_stream`,
however that will change.

* Change device lists stream to have one row per id.

This will make it possible to process the streams more incrementally,
avoiding having to process large chunks at once.

* Change device list replication to match new semantics.

Instead of sending down batches of user ID/host tuples, send down a row
per entity (user ID or host).

* Newsfile

* Remove handling of multiple rows per ID

* Fix worker handling

* Comments from review
 2020-03-19 11:36:53 +00:00
Erik Johnston
4a17a647a9
Improve get auth chain difference algorithm. (#7095) 
It was originally implemented by pulling the full auth chain of all
state sets out of the database and doing set comparison. However, that
can take a lot work if the state and auth chains are large.

Instead, lets try and fetch the auth chains at the same time and
calculate the difference on the fly, allowing us to bail early if all
the auth chains converge. Assuming that the auth chains do converge more
often than not, this should improve performance. Hopefully.
 2020-03-18 16:46:41 +00:00
Neil Johnson
1d66dce83e
Break down monthly active users by appservice_id (#7030) 
* Break down monthly active users by appservice_id and emit via prometheus.

Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
 2020-03-06 18:14:19 +00:00
Erik Johnston
f70f44abc7 Remove handling of multiple rows per ID 2020-02-28 11:45:35 +00:00
Patrick Cloke
509e381afa
Clarify list/set/dict/tuple comprehensions and enforce via flake8 (#6957) 
Ensure good comprehension hygiene using flake8-comprehensions.
 2020-02-21 07:15:07 -05:00
Richard van der Hoff
43b2be9764 Replace _event_dict_property with DictProperty 
this amounts to the same thing, but replaces `_event_dict` with `_dict`, and
removes some of the function layers generated by `property`.
 2020-02-14 16:42:37 +00:00
Erik Johnston
b9391c9575
Add typing to SyncHandler (#6821) 
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
 2020-02-03 18:05:44 +00:00
Richard van der Hoff
5a246611e3
Type defintions for use in refactoring for redaction changes (#6803) 
* Bump signedjson to 1.1

... so that we can use the type definitions

* Fix breakage caused by upgrade to signedjson 1.1

Thanks, @illicitonion...
 2020-01-30 11:25:59 +00:00
Erik Johnston
8df862e45d
Add rooms.room_version column (#6729) 
This is so that we don't have to rely on pulling it out from `current_state_events` table.
 2020-01-27 14:30:57 +00:00
Erik Johnston
fa4d609e20
Make 'event.redacts' never raise. (#6771) 
There are quite a few places that we assume that a redaction event has a
corresponding `redacts` key, which is not always the case. So lets
cheekily make it so that event.redacts just returns None instead.
 2020-01-23 15:19:03 +00:00
Erik Johnston
28c98e51ff
Add local_current_membership table (#6655) 
Currently we rely on `current_state_events` to figure out what rooms a
user was in and their last membership event in there. However, if the
server leaves the room then the table may be cleaned up and that
information is lost. So lets add a table that separately holds that
information.
 2020-01-15 14:59:33 +00:00
Manuel Stahl
d2906fe666 Allow admin users to create or modify users without a shared secret (#6495) 
Signed-off-by: Manuel Stahl <manuel.stahl@awesome-technologies.de>
 2020-01-09 13:31:00 +00:00
Richard van der Hoff
d20c346544
port BackgroundUpdateTestCase to HomeserverTestCase (#6653) 2020-01-07 14:09:07 +00:00
Richard van der Hoff
dc41fbf0dd Remove unused get_prev_events_and_hashes_for_room 2020-01-06 13:45:33 +00:00
Richard van der Hoff
5a04781643 rename get_prev_events_for_room to get_prev_events_and_hashes_for_room 
... to make way for a new method which just returns the event ids
 2020-01-06 13:45:33 +00:00
Erik Johnston
75d8f26ac8
Split state groups into a separate data store (#6296) 2019-12-20 10:48:24 +00:00
Erik Johnston
2284eb3a53
Add database config class (#6513) 
This encapsulates config for a given database and is the way to get new
connections.
 2019-12-18 10:45:12 +00:00
Richard van der Hoff
40eda84933
Fix race which caused deleted devices to reappear (#6514) 
Stop the `update_client_ips` background job from recreating deleted devices.
 2019-12-10 16:22:29 +00:00
Erik Johnston
852f80d8a6 Fixup tests 2019-12-06 16:02:50 +00:00
Erik Johnston
9a4fb457cf Change DataStores to accept 'database' param. 2019-12-06 13:30:06 +00:00
Erik Johnston
4a33a6dd19 Move background update handling out of store 2019-12-05 11:11:26 +00:00
Erik Johnston
756d4942f5 Move DB pool and helper functions into dedicated Database class 2019-12-05 10:46:37 +00:00
Erik Johnston
ee86abb2d6 Remove underscore from SQLBaseStore functions 2019-12-04 16:23:43 +00:00
Amber Brown
0f87b912ab
Implementation of MSC2314 (#6176) 2019-11-28 08:54:07 +11:00
Hubert Chathi
0d27aba900
add etag and count to key backup endpoints (#5858) 2019-11-27 16:14:44 -05:00
Erik Johnston
6a0092d371 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-11-04 13:29:35 +00:00
Hubert Chathi
6f4bc6d01d Merge branch 'develop' into cross-signing_federation 2019-10-31 22:38:21 -04:00
Erik Johnston
cd581338cf Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_purge_history 2019-10-31 15:19:26 +00:00
Hubert Chathi
bb6cec27a5 rename get_devices_by_remote to get_device_updates_by_remote 2019-10-30 14:57:34 -04:00
Erik Johnston
7c8c97e635 Split purge API into events vs state 2019-10-30 15:23:37 +00:00
Erik Johnston
69f0054ce6 Port to use state storage 2019-10-30 14:46:54 +00:00
Hubert Chathi
7d7eac61be Merge branch 'develop' into cross-signing_federation 2019-10-30 10:17:10 -04:00
Erik Johnston
ec6de1cc7d Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_out_persistence_store 2019-10-30 13:37:04 +00:00
Erik Johnston
326b3dace7 Make ObservableDeferred.observe() always return deferred. 
This makes it easier to use in an async/await world.

Also fixes a bug where cache descriptors would occaisonally return a raw
value rather than a deferred.
 2019-10-30 11:35:46 +00:00
Hubert Chathi
4697c0de0b remove unneeded imports 2019-10-25 10:47:02 -04:00
Hubert Chathi
4cf3a30a20 switch to using HomeserverTestCase 2019-10-25 10:42:07 -04:00
Hubert Chathi
c40d7244f8 Merge branch 'develop' into cross-signing_federation 2019-10-24 22:31:25 -04:00
Hubert Chathi
29a0bc5637 remove some unnecessary lines 2019-10-24 21:43:02 -04:00
Hubert Chathi
848cd388d9 delete keys when deleting backups 2019-10-24 21:21:51 -04:00
Erik Johnston
3ca4c7c516 Use new EventPersistenceStore 2019-10-23 16:15:03 +01:00
Hubert Chathi
3e3f9b684e fix unit test 2019-10-22 22:26:30 -04:00
Erik Johnston
c66a06ac6b Move storage classes into a main "data store". 
This is in preparation for having multiple data stores that offer
different functionality, e.g. splitting out state or event storage.
 2019-10-21 16:05:06 +01:00
Hubert Chathi
06fc66c81e
Merge pull request #6193 from matrix-org/uhoreg/interpret_device_key_in_storage 
make storage layer in charge of interpreting the device key data
 2019-10-11 15:19:06 -04:00
Neil Johnson
a0d0ba7862
Fix MAU reaping where reserved users are specified. (#6168) 2019-10-11 09:38:26 +01:00
Hubert Chathi
4908fb3b30 make storage layer in charge of interpreting the device key data 2019-10-10 19:15:30 -04:00
Erik Johnston
bc244627ac Fix postgres unit tests 2019-10-10 15:37:53 +01:00
Erik Johnston
b4fe7e19c0
Merge pull request #6146 from matrix-org/erikj/fix_destination_retry_timings 
Fix errors storing large retry intervals.
 2019-10-02 11:05:22 +01:00
Erik Johnston
f44f1d2e83 Fix errors storing large retry intervals. 
We have set the max retry interval to a value larger than a postgres or
sqlite int can hold, which caused exceptions when updating the
destinations table.

To fix postgres we need to change the column to a bigint, and for sqlite
we lower the max interval to 2**62 (which is still incredibly long).
 2019-10-02 10:36:27 +01:00
Erik Johnston
ce7a3e7e27 Fix fetching censored redactions from DB 
Fetching a censored redactions caused an exception due to the code
expecting redactions to have a `redact` key, which redacted redactions
don't have.
 2019-10-02 10:14:01 +01:00
Neil Johnson
034db2ba21 Fix dummy event insertion consent bug (#6053) 
Fixes #5905
 2019-09-26 11:47:53 +01:00
Erik Johnston
faac453f08 Test that pruning of old user IPs works 2019-09-24 15:53:17 +01:00
Erik Johnston
acb62a7cc6 Test background update 2019-09-23 16:59:45 +01:00
Erik Johnston
51d28272e2 Query devices table for last seen info. 
This is a) simpler than querying user_ips directly and b) means we can
purge older entries from user_ips without losing the required info.

The storage functions now no longer return the access_token, since it
was unused.
 2019-09-23 16:59:45 +01:00
Richard van der Hoff
1e19ce00bf
Add 'failure_ts' column to 'destinations' table (#6016) 
Track the time that a server started failing at, for general analysis purposes.
 2019-09-17 11:41:54 +01:00
Erik Johnston
470dc621ae
Merge pull request #5934 from matrix-org/erikj/censor_redactions 
Censor redactions in DB after a month
 2019-09-09 15:29:39 +01:00
Amber Brown
55d5b3af88
Servers-known-about statistic (#5981) 2019-09-07 01:45:51 +10:00
Erik Johnston
ad9b64b496 Fix test 2019-09-05 17:17:47 +01:00
Erik Johnston
3ff0422d2d Make redaction retention period configurable 2019-09-05 17:16:45 +01:00
Erik Johnston
a4bf72c30c Censor redactions in DB after a month 2019-08-30 11:29:17 +01:00
Half-Shot
0fb5189072 Fix registration test 2019-08-23 09:25:35 +01:00
Richard van der Hoff
fb86217553
Merge pull request #5788 from matrix-org/rav/metaredactions 
Fix handling of redactions of redactions
 2019-08-12 12:25:19 +01:00
Erik Johnston
63eb4a1b62
Merge pull request #5746 from matrix-org/erikj/test_bg_update_currnet_state 
Add unit test for current state membership bg update
 2019-07-30 10:00:02 +01:00
Richard van der Hoff
4e97eb89e5 Handle loops in redaction events 2019-07-30 00:00:34 +01:00
Richard van der Hoff
1cad8d7b6f
Convert RedactionTestCase to modern test style (#5768) 2019-07-26 07:38:55 +01:00
Erik Johnston
73bbaf2bc6 Add unit test for current state membership bg update 2019-07-23 17:00:22 +01:00
Amber Brown
4806651744
Replace returnValue with return (#5736) 2019-07-23 23:00:55 +10:00
Amber Brown
7ad1d76356
Support Prometheus_client 0.4.0+ (#5636) 2019-07-18 23:57:15 +10:00
Richard van der Hoff
5f158ec039
Implement access token expiry (#5660) 
Record how long an access token is valid for, and raise a soft-logout once it
expires.
 2019-07-12 17:26:02 +01:00
Richard van der Hoff
953dbb7980
Remove access-token support from RegistrationStore.register (#5642) 
The 'token' param is no longer used anywhere except the tests, so let's kill
that off too.
 2019-07-10 16:26:49 +01:00
Amber Brown
32e7c9e7f2
Run Black. (#5482) 2019-06-20 19:32:02 +10:00
Erik Johnston
554609288b Run as background process and fix comments 2019-06-19 11:33:03 +01:00
Erik Johnston
b42f90470f Add experimental option to reduce extremities. 
Adds new config option `cleanup_extremities_with_dummy_events` which
periodically sends dummy events to rooms with more than 10 extremities.

THIS IS REALLY EXPERIMENTAL.
 2019-06-18 15:02:18 +01:00
Erik Johnston
3ed595e327 Prometheus histograms are cumalative 2019-06-14 14:07:32 +01:00
Amber H. Brown
a10c8dae85 fix prometheus rendering error 2019-06-14 21:09:33 +10:00
Amber Brown
6312d6cc7c
Expose statistics on extrems to prometheus (#5384) 2019-06-13 22:40:52 +10:00
Andrew Morgan
2d1d7b7e6f Prevent multiple device list updates from breaking a batch send (#5156) 
fixes #5153
 2019-06-06 23:54:00 +01:00
Neil Johnson
a11865016e
Set default room version to v4. (#5379) 
Set default room version to v4.
 2019-06-06 20:13:47 +01:00
Erik Johnston
d7add713a8 Add test 2019-05-29 15:59:06 +01:00
Richard van der Hoff
b75537beaf Store key validity time in the storage layer 
This is a first step to checking that the key is valid at the required moment.

The idea here is that, rather than passing VerifyKey objects in and out of the
storage layer, we instead pass FetchKeyResult objects, which simply wrap the
VerifyKey and add a valid_until_ts field.
 2019-05-23 11:52:22 +01:00
Richard van der Hoff
2e052110ee
Rewrite store_server_verify_key to store several keys at once (#5234) 
Storing server keys hammered the database a bit. This replaces the
implementation which stored a single key, with one which can do many updates at
once.
 2019-05-23 11:45:39 +01:00
Erik Johnston
4fb44fb5b9 Expose DataStore._get_events as get_events_as_list 
This is in preparation for reaction work which requires it.
 2019-05-14 13:37:44 +01:00
Amber Brown
b36c82576e
Run Black on the tests again (#5170) 2019-05-10 00:12:11 -05:00
Richard van der Hoff
12f9d51e82
Add admin api for sending server_notices (#5121) 2019-05-02 11:59:16 +01:00
Richard van der Hoff
40e576e29c Move admin api impl to its own package 
It doesn't really belong under rest/client/v1 any more.
 2019-05-01 15:44:30 +01:00
Richard van der Hoff
18b69be00f Rewrite Datastore.get_server_verify_keys 
Rewrite this so that it doesn't hammer the database.
 2019-04-09 00:00:10 +01:00
Richard van der Hoff
0084309cd2 Rewrite test_keys as a HomeserverTestCase 2019-04-08 23:59:57 +01:00
Neil Johnson
e8419554ff
Remove presence lists (#4989) 
Remove presence list support as per MSC 1819
 2019-04-03 11:11:15 +01:00
Richard van der Hoff
54a87a7b08
Collect room-version variations into one place (#4969) 
Collect all the things that make room-versions different to one another into
one place, so that it's easier to define new room versions.
 2019-04-01 10:24:38 +01:00
Amber Brown
282c97327f
Migrate the user directory initial population to a background task (#4864) 2019-03-19 04:50:24 +11:00
Amber Brown
d306bd1b26 fixup 2019-03-12 22:38:01 +11:00
Amber Brown
f6135d06cf
Rewrite userdir to be faster (#4537) 2019-03-07 01:22:53 -08:00
Amber Brown
a06614bd2a
UPSERT many functionality (#4644) 2019-02-20 23:03:30 +11:00
Amber Brown
3f189c902e
Fix flake8 (#4519) 2019-01-30 10:53:17 +00:00
Amber Brown
7072fe3084
Fix UPSERTs on SQLite 3.24+ (#4477) 2019-01-28 15:43:32 +00:00
Erik Johnston
9770ed91c2 Fix tests 2019-01-25 10:32:26 +00:00
Erik Johnston
be6a7e47fa
Revert "Require event format version to parse or create events" 2019-01-25 10:23:51 +00:00
Erik Johnston
efb8ed1d45 Merge branch 'develop' of github.com:matrix-org/synapse into erikj/require_format_version 2019-01-24 18:52:34 +00:00
Amber Brown
58f6c48183
Use native UPSERTs where possible (#4306) 2019-01-24 21:31:54 +11:00
Erik Johnston
e79ba9eb34 Fix tests 2019-01-24 09:28:16 +00:00
Amber Brown
7960c26fda
Fix adding new rows instead of updating them if one of the key values is a NULL in upserts. (#4369) 2019-01-09 22:26:25 +11:00
Neil Johnson
d2f7c4e6b1
create support user (#4141) 
Allow for the creation of a support user.

A support user can access the server, join rooms, interact with other users, but does not appear in the user directory nor does it contribute to monthly active user limits.
 2018-12-14 18:20:59 +00:00
Richard van der Hoff
30da50a5b8
Initialise user displayname from SAML2 data (#4272) 
When we register a new user from SAML2 data, initialise their displayname
correctly.
 2018-12-07 14:44:46 +01:00
Travis Ralston
835779f7fb Add option to track MAU stats (but not limit people) (#3830) 2018-11-15 18:08:27 +00:00
Erik Johnston
abaa93c158 Add test to assert set_e2e_device_keys correctly returns False on no-op 2018-11-08 14:06:44 +00:00
Erik Johnston
cb53ce9d64
Refactor state group lookup to reduce DB hits (#4011) 
Currently when fetching state groups from the data store we make two
hits two the database: once for members and once for non-members (unless
request is filtered to one or the other). This adds needless load to the
datbase, so this PR refactors the lookup to make only a single database
hit.
 2018-10-25 17:49:55 +01:00
Neil Johnson
ea69a84bbb fix style inconsistencies 2018-10-24 17:18:08 +01:00
Neil Johnson
6105c6101f fix race condiftion in calling initialise_reserved_users 2018-10-23 15:24:58 +01:00
Erik Johnston
9c834b8ee9 Add tests 2018-10-02 16:22:39 +01:00
Amber Brown
6e05fd032c
Fix userconsent on Python 3 (#3938) 2018-10-02 00:11:58 +10:00
Richard van der Hoff
ae6ad4cf41
docstrings and unittests for storage.state (#3958) 
I spent ages trying to figure out how I was going mad...
 2018-09-27 11:22:25 +01:00
Amber Brown
1f3f5fcf52
Fix client IPs being broken on Python 3 (#3908) 2018-09-20 20:14:34 +10:00
Neil Johnson
8decd6233d improve naming 2018-09-12 16:22:15 +01:00
Neil Johnson
0ddf486724 expose number of real reserved users 2018-09-12 11:58:52 +01:00
Amber Brown
52ec6e9dfa
Port tests/ to Python 3 (#3808) 2018-09-07 02:58:18 +10:00
Neil Johnson
84a750e0c3 ensure guests never enter mau list 2018-09-06 17:22:53 +01:00
Neil Johnson
61b05727fa guest users should not be part of mau total 2018-09-05 22:30:36 +01:00
Amber Brown
77055dba92
Fix tests on postgresql (#3740) 2018-09-04 02:21:48 +10:00
Amber Brown
14e4d4f4bf
Port storage/ to Python 3 (#3725) 2018-08-31 00:19:58 +10:00
Richard van der Hoff
414fa36f3e Fix up tests 2018-08-28 17:21:05 +01:00
Erik Johnston
cd77270a66 Implement trail users 2018-08-23 19:17:19 +01:00
Erik Johnston
3bf8bab8f9
Merge pull request #3673 from matrix-org/erikj/refactor_state_handler 
Refactor state module to support multiple room versions
 2018-08-22 10:04:55 +01:00
Matthew Hodgson
bb81e78ec6
Split the state_group_cache in two (#3726) 
Splits the state_group_cache in two.

One half contains normal state events; the other contains member events.

The idea is that the lazyloading common case of: "I want a subset of member events plus all of the other state" can be accomplished efficiently by splitting the cache into two, and asking for "all events" from the non-members cache, and "just these keys" from the members cache.  This means we can avoid having to make DictionaryCache aware of these sort of complicated queries, whilst letting LL requests benefit from the caching.

Previously we were unable to sensibly use the caching and had to pull all state from the DB irrespective of the filtering, which made things slow.  Hopefully fixes https://github.com/matrix-org/synapse/issues/3720.
 2018-08-22 00:56:37 +02:00
Erik Johnston
4d664278af Merge branch 'develop' of github.com:matrix-org/synapse into erikj/refactor_state_handler 2018-08-20 14:49:43 +01:00
Neil Johnson
bcfeb44afe call reap on start up and fix under reaping bug 2018-08-16 22:55:32 +01:00
Matthew Hodgson
2f78f432c4 speed up /members and add at= and membership params (#3568) 2018-08-15 16:35:22 +01:00
Neil Johnson
1522ed9c07 in case max_mau is less than I think 2018-08-14 16:52:30 +01:00
Neil Johnson
414d54b61a
Merge pull request #3670 from matrix-org/neilj/mau_sync_block 
Block ability to read via sync if mau limit exceeded
 2018-08-14 15:21:31 +00:00
Amber Brown
99dd975dae
Run tests under PostgreSQL (#3423) 2018-08-13 16:47:46 +10:00
black
8b3d9b6b19 Run black. 2018-08-10 23:54:09 +10:00
Amber Brown
b37c472419
Rename async to async_helpers because async is a keyword on Python 3.7 (#3678) 2018-08-10 23:50:21 +10:00
Neil Johnson
885ea9c602 rename _user_last_seen_monthly_active 2018-08-09 18:02:12 +01:00
Erik Johnston
3e19beb941 Fix tests 2018-08-09 14:58:49 +01:00
Amber Brown
2511f3f8a0
Test fixes for Python 3 (#3647) 2018-08-09 12:22:01 +10:00
Neil Johnson
e8eba2b4e3 implement reserved users for mau limits 2018-08-07 17:49:43 +01:00
Neil Johnson
e10830e976 wip commit - tests failing 2018-08-03 17:55:50 +01:00
Neil Johnson
5593ff6773 fix (lots of) py3 test failures 2018-08-03 14:59:17 +01:00
Neil Johnson
b2aab04d2c fix py3 test failure 2018-08-03 14:12:56 +01:00
Neil Johnson
950807d93a fix caching and tests 2018-08-03 13:49:53 +01:00
Neil Johnson
9180061b49 remove unused count_monthly_users 2018-08-02 15:55:29 +01:00
Neil Johnson
c4ffbecb68 fix test, update constructor call 2018-08-02 13:51:05 +01:00
Neil Johnson
00f99f74b1 insertion into monthly_active_users 2018-08-02 13:47:19 +01:00
Neil Johnson
165e067033 Revert "change monthly_active_users table to be a single column" 
This reverts commit ec716a35b2.
 2018-08-02 10:59:58 +01:00
Neil Johnson
ec716a35b2 change monthly_active_users table to be a single column 2018-08-01 17:54:37 +01:00
Neil Johnson
d766f26de9 Merge branch 'develop' of github.com:matrix-org/synapse into neilj/mau_tracker 2018-08-01 17:49:41 +01:00
Neil Johnson
c480c4c962 fix isort 2018-08-01 14:25:58 +01:00
Neil Johnson
4e6e00152c fix known broken test 2018-08-01 11:48:37 +01:00
Neil Johnson
6ef983ce5c api into monthly_active_users table 2018-07-31 16:36:24 +01:00
Neil Johnson
df2235e7fa coding style 2018-07-31 13:16:20 +01:00
Neil Johnson
7d05406a07 fix user_ips counting 2018-07-31 12:03:23 +01:00
Neil Johnson
cefac79c10 monthly_active_tests 2018-07-30 22:08:09 +01:00