Commit graph

14855 commits

Author SHA1 Message Date
Olivier Wilkinson (reivilibre)
703f9ff3c9 Merge branch 'develop' into rei/room_stats_separated 2019-08-14 08:56:24 +01:00
Olivier Wilkinson (reivilibre)
95a30250b2 Newsfile
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-14 08:21:21 +01:00
Olivier Wilkinson (reivilibre)
f9f551fb93 Clarify docstrings in storage.stats
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-14 08:19:19 +01:00
Brendan Abolivier
f70d0a1dd9 1.3.0rc1 2019-08-13 18:20:09 +01:00
Brendan Abolivier
3039be82ce
Merge pull request #5848 from matrix-org/hawkowl/fix-mediarepo-worker-startup
Fix mediarepo worker startup
2019-08-13 17:38:11 +01:00
Amber H. Brown
28bce1ac7c changelog 2019-08-14 02:08:24 +10:00
Amber H. Brown
18bdac8ee4 fix config being a dict, actually 2019-08-14 02:06:42 +10:00
Olivier Wilkinson (reivilibre)
de6b266817 Linting
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-13 16:19:36 +01:00
Olivier Wilkinson (reivilibre)
0e6f700a13 Remove clean-up handler and replace with no-op as not currently needed.
needed.

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-13 15:49:24 +01:00
Olivier Wilkinson (reivilibre)
2da4b41c63 Remove obsolete function
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-13 15:48:50 +01:00
Olivier Wilkinson (reivilibre)
96fa239b8a Fix up stats_separated1.sql
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-13 15:32:14 +01:00
Olivier Wilkinson (reivilibre)
16e2ffd166 Initial room and user statistics documentation
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-13 15:08:48 +01:00
Olivier Wilkinson (reivilibre)
20ec9698ac Fix issue with not selecting a needed column
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-13 15:08:32 +01:00
Olivier Wilkinson (reivilibre)
5b54411213 Add SQLite support by working around missing syntax 2019-08-13 15:07:57 +01:00
Olivier Wilkinson (reivilibre)
fd184f6cd0 Fix generality of query 2019-08-13 15:07:00 +01:00
Erik Johnston
af187805b3
Merge pull request #5809 from matrix-org/erikj/handle_pusher_stop
Handle pusher being deleted during processing.
2019-08-13 14:08:29 +01:00
Erik Johnston
96bdd661b8
Remove redundant return 2019-08-13 12:50:36 +01:00
Amber Brown
0b6fbb28a8
Don't load the media repo when configured to use an external media repo (#5754) 2019-08-13 21:49:28 +10:00
Erik Johnston
e9906b0772
Merge pull request #5836 from matrix-org/erikj/lower_bound_ttl_well_known
Add a lower bound to well-known TTL.
2019-08-13 12:41:16 +01:00
Erik Johnston
fb3469f53a
Clarify docstring
Co-Authored-By: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2019-08-13 10:17:23 +01:00
Erik Johnston
f218705d2a Make default well known cache global again. 2019-08-13 10:06:51 +01:00
Erik Johnston
2546f32b90
Merge pull request #5826 from matrix-org/erikj/reduce_event_pauses
Don't unnecessarily block notifying of new events.
2019-08-13 09:36:25 +01:00
Erik Johnston
9d9cf3583b
Merge pull request #5843 from matrix-org/erikj/workers_hist_vis
Whitelist history visbility sytests for worker mode
2019-08-12 18:02:19 +01:00
Olivier Wilkinson (reivilibre)
1e0bd9a9c0 Fix stats tests and their expectations of the number of events in fresh
rooms – varies depending upon room publicness.

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-12 16:23:29 +01:00
Olivier Wilkinson (reivilibre)
182cdcbf24 Docstrings in storage.stats.
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-12 16:22:18 +01:00
Olivier Wilkinson (reivilibre)
314567d62d Split out partial indices from theschema delta, thus supporting SQLite.
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-12 16:21:35 +01:00
Erik Johnston
2bec3a4953
Merge pull request #5839 from tcitworld/fix-purge-remote-media-script
Fix curl command typo in purge_remote_media.sh
2019-08-12 14:51:27 +01:00
Erik Johnston
3de6cc245f
Changelogs should end in '.' or '!' 2019-08-12 14:16:42 +01:00
Erik Johnston
156a461cbd Newsfile 2019-08-12 13:57:52 +01:00
Erik Johnston
c9456193d3 Whitelist history visbility sytests for worker mode 2019-08-12 13:56:26 +01:00
Richard van der Hoff
fb86217553
Merge pull request #5788 from matrix-org/rav/metaredactions
Fix handling of redactions of redactions
2019-08-12 12:25:19 +01:00
Erik Johnston
41546f946e Newsfile 2019-08-12 09:56:58 +01:00
Thomas Citharel
a7f0161276
Fix curl command typo in purge_remote_media.sh
Was verbose option instead of -X, command didn't work

Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2019-08-09 18:36:12 +02:00
Olivier Wilkinson (reivilibre)
d54ae7118d Move back to defer.inlineCallbacks from async as it makes stats
unergonomic if we move to `async` from the bottom-up.

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-09 15:51:42 +01:00
Olivier Wilkinson (reivilibre)
9064f28e59 Add initial batch of stats tests
Still want to add more, concerning behaviour in the different states
of the background updater completion.

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 16:38:23 +01:00
Olivier Wilkinson (reivilibre)
9ee50ccdf2 Fix stats_separated SQL
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 16:36:16 +01:00
Olivier Wilkinson (reivilibre)
69234072d3 Remove obsolete functions for updating stats absolutely.
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:48:23 +01:00
Olivier Wilkinson (reivilibre)
b3844451f9 Introduce get_room_state; a way to get state for a single room
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:47:02 +01:00
Olivier Wilkinson (reivilibre)
4a3fec1f3b Fix tests
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:46:16 +01:00
Olivier Wilkinson (reivilibre)
7f2ec2e954 Handle user registration and ensure they start accruing statistics
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:44:15 +01:00
Olivier Wilkinson (reivilibre)
1c9732d64b Update incremental processor to use new interfaces and track total_events
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:43:50 +01:00
Olivier Wilkinson (reivilibre)
a59da511ce Update public room visibility change handler
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:43:25 +01:00
Olivier Wilkinson (reivilibre)
4a45fb5ab8 Adapt state delta handling to match the new interface
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:40:51 +01:00
Olivier Wilkinson (reivilibre)
cf9f7ae366 Introduce total_events tracking and rework statistics tracking.
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:37:29 +01:00
Olivier Wilkinson (reivilibre)
5ca4cd5ad4 Track more stats positions
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:32:17 +01:00
Olivier Wilkinson (reivilibre)
4cb921c0f9 Fix type signature in get_current_state_deltas
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:29:31 +01:00
Olivier Wilkinson (reivilibre)
5216299124 Use threading.Lock to prevent concurrent incremental position updates
Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:28:42 +01:00
Olivier Wilkinson (reivilibre)
259014b7ad Schema delta for separated statistics
Separated by current/historical

Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
2019-08-08 11:25:02 +01:00
Erik Johnston
107ad133fc Move well known lookup into a separate clas 2019-08-07 15:36:38 +01:00
Erik Johnston
af9f1c0764 Add a lower bound for TTL on well known results.
It costs both us and the remote server for us to fetch the well known
for every single request we send, so we add a minimum cache period. This
is set to 5m so that we still honour the basic premise of "refetch
frequently".
2019-08-06 17:01:23 +01:00