7493 commits

Author	SHA1	Message	Date
Erik Johnston	f793bc3877	Split out stream store	2018-03-01 15:13:08 +00:00
Erik Johnston	784f036306	Move RoomMemberHandler out of Handlers	2018-03-01 14:36:50 +00:00
Erik Johnston	6411f725be	Calculate stream_ordering_month_ago correctly on workers	2018-03-01 14:20:53 +00:00
Erik Johnston	a9a2d66cdd	Split out SignatureStore and EventFederationStore	2018-03-01 14:17:53 +00:00
Erik Johnston	0c8ba5dd1c	Split up RoomStore	2018-03-01 14:01:19 +00:00
Richard van der Hoff	3a75de923b	Rewrite make_deferred_yieldable avoiding inlineCallbacks ... ... because (a) it's actually simpler (b) it might be marginally more performant?	2018-03-01 12:40:05 +00:00
Erik Johnston	126b9bf96f	Log in the correct places	2018-03-01 12:05:33 +00:00
Erik Johnston	157298f986	Don't do preserve_fn for every request	2018-03-01 11:59:45 +00:00
Erik Johnston	89f90d808a	Add some logging	2018-03-01 11:59:16 +00:00
Erik Johnston	8ded8ba2c7	Make repl send_event idempotent and retry on timeouts ... If we treated timeouts as failures on the worker we would attempt to clean up e.g. push actions while the master might still process the event.	2018-03-01 11:20:34 +00:00
Erik Johnston	f381d63813	Check event auth on the worker	2018-03-01 10:18:37 +00:00
Erik Johnston	6b8604239f	Correctly send ratelimit and extra_users params	2018-03-01 10:08:39 +00:00
Erik Johnston	f756f961ea	Fixup comments	2018-03-01 10:05:27 +00:00
Erik Johnston	28e973ac11	Calculate push actions on worker	2018-02-28 18:02:30 +00:00
Erik Johnston	493e25d554	Move storage functions for push calculations ... This will allow push actions for an event to be calculated on workers.	2018-02-27 13:58:16 +00:00
Erik Johnston	3594dbc6dc	Merge pull request #2904 from matrix-org/erikj/receipt_cache_invalidation ... Fix missing invalidations for receipt storage	2018-02-27 11:34:26 +00:00
Erik Johnston	2311189ee4	Merge pull request #2903 from matrix-org/erikj/split_roommember_store ... Split out RoomMemberStore	2018-02-27 11:32:10 +00:00
Erik Johnston	c57607874c	Merge pull request #2901 from matrix-org/erikj/split_as_stores ... Split AS stores	2018-02-27 10:07:07 +00:00
Erik Johnston	8956f0147a	Add comment	2018-02-27 10:06:51 +00:00
Erik Johnston	e5b4a208ce	Merge pull request #2892 from matrix-org/erikj/batch_inserts_push_actions ... Batch inserts into event_push_actions_staging	2018-02-26 14:45:40 +00:00
Erik Johnston	45b5fe9122	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/handle_unpersisted_events_push	2018-02-26 13:49:24 +00:00
Erik Johnston	d62ce972f8	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_roommember_store	2018-02-23 11:46:24 +00:00
Erik Johnston	6ae9a3d2a6	Update copyright	2018-02-23 11:44:49 +00:00
Erik Johnston	a90c60912f	Merge branch 'develop' of github.com:matrix-org/synapse into erikj/split_event_push_actions	2018-02-23 11:26:31 +00:00
Erik Johnston	50e8657867	Merge pull request #2902 from matrix-org/erikj/split_events_store ... Split out get_events and co into a worker store	2018-02-23 11:23:52 +00:00
Erik Johnston	1cf9e071dd	Merge pull request #2899 from matrix-org/erikj/split_pushers ... Split PusherStore	2018-02-23 11:23:35 +00:00
Erik Johnston	d0957753bf	Merge pull request #2898 from matrix-org/erikj/split_push_rules_store ... Split PushRulesStore	2018-02-23 11:23:23 +00:00
Erik Johnston	199dba6c15	Merge pull request #2897 from matrix-org/erikj/split_account_data ... Split AccountDataStore and TagStore	2018-02-23 11:23:11 +00:00
Erik Johnston	70349872c2	Update copyright	2018-02-23 11:14:35 +00:00
Erik Johnston	eba93b05bf	Split EventsWorkerStore into separate file	2018-02-23 11:01:21 +00:00
Erik Johnston	bf8a36e080	Update copyright	2018-02-23 10:52:10 +00:00
Erik Johnston	5d0f665848	Remove redundant clock	2018-02-23 10:49:58 +00:00
Erik Johnston	3bd760628b	_event_persist_queue shouldn't be in worker store	2018-02-23 10:49:18 +00:00
Erik Johnston	eb9b5eec81	Update copyright	2018-02-23 10:42:39 +00:00
Erik Johnston	c2ecfcc3a4	Update copyright	2018-02-23 10:41:34 +00:00
Erik Johnston	7e6cf89dc2	Update copyright	2018-02-23 10:39:19 +00:00
Erik Johnston	26d37f7a63	Update copyright	2018-02-23 10:33:55 +00:00
Erik Johnston	bb73f55fc6	Use absolute imports	2018-02-23 10:31:16 +00:00
Erik Johnston	faeb369f15	Fix missing invalidations for receipt storage	2018-02-21 15:19:54 +00:00
Erik Johnston	3dec9c66b3	Split out RoomMemberStore	2018-02-21 12:07:26 +00:00
Erik Johnston	46244b2759	Split AS stores	2018-02-21 11:49:34 +00:00
Erik Johnston	27b094f382	Split out get_events and co into a worker store	2018-02-21 11:41:48 +00:00
Erik Johnston	573712da6b	Update comments	2018-02-21 11:29:49 +00:00
Erik Johnston	c96d547f4d	Actually use new param	2018-02-21 11:03:42 +00:00
Erik Johnston	d15d237b0d	Split out EventPushActionWorkerStore	2018-02-21 11:01:13 +00:00
Erik Johnston	27939cbb0e	Merge pull request #2893 from matrix-org/erikj/delete_from_staging_fed ... Delete from push_actions_staging in federation too	2018-02-21 11:00:06 +00:00
Erik Johnston	6f72765371	Split PusherStore	2018-02-21 10:54:21 +00:00
Erik Johnston	cbaad969f9	Split PushRulesStore	2018-02-21 10:43:31 +00:00
Erik Johnston	ca9b9d9703	Split AccountDataStore and TagStore	2018-02-21 10:15:04 +00:00
Erik Johnston	8fbb4d0d19	Raise exception in abstract method	2018-02-20 17:59:23 +00:00
Erik Johnston	95e4cffd85	Fix comment	2018-02-20 17:58:40 +00:00
Erik Johnston	e316bbb4c0	Use abstract base class to access stream IDs	2018-02-20 17:43:57 +00:00
Erik Johnston	f5ac4dc2d4	Split ReceiptsStore	2018-02-20 16:28:28 +00:00
Erik Johnston	24087bffa9	Ensure all push actions are deleted from staging	2018-02-20 12:34:31 +00:00
Erik Johnston	ad0ccf15ea	Refactor _set_push_actions_for_event_and_users_txn to use events_and_contexts	2018-02-20 12:34:28 +00:00
Erik Johnston	d874d4f2d7	Delete from push_actions_staging in federation too	2018-02-20 11:37:52 +00:00
Erik Johnston	6ff8c87484	Batch inserts into event_push_actions_staging	2018-02-20 11:33:07 +00:00
Erik Johnston	324c3e9399	Merge pull request #2868 from matrix-org/erikj/refactor_media_storage ... Make store_file use store_into_file	2018-02-20 11:31:24 +00:00
Pascal Bach	3acd616979	Update pynacl dependency to 1.2.1 or higher ... Signed-off-by: Pascal Bach <pascal.bach@nextrem.ch>	2018-02-19 10:45:22 +01:00
Richard van der Hoff	d1a3325f99	(Really) fix tablescan of event_push_actions on purge ... commit 278d21b5 added new code to avoid the tablescan, but didn't remove the old :/	2018-02-16 14:02:31 +00:00
Erik Johnston	6af025d3c4	Fix typo of double is_highlight	2018-02-16 11:35:31 +00:00
Erik Johnston	012e8e142a	Comments	2018-02-16 11:35:01 +00:00
Erik Johnston	b96278d6fe	Ensure that we delete staging push actions on errors	2018-02-15 15:47:06 +00:00
Erik Johnston	4810f7effd	Remove context.push_actions	2018-02-15 15:47:06 +00:00
Erik Johnston	c714c61853	Update event_push_actions table from staging table	2018-02-15 15:47:06 +00:00
Erik Johnston	acac21248c	Store push actions in staging area	2018-02-15 15:47:04 +00:00
Erik Johnston	106906a65e	Don't serialize current state over replication	2018-02-15 13:53:18 +00:00
Erik Johnston	5fb347fc41	Merge pull request #2872 from matrix-org/erikj/event_worker_dont_log ... Don't log errors propogated from send_event	2018-02-15 12:31:49 +00:00
Erik Johnston	fd1601c596	Fix state group storage bug in workers ... We needed to move `_count_state_group_hops_txn` to the StateGroupWorkerStore.	2018-02-15 11:04:32 +00:00
Erik Johnston	ef344b10e5	Don't log errors propogated from send_event	2018-02-15 11:03:49 +00:00
Richard van der Hoff	b8d821aa68	Merge pull request #2867 from matrix-org/rav/rework_purge ... purge_history cleanups	2018-02-15 09:49:07 +00:00
Erik Johnston	92c52df702	Make store_file use store_into_file	2018-02-14 17:55:18 +00:00
Richard van der Hoff	d28ec43e15	Merge pull request #2769 from matrix-org/matthew/hit_the_gin ... switch back from GIST to GIN indexes	2018-02-14 16:59:03 +00:00
Richard van der Hoff	39bf47319f	purge_history: fix sqlite syntax error ... apparently sqlite insists on indexes being named	2018-02-14 16:42:19 +00:00
Richard van der Hoff	ac27f6a35e	purge_history: handle sqlite asshattery ... apparently creating a temporary table commits the transaction. because that's a useful thing.	2018-02-14 16:41:12 +00:00
Richard van der Hoff	5978dccff0	remove overzealous exception handling	2018-02-14 15:54:09 +00:00
Richard van der Hoff	278d21b5e4	purge_history: fix index use ... event_push_actions doesn't have an index on event_id, so we need to specify room_id.	2018-02-14 15:44:51 +00:00
Richard van der Hoff	5fcbf1e07c	Rework event purge to use a temporary table ... ... which should speed things up by reducing the amount of data being shuffled across the connection	2018-02-14 11:02:22 +00:00
Erik Johnston	c0c9327fe0	Merge pull request #2854 from matrix-org/erikj/event_create_worker ... Create a worker for event creation	2018-02-13 18:07:10 +00:00
Richard van der Hoff	d627174da2	Fix log message in purge_history ... (we don't just remove remote events)	2018-02-13 16:51:21 +00:00
Richard van der Hoff	ddb6a79b68	Merge branch 'matthew/gin_work_mem' into matthew/hit_the_gin	2018-02-13 16:45:36 +00:00
Richard van der Hoff	0b27ae8dc3	move search reindex to schema 47 ... We're up to schema v47 on develop now, so this will have to go in there to have an effect. This might cause an error if somebody has already run it in the v46 guise, and runs it again in the v47 guise, because it will cause a duplicate entry in the bbackground_updates table. On the other hand, the entry is removed once it is complete, and it is unlikely that anyone other than matrix.org has run it on v46. The update itself is harmless to re-run because it deliberately copes with the index already existing.	2018-02-13 16:44:46 +00:00
Richard van der Hoff	4a6d551704	GIN reindex: Fix syntax errors, improve exception handling	2018-02-13 16:44:46 +00:00
Richard van der Hoff	bfdf7b9237	Merge pull request #2864 from matrix-org/rav/persist_event_caching ... Use StateResolutionHandler to resolve state in persist_events	2018-02-13 14:45:57 +00:00
Richard van der Hoff	630caf8a70	style nit	2018-02-13 14:29:22 +00:00
Richard van der Hoff	8fd1a32456	Fix typos in purge api & doc ... * It's supposed to be purge_local_events, not ..._history * Fix the doc to have valid json	2018-02-13 13:09:39 +00:00
Richard van der Hoff	4d09366656	Merge pull request #2695 from okurz/feature/allow_recent_pysaml ... Allow use of higher versions of saml2	2018-02-13 12:24:08 +00:00
Richard van der Hoff	a9b712e9dc	Merge branch 'develop' into matthew/gin_work_mem	2018-02-13 12:16:01 +00:00
Erik Johnston	1026690cd2	Merge pull request #2857 from matrix-org/erikj/upload_store ... Tell storage providers about new file so they can upload	2018-02-12 13:52:58 +00:00
Richard van der Hoff	10b34dbb9a	Merge pull request #2858 from matrix-org/rav/purge_updates ... delete_local_events for purge_room_history	2018-02-09 14:11:00 +00:00
Richard van der Hoff	39a6b35496	purge: move room_depth update to end ... ... to avoid locking the table for too long	2018-02-09 13:07:41 +00:00
Richard van der Hoff	74fcbf741b	delete_local_events for purge_history ... Add a flag which makes the purger delete local events	2018-02-09 13:07:41 +00:00
Richard van der Hoff	e571aef06d	purge: Move cache invalidation to more appropriate place ... it was a bit of a non-sequitur there	2018-02-09 13:07:41 +00:00
Richard van der Hoff	61ffaa8137	bump purge logging to info ... this thing takes ages and the only sign of any progress is the logs, so having some logs is useful.	2018-02-09 13:07:41 +00:00
Richard van der Hoff	671540dccf	rename delete_old_state -> purge_history ... (beacause it deletes more than state)	2018-02-09 13:07:41 +00:00
Erik Johnston	5fa571a91b	Tell storage providers about new file so they can upload	2018-02-07 13:35:08 +00:00
Erik Johnston	50fe92cd26	Move presence handling into handle_new_client_event ... As we want to have it run on the main synapse instance	2018-02-07 10:34:09 +00:00
Erik Johnston	8ec2e638be	Add event_creator worker	2018-02-07 10:32:32 +00:00
Erik Johnston	24dd73028a	Add replication http endpoint for event sending	2018-02-07 10:32:32 +00:00
Erik Johnston	e3624fad5f	Remove pointless ratelimit check ... The intention was for the check to be called as early as possible in the request, but actually was called just before the main ratelimit check, so was fairly pointless.	2018-02-07 10:30:25 +00:00
Erik Johnston	617199d73d	Merge pull request #2847 from matrix-org/erikj/separate_event_creation ... Split event creation into a separate handler	2018-02-06 17:01:17 +00:00
Erik Johnston	3e1e69ccaf	Update copyright	2018-02-06 16:40:38 +00:00
Erik Johnston	770b2252ca	s/_create_new_client_event/create_new_client_event/	2018-02-06 16:40:30 +00:00
Erik Johnston	3d33eef6fc	Store state groups separately from events (#2784) ... * Split state group persist into seperate storage func * Add per database engine code for state group id gen * Move store_state_group to StateReadStore This allows other workers to use it, and so resolve state. * Hook up store_state_group * Fix tests * Rename _store_mult_state_groups_txn * Rename StateGroupReadStore * Remove redundant _have_persisted_state_group_txn * Update comments * Comment compute_event_context * Set start val for state_group_id_seq ... otherwise we try to recreate old state groups * Update comments * Don't store state for outliers * Update comment * Update docstring as state groups are ints	2018-02-06 14:31:24 +00:00
Richard van der Hoff	b31bf0bb51	Merge pull request #2849 from matrix-org/rav/clean_up_state_delta ... Remove redundant return value from _calculate_state_delta	2018-02-05 17:42:20 +01:00
Richard van der Hoff	ebfe64e3d6	Use StateResolutionHandler to resolve state in persist events ... ... and thus benefit (hopefully) from its cache.	2018-02-05 16:23:26 +00:00
Richard van der Hoff	225dc3b4cb	Flatten _get_new_state_after_events ... rejig the if statements to simplify the logic and reduce indentation	2018-02-05 16:23:25 +00:00
Richard van der Hoff	9fcbbe8e7d	Check that events being persisted have state_group	2018-02-05 16:23:25 +00:00
Richard van der Hoff	447aed42d2	Add event_map param to resolve_state_groups	2018-02-05 16:23:25 +00:00
Richard van der Hoff	ee6fb4cf85	Remove redundant return value from _calculate_state_delta ... we already have the state from _get_new_state_after_events, so returning it from _calculate_state_delta is just confusing.	2018-02-05 16:23:20 +00:00
Richard van der Hoff	3c7b480ba3	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-05 16:12:14 +00:00
Erik Johnston	3fa362502c	Update places where we create events	2018-02-05 16:01:48 +00:00
Erik Johnston	5ff3d23564	Split event creation into a separate handler	2018-02-05 16:01:48 +00:00
Richard van der Hoff	c46e75d3d8	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-05 15:43:22 +00:00
Richard van der Hoff	db91e72ade	Merge pull request #2844 from matrix-org/rav/evicted_metrics ... montoring metrics for number of cache evictions	2018-02-05 16:34:36 +01:00
Richard van der Hoff	bc496df192	report metrics on number of cache evictions	2018-02-05 15:34:01 +00:00
Erik Johnston	1f881e0746	Merge pull request #2791 from matrix-org/erikj/media_storage_refactor ... Ensure media is in local cache before thumbnailing	2018-02-05 11:28:52 +00:00
Richard van der Hoff	80b8a28100	Factor out common code for search insert ... we can reuse the same code as is used for event insert, for doing the background index population.	2018-02-04 00:23:06 +00:00
Richard van der Hoff	bd25f9cf36	Clean up work_mem handling ... Add some comments and improve exception handling when twiddling work_mem for the search update	2018-02-03 23:05:41 +00:00
Richard van der Hoff	4eeae7ad65	Move store_event_search_txn to SearchStore ... ... as a precursor to making event storing and doing the bg update share some code.	2018-02-03 22:59:45 +00:00
Richard van der Hoff	bb9f0f3cdb	Merge branch 'develop' into matthew/gin_work_mem	2018-02-03 22:40:28 +00:00
Richard van der Hoff	6b02fc80d1	Reinstate event_search_postgres_gist handler ... People may have queued updates for this, so we can't just delete it.	2018-02-02 14:32:51 +00:00
Richard van der Hoff	9c9356512e	Merge pull request #2845 from matrix-org/rav/urlcache_error_handling ... Handle url_previews with no content-type	2018-02-02 15:27:52 +01:00
Richard van der Hoff	18eae413af	Merge pull request #2842 from matrix-org/rav/state_resolution_handler ... Factor out resolve_state_groups to a separate handler	2018-02-02 15:27:35 +01:00
Richard van der Hoff	78d6ddba86	Merge pull request #2841 from matrix-org/rav/refactor_calc_state_delta ... factor _get_new_state_after_events out of _calculate_state_delta	2018-02-02 15:27:15 +01:00
Richard van der Hoff	9dcd667ac2	Merge pull request #2836 from matrix-org/rav/resolve_state_events_docstring ... Docstring fixes	2018-02-02 15:26:49 +01:00
Travis Ralston	6e87b34f7b	Merge branch 'develop' into travis/admin-list-media	2018-02-01 18:05:47 -07:00
Richard van der Hoff	d5352cbba8	Handle url_previews with no content-type ... avoid failing with an exception if the remote server doesn't give us a Content-Type header. Also, clean up the exception handling a bit.	2018-02-02 00:53:46 +00:00
Richard van der Hoff	14737ba495	doc arg types for _seperate	2018-02-01 12:41:34 +00:00
Richard van der Hoff	e15d4ea248	More docstring fixes ... Fix a couple of errors in docstrings	2018-02-01 12:41:34 +00:00
Richard van der Hoff	a18828c129	Fix docstring for StateHandler.resolve_state_groups ... The return type was a complete lie, so fix it	2018-02-01 12:41:34 +00:00
Richard van der Hoff	6da4c4d3bd	Factor out resolve_state_groups to a separate handler ... We extract the storage-independent bits of the state group resolution out to a separate functiom, and stick it in a new handler, in preparation for its use from the storage layer.	2018-02-01 12:40:04 +00:00
Richard van der Hoff	0cbda53819	Rename resolve_state_groups -> resolve_state_groups_for_events ... (to make way for a method that actually just does the state group resolution)	2018-02-01 12:40:00 +00:00
Richard van der Hoff	77c0629ebc	Merge pull request #2837 from matrix-org/rav/fix_quarantine_media ... Fix sql error in quarantine_media	2018-02-01 11:45:58 +01:00
Travis Ralston	e16e45b1b4	pep8 ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 15:30:38 -07:00
Richard van der Hoff	e1e4ec9f9d	factor _get_new_state_after_events out of _calculate_state_delta ... This reduces the scope of a bunch of variables	2018-01-31 21:32:09 +00:00
Richard van der Hoff	78e7e05188	Merge pull request #2838 from matrix-org/rav/fix_logging_on_dns_fail ... Remove spurious log argument	2018-01-31 22:18:46 +01:00
Richard van der Hoff	ad48dfe73d	Merge pull request #2835 from matrix-org/rav/remove_event_type_param ... Remove unused "event_type" param on state.get_current_state_ids	2018-01-31 22:17:57 +01:00
Richard van der Hoff	518a74586c	Merge pull request #2834 from matrix-org/rav/better_persist_event_exception_handling ... Improve exception handling in persist_event	2018-01-31 22:17:38 +01:00
Richard van der Hoff	d1fe4db882	Merge pull request #2833 from matrix-org/rav/pusher_hacks ... Logging and metrics for the http pusher	2018-01-31 22:16:59 +01:00
Richard van der Hoff	421d68ca8c	Merge pull request #2817 from matrix-org/rav/http_conn_pool ... Use a connection pool for the SimpleHttpClient	2018-01-31 22:14:22 +01:00
Travis Ralston	63c4383927	Documentation and naming ... Signed-off-by: Travis Ralston <travpc@gmail.com>	2018-01-31 08:07:52 -07:00
Richard van der Hoff	af19f5e9aa	Remove spurious log argument ... ... which would cause scary-looking and unhelpful errors in the log on dns fail	2018-01-30 17:52:03 +00:00
Richard van der Hoff	773f0eed1e	Fix sql error in quarantine_media	2018-01-30 15:02:51 +00:00
Richard van der Hoff	adfc0c9539	docstring for get_current_state_ids	2018-01-29 17:39:55 +00:00
Richard van der Hoff	d413a2ba98	Remove unused "event_type" param on state.get_current_state_ids ... this param doesn't seem to be used, and is a bit pointless anyway because it can easily be replicated by the caller. It is also horrible, because it changes the return type of the method.	2018-01-29 17:06:57 +00:00
Richard van der Hoff	b387ee17b6	Improve exception handling in persist_event ... 1. use `deferred.errback()` instead of `deferred.errback(e)`, which means that a Failure object will be constructed using the current exception state, *including* its stack trace - so the stack trace is saved in the Failure, leading to better exception reports. 2. Set `consumeErrors=True` on the ObservableDeferred, because we know that there will always be at least one observer - which avoids a spurious "CRITICAL: unhandled exception in Deferred" error in the logs	2018-01-29 17:05:33 +00:00
Richard van der Hoff	03dd745fe2	Better logging when pushes fail	2018-01-29 15:49:06 +00:00
Richard van der Hoff	e051abd20b	add appid/device_display_name to to pusher logging	2018-01-29 15:04:16 +00:00
Richard van der Hoff	02ba118f81	Increase http conn pool size	2018-01-29 14:30:15 +00:00