Commit graph

15479 commits

Author SHA1 Message Date
Kevin Ottens
4117ac2913
Merge pull request #2268 from nextcloud/cherry_pick_testchunkingng_related_improvements
Cherry pick TestChunkingNG related improvements
2020-08-13 17:26:33 +02:00
Kevin Ottens
ef6d2600ca
Size cleanups in TestChunkingNG
Some of the comments didn't match the size or were missing. This also
means reducing one of the 150 MB payloads left behind reducing the
execution time by a few more seconds. This is now around 30s execution
time which is more acceptable.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-13 16:46:14 +02:00
Olivier Goffart
a7847a4e82
Upload: Store the size in the UploadInfo, and compare it when resolving potential conflict
This is about the conflicts that happens when the file has been uploaded
correctly to the server, but the etag was not recieved because the connection
was closed before we got the reply.

We used to compare only the mtime when comparing the uploaded file and the
existing file.  However, to be perfectly correct, we also should check the
size.

This was found because TestChunkingNG::connectionDroppedBeforeEtagRecieved is
flaky. Example of faillure found in https://drone.owncloud.com/owncloud/client/481/5
while testing PR #6626

(very trimmed log:)

06-29 07:58:02:015 [ info sync.csync.csync ]:	## Starting local discovery ##
06-29 07:58:02:016 [ info sync.csync.updater ]:	Database entry found, compare: 1530259082 <-> 1530259051, etag:  <-> 1644a8c8750, inode: 1935629 <-> 1935629, size: 301 <-> 300, perms: 0 <-> ff, type: 0 <-> 0, checksum:  <-> SHA1:cc9adedebe27a6259efb8d6ed09f4f2eff559ad1, ignore: 0
06-29 07:58:02:016 [ info sync.csync.updater ]:	file: A/a0, instruction: INSTRUCTION_EVAL <<=
06-29 07:58:02:972 [ warning sync.networkjob ]:	QNetworkReply::NetworkError(OperationCanceledError) "Connection timed out" QVariant(Invalid)
.. next sync...
06-29 07:58:02:980 [ info sync.engine ]:	#### Discovery start ####################################################
06-29 07:58:02:981 [ info sync.csync.csync ]:	## Starting local discovery ##
06-29 07:58:02:983 [ info sync.csync.updater ]:	Database entry found, compare: 1530259082 <-> 1530259051, etag:  <-> 1644a8c8750, inode: 1935629 <-> 1935629, size: 302 <-> 300, perms: 0 <-> ff, type: 0 <-> 0, checksum:  <-> SHA1:cc9adedebe27a6259efb8d6ed09f4f2eff559ad1, ignore: 0
06-29 07:58:02:983 [ info sync.csync.updater ]:	file: A/a0, instruction: INSTRUCTION_EVAL <<=
06-29 07:58:02:985 [ info sync.csync.csync ]:	## Starting remote discovery ##
06-29 07:58:02:985 [ info sync.networkjob ]:	OCC::LsColJob created for "http://localhost/owncloud" + "" "OCC::DiscoverySingleDirectoryJob"
06-29 07:58:02:987 [ info sync.csync.updater ]:	Database entry found, compare: 1530259082 <-> 1530259051, etag: 1644a8c8b26 <-> 1644a8c8750, inode: 0 <-> 1935629, size: 301 <-> 300, perms: ff <-> ff, type: 0 <-> 0, checksum: SHA1:5adcdac9608ae0811247f07f4cf1ab0a2ef99154 <-> SHA1:cc9adedebe27a6259efb8d6ed09f4f2eff559ad1, ignore: 0
06-29 07:58:02:987 [ info sync.csync.updater ]:	file: A/a0, instruction: INSTRUCTION_EVAL <<=
06-29 07:58:02:989 [ info sync.csync.csync ]:	Update detection for remote replica took 0.004 seconds walking 13 files
06-29 07:58:02:990 [ info sync.engine ]:	#### Discovery end ####################################################  9 ms
06-29 07:58:02:990 [ info sync.database ]:	Updating file record for path: "A/a0" inode: 1935629 modtime: 1530259082 type: 0 etag: "1644a8c8b26" fileId: "16383ea4" remotePerm: "WDNVCKR" fileSize: 301 checksum: "SHA1:cc9adedebe27a6259efb8d6ed09f4f2eff559ad1"
06-29 07:58:02:990 [ info sync.csync.reconciler ]:	INSTRUCTION_UPDATE_METADATA    client file: A/a0
06-29 07:58:02:990 [ info sync.csync.csync ]:	Reconciliation for local replica took  0 seconds visiting  13  files.
06-29 07:58:02:990 [ info sync.csync.reconciler ]:	INSTRUCTION_UPDATE_METADATA    server dir:  A
06-29 07:58:02:990 [ info sync.csync.csync ]:	Reconciliation for remote replica took  0 seconds visiting  13  files.
06-29 07:58:02:990 [ info sync.engine ]:	#### Reconcile end ####################################################  9 ms
06-29 07:58:02:990 [ info sync.database ]:	Updating local metadata for: "A/a0" 1530259082 302 1935629
FAIL!  : TestChunkingNG::connectionDroppedBeforeEtagRecieved(small file) '!fakeFolder.syncOnce()' returned FALSE. ()
2020-08-13 16:46:08 +02:00
Olivier Goffart
c2324327ef
TestChunkingNG: Make it faster
Use smaller files so the test run faster.
Particulary usefull for TestChunkingNG::connectionDroppedBeforeEtagRecieved
Which had become so much slower after 2638332dc6
increased the timeout for bigger files
2020-08-13 16:46:03 +02:00
Christian Kamm
c74c0d8e6e
Upload: Add more chunk resuming unit tests #6405 2020-08-13 16:45:55 +02:00
Kevin Ottens
2684ba149e
Merge pull request #2269 from nextcloud/enable_modernize_loop_convert
Enable the modernize-loop-convert check on clang-tidy
2020-08-13 16:37:41 +02:00
Kevin Ottens
866ffc2a6b Enable the modernize-loop-convert check on clang-tidy
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-13 12:55:14 +00:00
Kevin Ottens
a5b4922a0e Remove unused method
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-13 12:55:14 +00:00
Kevin Ottens
0e2af4b502 Enable the modernize-deprecated-headers check on clang-tidy
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-13 14:53:52 +02:00
Kevin Ottens
2f8c29082a Enable the modernize-use-equals-default check on clang-tidy
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-13 14:19:42 +02:00
Nextcloud bot
2d4d66b998
[tx-robot] updated from transifex 2020-08-13 03:30:50 +00:00
Kevin Ottens
cdf6e7d72b Enable the modernize-user-nullptr check on clang-tidy
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 19:45:25 +02:00
Kevin Ottens
3ff899cff9
Merge pull request #2264 from nextcloud/enable_modernize_use_using
Enable the modernize-use-using check on clang-tidy
2020-08-12 19:29:17 +02:00
Kevin Ottens
6ae1d0ddf6
Get rid of useless using constructs
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 18:18:57 +02:00
Kevin Ottens
97af14a4f0
Make QTokenizer use simpler by having optional second parameter
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 18:18:57 +02:00
Kevin Ottens
d58ec4bf70
Enable the modernize-use-using check on clang-tidy
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 18:18:57 +02:00
Kevin Ottens
7bb54e34d7
Merge pull request #2262 from nextcloud/enable_trivial_clang_tidy_modernize_checks
Enable "trivial" modernize clang-tidy checks
2020-08-12 17:16:38 +02:00
Kevin Ottens
07b7753216 Enable "trivial" modernize clang-tidy checks
This is a bunch of modernize checks which we don't hit at all. This is
an easy win and also make sure those old constructs won't creep up
behind our back.

Note that some of them won't trigger until we bump our C++ baseline
standard to C++17. But I'd say that's OK, that'll force our hand to do
those C++17 related cleanups when the time comes.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 13:13:08 +00:00
Kevin Ottens
1767c0295f
Merge pull request #2037 from nextcloud/enable_modernize_raw_string_literal
Enable modernize raw string literal
2020-08-12 15:02:55 +02:00
Kevin Ottens
3e79e1861b
We don't really need two \ here
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 09:48:33 +02:00
Kevin Ottens
c50a968a1e
Add more raw string literals missed previously
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 09:48:29 +02:00
Kevin Ottens
ed1c2eaab4
Merge pull request #2247 from nextcloud/improve_drone_config
Improve drone config
2020-08-12 09:45:26 +02:00
Kevin Ottens
14bfa5025f Make sure wget looks for the certificates at the right place
Might happen because we got too many copies of OpenSSL around

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 06:45:13 +00:00
Kevin Ottens
571e15c6eb Switch to Ninja for the clang build
Clang is generally slower to compile than GCC so use a faster Make for
that build variant. This also allows us to verify our CMake files don't
do anything Make specific.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 06:45:13 +00:00
Kevin Ottens
5116f4d255 Parallelize the clang-tidy runs
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 06:45:13 +00:00
Kevin Ottens
db5c186202 Split cmake and compile into separate pipeline steps
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 06:45:13 +00:00
Kevin Ottens
725f1e742d Switch to actual commands inside of the steps
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 06:45:12 +00:00
Kevin Ottens
6da67ef653 Split the CI pipelines into proper steps
Will make it easier to navigate the logs

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 06:45:12 +00:00
Kevin Ottens
65407483e5 Now make sure we're not picking up generated headers
This could happen (and started to happen more with clang-tidy 10) that
clang-tidy picked up errors in headers from the build directory. Now
that we moved the build directory out of source, we can simply filter
headers based on the pwd.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 06:45:12 +00:00
Kevin Ottens
1e8340cd48 Move the build in a volume to be reused between steps
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 06:45:12 +00:00
Kevin Ottens
9016ffe4b7 Remove unneeded environment variables
Those are set by the base CI image anyway, no need to do the work again.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 06:45:12 +00:00
Kevin Ottens
be6a9dab01 Switch to the newer base CI image
We can now spare building QtKeychain over and over shortening a bit the
build. We also update to latest gcc and clang.

Note that I didn't switch the AppImage pipeline to the newer image to
keep the binary compatibility promise required by linuxdeployqt.
Probably would make sense to do a specific image for AppImage with some
other of the cleanups I did there (in particular to avoid building
qtkeychain every time).

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 06:45:12 +00:00
Kevin Ottens
d264f7078b Parallelize the CI build
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-12 06:45:12 +00:00
Nextcloud bot
c3cd06ddad
[tx-robot] updated from transifex 2020-08-12 03:30:27 +00:00
Kevin Ottens
c5343e10f5
Merge pull request #2258 from nextcloud/have_a_way_to_display_e2ee_mnemonic
Keep the E2EE info message around to allow displaying mnemonic
2020-08-11 15:11:40 +02:00
Kevin Ottens
37e20698ca
Keep the E2EE info message around to allow displaying mnemonic
We lost the ability to display the E2EE mnemonic during the GUI
redesign and the info message wasn't displayed again on restart. So now
we display it every time, it still can be dismissed and the button text
is different in such a case to make the intent clearer.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-11 14:55:51 +02:00
Nextcloud bot
a1dbfd71cb
[tx-robot] updated from transifex 2020-08-11 03:30:03 +00:00
Kevin Ottens
d08c160b69
Merge pull request #2255 from nextcloud/restore_e2ee_old_key_storage_format_compatibility
Restore E2EE old key storage format compatibility
2020-08-10 16:07:06 +02:00
Kevin Ottens
ef3c516598
Add support for BASE64 encoded '|' when decrypting
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-10 13:13:19 +02:00
Kevin Ottens
595eb78c8a
Move the private key salt handling in its own function
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-10 13:13:19 +02:00
Kevin Ottens
d5339265fb
Ignore the salt part of the key during decryption
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-10 13:13:19 +02:00
Kevin Ottens
54a19945fd
Restore the symmetry between *StringSymmetric functions
If we receive data without base64 encoding for encryption, it makes
sense to get it without base64 encoding out of decryption.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-10 13:13:19 +02:00
Kevin Ottens
26b88131cc
Add unit tests for the encryption helpers
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-10 13:13:19 +02:00
Nextcloud bot
aac1f2ae56
[tx-robot] updated from transifex 2020-08-09 03:29:44 +00:00
Nextcloud bot
7e2c3b2516
[tx-robot] updated from transifex 2020-08-08 03:31:51 +00:00
Kevin Ottens
a813602124 Master is now what will become 3.1
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-07 14:36:44 +02:00
Nextcloud bot
53f346fa9b
[tx-robot] updated from transifex 2020-08-07 03:32:17 +00:00
Kevin Ottens
ff60f75b2a
Merge pull request #2248 from nextcloud/bump_version_to_3.0
There won't be a 2.7, this is called 3.0 instead
2020-08-06 14:25:00 +02:00
Kevin Ottens
39de633828
There won't be a 2.7, this is called 3.0 instead
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-08-06 13:40:00 +02:00
Nextcloud bot
2ebc4041ac
[tx-robot] updated from transifex 2020-08-06 03:31:10 +00:00