Commit graph

50 commits

Author SHA1 Message Date
Kevin Ottens
712869db9a Use auto to avoiding repeating type names
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-05-20 03:54:41 +02:00
Christian Kamm
cba0627a9c
Ensure GETFileJob notices finishing #6581
It could happen that readyRead was emitted for incoming data while the
download was not yet finished. Then the network job could finish with
no more data arriving - so readyRead wasn't emitted again.

To fix this, the finished signal also gets connected to the readyRead
slot.
2018-09-09 11:50:00 +02:00
Roeland Jago Douma
1262cbf4ef
Merge pull request #314 from nextcloud/upstream/pr/6373
Make sure ignored and conflict files show up in the issues tab even with partial local discovery
2018-06-02 20:20:19 +02:00
Roeland Jago Douma
bb1b86b087
Merge pull request #307 from nextcloud/upstream/pr/6366
test/testsyncengine: testNoLocalEncoding is for Linux
2018-05-26 21:30:21 +02:00
Christian Kamm
75194d1821
SyncEngine: Make "local discovery?" question available
Also fix the minor bug that was mentioned and add tests.
2018-05-16 16:46:40 +02:00
Markus Goetz
f6b67a866a
test/testsyncengine: testNoLocalEncoding is for Linux 2018-05-15 23:06:39 +02:00
Christian Kamm
ce0e2d0b88
SyncJournal: Don't use LIKE with paths
Paths can contain the wildcards % and _ and that would lead to odd
behavior.

This patch also clarifies the behavior of avoidReadFromDbOnNextSync()
which previously dependend on whether "foo/bar" or "foo/bar/" was
passed as input.

Possibly affects #6322
2018-05-15 19:44:00 +02:00
Olivier Goffart
9613f6437d Merge remote-tracking branch 'origin/2.4'
Conflicts:
	src/libsync/networkjobs.cpp
2018-01-11 10:46:50 +01:00
Christian Kamm
7d70f1becb Ignore files that can't be encoded for the filesystem
There's an upstream bug where QTextCodec::canEncode returns true even
though it should be false. This works around that issue and adds a test.

The original work was done in 72809ef5b1

See #6287, #5676, #5719
See https://bugreports.qt.io/browse/QTBUG-6925
2018-01-10 13:36:00 +01:00
Olivier Goffart
482ee875a1 Fix TestSyncEngine on windows 2017-12-15 14:39:01 +01:00
Olivier Goffart
4369853ddb TestSystem: Add QIODevice in the serverOverride function, and add a DelayedReply
Preparing to add test that needs the QIODevice.
Also make the DelayedReply so we can generalize the existing delay on FakeChunkMoveReply
to any reply.
2017-12-14 11:56:12 +01:00
Christian Kamm
aafe7c4732 Merge remote-tracking branch 'origin/2.4' 2017-11-20 11:54:56 +01:00
Olivier Goffart
480932a58a Checksum: Ignore unkown OC-Checksum header when downloading...
And if there are several checksums, pick the "best" one.

The case of several checksum was reported in
https://github.com/nextcloud/client_theming/issues/213
2017-11-20 10:51:15 +01:00
Olivier Goffart
2192386d86 test/testsyncengine: test for PR #6168 2017-11-16 13:31:54 +01:00
Christian Kamm
e694ffcd03 PropagateDownload: Adjustments to skipping downloads #6153
Previously we required matching mtimes but that's actually
unnecessary when the question is about whether to skip the
download. We will still update the file's metadata.

Also, adjust behavior when the checksum is weak (Adler32):
in these cases we still depend on equal mtimes.
2017-11-16 13:31:54 +01:00
Christian Kamm
37c5612aee Discovery: Treat files starting with '.' as hidden #6145
This bug was introduced when strcmp(a, b) != 0 was accidentally
converted to a == b.
2017-11-08 12:53:46 +01:00
Christian Kamm
05c1bfb6cf Merge remote-tracking branch 'origin/2.4' 2017-10-27 10:29:31 +02:00
Olivier Goffart
ee63b36ed3 SyncFileStatusTracker: Detect changed in the shared flag
... even if the file is not changed.

We get an UPDATE_METADATA in that case, so make sure we let the
SyncFileStatusTracker know about it.
That means we need to filter out UPDATE_METADATA in the other listeners
of this signal.

Issue #6098
2017-10-24 15:50:14 +02:00
Christian Kamm
9c7ee6ef85 Reconcile: Rename handling fixes: duplicate file ids
When users share the same tree several times (say A/ and A/B/ are both
shared) the remote tree can have several entries that have the same
file id. This needs to be respected in rename detection.

Also adds several tests and fixes for issues noticed during testing.

See #6096
2017-10-24 10:54:23 +02:00
Christian Kamm
e85a339d94 Local discovery: Use db instead of filesystem
We mostly trust the file watchers meaning that we don't re-scan the
local tree if we have done that recently and no file watcher events
have arrived. If the file watchers invalidate a subtree, we rescan
only that subtree.

Since we're not entirely sure the file watchers are reliable, we still
do full local discoveries regularly (1h by default). There is a config
file setting as well as an environment variable to control the interval.
2017-10-24 10:00:20 +02:00
Christian Kamm
bf39343920 Sync: Add capability for invalid filename regexes #6092 2017-10-17 17:15:49 +02:00
Christian Kamm
01c2ffe2ae PropagateDownload: Read Content-md5 header #6088 2017-10-11 09:06:23 +02:00
Christian Kamm
90befac901 Rename detection: File size must be equal
Comparison of file sizes for potential conflicts was added in
0eb9401c62, but did not extend to checking
the file size in case of potential local moves.

This commit adds this check and adds tests for various move+change
scenarios.
2017-10-05 14:17:07 +02:00
Olivier Goffart
13e0cc6b50 Use QDateTime::currentDateTimeUtc instead of the non utc version
QDateTime::currentDateTime is terribly slow.
2017-09-26 16:14:17 +02:00
Jocelyn Turcotte
a034ee894c Use SyncJournalDb in csync
This gets rid of the csync_statedb sqlite layer and use
the same code and same connection as the rest of the SyncEngine.

Missing functions are added to SyncJournalDb and change a few minor
things (like changing SyncJournalFileRecord::_modtime to be an int64
instead of a QDateTime, like it was in csync).
2017-09-26 11:03:23 +02:00
Jocelyn Turcotte
6f46764daa SyncJournalDB: Allow callers of getFileRecord if the query failed
The current implementation would return the same value whether the query failed
or if no row would be found. This is something that is currently checked by csync
and needs to be provided if we want to use SyncJournalDB there.

Adjusted all call sites to also check the return value even though they
could still just rely on rec.isValid(), but makes it more explicit as to what
happens for database errors in those cases, if we ever want to gracefully handle
them.
2017-09-26 11:03:23 +02:00
Christian Kamm
78212e03d6 Checksums: Clearer behavior and added testing 2017-09-18 12:52:09 +02:00
Jocelyn Turcotte
a8ea7b0858 Remove SyncFileItem::_isDirectory
It's always equivalent to _type == SyncFileItem::Directory.
2017-09-18 11:46:09 +02:00
Jocelyn Turcotte
0e97770c47 Remove SyncFileItem::log
This remove the remaining "other" fields of the sync log to save a
bit of memory.

other_etag and other_fileId don't give much information to the users
and other_instruction will always be INST_NONE anyway.

other_modtime and other_size are kept since they are sometimes used.
They were renamed to have a bit more meaningful name.

SyncEngine::checkPermissions will now fetch its information from the
csync trees since they are now preserved until right after this point.

Fixes #3213
2017-09-18 11:46:09 +02:00
Christian Kamm
3a55ff2119 PropagateUpload: Model of remote quota, avoid some uploads #5537
When we see a 507 error, assume that quota is < uploaded size.
2017-09-15 09:27:07 +02:00
Christian Kamm
e7a0c1b6d0 SyncEngineTest: _size now correct for uploads #5855 2017-07-12 10:09:04 +02:00
Christian Kamm
0db095e02a SyncEngineTest: Fix test reliability
There was a rounding issue in the mtimes which sometimes resulted in an
off-by-one error. Caused by storing a full QDateTime in the FileInfo but
the mtime saved to the disk being truncated to seconds.
2017-07-04 09:57:22 +02:00
Christian Kamm
851a3128e4 SyncEngine: Add unittest for SyncFileItem properties #5855
Checks instruction, direction, size, modtime for three common cases.
2017-07-03 12:54:24 +02:00
Christian Kamm
8160963110 Compare the hash of files with identical mtime/size #5589
* For conflicts where mtime and size are identical:

  a) If there's no remote checksum, skip (unchanged)
  b) If there's a remote checksum that's a useful hash, create a
     PropagateDownload job and compute the local hash. If the hashes
     are identical, don't download the file and just update metadata.

* Avoid exposing the existence of checksumTypeId beyond the database
  layer. This makes handling checksums easier in general because they
  can usually be treated as a single blob.

  This change was prompted by the difficulty of producing file_stat_t
  entries uniformly from PROPFINDs and the database.
2017-06-15 13:54:16 +02:00
Olivier Goffart
3314a472c3 SyncEngine: SelectiveSync: Remove local files of undelected folder despite other modified files
Issue #5783

When the directry that should be removed by selective sync contains changes,
we ignore the whole sub tree instead of only ignoreing new files.
We cannot ignore the whole directory, we need to ignore only the directory
that do not have files to remove
2017-06-08 14:53:54 +02:00
Christian Kamm
5eaae83a01 Merge remote-tracking branch 'origin/2.3' 2017-05-17 13:10:19 +02:00
Christian Kamm
c5a0ce5a43 Selective sync: Skip excluded folders when reading db
When a new folder becomes selective-sync excluded, we already mark it
and all its parent folders with _invalid_ etags to force rediscovery.

That's not enough however. Later calls to csync_statedb_get_below_path
could still pull data about the excluded files into the remote tree.

That lead to incorrect behavior, such as uploads happening for folders
that had been explicitly excluded from sync.

To fix the problem, statedb_get_below_path is adjusted to not read the
data about excluded folders from the database.

Currently we can't wipe this data from the database outright because we
need it to determine whether the files in the excluded folder can be
wiped away or not.

See owncloud/enterprise#1965
2017-05-16 13:58:45 +02:00
Olivier Goffart
1a98b49f1b Fix TestSyncEngine::testDirDownloadWithError
On master, the scheduling is different and the tasks abort in another
order
2017-03-09 10:35:58 +01:00
Olivier Goffart
0e8f0481d4 Merge remote-tracking branch 'origin/2.3'
Conflicts:
	src/libsync/owncloudpropagator.cpp
2017-03-09 09:54:14 +01:00
Olivier Goffart
f862c626a1 Propagator: Fix finished signal of directory being emited twice and causing crash
When there is a FatalError, we ended up emiting the finished signal for
the directory job several times, which would lead to crashes

Issue #5578
2017-03-06 16:14:04 +01:00
Jocelyn Turcotte
ddb6b3b262 Fix TestSyncEngine::abortAfterFailedMkdir (again)
It could be possible that _firstJob is marked as finished if
aborted before its parent PropagateDirectory was marked as finished,
allowing a posted scheduleNextJob call to schedule the child job
in-between.
2017-02-23 17:54:33 +01:00
Jocelyn Turcotte
824be789bf Revert "Fix TestSyncEngine::abortAfterFailedMkdir"
This reverts commit 1cec2ca13d
but keeps the reset() -> take() part that is probably the cause
of crashes we've seen in the crash reporter.
2017-02-17 14:06:30 +01:00
Jocelyn Turcotte
1cec2ca13d Fix TestSyncEngine::abortAfterFailedMkdir
It could be possible that _firstJob is marked as finished if
aborted before its parent PropagateDirectory was marked as finished,
allowing a posted scheduleNextJob call to schedule the child job
in-between.
2017-02-08 17:32:04 +01:00
Jocelyn Turcotte
1d9e08d109 Skip TestSyncEngine::abortAfterFailedMkdir 2017-02-08 17:25:32 +01:00
Piotr Mrówczyński
c3ae5123cb Delete finished propagation jobs in PropagateDirectory #5269 (#5400) 2017-01-26 10:03:22 +01:00
Jocelyn Turcotte
1fc5a76622 Pass the SyncFileItem as SyncFileItemPtr in itemCompleted
This will allow us to keep a reference on the items in connected slots.
2017-01-25 23:26:23 +01:00
Jocelyn Turcotte
a764d7eb86 Don't pass the PropagatorJob in itemCompleted
This was to catch duplicate emissions for PropagateDirectory but we
don't emit this signal anymore from there.
This fixes a warning about PropagatorJob not being a registered metatype.

This reverts commit fe42c1a818.
2017-01-25 23:26:23 +01:00
Olivier Goffart
d80d5a8ee4 SyncEngine: Fix renaming a folder should keep the selective sync state
Issue #5224

Two problems:

 - In the discovery phase, we need to check the selective sync entries of
   the source path in case of renames.

 - When the rename is done, we need to actually update the black list in the
   database.
2016-10-11 14:38:25 +02:00
Olivier Goffart
85b8ab178e SyncEngine: Fix renaming of folder when file are changed (#5195)
Two bugs:
 - The change filed are not considered as move, they are re-downloaded
   but the old file was not removed from the database. The change in
   owncloudpropagator.cpp takes care of removing the old entries.

 - Next sync would then remove the file in the server in the old folder
   This was not a problem until we start reusing the sync engine, and
   that the _renamedFolders map is not cleared. We were before deleting
   a non-existing file. But now we delete the actual file.

Also improve the tests to be able to do move on the server.
This include support for file id.

Issue #5192
2016-09-22 09:02:47 +02:00
Jocelyn Turcotte
b7ff4a76e8 Add TestSyncEngine and TestSyncFileStatusTracker auto tests
To be able to test the SyncEngine efficiently, a set of server
mocking classes have been implemented on top of QNetworkAccessManager.

The local disk side hasn't been mocked since this would require adding
a large abstraction layer in csync. The SyncEngine is instead pointed
to a different temporary dir in each test and we test by interacting
with files in this directory instead.

The FakeFolder object wraps the SyncEngine with those abstractions
and allow controlling the local files, and the fake remote state
through the FileModifier interface, using a FileInfo tree structure
for the remote-side implementation as well as feeding and comparing
the states on both side in tests.

Tests run fast and require no setup to be run, but each server feature
that we want to test on the client side needs to be implemented in
this fake objects library. For example, the OC-FileId header isn't
set as of this commit, and we can't test the file move logic properly
without implementing it first.

The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL
for what I esteem being issues that need to be fixed in order to catch
up on our test coverage without making this patch too huge.
2016-08-17 15:39:30 +02:00