Commit graph

525 commits

Author SHA1 Message Date
Christian Kamm
8160963110 Compare the hash of files with identical mtime/size #5589
* For conflicts where mtime and size are identical:

  a) If there's no remote checksum, skip (unchanged)
  b) If there's a remote checksum that's a useful hash, create a
     PropagateDownload job and compute the local hash. If the hashes
     are identical, don't download the file and just update metadata.

* Avoid exposing the existence of checksumTypeId beyond the database
  layer. This makes handling checksums easier in general because they
  can usually be treated as a single blob.

  This change was prompted by the difficulty of producing file_stat_t
  entries uniformly from PROPFINDs and the database.
2017-06-15 13:54:16 +02:00
Christian Kamm
5eaae83a01 Merge remote-tracking branch 'origin/2.3' 2017-05-17 13:10:19 +02:00
Christian Kamm
c5a0ce5a43 Selective sync: Skip excluded folders when reading db
When a new folder becomes selective-sync excluded, we already mark it
and all its parent folders with _invalid_ etags to force rediscovery.

That's not enough however. Later calls to csync_statedb_get_below_path
could still pull data about the excluded files into the remote tree.

That lead to incorrect behavior, such as uploads happening for folders
that had been explicitly excluded from sync.

To fix the problem, statedb_get_below_path is adjusted to not read the
data about excluded folders from the database.

Currently we can't wipe this data from the database outright because we
need it to determine whether the files in the excluded folder can be
wiped away or not.

See owncloud/enterprise#1965
2017-05-16 13:58:45 +02:00
Jocelyn Turcotte
cc9680c1e8 Also pass csync logs through QMessageLogger
This allows using logging rules and custom message patterns with csync.
This also pass the a relative verbosity to the message logger.

Issue #5647
2017-05-11 17:22:59 +02:00
Olivier Goffart
13705999f7 Discovery: Increase the MAX_DEPTH and show deep folder as ignored
Before this patch, to deep folder would just be ignored, without any feedback.
This patch makes it so deep folder are properly shown as ignored in the UI.

Also increase the MAX_DEPTH

Issue: #1067
2017-05-08 11:29:51 +02:00
Jocelyn Turcotte
b26db062d2 Fix the UNIT_TESTING build on Windows
- Put all tests in the bin directory so that DLLs can be loaded
- Add missing exports
- Skip tests that use code depending on zlib
- The "GMT" timezone is named differently, use the int constructor instead

5 tests are still failing, it's not really worth fixing at the moment
since no developper is currently using Windows as its main platform.
2017-02-14 14:32:41 +01:00
Olivier Goffart
0d16cf41fe Merge remote-tracking branch 'origin/master' into 2.3
We can do that because the only changes that were in master but not in 2.3 were the
translations change and documentation change, and the support for the 'M' permission
which we want in 2.3.
2017-01-27 17:45:05 +01:00
Olivier Goffart
70da607e06 Discovery: Add branding option to disable default sync of 'M' directories
Issue #5331 and https://github.com/owncloud/enterprise/issues/1594
2017-01-27 15:59:59 +01:00
Jocelyn Turcotte
f1e9be4fa8 Don't use the StopWatch for each job in production builds #5456
This accounted for more than 10% of the peak memory usage while only
providing value through the debugging log.
2017-01-26 17:13:54 +01:00
Markus Goetz
c6f4f44619 Fix up SSL client certificates #5213 #69 (#5289)
The re-enables the UI, uses Qt API for importing and
stores the certificate/key in the system keychain.
People who had set up client certs need to re-setup the account. This is ok
since it was an undocumented feature anyway.
2017-01-02 08:34:02 +01:00
Christian Kamm
276985f6c3 Fix perl tests for sync journal db name 2016-11-25 13:21:34 +01:00
Christian Kamm
9641c7a1e7 Rename sync journal to ._sync_xxx.db.
The added underscore means that older clients will also ignore the
file.
2016-11-23 16:48:15 +01:00
Christian Kamm
3e59a9b316 Merge branch 'master' into dbjournal_per_account 2016-11-23 16:47:56 +01:00
Olivier Goffart
a1dc4069c9 libsync: Don't store the remote URI in the csync or in the SyncEngine
We are going to change the webdav path depending on the capabilities.
But the SyncEngine and csync might have been created before the capabilities
are retrieved.

The main raison why we gave the path to the sync engine was to pass it to csync.
But the thing is that csync don't need anymore this url as everything is done by the
discovery classes in libsync that use the network jobs that use the account for the urls.
So csync do not need the remote URI.

shortenFilename in folderstatusmodel.cpp was useless because the string is the
_file of a SyncFileItem which is the relative file name, that name never
starts with owncloud://.

All the csync test creates the folder because csync use to check if the folder
exists. But we don't need to do that anymore
2016-11-21 08:09:11 +01:00
ckamm
36d61ef3a9 Doc: Add sync algorithm overview and comments (#5277)
* Doc: Add sync algorithm overview and comments
2016-11-08 16:10:55 +01:00
Christian Kamm
f1f27221a7 Conflicts: Use the local mtime for the conflict file name #5273
Otherwise local conflict files may be overridden in a restore
situation. See ticket for details.
2016-11-08 14:47:51 +01:00
Olivier Goffart
d80d5a8ee4 SyncEngine: Fix renaming a folder should keep the selective sync state
Issue #5224

Two problems:

 - In the discovery phase, we need to check the selective sync entries of
   the source path in case of renames.

 - When the rename is done, we need to actually update the black list in the
   database.
2016-10-11 14:38:25 +02:00
Klaas Freitag
0c9dcdafc2 csync: Only free the statedb filename on destroy.
The filename must not be wiped in csync_commit as that is
happening after every sync. It is only set once in the
constructor of the SyncEngine in csync_init().
2016-09-29 16:36:14 +02:00
Christian Kamm
598a7cec5d Merge remote-tracking branch 'origin/2.2' 2016-09-28 12:07:45 +02:00
Christian Kamm
383479fbfd Fix Windows build with WITH_UNIT_TESTING 2016-09-22 13:55:48 +02:00
Jocelyn Turcotte
2b274e9460 Fix the MSVC build
- Replace functions that are provided by MinGW with a Win32-based
  implementation
- Explicitly export needed symbols from ocsync.dll
- Rename share.h to sharemanager.h since the name clashes with one
  of the Windows headers and get included from there
- Remove the timestamp from the fallback csync stderr logging, it's
  not used since we always provide a log callback
2016-09-16 17:43:39 +02:00
Christian Kamm
3e1690ff7a Improvements to file recall 2016-09-14 15:42:19 +02:00
Christian Kamm
4abaee7736 Recall: Copy instead of move recalled file #5150
That was an unintentional change in
2662203fb7

Also expand the test case to cover this.

(cherry picked from commit af9c4d0e2f)
2016-09-12 14:09:12 +02:00
Christian Kamm
af9c4d0e2f Recall: Copy instead of move recalled file #5150
That was an unintentional change in
2662203fb7

Also expand the test case to cover this.
2016-09-06 10:42:59 +02:00
Klaas Freitag
6fd930908c csync_tests: Adopted to new cmocka API. 2016-09-02 15:49:54 +02:00
Klaas Freitag
62125a442d csync_exclude: Also exclude the new syncjournal filename. 2016-09-02 15:49:21 +02:00
Klaas Freitag
2d54fb2ff9 csync_update: Do not check to exclude .csync_journal.db
It is indeed already handled by csync_exclude.
2016-09-02 15:48:45 +02:00
Jocelyn Turcotte
e974771796 csync: Use an explicit instruction for should_update_metadata
The current way of tracking the need to update the metadata without
propagation using a separate flag makes it difficult to track
priorities between the local and remote tree. The logic is also
difficult to logically cover since the possibilities matrix isn't
100% covered, leaving the flag only used in a few situations
(mostly involving folders, but not only).

The reason we need to change this is to be able to track the sync
state of files for overlay icons. The instruction alone can't be
used since CSYNC_INSTRUCTION_SYNC is used for folders even though
they won't be propagated. Removing this logic is however not possible
without using something else than CSYNC_INSTRUCTION_NONE since too
many codepath interpret (rightfully) this as meaning "nothing to do".

This patch adds a new CSYNC_INSTRUCTION_UPDATE_METADATA instruction
to let the update and reconcile steps tell the SyncEngine to update
the metadata of a file without any propagation. Other flags are left
to be interpretted by the implementation as implicitly needing
metadata update or not, as this was already the case for most file
propagation jobs. For example, CSYNC_INSTRUCTION_NEW for directories
now also implicitly update the metadata.

Since it's not impossible for folders to emit CSYNC_INSTRUCTION_SYNC
or CSYNC_INSTRUCTION_CONFLICT, the corresponding code paths in the
sync engine have been removed.

Since the reconcile step can now know if the local tree needs metadata
update while the remote side might want propagation, the
localMetadataUpdate logic in SyncEngine::treewalkFile now simply use
a CSYNC_INSTRUCTION_UPDATE_METADATA for the local side, which is now
implemented as a different database query.
2016-08-17 15:39:31 +02:00
Jocelyn Turcotte
b7ff4a76e8 Add TestSyncEngine and TestSyncFileStatusTracker auto tests
To be able to test the SyncEngine efficiently, a set of server
mocking classes have been implemented on top of QNetworkAccessManager.

The local disk side hasn't been mocked since this would require adding
a large abstraction layer in csync. The SyncEngine is instead pointed
to a different temporary dir in each test and we test by interacting
with files in this directory instead.

The FakeFolder object wraps the SyncEngine with those abstractions
and allow controlling the local files, and the fake remote state
through the FileModifier interface, using a FileInfo tree structure
for the remote-side implementation as well as feeding and comparing
the states on both side in tests.

Tests run fast and require no setup to be run, but each server feature
that we want to test on the client side needs to be implemented in
this fake objects library. For example, the OC-FileId header isn't
set as of this commit, and we can't test the file move logic properly
without implementing it first.

The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL
for what I esteem being issues that need to be fixed in order to catch
up on our test coverage without making this patch too huge.
2016-08-17 15:39:30 +02:00
Jocelyn Turcotte
2507ba9818 tx.pl: Fix each put_to_dir taking 10 seconds
The open function expects a URL, passing only the directory name would
lead HTTP::DAV to try looking it as an hostname on the network and
only return after it timed out.
2016-08-15 15:35:53 +02:00
Christian Kamm
eadecc5802 Excludes: Fix a Windows specific bug #5039
It meant that almost nothing was being excluded.

Broken since 87b4693a9d
2016-07-19 15:38:57 +02:00
Klaas Freitag
f3cfd2b70b csync: Free statedb file member before realloc 2016-07-17 21:13:17 +02:00
Olivier Goffart
7c671756e6 Merge remote-tracking branch 'origin/2.2'
Conflicts:
	src/libsync/syncfilestatustracker.cpp
2016-07-12 17:38:15 +02:00
Klaas Freitag
b9ea7c3414 csync: Do not compute the journal name in csync_update any more.
Rely on the name that was passed to csync_init()
2016-07-10 12:54:33 +02:00
ckamm
11b44358f6 Windows: Skip symlinks and junctions again #5019 (#5036)
Fixes an accidental behavior change introduced in
055c2ef73f

Affects #4056 and owncloud/enterprise#1225.
2016-07-07 08:07:19 +02:00
Christian Kamm
6e9df8673f Merge remote-tracking branch 'origin/2.2' 2016-06-22 15:41:37 +02:00
ckamm
32bb8295a2 csync: Use the full file stat destructors (#4992)
* For csync_file_stat_t and csync_vio_file_stat_t
* Add original_name to the vio file stat destructor
2016-06-21 12:06:21 +02:00
Klaas Freitag
191177ff22 excludes: Only log if the pattern was really logged. (#4989) 2016-06-20 15:50:20 +02:00
ckamm
43a2aec92b Performance: Don't redundantly add the same exclude files #4967 (#4988)
Excludes: Don't redundantly add the same exclude files #4967, Never accept duplicate exclude patterns #4967
2016-06-20 15:14:13 +02:00
Christian Kamm
87b4693a9d Add error message for trailing spaces on Windows #4878 2016-06-09 12:41:02 +02:00
Christian Kamm
6d02d76cfd Fix compile warnings in tests 2016-06-07 09:43:04 +02:00
Olivier Goffart
159535fe90 Merge branch '2.2' 2016-05-25 16:30:17 +02:00
Daniel Molkentin
eacc0c8bd6 Fix pot. mem leak found by coverity (CID 12900) 2016-05-24 11:21:33 +02:00
orion1024
9f3e576348
Adding CURL check in t6.pl
Aborts immediately if CURL not found, to avoid misleading errors later.
2016-04-30 11:44:16 +02:00
Klaas Freitag
6814342d1f Merge pull request #4748 from owncloud/win_stat_revisited
vio for windows: Use more graceful create file params.
2016-04-28 13:45:30 +02:00
Christian Kamm
68126dcff6 Checksums: Skip uploads of .eml files only #4754
This is a workaround. A correct solution would still update file metadata
such as the mtime. See #4755.
2016-04-28 12:59:39 +02:00
Christian Kamm
e7f00339e6 Fix perl tests #4722
* The dav auth realm is "sabre/dav" for newer server versions
* Content checksums are enabled generally, so conflict detection
  is smarter
2016-04-28 10:44:37 +02:00
Klaas Freitag
051a348afd Vio for windows: Removed the overlap param again.
After discussion and studying docs again we agreed that
the parameter is not neccessary here as it only affects
the process that opens the handle. And we do not want to
do async stuff.
2016-04-27 11:37:21 +02:00
Klaas Freitag
07bdd519e9 vio for windows: Use more graceful create file params. 2016-04-27 11:17:42 +02:00
Klaas Freitag
6ba7c6e2d3 Windows VIO: Remove unused variable. 2016-04-21 17:40:01 +02:00