Commit graph

287 commits

Author SHA1 Message Date
Olivier Goffart
1ed4eb46f2 Merge remote-tracking branch 'origin/2.3'
Conflicts:
	VERSION.cmake
2017-04-13 10:19:35 +02:00
Christian Kamm
6abc2bf72d Transfers: Show a decimal for single digit MB rates #4428 2017-04-04 09:54:06 +02:00
Christian Kamm
f67989afea Wizards: Never propose an existing folder for syncing #5597 2017-03-21 17:07:35 +01:00
Olivier Goffart
1a98b49f1b Fix TestSyncEngine::testDirDownloadWithError
On master, the scheduling is different and the tasks abort in another
order
2017-03-09 10:35:58 +01:00
Olivier Goffart
0e8f0481d4 Merge remote-tracking branch 'origin/2.3'
Conflicts:
	src/libsync/owncloudpropagator.cpp
2017-03-09 09:54:14 +01:00
Christian Kamm
4a1a5fa076 AbstractNetworkJob: Improve redirect handling #5555
* For requests:
  - reuse the original QNetworkRequest, so headers and attributes
    are the same as in the original request
  - determine the original http method from the reply and the request
    attributes
  - keep the original request body around such that it can be sent
    again in case the request is redirected

* Simplify the interface that is used for creating new requests in
  AbstractNetworkJob.
2017-03-07 13:18:01 +01:00
Olivier Goffart
f862c626a1 Propagator: Fix finished signal of directory being emited twice and causing crash
When there is a FatalError, we ended up emiting the finished signal for
the directory job several times, which would lead to crashes

Issue #5578
2017-03-06 16:14:04 +01:00
Jocelyn Turcotte
ddb6b3b262 Fix TestSyncEngine::abortAfterFailedMkdir (again)
It could be possible that _firstJob is marked as finished if
aborted before its parent PropagateDirectory was marked as finished,
allowing a posted scheduleNextJob call to schedule the child job
in-between.
2017-02-23 17:54:33 +01:00
Jocelyn Turcotte
824be789bf Revert "Fix TestSyncEngine::abortAfterFailedMkdir"
This reverts commit 1cec2ca13d
but keeps the reset() -> take() part that is probably the cause
of crashes we've seen in the crash reporter.
2017-02-17 14:06:30 +01:00
Olivier Goffart
e8c10501a5 Folder: Keep files option after aboutToRemoveAllFiles should not clear selective sync list (#5531)
We were removing the wholme journal db when the user wanted to keep all files,
But that would also remove the selective sync lists.
We should only remove the metadata table.

Issue #5484
2017-02-15 13:27:41 +01:00
Olivier Goffart
172689d35c Merge remote-tracking branch 'origin/2.3' 2017-02-14 15:08:10 +01:00
Jocelyn Turcotte
60a4a742a3 Disable tests uncompatible with Qt4 for that build 2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
b26db062d2 Fix the UNIT_TESTING build on Windows
- Put all tests in the bin directory so that DLLs can be loaded
- Add missing exports
- Skip tests that use code depending on zlib
- The "GMT" timezone is named differently, use the int constructor instead

5 tests are still failing, it's not really worth fixing at the moment
since no developper is currently using Windows as its main platform.
2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
611f633ba8 Fix FolderManTest on macOS
On macOS /var is a symlink to /private/var and we have to make sure that we
use the canonical path before and after it enters the code to make sure we
compare paths correctly.
2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
3265948458 Fix ExcludedFilesTest on macOS
Use the same logic to find sync-exclude as check_csync_exclude.c
2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
0e3e9d5991 Fix FileSystemTest on macOS 2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
0be6cd3bf7 Fix FolderWatcherTest on macOS and Windows
- We need to use a QGuiApplication on macOS or else we don't get notifications
- Switch to use QSignalSpy rather than lists and sleeps
- Use system() for all modifications since we pass kFSEventStreamCreateFlagIgnoreSelf
- Keep using the local process on Windows since it catches its own events
2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
867b78052d Fix UtilityTest on macOS 2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
1cec2ca13d Fix TestSyncEngine::abortAfterFailedMkdir
It could be possible that _firstJob is marked as finished if
aborted before its parent PropagateDirectory was marked as finished,
allowing a posted scheduleNextJob call to schedule the child job
in-between.
2017-02-08 17:32:04 +01:00
Jocelyn Turcotte
1d9e08d109 Skip TestSyncEngine::abortAfterFailedMkdir 2017-02-08 17:25:32 +01:00
Markus Goetz
df773ea8bb CentOS: More build fixes 2017-02-08 11:39:28 +01:00
Markus Goetz
9614a94035 Revert "Disable RTTI"
This reverts commit 85d3de1589.
This breaks our old Qt on CentOS.

We can re-introduce this with #5470
2017-02-07 17:35:04 +01:00
Piotr Mrówczyński
c3ae5123cb Delete finished propagation jobs in PropagateDirectory #5269 (#5400) 2017-01-26 10:03:22 +01:00
Jocelyn Turcotte
1fc5a76622 Pass the SyncFileItem as SyncFileItemPtr in itemCompleted
This will allow us to keep a reference on the items in connected slots.
2017-01-25 23:26:23 +01:00
Jocelyn Turcotte
a764d7eb86 Don't pass the PropagatorJob in itemCompleted
This was to catch duplicate emissions for PropagateDirectory but we
don't emit this signal anymore from there.
This fixes a warning about PropagatorJob not being a registered metatype.

This reverts commit fe42c1a818.
2017-01-25 23:26:23 +01:00
Jocelyn Turcotte
605a18ff73 Tests: Add a large sync benchmark
This simulates a ~50k files sync that can be used to measure memory
usage without having to wait for a server.
2017-01-25 23:26:23 +01:00
Olivier Goffart
268fc97a71 Merge pull request #5480 from owncloud/chunking-ng-cleanup
Chunking ng cleanup
2017-01-25 19:19:48 +01:00
Olivier Goffart
d6fdda8efa ChunkingNG: add '0' padding on the filename (#5476)
The server sorts the chunk by name alphabetically. So if we want to keep
the chunk in order, we need to add a few '0' in front of the chunk name
2017-01-20 16:03:50 +01:00
Olivier Goffart
8a70d22af7 ChunkingNG: Add some tests
- Test that we recover correctly if the chunks were removed on the server.
 - Test changing the file localy while uploading.
2017-01-20 14:48:53 +01:00
Olivier Goffart
a63d970e5e ChunkingNG: remove stale chunks when cleaning the uploadInfo table
Stale chunks might be there because a file was removed or would just not
be uploaded, for any reason.
We just start the DeleteJob but we don't care if it success or not.

Relates to https://github.com/owncloud/core/issues/26981

One of the test is testing the case where the file is modified on the server
during the upload. So this test the precondition failed error.
The FakeGetReply logic was modified because resizing a 150MB big QByteArray
by increment of 16k just did not scale when downloading a big file.
2017-01-20 14:48:53 +01:00
Olivier Goffart
f6c77fad17 ChunkingNG: delete stale chunks if the file was changed locally
Relates to https://github.com/owncloud/core/issues/26981

We do not track the success or error of the DeleteJob because it does not
matter. If it fails, it might be because the chunks were already removed.
If not, the chunks will be stale, but the server must anyway do a few
cleanup from time to time because we do not always remove the chunks
2017-01-20 14:48:53 +01:00
Jocelyn Turcotte
b68a28de8d StatusTracker: Emit OK for the last child before parents (#5467)
The current logic tried to avoid a DB lookup just to fetch whether
the file is shared or not since that info is already in the
SyncFileItem. The implementation would however need to decrease the
sync count for itself (and parents) before emitting the new status,
thus emitting the OK status for parents before that last child that
ended the propagation for that folder.

Change the implementation to achieve what we want: give the
possibility to decSyncCount to use a pre-fetched sharing state while
still doing the emission for all involved files. This ensures that
the leaf file also gets its status emitted before its parents.

Issue #4797
2017-01-18 15:40:52 +01:00
Markus Goetz
85d3de1589 Disable RTTI
Shrinks owncloud binary by 24 KB and libowncloudsync by 14 KB.
I don't know if it has influence on memory usage or runtime speed though.

Was worth a try.
2017-01-17 14:52:10 +01:00
ckamm
d76e0ec6d8 Reset stuck chunked uploads eventually #5344 (#5443)
Previously this wasn't happening for errors that were not
NormalErrors because they don't end up in the blacklist.

This revises the resetting logic to be independent of the
error blacklist and make use of UploadInfo::errorCount
instead.

412 errors should reset chunked uploads because they might be
indicative of a checksum error.

Additionally, server bugs might require that additional
errors cause an upload reset. To allow that, a new capability
is added that can be used to advise the client about this.
2017-01-13 14:44:45 +01:00
Christian Kamm
6775292d63 Fix compile of tests 2017-01-03 10:52:39 +01:00
Christian Kamm
13e624c38f Fix compilation with Qt < 5.1
For review of #5045
2016-11-29 10:39:08 +01:00
Christian Kamm
49f8143f00 Bugfixes for sync journal name generation and usage
* Use 'user' value for journal name generation
* Save journal name in settings
* Make owncloudcmd choose the right db
2016-11-23 16:48:15 +01:00
Christian Kamm
9641c7a1e7 Rename sync journal to ._sync_xxx.db.
The added underscore means that older clients will also ignore the
file.
2016-11-23 16:48:15 +01:00
Christian Kamm
3e59a9b316 Merge branch 'master' into dbjournal_per_account 2016-11-23 16:47:56 +01:00
Olivier Goffart
a1dc4069c9 libsync: Don't store the remote URI in the csync or in the SyncEngine
We are going to change the webdav path depending on the capabilities.
But the SyncEngine and csync might have been created before the capabilities
are retrieved.

The main raison why we gave the path to the sync engine was to pass it to csync.
But the thing is that csync don't need anymore this url as everything is done by the
discovery classes in libsync that use the network jobs that use the account for the urls.
So csync do not need the remote URI.

shortenFilename in folderstatusmodel.cpp was useless because the string is the
_file of a SyncFileItem which is the relative file name, that name never
starts with owncloud://.

All the csync test creates the folder because csync use to check if the folder
exists. But we don't need to do that anymore
2016-11-21 08:09:11 +01:00
ckamm
ec7333a4bf Merge pull request #5272 from owncloud/licensefix-pending
License: Adjust license of GPLv2 source files to GPLv2+
2016-11-18 15:14:47 +01:00
Olivier Goffart
efa7821dd2 Revert "Discovery: consider also the "shared by me" as shared"
This reverts pull request 5313 (commit
2d6e473a40 and
2f3db04e87)

The problem is that this loads the server too much.
2016-11-16 17:45:40 +01:00
Markus Goetz
52552a4204 Merge pull request #5102 from owncloud/chunking-ng
Chunking ng
2016-11-15 15:56:52 +01:00
Olivier Goffart
2f3db04e87 Rename SyncFileStatus::sharedWithMe to shared
Last commit for issue #4788 made the meaning to be simply shared
(with the user or from the user)
2016-11-15 14:32:20 +01:00
Olivier Goffart
2d6e473a40 Discovery: consider also the "shared by me" as shared
The "S" in the permission is only for the "Shared with me" files.
It is only used to show the shared status in the overlay icons.
But we also wish to show the shared status for files that are shared
"by" the users. We can find that out using the 'share-types' webdav
property. If set, then we are sharing the object.
We fake a 'S' in the permission as for our purpose, they mean the same.

Issue #4788
2016-11-15 14:32:20 +01:00
Olivier Goffart
92027e8692 SyncEngineTestUtils: Do don't allocate a buffer for the whole file
As the file can be some hunreds of megabytes, allocating such big arrays may
cause problems.

Also make the timeout a bit bigger so the test can rununder valgrind.
2016-11-15 11:39:40 +01:00
Christian Kamm
b0c45cfc89 Test: Don't fail when DST change is near #5284 2016-11-08 11:13:19 +01:00
Olivier Goffart
15f2b911d9 ChunkingNG: remove stale files when resuming 2016-11-04 16:43:01 +01:00
Olivier Goffart
c8014a0afd ChunkingNG: Add Test 2016-10-31 15:16:53 +01:00
Christian Kamm
10644d3568 Move concatUrl and settingsWithGroup to Utility
There was little reason to keep them cluttering Account.
2016-10-25 12:05:28 +02:00
Christian Kamm
cf48ea2e00 Remove unused functions
Account::changed and AbstractCredentials::changed have not been needed
in a long while.
2016-10-25 11:33:38 +02:00
Christian Kamm
db24f60ae3 License: Adjust license of GPLv2 source files to GPLv2+
See #5180
2016-10-25 11:06:54 +02:00
Olivier Goffart
766d9ae1ac QProgressIndicator: update copy from upstream
Update from commit 05ce8a23cdc12e825532dc6de06c267fb8d48b4f from
https://github.com/dragotin/QProgressIndicator
Which itself is forked from commit e5ba0fd09bfd43b067ee3646d70b294c7efcb558 from
upstream, with additional license header.

It was relicensed to MIT according to
14bb9d10e2

Relates to issues #5180 and #5184
2016-10-13 12:05:59 +02:00
Klaas Freitag
27d23edacc Utility: Add a function to check if two filenames are equal plus test.
It calls canonical path always and works with the correct case preserving
depending on the platform.
2016-10-12 14:48:00 +02:00
Olivier Goffart
d80d5a8ee4 SyncEngine: Fix renaming a folder should keep the selective sync state
Issue #5224

Two problems:

 - In the discovery phase, we need to check the selective sync entries of
   the source path in case of renames.

 - When the rename is done, we need to actually update the black list in the
   database.
2016-10-11 14:38:25 +02:00
Olivier Goffart
e0a36ab313 TestFolderMan: don't pollute user's settings 2016-10-11 14:27:14 +02:00
Klaas Freitag
5d13f9290f Fix the folderman test, adopting the checkPathValidity method tests. 2016-10-07 16:24:09 +02:00
Olivier Goffart
85b8ab178e SyncEngine: Fix renaming of folder when file are changed (#5195)
Two bugs:
 - The change filed are not considered as move, they are re-downloaded
   but the old file was not removed from the database. The change in
   owncloudpropagator.cpp takes care of removing the old entries.

 - Next sync would then remove the file in the server in the old folder
   This was not a problem until we start reusing the sync engine, and
   that the _renamedFolders map is not cleared. We were before deleting
   a non-existing file. But now we delete the actual file.

Also improve the tests to be able to do move on the server.
This include support for file id.

Issue #5192
2016-09-22 09:02:47 +02:00
Jocelyn Turcotte
a221ac1e4d Fix unit testing build warning on Linux
The compiler complained about the PathComponent reference on the lambdas comes from the anonymous namespace.
2016-09-16 17:55:28 +02:00
Jocelyn Turcotte
b28123bed9 Fix the Qt4 unit testing build 2016-08-18 10:34:07 +02:00
Jocelyn Turcotte
efb6b8c2c9 [overlays] Fix folders appearing as OK even though children are still syncing #4797
This would happen if the directory would first need to be created
through an mkdir propagation job. This job's itemCompleted signal
would trigger the directory to show as SYNC even though its children
are still propagating.

Fix the issue by tracking the sync count for each file, affecting
its parents. This allows us to get rid of the O(n) vector lookup
for each status query, and properly track the hierachical sync
status of a directory.

This also removes the itemCompleted signal emission from the
PropagateDirectory job. Since we only needed for overlay icons, and
since this job doesn't do any direct propagation, we can remove it
to ensure that we won't call itemCompleted twice for the item attached
to Propagate*Mkdir jobs (since the PropagateDirectory is backed by
the same SyncFileItem, instruction and status).
2016-08-17 15:40:02 +02:00
Jocelyn Turcotte
82ef1bcfe0 Make sure that we invalidate parents on blacklisted items
Add a missing call that we currently only do in slotItemCompleted.
This would normally only affect the first sync and would have
gotten properly update at the end of the sync anyway.
2016-08-17 15:39:30 +02:00
Jocelyn Turcotte
b7ff4a76e8 Add TestSyncEngine and TestSyncFileStatusTracker auto tests
To be able to test the SyncEngine efficiently, a set of server
mocking classes have been implemented on top of QNetworkAccessManager.

The local disk side hasn't been mocked since this would require adding
a large abstraction layer in csync. The SyncEngine is instead pointed
to a different temporary dir in each test and we test by interacting
with files in this directory instead.

The FakeFolder object wraps the SyncEngine with those abstractions
and allow controlling the local files, and the fake remote state
through the FileModifier interface, using a FileInfo tree structure
for the remote-side implementation as well as feeding and comparing
the states on both side in tests.

Tests run fast and require no setup to be run, but each server feature
that we want to test on the client side needs to be implemented in
this fake objects library. For example, the OC-FileId header isn't
set as of this commit, and we can't test the file move logic properly
without implementing it first.

The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL
for what I esteem being issues that need to be fixed in order to catch
up on our test coverage without making this patch too huge.
2016-08-17 15:39:30 +02:00
Klaas Freitag
78caa1a712 Fix tests for syncjournaldb 2016-07-17 21:10:07 +02:00
Olivier Goffart
927a8b5071 Qt4: don't require a X server in the tests
Previous commit was not enough
2016-07-12 11:07:12 +02:00
Olivier Goffart
a4310f0f5c Qt4: don't require a X server in the tests 2016-07-11 14:04:58 +02:00
Christian Kamm
6d02d76cfd Fix compile warnings in tests 2016-06-07 09:43:04 +02:00
hefee
669a72f0e1 A final release version number breaks this regex (#4835)
f.ex. a valid verson nr: 2.2.0
-> fist \d+ = 2
-> second \d+ = 2
-> the last \d+ = 0
-> .+ in not matching, because there is nothing to match
2016-05-13 16:55:11 +02:00
Hefee
edb3759684 For tests we do not need a xserver running.
Use for tests QTEST_GUILESS_MAIN and QTEST_APPLESS_MAIN.
2016-05-12 11:34:21 +02:00
Christian Kamm
8934979ba1 timeAgo test: Fix for qt4, clean up 2016-05-12 09:25:15 +02:00
Klaas Freitag
9b1f46e560 Fix exclude unit test, adopt to new Iface of isExcluded. 2016-05-10 16:46:08 +02:00
Klaas Freitag
9e7a8e619b Fix utility test: Pass a command name to do the version check. 2016-05-10 16:26:42 +02:00
ckamm
e6b937f508 LockWatcher: Keep an eye on Windows file locks (#4758)
When a conflict-rename or a temporary-rename fails, notify the
LockWatcher. It'll regularly check whether the file has become
accesible again. When it has, another sync is triggered.

owncloud/enterprise#1288
2016-04-29 16:14:18 +02:00
Daniel Molkentin
e29d7e0128 Use QTokenizer to properly parse netrc
Addresses #4691
2016-04-21 12:46:03 +02:00
Christian Kamm
844777d43f Fix unit test build 2016-04-12 13:43:17 +02:00
Jocelyn Turcotte
f427955512 Simplify the build of auto tests
Remove all configure_files:
- Move all tests to cpp files
- Use the QTEST_MAIN macro instead of a generated main.cpp
- Include test*.moc in the cpp to let CMAKE_AUTOMOC call moc
- Pass info through add_definitions instead of generating oc_bin.h with them

This makes sure that build errors points to the original test source
file instead of the generated one in the build directory to be able to
jump and fix errors directly from the IDE's error pane.
2016-03-30 18:00:22 +02:00
Olivier Goffart
cdbc25ede8 Merge pull request #4615 from owncloud/socketApiRefactor
Socket API refactor
2016-03-30 13:24:10 +02:00
Christian Kamm
41f43feecf Add utility function for shorter time duration strings #3403 2016-03-29 12:26:09 +02:00
Christian Kamm
4915bbf8f3 Adjust size description strings #3403
@jancorchardt suggested to not have decimal places on file
sizes, except for GB.
2016-03-29 12:26:09 +02:00
Jocelyn Turcotte
df386b64ba Make the AccountState a construction argument of the Folder
This will help moving the SyncEngine construction in the constructor
and allow moving functionalities from Folder to SyncEngine or its
delegated objects.
2016-03-28 13:07:28 +02:00
Olivier Goffart
54612455e6 Merge pull request #4513 from owncloud/csyncContextToSyncEngine
Move the csync_context creation in SyncEngine
2016-03-08 18:12:31 +01:00
Christian Kamm
d521232587 AccountState: Allow storing state in settings
This will be useful if we ever want to store account-level gui state.
I built this originally because I thought a paused account would be
this kind of state.
2016-03-01 16:08:23 +01:00
Jocelyn Turcotte
75c99bf2b1 Fix TestXmlParse::testParser1 2016-02-25 20:53:13 +01:00
Klaas Freitag
e846b36bf6 Tests: Add a test for the timeAgoInWords function 2016-02-15 17:59:24 +01:00
Klaas Freitag
e3a846bb1f SyncJournalTest: Use valid time for the tests. 2015-11-25 22:16:29 +01:00
Klaas Freitag
aaa5c1bc5f Test syncjournaldb: Fix comparison of two QDateTime objects.
Only compare full seconds as milliseconds may float.
2015-11-25 21:13:50 +01:00
Christian Kamm
10db6cee6c Rename files 2015-11-23 12:09:25 +01:00
Christian Kamm
a25f094c4c Checksums: Don't reupload if size and checksum are unchanged #3235
* Compute the content checksum (in addition to the optional
  transmission checksum) during upload (.eml files only)

* Add hook to compute and compare the checksum in csync_update

* Add content checksum to database, remove transmission checksum
2015-11-23 11:59:56 +01:00
Christian Kamm
eb3388de68 Fix OwnSql test
3993a7f636 added asserts for failing
prepare() calls
2015-11-11 09:42:31 +01:00
Christian Kamm
25c177ca3b SyncJournalFileRecord: Remove unused mode field 2015-11-11 09:42:31 +01:00
Christian Kamm
17dd199cba Checksums: Treat more carefully in db #4034
In particular, preserve them on local rename or remote move.
2015-11-10 15:05:00 +01:00
ckamm
251679253a Merge pull request #3951 from ckamm/checksum
Checksums stored in database #3735
2015-10-29 10:40:24 +01:00
Christian Kamm
496b1e907d Checksum: Don't lose it on metadata update #3735
Also improve tests.
2015-10-28 14:49:55 +01:00
Christian Kamm
b7823dc648 Checksum: Put checksum type into separate column #3735 2015-10-28 14:46:20 +01:00
Markus Goetz
a6c9e8c5b4 Merge branch 'kill_legacy_propagator' 2015-10-28 11:06:50 +01:00
Christian Kamm
0354289795 Checksums: Improvement in async computation and validation #3735 2015-10-28 09:56:39 +01:00
Olivier Goffart
455c3ae57d test/CMakeLists.txt: remove comments 2015-10-22 12:32:53 +02:00
Olivier Goffart
9c388787bb csync_update: Don't fetch the etag in the local discovery from the DB
We don't need it, and it's slow.
This saves a lot of DB queries

(Also replaced a strlen>0 with a faster check)
2015-10-21 16:38:26 +02:00
Markus Goetz
c8590c4468 Remove legacy propagator and neon
The code was already uneeded/unbuilt on Windows and OS X.
2015-10-20 17:57:43 +02:00
Christian Kamm
6e09e3af86 Fix test compile
I regularly fix errors in the copies of the test sources that are
made in build/. Refactoring how these tests work could be worthwhile.
2015-10-15 15:52:58 +02:00