Commit graph

234 commits

Author SHA1 Message Date
Christian Kamm
c5a0ce5a43 Selective sync: Skip excluded folders when reading db
When a new folder becomes selective-sync excluded, we already mark it
and all its parent folders with _invalid_ etags to force rediscovery.

That's not enough however. Later calls to csync_statedb_get_below_path
could still pull data about the excluded files into the remote tree.

That lead to incorrect behavior, such as uploads happening for folders
that had been explicitly excluded from sync.

To fix the problem, statedb_get_below_path is adjusted to not read the
data about excluded folders from the database.

Currently we can't wipe this data from the database outright because we
need it to determine whether the files in the excluded folder can be
wiped away or not.

See owncloud/enterprise#1965
2017-05-16 13:58:45 +02:00
Jocelyn Turcotte
879dadba78 Use QTemporaryDir in TestOwnSql
This will improve the cleanup and possibly help for #5366.
2017-05-08 23:18:06 +02:00
Christian Kamm
6abc2bf72d Transfers: Show a decimal for single digit MB rates #4428 2017-04-04 09:54:06 +02:00
Christian Kamm
4a1a5fa076 AbstractNetworkJob: Improve redirect handling #5555
* For requests:
  - reuse the original QNetworkRequest, so headers and attributes
    are the same as in the original request
  - determine the original http method from the reply and the request
    attributes
  - keep the original request body around such that it can be sent
    again in case the request is redirected

* Simplify the interface that is used for creating new requests in
  AbstractNetworkJob.
2017-03-07 13:18:01 +01:00
Olivier Goffart
f862c626a1 Propagator: Fix finished signal of directory being emited twice and causing crash
When there is a FatalError, we ended up emiting the finished signal for
the directory job several times, which would lead to crashes

Issue #5578
2017-03-06 16:14:04 +01:00
Jocelyn Turcotte
824be789bf Revert "Fix TestSyncEngine::abortAfterFailedMkdir"
This reverts commit 1cec2ca13d
but keeps the reset() -> take() part that is probably the cause
of crashes we've seen in the crash reporter.
2017-02-17 14:06:30 +01:00
Olivier Goffart
e8c10501a5 Folder: Keep files option after aboutToRemoveAllFiles should not clear selective sync list (#5531)
We were removing the wholme journal db when the user wanted to keep all files,
But that would also remove the selective sync lists.
We should only remove the metadata table.

Issue #5484
2017-02-15 13:27:41 +01:00
Olivier Goffart
172689d35c Merge remote-tracking branch 'origin/2.3' 2017-02-14 15:08:10 +01:00
Jocelyn Turcotte
60a4a742a3 Disable tests uncompatible with Qt4 for that build 2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
b26db062d2 Fix the UNIT_TESTING build on Windows
- Put all tests in the bin directory so that DLLs can be loaded
- Add missing exports
- Skip tests that use code depending on zlib
- The "GMT" timezone is named differently, use the int constructor instead

5 tests are still failing, it's not really worth fixing at the moment
since no developper is currently using Windows as its main platform.
2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
611f633ba8 Fix FolderManTest on macOS
On macOS /var is a symlink to /private/var and we have to make sure that we
use the canonical path before and after it enters the code to make sure we
compare paths correctly.
2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
3265948458 Fix ExcludedFilesTest on macOS
Use the same logic to find sync-exclude as check_csync_exclude.c
2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
0e3e9d5991 Fix FileSystemTest on macOS 2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
0be6cd3bf7 Fix FolderWatcherTest on macOS and Windows
- We need to use a QGuiApplication on macOS or else we don't get notifications
- Switch to use QSignalSpy rather than lists and sleeps
- Use system() for all modifications since we pass kFSEventStreamCreateFlagIgnoreSelf
- Keep using the local process on Windows since it catches its own events
2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
867b78052d Fix UtilityTest on macOS 2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
1cec2ca13d Fix TestSyncEngine::abortAfterFailedMkdir
It could be possible that _firstJob is marked as finished if
aborted before its parent PropagateDirectory was marked as finished,
allowing a posted scheduleNextJob call to schedule the child job
in-between.
2017-02-08 17:32:04 +01:00
Jocelyn Turcotte
1d9e08d109 Skip TestSyncEngine::abortAfterFailedMkdir 2017-02-08 17:25:32 +01:00
Markus Goetz
df773ea8bb CentOS: More build fixes 2017-02-08 11:39:28 +01:00
Markus Goetz
9614a94035 Revert "Disable RTTI"
This reverts commit 85d3de1589.
This breaks our old Qt on CentOS.

We can re-introduce this with #5470
2017-02-07 17:35:04 +01:00
Piotr Mrówczyński
c3ae5123cb Delete finished propagation jobs in PropagateDirectory #5269 (#5400) 2017-01-26 10:03:22 +01:00
Jocelyn Turcotte
1fc5a76622 Pass the SyncFileItem as SyncFileItemPtr in itemCompleted
This will allow us to keep a reference on the items in connected slots.
2017-01-25 23:26:23 +01:00
Jocelyn Turcotte
a764d7eb86 Don't pass the PropagatorJob in itemCompleted
This was to catch duplicate emissions for PropagateDirectory but we
don't emit this signal anymore from there.
This fixes a warning about PropagatorJob not being a registered metatype.

This reverts commit fe42c1a818.
2017-01-25 23:26:23 +01:00
Jocelyn Turcotte
605a18ff73 Tests: Add a large sync benchmark
This simulates a ~50k files sync that can be used to measure memory
usage without having to wait for a server.
2017-01-25 23:26:23 +01:00
Olivier Goffart
268fc97a71 Merge pull request #5480 from owncloud/chunking-ng-cleanup
Chunking ng cleanup
2017-01-25 19:19:48 +01:00
Olivier Goffart
d6fdda8efa ChunkingNG: add '0' padding on the filename (#5476)
The server sorts the chunk by name alphabetically. So if we want to keep
the chunk in order, we need to add a few '0' in front of the chunk name
2017-01-20 16:03:50 +01:00
Olivier Goffart
8a70d22af7 ChunkingNG: Add some tests
- Test that we recover correctly if the chunks were removed on the server.
 - Test changing the file localy while uploading.
2017-01-20 14:48:53 +01:00
Olivier Goffart
a63d970e5e ChunkingNG: remove stale chunks when cleaning the uploadInfo table
Stale chunks might be there because a file was removed or would just not
be uploaded, for any reason.
We just start the DeleteJob but we don't care if it success or not.

Relates to https://github.com/owncloud/core/issues/26981

One of the test is testing the case where the file is modified on the server
during the upload. So this test the precondition failed error.
The FakeGetReply logic was modified because resizing a 150MB big QByteArray
by increment of 16k just did not scale when downloading a big file.
2017-01-20 14:48:53 +01:00
Olivier Goffart
f6c77fad17 ChunkingNG: delete stale chunks if the file was changed locally
Relates to https://github.com/owncloud/core/issues/26981

We do not track the success or error of the DeleteJob because it does not
matter. If it fails, it might be because the chunks were already removed.
If not, the chunks will be stale, but the server must anyway do a few
cleanup from time to time because we do not always remove the chunks
2017-01-20 14:48:53 +01:00
Jocelyn Turcotte
b68a28de8d StatusTracker: Emit OK for the last child before parents (#5467)
The current logic tried to avoid a DB lookup just to fetch whether
the file is shared or not since that info is already in the
SyncFileItem. The implementation would however need to decrease the
sync count for itself (and parents) before emitting the new status,
thus emitting the OK status for parents before that last child that
ended the propagation for that folder.

Change the implementation to achieve what we want: give the
possibility to decSyncCount to use a pre-fetched sharing state while
still doing the emission for all involved files. This ensures that
the leaf file also gets its status emitted before its parents.

Issue #4797
2017-01-18 15:40:52 +01:00
Markus Goetz
85d3de1589 Disable RTTI
Shrinks owncloud binary by 24 KB and libowncloudsync by 14 KB.
I don't know if it has influence on memory usage or runtime speed though.

Was worth a try.
2017-01-17 14:52:10 +01:00
ckamm
d76e0ec6d8 Reset stuck chunked uploads eventually #5344 (#5443)
Previously this wasn't happening for errors that were not
NormalErrors because they don't end up in the blacklist.

This revises the resetting logic to be independent of the
error blacklist and make use of UploadInfo::errorCount
instead.

412 errors should reset chunked uploads because they might be
indicative of a checksum error.

Additionally, server bugs might require that additional
errors cause an upload reset. To allow that, a new capability
is added that can be used to advise the client about this.
2017-01-13 14:44:45 +01:00
Christian Kamm
6775292d63 Fix compile of tests 2017-01-03 10:52:39 +01:00
Christian Kamm
13e624c38f Fix compilation with Qt < 5.1
For review of #5045
2016-11-29 10:39:08 +01:00
Christian Kamm
49f8143f00 Bugfixes for sync journal name generation and usage
* Use 'user' value for journal name generation
* Save journal name in settings
* Make owncloudcmd choose the right db
2016-11-23 16:48:15 +01:00
Christian Kamm
9641c7a1e7 Rename sync journal to ._sync_xxx.db.
The added underscore means that older clients will also ignore the
file.
2016-11-23 16:48:15 +01:00
Christian Kamm
3e59a9b316 Merge branch 'master' into dbjournal_per_account 2016-11-23 16:47:56 +01:00
Olivier Goffart
a1dc4069c9 libsync: Don't store the remote URI in the csync or in the SyncEngine
We are going to change the webdav path depending on the capabilities.
But the SyncEngine and csync might have been created before the capabilities
are retrieved.

The main raison why we gave the path to the sync engine was to pass it to csync.
But the thing is that csync don't need anymore this url as everything is done by the
discovery classes in libsync that use the network jobs that use the account for the urls.
So csync do not need the remote URI.

shortenFilename in folderstatusmodel.cpp was useless because the string is the
_file of a SyncFileItem which is the relative file name, that name never
starts with owncloud://.

All the csync test creates the folder because csync use to check if the folder
exists. But we don't need to do that anymore
2016-11-21 08:09:11 +01:00
ckamm
ec7333a4bf Merge pull request #5272 from owncloud/licensefix-pending
License: Adjust license of GPLv2 source files to GPLv2+
2016-11-18 15:14:47 +01:00
Olivier Goffart
efa7821dd2 Revert "Discovery: consider also the "shared by me" as shared"
This reverts pull request 5313 (commit
2d6e473a40 and
2f3db04e87)

The problem is that this loads the server too much.
2016-11-16 17:45:40 +01:00
Markus Goetz
52552a4204 Merge pull request #5102 from owncloud/chunking-ng
Chunking ng
2016-11-15 15:56:52 +01:00
Olivier Goffart
2f3db04e87 Rename SyncFileStatus::sharedWithMe to shared
Last commit for issue #4788 made the meaning to be simply shared
(with the user or from the user)
2016-11-15 14:32:20 +01:00
Olivier Goffart
2d6e473a40 Discovery: consider also the "shared by me" as shared
The "S" in the permission is only for the "Shared with me" files.
It is only used to show the shared status in the overlay icons.
But we also wish to show the shared status for files that are shared
"by" the users. We can find that out using the 'share-types' webdav
property. If set, then we are sharing the object.
We fake a 'S' in the permission as for our purpose, they mean the same.

Issue #4788
2016-11-15 14:32:20 +01:00
Olivier Goffart
92027e8692 SyncEngineTestUtils: Do don't allocate a buffer for the whole file
As the file can be some hunreds of megabytes, allocating such big arrays may
cause problems.

Also make the timeout a bit bigger so the test can rununder valgrind.
2016-11-15 11:39:40 +01:00
Christian Kamm
b0c45cfc89 Test: Don't fail when DST change is near #5284 2016-11-08 11:13:19 +01:00
Olivier Goffart
15f2b911d9 ChunkingNG: remove stale files when resuming 2016-11-04 16:43:01 +01:00
Olivier Goffart
c8014a0afd ChunkingNG: Add Test 2016-10-31 15:16:53 +01:00
Christian Kamm
10644d3568 Move concatUrl and settingsWithGroup to Utility
There was little reason to keep them cluttering Account.
2016-10-25 12:05:28 +02:00
Christian Kamm
cf48ea2e00 Remove unused functions
Account::changed and AbstractCredentials::changed have not been needed
in a long while.
2016-10-25 11:33:38 +02:00
Christian Kamm
db24f60ae3 License: Adjust license of GPLv2 source files to GPLv2+
See #5180
2016-10-25 11:06:54 +02:00
Olivier Goffart
766d9ae1ac QProgressIndicator: update copy from upstream
Update from commit 05ce8a23cdc12e825532dc6de06c267fb8d48b4f from
https://github.com/dragotin/QProgressIndicator
Which itself is forked from commit e5ba0fd09bfd43b067ee3646d70b294c7efcb558 from
upstream, with additional license header.

It was relicensed to MIT according to
14bb9d10e2

Relates to issues #5180 and #5184
2016-10-13 12:05:59 +02:00