Commit graph

1400 commits

Author SHA1 Message Date
Tomaz Canabrava
3f4d915a17
[cse] Add files to handle client side encryption
This will be the responsible for encryption,
decryption and talking with the server.
2017-10-18 21:51:36 +02:00
Tomaz Canabrava
7e2085375f
[cse] Invalid read of 1 2017-10-18 21:51:36 +02:00
Tomaz Canabrava
1c85f94b00
[cse] Use server capabilities to query for cse 2017-10-18 21:51:36 +02:00
Tomaz Canabrava
920047fa70
[CSE] Remove uneeded includes 2017-10-18 21:51:36 +02:00
Tomaz Canabrava
5b51346e83
[CSE] New Network Job: DeleteApiJob
This network job does a DELETE http request on a URL. It's the
second class that does basically the same, but this one returns
the http return code, and it's set to do a api call.
2017-10-18 21:51:36 +02:00
Tomaz Canabrava
9870f39dcb
[CSE] Fetch file-id for subfolders
File id is a must if we want to call any API.

Signed-off-by: Tomaz Canabrava <tcanabrava@kde.org>
2017-10-18 21:51:36 +02:00
Tomaz Canabrava
9318c487b9
[CSE] Display menu for encrytp / decryot folders
If the server supports client syde encryption, display
a menu on right click that should display encrypt and decrypt.
ideally it would show the encrypt if the folder is decrypted, and
decrypt if the folder is encrypted but currently there's no way
for the client to know that.
2017-10-18 21:51:36 +02:00
Tomaz Canabrava
1c32f6bb2e
[CSE] Check the server for client side encryption
Call the apps api and check if the response has
the client_side_encryption key.

Signed-off-by: Tomaz Canabrava <tcanabrava@kde.org>
2017-10-18 21:51:36 +02:00
Christian Kamm
bf39343920 Sync: Add capability for invalid filename regexes #6092 2017-10-17 17:15:49 +02:00
Christian Kamm
b2a8ffc577 Abort: Fix crash with early aborts
_chunkCount could be 0, leading to a floating point exception

I also added initializers for several uninitialized integers in the
upload jobs.
2017-10-17 09:44:52 +02:00
Christian Kamm
e2711224ed Propagator: Avoid duplicate async abort 2017-10-17 09:44:52 +02:00
Christian Kamm
111bb485ec UploadNG: Avoid div-by-zero for super fast uploads 2017-10-17 09:44:52 +02:00
Piotr Mrowczynski
e10775d34f Fix paused sync file move issue #5949
Dont abort final chunk immedietally

Use sync and async aborts
2017-10-17 09:44:52 +02:00
Christian Kamm
f598ac89ac HttpCreds: Fix retry after wrong password #5989
This is an ugly solution.
2017-10-13 14:24:37 +02:00
Christian Kamm
01c2ffe2ae PropagateDownload: Read Content-md5 header #6088 2017-10-11 09:06:23 +02:00
Thomas Müller
315e38e814 Use display-name from the ocs call in the settings dialog 2017-10-06 10:59:20 +02:00
Markus Goetz
3a1a56c7ad Excludes: During directory traversal, use QRegularExpression
On Mac, this halves the time spent in csync_excluded_traversal
when using check_csync_excluded_performance. A similar performance
increase is seen on linux.
2017-10-05 10:19:48 +02:00
Olivier Goffart
800b9cf167 SyncEngine: _hasNoneFiles should be set when there are INSTRUCTION_UPDATE_METADATA 2017-09-28 11:15:36 +02:00
Olivier Goffart
d68b07dfc7 Fix encoding of version string 2017-09-27 12:06:49 +02:00
Dominik Schmidt
a2ce739192 Set CMAKE_INSTALL_RPATH as RPATH for installed executables 2017-09-27 09:13:44 +02:00
Christian Kamm
844bfc5f25 Move the --version output text to Theme #5957
Extracted and adjusted from packaging.diff by @jnweiger and @dschmidt
2017-09-27 09:13:44 +02:00
Olivier Goffart
13e0cc6b50 Use QDateTime::currentDateTimeUtc instead of the non utc version
QDateTime::currentDateTime is terribly slow.
2017-09-26 16:14:17 +02:00
Jocelyn Turcotte
54c522c861 SyncJournalDb: Reimplement the db_is_empty logic from csync
This reduces the initial sync local discovery time from
2.0 to 0.6 seconds in LargeSyncBench on my machine.
2017-09-26 11:03:23 +02:00
Jocelyn Turcotte
a034ee894c Use SyncJournalDb in csync
This gets rid of the csync_statedb sqlite layer and use
the same code and same connection as the rest of the SyncEngine.

Missing functions are added to SyncJournalDb and change a few minor
things (like changing SyncJournalFileRecord::_modtime to be an int64
instead of a QDateTime, like it was in csync).
2017-09-26 11:03:23 +02:00
Jocelyn Turcotte
6f46764daa SyncJournalDB: Allow callers of getFileRecord if the query failed
The current implementation would return the same value whether the query failed
or if no row would be found. This is something that is currently checked by csync
and needs to be provided if we want to use SyncJournalDB there.

Adjusted all call sites to also check the return value even though they
could still just rely on rec.isValid(), but makes it more explicit as to what
happens for database errors in those cases, if we ever want to gracefully handle
them.
2017-09-26 11:03:23 +02:00
Olivier Goffart
090abdee87 bandwidthmanager: Fix unregistering devices on delete
from the destroyed signal, qobject_cast won't work because the object
is already destroyed. One must use reinterpret_cast then
2017-09-25 12:56:20 +02:00
Olivier Goffart
ce04a76b17 PropagateUpload: Fix another non-normalized connect that appears while profiling 2017-09-25 12:49:55 +02:00
Olivier Goffart
0cec6f08ca OAuth2: Fix double slash in URL
We need to use concatPath to avoid possible double '/' in the URLs if the
account url() ends with '/'.

This has become even more of a problem since commit
d1b8370a4a which was resolving the url after
a redirect where most server actually add a '/' if the url is a folder
2017-09-23 10:10:40 +02:00
Christian Kamm
eae3f7ce0a SyncEngine: Adjust free space messaging
Before, the way the info message was phrased implied a failure. And
since the large numbers can be hard to compare that can be confusing.
2017-09-22 11:07:27 +02:00
Olivier Goffart
95d23b1914 RemotePermissions: Store in a class rather than in a QByteArray to save memory
Create a specific type that parses the permissions so we can store
it in a short rather than in a QByteArray

Note: in RemotePermissions::toString, we make sure the string is not
empty by adding a space, this was already existing before commit
e8f7adc7ca where it was removed by mistake.
2017-09-22 09:29:08 +02:00
Olivier Goffart
35e4fe061d Port to new signal-slot syntax what cannot be done automatically
Some slot were protected or private but needed to be public.
Some needed a static_cast (can't use qOverload because it is in Qt 5.7)

This is not only a partial change.
2017-09-21 14:05:39 +02:00
Olivier Goffart
02924ad83e DiscoveryPhase: use the new syntax in disconnect
Since we used the new syntax in connect, we need to use it in disconnect
2017-09-21 14:05:39 +02:00
Olivier Goffart
ff4213b59f Use the Qt5 connection syntax (automated with clazy)
This is motivated by the fact that QMetaObject::noralizeSignature takes 7.35%
CPU of the LargeSyncBench. (Mostly from ABstractNetworkJob::setupConnections and
PropagateUploadFileV1::startNextChunk). It could be fixed by using normalized
signature in the connection statement, but i tought it was a good oportunity
to modernize the code.

This commit only contains calls that were automatically converted with clazy.
2017-09-21 14:05:39 +02:00
Olivier Goffart
1218a38f2c Include <QAuthenticator> from code that connect to signal that uses it
The new connection syntax will need that
2017-09-21 14:05:39 +02:00
Christian Kamm
53195b064a Conflicts: Upload them files if env variable says so
Set OWNCLOUD_UPLOAD_CONFLICT_FILES=1 to trigger this behavior.

Note that this is experimental and unsupported. The real feature is
likely to end up in 2.5.

Uploading conflict files is simply done by removing the pattern from
csync_exclude. The rest here deals with making the conflict notification
ui approximately work.

There are still some concerns about where an uploaded conflict file
appears in the sync protocol and issues list (it should be in both, but
is only in one of them currently!).

See #4557.
2017-09-19 11:45:38 +02:00
Christian Kamm
dca2664707 Private links: Retrieve link through propfind property #6020
* The sharing ui does a propfind anyway: use that to query the new
property as well!
* For the socket api, asynchronously query the server for the right url
when an action that needs it is triggered.

The old, manually generated URL will be used as fallback in case the
server doesn't support the new property or the property can't be
retrieved for some reason.

Depends on owncloud/core#29021
2017-09-19 10:51:03 +02:00
Jocelyn Turcotte
2e90e9c35b Fix the build (make install) 2017-09-18 15:32:05 +02:00
Jocelyn Turcotte
a1f1775d15 Move SyncJournalDB to src/common 2017-09-18 14:00:52 +02:00
Jocelyn Turcotte
5fbed0d1cd Reverse the dependency between SyncJournalFileRecord and SyncFileItem
This will allow us to also use the SyncJournalDB in csync.
2017-09-18 14:00:52 +02:00
Jocelyn Turcotte
bd25225045 Move non-csync-dependent parts of FileSystem to src/common
This keep the csync-dependent parts in src/libsync, slitting the namespace
over two files.
This will allow moving SyncJournalDB to src/common as well.
2017-09-18 14:00:52 +02:00
Christian Kamm
78212e03d6 Checksums: Clearer behavior and added testing 2017-09-18 12:52:09 +02:00
Jocelyn Turcotte
a8ea7b0858 Remove SyncFileItem::_isDirectory
It's always equivalent to _type == SyncFileItem::Directory.
2017-09-18 11:46:09 +02:00
Jocelyn Turcotte
0e97770c47 Remove SyncFileItem::log
This remove the remaining "other" fields of the sync log to save a
bit of memory.

other_etag and other_fileId don't give much information to the users
and other_instruction will always be INST_NONE anyway.

other_modtime and other_size are kept since they are sometimes used.
They were renamed to have a bit more meaningful name.

SyncEngine::checkPermissions will now fetch its information from the
csync trees since they are now preserved until right after this point.

Fixes #3213
2017-09-18 11:46:09 +02:00
Jocelyn Turcotte
3d7bd7583b Remove SyncEngine::_remotePerms
Now that csync is using a more convenient data structure for
its file trees, wait a little bit longer before destroying them and
fetch the remote permissions from the remote tree there instead.
2017-09-18 11:46:09 +02:00
Jocelyn Turcotte
d66c2b5fae Replace the custom rbtree by a std::map 2017-09-18 11:46:09 +02:00
Christian Kamm
be5d5aea6d AbstractNetworkJob: Allow relative redirect
Previously all redirects were considered absolute.
2017-09-15 15:25:10 +02:00
Christian Kamm
8635b8ac84 Reduce timeout for some admin jobs
The oauth token jobs and the wizard redirect check job shouldn't have
5min timeouts.
2017-09-15 15:25:10 +02:00
Christian Kamm
1f57fd0407 HttpCreds: Remove migration code from 1.7 2017-09-15 09:29:05 +02:00
Christian Kamm
671599c8b2 Credentials: Use per-account keychain entries #5830
This requires a lot of migration code: the old entries need to be read,
saved to the new locations and then deleted.
2017-09-15 09:29:05 +02:00
Christian Kamm
0b4fd52d63 Journal: Second attempt at journal mode fallback #5723
Some filesystems, vms or other limitations make using the WAL journal
mode impossible. We are notified of this problem through an sqlite
IOERR for SHMMAP. In that case We want to attempt to fall back to the
DELETE journal mode.
2017-09-15 09:28:35 +02:00
Christian Kamm
e05f5fc50d OAuth: Don't use implicit POST bodies
The query args of POST requests become the request body. If there's a
redirect, the redirected url will therefore not contain the query
arguments. Use an explicit request body to make the redirection work.
2017-09-15 09:28:03 +02:00
Christian Kamm
7d075cdcb7 OAuth: Use redirectable jobs for oauth token management 2017-09-15 09:28:03 +02:00
Christian Kamm
3a55ff2119 PropagateUpload: Model of remote quota, avoid some uploads #5537
When we see a 507 error, assume that quota is < uploaded size.
2017-09-15 09:27:07 +02:00
Christian Kamm
b5b055e1ae SyncOptions/Propagator: Parallelism as an option
Very useful for changing it on a per-test basis.
2017-09-15 09:27:07 +02:00
Christian Kamm
0c99673581 Make DetailError different from BlacklistedError
It's quite different in regard to blacklist handling and overall
sync failure changes.
2017-09-15 09:27:07 +02:00
Helmut K. C. Tessarek
709aa27031 remove qt4 code 2017-09-15 07:11:05 +02:00
Christian Kamm
70aafd74a8 Journal: Fall back to DELETE on IO error #5723
Also add logging of extended error codes for this IO error, maybe we can
become more specific about which situations should trigger a journal
mode switch.
2017-09-12 14:12:24 +02:00
Christian Kamm
09173fb727 Update server url in case of permanent redirection #5972
This is the first time the account url may update outside of
account setup.

Summary of redirection handling:
1. During account setup (wizard)
   - status.php gets permanently redirected -> adjust url
   - authed PROPFIND gets *any* redirection -> adjust url
2. During connectivity ping (ConnectionValidator)
   - status.php gets permanently redirected -> adjust url (new!)

All other redirections should be followed transparently and
don't update the account url in the settings.
2017-09-11 19:27:36 +02:00
Christian Kamm
de5de6284c Use DetermineAuthTypeJob in HttpCredentials
* Move it to networkjobs
* Minor adjustments to its logic
* Fixes redirect handling for oauth/basic http auth check #6003
2017-09-11 19:15:43 +02:00
Jocelyn Turcotte
715b8c4683 Fix build after C++ conversion of csync
Use the same macro as in SyncFileItem for bitfields on enums.

Fixes #5993
2017-09-06 13:08:23 +02:00
Jocelyn Turcotte
b1bb7ec1a8 Use a C++ contructor and destructor for CSYNC
Merge csync_create and csync_init into the constructor and
replace csync_destroy with the destructor.

Also use a QByteArray for csync_s::root_perms and flatten
csync_rename_s as a rename sub-struct of csync_s since it
can now handle C++ types.
2017-09-06 13:08:23 +02:00
Jocelyn Turcotte
cb49635231 Replace TREE_WALK_FILE with csync_file_stat_t
Just expose csync_file_stat_t since we don't need an abstraction layer
anymore. Also pass the nodes of both trees directly to the visitor
function.

Issue #1817
2017-09-06 13:08:23 +02:00
Jocelyn Turcotte
e8f7adc7ca Replace csync_vio_file_stat_t with csync_file_stat_t
Also move csync_normalize_etag to common/utility since we
don't need the char* function anymore.

Remove the single space file_stat->remotePerm codepath since
this won't be used in csync anymore since
8de3bda0b1.

Issue #1817
2017-09-06 13:08:23 +02:00
Jocelyn Turcotte
704ca366b3 Make csync_file_stat_t public and partly convert to C++
This is the first commit trying to unify csync_file_stat_s,
csync_vio_file_stat_s and csync_tree_walk_file_s. Use QByteArray
and unique_ptr already since I'm not used to track memory allocations
and this will make the transition easier.

Issue #1817
2017-09-05 17:25:19 +02:00
Jocelyn Turcotte
cf15cbf0b3 Move Utility to a new common static library
Now that csync builds as C++, this will avoid having to implement
functionalities needed by csync mandatorily in csync itself.

This library is built as part of libocsync and symbols exported
through it.
This requires a relicense of Utility as LGPL. All classes moved into
this library from src/libsync will need to be relicensed as well.
2017-09-05 17:25:19 +02:00
Jocelyn Turcotte
48d2fc1599 StatusTracker: Fix different case paths not matching (#5981)
Use a custom std::map comparator functor to do all comparisons
on contained QStrings using Qt::CaseInsensitive on macOS and Windows.

Issue #5257
2017-08-31 13:32:00 +02:00
Jocelyn Turcotte
4e5898efd7 Compile almost all of csync as C++
This will allow us to unify data structures between csync and libsync.

Utility functions like csync_time and c_std are still compiled as C
since we won't need to be coupled with Qt in the short term.
2017-08-22 15:22:56 +02:00
Jocelyn Turcotte
28a0971cdf Move csync to src/csync 2017-08-22 11:18:22 +02:00
Jocelyn Turcotte
5c9827a33c Move csync/tests to test/csync 2017-08-22 11:18:22 +02:00
Jocelyn Turcotte
251f1d0047 Windows: Use the application icon for the sidebar
By setting the icon in Desktop.ini of the root folder, this adds the icon
both when browsing the folder directly and to the sidebar shortcut.

To avoid overwriting any user setting that could exist in Desktop.ini,
only do this if the file doesn't exist. Editing .ini files on Windows
isn't trivial and isn't worth it given that this file won't exist most
of the time.

Fixes #2446
2017-08-15 13:37:33 +02:00
Markus Goetz
4aad318446 Upload: Log the checksum / file hash 2017-08-04 15:42:11 +02:00
Olivier Goffart
6ae88514d8 OAuth: clear refresh token when the server claim not to support oauth
Allow upgrade path when the server removes support for oauth
Relates: https://github.com/owncloud/client/issues/5848#issuecomment-317353049

We also need to force the account to commit the config to the disk,
otherwise we may not register we are no longer using owncloud and we
risk sending the password as the token to the token refresh API call
2017-07-25 12:34:13 +02:00
Olivier Goffart
c043840cb1 OAuth: Fix refresh of token after expiration
Before commit d3b00532b1,
fetchFromKeychain was called everytime we detect that the creds are
invalid (in AccountState::slotInvalidCredentials)
But since that commit, AccountState was calling askFromUser directly,
breaking the refresh of the token.

So I made sure AccountState::slotInvalidCredentials still calls
refreshAccessToken.

Another change that was made was too be sure to clear the cookies
in HttpCredentials::invalidateToken even when we are only clearing the
access_token. That's because the session with a cookie may stay valid
longer than the access_token
2017-07-25 12:34:13 +02:00
Olivier Goffart
a831164d65 Revert "Revert "Discovery: consider also the "shared by me" as shared""
This reverts commit efa7821dd2.

This reverts the revert, but also add a check that the server version
is bigger than 10.0

Issue #4788
2017-07-25 12:11:33 +02:00
Jocelyn Turcotte
a41dc00160 Don't keep the list of touched files for the whole sync
We only want to know if they were touched within the last 15 seconds,
so change the data structure to use a QMultiMap, and sort them by
QElapsedTimer. This allows us to iterate over old entries ordered by
time and to stop once we find a recent entry.

This makes the look-up slower but in most cases the folder watcher
will report any change within milliseconds, and we start from the
most recent. What this really makes slower are actual user file
changes while a fast sync is underways which will need to iterate
over the whole map to find out the file isn't there.

This reduces the growth of the memory usage when downloading a large
amount of files.
2017-07-24 17:54:29 +02:00
Jocelyn Turcotte
5a1bf7d8fe windows: Fix a memory leak in FileSystem::longWinPath
Also use c_path_to_UNC directly instead of doing an extra
UTF-8->wchar conversion.
2017-07-24 17:54:29 +02:00
Olivier Goffart
d9813dbc90 Propagator: fix a qMax which should have been a qMin 2017-07-18 08:58:07 +02:00
Olivier Goffart
3cea550d08 PropagateUpload: Remove the Qt workaround against data corruption
This was only for Qt <= 5.4, which we don't really support anymore.
And even then, the data corruption is happens very seldomly anyway.
2017-07-17 10:12:48 +02:00
Olivier Goffart
b1363d1a79 AccessManager: Disable HTTP/2 without TLS
Qt would otherwise still try to do HTTP/2 connection even over "http://".
And that does not work with server that does not support it
2017-07-17 08:20:17 +02:00
Olivier Goffart
520923b5a7 HTTP/2 Support
We need Qt 5.9 for HTTP2 because, even if Qt 5.8 already has support
for it, there is some critical bug in the HTTP2 implementation which
make it unusable [ https://codereview.qt-project.org/186050 and
https://codereview.qt-project.org/186066 ]

When using HTTP2, we can use many more parallel network request, this
is especially good for small file handling

Lower the priority of the GET and PUT propagation jobs, so the quota
or selective sync ui PROPFIND will not be blocked by them
2017-07-17 08:20:17 +02:00
Markus Goetz
1fb68e8711 X-Request-ID: Send per request not per account #5853 2017-07-14 09:14:01 +02:00
Christian Kamm
1c2d5f16c8 Account: Send a X-Request-ID header #5853 2017-07-13 18:06:40 +02:00
Christian Kamm
2cdf5517cb Conflicts: Detect and show in issues tab
Incidentally fixes a potential issue where conflicts were silently-
ignored and thus deleted if the parent folder was deleted.
2017-07-13 11:48:10 +02:00
Christian Kamm
da54b3d7e5 Merge remote-tracking branch 'origin/2.3' 2017-07-12 10:06:04 +02:00
Christian Kamm
971abaea80 IssuesWidget: Add button to retry 507 errors #5537
Since these errors are blacklisted, it can take up to 24h to retry items
that had a 507 error for a while. This way users can intervene and cause
an upload attempt immediately.
2017-07-12 09:04:27 +02:00
Christian Kamm
cd1b89475c PropagateUpload: Better messaging for 507 #5537
It now produces a summary error message indicating the problem.

Adjust blacklist database table to contain 'errorCategory'. This is
useful for two things:
  - Reestablishing summary messages based on blacklisted errors. For
    example if we don't retry a 507ed file, we still want to show the
    message about space on the server
  - Selectively wiping the blacklist: When we have ui for something like
    "I deleted some files, please retry all files now!", we want to
    delete all blacklist entries of a specific category only.
2017-07-12 09:04:27 +02:00
Christian Kamm
5d90b48790 PropagateUpload: Put upload error handling in one function 2017-07-12 09:04:27 +02:00
Christian Kamm
c3cb186952 ProgressInfo: Carry a sync status
* A bunch of code was determining sync status by ad-hoc comparing some
  progress info fields. It can now just check the status, making it
  easier to comprehend.
* There's a clear indication for "a new sync is starting", which helps
  wiping the issues tab at the right time.
2017-07-12 09:04:27 +02:00
Christian Kamm
5ca743dd25 SyncEngine: Introduce overall errors that are not tied to a file #5746
For now we use them for:
* csync errors: This allows them to appear in the sync issues tab
* insufficient local disk space, as a summary of individual file errors

Insufficient remote space will use them too, as might other issues that
are bigger than a single sync item.
2017-07-12 09:04:27 +02:00
Christian Kamm
a5ace5e71d Account/Credentials: Have identical lifetimes
The QNAM may continue to outlive both.

Rename Credentials::getQNAM() to createQNAM() while we're at it - it's
used to make a new QNAM that will subsequently be owned by the Account
object.

See d01065b9a1 for rationale.

Relates to
d40c56eda5
147cf798a6
2017-07-08 13:07:13 +02:00
Christian Kamm
e54be1c4ee Link shares: Add "show file listing" option #5837 2017-07-07 11:23:32 +02:00
Christian Kamm
0238a29c7c Introduce private link sharing #5023
* SocketAPI has COPL_LOCAL_LINK / EMAIL_LOCAL_LINK commands
* The nautilus and dolphing shell integrations show a submenu from which
  one can share as well as access the private link.
* The SocketAPI provides a new GET_STRINGS command to access localized
  strings.
* The private link can also be accessed from the user/group sharing
  dialog.
* The numeric file id is extracted from the full id to create the
  private link url.
2017-07-07 10:49:51 +02:00
Christian Kamm
d01065b9a1 Fix crash on account deletion
Calling forgetSensitiveData() on account deletion leads to a timer for
clearQNAMCache() being queued. Then the Account object is deleted. The
Credentials object stays alive for now because it has a deleteLater
deleter.

If the timer calls into a slot on the Credentials object, the _account
pointer will be invalid at this time.

As a workaround, move the target slot to Account - that way it will not
be called as the account object is already destroyed.

However since Account and Credentials are mutually dependent, it would
be much preferable if their lifetimes were linked, avoiding this
category of bugs.

The current behavior was introduced in
d40c56eda5 and I currently don't
understand why - maybe there's another way of dealing with the problem
that existed then.
2017-07-06 14:34:02 +02:00
Christian Kamm
b1aaf055b1 Blacklist: Don't let errors become warnings #5516
Before, blacklisted errors were set to FileIgnored status and hence
displayed as warnings. Now, they have their own BlacklistedError
category which allows them to appear as errors in the issues list and in
the shell integration icons.
2017-07-04 13:07:51 +02:00
Markus Goetz
f4d1a07b72 SyncEngine: Keep local size in SyncItem #5855 2017-07-04 11:42:44 +02:00
Piotr Mrowczynski
d1e00099dc Classify chunked items correctly. Issue #5850 2017-07-03 14:41:53 +02:00
Samuel Alfageme
aeea27b57b Update central.owncloud.org link
Desktop client category was renamed, old link was 404 - see: https://central.owncloud.org/t/new-central-categories-you-decide
2017-06-29 17:54:42 +02:00
Samuel Alfageme
0f70bd7913 Update central.owncloud.org link
Desktop client category was renamed, old link was 404 - see: https://central.owncloud.org/t/new-central-categories-you-decide
2017-06-29 17:52:00 +02:00
Christian Kamm
15ee7b39ac Merge remote-tracking branch 'origin/2.3' 2017-06-22 14:06:31 +02:00
Christian Kamm
4291ea47f7 Journal: Don't use a ._ path if it won't work #5633
When synchronizing a folder on a samba share, creating files that begin
with ._ is often forbidden. This prevented the client from creating
its ._sync_abcdef.db file.

Now, it'll check whether the preferred filename is creatable, and if
it isn't it'll use .sync_abcdef.db instead.

The disadvantage is that this alternative path won't be ignored by
older clients - that was the reason for the ._ prefix.
2017-06-20 13:35:34 +02:00
Jocelyn Turcotte
5d24a2dc0f Remove debug messages by default from sync.engine
Other categories were already set properly.
2017-06-15 20:10:30 +02:00
Christian Kamm
8160963110 Compare the hash of files with identical mtime/size #5589
* For conflicts where mtime and size are identical:

  a) If there's no remote checksum, skip (unchanged)
  b) If there's a remote checksum that's a useful hash, create a
     PropagateDownload job and compute the local hash. If the hashes
     are identical, don't download the file and just update metadata.

* Avoid exposing the existence of checksumTypeId beyond the database
  layer. This makes handling checksums easier in general because they
  can usually be treated as a single blob.

  This change was prompted by the difficulty of producing file_stat_t
  entries uniformly from PROPFINDs and the database.
2017-06-15 13:54:16 +02:00
Olivier Goffart
5100a2daf1 OAuth: cleanup debug messages
- Add category to the all messages (they did not have it was merged right after
the patch to add category everywhere, but this code did not have it.)

- Make sure there is no warnings in the normal flow. (The wizard does a request
without authentication to determine the auth type)
2017-06-13 12:08:24 +02:00
Olivier Goffart
6e57b0219e Don't link dirrectly against openssl
All our crypto code is handled by qt nodaways.
No need to carry this dependency.

Especially since it causes warnings on system where there are
twp openssl version installed:
/usr/bin/ld: warning: libcrypto.so.1.0.0, needed by /usr/lib/libQt5Network.so.5.9.0, may conflict with libcrypto.so.1.1
2017-06-13 11:27:23 +02:00
Olivier Goffart
3314a472c3 SyncEngine: SelectiveSync: Remove local files of undelected folder despite other modified files
Issue #5783

When the directry that should be removed by selective sync contains changes,
we ignore the whole sub tree instead of only ignoreing new files.
We cannot ignore the whole directory, we need to ignore only the directory
that do not have files to remove
2017-06-08 14:53:54 +02:00
Christian Kamm
aeeb408c74 Merge branch '2.3' 2017-06-08 13:02:34 +02:00
Olivier Goffart
18c39197e5 ownsql: fix build on jenkins
sqlite3_system_errno was only added in sqlite 3.12

(cherry picked from commit e2e56f45af)
2017-06-08 12:53:45 +02:00
Christian Kamm
7eb8219419 Add extra logging for SQLITE_CANTOPEN errors #5633
(cherry picked from commit 05c1dcef9b)
2017-06-08 12:52:58 +02:00
Olivier Goffart
63c40b45e4 OAuth: Fix the secret to the one from commit 05fd985f9f in the oauth2 repository
05fd985f9f
2017-06-06 14:49:34 +02:00
Olivier Goffart
e2e56f45af ownsql: fix build on jenkins
sqlite3_system_errno was only added in sqlite 3.12
2017-06-06 14:47:11 +02:00
Christian Kamm
05c1dcef9b Add extra logging for SQLITE_CANTOPEN errors #5633 2017-06-06 12:59:57 +02:00
Christian Kamm
cc5fe558ba Fix maintenance mode detection #5755
The server sends "maintenance": "true" which isn't a json boolean...
2017-06-06 10:45:41 +02:00
Olivier Goffart
3d93527a8e Authentication with OAuth2
When the OAuth2 app (https://github.com/owncloud/oauth2) is enabled,
We will open a browser and perform the OAuth2 authentication

Issue: #4798 and https://github.com/owncloud/platform/issues/17
2017-06-01 10:39:33 +02:00
Christian Kamm
577a2715bd Download: Retrigger folder discovery on 404
See owncloud/enterprise#1966

If the server and the client's database go out of sync, there could be
persistent 404 errors. This change ensures that the problem corrects
itself eventually by triggering a remote discovery of the file's
parent folders.

It does not address the root cause that might have lead to the
divergence.
2017-05-29 21:41:10 +02:00
Olivier Goffart
f88431adbf Don't call QElapsedTimer::restart on uninitialized timer
That's an undefined behavior.
Since we don't use the return value anyway, we should just use start()
2017-05-29 10:51:05 +02:00
Christian Kamm
d3b00532b1 Credentials: Simplify credential flow #5728
And as a side effect: don't ask for user password when we can't
connect to the server in the first place.
2017-05-22 10:52:18 +02:00
Christian Kamm
5eaae83a01 Merge remote-tracking branch 'origin/2.3' 2017-05-17 13:10:19 +02:00
Christian Kamm
c8d0f788e0 Apply clang-format 2017-05-17 12:26:27 +02:00
Christian Kamm
ae263d60bd Reformatting: Reformat some lines that would become really long 2017-05-17 12:26:27 +02:00
Christian Kamm
df3fe25702 Reformatting: Adjust trailing comments
These would otherwise be line-wrapped by clang-format,
and then consecutive reformattings remove the aligned
comment indentation

Example:

int a; // too long comment

->

int a; // too long
       // comment

->

int a; // too long
// comment
2017-05-17 12:26:27 +02:00
Christian Kamm
c5a0ce5a43 Selective sync: Skip excluded folders when reading db
When a new folder becomes selective-sync excluded, we already mark it
and all its parent folders with _invalid_ etags to force rediscovery.

That's not enough however. Later calls to csync_statedb_get_below_path
could still pull data about the excluded files into the remote tree.

That lead to incorrect behavior, such as uploads happening for folders
that had been explicitly excluded from sync.

To fix the problem, statedb_get_below_path is adjusted to not read the
data about excluded folders from the database.

Currently we can't wipe this data from the database outright because we
need it to determine whether the files in the excluded folder can be
wiped away or not.

See owncloud/enterprise#1965
2017-05-16 13:58:45 +02:00
Jocelyn Turcotte
7fd2f292e7 Promote a few more logs to info/warning 2017-05-11 17:22:59 +02:00
Jocelyn Turcotte
cf058bc537 Add the possiblility to enable debug logging categories
Add a checkbox in the log window as well as a --logdebug command-line
option that should have the same effect.

Issue #5647
2017-05-11 17:22:59 +02:00
Jocelyn Turcotte
cc9680c1e8 Also pass csync logs through QMessageLogger
This allows using logging rules and custom message patterns with csync.
This also pass the a relative verbosity to the message logger.

Issue #5647
2017-05-11 17:22:59 +02:00
Jocelyn Turcotte
b7553d5bdf Upgrade some qCDebug to qCInfo or qCWarning
Use qCInfo for anything that has general value for support and
development. Use qCWarning for any recoverable error and qCCritical
for anything that could result in data loss or would identify a serious
issue with the code.

Issue #5647
2017-05-11 17:22:59 +02:00
Jocelyn Turcotte
4ad190a558 Use Qt logging categories for logging
This gives more insight about the logs and allow setting fine-tuned
logging rules. The categories are set to only output Info by default
so this allows us to provide more concise logging while keeping the
ability to extract more information for a specific category when
developping or debugging customer issues.

Issue #5647
2017-05-11 17:22:59 +02:00
Jocelyn Turcotte
3a193655b3 Improve the logging message pattern
Add the log level and category name in the output. Only output the
thread ID and function name for qCDebug statements as they are not
necessary for general use and make the log harder to read.

Also make sure that the message pattern is set when NO_MSG_HANDLER is
used. Using an environment variable should have priority over it anyway.
2017-05-11 17:22:59 +02:00
Christian Kamm
4b6edaa01a Detect maintenance mode #4485
When we first detect a 503 (probably from a PROPFIND) and enter the
ServiceUnavailable state, we new trigger a status.php query that will
switch the state to MaintenanceMode if necessary.
2017-05-11 11:39:49 +02:00
Jocelyn Turcotte
4d3212b31f Merge branch 2.3 into master 2017-05-09 18:23:01 +02:00
Christian Kamm
3da1e9a4b9 Remove 3rdparty qjson #5710 2017-05-08 11:50:33 +02:00
Christian Kamm
f1b42cedda Port PollJob responses to Qt5's QJson #5710 2017-05-08 11:50:33 +02:00
Christian Kamm
2598579d84 Switch JsonApiJob to Qt5's QJson #5710 2017-05-08 11:50:33 +02:00
Olivier Goffart
13705999f7 Discovery: Increase the MAX_DEPTH and show deep folder as ignored
Before this patch, to deep folder would just be ignored, without any feedback.
This patch makes it so deep folder are properly shown as ignored in the UI.

Also increase the MAX_DEPTH

Issue: #1067
2017-05-08 11:29:51 +02:00
Christian Kamm
aff8008baf Downloads: Remove empty temporary if disk space full #5746 2017-05-04 15:12:14 +02:00
Christian Kamm
c922015521 SSL error handling: Remove unnecessary confusing code
I'm confident this is unnecessary. The original bug in #3283 was
to call ignoreSslErrors() without an argument in the 'accept'
case, which meant ignoring *all* subsequent SSL errors.

With that fixed, explicitly aborting the reply and resetting QNAM
is not needed since not ignoring the error will lead to the SSL
handshake failing.

See also:
  75b38d1a2f (workaround introduced)
  89376e14d6 (real fix)
  76ce5adbf0 (cherry-pick of workaround)
2017-05-04 09:19:01 +02:00
Olivier Goffart
bb244d527c Better words for the error message when one cannot create the database.
Issue #5291
2017-05-03 17:53:46 +02:00
Luca Lorenzetto
3db8d3dfb6 When creating explorer favorite use more specific windows functions (#5690)
Fixes https://github.com/owncloud/client/issues/2719
2017-04-27 10:34:07 +02:00
Tommy Sparber
72a3e08907 Move check for mono icons into separate function 2017-04-26 17:39:01 +02:00
Christian Kamm
f8c0e796df SqlDatabase: Ask freeSpace for directory, fix for Windows
See owncloud/enterprise#1955
2017-04-26 12:38:04 +02:00
Christian Kamm
cd4e647816 Merge remote-tracking branch 'origin/2.3' 2017-04-24 10:32:53 +02:00
Kiril Vladimiroff
3a31694147 Remove all conditional checks for Qt 4.x (#5505) 2017-04-21 18:13:32 +02:00
Christian Kamm
05236f0a7a Sharing multiple public links: Depend on new capability
See owncloud/core#27709: files_sharing/public/multiple
2017-04-21 14:08:00 +02:00
Kaustubh Welankar
70c3514996 Removed Magic Numbers in propagator #5685 (#5705) 2017-04-20 11:14:53 +02:00
ckamm
543e8a224c Fix a crash in ProxyAuthHandler (#5711)
See
https://sentry.io/owncloud/desktop-win-and-mac/issues/243433178/activity/
https://sentry.io/owncloud/desktop-win-and-mac/issues/234182688/activity/

The problem was that an account's QNetworkAccessManager can be deleted
when reentering the event loop.
2017-04-20 09:21:33 +02:00
Markus Goetz
5ac58d3b83 Server: Parse version from capabilities too #5691 (#5698)
Newer servers will have the option of hiding version, versionstring, edition
and productname. They will always send the full information in the capabilities.
2017-04-19 11:02:03 +02:00
Olivier Goffart
8cb3a77022 Merge remote-tracking branch 'origin/avatar_pics'
Also fix compilation because of 22370fdbdadd06f0cacd249a8d7a32f0d3c1374e

Pull request #5482
2017-04-13 11:54:28 +02:00
Olivier Goffart
1ed4eb46f2 Merge remote-tracking branch 'origin/2.3'
Conflicts:
	VERSION.cmake
2017-04-13 10:19:35 +02:00
Christian Kamm
6abc2bf72d Transfers: Show a decimal for single digit MB rates #4428 2017-04-04 09:54:06 +02:00
Christian Kamm
35af03b2e5 Improve http error messages; cleanup
By default QNetworkReply::errorString() often produces messages like
   "Error downloading <url> - server replied: <reason>"
but the "downloading" part invariably confuses people since the
error might very well have been produced by a PUT request.

This commit produces clearer error messages for HTTP errors.

Additionally:
* Remove some unnecessary null checks from slots connected to
  network job signals and document that these signals never send
  null replies.
* There was a bug where AbstractNetworkJob::_timedout wasn't
  set when derived classes overrode slotTimeout. We now ensure
  it's always set by disallowing overrides of slotTimeout.
  Instead it now calls onTimedOut, which allows custom handling.
* Several subclasses declared errorString, isTimedOut. Move
  these to AbstractNetworkJob.
* Unify handling of OC-ErrorString (via the new, general
  Job::errorString)
* Add documentation in various places.
2017-04-04 09:27:37 +02:00
Markus Goetz
28e9c56a4b Chunking NG: Also send OC-Total-Length on MOVE
For https://github.com/owncloud/core/issues/26988
2017-03-29 19:51:02 +02:00
Olivier Goffart
039ff0ca8e PropagateDownload: fix possible crash
Backtrace looks like this:
Crash: EXCEPTION_ACCESS_VIOLATION_READ at 0x0
  File "propagatedownload.cpp", line 234, in OCC::GETFileJob::slotReadyRead
  File "moc_propagatedownl_CA5CFSHZDTX34X.cpp", line 86, in OCC::GETFileJob::qt_static_metacall
  File "qobject.cpp", line 495, in QMetaCallEvent::placeMetaCall
  File "qobject.cpp", line 1256, in QObject::event
  File "qapplication.cpp", line 3804, in QApplicationPrivate::notify_helper

GETFileJob::slotReadyRead can be called with a QueuedConnection when the
bendwith manager is involved. In that case, if the QNAM was reset
in between, the reply might have been destroyed.
(This is only speculation based on the backtrace)
2017-03-28 18:04:19 +02:00
Christian Kamm
e86499d990 dynamic chunking: cleanup, fixes, improvements
* make target duration a client option instead of a capability
* simplify algorithm for determining chunk size significantly
* preserve chunk size for the whole propagation, not just per upload
* move options to SyncOptions to avoid depending on ConfigFile
  in the propagator
* move chunk-size adjustment to after a chunk finishes, not when
  a new chunk starts
2017-03-28 11:32:10 +02:00
Piotr M
88e0f97798 dynamic chunking prototype 2017-03-27 13:43:39 +02:00
Olivier Goffart
4597201560 Merge remote-tracking branch 'origin/2.3'
Conflicts:
	src/libsync/owncloudpropagator.h
2017-03-16 14:35:08 +01:00
Olivier Goffart
21909cae04 Propagator: change order of destruction
The destructor of the PropagateItemJob will access the propagator's
_activeJobList. So the _rootJob needs to be destroyed before it.
Order of destruction is the reverse of the order of the members in
the class. So put it at the end so it can be destroyed first.

(This made TestSyncEngine::testDirDownloadWithError crash sometimes
in the master branch)
2017-03-16 14:24:25 +01:00
Olivier Goffart
574435687d Merge remote-tracking branch 'origin/2.3'
Conflicts:
	src/libsync/owncloudpropagator.cpp
2017-03-16 13:51:53 +01:00
Christian Kamm
1a279ca158 About: Show the runtime versions of Qt and OpenSSL #5592
I didn't want to touch this visible translated string in a
patch release so I just switched the data.

For Qt versions before 5, we continue to show the build-time version
of OpenSSL but it's clearly marked as such.
2017-03-16 12:40:42 +01:00
Olivier Goffart
0ed929f24b SyncEngine: fix test regression in TestChunkingNG::testCreateConflictWhileSyncing
In 8ef11a38c9, we started blacklisting
SoftError for 0 seconds.  But if the two sync happen with less than
1s interval, we would still prevent them to happen.
So make sure we expire if 0 seconds have expired
2017-03-16 10:17:08 +01:00
Christian Kamm
b98876e265 Account server version: Helper to create versions
Hex literals don't work well with version 10: 0x100000 doesn't do
the right thing.
2017-03-15 16:30:08 +01:00
Christian Kamm
13b2568358 Windows/NTFS: Do not attempt to upload inaccessible files #5544
It is possible to create files with filenames that differ
only by case in NTFS, but most operations such as stat and
open only target one of these by default.

When that happens, we want to avoid uploading incorrect data
and give up on the file.

Typically this situation should never occurr during normal use
of Windows. It can happen, however, when a NTFS partition is
mounted in another OS.
2017-03-14 16:03:08 +01:00
Christian Kamm
8ef11a38c9 Blacklist: Escalate repeated SoftError to NormalError #5500 2017-03-10 15:51:55 +01:00
Klaas Freitag
5e33898a08 Avatar: Use QImage instead of QPixmap to avoid dep on QApplication.
That fixes the test suite.
2017-03-09 22:34:36 +01:00
Olivier Goffart
9c83c57f1c Propagator: Attempt to fix a crash in OwncloudPropagator::scheduleNextJob
The crash reporter shows many crashes in OwncloudPropagator::scheduleNextJob.
We don't really know what could be the cause, but it's probably because
the _activeJobList contains dangling pointer.
So this patch makes sure to remove all the jobs from this list as they get
destroyed.
2017-03-09 15:48:52 +01:00
Olivier Goffart
0e8f0481d4 Merge remote-tracking branch 'origin/2.3'
Conflicts:
	src/libsync/owncloudpropagator.cpp
2017-03-09 09:54:14 +01:00
Christian Kamm
4a1a5fa076 AbstractNetworkJob: Improve redirect handling #5555
* For requests:
  - reuse the original QNetworkRequest, so headers and attributes
    are the same as in the original request
  - determine the original http method from the reply and the request
    attributes
  - keep the original request body around such that it can be sent
    again in case the request is redirected

* Simplify the interface that is used for creating new requests in
  AbstractNetworkJob.
2017-03-07 13:18:01 +01:00
Olivier Goffart
f862c626a1 Propagator: Fix finished signal of directory being emited twice and causing crash
When there is a FatalError, we ended up emiting the finished signal for
the directory job several times, which would lead to crashes

Issue #5578
2017-03-06 16:14:04 +01:00
Olivier Goffart
13df7fcee0 Merge branch '2.3'
Conflicts:
	src/libsync/owncloudpropagator.cpp
2017-03-03 10:57:23 +01:00
Jocelyn Turcotte
384f18da38 Potential fix for crash in PropagateDirectory::abort (#5564) 2017-02-27 17:09:29 +01:00
Jocelyn Turcotte
793a994ce6 Don't abort propagation job abortions synchronously from finished
This leads to crashes since we changed the connection to the parent
jobs not to be queued anymore.
We don't really need to bubble up the finished state through
parents in that case, and it would also mean that we'd recurse
all the way through leaves as we go up to each parent. So just call
abort directly on the OwncloudPropagator and make sure the abortion
call is posted to the event loop.
2017-02-23 17:54:33 +01:00
Jocelyn Turcotte
19bf5e2ff1 Reduce the connection data used by PropagateDirectory objects
Avoid using connections to report up the job tree for signals
that we can directly communicate to the OwncloudPropagator.
This slightly reduces the memory usage and avoid passing those calls
through the whole parent chain.
2017-02-23 17:54:33 +01:00
Jocelyn Turcotte
16ad3d5c88 Create PropagateItemJobs only before starting them 2017-02-23 17:54:33 +01:00
Jocelyn Turcotte
ff2d98596f Split running jobs into a separate vector 2017-02-23 17:54:33 +01:00
Jocelyn Turcotte
072698d606 Split the subjob logic out of PropagateDirectory 2017-02-23 17:54:33 +01:00
Jocelyn Turcotte
0b098045f1 Replace WaitForFinishedInParentDirectory with WaitForFinished for directory moves
In preparation for the PropagateDirectory refactoring, simplify things
by removing WaitForFinishedInParentDirectory, which is currently
implemented as a one-level check.

This value is important for directory items, but is however never
used since a directory CSYNC_INSTRUCTION_RENAME item will always be in
PropagateDirectory::_firstJob, which will have to pass through its own
PropagateDirectory job's parallelism() before reaching the parent's
_subJobs optimization.
Since PropagateDirectory::parallelism can only return WaitForFinished
or FullParallelism, that value is lost. So this commit doesn't
change the behavior for directories, and allow file renames to be
scheduled in parallel across directories (which isn't a problem).
2017-02-23 17:54:33 +01:00
Jocelyn Turcotte
ddb6b3b262 Fix TestSyncEngine::abortAfterFailedMkdir (again)
It could be possible that _firstJob is marked as finished if
aborted before its parent PropagateDirectory was marked as finished,
allowing a posted scheduleNextJob call to schedule the child job
in-between.
2017-02-23 17:54:33 +01:00
Olivier Goffart
7bfe061382 Verify that all strings are properly escaped (#5558)
- I checked every occurence of a '%2' and make correct use of the
QString::arg overload that takes several argument instead of chaining
them, because the first argument can contains a '%1'

 - I tried to look for every label that they either use plain text or richtext
and escape the user provided strings in there.
2017-02-23 14:54:17 +01:00
Christian Kamm
21a09df7d1 Wizard: Handle PROPFIND redirects #5553
By default, followRedirects is true for all requests, to transparently
handle redirections. In the wizard, we have special redirect-handling
code though and that was being skipped.

Setting the flag to false allows the wizard to be aware of redirects
and to handle them in the correct way. Tested with the server described
in
https://github.com/owncloud/administration/tree/master/redirectServer

There's a second bug here, where followRedirects always converts
redirected requests to the GET verb. That means redirected PROPFINDs
will never have worked. This change un-breaks them for the wizard only.
There should be no case that previously worked that stops working now.
2017-02-22 13:24:13 +01:00
Olivier Goffart
112a7dba73 Propagator: Fix t8.pl
The test sets OWNCLOUD_MAX_PARALLEL to 1 to disable parallelism.
But since the max amount of parallelism is twice as much, that does not
work.
So change the way we compute the hardMaximumActiveJob: Use the value of
OWNCLOUD_MAX_PARALLEL to maximize this amount and base the maximum amount
of transfer jobs on it instead of the other way.

A result of this change is that, in case of bandwidth limit, we keep the
default of 6 non-transfer jobs in parallel. I believe that's fine since
the short jobs do not really use bandwidth, so we can still keep the same
amount of small jobs.
2017-02-17 16:56:57 +01:00
Christian Kamm
ee11b5d9be Revert "Don't blacklist 507 Insufficent Storage #5346 (#5424)"
This reverts commit e1f5a49c21.

Retrying uploads with insufficent storage errors frequently leads to
high server traffic. See #5537 for links and a sketch of a correct
solution.
2017-02-17 16:21:54 +01:00
Jocelyn Turcotte
824be789bf Revert "Fix TestSyncEngine::abortAfterFailedMkdir"
This reverts commit 1cec2ca13d
but keeps the reset() -> take() part that is probably the cause
of crashes we've seen in the crash reporter.
2017-02-17 14:06:30 +01:00
Olivier Goffart
a3f4907e0b AccessManager: don't set an invalid network configuration on linux (#5538)
If we call
 setConfiguration(QNetworkConfiguration());
This sets an invalid configuration on the QNAM.
But later, when we really go online because interfaces are discovered,
QNetworkAccessManagerPrivate::_q_onlineStateChanged is called (with isOnline=true).
And this will set the state to disconnected because customNetworkConfiguration is
true, and the networkConfiguration state is disabled.

The workaround we to fix another bug on Windows in which the default network
configuration was not behaving properly.

The issue on linux is hard to reproduce and only happen in some condition,
but it was reproduced on smashbox when they run two owncloudcmd at the same time.

Issues: #4720 , #3600
2017-02-16 16:14:55 +01:00
Christian Kamm
4d7dd0c909 Recent Changes: Fix duplicates #5256
The transmissionProgress emission when the whole sync finishes
shouldn't look exactly like the one for the last file.
2017-02-16 15:10:46 +01:00
Olivier Goffart
e8c10501a5 Folder: Keep files option after aboutToRemoveAllFiles should not clear selective sync list (#5531)
We were removing the wholme journal db when the user wanted to keep all files,
But that would also remove the selective sync lists.
We should only remove the metadata table.

Issue #5484
2017-02-15 13:27:41 +01:00
Olivier Goffart
172689d35c Merge remote-tracking branch 'origin/2.3' 2017-02-14 15:08:10 +01:00
Jocelyn Turcotte
b26db062d2 Fix the UNIT_TESTING build on Windows
- Put all tests in the bin directory so that DLLs can be loaded
- Add missing exports
- Skip tests that use code depending on zlib
- The "GMT" timezone is named differently, use the int constructor instead

5 tests are still failing, it's not really worth fixing at the moment
since no developper is currently using Windows as its main platform.
2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
867b78052d Fix UtilityTest on macOS 2017-02-14 14:32:41 +01:00
Jocelyn Turcotte
1cec2ca13d Fix TestSyncEngine::abortAfterFailedMkdir
It could be possible that _firstJob is marked as finished if
aborted before its parent PropagateDirectory was marked as finished,
allowing a posted scheduleNextJob call to schedule the child job
in-between.
2017-02-08 17:32:04 +01:00
ckamm
7879c470b3 Merge pull request #5518 from ckamm/asserts
Improve usage of asserts
2017-02-08 15:25:02 +01:00
Christian Kamm
4c1fdf1dee Double check usage of asserts #5429
A few are supposed to be fatal.
2017-02-08 15:15:19 +01:00
Christian Kamm
18e25122db Performance: reserve() _remotePerm #5496 2017-02-08 15:06:09 +01:00
Markus Goetz
6a4ae63f14 CentOS: Another fix for old compiler version (2) 2017-02-07 19:22:04 +01:00
Markus Goetz
19dd656989 CentOS: Another fix for old compiler version 2017-02-07 18:32:34 +01:00
Markus Goetz
d9fac50e9b CentOS: Fix build for old compiler version 2017-02-07 18:05:29 +01:00
Markus Goetz
9614a94035 Revert "Disable RTTI"
This reverts commit 85d3de1589.
This breaks our old Qt on CentOS.

We can re-introduce this with #5470
2017-02-07 17:35:04 +01:00
Olivier Goffart
0d16cf41fe Merge remote-tracking branch 'origin/master' into 2.3
We can do that because the only changes that were in master but not in 2.3 were the
translations change and documentation change, and the support for the 'M' permission
which we want in 2.3.
2017-01-27 17:45:05 +01:00
Olivier Goffart
38cf459b3e Confirm External Storage: adjust the notification message
We need to forward the information that the folder is an external storage
for the notification message.

Issue: https://github.com/owncloud/client/pull/5340#issuecomment-274878023
2017-01-27 15:59:59 +01:00
Olivier Goffart
ed7416098e Wizard: Increase the size
The advanced page has become quite complex and does not fit on the screen
anymore if the fonts are too big
2017-01-27 15:59:59 +01:00
Olivier Goffart
f89bc09fd1 Discovery: Filter 'M' out of permissions for non root storage
The sync engine rely on the 'M' in premission to ask for confirmation
(As requested in issue #5340)
But we only want to ask the premission for the 'root' of the mounting point and not
for every subfolders within it.
So we change the discovery phase in a way that it does not keep the 'M' for
children within the external storage.
2017-01-27 15:59:59 +01:00
Olivier Goffart
f854c5263b Wizard: Add options to ask confirmation for external storage
Added two checkboxes in the Account Wizard in the advanced page to change the first options.
Also added a checkbox in the general settings to ask for confirmation for external storages.

Theme options allow to hide the checkboxes in the wizard.

As described in issue #5340
2017-01-27 15:59:59 +01:00
Olivier Goffart
70da607e06 Discovery: Add branding option to disable default sync of 'M' directories
Issue #5331 and https://github.com/owncloud/enterprise/issues/1594
2017-01-27 15:59:59 +01:00
Jocelyn Turcotte
8884fe7236 Remove PropagateDirectory::_runningNow
This variable isn't used anymore except in a stray Q_ASSERT.
2017-01-26 17:13:54 +01:00
Jocelyn Turcotte
bb3efc5988 Clear SyncEngine::_remotePerms after the sync
Also make reuse the SyncFileItem::_remotePerm QByteArray allocation
instead of redoing the copy from the csync char*.
2017-01-26 17:13:54 +01:00
Jocelyn Turcotte
f6665ccc81 Pack boolean members in PropagateUploadFileCommon #5457 2017-01-26 17:13:54 +01:00
Jocelyn Turcotte
f1e9be4fa8 Don't use the StopWatch for each job in production builds #5456
This accounted for more than 10% of the peak memory usage while only
providing value through the debugging log.
2017-01-26 17:13:54 +01:00
Jocelyn Turcotte
9db23d4df1 Remove SyncFileItem::_requestDuration #5456
This will save a QElapsedTimer in each job that was used only for
.owncloudsync.log (for which the request duration doesn't bring much
value).
2017-01-26 17:13:54 +01:00
Jocelyn Turcotte
9d16ad844c Don't reallocate a QByteArray for each contentChecksumType call #5481 2017-01-26 17:13:54 +01:00
Jocelyn Turcotte
b0700ebbab Remove duplicate SyncFileItem entries for the log
The accuracy of that log isn't as important as the few bytes those
fields take as hostage for the whole sync.
2017-01-26 17:13:54 +01:00
Markus Goetz
e859d220be Cookies: Use different DB for different accounts (#5490)
This is a follow up to #5469
2017-01-26 10:54:03 +01:00
Piotr Mrówczyński
c3ae5123cb Delete finished propagation jobs in PropagateDirectory #5269 (#5400) 2017-01-26 10:03:22 +01:00
Jocelyn Turcotte
ee211d7609 Release SyncFileItem objects with their job
We now delete subjobs as their propagation is complete. This allows us
to also release the item by making sure that nothing else is holding a
reference to it.

Remove the stored SyncFileItemVector from SyncEngine and SyncResult
and instead gather the needed info progressively as each itemCompleted
signal is emitted.

This frees some holes on the heap as propagation goes, allowing many
memory allocations without the need of requesting more virtual memory
from the OS, preventing the memory usage from increasingly growing.
2017-01-25 23:26:23 +01:00
Jocelyn Turcotte
1fc5a76622 Pass the SyncFileItem as SyncFileItemPtr in itemCompleted
This will allow us to keep a reference on the items in connected slots.
2017-01-25 23:26:23 +01:00
Jocelyn Turcotte
a764d7eb86 Don't pass the PropagatorJob in itemCompleted
This was to catch duplicate emissions for PropagateDirectory but we
don't emit this signal anymore from there.
This fixes a warning about PropagatorJob not being a registered metatype.

This reverts commit fe42c1a818.
2017-01-25 23:26:23 +01:00
Olivier Goffart
268fc97a71 Merge pull request #5480 from owncloud/chunking-ng-cleanup
Chunking ng cleanup
2017-01-25 19:19:48 +01:00
Klaas Freitag
c00e3e8c0a ConnectionValidator: Set a 20 second timeout for Avatar request. 2017-01-23 21:35:12 +01:00
Klaas Freitag
2a12610a46 Update ASCII-Art describing the connection check flow.
Added Avatar fetching
2017-01-23 21:34:34 +01:00
Klaas Freitag
e05d6bfcdc Make the Account store the avatar pixmap.
The avatar pixmap is fetched from the server by the Connectionvalidator,
once it has validated the user name, it queries the avatar pixmap.

If the server does not have the avatar route, an empty pixmap is stored.
2017-01-22 13:55:08 +01:00
Klaas Freitag
b49dd02e3d New network job AvatarJob: GETs the account avatar from server. 2017-01-22 13:52:19 +01:00
Olivier Goffart
d6fdda8efa ChunkingNG: add '0' padding on the filename (#5476)
The server sorts the chunk by name alphabetically. So if we want to keep
the chunk in order, we need to add a few '0' in front of the chunk name
2017-01-20 16:03:50 +01:00
Olivier Goffart
a63d970e5e ChunkingNG: remove stale chunks when cleaning the uploadInfo table
Stale chunks might be there because a file was removed or would just not
be uploaded, for any reason.
We just start the DeleteJob but we don't care if it success or not.

Relates to https://github.com/owncloud/core/issues/26981

One of the test is testing the case where the file is modified on the server
during the upload. So this test the precondition failed error.
The FakeGetReply logic was modified because resizing a 150MB big QByteArray
by increment of 16k just did not scale when downloading a big file.
2017-01-20 14:48:53 +01:00
Olivier Goffart
f6c77fad17 ChunkingNG: delete stale chunks if the file was changed locally
Relates to https://github.com/owncloud/core/issues/26981

We do not track the success or error of the DeleteJob because it does not
matter. If it fails, it might be because the chunks were already removed.
If not, the chunks will be stale, but the server must anyway do a few
cleanup from time to time because we do not always remove the chunks
2017-01-20 14:48:53 +01:00
Jocelyn Turcotte
b68a28de8d StatusTracker: Emit OK for the last child before parents (#5467)
The current logic tried to avoid a DB lookup just to fetch whether
the file is shared or not since that info is already in the
SyncFileItem. The implementation would however need to decrease the
sync count for itself (and parents) before emitting the new status,
thus emitting the OK status for parents before that last child that
ended the propagation for that folder.

Change the implementation to achieve what we want: give the
possibility to decSyncCount to use a pre-fetched sharing state while
still doing the emission for all involved files. This ensures that
the leaf file also gets its status emitted before its parents.

Issue #4797
2017-01-18 15:40:52 +01:00
Jocelyn Turcotte
d3a0608bd5 SocketAPI: Only push status for files requested by the shell #5361
We currently push the SYNC status for all files that will be propagated,
and then the OK status when those files are propagated.
On top of this, we send those statuses to all clients connected, even
if the socket is kept open by an application that only needed to show
a file open dialog. On macOS we're also using an NSConnection which
means that we have to wait for the RPC call to return from the
extension, which makes bulk status changes possibly heavy.

Reduce the time spent needlessly sending status pushes by limiting
them to files requested through that socket since it connected.
To limit the data to store, only remember the parent directory of
files requested, and store those in a bloom filter.

Note that this adds a requirement to shell extensions: they should
make sure that the status cache only contains entries that have been
requested through the socket API. In other words, the status cache
must be empty when each socket client connects to the socket API.
Otherwise the cached icon type will be shown to the user, and the
SocketAPI won't push new status for that file if it didn't receive
a RETRIEVE_FILE_STATUS.
2017-01-18 12:15:52 +01:00
Olivier Goffart
9b96899d75 Sync Engine: Specify what chars are not syncable in the error message #1733 (#5449) 2017-01-18 11:45:46 +01:00
Olivier Goffart
b046cca010 Fix more compilation errors with Qt 4
Ammand last commit which was a fixup to the change from pull request #5468
for issue #5453
2017-01-18 10:19:51 +01:00
Olivier Goffart
edf0a99a8a Fix compilation with Qt 4
Fixup to the change from pull request #5468 for issue #5453
2017-01-18 09:57:36 +01:00
Markus Goetz
85d3de1589 Disable RTTI
Shrinks owncloud binary by 24 KB and libowncloudsync by 14 KB.
I don't know if it has influence on memory usage or runtime speed though.

Was worth a try.
2017-01-17 14:52:10 +01:00
Christian Kamm
cd9e88ad22 PropagatorJob: Remove unneccessary pointer #5453 2017-01-17 14:47:38 +01:00
Christian Kamm
29b39acfbe PropagatorJob: Access propagator through function #5453 2017-01-17 14:47:38 +01:00
Markus Goetz
9d7425b201 DB: Also migrate -shm and -wal to new name #3764 #5045 (#5459) 2017-01-16 15:42:11 +01:00
Markus Goetz
7eb43d7f7e PropagateUploadFileNG: Delete unused map contents to free memory 2017-01-13 19:00:04 +01:00
ckamm
d76e0ec6d8 Reset stuck chunked uploads eventually #5344 (#5443)
Previously this wasn't happening for errors that were not
NormalErrors because they don't end up in the blacklist.

This revises the resetting logic to be independent of the
error blacklist and make use of UploadInfo::errorCount
instead.

412 errors should reset chunked uploads because they might be
indicative of a checksum error.

Additionally, server bugs might require that additional
errors cause an upload reset. To allow that, a new capability
is added that can be used to advise the client about this.
2017-01-13 14:44:45 +01:00
Christian Kamm
adbe5ecf55 ownsql: qFatal on prepare errors #5357 2017-01-04 15:15:06 +01:00
Christian Kamm
084146756b Account: Fix displayName() #5413
This displayName() seemed to be based on Account::user() which used
to call _credentials->user(). But then we repurposed user() to be
davUser() and this usage wasn't updated to point back to the username
used for the credentials.
2017-01-03 13:54:59 +01:00
ckamm
e1f5a49c21 Don't blacklist 507 Insufficent Storage #5346 (#5424) 2017-01-03 13:48:32 +01:00
Olivier Goffart
d433f0e08e HTTPCreds: remove all cookies when logging out (#5383)
Some custom server use persistent cookies with the auth token. So we should
clear all the cookies when disconnecting.
Account::clearCookieJar is only called from the HTTPCredentials. This funciton
is not used for shibboleth.
There is probably no reasons to keep the HTTP cookie anyway.

Issue #5370
2017-01-03 11:39:10 +01:00
Christian Kamm
6835429a28 Sql: Allow prepare fail for PRAGMA quick_check #5357
This actually happens in low-disk situations.
2017-01-03 10:43:29 +01:00
Christian Kamm
e306f4611c Reschedule a folder regularly with some delay for some errors
Like "folder doesn't exist" - such that we will detect when the folder
becomes available and start syncing.

See #5317
2017-01-02 08:52:26 +01:00
Markus Goetz
c6f4f44619 Fix up SSL client certificates #5213 #69 (#5289)
The re-enables the UI, uses Qt API for importing and
stores the certificate/key in the system keychain.
People who had set up client certs need to re-setup the account. This is ok
since it was an undocumented feature anyway.
2017-01-02 08:34:02 +01:00
Piotr Mrówczyński
d781e63fab Add capability to disable parallel chunked upload #5364 - technical review (#5403) 2016-12-21 15:08:45 +01:00
Christian Kamm
030e3a5d4a Increase initial bandwidth estimates #4428 #5390
To 2 MB/s and 10 files/s.
2016-12-21 08:53:47 +01:00
Christian Kamm
3912dba33a Move qDeclareMetatype from Folder to SyncEngine 2016-12-20 11:43:39 +01:00
Christian Kamm
d8c479ab1e Don't wipe db if check fails while disk space is low #5357
An experimental fix for the issue, merged because it's low risk and
makes testing easier.
2016-12-20 09:33:59 +01:00
Olivier Goffart
ac95844ebd Propagator: make sure every network job has a parent
This could make sure that the network job gets deleted if the parent job gets
deleted, and would avoid crashes like:

Crash: EXCEPTION_ACCESS_VIOLATION_READ at 0xffffffff8b008a04
  File "qiodevice.cpp", line 1617, in QIODevice::errorString
  File "propagatedownload.cpp", line 264, in OCC::GETFileJob::slotReadyRead
  File "moc_propagatedownload.cpp", line 85, in OCC::GETFileJob::qt_static_metacall
  File "qobject.cpp", line 3716, in QMetaObject::activate
  File "moc_qiodevice.cpp", line 154, in QIODevice::readyRead
  File "qnetworkreplyhttpimpl.cpp", line 1045, in QNetworkReplyHttpImplPrivate::replyDownloadData

(#5329)
2016-12-14 14:13:47 +01:00
Olivier Goffart
c5b90d9507 Account::slotHandleSslErrors: add guards in case the reply gets deleted in the dialog event loop
Since slotHandleSslErrors will show a dialog with an eventloop, it could be
That the reply gets deleted. Guard against that.

(#5329)
2016-12-14 14:13:47 +01:00
Jocelyn Turcotte
f985111b62 Fix the log window not showing csync logs
The csync log level was only set up on startup, and for log files.
Fix the issue by making Logger::isNoop rely on being explicitly activated
for the log window instead of relying on the presence of a connected
signal, and move the csync log level logic in Logger.
2016-12-09 10:03:46 +01:00
Jocelyn Turcotte
e8d734b1c2 Fix the sync direction being wrong on MSVC
The compiler seems to use signed enums and we need to reserve an extra
bit for the sign to avoid the 2 value to overflow and being interpreted
as -2 when read, and thus not being correctly compared to the full enum
value.
2016-12-09 10:03:46 +01:00
ckamm
5bef1aa402 Merge pull request #5045 from owncloud/dbjournal_per_account
Allow a folder to be synced to several accounts. This changes the path of the sync journal file!
2016-12-06 10:55:58 +01:00
Olivier Goffart
4ac2b764a4 libsync: Clear the upload info after several retries (#5345)
Issue #5344
2016-12-05 17:21:36 +01:00
Olivier Goffart
19d2677db1 libsync: use the new webdav url if the server reports it
The rules to select the webdav url are now:

 - If the server reports that the new chunking algorithm is working,
   always use remote.php/dav/files/<username>
   This capability can be overriden with an environment variable

 - Otherwise, use the dav path provided by the theme, which defaults to
   remote.php/webdav

This means that with the newer server, the branding can no longer override
the webdav URL. If there is still an usecase for the branding to do so, we
need to find another way to override it. But it is now more complicated to
configure as might need include the username and need different endpoint
depending on the operations (chunks or not)

Issue #4007
2016-11-29 16:16:20 +01:00
Christian Kamm
045498c71a Account: Rename user() to davUser()
Avoids potential confusion with credentials->user()
2016-11-23 17:08:17 +01:00