Commit graph

2155 commits

Author SHA1 Message Date
Christian Kamm
61972c35a8
Discovery: Parse etag to be consistent with RequestEtagJob
This avoids unnecessary sync runs.

For #7345
2020-12-15 10:58:54 +01:00
Christian Kamm
55ee3f440b
Vfs: Remove old db record when dehydrating via rename
For #7338
2020-12-15 10:58:54 +01:00
Christian Kamm
e6990bd04e
Vfs: Make move detection work with virtual files #7001
Previously a checksum computation could be done on a suffix-placeholder
file, making discovery believe that no move took place.
2020-12-15 10:58:54 +01:00
Christian Kamm
00dcf3ef59
Vfs: Ensure pins change with (de-)hydration
Previously an implicit hydration of a file in an online-only folder
would not change the pin state and cause a dehydration on the next
sync.
2020-12-15 10:58:53 +01:00
Christian Kamm
0df3b83bd2
Vfs: Add hook to allow update-metadata for unchanged files
Allows winvfs to convert files to placeholders when vfs is enabled. This
is required to mark files as in-sync #7329.
2020-12-15 10:58:53 +01:00
Christian Kamm
5bc2180478
Vfs: Preserve pin state on hydration
For #7322 and #7323
2020-12-15 10:58:53 +01:00
Dominik Schmidt
92f6d866e1
Add missing OWNCLOUDSYNC_EXPORT 2020-12-15 10:58:52 +01:00
Christian Kamm
5acb157a7e
VfsSuffix: Wipe stale pin states #7273
Previously the pin states of deleted files stayed in the 'flags'
database and could be inadvertently reused when a new file with the same
name appeared. Now they are deleted.

To make this work right, the meaning of the 'path' column in the 'flags'
table was changed: Previously it never had the .owncloud file suffix.
Now it's the same as in metadata.path.

This takes the safe parts from #7274 for inclusion in 2.6. The more
elaborate database schema changes (why use 'path' the join the two
tables in the first place?) shall go into master.
2020-12-15 10:58:52 +01:00
Christian Kamm
0e9f030b0f
RequestEtagJob: Consistently parse etags #7271
Previously RequestEtagJob did return the etag verbatim (including extra
quotes) while the db had the parsed form. That caused the etag
comparison during discovery move detection to always fail. The test
didn't catch it because the etags there didn't have quotes.

Now:
- RequestEtagJob will parse the etag, leading to a consistent format
- Tests have etags with quotes, detecting the problem
2020-12-15 10:58:52 +01:00
Christian Kamm
69fa1e4775
UploadDevice: Fix windows issues #7264
- Close the UploadDevice to close the QFile after the PUT job is done.
  This allows winvfs to get an oplock on the file later.

- Don't rely on QFile::fileName() to be valid after
  openAndSeekFileSharedRead() was called. The way it is openend on
  Windows makes it have an empty filename.
2020-12-15 10:58:52 +01:00
Christian Kamm
fb47419e84
Vfs: Move pin state if files move #7250
Previously renames of items didn't carry the pin state with them.
2020-12-15 10:58:52 +01:00
Christian Kamm
34dc5e4e88
Vfs: Don't let new local files start out unpinned #7250
If one adds a new file to an online-only folder the previous behavior
was to upload the file in one sync and dehydrate it in the next. Now
these new files get set to Unspecified pin state, making them retain
their data.
2020-12-15 10:58:52 +01:00
Markus Goetz
62d876b09a
OAuth2: Better error logging
This does not fix a bug, just was found while spotting a bug that was no bug.
For https://github.com/owncloud/enterprise/issues/2951
2020-12-15 10:58:51 +01:00
Christian Kamm
452ed56571
Upload: Read file chunks gradually #7226
Instead of all at once, to reduce peak memory use.

Changing UploadDevice in this way requires keeping the file open for the
duration of the upload. It also means changes to open(), seek(), close()
to ensure that uses of the device work right when a request needs to
be resent.
2020-12-15 10:58:51 +01:00
Christian Kamm
a7852e3aba
Http2: Resend requests on ContentReSend error #7174
Since Qt does not yet transparently resend HTTP2 requests in some cases
we do it manually.

The test showed a problem where the initial non-200 reply would close
the target temporary file and the follow-up request couldn't store any
data. Removing that close() call is safe because there also is a
_saveBodyToFile flag that guards writes to the target file.
2020-12-15 10:58:51 +01:00
Olivier Goffart
c335f69a65
Discovery: Do not abort the sync in case of error 404 (or 500)
Issue: #7199
2020-12-15 10:58:51 +01:00
Olivier Goffart
9f08636a4a
SyncEngine: Fix renaming a single file cause the "delete all file" popup
Possibly a regression, since the new discovery discovers rist the renamed
files as removed

Issue #7204
2020-12-15 10:58:51 +01:00
Christian Kamm
edad7ce7d7
Vfs: Mark sqlite temporaries excluded on db-open #7141
The previous patch ensured that the sqlite temporaries weren't deleted
and recreated for every sync run, but there was still time between
client startup and the first sync run where they would have the
"needs-sync" icon.
2020-12-15 10:58:50 +01:00
Christian Kamm
7774b8049e
Vfs: Distinguish availability error kinds #7143
Previously "no-availability" meant db-error and querying the
availability of a nonexistant path returned AllHydrated.

Now, the availability has a DbError and a NoSuchItem error case.
2020-12-15 10:58:49 +01:00
Christian Kamm
21cb93e3ce
SyncEngine: Don't close db when done #7141
The db-close operation is likely a leftover from when the SyncEngine
owned its own db connection and serves no purpose anymore.

Closing the db causes the removal of the temporary wal and shm files.
These files are recreated when the db is opened again, which happens
almost immediately.

This is a problem for winvfs because the delete-recreate step wipes the
exclusion state on these files just after the sync is done. That meant
that the db temporaries permanently had a "needs sync" icon marker shown
in the explorer.

Avoiding reopening the db also reduces the number of log messages per
sync.
2020-12-15 10:58:48 +01:00
Christian Kamm
cfbcdc01db
Discovery: Improvements to doc comments 2020-12-15 10:58:48 +01:00
Christian Kamm
fbe2dbf4ab
Discovery: Query data-fingerprint on root item
Previously the property wasn't queried, meaning the fingerprint logic
couldn't get triggered.
2020-12-15 10:58:48 +01:00
Christian Kamm
c50f041c5b
Discovery: 403 and 503 on root cause error
Previously these result codes during remote discovery of the sync root
would not cause an error and the discovery would get stuck.

Also extends RemoteDiscovery tests to check for errors on the root item.
2020-12-15 10:58:48 +01:00
Christian Kamm
7b96321df2
PropagateIgnore: Default to NormalError for INSTRUCTION_ERROR
Previously if one set the instruction to ERROR while forgetting to set
an error status, it'd propagate as FileIgnored. Now the default is
NormalError for INSTRUCTION_ERROR and FileIgnored for
INSTRUCTION_IGNORE.
2020-12-15 10:58:47 +01:00
Christian Kamm
da40e84aec
About: Add remark about vfs plugin that's in use #7137 2020-12-15 10:58:47 +01:00
Christian Kamm
2bffde2600
Fix windows build 2020-12-15 10:58:47 +01:00
Christian Kamm
22234e0e73
LogWindow: Remove output, add "go to log folder" button #6475 2020-12-15 10:58:47 +01:00
Christian Kamm
dcf34316fd
Vfs: Add 'availability', a simplified, user-facing pin state #7111
The idea is that the user's question is "is this folder's data available
offline?" and not "does this folder have AlwaysLocal pin state?".
The the answers to the two questions can differ: an always-local
folder can have subitems that are not always-local and are dehydrated.

The new availability enum intends to describe the answer to the user's
actual question and can be derived from pin states. If pin states aren't
stored in the database the way of calculating availability will depend
on the vfs plugin.
2020-12-15 10:58:47 +01:00
Christian Kamm
590db28541
Vfs: Clear up relationship between _type and pin state
The pin state is a per-item attribute that has an effect on _type:
AlwaysLocal dehydrated files will be marked for hydration and OnlineOnly
hydrated files will be marked for dehydration.

Where exactly this effect materializes depends on how the pin states are
stored. If they're stored in the db (suffix) the dbEntry._type is
changed during the discovery.

If the pin state is stored in the filesystem, the localEntry._type must
be adjusted by the plugin's stat callback.

This patch makes pin states behave more consistently between plugins.
Previously with suffix-vfs pin states only had an effect on new remote
files. Now the effect of pinning or unpinning files or directories is as
documented and similar to other plugins.
2020-12-15 10:58:46 +01:00
Christian Kamm
cd10e3d28c
PropagateDirectory: Set initial dir mtime to server mtime #7119
It's still not synced in any way later.
2020-12-15 10:58:46 +01:00
Christian Kamm
69887c531e
PropagateDirectory: Remove dead code
1. The _firstJob is usually deleted by the time the PropagateDirectory
   finishes. (deleteLater() is called early)
2. The PropagateDirectory::_item and PropagateRemoteMkdir::_item point
   to the same SyncFileItem anyway. This code is a leftover from when
   each job had its own instance.
2020-12-15 10:58:46 +01:00
Christian Kamm
fd9b01981b
Detect missing server data during discovery #7112
This has two positive effects:
- We can put the error on the particular file that has missing data
- We can sync all other files
2020-12-15 10:58:46 +01:00
Christian Kamm
4bab93b246
Vfs: Better handling and more tests for suffix file renames
Previously removing the vfs suffix of a file always triggered a
conflict. Now it may just cause a file download.

This was done because users expected symmetry in the rename actions and
renaming foo -> foo.owncloud already triggers the "make the file
virtual" action. Now foo.owncloud -> foo triggers the "download the
contents" action.
2020-12-15 10:58:46 +01:00
Christian Kamm
5e5b0b3f76
Vfs suffix: Require suffix when creating placeholder files 2020-12-15 10:58:45 +01:00
Christian Kamm
3f55f9302e
Vfs: Hydrating a virtual is SYNC not NEW #7101
Previously it'd be NEW(ItemTypeFile), but now it has changed to be
SYNC(ItemTypeVirtualFileDownload) which allows better classification.
2020-12-15 10:58:45 +01:00
Christian Kamm
0c0049c908
Local discovery tracking: On success, also wipe _renameTarget
It's possible that the rename target was in the local discovery list.
2020-12-15 10:58:45 +01:00
Christian Kamm
47f10fbf96
Fix and test _file and _renameTarget
There was a bunch of inconsistency around whether _file was set to
_renameTarget or not. This is now never done, passing on more
information.
2020-12-15 10:58:44 +01:00
Christian Kamm
0cf19123a7
VFS: Unbreak behavior for rename+hydrate #7001
Users can rename a file *and* add/remove the vfs suffix at the same time
leading to very complex sync actions. This patch doesn't add support for
them, but adds tests and makes sure these cases do not cause unintened
behavior.

The rename will be propagated, but the users's hydrate/dehydrate request
will be ignored.
2020-12-15 10:58:44 +01:00
Olivier Goffart
4346567a03
Async Poll: keep the size in the database
This was not required with 2.5 because a size of 0 was ignorted when comparing
size by the csync updater, to be compatible with very old version of the database.
But the we discovery will still think the file is changed if the database contains
a size of 0
2020-12-15 10:58:44 +01:00
Olivier Goffart
46bf3ed31a
Upload: asynchronious operations
Implements https://github.com/owncloud/core/pull/31851
2020-12-15 10:58:44 +01:00
Markus Goetz
32c60c2f5d
macOS: Fix vfs suffix plugin paths #7090 2020-12-15 10:58:44 +01:00
Christian Kamm
ee6a48b3dc
Client certs: Store pkcs12 in config, password in keychain
It still reads and writes the old format too, but all newly stored
client certs will be in the new form.

For #6776 because Windows limits credential data to 512 bytes in older
versions.
2020-12-15 10:58:43 +01:00
Christian Kamm
cc840534c0
Add PLUGINDIR cmake setting and define #7027
By default, plugins are only searched next to the binary or next to the
other Qt plugins. This optional build variable allows another path to be
configured.

The idea is that on linux the oC packaging probably wants the binary in
something like /opt/owncloud/bin and the plugins in
/opt/owncloud/lib/plugins.

Similarly, distribution packagers probably don't want the plugins next
to the binary or next to the other Qt plugins. This flag allows them to
configure another path that the executable will look in.
2020-12-15 10:58:43 +01:00
Christian Kamm
93afc2a04b
Discovery win: Fix detection of case-only renames
Previously they were detected as DELETE+NEW because if "a" is renamed to
"A" then QFile::exists("a") will still return true on Windows.
2020-12-15 10:58:42 +01:00
Christian Kamm
238ac53666
Ensure local discovery on selective sync changes
As far as I'm aware local discovery can be skipped on folders that are
selective-sync blacklisted, so a local discovery is required when an
entry is removed from the blacklist.

Also rename
avoidReadFromDbOnNextSync() -> schedulePathForRemoteDiscovery()
since the old name might also imply it's not read from db in the local
discovery - which is not the case. Use Folder::
schedulePathForLocalDiscovery() for that.
2020-12-15 10:58:41 +01:00
Christian Kamm
4c04351360
Fix warnings about signedness
Sizes are always qint64, not unsigned.
TransferIds are always uint.
2020-12-15 10:58:41 +01:00
Christian Kamm
910ccaf600
Vfs: Improve sync protocol entries for actions
Creating a new virtual file and replacing a file with a virtual one now
have their own text in the protocol, not just "Downloaded".

To do this, the SyncFileItem type is kept as
ItemTypeVirtualFileDehydration for these actions. Added new code to
ensure the type isn't written to the database.

While looking at this, I've also added documentation on SyncFileItem's
_file, _renameTarget, _originalFile and destination() because some of
the semantics weren't clear.
2020-12-15 10:58:41 +01:00
Christian Kamm
f24687ec93
Vfs: Make dehydration a SYNC and not a NEW action
That change will be useful for the notifications. Previously the
dehydrated files were reported as "newly downloaded", now they're
reported as "updated".
2020-12-15 10:58:40 +01:00
Christian Kamm
797734870f
PropagateDownload: Create conflict even if local file changed
Fixes a bug introduced while moving the attribute propagation before the
conflict-renaming.
2020-12-15 10:58:40 +01:00
Christian Kamm
8a8e93827f
Vfs suffix: Fix dehydration creating the wrong db entry 2020-12-15 10:58:40 +01:00
Christian Kamm
6a977edeee
Vfs: Remove VfsDefaults
That just complicated things. It's ok if Vfs is not a fully abstract
interface class.

The pinstate-in-db methods are instead provided directly on Vfs and
VfsSuffix and VfsOff use them to implement pin states.

The start() method is simply non-virtual and calls into startImpl() for
the plugin-specific startup code.
2020-12-15 10:58:40 +01:00
Christian Kamm
e9cbe13598
Vfs: Add option to hide socketapi pin actions
Because some plugins provide alternative ui.
2020-12-15 10:58:40 +01:00
Christian Kamm
e6ee5d0f8a
PropagateDownload: Conflict-rename later
The block of code that propagated attributes etc from the previously
existing file was placed *after* the block that renamed the previously
existing file to a conflict name. That meant the propagation didn't work
in the conflict case.
2020-12-15 10:58:39 +01:00
Christian Kamm
6c5fa1dadd
Vfs: dehydration is separate action
Allows for better attribute preservation.

Also add verifyFileUnchanged() call before dehydration to avoid data
loss when discovery takes a while.
2020-12-15 10:58:39 +01:00
Christian Kamm
83a818678f
PinStates cleanup
- SyncJournalDB functions now behind internalPinStates() to avoid
accidental usage, when nearly everyone should go through Vfs.
- Rename Vfs::getPinState() to Vfs::pinState()
2020-12-15 10:58:39 +01:00
Christian Kamm
597cc60f52
Vfs: Enable propagating attributes on download 2020-12-15 10:58:38 +01:00
Christian Kamm
01f7bc7b49
SyncFileStatusTracker: Distinguish Warning and Excluded
Any folder with a (potentially deeply) contained error will have
StatusWarning. StatusExcluded marks exclusions. The difference is useful
to know for VFS.
2020-12-15 10:58:38 +01:00
Christian Kamm
7f400e3226
Pin state updates
- unspecified and inherited are different
- move enum to header in common/
- access through Vfs instead of directly in Journal
2020-12-15 10:58:38 +01:00
Christian Kamm
f89450648f
Vfs: Clarify SyncEngine::wipeVirtualFiles()
Possibly the behavior should actually change and the function should
de-placeholder all items, not just dehydrated ones.
2020-12-15 10:58:38 +01:00
Christian Kamm
5820ac8b41
Discovery: Files can have dehydrate/download actions
This will be used in conjunction with vfs plugins that detect whether a
file has a pending hydration/dehydration through independent means and
communicate that to the discovery through local file type.
2020-12-15 10:58:38 +01:00
Christian Kamm
af1666788e
Discovery: Add signal for silentlyExcluded files
This allows SyncFileStatusTracker to also know about these. After all
its information is used to provide icons for them too.
2020-12-15 10:58:37 +01:00
Christian Kamm
e2eea24a03
Vfs: Send SyncFileStatusTracker data to vfs plugins 2020-12-15 10:58:37 +01:00
Christian Kamm
31394f14b5
Vfs: Make files that end up in db placeholders
Since 'placeholder' just means that it's an item of the special type
that the vfs plugin can deal with - no matter whether hydrated or
dehydrated - all done items should become placeholders. Even
directories.

Now every file that passes through updateMetadata() will be converted to
a placeholder if necessary.
2020-12-15 10:58:37 +01:00
Christian Kamm
ca0323e1e7
Propagator: Helper for updating db
Similar steps were done in many propagation jobs.

This also updates the db entry to always have the item.destination() as
file path.
2020-12-15 10:58:37 +01:00
Christian Kamm
fe27804afb
Move SyncFileStatus to libcommon
It'll be needed in vfs plugins so they can connect to the data coming
out of SyncFileStatusTracker.
2020-12-15 10:58:37 +01:00
Christian Kamm
4d58208676
File watcher: Reduce touch ignore duration
On Linux and Windows the file watcher can't distinguish between changes
that were caused by the process itself, like during a sync operation,
and external changes. To work around that the client keeps a list of
files it has touched and blocks notifications on these files for a bit.

The duration of this block was originally and arbitrarily set at 15
seconds. During manual tests I regularly thought there was a bug when
syncs didn't trigger, when the only problem was that my changes happened
too close to a previous sync operation.

This change reduces the duration to three seconds. I imagine that this
is still enough.

Also use std::chrono while at it.
2020-12-15 10:58:36 +01:00
Christian Kamm
13890c04a7
HttpCreds: Warn in log if keychain-write jobs fail #6776
Also, calling deleteLater() on jobs is unnecessary (they autodelete
after finished()) and deleting the attached QSettings is also
unnecessary because the settings object is parented to the job.
2020-12-15 10:58:36 +01:00
Christian Kamm
1a250bc3c7
Download: Ignore content-length for compressed HTTP2/SPDY replies #6885
It contains the compressed size.

See https://bugreports.qt.io/browse/QTBUG-73364
2020-12-15 10:58:36 +01:00
Olivier Goffart
6f4bf585f0
Move: Fix move detection in directory move on the other side 2020-12-15 10:58:35 +01:00
Olivier Goffart
88d02a887f
Move: add comments and tests 2020-12-15 10:58:35 +01:00
Olivier Goffart
1fb4c22adf
Move: add more test and fix move within moves 2020-12-15 10:58:35 +01:00
Olivier Goffart
ade4c11de3
Rename: fix renamed folder moved into renamed folder issue
Issue #6694
2020-12-15 10:58:35 +01:00
Christian Kamm
b91839b760
SyncEngine: Rename phase finishing slots
slotDiscoveryJobFinished -> slotDiscoveryFinished
slotFinished -> slotPropagationFinished

This should be clearer. Particular the
slotFinished -> finalize -> emit finished()
chain was confusing before.
2020-12-15 10:58:34 +01:00
Christian Kamm
9f8505c736
Discovery: Fix aborting during discovery #6972 2020-12-15 10:58:34 +01:00
Christian Kamm
836d298141
vfs suffix: Ignore server files or synced files with the suffix #6953 2020-12-15 10:58:34 +01:00
Christian Kamm
e774c6c739
Discovery: Ensure selective sync lists are sorted #6958 2020-12-15 10:58:34 +01:00
Dominik Schmidt
17465a6f94
Fix lookup of system wide settings 2020-12-15 10:58:32 +01:00
Olivier Goffart
51d2e41d8b
Chunking Move: Fix too many starting slashes in the destination header
Commit 6ca724f fixed it for the move case. But the upload MOVE did the same

Issue #6904
2020-12-15 10:58:31 +01:00
Christian Kamm
d956f518a8
vfs: Remove newFilesAreVirtual - use root PinState instead
This unifies how to deal with pin states.

Also enable reading a folders direct pin state vs its effective pin
state.
2020-12-15 10:58:30 +01:00
Olivier Goffart
62ec4c9330
Discovery: Handle the blacklistFiles from the server capabilities
Issue #434

Ideally one could add the blacklist to the exlucde reggexp, but this
is simpler
2020-12-15 10:58:30 +01:00
Christian Kamm
8fecff5153
Progress: Virtual file creation needs no transfer progress #6933
Treat virtual file creation as a size-less action, similar to
propagating renames or deletions.
2020-12-15 10:58:30 +01:00
Christian Kamm
486c25cb47
vfs: Use PinState in sync algorithm #6815
New files are virtual if the file's pin state is OnlineOnly.
2020-12-15 10:58:29 +01:00
Olivier Goffart
a29320b18d
Discovery: Set right direction when restoring deleted discovery because it has modified files
(Catched by a faillure of t1.pl)
2020-12-15 10:58:29 +01:00
Christian Kamm
30294e0c9a
Vfs: Have a static list of potential plugins for now
Fixes in-source builds and other cases where more non-plugin directories
are created in src/libsync/vfs.
2020-12-15 10:58:28 +01:00
Olivier Goffart
fd410a5a84
SyncEngine: Ensure that the paths passed to the discovery ends with slashes
This was making the tx.pl test fail
2020-12-15 10:58:28 +01:00
Olivier Goffart
edd806960d
Propagator: Don't abort sync on error 503
Only do it when it is actually a maintenance mode

Issues #5088, #5859, https://github.com/owncloud/enterprise/issues/2637
2020-12-15 10:58:27 +01:00
Dominik Schmidt
687eb9665d
Fix installation of vfs suffix plugin 2020-12-15 10:58:27 +01:00
Olivier Goffart
a7a54b6d11
VFS: Plugin needs to be installed, otherwise it can't be loaded 2020-12-15 10:58:27 +01:00
Olivier Goffart
47b2913357
Fix compilation warning
src/libsync/propagatorjobs.cpp:63:10: warning: lambda capture 'this' is not used [-Wunused-lambda-capture]
        [this, &deleted](const QString &path, bool isDir) {
         ^~~~~
2020-12-15 10:58:27 +01:00
Christian Kamm
305d439c41
vfs: Separate vfs availability from new-files-virtual
This helps support 2.5 settings where there are virtual files in the
tree but new files aren't created virtual.

It's also a prelude for #6815

There's currently no way of
- upgrading vfs plugins (a silent suffix->winvfs upgrade is attempted
  once only, when moving to master)
- disabling vfs capabilities outright
2020-12-15 10:58:27 +01:00
Christian Kamm
1e5e884805
vfs: Don't always load plugins, check metadata 2020-12-15 10:58:26 +01:00
Christian Kamm
b30f79edf6
vfs: Ensure SyncOptions::_vfs is never null
- Create a VfsOff derived class
- Make it a shared pointer shared with Folder::_vfs
2020-12-15 10:58:26 +01:00
Christian Kamm
842577e014
vfs: Switch order of deletion and journal removal in dehydration
To be more crash-resilient.
2020-12-15 10:58:26 +01:00
Christian Kamm
4e40b635da
vfs: Update outdated comment 2020-12-15 10:58:26 +01:00
Christian Kamm
9196aa8e0a
vfs: Remove VfsSuffixPrivate 2020-12-15 10:58:25 +01:00
Christian Kamm
f502a526fa
Generalize Result<> class, add Optional<>
To make it nicer to use outside of HTTP results.
2020-12-15 10:58:24 +01:00
Dominik Schmidt
1104883fba
Cleanup CMake (output) 2020-12-15 10:58:23 +01:00
Christian Kamm
f074c1a07b
vfs: Add check for presence of win plugin 2020-12-15 10:58:23 +01:00
Christian Kamm
fa6f3cd847
vfs: Fix plugin decision in wizards, sanitize loading 2020-12-15 10:58:23 +01:00
Christian Kamm
d5a3604d39
vfs: Reenable local metadata updating of vfs files 2020-12-15 10:58:23 +01:00
Christian Kamm
96f4fd46e5
vfs: Make some behaviors suffix-vfs specific 2020-12-15 10:58:23 +01:00
Christian Kamm
90827c8162
vfs: Allow remote renames to propagate as such 2020-12-15 10:58:22 +01:00
Christian Kamm
e39d751b59
Fix big-folder detection when vfs is disabled 2020-12-15 10:58:22 +01:00
Christian Kamm
bee1b0edd4
vfs: Implement dehydration tagging 2020-12-15 10:58:22 +01:00
Christian Kamm
ae9a7e088f
vfs: Fix suffix detection and handling 2020-12-15 10:58:22 +01:00
Christian Kamm
1ed005aafe
winvfs: Better error reporting
- fix download and validation error reporting
- add tests
2020-12-15 10:58:22 +01:00
Christian Kamm
0f2ef42ba2
Fixes after rebase to master
- use vfs suffix in ProcessDirectoryJob
- fix include vfs.h
- fix local vio passing vfs
- fix checksum computation
- vfs mode use
- mingw lambda compile issue
2020-12-15 10:58:21 +01:00
Christian Kamm
2b20985875
winvfs: initial work
Done by ckamm and dschmidt
2020-12-15 10:58:21 +01:00
Christian Kamm
22e08cf6ad
Checksums: Work on QIODevice*s
Needed for cfapi where we want to feed data through a custom device
which retrieves data from the windows api.
2020-12-15 10:58:21 +01:00
Olivier Goffart
7c1871ae65
Account: Fix crash when deleting an account
We should not have request parented to the account, otherwise we might get
a loop on the deletion order.

Issue #6893
2020-12-15 10:58:21 +01:00
Olivier Goffart
5e05b61d2f
csync_vio_locale: use QString for the path in opendir
So we don't have to convert to utf8 and back again
2020-12-15 10:58:20 +01:00
Olivier Goffart
3c3619f99a
Exclude: do everything with QString wiuthout converting to char* 2020-12-15 10:58:19 +01:00
Olivier Goffart
164051b0c9
ProcessDirectoryJob::process: optimize so there is only one map 2020-12-15 10:58:19 +01:00
Olivier Goffart
1783db5811
PropagateUpload: Avoid many allocations by using QByteArrayLiteral 2020-12-15 10:58:19 +01:00
Olivier Goffart
69de2d5180
Sync: optimize by removing setFileRecordMetadata
Inh most case we already have a record from before, so avoid doing a useless
lookup in the database.
In owncloudpropagator.cpp, directories do not have a checksum so no need
to call a function that preserves it
2020-12-15 10:58:19 +01:00
Gerhard Gappmeier
b3792ac1f0
libsync: Fix build error with TOKEN_AUTH_ONLY
When enabling TOKEN_AUTH_ONLY, the code path using QPainter is disabled.
So we also don't need the includes.
This header is not available for Remarkable.
2020-12-15 10:58:18 +01:00
Kevin Ottens
9131a68ec4
Get DatabaseErrorTest to pass
The E2EE code path would get the engine to go wrong in case of db error.
It's just better to have a failing upload or failing mkdir later in those
cases.

Emitting signals from a ctor is a bad idea anyway

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-12-15 10:58:18 +01:00
Olivier Goffart
b79e57d1c1
Discovery: fix double emission of finished in case of error 2020-12-15 10:58:17 +01:00
Olivier Goffart
7061f31887
Discovery: Fix downloading files when database is used for local discovery
This also fix the currently failling LockedFilesTest
2020-12-15 10:58:17 +01:00
Christian Kamm
815e0fee8f
Propagator: Add assert against duplicate done() calls 2020-12-15 10:58:16 +01:00
Olivier Goffart
eb23776f16
Discovery: Fix renaming on windows
buf.type is ItemFileSkip because csync_vio_local_stat does not set this field
2020-12-15 10:58:16 +01:00
Olivier Goffart
beee123c80
New Discovery: Fix trailing slash causing failure on windows 2020-12-15 10:58:16 +01:00
Olivier Goffart
e20e1d110f
Move: Fix too many starting slashes in the destination header
QDir::cleanPath does not remove starting slashes on windows.
So use account::davUrl which is already cleaned

Issue: #6824
2020-12-15 10:58:15 +01:00
Olivier Goffart
35967fc2d6
OAuth2: Refresh the token without aborting the sync
OAuth2 access token typically only has a token valid for 1 hour.
Before this patch, when the token was timing out during the sync, the
sync was aborted, and the ConnectionValidator was then requesting a new
token, so the sync can be started over.
If the discovery takes longer than the oauth2 validity, this means that
the sync can never proceed, as it would be always restarted from scratch.

With this patch, we try to transparently renew the OAuth2 token and restart
the jobs that failed because the access token was invalid.

Note that some changes were required in the GETFile job because it handled
the error itself and so it was erroring the jobs before its too late.

Issue #6814
2020-12-15 10:58:14 +01:00
Olivier Goffart
53a14c2041
HttpCredentials: initialize all member inline 2020-12-15 10:58:14 +01:00
Christian Kamm
00edcf98a1
Discovery: Virtual file handling adjustments
- adjust virtual file path handing
- helpers for vfs suffix adding/removal
- helpers for isDirectory/isVirtual on SyncJournalRecords
- be clear about what PathTuple _local/_server mean
2020-12-15 10:58:14 +01:00
Olivier Goffart
58eaf9940a
Database: Add an index on the parent path
So we can quickly query the items in a parent directory

This uses a custom slite3 function, which means that when downgrading the client,
or using another tool to add entries in the database, any insertion in the metadata
table will produce an error: "unknown function: parent_hash()"
(This will crash the client 2.5)
2020-12-15 10:58:13 +01:00
Olivier Goffart
e45e57982d
Add missing export 2020-12-15 10:58:13 +01:00
Christian Kamm
b431f4ef0c
Discovery: Fix log output for instructions 2020-12-15 10:58:13 +01:00
Christian Kamm
ce420d77a8
Discovery: Don't rebuild invalidFilname regex each call 2020-12-15 10:58:13 +01:00
Christian Kamm
113124cde5
Discovery: Introduce smaller functions 2020-12-15 10:58:13 +01:00
Christian Kamm
4f6f706f40
Excludes: drop csyncTraversalMatchFun()
The new discovery can call the traversal match function directly.
2020-12-15 10:58:13 +01:00
Christian Kamm
60de1c9720
virtual files: show option only when branding allows it 2020-12-15 10:58:12 +01:00
Olivier Goffart
18f6e346b8
Upload: Do not error out if the server do not send the X-OC-MTime: accepted header
If the server does not set the mtime, it is not a big problem for the
synchronisation.

The test was used before so we could do a PROPPATCH for server that did not
support this header. But now that all server supports that we don't need to
to the check. (We do not do the PROPPATCH since we got rid of the neon
dependency)

Apparently, it may happen that some backend don't support setting mtime
and this can lead to this error.

https://github.com/owncloud/client/issues/6797
2020-12-15 10:58:11 +01:00
Olivier Goffart
15eab07866
OAuth2: Store 'Account::davUser' in the config, and use that user for connecting
We need to use the user id to check if we are connected to the right account.
These might be different from the HTTP Basic Auth login. (LDAP setups)

When the account was configured as an oauth2 account form the wisard, the
http_user was already set correctly to the user id. But when the server is
upgrading from basic auth to oauth2, we need to pick the right login.

Note that Account::davUser() already defaults to the HTTP user when none
is set, so this means the upgrade will be fine if this is not set in the
config.

Issues:
https://github.com/owncloud/oauth2/issues/109
https://github.com/owncloud/enterprise/issues/2781
2020-12-15 10:58:11 +01:00
Olivier Goffart
75f66ddaa1
Local discovery: always recurse within touched directory
If the file system watcher tells us a directory was modified, we should
recurse into it because it means it is probably a new directory

Issue #6804
2020-12-15 10:58:11 +01:00
Christian Kamm
e10e953c66
PropagateUpload: Avoid crash due to cascading aborts
https://sentry.io/owncloud/desktop-win-and-mac/issues/698694072/activity/
2020-12-15 10:58:11 +01:00
Christian Kamm
d1aedcfd3c
Discovery: restructure processFileAnalyzeLocalInfo 2020-12-15 10:58:10 +01:00
Christian Kamm
a2839bd40a
Discovery: easy conditions first in processFileAnalyzeRemoteInfo
Removing two levels of indent that way
2020-12-15 10:58:10 +01:00
Christian Kamm
76341904e9
Discovery: Add comments 2020-12-15 10:58:10 +01:00
Olivier Goffart
5a1c93d0ac
Discovery: make sure finished is not called twice, even in case of errors 2020-12-15 10:58:10 +01:00
Olivier Goffart
b86e1efc9a
Remove the backup deteciton code which was used for server < 9.1 2020-12-15 10:58:10 +01:00
Olivier Goffart
f666511a4b
Discovery: Remove stale DB entries
And test the Remove/Remove case.

This means we need to always query the database for all the entries.
This showed another small bug in the test in which sync item for virtual
files at the root could have a slash in front of them.
2020-12-15 10:58:10 +01:00
Christian Kamm
5683278fab
Discovery: Comments and visibility adjustments 2020-12-15 10:58:09 +01:00
Olivier Goffart
aa18e10ff5
Discovery: cleanups and comments 2020-12-15 10:58:09 +01:00
Olivier Goffart
b10b3e5eeb
Discovery: move checkMovePermissions to its own function 2020-12-15 10:58:09 +01:00
Olivier Goffart
edd866b32b
Discovery: Adjust the instruction in case of resolved conflict
When resolving a conflict because the file was just updated on the server,
we write all the metadata on the database immediatly, so INSTRUCITON_NONE
is enough and UPDATE_METADATA is not needed
2020-12-15 10:58:09 +01:00
Olivier Goffart
7cddaf82ab
ProcessDirectoryJob: always set _childModified to true, regardless the direction
This was like that to handle the case of CSYNC_INSTRUCTION_TYPE_CHANGE, but just add
a condition in this location.
2020-12-15 10:58:09 +01:00
Olivier Goffart
46510c2f39
Discovery phase: refactor some code in DiscoveryPhase::findAndCancelDeletedJob
Less code duplication
2020-12-15 10:58:09 +01:00
Olivier Goffart
1c2a3279bb
New Discovery Algorithm: more cleanups
- rename progress() to be more explicit
 - Make some more member of the discovery phase private
2020-12-15 10:58:09 +01:00