If one adds a new file to an online-only folder the previous behavior
was to upload the file in one sync and dehydrate it in the next. Now
these new files get set to Unspecified pin state, making them retain
their data.
Instead of all at once, to reduce peak memory use.
Changing UploadDevice in this way requires keeping the file open for the
duration of the upload. It also means changes to open(), seek(), close()
to ensure that uses of the device work right when a request needs to
be resent.
Since Qt does not yet transparently resend HTTP2 requests in some cases
we do it manually.
The test showed a problem where the initial non-200 reply would close
the target temporary file and the follow-up request couldn't store any
data. Removing that close() call is safe because there also is a
_saveBodyToFile flag that guards writes to the target file.
The previous patch ensured that the sqlite temporaries weren't deleted
and recreated for every sync run, but there was still time between
client startup and the first sync run where they would have the
"needs-sync" icon.
Previously "no-availability" meant db-error and querying the
availability of a nonexistant path returned AllHydrated.
Now, the availability has a DbError and a NoSuchItem error case.
The db-close operation is likely a leftover from when the SyncEngine
owned its own db connection and serves no purpose anymore.
Closing the db causes the removal of the temporary wal and shm files.
These files are recreated when the db is opened again, which happens
almost immediately.
This is a problem for winvfs because the delete-recreate step wipes the
exclusion state on these files just after the sync is done. That meant
that the db temporaries permanently had a "needs sync" icon marker shown
in the explorer.
Avoiding reopening the db also reduces the number of log messages per
sync.
Previously these result codes during remote discovery of the sync root
would not cause an error and the discovery would get stuck.
Also extends RemoteDiscovery tests to check for errors on the root item.
Previously if one set the instruction to ERROR while forgetting to set
an error status, it'd propagate as FileIgnored. Now the default is
NormalError for INSTRUCTION_ERROR and FileIgnored for
INSTRUCTION_IGNORE.
The idea is that the user's question is "is this folder's data available
offline?" and not "does this folder have AlwaysLocal pin state?".
The the answers to the two questions can differ: an always-local
folder can have subitems that are not always-local and are dehydrated.
The new availability enum intends to describe the answer to the user's
actual question and can be derived from pin states. If pin states aren't
stored in the database the way of calculating availability will depend
on the vfs plugin.
The pin state is a per-item attribute that has an effect on _type:
AlwaysLocal dehydrated files will be marked for hydration and OnlineOnly
hydrated files will be marked for dehydration.
Where exactly this effect materializes depends on how the pin states are
stored. If they're stored in the db (suffix) the dbEntry._type is
changed during the discovery.
If the pin state is stored in the filesystem, the localEntry._type must
be adjusted by the plugin's stat callback.
This patch makes pin states behave more consistently between plugins.
Previously with suffix-vfs pin states only had an effect on new remote
files. Now the effect of pinning or unpinning files or directories is as
documented and similar to other plugins.
1. The _firstJob is usually deleted by the time the PropagateDirectory
finishes. (deleteLater() is called early)
2. The PropagateDirectory::_item and PropagateRemoteMkdir::_item point
to the same SyncFileItem anyway. This code is a leftover from when
each job had its own instance.
Previously removing the vfs suffix of a file always triggered a
conflict. Now it may just cause a file download.
This was done because users expected symmetry in the rename actions and
renaming foo -> foo.owncloud already triggers the "make the file
virtual" action. Now foo.owncloud -> foo triggers the "download the
contents" action.
Users can rename a file *and* add/remove the vfs suffix at the same time
leading to very complex sync actions. This patch doesn't add support for
them, but adds tests and makes sure these cases do not cause unintened
behavior.
The rename will be propagated, but the users's hydrate/dehydrate request
will be ignored.
This was not required with 2.5 because a size of 0 was ignorted when comparing
size by the csync updater, to be compatible with very old version of the database.
But the we discovery will still think the file is changed if the database contains
a size of 0
It still reads and writes the old format too, but all newly stored
client certs will be in the new form.
For #6776 because Windows limits credential data to 512 bytes in older
versions.
By default, plugins are only searched next to the binary or next to the
other Qt plugins. This optional build variable allows another path to be
configured.
The idea is that on linux the oC packaging probably wants the binary in
something like /opt/owncloud/bin and the plugins in
/opt/owncloud/lib/plugins.
Similarly, distribution packagers probably don't want the plugins next
to the binary or next to the other Qt plugins. This flag allows them to
configure another path that the executable will look in.
As far as I'm aware local discovery can be skipped on folders that are
selective-sync blacklisted, so a local discovery is required when an
entry is removed from the blacklist.
Also rename
avoidReadFromDbOnNextSync() -> schedulePathForRemoteDiscovery()
since the old name might also imply it's not read from db in the local
discovery - which is not the case. Use Folder::
schedulePathForLocalDiscovery() for that.
Creating a new virtual file and replacing a file with a virtual one now
have their own text in the protocol, not just "Downloaded".
To do this, the SyncFileItem type is kept as
ItemTypeVirtualFileDehydration for these actions. Added new code to
ensure the type isn't written to the database.
While looking at this, I've also added documentation on SyncFileItem's
_file, _renameTarget, _originalFile and destination() because some of
the semantics weren't clear.
That change will be useful for the notifications. Previously the
dehydrated files were reported as "newly downloaded", now they're
reported as "updated".
That just complicated things. It's ok if Vfs is not a fully abstract
interface class.
The pinstate-in-db methods are instead provided directly on Vfs and
VfsSuffix and VfsOff use them to implement pin states.
The start() method is simply non-virtual and calls into startImpl() for
the plugin-specific startup code.
The block of code that propagated attributes etc from the previously
existing file was placed *after* the block that renamed the previously
existing file to a conflict name. That meant the propagation didn't work
in the conflict case.
- SyncJournalDB functions now behind internalPinStates() to avoid
accidental usage, when nearly everyone should go through Vfs.
- Rename Vfs::getPinState() to Vfs::pinState()
Any folder with a (potentially deeply) contained error will have
StatusWarning. StatusExcluded marks exclusions. The difference is useful
to know for VFS.
This will be used in conjunction with vfs plugins that detect whether a
file has a pending hydration/dehydration through independent means and
communicate that to the discovery through local file type.
Since 'placeholder' just means that it's an item of the special type
that the vfs plugin can deal with - no matter whether hydrated or
dehydrated - all done items should become placeholders. Even
directories.
Now every file that passes through updateMetadata() will be converted to
a placeholder if necessary.
On Linux and Windows the file watcher can't distinguish between changes
that were caused by the process itself, like during a sync operation,
and external changes. To work around that the client keeps a list of
files it has touched and blocks notifications on these files for a bit.
The duration of this block was originally and arbitrarily set at 15
seconds. During manual tests I regularly thought there was a bug when
syncs didn't trigger, when the only problem was that my changes happened
too close to a previous sync operation.
This change reduces the duration to three seconds. I imagine that this
is still enough.
Also use std::chrono while at it.
Also, calling deleteLater() on jobs is unnecessary (they autodelete
after finished()) and deleting the attached QSettings is also
unnecessary because the settings object is parented to the job.
slotDiscoveryJobFinished -> slotDiscoveryFinished
slotFinished -> slotPropagationFinished
This should be clearer. Particular the
slotFinished -> finalize -> emit finished()
chain was confusing before.
This helps support 2.5 settings where there are virtual files in the
tree but new files aren't created virtual.
It's also a prelude for #6815
There's currently no way of
- upgrading vfs plugins (a silent suffix->winvfs upgrade is attempted
once only, when moving to master)
- disabling vfs capabilities outright
Inh most case we already have a record from before, so avoid doing a useless
lookup in the database.
In owncloudpropagator.cpp, directories do not have a checksum so no need
to call a function that preserves it
When enabling TOKEN_AUTH_ONLY, the code path using QPainter is disabled.
So we also don't need the includes.
This header is not available for Remarkable.
The E2EE code path would get the engine to go wrong in case of db error.
It's just better to have a failing upload or failing mkdir later in those
cases.
Emitting signals from a ctor is a bad idea anyway
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
OAuth2 access token typically only has a token valid for 1 hour.
Before this patch, when the token was timing out during the sync, the
sync was aborted, and the ConnectionValidator was then requesting a new
token, so the sync can be started over.
If the discovery takes longer than the oauth2 validity, this means that
the sync can never proceed, as it would be always restarted from scratch.
With this patch, we try to transparently renew the OAuth2 token and restart
the jobs that failed because the access token was invalid.
Note that some changes were required in the GETFile job because it handled
the error itself and so it was erroring the jobs before its too late.
Issue #6814
- adjust virtual file path handing
- helpers for vfs suffix adding/removal
- helpers for isDirectory/isVirtual on SyncJournalRecords
- be clear about what PathTuple _local/_server mean
So we can quickly query the items in a parent directory
This uses a custom slite3 function, which means that when downgrading the client,
or using another tool to add entries in the database, any insertion in the metadata
table will produce an error: "unknown function: parent_hash()"
(This will crash the client 2.5)
If the server does not set the mtime, it is not a big problem for the
synchronisation.
The test was used before so we could do a PROPPATCH for server that did not
support this header. But now that all server supports that we don't need to
to the check. (We do not do the PROPPATCH since we got rid of the neon
dependency)
Apparently, it may happen that some backend don't support setting mtime
and this can lead to this error.
https://github.com/owncloud/client/issues/6797
We need to use the user id to check if we are connected to the right account.
These might be different from the HTTP Basic Auth login. (LDAP setups)
When the account was configured as an oauth2 account form the wisard, the
http_user was already set correctly to the user id. But when the server is
upgrading from basic auth to oauth2, we need to pick the right login.
Note that Account::davUser() already defaults to the HTTP user when none
is set, so this means the upgrade will be fine if this is not set in the
config.
Issues:
https://github.com/owncloud/oauth2/issues/109https://github.com/owncloud/enterprise/issues/2781
And test the Remove/Remove case.
This means we need to always query the database for all the entries.
This showed another small bug in the test in which sync item for virtual
files at the root could have a slash in front of them.
When resolving a conflict because the file was just updated on the server,
we write all the metadata on the database immediatly, so INSTRUCITON_NONE
is enough and UPDATE_METADATA is not needed
The previous code considered the also HTTP 207 code without the
application/xml header to have this message.
httpCode 0 does not make much sense anyway.
This change the behavior to consider any 2xx without the xml header
to show this error message
Since we do not recurse within some directories, many files are not seen.
The stale entry will cleanup by themself as the sync engine try to remove
the files that are already removed.
Should we need to actually do this cleanup, it should be dotected in the
discovery.
Move the finialization in its own function.
This allow to save a bit of code duplication.
Also change the order of the parameter in the constructor for consistency
Unfortunately to do this, the local update phase must write to the
database, creating a new side-effect and order dependency (local update
must run before remote update).
Fixes two bugs that appeared since the introduction of the struct:
- when reading permissions from the journal, null ("") was read as
empty-not-null
- when reading permissinos from the server, empty ("") was read as null
Addresses #4608
The check was added for #6317 in commit
13eb64584f.
We did see missing mtimes in replies in tests with live servers though.
Possibly those were old incomplete responses cached in the stat cache?
Installing to lib/${APPLICATION_EXECUTABLE} has caused a bunch of
irritations in the past and subtle annoying to fix bugs. To avoid name
clashes with branded clients ${APPLICATION_EXECUTABLE} becomes now
part of the filename instead of the subfolder.
The concrete motivation to change this now is that on Windows there
is no RPATH and it's not possible to run owncloud directly from the
Craft Root folder, which is nice when you're developing on Windows.
It would have been possible to change this just for Windows but as
written earlier this has caused lots of issues and thus I think it's
a good idea to just stay consistent accross platforms when touching it.
The problem here is that we were sometimes allocating the error_string with
qstrdup, which need to be released with delete[] and not free().
Simplify the code by using QString instead.
```
==7230== Mismatched free() / delete / delete []
==7230== at 0x4C2E10B: free (vg_replace_malloc.c:530)
==7230== by 0x57C2321: csync_s::reinitialize() (csync.cpp:247)
==7230== by 0x548130F: OCC::SyncEngine::finalize(bool) (syncengine.cpp:1212)
==7230== by 0x5481223: OCC::SyncEngine::handleSyncError(csync_s*, char const*) (syncengine.cpp:746)
==7230== by 0x5483E78: OCC::SyncEngine::slotDiscoveryJobFinished(int) (syncengine.cpp:965)
==7230== by 0x5495E34: QtPrivate::FunctorCall<QtPrivate::IndexesList<0>, QtPrivate::List<int>, void, void (OCC::SyncEngine::*)(int)>::call(void (OCC::SyncEngine::*)(int), OCC::SyncEngine*, void**) (qobjectdefs_impl.h:134)
==7230== by 0x5495D92: void QtPrivate::FunctionPointer<void (OCC::SyncEngine::*)(int)>::call<QtPrivate::List<int>, void>(void (OCC::SyncEngine::*)(int), OCC::SyncEngine*, void**) (qobjectdefs_impl.h:167)
==7230== by 0x5495CB5: QtPrivate::QSlotObject<void (OCC::SyncEngine::*)(int), QtPrivate::List<int>, void>::impl(int, QtPrivate::QSlotObjectBase*, QObject*, void**, bool*) (qobjectdefs_impl.h:396)
==7230== by 0xA9BF2E1: QObject::event(QEvent*) (in /usr/lib/libQt5Core.so.5.11.0)
==7230== by 0x64BE983: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib/libQt5Widgets.so.5.11.0)
==7230== by 0x64C625A: QApplication::notify(QObject*, QEvent*) (in /usr/lib/libQt5Widgets.so.5.11.0)
==7230== by 0xA994BC8: QCoreApplication::notifyInternal2(QObject*, QEvent*) (in /usr/lib/libQt5Core.so.5.11.0)
==7230== Address 0x225b2640 is 0 bytes inside a block of size 50 alloc'd
==7230== at 0x4C2DC6F: operator new[](unsigned long) (vg_replace_malloc.c:423)
==7230== by 0xA7E8FC8: qstrdup(char const*) (in /usr/lib/libQt5Core.so.5.11.0)
==7230== by 0x53F5750: OCC::DiscoveryJob::remote_vio_opendir_hook(char const*, void*) (discoveryphase.cpp:666)
==7230== by 0x57E1278: csync_vio_opendir(csync_s*, char const*) (csync_vio.cpp:39)
==7230== by 0x57D718F: csync_ftw(csync_s*, char const*, int (*)(csync_s*, std::unique_ptr<csync_file_stat_s, std::default_delete<csync_file_stat_s> >), unsigned int) (csync_update.cpp:674)
==7230== by 0x57C1B05: csync_update(csync_s*) (csync.cpp:109)
==7230== by 0x53F5BCC: OCC::DiscoveryJob::start() (discoveryphase.cpp:718)
==7230== by 0x54B8F74: OCC::DiscoveryJob::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) (moc_discoveryphase.cpp:494)
==7230== by 0xA9BF2E1: QObject::event(QEvent*) (in /usr/lib/libQt5Core.so.5.11.0)
==7230== by 0x64BE983: QApplicationPrivate::notify_helper(QObject*, QEvent*) (in /usr/lib/libQt5Widgets.so.5.11.0)
==7230== by 0x64C625A: QApplication::notify(QObject*, QEvent*) (in /usr/lib/libQt5Widgets.so.5.11.0)
==7230== by 0xA994BC8: QCoreApplication::notifyInternal2(QObject*, QEvent*) (in /usr/lib/libQt5Core.so.5.11.0)
==7230==
```
We want to keep the body so we can get the message from it
(Issue #6459)
TestDownload::testErrorMessage did not fail because the FakeErrorReply
did not emit readyRead and did not implement bytesAvailable.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
Otherwise a interrupted or unsuccessful download would mean that the
download-intend was forgotten. The next sync would reestablish the
virtual file instead.
Also, if there is too-new configuration, backup the file, show a
warning message asking the user whether it's ok to discard the
configuration from the future.
See #6504
Issue #6420
Store the X-Request-ID in the SyncFileItem and also in the blacklist.
Note that for consistency reason, the X-Request-ID is also in the
SyncFileItem if the request succeeds.
Currently there is no UI to access it, but it can be queried with sql
commands
Similar to the use for the checksum.
I know that zlib is required in principle, but i don't have it
in one of my test building environment, and it is easier to just
disable it.
From issue #7015, the code is wrong because the path is the file system path and
not the path on the DB.
But since this is a conflict, this means the reconcile will still want to download
the file from the server next sync, so we need not to worry about this case
Now the db entries for placeholders will have the full placeholder
paths. That way older clients will, on remote discovery, delete the
placeholders and download the real files.
- Controled by an option.
- New remote files start out as ItemTypePlaceholder, are created with a
.owncloud extension.
- When their db entry is set to ItemTypePlaceholderDownload the next
sync run will download them.
- Files that aren't in the placeholder state sync as usual.
- See test cases in testsyncplaceholders.
Missing:
- User ui for triggering placeholder file download
- Maybe: Going back from file to placeholder?
QtKeychain provides Qt5KeychainConfig.cmake and friends nowadays, so no
need to have a less reliable and outdated find module on our end.
Also this shows that we were including keychain.h in the wrong way and
were not using the link target, so both got fixed as well.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
For conflicts generally as well as new files in read-only directories
the context menu will now present delete and move options.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
This way we avoid the expensive SQL query on the server at the price of
more round-trips since we're doing the recursive traversal by hand now.
Also it turns out this depth was used for all the other propfind calls
during sync when we want fresher information regarding a folder. This
was very inefficient in all cases and won't happen anymore.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
Now that they are used from gui/ they need to be properly exported so
that linking doesn't fail when visibility is activated (only on our
Windows build it seems).
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
It is better to rely on the Logger state to know exactly where we're
logging. Indeed due to the the various ways to impact its state the
config alone might not now where we're logging.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
Turn on the logging by default for everyone. Let's use a log dir within
the config directory of the application and have debug logs expiring
after a day.
This obviously means we'll generate quite some logs but with the
automated compression it shouldn't be too horrible. Obviously that
scales with the amount of files and syncs occurring. In our tests with a
large setup we're around 100 MB for a day worth of logs, this shouldn't
be too much of an issue on today's average desktop/laptop.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
APPLICATION_SERVER_URL can be either empty or be specified. This commit adds the new CMake option APPLICATION_SERVER_URL_ENFORCE to decide whether to enforce the
URL's unmodified use (like before, default: ON) or to allow modification by the user (new).
By default APPLICATION_SERVER_URL_ENFORCE is set to ON, to no break with the previous implementation's expectations.
If APPLICATION_SERVER_URL is empty, APPLICATION_SERVER_URL_ENFORCE will be ignored by the Account Wizard.
The previous behaviour confused me a bit with branded builds. When the URL was (usually) specified but not forced, it was simply discarded, forcing the user to
manually supply it.
Signed-off-by: Michael Schuster <michael@schuster.ms>
We don't do much with that mimetype on our end, but other clients
somehow don't expect inode/directory to let's lie. ;-)
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
Now the only remaining reference is in:
src/3rdparty/libcrashreporter-qt/3rdparty/drkonqi-parser/backtraceparser.cpp:269: || line.functionName().startsWith(QLatin1String("QLinkedList")
Signed-off-by: Michael Schuster <michael@schuster.ms>
