Issue #7506
This is a regression introduced by the delta sync feature (as the chunk offset
changed from being the chunk number to be the byte offset, it needs to be a
qint64 now)
Previously the source was deleted (or attempted to be deleted), even if
the new location was not acceptable for upload. This could make data
unavilable on the server.
For #7410
By introducing a PropagateRootDirectory job that explicitly
separates the directory deletion jobs from all the other jobs.
Note that this means that if there are errors in subJobs the
dirDeletionJobs won't get executed.
Previously the pin states of deleted files stayed in the 'flags'
database and could be inadvertently reused when a new file with the same
name appeared. Now they are deleted.
To make this work right, the meaning of the 'path' column in the 'flags'
table was changed: Previously it never had the .owncloud file suffix.
Now it's the same as in metadata.path.
This takes the safe parts from #7274 for inclusion in 2.6. The more
elaborate database schema changes (why use 'path' the join the two
tables in the first place?) shall go into master.
Previously RequestEtagJob did return the etag verbatim (including extra
quotes) while the db had the parsed form. That caused the etag
comparison during discovery move detection to always fail. The test
didn't catch it because the etags there didn't have quotes.
Now:
- RequestEtagJob will parse the etag, leading to a consistent format
- Tests have etags with quotes, detecting the problem
If one adds a new file to an online-only folder the previous behavior
was to upload the file in one sync and dehydrate it in the next. Now
these new files get set to Unspecified pin state, making them retain
their data.
Since Qt does not yet transparently resend HTTP2 requests in some cases
we do it manually.
The test showed a problem where the initial non-200 reply would close
the target temporary file and the follow-up request couldn't store any
data. Removing that close() call is safe because there also is a
_saveBodyToFile flag that guards writes to the target file.
Previously these result codes during remote discovery of the sync root
would not cause an error and the discovery would get stuck.
Also extends RemoteDiscovery tests to check for errors on the root item.
The idea is that the user's question is "is this folder's data available
offline?" and not "does this folder have AlwaysLocal pin state?".
The the answers to the two questions can differ: an always-local
folder can have subitems that are not always-local and are dehydrated.
The new availability enum intends to describe the answer to the user's
actual question and can be derived from pin states. If pin states aren't
stored in the database the way of calculating availability will depend
on the vfs plugin.
The pin state is a per-item attribute that has an effect on _type:
AlwaysLocal dehydrated files will be marked for hydration and OnlineOnly
hydrated files will be marked for dehydration.
Where exactly this effect materializes depends on how the pin states are
stored. If they're stored in the db (suffix) the dbEntry._type is
changed during the discovery.
If the pin state is stored in the filesystem, the localEntry._type must
be adjusted by the plugin's stat callback.
This patch makes pin states behave more consistently between plugins.
Previously with suffix-vfs pin states only had an effect on new remote
files. Now the effect of pinning or unpinning files or directories is as
documented and similar to other plugins.
Previously removing the vfs suffix of a file always triggered a
conflict. Now it may just cause a file download.
This was done because users expected symmetry in the rename actions and
renaming foo -> foo.owncloud already triggers the "make the file
virtual" action. Now foo.owncloud -> foo triggers the "download the
contents" action.
Users can rename a file *and* add/remove the vfs suffix at the same time
leading to very complex sync actions. This patch doesn't add support for
them, but adds tests and makes sure these cases do not cause unintened
behavior.
The rename will be propagated, but the users's hydrate/dehydrate request
will be ignored.
This could fix a problem where the client incorrectly decides to delete
local data.
Previously any sqlite3_step() return value that wasn't SQLITE_ROW would
be interpreted as "there's no more data here". Thus an sqlite error at a
bad time could cause the remote discovery to fail to read an unchanged
subtree from the database. These files would then be deleted locally.
With this change sqlite errors from sqlite3_step are detected and
logged. For the particular case of SyncJournalDb::getFilesBelowPath()
the error will now be propagated and the sync run will fail instead of
performing spurious deletes.
Note that many other database functions still don't distinguish
not-found from error cases. Most of them won't have as severe effects on
affected sync runs though.
As far as I'm aware local discovery can be skipped on folders that are
selective-sync blacklisted, so a local discovery is required when an
entry is removed from the blacklist.
Also rename
avoidReadFromDbOnNextSync() -> schedulePathForRemoteDiscovery()
since the old name might also imply it's not read from db in the local
discovery - which is not the case. Use Folder::
schedulePathForLocalDiscovery() for that.
Creating a new virtual file and replacing a file with a virtual one now
have their own text in the protocol, not just "Downloaded".
To do this, the SyncFileItem type is kept as
ItemTypeVirtualFileDehydration for these actions. Added new code to
ensure the type isn't written to the database.
While looking at this, I've also added documentation on SyncFileItem's
_file, _renameTarget, _originalFile and destination() because some of
the semantics weren't clear.
That change will be useful for the notifications. Previously the
dehydrated files were reported as "newly downloaded", now they're
reported as "updated".
- SyncJournalDB functions now behind internalPinStates() to avoid
accidental usage, when nearly everyone should go through Vfs.
- Rename Vfs::getPinState() to Vfs::pinState()
Any folder with a (potentially deeply) contained error will have
StatusWarning. StatusExcluded marks exclusions. The difference is useful
to know for VFS.
On Linux and Windows the file watcher can't distinguish between changes
that were caused by the process itself, like during a sync operation,
and external changes. To work around that the client keeps a list of
files it has touched and blocks notifications on these files for a bit.
The duration of this block was originally and arbitrarily set at 15
seconds. During manual tests I regularly thought there was a bug when
syncs didn't trigger, when the only problem was that my changes happened
too close to a previous sync operation.
This change reduces the duration to three seconds. I imagine that this
is still enough.
Also use std::chrono while at it.
The idea is to allow folders (and later maybe files?) to be
- pinned to be available locally
- pinned to be online only
- inherit their pin from the parent
Where this pinning only controls the default for new files.
Subfolders may have a different pin state, and contained files
may be hydrated or dehydrated based on user actions.
This value is stored in a new 'flags' table. The idea is to store
data there that doesn't necessarily exist for each metadata entry.
The selective sync state could be migrated to this table.
This is just a port to QtTest, I did not change the layout of the test.
I did search and replace to replace the assert with QCOMPARE/QVERIFY
I still call setup and setup_init like before (only explicitly, now)
Also ported the preformence tests to QBENCHMAK because windows don't have
gettimeofday.
Relates #6358
This is to avoid issues on OSX, where the ._ prefix has special meaning.
Originally (before 2.3.2) ._ was necessary to guarantee exclusion. But
since then the .sync_ prefix is excluded as well.
This does not affect existing database files.
This helps support 2.5 settings where there are virtual files in the
tree but new files aren't created virtual.
It's also a prelude for #6815
There's currently no way of
- upgrading vfs plugins (a silent suffix->winvfs upgrade is attempted
once only, when moving to master)
- disabling vfs capabilities outright
Inh most case we already have a record from before, so avoid doing a useless
lookup in the database.
In owncloudpropagator.cpp, directories do not have a checksum so no need
to call a function that preserves it
