The idea is that the user's question is "is this folder's data available
offline?" and not "does this folder have AlwaysLocal pin state?".
The the answers to the two questions can differ: an always-local
folder can have subitems that are not always-local and are dehydrated.
The new availability enum intends to describe the answer to the user's
actual question and can be derived from pin states. If pin states aren't
stored in the database the way of calculating availability will depend
on the vfs plugin.
The pin state is a per-item attribute that has an effect on _type:
AlwaysLocal dehydrated files will be marked for hydration and OnlineOnly
hydrated files will be marked for dehydration.
Where exactly this effect materializes depends on how the pin states are
stored. If they're stored in the db (suffix) the dbEntry._type is
changed during the discovery.
If the pin state is stored in the filesystem, the localEntry._type must
be adjusted by the plugin's stat callback.
This patch makes pin states behave more consistently between plugins.
Previously with suffix-vfs pin states only had an effect on new remote
files. Now the effect of pinning or unpinning files or directories is as
documented and similar to other plugins.
Previously removing the vfs suffix of a file always triggered a
conflict. Now it may just cause a file download.
This was done because users expected symmetry in the rename actions and
renaming foo -> foo.owncloud already triggers the "make the file
virtual" action. Now foo.owncloud -> foo triggers the "download the
contents" action.
Users can rename a file *and* add/remove the vfs suffix at the same time
leading to very complex sync actions. This patch doesn't add support for
them, but adds tests and makes sure these cases do not cause unintened
behavior.
The rename will be propagated, but the users's hydrate/dehydrate request
will be ignored.
This could fix a problem where the client incorrectly decides to delete
local data.
Previously any sqlite3_step() return value that wasn't SQLITE_ROW would
be interpreted as "there's no more data here". Thus an sqlite error at a
bad time could cause the remote discovery to fail to read an unchanged
subtree from the database. These files would then be deleted locally.
With this change sqlite errors from sqlite3_step are detected and
logged. For the particular case of SyncJournalDb::getFilesBelowPath()
the error will now be propagated and the sync run will fail instead of
performing spurious deletes.
Note that many other database functions still don't distinguish
not-found from error cases. Most of them won't have as severe effects on
affected sync runs though.
As far as I'm aware local discovery can be skipped on folders that are
selective-sync blacklisted, so a local discovery is required when an
entry is removed from the blacklist.
Also rename
avoidReadFromDbOnNextSync() -> schedulePathForRemoteDiscovery()
since the old name might also imply it's not read from db in the local
discovery - which is not the case. Use Folder::
schedulePathForLocalDiscovery() for that.
Creating a new virtual file and replacing a file with a virtual one now
have their own text in the protocol, not just "Downloaded".
To do this, the SyncFileItem type is kept as
ItemTypeVirtualFileDehydration for these actions. Added new code to
ensure the type isn't written to the database.
While looking at this, I've also added documentation on SyncFileItem's
_file, _renameTarget, _originalFile and destination() because some of
the semantics weren't clear.
That change will be useful for the notifications. Previously the
dehydrated files were reported as "newly downloaded", now they're
reported as "updated".
- SyncJournalDB functions now behind internalPinStates() to avoid
accidental usage, when nearly everyone should go through Vfs.
- Rename Vfs::getPinState() to Vfs::pinState()
Any folder with a (potentially deeply) contained error will have
StatusWarning. StatusExcluded marks exclusions. The difference is useful
to know for VFS.
On Linux and Windows the file watcher can't distinguish between changes
that were caused by the process itself, like during a sync operation,
and external changes. To work around that the client keeps a list of
files it has touched and blocks notifications on these files for a bit.
The duration of this block was originally and arbitrarily set at 15
seconds. During manual tests I regularly thought there was a bug when
syncs didn't trigger, when the only problem was that my changes happened
too close to a previous sync operation.
This change reduces the duration to three seconds. I imagine that this
is still enough.
Also use std::chrono while at it.
The idea is to allow folders (and later maybe files?) to be
- pinned to be available locally
- pinned to be online only
- inherit their pin from the parent
Where this pinning only controls the default for new files.
Subfolders may have a different pin state, and contained files
may be hydrated or dehydrated based on user actions.
This value is stored in a new 'flags' table. The idea is to store
data there that doesn't necessarily exist for each metadata entry.
The selective sync state could be migrated to this table.
This is just a port to QtTest, I did not change the layout of the test.
I did search and replace to replace the assert with QCOMPARE/QVERIFY
I still call setup and setup_init like before (only explicitly, now)
Also ported the preformence tests to QBENCHMAK because windows don't have
gettimeofday.
Relates #6358
This is to avoid issues on OSX, where the ._ prefix has special meaning.
Originally (before 2.3.2) ._ was necessary to guarantee exclusion. But
since then the .sync_ prefix is excluded as well.
This does not affect existing database files.
This helps support 2.5 settings where there are virtual files in the
tree but new files aren't created virtual.
It's also a prelude for #6815
There's currently no way of
- upgrading vfs plugins (a silent suffix->winvfs upgrade is attempted
once only, when moving to master)
- disabling vfs capabilities outright
Inh most case we already have a record from before, so avoid doing a useless
lookup in the database.
In owncloudpropagator.cpp, directories do not have a checksum so no need
to call a function that preserves it
And test the Remove/Remove case.
This means we need to always query the database for all the entries.
This showed another small bug in the test in which sync item for virtual
files at the root could have a slash in front of them.
Unfortunately to do this, the local update phase must write to the
database, creating a new side-effect and order dependency (local update
must run before remote update).
Fixes two bugs that appeared since the introduction of the struct:
- when reading permissions from the journal, null ("") was read as
empty-not-null
- when reading permissinos from the server, empty ("") was read as null
Addresses #4608
Installing to lib/${APPLICATION_EXECUTABLE} has caused a bunch of
irritations in the past and subtle annoying to fix bugs. To avoid name
clashes with branded clients ${APPLICATION_EXECUTABLE} becomes now
part of the filename instead of the subfolder.
The concrete motivation to change this now is that on Windows there
is no RPATH and it's not possible to run owncloud directly from the
Craft Root folder, which is nice when you're developing on Windows.
It would have been possible to change this just for Windows but as
written earlier this has caused lots of issues and thus I think it's
a good idea to just stay consistent accross platforms when touching it.
We want to keep the body so we can get the message from it
(Issue #6459)
TestDownload::testErrorMessage did not fail because the FakeErrorReply
did not emit readyRead and did not implement bytesAvailable.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
Otherwise a interrupted or unsuccessful download would mean that the
download-intend was forgotten. The next sync would reestablish the
virtual file instead.
Issue #6420
Store the X-Request-ID in the SyncFileItem and also in the blacklist.
Note that for consistency reason, the X-Request-ID is also in the
SyncFileItem if the request succeeds.
Currently there is no UI to access it, but it can be queried with sql
commands
To do this, we add the placeholder extension to the user exclude file
automatically. However, newer clients shouldn't use that exclude
pattern: so we also add version directives that allow making exclude
patterns dependent on the client version.
Now the db entries for placeholders will have the full placeholder
paths. That way older clients will, on remote discovery, delete the
placeholders and download the real files.
- Controled by an option.
- New remote files start out as ItemTypePlaceholder, are created with a
.owncloud extension.
- When their db entry is set to ItemTypePlaceholderDownload the next
sync run will download them.
- Files that aren't in the placeholder state sync as usual.
- See test cases in testsyncplaceholders.
Missing:
- User ui for triggering placeholder file download
- Maybe: Going back from file to placeholder?
Previously it depended on addFolder() / removeFolder() calls to adjust
watchers when new folders were added or removed. There also needed to be
complex move handling.
Now, any folder creation/move-in notifications automatically trigger
watcher additions and folder deletion/move-out triggers removal.
If a folder was renamed A -> B, the folder watcher for the inode
would be unaware and still report changes for A/foo. Now directory
renames in the watched folders are tracked and paths are updated
accordingly.
The socket api move and delete commands are not strictly about conflicts
since they also deal with files which couldn't be uploaded for some
other reason. Still the new ConflictSolver could be used in those cases.
This opens the door at reusing that logic in other places.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
For conflicts generally as well as new files in read-only directories
the context menu will now present delete and move options.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
This flagged mostly unused parameters. Didn't enable the
misc-non-private-member-variables-in-classes check as we got a lot of
those. Hopefully we'll get to fix them at some point but that feels too
early and too much work for now.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
Some of the comments didn't match the size or were missing. This also
means reducing one of the 150 MB payloads left behind reducing the
execution time by a few more seconds. This is now around 30s execution
time which is more acceptable.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
This is about the conflicts that happens when the file has been uploaded
correctly to the server, but the etag was not recieved because the connection
was closed before we got the reply.
We used to compare only the mtime when comparing the uploaded file and the
existing file. However, to be perfectly correct, we also should check the
size.
This was found because TestChunkingNG::connectionDroppedBeforeEtagRecieved is
flaky. Example of faillure found in https://drone.owncloud.com/owncloud/client/481/5
while testing PR #6626
(very trimmed log:)
06-29 07:58:02:015 [ info sync.csync.csync ]: ## Starting local discovery ##
06-29 07:58:02:016 [ info sync.csync.updater ]: Database entry found, compare: 1530259082 <-> 1530259051, etag: <-> 1644a8c8750, inode: 1935629 <-> 1935629, size: 301 <-> 300, perms: 0 <-> ff, type: 0 <-> 0, checksum: <-> SHA1:cc9adedebe27a6259efb8d6ed09f4f2eff559ad1, ignore: 0
06-29 07:58:02:016 [ info sync.csync.updater ]: file: A/a0, instruction: INSTRUCTION_EVAL <<=
06-29 07:58:02:972 [ warning sync.networkjob ]: QNetworkReply::NetworkError(OperationCanceledError) "Connection timed out" QVariant(Invalid)
.. next sync...
06-29 07:58:02:980 [ info sync.engine ]: #### Discovery start ####################################################
06-29 07:58:02:981 [ info sync.csync.csync ]: ## Starting local discovery ##
06-29 07:58:02:983 [ info sync.csync.updater ]: Database entry found, compare: 1530259082 <-> 1530259051, etag: <-> 1644a8c8750, inode: 1935629 <-> 1935629, size: 302 <-> 300, perms: 0 <-> ff, type: 0 <-> 0, checksum: <-> SHA1:cc9adedebe27a6259efb8d6ed09f4f2eff559ad1, ignore: 0
06-29 07:58:02:983 [ info sync.csync.updater ]: file: A/a0, instruction: INSTRUCTION_EVAL <<=
06-29 07:58:02:985 [ info sync.csync.csync ]: ## Starting remote discovery ##
06-29 07:58:02:985 [ info sync.networkjob ]: OCC::LsColJob created for "http://localhost/owncloud" + "" "OCC::DiscoverySingleDirectoryJob"
06-29 07:58:02:987 [ info sync.csync.updater ]: Database entry found, compare: 1530259082 <-> 1530259051, etag: 1644a8c8b26 <-> 1644a8c8750, inode: 0 <-> 1935629, size: 301 <-> 300, perms: ff <-> ff, type: 0 <-> 0, checksum: SHA1:5adcdac9608ae0811247f07f4cf1ab0a2ef99154 <-> SHA1:cc9adedebe27a6259efb8d6ed09f4f2eff559ad1, ignore: 0
06-29 07:58:02:987 [ info sync.csync.updater ]: file: A/a0, instruction: INSTRUCTION_EVAL <<=
06-29 07:58:02:989 [ info sync.csync.csync ]: Update detection for remote replica took 0.004 seconds walking 13 files
06-29 07:58:02:990 [ info sync.engine ]: #### Discovery end #################################################### 9 ms
06-29 07:58:02:990 [ info sync.database ]: Updating file record for path: "A/a0" inode: 1935629 modtime: 1530259082 type: 0 etag: "1644a8c8b26" fileId: "16383ea4" remotePerm: "WDNVCKR" fileSize: 301 checksum: "SHA1:cc9adedebe27a6259efb8d6ed09f4f2eff559ad1"
06-29 07:58:02:990 [ info sync.csync.reconciler ]: INSTRUCTION_UPDATE_METADATA client file: A/a0
06-29 07:58:02:990 [ info sync.csync.csync ]: Reconciliation for local replica took 0 seconds visiting 13 files.
06-29 07:58:02:990 [ info sync.csync.reconciler ]: INSTRUCTION_UPDATE_METADATA server dir: A
06-29 07:58:02:990 [ info sync.csync.csync ]: Reconciliation for remote replica took 0 seconds visiting 13 files.
06-29 07:58:02:990 [ info sync.engine ]: #### Reconcile end #################################################### 9 ms
06-29 07:58:02:990 [ info sync.database ]: Updating local metadata for: "A/a0" 1530259082 302 1935629
FAIL! : TestChunkingNG::connectionDroppedBeforeEtagRecieved(small file) '!fakeFolder.syncOnce()' returned FALSE. ()
Use smaller files so the test run faster.
Particulary usefull for TestChunkingNG::connectionDroppedBeforeEtagRecieved
Which had become so much slower after 2638332dc6
increased the timeout for bigger files
If we receive data without base64 encoding for encryption, it makes
sense to get it without base64 encoding out of decryption.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
Make the codebase consistent, we already have a lot of implicit pointer comparisons.
Exception: Stay explicit on return's, example:
return _db != nullptr;
Signed-off-by: Michael Schuster <michael@schuster.ms>
We keep NULL in the pure C files in src/csync/std and test/csync.
We also replace Doxygen documentation referring to "NULL" to
"\c nullptr" (formatted as code).
Signed-off-by: Stephan Beyer <s-beyer@gmx.net>
This test was failing locally for me. Indeed, through QStandardPaths it
was finding the user settings of my production client and not having the
initial state it expected. Using QStandardPaths test mode then it starts
from a clean slate every time.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
We simply use a static QObject using Q_GLOBAL_STATIC()
instead of allocating a leaking QObject on the heap.
Signed-off-by: Stephan Beyer <s-beyer@gmx.net>
In case of denormalized paths in the dav href (presence of . or .. in
the path) simple string startsWith comparison wasn't enough to know if
said href ended up in the right namespace. That's why we're now using
QUrl (pretending local file since we don't have a full URL in the href)
to normalize the path before comparison.
This could happen with broken proxies for instance where we would
wrongly validate the dav information resulting in potentially surprising
syncing and name collisions.
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
- When the the users logs because of 401 or 403 errors, it checks if the
server requested the remote wipe. If yes, locally deletes account and folders
connected to the account and notify the server. If no, proceeds to ask the
user to login again.
- The app password is restored in the keychain.
- WIP: The change also includes a test class for RemoteWipe.
Signed-off-by: Camila San <hello@camila.codes>
Add a test to test the data fingerprint feature make me realize it was broken.
The code was relying in the distinction between empty and null QByteArray,
but this was a bad idea as this difference is lost when going through QString.
Note that we also needed to adjust the server url to contains the user name
in the folder wizard. (As checkPathValidityForNewFolder expect the user name)
Issue #6654
Fix a strange warning seen on the log from the CI on Windows in
https://github.com/owncloud/client/pull/6621
The test shows, at the beginning
QObject::connect: No such signal DesktopServiceHook::destroyed(QObject*)
And crashes at the and.
My guess is that when QDesktopServices::setUrlHandler is called, the
QMetaObject is not yet initialized
But this is probably not the reason of the crash
It could happen that readyRead was emitted for incoming data while the
download was not yet finished. Then the network job could finish with
no more data arriving - so readyRead wasn't emitted again.
To fix this, the finished signal also gets connected to the readyRead
slot.
Since commit 4dc49ff3, we store an entry in the upload info table even
for non chunked uploads. However, if this fails we don't want to remove
non-existant stale chunks if the upload fails.
Without this commit, we would send a DELETE command to clean non-existant
chunks in the dav/uploads/ namespace.
Since the release package will be build with unit test, we don't
want to query the env variable at every call to fsCasePreserving.
So only test the env variable at startup.
And the testutility can still change the value.
(The env variable is still used from t8.pl and maybe smashbox)
Issue #6318
Paths can contain the wildcards % and _ and that would lead to odd
behavior.
This patch also clarifies the behavior of avoidReadFromDbOnNextSync()
which previously dependend on whether "foo/bar" or "foo/bar/" was
passed as input.
Possibly affects #6322
Previously conflicts with a different type on both ends lead to sync
errors. Now they are handled in the expected way: the local item gets
renamed and the remote item gets propagated downwards.
This also adds a unittest for the TYPE_CHANGE case. That one looks like
parts of it might be unified with CONFLICT cases.
Previously we'd use the full regex when the bname triggered a full-path
matching to take place. Now we have a simplified full-traversal regex
for this case that can be significantly faster to apply.
Triggered by #5017 but doesn't actually solve it.
Mainly uses target_include_directories instead of include_directories
so libraries public include directory get automatically added when adding
the target in target_link_library
Unfortunately matching behaved differently on Windows. This patch
restores the previous matching behavior but still uses the new regular
expression based matching.
Further work will hopefully unify the behavior between platforms without
breaking backwards compatibility.
fopen does not work well with relative path tand forward slashes on windows
This fix the windows textexcludedfiles test.
And also make the code simpler.
Note that the 'trimmed' might be a behavior change, but i think it is ok
If the server has the 'uploadConflictFiles' capability conflict
files will be uploaded instead of ignored.
Uploaded conflict files have the following headers set during upload
OC-Conflict: 1
OC-ConflictBaseFileId: 172489174instanceid
OC-ConflictBaseMtime: 1235789213
OC-ConflictBaseEtag: myetag
when the data is available. Downloads accept the same headers in return
when downloading a conflict file.
In the absence of server support clients will identify conflict files
through the file name pattern and attempt to deduce the base fileid.
Base etag and mtime can't be deduced though.
The upload job for a new conflict file will be triggered directly from
the job that created the conflict file now. No second sync run is
necessary anymore.
This commit does not yet introduce a 'username' like identifier that
automatically gets added to conflict file filenames (to name the files
foo_conflict-Fred-1345.txt instead of just foo_conflict-1345.txt).
Previously, there was csync_ftw_type_e and SyncFileItem::Type. Having
two enums lead to a bug where Type::Unknown == Type::File that went
unnoticed for a good while.
This patch keeps only a single enum.
This can happen if the upload of a file is finished, but we just got
disconnected right before recieving the reply containing the etag.
So nothing was save din the DB, and we are not sure if the server
recieved the file properly or not. Further local update of the file
will cause a conflict.
In order to fix this, store the checksum of the uploading file in
the uploadinfo table of the local db (even if there is no chunking
involved). And when we have a conflict, check that it is not because
of this situation by checking the entry in the uploadinfo table.
Issue #5106
In addition to using the right function when retrieving inodes this
*also* fixes a more general bug ownsql had with storing uint64 values
that didn't fit into an int64.
Improves full matches by more than an order of magnitude
and also improves speed of traversal matches by roughly 20%,
judging by the check_csync_exclude performance test.
Make ExcludedFiles something that is instantiated outside of
the CSYNC context and then given to it as a hook.
ExcludedFiles still lives in csync_exclude and the internal
workings haven't been touched.
For duplicate file ids the update phase and reconcile phase determined
the rename mappings independently. If they disagreed (due to different
order of processing), complicated misbehavior would result.
This patch fixes it by letting reconcile try to use the mapping that the
update phase has computed first.
Previously we required matching mtimes but that's actually
unnecessary when the question is about whether to skip the
download. We will still update the file's metadata.
Also, adjust behavior when the checksum is weak (Adler32):
in these cases we still depend on equal mtimes.
... even if the file is not changed.
We get an UPDATE_METADATA in that case, so make sure we let the
SyncFileStatusTracker know about it.
That means we need to filter out UPDATE_METADATA in the other listeners
of this signal.
Issue #6098
When users share the same tree several times (say A/ and A/B/ are both
shared) the remote tree can have several entries that have the same
file id. This needs to be respected in rename detection.
Also adds several tests and fixes for issues noticed during testing.
See #6096
We mostly trust the file watchers meaning that we don't re-scan the
local tree if we have done that recently and no file watcher events
have arrived. If the file watchers invalidate a subtree, we rescan
only that subtree.
Since we're not entirely sure the file watchers are reliable, we still
do full local discoveries regularly (1h by default). There is a config
file setting as well as an environment variable to control the interval.
Comparison of file sizes for potential conflicts was added in
0eb9401c62, but did not extend to checking
the file size in case of potential local moves.
This commit adds this check and adds tests for various move+change
scenarios.
On Mac, this halves the time spent in csync_excluded_traversal
when using check_csync_excluded_performance. A similar performance
increase is seen on linux.
This gets rid of the csync_statedb sqlite layer and use
the same code and same connection as the rest of the SyncEngine.
Missing functions are added to SyncJournalDb and change a few minor
things (like changing SyncJournalFileRecord::_modtime to be an int64
instead of a QDateTime, like it was in csync).
The current implementation would return the same value whether the query failed
or if no row would be found. This is something that is currently checked by csync
and needs to be provided if we want to use SyncJournalDB there.
Adjusted all call sites to also check the return value even though they
could still just rely on rec.isValid(), but makes it more explicit as to what
happens for database errors in those cases, if we ever want to gracefully handle
them.
The code that was creating the files in the directory was removed in
commit 6906b8d30c. The directory is empty
so the result is expected to be null. It was passing before because the
code was returning an entry for . and .., but since commit
35f80bd439 this is no longer the case
Create a specific type that parses the permissions so we can store
it in a short rather than in a QByteArray
Note: in RemotePermissions::toString, we make sure the string is not
empty by adding a space, this was already existing before commit
e8f7adc7ca where it was removed by mistake.
This remove the remaining "other" fields of the sync log to save a
bit of memory.
other_etag and other_fileId don't give much information to the users
and other_instruction will always be INST_NONE anyway.
other_modtime and other_size are kept since they are sometimes used.
They were renamed to have a bit more meaningful name.
SyncEngine::checkPermissions will now fetch its information from the
csync trees since they are now preserved until right after this point.
Fixes#3213
Only store the path since they represent the same thing, and do the
phash conversion during DB lookup like done in libsync.
We could get rid of everything since we also have an index on the path
column, but since it's the primary key this makes the migration non-trivial.
Now that they use the same structure, avoid _csync_detect_update
having to recreate another instance and transfer everything manually.
Any instance created during discovery should now be used all the way
up to SyncEngine::treewalkFile.
This also makes sure that the path and types are properly set in that
object instead of having to pass everything as separate parameters.
This gets rid of csync_ftw_flags_e which was now converted from,
and to csync_ftw_type_e, already in the csync_file_stat_t.
Issue #1817
Otherwise adding patterns that start with # are impossible to add, since
they get treated as comments. Also add this escaping for patterns added
in the ui.
Merge csync_create and csync_init into the constructor and
replace csync_destroy with the destructor.
Also use a QByteArray for csync_s::root_perms and flatten
csync_rename_s as a rename sub-struct of csync_s since it
can now handle C++ types.
The only difference with csync_s::current is that it's
assigned the value of csync_s::local::type and
csync_s::remote::type, which never change. So might as
well only use the "current" field with constants.
Also move csync_normalize_etag to common/utility since we
don't need the char* function anymore.
Remove the single space file_stat->remotePerm codepath since
this won't be used in csync anymore since
8de3bda0b1.
Issue #1817
This is the first commit trying to unify csync_file_stat_s,
csync_vio_file_stat_s and csync_tree_walk_file_s. Use QByteArray
and unique_ptr already since I'm not used to track memory allocations
and this will make the transition easier.
Issue #1817
Now that csync builds as C++, this will avoid having to implement
functionalities needed by csync mandatorily in csync itself.
This library is built as part of libocsync and symbols exported
through it.
This requires a relicense of Utility as LGPL. All classes moved into
this library from src/libsync will need to be relicensed as well.
This will allow us to unify data structures between csync and libsync.
Utility functions like csync_time and c_std are still compiled as C
since we won't need to be coupled with Qt in the short term.
The QNAM may continue to outlive both.
Rename Credentials::getQNAM() to createQNAM() while we're at it - it's
used to make a new QNAM that will subsequently be owned by the Account
object.
See d01065b9a1 for rationale.
Relates to
d40c56eda5147cf798a6
* SocketAPI has COPL_LOCAL_LINK / EMAIL_LOCAL_LINK commands
* The nautilus and dolphing shell integrations show a submenu from which
one can share as well as access the private link.
* The SocketAPI provides a new GET_STRINGS command to access localized
strings.
* The private link can also be accessed from the user/group sharing
dialog.
* The numeric file id is extracted from the full id to create the
private link url.
There was a rounding issue in the mtimes which sometimes resulted in an
off-by-one error. Caused by storing a full QDateTime in the FileInfo but
the mtime saved to the disk being truncated to seconds.
* For conflicts where mtime and size are identical:
a) If there's no remote checksum, skip (unchanged)
b) If there's a remote checksum that's a useful hash, create a
PropagateDownload job and compute the local hash. If the hashes
are identical, don't download the file and just update metadata.
* Avoid exposing the existence of checksumTypeId beyond the database
layer. This makes handling checksums easier in general because they
can usually be treated as a single blob.
This change was prompted by the difficulty of producing file_stat_t
entries uniformly from PROPFINDs and the database.
This is useful for monitoring what kind of network requests are
sent to the fake server. Such as "did this sync cause an upload?"
and "was there a propfind for this path?". It can also inject
custom replies.
The client is very picky about date strings it accepts. If dates are
formatted with a non-C locale (such as localized weekday names), it
fails to parse it and tests fail in subtle ways.
Issue #5783
When the directry that should be removed by selective sync contains changes,
we ignore the whole sub tree instead of only ignoreing new files.
We cannot ignore the whole directory, we need to ignore only the directory
that do not have files to remove
When a new folder becomes selective-sync excluded, we already mark it
and all its parent folders with _invalid_ etags to force rediscovery.
That's not enough however. Later calls to csync_statedb_get_below_path
could still pull data about the excluded files into the remote tree.
That lead to incorrect behavior, such as uploads happening for folders
that had been explicitly excluded from sync.
To fix the problem, statedb_get_below_path is adjusted to not read the
data about excluded folders from the database.
Currently we can't wipe this data from the database outright because we
need it to determine whether the files in the excluded folder can be
wiped away or not.
See owncloud/enterprise#1965
* For requests:
- reuse the original QNetworkRequest, so headers and attributes
are the same as in the original request
- determine the original http method from the reply and the request
attributes
- keep the original request body around such that it can be sent
again in case the request is redirected
* Simplify the interface that is used for creating new requests in
AbstractNetworkJob.
It could be possible that _firstJob is marked as finished if
aborted before its parent PropagateDirectory was marked as finished,
allowing a posted scheduleNextJob call to schedule the child job
in-between.
We were removing the wholme journal db when the user wanted to keep all files,
But that would also remove the selective sync lists.
We should only remove the metadata table.
Issue #5484
- Put all tests in the bin directory so that DLLs can be loaded
- Add missing exports
- Skip tests that use code depending on zlib
- The "GMT" timezone is named differently, use the int constructor instead
5 tests are still failing, it's not really worth fixing at the moment
since no developper is currently using Windows as its main platform.
On macOS /var is a symlink to /private/var and we have to make sure that we
use the canonical path before and after it enters the code to make sure we
compare paths correctly.
- We need to use a QGuiApplication on macOS or else we don't get notifications
- Switch to use QSignalSpy rather than lists and sleeps
- Use system() for all modifications since we pass kFSEventStreamCreateFlagIgnoreSelf
- Keep using the local process on Windows since it catches its own events
It could be possible that _firstJob is marked as finished if
aborted before its parent PropagateDirectory was marked as finished,
allowing a posted scheduleNextJob call to schedule the child job
in-between.
This was to catch duplicate emissions for PropagateDirectory but we
don't emit this signal anymore from there.
This fixes a warning about PropagatorJob not being a registered metatype.
This reverts commit fe42c1a818.
Stale chunks might be there because a file was removed or would just not
be uploaded, for any reason.
We just start the DeleteJob but we don't care if it success or not.
Relates to https://github.com/owncloud/core/issues/26981
One of the test is testing the case where the file is modified on the server
during the upload. So this test the precondition failed error.
The FakeGetReply logic was modified because resizing a 150MB big QByteArray
by increment of 16k just did not scale when downloading a big file.
Relates to https://github.com/owncloud/core/issues/26981
We do not track the success or error of the DeleteJob because it does not
matter. If it fails, it might be because the chunks were already removed.
If not, the chunks will be stale, but the server must anyway do a few
cleanup from time to time because we do not always remove the chunks
The current logic tried to avoid a DB lookup just to fetch whether
the file is shared or not since that info is already in the
SyncFileItem. The implementation would however need to decrease the
sync count for itself (and parents) before emitting the new status,
thus emitting the OK status for parents before that last child that
ended the propagation for that folder.
Change the implementation to achieve what we want: give the
possibility to decSyncCount to use a pre-fetched sharing state while
still doing the emission for all involved files. This ensures that
the leaf file also gets its status emitted before its parents.
Issue #4797
Shrinks owncloud binary by 24 KB and libowncloudsync by 14 KB.
I don't know if it has influence on memory usage or runtime speed though.
Was worth a try.
Previously this wasn't happening for errors that were not
NormalErrors because they don't end up in the blacklist.
This revises the resetting logic to be independent of the
error blacklist and make use of UploadInfo::errorCount
instead.
412 errors should reset chunked uploads because they might be
indicative of a checksum error.
Additionally, server bugs might require that additional
errors cause an upload reset. To allow that, a new capability
is added that can be used to advise the client about this.
We are going to change the webdav path depending on the capabilities.
But the SyncEngine and csync might have been created before the capabilities
are retrieved.
The main raison why we gave the path to the sync engine was to pass it to csync.
But the thing is that csync don't need anymore this url as everything is done by the
discovery classes in libsync that use the network jobs that use the account for the urls.
So csync do not need the remote URI.
shortenFilename in folderstatusmodel.cpp was useless because the string is the
_file of a SyncFileItem which is the relative file name, that name never
starts with owncloud://.
All the csync test creates the folder because csync use to check if the folder
exists. But we don't need to do that anymore
The "S" in the permission is only for the "Shared with me" files.
It is only used to show the shared status in the overlay icons.
But we also wish to show the shared status for files that are shared
"by" the users. We can find that out using the 'share-types' webdav
property. If set, then we are sharing the object.
We fake a 'S' in the permission as for our purpose, they mean the same.
Issue #4788
As the file can be some hunreds of megabytes, allocating such big arrays may
cause problems.
Also make the timeout a bit bigger so the test can rununder valgrind.
Update from commit 05ce8a23cdc12e825532dc6de06c267fb8d48b4f from
https://github.com/dragotin/QProgressIndicator
Which itself is forked from commit e5ba0fd09bfd43b067ee3646d70b294c7efcb558 from
upstream, with additional license header.
It was relicensed to MIT according to
14bb9d10e2
Relates to issues #5180 and #5184
Issue #5224
Two problems:
- In the discovery phase, we need to check the selective sync entries of
the source path in case of renames.
- When the rename is done, we need to actually update the black list in the
database.
Two bugs:
- The change filed are not considered as move, they are re-downloaded
but the old file was not removed from the database. The change in
owncloudpropagator.cpp takes care of removing the old entries.
- Next sync would then remove the file in the server in the old folder
This was not a problem until we start reusing the sync engine, and
that the _renamedFolders map is not cleared. We were before deleting
a non-existing file. But now we delete the actual file.
Also improve the tests to be able to do move on the server.
This include support for file id.
Issue #5192
This would happen if the directory would first need to be created
through an mkdir propagation job. This job's itemCompleted signal
would trigger the directory to show as SYNC even though its children
are still propagating.
Fix the issue by tracking the sync count for each file, affecting
its parents. This allows us to get rid of the O(n) vector lookup
for each status query, and properly track the hierachical sync
status of a directory.
This also removes the itemCompleted signal emission from the
PropagateDirectory job. Since we only needed for overlay icons, and
since this job doesn't do any direct propagation, we can remove it
to ensure that we won't call itemCompleted twice for the item attached
to Propagate*Mkdir jobs (since the PropagateDirectory is backed by
the same SyncFileItem, instruction and status).
Add a missing call that we currently only do in slotItemCompleted.
This would normally only affect the first sync and would have
gotten properly update at the end of the sync anyway.
To be able to test the SyncEngine efficiently, a set of server
mocking classes have been implemented on top of QNetworkAccessManager.
The local disk side hasn't been mocked since this would require adding
a large abstraction layer in csync. The SyncEngine is instead pointed
to a different temporary dir in each test and we test by interacting
with files in this directory instead.
The FakeFolder object wraps the SyncEngine with those abstractions
and allow controlling the local files, and the fake remote state
through the FileModifier interface, using a FileInfo tree structure
for the remote-side implementation as well as feeding and comparing
the states on both side in tests.
Tests run fast and require no setup to be run, but each server feature
that we want to test on the client side needs to be implemented in
this fake objects library. For example, the OC-FileId header isn't
set as of this commit, and we can't test the file move logic properly
without implementing it first.
The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL
for what I esteem being issues that need to be fixed in order to catch
up on our test coverage without making this patch too huge.
When a conflict-rename or a temporary-rename fails, notify the
LockWatcher. It'll regularly check whether the file has become
accesible again. When it has, another sync is triggered.
owncloud/enterprise#1288
Remove all configure_files:
- Move all tests to cpp files
- Use the QTEST_MAIN macro instead of a generated main.cpp
- Include test*.moc in the cpp to let CMAKE_AUTOMOC call moc
- Pass info through add_definitions instead of generating oc_bin.h with them
This makes sure that build errors points to the original test source
file instead of the generated one in the build directory to be able to
jump and fix errors directly from the IDE's error pane.
This will help moving the SyncEngine construction in the constructor
and allow moving functionalities from Folder to SyncEngine or its
delegated objects.
This will be useful if we ever want to store account-level gui state.
I built this originally because I thought a paused account would be
this kind of state.
* Compute the content checksum (in addition to the optional
transmission checksum) during upload (.eml files only)
* Add hook to compute and compare the checksum in csync_update
* Add content checksum to database, remove transmission checksum
Introduce a global ExcludedFiles instance to avoid loading the global
exclude lists several times.
One could still add per-folder exclude lists by checking these after
the global ones.
The owncloud wizard already created the directory, but the recent addition
of FolderMan::checkPathValidityForNewFolder stopped allowing unexisting directories.
So change FolderMan::checkPathValidityForNewFolder to allow non existing directory
and whange the FolderWizard to create the directory if it does not exist.
Issue #3492
factorize the function to check the validity of a new folder out of
the folder wizard to the FolderMan.
There is even a test for it now.
That way we can also use it in the normal wizard
- Make it translatable
- Make use of the %n so that it supports plurals (but this will only
take effect if we have an english translation)
- Only put two units. eg: "5 years 7 months" instead of
"5 years 7 months 12 days 34 minutes 23 seconds"
Even when it is "2 hours 23 minutes" the amount of second does not
matter, especially since the estimation is likely to be boggus anyway
Issues #2672 and #3097
