Previously rejecting any kind of certificate meant that the user
was never asked again, even if the certificate changed.
Now we keep track of which certificates were rejected and ask again
if the ones mentioned in the ssl errors change.
mitmproxy is excellent for testing this.
* Progress: Don't count dirs without propagation jobs #4856
These directory SyncFileItems are necessary for bookkeeping
but should not influence the progress display at all.
* Progress: Skip ignored files #4856
The problem in this case is if we rename the file "xxx" to "invalid\file".
The rename will fail because the new filename constains a slash, and it
will be blacklisted.
But then if the user re-rename the file to "valid_name", then we should
invalidate the blacklist entry and retry to upload. But we did not do
that because renaming don't change the mtime and we did not store the
rename target in the database
IL issue 558
This fixes an issue in which too many jobs are started un parallel
while uploading many files, which could cause too much memory usage as the
chunks are stored in memory.
Probably the fix for #4611
Issue #4855
A typo in the context string made the translation lookup fail.
But also the %Ln was not recognized as a plural form by transifex, so only
the singular was translated
The assert was there to make sure that this case wasn't happening
to eventually be properly tested. Remove the assert for now but this
codepath should eventually be unit tested using this specific situation.
Since the windows implementation first does cache lookups using the
path string, directories need to be passed identically as through
RETRIEVE_FILE_STATUS.
Change the convention to never have a trailing slash for directories
in the protocol. This allows the convention to be applied without
having to access the disk (since we'd need to know if the path is
represented by a directory) and also matches the convention of the
rest of the sync engine. Individual file manager plugins are then
responsible of handling pushed paths as not ending with a trailing
slash.
This also:
- Moves the trailing slash removal logic from the SyncFileStatusTracker
to the SocketApi class
- Remove the unneeded QString::normalized call in fileStatus, since
this should already be done by the FolderWatcher and plugins
Go through fileStatus like other cases to make sure that all use
cases go through the same code path. This also makes sure to use
lookupProblem which will use lower_bound which is more efficient
for larger sets of sync problems.
This also fixes the issue with lookupProblem that prevented it to
properly match an empty pathToMatch, caused by the fact that the
problem map contains relative paths not starting with a slash.
Make sure that we push the new status when the status of the SyncEngine
changed. SyncEngine::started comes a bit late, only when the propagation
starts, although it's better in this case since child folders will
only switch to Sync in aboutToPropagate.
Also fix an issue with SyncEngine::findSyncItem when using an empty
fileName; this would match and return the wrong item, even though
not currently happening with the code since fileStatus won't call
it with an empty fileName anymore.
As before, we rely on metadata-update SyncFileItem entries for parent
directories to notify us that a directory contains files to propagate,
and to know when all children were propagated through its itemCompleted
signal.
Those metadata SyncFileItems however have a None direction and we need
to add a explicit directory check to show them as Sync.
This fix also handles new files as well as existing ones, so no need
to keep a separate logic for new files.
propagatedownload.cpp:712:35: error: 'seenLockedFile' is a protected member of 'OCC::OwncloudPropagator'
Signals are protected in Qt4 but public in Qt5, mark the class accessing it
as friend when compiling with Qt4
When a conflict-rename or a temporary-rename fails, notify the
LockWatcher. It'll regularly check whether the file has become
accesible again. When it has, another sync is triggered.
owncloud/enterprise#1288
If the downloaded file is empty but the PROPFIND previously announced it
should not have been empty, this might mean the file was somehow corrupted
because of a bug on the server and that we should therefore not accept
the file.
Normaly we accept a change between the actual size of the file and what we
got during discovery because the file might have been updated to a new version
inbetween. But after this patch we won't accept the file if it was replaced
by an empty file.
Will help for issue #4583
Also requested by IL for issue 548
This uses the file watcher to keep track of files that were modified
in order to assign them the blue icon.
This is transient state that's not persisted across restarts.
As discussed on issue ##4460
Having the quote to be queried on subfolder is wrong in the generic case,
so add a branding option to configure it.
This partially reverts commit ff4cdc3161
In the before-propagate slot, new files that wait to be
pushed to the server are remembered in the _syncProblems
map. That way, the parents show a sync icon properly as
asked for in #4682.
After the item has been transfered properly, the item is
removed from the map again because success is the default.
Added in previous commit from pull request #4663
As discussed, we do not need this option so no need to introduce
a new dependency on the config file in the sync engine
* Add checksums/supportedTypes and checksums/preferredUploadType
capabilities. The default is that no checksum types are supported.
* Remove the transmissionChecksum config option. Servers must now
use the capabilities to indicate that they are fine with the
client sending checksums.
Note: This intentionally breaks brandings that overrode
Theme::transmissionChecksum. The override must be removed and the
server's capabilities must be adjusted to include the new values.
SQLITE_DONE is the indicator for not more query results, which is a legal
thing and not an error condition.
Also, check _getFileRecordQuery for null pointer, as close() wipes it.
The idea is that the next call to any database operation will try to
reopen the database through the checkConnect() method. So even if there
was a disconnect trom the db file, this will reestablish the connection.
Imagine tgus scenario on a read only share that you move file from
one location to a new directory in the read only share.
Creating the read only directory fails for permission error.
But we should also restore the files that have been moved.
IL issue 542
Bring back the hardcoded status logic for excluded files.
Since the activity log doesn't even mention those files on purpose,
we can't rely on the SyncEngine to notify us about the status of those files.
Looking up a/aa while an error is present in a/aab/aaba would return
a warning status since a/aa is a substring of a/aab.
Fix the issue by checking if the following character is a slash.
This prevents having to define a Problem structure with dubious
operator overloads to accomplish the same.
Also use std::map::lower_bound to quickly iterate over the
list of problems.
This also remove all smartness from the SocketApi about the status
of a file and solely use info from the current and last sync.
This simplifies the logic a lot and prevents any discrepancy between
the status shown in the activity log and the one displayed on the
overlay icon of a file.
The main benefit of the additional simplicity is that we are able
to push all new status of a file reliably (including warnings for
parent folders) to properly update the icon on overlay implementations
that don't allow us invalidating the status cache, like on OS X.
Both errors and warning from the last sync are now kept in a set,
which is used to also affect parent folders of an error.
To make sure that errors don't become warning icons on a second
sync, SyncFileItem::_hasBlacklistEntry is also interpreted as an error.
This also renames StatusIgnore to StatusWarning to match this semantic.
SyncEngine::aboutToPropagate is used in favor of SyncEngine::syncItemDiscovered
since the latter is emitted before file permission warnings are set on the
SyncFileItem. SyncEngine::finished is not used since we have all the
needed information in SyncEngine::itemCompleted.
SyncFileStatus' purpose is to track overlay icon status.
Instead of putting comments and default: clauses in switch
on both sides about unused enums, use different enums.
This also remove STATUS_NEW which is the equivalent of
STATUS_SYNC in all shell extension implementations, and
remove STATUS_UPDATED and STATUS_STAT_ERROR which have
the same semantic as STATUS_UPTODATE and STATUS__ERROR.
This will help moving the SyncEngine construction in the constructor
and allow moving functionalities from Folder to SyncEngine or its
delegated objects.
Soldiering on with a broken or incomplete response could lead to
incorrect sync behavior.
Since discovery uses LsCol jobs which already handle errors
correctly, this should not have a significant impact.
the size on the server might be different from the size on the client
with certain backend so it should be ignored.
(cherry picked from commit 9222db6df9b19a21e1bea5a238d745d96a6385e3)
In SQLite bindings are not cleared by sqlite3_reset() calls, so
skipping a sqlite3_bind call to create a NULL value doesn't work,
instead the previous value will be written.
To fix this, I clear all bindings in SqlQuery::reset and make sure
to explicitly bind NULL when desired in SqlQuery::bind.
To make sure there's no confusion about SqlQuery::reset and
sqlite3_reset, I rename our method to reset_and_clear_bindings().
(cherry picked from commit 7bd4f95b8c)
In SQLite bindings are not cleared by sqlite3_reset() calls, so
skipping a sqlite3_bind call to create a NULL value doesn't work,
instead the previous value will be written.
To fix this, I clear all bindings in SqlQuery::reset and make sure
to explicitly bind NULL when desired in SqlQuery::bind.
To make sure there's no confusion about SqlQuery::reset and
sqlite3_reset, I rename our method to reset_and_clear_bindings().
The isValid check should be used everywhere the capabilities
are used as the loading of the capabilities is happening
in parallel of the startup, so it is not guaranteed to be
available always.
Helps with small file sync #331
When I benchmarked this, it went up to 6 parallelism and
was about 1/3 faster than the previous fixed 3 parallelism.
Doing more than 6 is dangerous because QNAM limits to 6 TCP
connections and also the server might become a bottleneck.
Should also help for #4081
This will be useful if we ever want to store account-level gui state.
I built this originally because I thought a paused account would be
this kind of state.
The creation doesn't need to be separated from the SyncEngine anymore.
This allows the SyncEngine to be created in fewer steps if we want to
use it in tests.
This moves most of the direct csync code from Folder into the SyncEngine.
The exclude file logic for the context has been wrapped using the
existing ExcludedFiles class as well.
As discussed with Klaas, this seems to be a better compromise.
10MB * 3 prarralel jobs = 30MB in memory, and to retry in case of
disconnection. Which is still reasonable. And might make the upload
almost twice as fast on fast network where the amount of chunk is the
bottleneck (because of more server processing)
Relates to issue #4354
If the PROPFIND return an invalid code (like 200) then we would
not recieve the error signal and we would never sync again.
Found while investigating https://github.com/owncloud/enterprise/issues/1068
The ".sys.admin#recall#" is the recall file and should not be ignored
even if hidden.
The remote discovery do not need to detect hidden files because it
is already detected by csync in all cases. So this avoid code duplication
Users have complained that they don't see the notification when it is
shown and are not aware that their files aren't syncing.
Remove the non-interactive credentials fetch logic and add make sure
that the shibboleth popup will flash in the taskbar instead.
This will still not allow the popup to show in front in all cases,
but this is a compromise that we have to chose.
This reverts commit dcb687929f.
Issue https://github.com/owncloud/enterprise/issues/990
This is the fix for issue #4370
Step to reproduce the bug:
1) have lots of files in directory "dir1"
2) do mkdir dir2 && mv dir1/* dir2
3) DURING the sync (which takes time because of the many moves) do mkdir dir3 && mv dir2/* dir3/
4) observe that files are PUT in the next sync
The problem is that SyncJournalFileRecord::SyncJournalFileRecord will fail to
get the inode after the forst move because the files are already moved on the
filesystem. Normaly it should use the inode from the discovery phase in that
case but that is not working because it comes from the remote node in case of
moves, so the code in SyncEngine::treewalkFile would not set the inode.
Test in https://github.com/owncloud/smashbox/pull/143
Added a new "chunkSize" entry in the General group of the owncloud.cfg
which can be set to the size, in bytes, of the chunks.
This allow user with hude bandwidth to select more optimal chunk size
Issue #4354
Server older than 8.1 cannot cope with invalid char in the filename
so we must not send them from the client. We were already checking
for new files, but not for renames or new directories.
https://github.com/owncloud/enterprise/issues/1009
Previously we would fail to start if the directory was not existing.
This was working for relative directory, but it should also work for
absolute ones
https://github.com/owncloud/enterprise/issues/970
* Ensure every time a file becomes a directory or the other way around
the item is flagged as INSTRUCTION_TYPE_CHANGE.
* Delete the badly-typed entity if necessary in the propagation jobs.
The Qt HTTP thread calls authenticationRequired (indirectly) using a
BlockingQueuedConnection. So when we call invalidateToken from slot
connected to this signal and end up calling QNAM::clearAccessCache which
waits on the thread for 5 seconds
Backtraces:
Qt HTTP thread:
#0 0x00007ffff20c707f in pthread_cond_wait@@GLIBC_2.3.2 ()
#1 0x00007ffff43f0c0b in QWaitConditionPrivate::wait
#2 QWaitCondition::wait
#3 0x00007ffff43ea06b in QSemaphore::acquire
#4 0x00007ffff45dcf6f in QMetaObject::activate
[...]
#9 0x00007ffff45dd607 in QMetaObject::activate
#10 0x00007ffff4edbaf7 in QHttpNetworkReply::authenticationRequired
#11 0x00007ffff4e0b2b4 in QHttpNetworkConnectionPrivate::handleAuthenticateChallenge
#12 0x00007ffff4e10753 in QHttpNetworkConnectionChannel::handleStatus
#13 0x00007ffff4e11cc9 in QHttpNetworkConnectionChannel::allDone
#14 0x00007ffff4e14605 in QHttpProtocolHandler::_q_receiveReply
Main Thread:
#0 0x00007ffff20c7428 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
#1 0x00007ffff43f0b56 in QWaitConditionPrivate::wait_relative (time=5000, this=0x136c580)
#2 QWaitConditionPrivate::wait (time=5000, this=0x136c580)
#3 QWaitCondition::wait (this=this@entry=0x136c788, mutex=mutex@entry=0x136c760, time=time@entry=5000)
#4 0x00007ffff43efa6e in QThread::wait (this=<optimized out>, time=time@entry=5000)
#5 0x00007ffff4e1edd3 in QNetworkAccessManagerPrivate::clearCache
#6 0x00007ffff7b6fb03 in OCC::HttpCredentials::invalidateToken()
#7 0x000000000057adb4 in OCC::AccountState::slotInvalidCredentials()
#8 0x000000000057ac76 in OCC::AccountState::slotConnectionValidatorResult(OCC::ConnectionValidator::Status, QStringList const&) ()
#9 0x00000000005ab45c in OCC::AccountState::qt_static_metacall(QObject*, QMetaObject::Call, int, void**)
#10 0x00007ffff45dcd30 in QMetaObject::activate
#11 0x00007ffff7b78671 in OCC::ConnectionValidator::connectionResult(OCC::ConnectionValidator::Status, QStringList) ()
#12 0x00007ffff7ae2514 in OCC::ConnectionValidator::reportResult(OCC::ConnectionValidator::Status) ()
#13 0x00007ffff7ae39b7 in OCC::ConnectionValidator::slotAuthFailed(QNetworkReply*) ()
#14 0x00007ffff7b784a9 in OCC::ConnectionValidator::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) ()
#15 0x00007ffff45dcd30 in QMetaObject::activate
#16 0x00007ffff7b766dc in OCC::AbstractNetworkJob::networkError(QNetworkReply*)
#17 0x00007ffff7af9f6e in OCC::AbstractNetworkJob::slotFinished()
#18 0x00007ffff7b7654d in OCC::AbstractNetworkJob::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) ()
#20 0x00007ffff45dd607 in QMetaObject::activate
#21 0x00007ffff4edd143 in QNetworkReply::finished
#22 0x00007ffff4e3fec7 in QNetworkReplyHttpImplPrivate::finished
#23 0x00007ffff4e41818 in QNetworkReplyHttpImpl::close
#24 0x00007ffff7b7047b in OCC::HttpCredentials::slotAuthentication(QNetworkReply*, QAuthenticator*) ()
#25 0x00007ffff7b79092 in OCC::HttpCredentials::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) ()
#27 0x00007ffff45dd607 in QMetaObject::activate
#28 0x00007ffff4e1d6fb in QNetworkAccessManager::authenticationRequired
#29 0x00007ffff4e1ea07 in QNetworkAccessManagerPrivate::authenticationRequired
#30 0x00007ffff4e3c784 in QNetworkReplyHttpImplPrivate::httpAuthenticationRequired
Another case of Main Thread:
#5 0x00007ffff4e1edd3 in QNetworkAccessManagerPrivate::clearCache
#6 0x00007ffff7b6fb03 in OCC::HttpCredentials::invalidateToken()
#7 0x000000000057b1e4 in OCC::AccountState::slotInvalidCredentials() ()
#8 0x00000000005abb8a in OCC::AccountState::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) ()
#9 0x00007ffff45dcd30 in QMetaObject::activate
#10 0x00007ffff7b76ed5 in OCC::Account::invalidCredentials() ()
#11 0x00007ffff7ad55f5 in OCC::Account::handleInvalidCredentials()
#12 0x00007ffff7afa69a in OCC::AbstractNetworkJob::slotFinished()
#13 0x00007ffff7b7654d in OCC::AbstractNetworkJob::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) ()
#15 0x00007ffff45dd607 in QMetaObject::activate
#16 0x00007ffff4edd143 in QNetworkReply::finished
#17 0x00007ffff4e3fec7 in QNetworkReplyHttpImplPrivate::finished
#18 0x00007ffff4e41818 in QNetworkReplyHttpImpl::close
#19 0x00007ffff7b7047b in OCC::HttpCredentials::slotAuthentication(QNetworkReply*, QAuthenticator*) ()
#20 0x00007ffff7b79092 in OCC::HttpCredentials::qt_static_metacall(QObject*, QMetaObject::Call, int, void**) ()
#22 0x00007ffff45dd607 in QMetaObject::activate
#23 0x00007ffff4e1d6fb in QNetworkAccessManager::authenticationRequired
The error status of children should only be used for the etag logic.
The SocketApi uses a path matching system to do this and the UI should
report errors only for individual involved files/directories.
Since the presence of any path in SyncEngine::_syncedItems
would translate in a SYNC status, platforms that don't refresh
all their status cache after an UPDATE_VIEW message like OS X
or Windows would keep displaying that status even after all
files are successfully synchronized.
- Read SyncFileItem::_status to determine the status to display mid-sync
- Match moved paths also to _renameTarget since this might be the
path to match
- Make sure that PropagateDirectory jobs also set SyncFileItem::_status
properly
If all the files bring us to past timestamp, it is possibly a backup
restoration in the server. In which case we want don't want to just
overwrite newer files with the older ones.
Issue #2325
Since owncloud 2.1, csync_vio_local_stat was optimized because readdir
would already fetch most of the info. This works for the discovery,
but not later. And we used this function later for symliks and co.
So this fixes the .lnk on windows
Issue #4300
* Compute the content checksum (in addition to the optional
transmission checksum) during upload (.eml files only)
* Add hook to compute and compare the checksum in csync_update
* Add content checksum to database, remove transmission checksum
The signal jsonReceived() now not only delivers the raw json string, but
also the status code that came as OCS reply.
Also, fixed a typo in the signals name (recieved => received).
A 403 is a reply code sent from the file firewall to indicate that
this directory is forbidden to use for the user.
The patch handles it by setting the state to IGNORED.
This addresses #3490
We changed the discovery code not to ignore files whose filename contains
charachter invalid on windows. (Because newer versions of the server
supports them)
Servers older than 8.1 will just say "Bad Request" as an error and it's a
regression against previous client version. So keep nice error even with
older server.
Relates to #3736
