Commit graph

181 commits

Author SHA1 Message Date
Kevin Ottens
3204c15911 Better separate between absolute and relative paths on uploads
Yes... I still wish this would be all driven by the type system, would be
much less error-prone.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-07-01 16:58:29 +00:00
Kevin Ottens
d7a9a4cd09 Fix parenting issues of PropagateUploadEncrypted
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-06-30 11:29:08 +02:00
Kevin Ottens
710c470c13 Assume that folder paths can be mangled
PropagateUploadEncrypted made the assumption of the folder names never
being mangled. This is not true since the previous commits so make sure
we properly deal with that using the journal db.

Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-06-30 11:29:08 +02:00
Kevin Ottens
76fbbfefa5 Fix typo
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-06-30 11:29:08 +02:00
Kevin Ottens
5afcf71e68 Fix typo
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-06-30 11:29:08 +02:00
Kevin Ottens
712869db9a Use auto to avoiding repeating type names
Signed-off-by: Kevin Ottens <kevin.ottens@nextcloud.com>
2020-05-20 03:54:41 +02:00
Mariusz Wasak
dcc33f3417 Fix for #1382 "linux client crashes for no discernable reason"
There in no "return" in
PropagateUploadFileCommon::slotStartUpload in if (prevModtime != _item-
>_modtime) {... }

There is possibility that
PropagateItemJob::done(status, errorString)
maybe called two times from PropagateUploadFileCommon::slotStartUpload
1. in if (prevModtime != _item->_modtime) {... }
2. in if (fileIsStillChanging(*_item)) {..}
if changes in files are frequent the second call is possible.

This two calls has effect in PropagatorCompositeJob::slotSubJobFinished
and job is removed two times in _runningJobs.remove(i);
(the second time with argumetnt -1 (because first call removed job).

This return was removed in commit
efc039863b - by accident I think.

Good simulation is to synchronize firefox profile with frequent page
refresh.

Signed-off-by: Mariusz Wasak <mawasak@gmail.com>
2019-09-18 18:18:12 +02:00
J-P Nurmi
1af9bf8abc libsync: run clang-tidy modernize-use-nullptr 2018-11-12 18:46:39 +01:00
Christian Kamm
5f793c151c
Upload: Adjust timeout for final job based on size #6527
Some servers have virus scanners and the like that can delay the
response of the final chunked upload assembly significantly, often
breaking the current 5min (!) timeout. See owncloud/enterprise#2480
for details.
2018-06-07 09:38:16 +02:00
Christian Kamm
d9640dd79e
PropagateUpload: Fix miscount in abort() #6516
Previously it tried to abort even jobs that had already finished, which
was not going to work as they wouldn't emit finished() again.

Also, in some cases the abortCount would never go to zero and that case
wasn't well documented.
2018-06-07 08:53:12 +02:00
Christian Kamm
96126a53d3
PropagateUpload: Remove unused PropagateUploadFileCommon::abort 2018-06-07 08:53:10 +02:00
Christian Kamm
a2675b559b
Upload: Simplify 'finished' flag
* Sometimes there was 'finished' and '_finished'
* Make each done() call automatically set _finished to true
2018-05-16 20:57:26 +02:00
Tomaz Canabrava
96115be0c6 Unlock folder in case of error 2018-03-25 21:54:08 +02:00
Daniel Nicoletti
99117078f3 Remove table for e2e and add an e2eMangledName column (#169)
If the code was not complex enough syncing two tables
already started to give UNIQUE constrains errors on
simple sync operations, this also adds initial support
remote delete of an encrypted file
2018-02-12 12:50:51 +01:00
Tomaz Canabrava
d24a1e542e Merge branch 'master' into clientSideEncryptionV3 2018-01-29 14:06:12 +01:00
Tomaz Canabrava
e8849a2cf9 [CSE] Save the relation of the file / encrypted file names on the db 2018-01-28 22:26:07 +01:00
Olivier Goffart
b8539eb329 Get rid of the StopWatch in the PropagateUpload job
For issue #6318

The StopWatch is using memory, and we are not really using it.
2018-01-19 13:37:48 +01:00
Daniel Nicoletti
e2c895e61f Fix encrypted file location on the server 2017-12-29 17:28:35 +01:00
Daniel Nicoletti
a63d34f870 Prepend "nextcloud" for all logging categories
Thus making easier to exclude logging from kio, qt
and only enable "nextcloud.*"
2017-12-28 17:33:10 -02:00
Tomaz Canabrava
efc039863b [CSE] Tons of debugs. 2017-12-24 17:08:04 +01:00
Tomaz Canabrava
06c34ed617 [CSE] Assorted fixes
- Unlock the folder even on error
- Use the correct name of the file for upload
2017-12-24 16:30:39 +01:00
Tomaz Canabrava
b72113a53d [CSE] Prepare the beginning of the upload.
I still need to send the metadata.
2017-12-21 00:35:23 +01:00
Tomaz Canabrava
7be5f0a736 [CSE] Update the finished signal 2017-12-20 23:36:49 +01:00
Tomaz Canabrava
aa36cc4a5c [CSE] Reduce probability of conflicts when rebasing
Creates a new file/class to handle upload,
propagateuploadencrypted.h/cpp. This should have all the logic to deal
with the encryption and upload.
2017-12-17 22:38:43 +01:00
Christian Kamm
a7c0cfc8eb Upload conflict files #4557
If the server has the 'uploadConflictFiles' capability conflict
files will be uploaded instead of ignored.

Uploaded conflict files have the following headers set during upload
  OC-Conflict: 1
  OC-ConflictBaseFileId: 172489174instanceid
  OC-ConflictBaseMtime: 1235789213
  OC-ConflictBaseEtag: myetag
when the data is available. Downloads accept the same headers in return
when downloading a conflict file.

In the absence of server support clients will identify conflict files
through the file name pattern and attempt to deduce the base fileid.
Base etag and mtime can't be deduced though.

The upload job for a new conflict file will be triggered directly from
the job that created the conflict file now. No second sync run is
necessary anymore.

This commit does not yet introduce a 'username' like identifier that
automatically gets added to conflict file filenames (to name the files
foo_conflict-Fred-1345.txt instead of just foo_conflict-1345.txt).
2017-12-15 18:03:53 +01:00
Tomaz Canabrava
2127b2629d [CSE] Move encryption related network jobs to its own file
Move encryption related network jobs to it's own file,
the original file was starting to be just way too big.
2017-12-12 19:36:47 +01:00
Tomaz Canabrava
0a58ea76e5 [CSE] Start the work with an existing metadata 2017-12-12 15:35:53 +01:00
Tomaz Canabrava
e6f835d138 [CSE] Unlock the folder after a lock
This makes sure the folder will be unlocked for next tests
2017-12-12 11:26:54 +01:00
Tomaz Canabrava
a08a32ceca [CSE] Fetch the metadata for the upload. 2017-12-08 10:32:05 +01:00
Tomaz Canabrava
6745777ca3 [CSE] Save folder id and token for the lock
So we can unlock it later. - perhaps we should store this
on the disk or something, to make it harder to lose.
2017-12-08 10:19:56 +01:00
Tomaz Canabrava
70d562668c [CSE] Add handling for side-by-side uploads. 2017-12-07 21:51:42 +01:00
Tomaz Canabrava
ccd8cff4db [CSE] Try to lock repeteadly the folder if needed.
The upload is made in an event loop with more than one
upload at the same time, this confuses the hell out of the
folder locking mechanism.

We need to lock the folder and ask the other trials to try
again in a few seconds in the future to give time for the
uploader to actually upload the current file that's locking
the folder.
2017-12-07 21:31:20 +01:00
Tomaz Canabrava
9f678652f8 [CSE] Lock the folder to upload a file. 2017-12-07 20:52:54 +01:00
Tomaz Canabrava
76916b6c48 [CSE] Request the Folder ID. 2017-12-07 19:39:30 +01:00
Tomaz Canabrava
7fe4dd2163 [CSE] Renane GetFolderEncryptStatus to GetFolderEncryptStatusJob
it's a network job after all
2017-12-07 18:06:55 +01:00
Tomaz Canabrava
863e86138f [CSE] Pass the folder to the GetEncryptionStatus
Sometimes we are only interested in folders.
2017-12-07 17:32:35 +01:00
Tomaz Canabrava
66aecb9626 [CSE] Handle normal uploads in CSE enabled instances 2017-12-04 22:45:10 +01:00
Tomaz Canabrava
6f35cbff6b [CSE] Fetch the Folder information for encryption status 2017-12-04 22:27:13 +01:00
Tomaz Canabrava
387737d908 [CSE] Start to create the upload logic. 2017-12-04 21:33:43 +01:00
Tomaz Canabrava
8c4928d9f6 Merge branch 'uploadImprovements' into clientSideEncryptionV3 2017-12-03 18:15:33 +01:00
Tomaz Canabrava
a623b14209 [uploadImprovements] use the new struct info in PropagateUploadV1
Add a new member for the UploadFileInfo in PropagateUploadCommon
to hold the full file path - as it can change if we use a temporary
file to upload.

Adapt propagateuploadv1 to use the new calls.
2017-11-28 22:17:29 +01:00
Tomaz Canabrava
f609336bfc Split file to upload from the file on disk
They can be conceptually equal - I can upload the file
on disk, and that's what I do right now. But if we want
to accept filters in the future, filters that change
the file on disk like shrinking an image, the current
information used is wrong and we need a way to separate those.

This patch introduces a new struct that holds the *actual*
file that will be uploaded, be it a temporary one or
the original file.
2017-11-28 21:28:06 +01:00
Tomaz Canabrava
c4d3d0987d [CSE] Find the start of the upload 2017-11-27 22:20:24 +01:00
Piotr Mrowczynski
e10775d34f Fix paused sync file move issue #5949
Dont abort final chunk immedietally

Use sync and async aborts
2017-10-17 09:44:52 +02:00
Olivier Goffart
13e0cc6b50 Use QDateTime::currentDateTimeUtc instead of the non utc version
QDateTime::currentDateTime is terribly slow.
2017-09-26 16:14:17 +02:00
Olivier Goffart
ff4213b59f Use the Qt5 connection syntax (automated with clazy)
This is motivated by the fact that QMetaObject::noralizeSignature takes 7.35%
CPU of the LargeSyncBench. (Mostly from ABstractNetworkJob::setupConnections and
PropagateUploadFileV1::startNextChunk). It could be fixed by using normalized
signature in the connection statement, but i tought it was a good oportunity
to modernize the code.

This commit only contains calls that were automatically converted with clazy.
2017-09-21 14:05:39 +02:00
Jocelyn Turcotte
a1f1775d15 Move SyncJournalDB to src/common 2017-09-18 14:00:52 +02:00
Jocelyn Turcotte
5fbed0d1cd Reverse the dependency between SyncJournalFileRecord and SyncFileItem
This will allow us to also use the SyncJournalDB in csync.
2017-09-18 14:00:52 +02:00
Christian Kamm
3a55ff2119 PropagateUpload: Model of remote quota, avoid some uploads #5537
When we see a 507 error, assume that quota is < uploaded size.
2017-09-15 09:27:07 +02:00
Christian Kamm
0c99673581 Make DetailError different from BlacklistedError
It's quite different in regard to blacklist handling and overall
sync failure changes.
2017-09-15 09:27:07 +02:00