mirrors/nextcloud-desktop
1
0
Fork 0
mirror of https://github.com/nextcloud/desktop.git synced 2024-12-13 15:46:01 +03:00
Code Issues Projects Releases Packages Wiki Activity
nextcloud-desktop/test/testsyncengine.cpp

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

627 lines
28 KiB
C++
Raw Normal View History

Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
/*
* This software is in the public domain, furnished "as is", without technical
* support, and with no warranty, express or implied, as to its usefulness for
* any purpose.
*
*/
#include <QtTest>
#include "syncenginetestutils.h"
SyncEngine: Fix renaming a folder should keep the selective sync state Issue #5224 Two problems: - In the discovery phase, we need to check the selective sync entries of the source path in case of renames. - When the rename is done, we need to actually update the black list in the database.
2016-10-10 17:55:31 +03:00
#include <syncengine.h>
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
using namespace OCC;
bool itemDidComplete(const QSignalSpy &spy, const QString &path)
{
for(const QList<QVariant> &args : spy) {
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
auto item = args[0].value<SyncFileItemPtr>();
if (item->destination() == path)
SyncFileStatusTracker: Detect changed in the shared flag ... even if the file is not changed. We get an UPDATE_METADATA in that case, so make sure we let the SyncFileStatusTracker know about it. That means we need to filter out UPDATE_METADATA in the other listeners of this signal. Issue #6098
2017-10-23 17:35:35 +03:00
return item->_instruction != CSYNC_INSTRUCTION_NONE && item->_instruction != CSYNC_INSTRUCTION_UPDATE_METADATA;
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
}
return false;
}
bool itemDidCompleteSuccessfully(const QSignalSpy &spy, const QString &path)
{
for(const QList<QVariant> &args : spy) {
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
auto item = args[0].value<SyncFileItemPtr>();
if (item->destination() == path)
return item->_status == SyncFileItem::Success;
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
}
return false;
}
class TestSyncEngine : public QObject
{
Q_OBJECT
private slots:
void testFileDownload() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.remoteModifier().insert("A/a0");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "A/a0"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testFileUpload() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.localModifier().insert("A/a0");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "A/a0"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testDirDownload() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.remoteModifier().mkdir("Y");
fakeFolder.remoteModifier().mkdir("Z");
fakeFolder.remoteModifier().insert("Z/d0");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Y"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Z"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Z/d0"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testDirUpload() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.localModifier().mkdir("Y");
fakeFolder.localModifier().mkdir("Z");
fakeFolder.localModifier().insert("Z/d0");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Y"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Z"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "Z/d0"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testLocalDelete() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.remoteModifier().remove("A/a1");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "A/a1"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testRemoteDelete() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.localModifier().remove("A/a1");
fakeFolder.syncOnce();
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "A/a1"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
void testEmlLocalChecksum() {
FakeFolder fakeFolder{FileInfo{}};
fakeFolder.localModifier().insert("a1.eml", 64, 'A');
fakeFolder.localModifier().insert("a2.eml", 64, 'A');
fakeFolder.localModifier().insert("a3.eml", 64, 'A');
Checksums: Clearer behavior and added testing
2017-09-15 14:30:29 +03:00
fakeFolder.localModifier().insert("b3.txt", 64, 'A');
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
// Upload and calculate the checksums
// fakeFolder.syncOnce();
fakeFolder.syncOnce();
Checksums: Clearer behavior and added testing
2017-09-15 14:30:29 +03:00
auto getDbChecksum = [&](QString path) {
SyncJournalDB: Allow callers of getFileRecord if the query failed The current implementation would return the same value whether the query failed or if no row would be found. This is something that is currently checked by csync and needs to be provided if we want to use SyncJournalDB there. Adjusted all call sites to also check the return value even though they could still just rely on rec.isValid(), but makes it more explicit as to what happens for database errors in those cases, if we ever want to gracefully handle them.
2017-09-13 20:02:38 +03:00
SyncJournalFileRecord record;
fakeFolder.syncJournal().getFileRecord(path, &record);
Checksums: Clearer behavior and added testing
2017-09-15 14:30:29 +03:00
return record._checksumHeader;
};
// printf 'A%.0s' {1..64} | sha1sum -
QByteArray referenceChecksum("SHA1:30b86e44e6001403827a62c58b08893e77cf121f");
QCOMPARE(getDbChecksum("a1.eml"), referenceChecksum);
QCOMPARE(getDbChecksum("a2.eml"), referenceChecksum);
QCOMPARE(getDbChecksum("a3.eml"), referenceChecksum);
QCOMPARE(getDbChecksum("b3.txt"), referenceChecksum);
Pass the SyncFileItem as SyncFileItemPtr in itemCompleted This will allow us to keep a reference on the items in connected slots.
2017-01-25 13:12:38 +03:00
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
// Touch the file without changing the content, shouldn't upload
fakeFolder.localModifier().setContents("a1.eml", 'A');
// Change the content/size
fakeFolder.localModifier().setContents("a2.eml", 'B');
fakeFolder.localModifier().appendByte("a3.eml");
Checksums: Clearer behavior and added testing
2017-09-15 14:30:29 +03:00
fakeFolder.localModifier().appendByte("b3.txt");
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
fakeFolder.syncOnce();
Checksums: Clearer behavior and added testing
2017-09-15 14:30:29 +03:00
QCOMPARE(getDbChecksum("a1.eml"), referenceChecksum);
QCOMPARE(getDbChecksum("a2.eml"), QByteArray("SHA1:84951fc23a4dafd10020ac349da1f5530fa65949"));
QCOMPARE(getDbChecksum("a3.eml"), QByteArray("SHA1:826b7e7a7af8a529ae1c7443c23bf185c0ad440c"));
QCOMPARE(getDbChecksum("b3.eml"), getDbChecksum("a3.txt"));
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
QVERIFY(!itemDidComplete(completeSpy, "a1.eml"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "a2.eml"));
QVERIFY(itemDidCompleteSuccessfully(completeSpy, "a3.eml"));
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
}
SyncEngine: Fix renaming of folder when file are changed (#5195) Two bugs: - The change filed are not considered as move, they are re-downloaded but the old file was not removed from the database. The change in owncloudpropagator.cpp takes care of removing the old entries. - Next sync would then remove the file in the server in the old folder This was not a problem until we start reusing the sync engine, and that the _renamedFolders map is not cleared. We were before deleting a non-existing file. But now we delete the actual file. Also improve the tests to be able to do move on the server. This include support for file id. Issue #5192
2016-09-22 10:02:47 +03:00
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
void testSelectiveSyncBug() {
// issue owncloud/enterprise#1965: files from selective-sync ignored
// folders are uploaded anyway is some circumstances.
FakeFolder fakeFolder{FileInfo{ QString(), {
FileInfo { QStringLiteral("parentFolder"), {
Use SyncJournalDb in csync This gets rid of the csync_statedb sqlite layer and use the same code and same connection as the rest of the SyncEngine. Missing functions are added to SyncJournalDb and change a few minor things (like changing SyncJournalFileRecord::_modtime to be an int64 instead of a QDateTime, like it was in csync).
2017-09-14 16:50:13 +03:00
FileInfo{ QStringLiteral("subFolderA"), {
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
{ QStringLiteral("fileA.txt"), 400 },
SyncEngine: SelectiveSync: Remove local files of undelected folder despite other modified files Issue #5783 When the directry that should be removed by selective sync contains changes, we ignore the whole sub tree instead of only ignoreing new files. We cannot ignore the whole directory, we need to ignore only the directory that do not have files to remove
2017-06-06 17:00:41 +03:00
{ QStringLiteral("fileB.txt"), 400, 'o' },
FileInfo { QStringLiteral("subsubFolder"), {
{ QStringLiteral("fileC.txt"), 400 },
{ QStringLiteral("fileD.txt"), 400, 'o' }
}},
FileInfo{ QStringLiteral("anotherFolder"), {
FileInfo { QStringLiteral("emptyFolder"), { } },
FileInfo { QStringLiteral("subsubFolder"), {
{ QStringLiteral("fileE.txt"), 400 },
{ QStringLiteral("fileF.txt"), 400, 'o' }
}}
}}
Use SyncJournalDb in csync This gets rid of the csync_statedb sqlite layer and use the same code and same connection as the rest of the SyncEngine. Missing functions are added to SyncJournalDb and change a few minor things (like changing SyncJournalFileRecord::_modtime to be an int64 instead of a QDateTime, like it was in csync).
2017-09-14 16:50:13 +03:00
}},
FileInfo{ QStringLiteral("subFolderB"), {} }
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
}}
}}};
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
auto expectedServerState = fakeFolder.currentRemoteState();
Use SyncJournalDb in csync This gets rid of the csync_statedb sqlite layer and use the same code and same connection as the rest of the SyncEngine. Missing functions are added to SyncJournalDb and change a few minor things (like changing SyncJournalFileRecord::_modtime to be an int64 instead of a QDateTime, like it was in csync).
2017-09-14 16:50:13 +03:00
// Remove subFolderA with selectiveSync:
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
fakeFolder.syncEngine().journal()->setSelectiveSyncList(SyncJournalDb::SelectiveSyncBlackList,
Use SyncJournalDb in csync This gets rid of the csync_statedb sqlite layer and use the same code and same connection as the rest of the SyncEngine. Missing functions are added to SyncJournalDb and change a few minor things (like changing SyncJournalFileRecord::_modtime to be an int64 instead of a QDateTime, like it was in csync).
2017-09-14 16:50:13 +03:00
{"parentFolder/subFolderA/"});
fakeFolder.syncEngine().journal()->avoidReadFromDbOnNextSync(QByteArrayLiteral("parentFolder/subFolderA/"));
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
SyncEngine: SelectiveSync: Remove local files of undelected folder despite other modified files Issue #5783 When the directry that should be removed by selective sync contains changes, we ignore the whole sub tree instead of only ignoreing new files. We cannot ignore the whole directory, we need to ignore only the directory that do not have files to remove
2017-06-06 17:00:41 +03:00
// But touch local file before the next sync, such that the local folder
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
// can't be removed
Use SyncJournalDb in csync This gets rid of the csync_statedb sqlite layer and use the same code and same connection as the rest of the SyncEngine. Missing functions are added to SyncJournalDb and change a few minor things (like changing SyncJournalFileRecord::_modtime to be an int64 instead of a QDateTime, like it was in csync).
2017-09-14 16:50:13 +03:00
fakeFolder.localModifier().setContents("parentFolder/subFolderA/fileB.txt", 'n');
fakeFolder.localModifier().setContents("parentFolder/subFolderA/subsubFolder/fileD.txt", 'n');
fakeFolder.localModifier().setContents("parentFolder/subFolderA/anotherFolder/subsubFolder/fileF.txt", 'n');
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
// Several follow-up syncs don't change the state at all,
// in particular the remote state doesn't change and fileB.txt
// isn't uploaded.
for (int i = 0; i < 3; ++i) {
fakeFolder.syncOnce();
{
// Nothing changed on the server
QCOMPARE(fakeFolder.currentRemoteState(), expectedServerState);
// The local state should still have subFolderA
auto local = fakeFolder.currentLocalState();
Use SyncJournalDb in csync This gets rid of the csync_statedb sqlite layer and use the same code and same connection as the rest of the SyncEngine. Missing functions are added to SyncJournalDb and change a few minor things (like changing SyncJournalFileRecord::_modtime to be an int64 instead of a QDateTime, like it was in csync).
2017-09-14 16:50:13 +03:00
QVERIFY(local.find("parentFolder/subFolderA"));
QVERIFY(!local.find("parentFolder/subFolderA/fileA.txt"));
QVERIFY(local.find("parentFolder/subFolderA/fileB.txt"));
QVERIFY(!local.find("parentFolder/subFolderA/subsubFolder/fileC.txt"));
QVERIFY(local.find("parentFolder/subFolderA/subsubFolder/fileD.txt"));
QVERIFY(!local.find("parentFolder/subFolderA/anotherFolder/subsubFolder/fileE.txt"));
QVERIFY(local.find("parentFolder/subFolderA/anotherFolder/subsubFolder/fileF.txt"));
QVERIFY(!local.find("parentFolder/subFolderA/anotherFolder/emptyFolder"));
QVERIFY(local.find("parentFolder/subFolderB"));
Selective sync: Skip excluded folders when reading db When a new folder becomes selective-sync excluded, we already mark it and all its parent folders with _invalid_ etags to force rediscovery. That's not enough however. Later calls to csync_statedb_get_below_path could still pull data about the excluded files into the remote tree. That lead to incorrect behavior, such as uploads happening for folders that had been explicitly excluded from sync. To fix the problem, statedb_get_below_path is adjusted to not read the data about excluded folders from the database. Currently we can't wipe this data from the database outright because we need it to determine whether the files in the excluded folder can be wiped away or not. See owncloud/enterprise#1965
2017-05-15 15:46:09 +03:00
}
}
}
Delete finished propagation jobs in PropagateDirectory #5269 (#5400)
2017-01-26 12:03:22 +03:00
void abortAfterFailedMkdir() {
FakeFolder fakeFolder{FileInfo{}};
QSignalSpy finishedSpy(&fakeFolder.syncEngine(), SIGNAL(finished(bool)));
fakeFolder.serverErrorPaths().append("NewFolder");
fakeFolder.localModifier().mkdir("NewFolder");
// This should be aborted and would otherwise fail in FileInfo::create.
fakeFolder.localModifier().insert("NewFolder/NewFile");
fakeFolder.syncOnce();
QCOMPARE(finishedSpy.size(), 1);
QCOMPARE(finishedSpy.first().first().toBool(), false);
}
Propagator: Fix finished signal of directory being emited twice and causing crash When there is a FatalError, we ended up emiting the finished signal for the directory job several times, which would lead to crashes Issue #5578
2017-03-06 15:07:08 +03:00
void testDirDownloadWithError() {
FakeFolder fakeFolder{FileInfo::A12_B12_C12_S12()};
QSignalSpy completeSpy(&fakeFolder.syncEngine(), SIGNAL(itemCompleted(const SyncFileItemPtr &)));
fakeFolder.remoteModifier().mkdir("Y");
fakeFolder.remoteModifier().mkdir("Y/Z");
fakeFolder.remoteModifier().insert("Y/Z/d0");
fakeFolder.remoteModifier().insert("Y/Z/d1");
fakeFolder.remoteModifier().insert("Y/Z/d2");
fakeFolder.remoteModifier().insert("Y/Z/d3");
fakeFolder.remoteModifier().insert("Y/Z/d4");
fakeFolder.remoteModifier().insert("Y/Z/d5");
fakeFolder.remoteModifier().insert("Y/Z/d6");
fakeFolder.remoteModifier().insert("Y/Z/d7");
fakeFolder.remoteModifier().insert("Y/Z/d8");
fakeFolder.remoteModifier().insert("Y/Z/d9");
fakeFolder.serverErrorPaths().append("Y/Z/d2", 503); // 503 is a fatal error
fakeFolder.serverErrorPaths().append("Y/Z/d3", 503); // 503 is a fatal error
QVERIFY(!fakeFolder.syncOnce());
QCoreApplication::processEvents(); // should not crash
QSet<QString> seen;
for(const QList<QVariant> &args : completeSpy) {
auto item = args[0].value<SyncFileItemPtr>();
Remove SyncFileItem::_isDirectory It's always equivalent to _type == SyncFileItem::Directory.
2017-08-24 18:31:46 +03:00
qDebug() << item->_file << item->isDirectory() << item->_status;
Propagator: Fix finished signal of directory being emited twice and causing crash When there is a FatalError, we ended up emiting the finished signal for the directory job several times, which would lead to crashes Issue #5578
2017-03-06 15:07:08 +03:00
QVERIFY(!seen.contains(item->_file)); // signal only sent once per item
seen.insert(item->_file);
Fix TestSyncEngine::testDirDownloadWithError On master, the scheduling is different and the tasks abort in another order
2017-03-09 12:35:58 +03:00
if (item->_file == "Y/Z/d2") {
Propagator: Fix finished signal of directory being emited twice and causing crash When there is a FatalError, we ended up emiting the finished signal for the directory job several times, which would lead to crashes Issue #5578
2017-03-06 15:07:08 +03:00
QVERIFY(item->_status == SyncFileItem::FatalError);
Fix TestSyncEngine::testDirDownloadWithError On master, the scheduling is different and the tasks abort in another order
2017-03-09 12:35:58 +03:00
} else if(item->_file == "Y/Z/d3") {
QVERIFY(item->_status != SyncFileItem::Success);
Propagator: Fix finished signal of directory being emited twice and causing crash When there is a FatalError, we ended up emiting the finished signal for the directory job several times, which would lead to crashes Issue #5578
2017-03-06 15:07:08 +03:00
}
QVERIFY(item->_file != "Y/Z/d9"); // we should have aborted the sync before d9 starts
}
}
test/testsyncengine: test for PR #6168
2017-11-16 14:35:40 +03:00
void testFakeConflict_data()
{
QTest::addColumn<bool>("sameMtime");
QTest::addColumn<QByteArray>("checksums");
QTest::addColumn<int>("expectedGET");
QTest::newRow("Same mtime, but no server checksum -> ignored in reconcile")
<< true << QByteArray()
<< 0;
QTest::newRow("Same mtime, weak server checksum differ -> downloaded")
<< true << QByteArray("Adler32:bad")
<< 1;
QTest::newRow("Same mtime, matching weak checksum -> skipped")
<< true << QByteArray("Adler32:2a2010d")
<< 0;
QTest::newRow("Same mtime, strong server checksum differ -> downloaded")
<< true << QByteArray("SHA1:bad")
<< 1;
QTest::newRow("Same mtime, matching strong checksum -> skipped")
<< true << QByteArray("SHA1:56900fb1d337cf7237ff766276b9c1e8ce507427")
<< 0;
QTest::newRow("mtime changed, but no server checksum -> download")
<< false << QByteArray()
<< 1;
QTest::newRow("mtime changed, weak checksum match -> download anyway")
<< false << QByteArray("Adler32:2a2010d")
<< 1;
QTest::newRow("mtime changed, strong checksum match -> skip")
<< false << QByteArray("SHA1:56900fb1d337cf7237ff766276b9c1e8ce507427")
<< 0;
}
Compare the hash of files with identical mtime/size #5589 * For conflicts where mtime and size are identical: a) If there's no remote checksum, skip (unchanged) b) If there's a remote checksum that's a useful hash, create a PropagateDownload job and compute the local hash. If the hashes are identical, don't download the file and just update metadata. * Avoid exposing the existence of checksumTypeId beyond the database layer. This makes handling checksums easier in general because they can usually be treated as a single blob. This change was prompted by the difficulty of producing file_stat_t entries uniformly from PROPFINDs and the database.
2017-06-14 13:14:46 +03:00
void testFakeConflict()
{
test/testsyncengine: test for PR #6168
2017-11-16 14:35:40 +03:00
QFETCH(bool, sameMtime);
QFETCH(QByteArray, checksums);
QFETCH(int, expectedGET);
Compare the hash of files with identical mtime/size #5589 * For conflicts where mtime and size are identical: a) If there's no remote checksum, skip (unchanged) b) If there's a remote checksum that's a useful hash, create a PropagateDownload job and compute the local hash. If the hashes are identical, don't download the file and just update metadata. * Avoid exposing the existence of checksumTypeId beyond the database layer. This makes handling checksums easier in general because they can usually be treated as a single blob. This change was prompted by the difficulty of producing file_stat_t entries uniformly from PROPFINDs and the database.
2017-06-14 13:14:46 +03:00
FakeFolder fakeFolder{ FileInfo::A12_B12_C12_S12() };
int nGET = 0;
fakeFolder.setServerOverride([&](QNetworkAccessManager::Operation op, const QNetworkRequest &) {
if (op == QNetworkAccessManager::GetOperation)
++nGET;
return nullptr;
});
// For directly editing the remote checksum
FileInfo &remoteInfo = dynamic_cast<FileInfo &>(fakeFolder.remoteModifier());
// Base mtime with no ms content (filesystem is seconds only)
Use QDateTime::currentDateTimeUtc instead of the non utc version QDateTime::currentDateTime is terribly slow.
2017-09-25 12:49:11 +03:00
auto mtime = QDateTime::currentDateTimeUtc().addDays(-4);
Compare the hash of files with identical mtime/size #5589 * For conflicts where mtime and size are identical: a) If there's no remote checksum, skip (unchanged) b) If there's a remote checksum that's a useful hash, create a PropagateDownload job and compute the local hash. If the hashes are identical, don't download the file and just update metadata. * Avoid exposing the existence of checksumTypeId beyond the database layer. This makes handling checksums easier in general because they can usually be treated as a single blob. This change was prompted by the difficulty of producing file_stat_t entries uniformly from PROPFINDs and the database.
2017-06-14 13:14:46 +03:00
mtime.setMSecsSinceEpoch(mtime.toMSecsSinceEpoch() / 1000 * 1000);
fakeFolder.localModifier().setContents("A/a1", 'C');
fakeFolder.localModifier().setModTime("A/a1", mtime);
fakeFolder.remoteModifier().setContents("A/a1", 'C');
test/testsyncengine: test for PR #6168
2017-11-16 14:35:40 +03:00
if (!sameMtime)
mtime = mtime.addDays(1);
Compare the hash of files with identical mtime/size #5589 * For conflicts where mtime and size are identical: a) If there's no remote checksum, skip (unchanged) b) If there's a remote checksum that's a useful hash, create a PropagateDownload job and compute the local hash. If the hashes are identical, don't download the file and just update metadata. * Avoid exposing the existence of checksumTypeId beyond the database layer. This makes handling checksums easier in general because they can usually be treated as a single blob. This change was prompted by the difficulty of producing file_stat_t entries uniformly from PROPFINDs and the database.
2017-06-14 13:14:46 +03:00
fakeFolder.remoteModifier().setModTime("A/a1", mtime);
test/testsyncengine: test for PR #6168
2017-11-16 14:35:40 +03:00
remoteInfo.find("A/a1")->checksums = checksums;
Compare the hash of files with identical mtime/size #5589 * For conflicts where mtime and size are identical: a) If there's no remote checksum, skip (unchanged) b) If there's a remote checksum that's a useful hash, create a PropagateDownload job and compute the local hash. If the hashes are identical, don't download the file and just update metadata. * Avoid exposing the existence of checksumTypeId beyond the database layer. This makes handling checksums easier in general because they can usually be treated as a single blob. This change was prompted by the difficulty of producing file_stat_t entries uniformly from PROPFINDs and the database.
2017-06-14 13:14:46 +03:00
QVERIFY(fakeFolder.syncOnce());
test/testsyncengine: test for PR #6168
2017-11-16 14:35:40 +03:00
QCOMPARE(nGET, expectedGET);
Compare the hash of files with identical mtime/size #5589 * For conflicts where mtime and size are identical: a) If there's no remote checksum, skip (unchanged) b) If there's a remote checksum that's a useful hash, create a PropagateDownload job and compute the local hash. If the hashes are identical, don't download the file and just update metadata. * Avoid exposing the existence of checksumTypeId beyond the database layer. This makes handling checksums easier in general because they can usually be treated as a single blob. This change was prompted by the difficulty of producing file_stat_t entries uniformly from PROPFINDs and the database.
2017-06-14 13:14:46 +03:00
PropagateDownload: Adjustments to skipping downloads #6153 Previously we required matching mtimes but that's actually unnecessary when the question is about whether to skip the download. We will still update the file's metadata. Also, adjust behavior when the checksum is weak (Adler32): in these cases we still depend on equal mtimes.
2017-11-16 12:13:21 +03:00
// check that mtime in journal and filesystem agree
QString a1path = fakeFolder.localPath() + "A/a1";
SyncJournalFileRecord a1record;
fakeFolder.syncJournal().getFileRecord(QByteArray("A/a1"), &a1record);
QCOMPARE(a1record._modtime, (qint64)FileSystem::getModTime(a1path));
Compare the hash of files with identical mtime/size #5589 * For conflicts where mtime and size are identical: a) If there's no remote checksum, skip (unchanged) b) If there's a remote checksum that's a useful hash, create a PropagateDownload job and compute the local hash. If the hashes are identical, don't download the file and just update metadata. * Avoid exposing the existence of checksumTypeId beyond the database layer. This makes handling checksums easier in general because they can usually be treated as a single blob. This change was prompted by the difficulty of producing file_stat_t entries uniformly from PROPFINDs and the database.
2017-06-14 13:14:46 +03:00
// Extra sync reads from db, no difference
QVERIFY(fakeFolder.syncOnce());
test/testsyncengine: test for PR #6168
2017-11-16 14:35:40 +03:00
QCOMPARE(nGET, expectedGET);
Compare the hash of files with identical mtime/size #5589 * For conflicts where mtime and size are identical: a) If there's no remote checksum, skip (unchanged) b) If there's a remote checksum that's a useful hash, create a PropagateDownload job and compute the local hash. If the hashes are identical, don't download the file and just update metadata. * Avoid exposing the existence of checksumTypeId beyond the database layer. This makes handling checksums easier in general because they can usually be treated as a single blob. This change was prompted by the difficulty of producing file_stat_t entries uniformly from PROPFINDs and the database.
2017-06-14 13:14:46 +03:00
}
SyncEngine: Add unittest for SyncFileItem properties #5855 Checks instruction, direction, size, modtime for three common cases.
2017-06-28 12:15:22 +03:00
/**
* Checks whether SyncFileItems have the expected properties before start
* of propagation.
*/
void testSyncFileItemProperties()
{
Use QDateTime::currentDateTimeUtc instead of the non utc version QDateTime::currentDateTime is terribly slow.
2017-09-25 12:49:11 +03:00
auto initialMtime = QDateTime::currentDateTimeUtc().addDays(-7);
auto changedMtime = QDateTime::currentDateTimeUtc().addDays(-4);
auto changedMtime2 = QDateTime::currentDateTimeUtc().addDays(-3);
SyncEngine: Add unittest for SyncFileItem properties #5855 Checks instruction, direction, size, modtime for three common cases.
2017-06-28 12:15:22 +03:00
// Base mtime with no ms content (filesystem is seconds only)
initialMtime.setMSecsSinceEpoch(initialMtime.toMSecsSinceEpoch() / 1000 * 1000);
changedMtime.setMSecsSinceEpoch(changedMtime.toMSecsSinceEpoch() / 1000 * 1000);
changedMtime2.setMSecsSinceEpoch(changedMtime2.toMSecsSinceEpoch() / 1000 * 1000);
SyncEngineTest: Fix test reliability There was a rounding issue in the mtimes which sometimes resulted in an off-by-one error. Caused by storing a full QDateTime in the FileInfo but the mtime saved to the disk being truncated to seconds.
2017-07-04 10:57:22 +03:00
// Ensure the initial mtimes are as expected
auto initialFileInfo = FileInfo::A12_B12_C12_S12();
initialFileInfo.setModTime("A/a1", initialMtime);
initialFileInfo.setModTime("B/b1", initialMtime);
initialFileInfo.setModTime("C/c1", initialMtime);
FakeFolder fakeFolder{ initialFileInfo };
SyncEngine: Add unittest for SyncFileItem properties #5855 Checks instruction, direction, size, modtime for three common cases.
2017-06-28 12:15:22 +03:00
// upload a
fakeFolder.localModifier().appendByte("A/a1");
fakeFolder.localModifier().setModTime("A/a1", changedMtime);
// download b
fakeFolder.remoteModifier().appendByte("B/b1");
fakeFolder.remoteModifier().setModTime("B/b1", changedMtime);
// conflict c
fakeFolder.localModifier().appendByte("C/c1");
fakeFolder.localModifier().appendByte("C/c1");
fakeFolder.localModifier().setModTime("C/c1", changedMtime);
fakeFolder.remoteModifier().appendByte("C/c1");
fakeFolder.remoteModifier().setModTime("C/c1", changedMtime2);
connect(&fakeFolder.syncEngine(), &SyncEngine::aboutToPropagate, [&](SyncFileItemVector &items) {
SyncFileItemPtr a1, b1, c1;
for (auto &item : items) {
if (item->_file == "A/a1")
a1 = item;
if (item->_file == "B/b1")
b1 = item;
if (item->_file == "C/c1")
c1 = item;
}
// a1: should have local size and modtime
QVERIFY(a1);
QCOMPARE(a1->_instruction, CSYNC_INSTRUCTION_SYNC);
QCOMPARE(a1->_direction, SyncFileItem::Up);
SyncEngineTest: _size now correct for uploads #5855
2017-07-12 11:09:04 +03:00
QCOMPARE(a1->_size, quint64(5));
SyncEngine: Add unittest for SyncFileItem properties #5855 Checks instruction, direction, size, modtime for three common cases.
2017-06-28 12:15:22 +03:00
QCOMPARE(Utility::qDateTimeFromTime_t(a1->_modtime), changedMtime);
Remove SyncFileItem::log This remove the remaining "other" fields of the sync log to save a bit of memory. other_etag and other_fileId don't give much information to the users and other_instruction will always be INST_NONE anyway. other_modtime and other_size are kept since they are sometimes used. They were renamed to have a bit more meaningful name. SyncEngine::checkPermissions will now fetch its information from the csync trees since they are now preserved until right after this point. Fixes #3213
2017-08-24 18:07:22 +03:00
QCOMPARE(a1->_previousSize, quint64(4));
QCOMPARE(Utility::qDateTimeFromTime_t(a1->_previousModtime), initialMtime);
SyncEngine: Add unittest for SyncFileItem properties #5855 Checks instruction, direction, size, modtime for three common cases.
2017-06-28 12:15:22 +03:00
// b2: should have remote size and modtime
QVERIFY(b1);
QCOMPARE(b1->_instruction, CSYNC_INSTRUCTION_SYNC);
QCOMPARE(b1->_direction, SyncFileItem::Down);
QCOMPARE(b1->_size, quint64(17));
QCOMPARE(Utility::qDateTimeFromTime_t(b1->_modtime), changedMtime);
Remove SyncFileItem::log This remove the remaining "other" fields of the sync log to save a bit of memory. other_etag and other_fileId don't give much information to the users and other_instruction will always be INST_NONE anyway. other_modtime and other_size are kept since they are sometimes used. They were renamed to have a bit more meaningful name. SyncEngine::checkPermissions will now fetch its information from the csync trees since they are now preserved until right after this point. Fixes #3213
2017-08-24 18:07:22 +03:00
QCOMPARE(b1->_previousSize, quint64(16));
QCOMPARE(Utility::qDateTimeFromTime_t(b1->_previousModtime), initialMtime);
SyncEngine: Add unittest for SyncFileItem properties #5855 Checks instruction, direction, size, modtime for three common cases.
2017-06-28 12:15:22 +03:00
// c1: conflicts are downloads, so remote size and modtime
QVERIFY(c1);
QCOMPARE(c1->_instruction, CSYNC_INSTRUCTION_CONFLICT);
QCOMPARE(c1->_direction, SyncFileItem::None);
QCOMPARE(c1->_size, quint64(25));
QCOMPARE(Utility::qDateTimeFromTime_t(c1->_modtime), changedMtime2);
Remove SyncFileItem::log This remove the remaining "other" fields of the sync log to save a bit of memory. other_etag and other_fileId don't give much information to the users and other_instruction will always be INST_NONE anyway. other_modtime and other_size are kept since they are sometimes used. They were renamed to have a bit more meaningful name. SyncEngine::checkPermissions will now fetch its information from the csync trees since they are now preserved until right after this point. Fixes #3213
2017-08-24 18:07:22 +03:00
QCOMPARE(c1->_previousSize, quint64(26));
QCOMPARE(Utility::qDateTimeFromTime_t(c1->_previousModtime), changedMtime);
SyncEngine: Add unittest for SyncFileItem properties #5855 Checks instruction, direction, size, modtime for three common cases.
2017-06-28 12:15:22 +03:00
});
QVERIFY(fakeFolder.syncOnce());
}
PropagateUpload: Model of remote quota, avoid some uploads #5537 When we see a 507 error, assume that quota is < uploaded size.
2017-07-12 10:58:15 +03:00
/**
* Checks whether subsequent large uploads are skipped after a 507 error
*/
void testInsufficientRemoteStorage()
{
FakeFolder fakeFolder{ FileInfo::A12_B12_C12_S12() };
// Disable parallel uploads
SyncOptions syncOptions;
syncOptions._parallelNetworkJobs = false;
fakeFolder.syncEngine().setSyncOptions(syncOptions);
// Produce an error based on upload size
int remoteQuota = 1000;
int n507 = 0, nPUT = 0;
auto parent = new QObject;
fakeFolder.setServerOverride([&](QNetworkAccessManager::Operation op, const QNetworkRequest &request) -> QNetworkReply * {
if (op == QNetworkAccessManager::PutOperation) {
nPUT++;
if (request.rawHeader("OC-Total-Length").toInt() > remoteQuota) {
n507++;
return new FakeErrorReply(op, request, parent, 507);
}
}
return nullptr;
});
fakeFolder.localModifier().insert("A/big", 800);
QVERIFY(fakeFolder.syncOnce());
QCOMPARE(nPUT, 1);
QCOMPARE(n507, 0);
nPUT = 0;
fakeFolder.localModifier().insert("A/big1", 500); // ok
fakeFolder.localModifier().insert("A/big2", 1200); // 507 (quota guess now 1199)
fakeFolder.localModifier().insert("A/big3", 1200); // skipped
fakeFolder.localModifier().insert("A/big4", 1500); // skipped
fakeFolder.localModifier().insert("A/big5", 1100); // 507 (quota guess now 1099)
fakeFolder.localModifier().insert("A/big6", 900); // ok (quota guess now 199)
fakeFolder.localModifier().insert("A/big7", 200); // skipped
fakeFolder.localModifier().insert("A/big8", 199); // ok (quota guess now 0)
fakeFolder.localModifier().insert("B/big8", 1150); // 507
QVERIFY(!fakeFolder.syncOnce());
QCOMPARE(nPUT, 6);
QCOMPARE(n507, 3);
}
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
// Checks whether downloads with bad checksums are accepted
void testChecksumValidation()
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
{
FakeFolder fakeFolder{ FileInfo::A12_B12_C12_S12() };
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
QObject parent;
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
QByteArray checksumValue;
QByteArray contentMd5Value;
fakeFolder.setServerOverride([&](QNetworkAccessManager::Operation op, const QNetworkRequest &request) -> QNetworkReply * {
if (op == QNetworkAccessManager::GetOperation) {
auto reply = new FakeGetReply(fakeFolder.remoteModifier(), op, request, &parent);
if (!checksumValue.isNull())
reply->setRawHeader("OC-Checksum", checksumValue);
if (!contentMd5Value.isNull())
reply->setRawHeader("Content-MD5", contentMd5Value);
return reply;
}
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
return nullptr;
});
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
// Basic case
fakeFolder.remoteModifier().create("A/a3", 16, 'A');
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
QVERIFY(fakeFolder.syncOnce());
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
// Bad OC-Checksum
checksumValue = "SHA1:bad";
fakeFolder.remoteModifier().create("A/a4", 16, 'A');
QVERIFY(!fakeFolder.syncOnce());
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
// Good OC-Checksum
checksumValue = "SHA1:19b1928d58a2030d08023f3d7054516dbc186f20"; // printf 'A%.0s' {1..16} | sha1sum -
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
QVERIFY(fakeFolder.syncOnce());
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
checksumValue = QByteArray();
// Bad Content-MD5
contentMd5Value = "bad";
fakeFolder.remoteModifier().create("A/a5", 16, 'A');
QVERIFY(!fakeFolder.syncOnce());
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
// Good Content-MD5
contentMd5Value = "d8a73157ce10cd94a91c2079fc9a92c8"; // printf 'A%.0s' {1..16} | md5sum -
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
QVERIFY(fakeFolder.syncOnce());
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
Checksum: Ignore unkown OC-Checksum header when downloading... And if there are several checksums, pick the "best" one. The case of several checksum was reported in https://github.com/nextcloud/client_theming/issues/213
2017-11-20 10:18:52 +03:00
// Invalid OC-Checksum is ignored
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
checksumValue = "garbage";
// contentMd5Value is still good
fakeFolder.remoteModifier().create("A/a6", 16, 'A');
Checksum: Ignore unkown OC-Checksum header when downloading... And if there are several checksums, pick the "best" one. The case of several checksum was reported in https://github.com/nextcloud/client_theming/issues/213
2017-11-20 10:18:52 +03:00
QVERIFY(fakeFolder.syncOnce());
contentMd5Value = "bad";
fakeFolder.remoteModifier().create("A/a7", 16, 'A');
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
QVERIFY(!fakeFolder.syncOnce());
Checksum: Ignore unkown OC-Checksum header when downloading... And if there are several checksums, pick the "best" one. The case of several checksum was reported in https://github.com/nextcloud/client_theming/issues/213
2017-11-20 10:18:52 +03:00
contentMd5Value.clear();
QVERIFY(fakeFolder.syncOnce());
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
// OC-Checksum contains Unsupported checksums
checksumValue = "Unsupported:XXXX SHA1:invalid Invalid:XxX";
fakeFolder.remoteModifier().create("A/a8", 16, 'A');
QVERIFY(!fakeFolder.syncOnce()); // Since the supported SHA1 checksum is invalid, no download
checksumValue = "Unsupported:XXXX SHA1:19b1928d58a2030d08023f3d7054516dbc186f20 Invalid:XxX";
QVERIFY(fakeFolder.syncOnce()); // The supported SHA1 checksum is valid now, so the file are downloaded
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
PropagateDownload: Read Content-md5 header #6088
2017-10-10 14:24:41 +03:00
}
Sync: Add capability for invalid filename regexes #6092
2017-10-13 15:56:40 +03:00
// Tests the behavior of invalid filename detection
void testInvalidFilenameRegex()
{
FakeFolder fakeFolder{ FileInfo::A12_B12_C12_S12() };
// For current servers, no characters are forbidden
fakeFolder.syncEngine().account()->setServerVersion("10.0.0");
fakeFolder.localModifier().insert("A/\\:?*\"<>|.txt");
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
QVERIFY(fakeFolder.syncOnce());
Sync: Add capability for invalid filename regexes #6092
2017-10-13 15:56:40 +03:00
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
// For legacy servers, some characters were forbidden by the client
fakeFolder.syncEngine().account()->setServerVersion("8.0.0");
fakeFolder.localModifier().insert("B/\\:?*\"<>|.txt");
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
QVERIFY(fakeFolder.syncOnce());
Sync: Add capability for invalid filename regexes #6092
2017-10-13 15:56:40 +03:00
QVERIFY(!fakeFolder.currentRemoteState().find("B/\\:?*\"<>|.txt"));
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
Sync: Add capability for invalid filename regexes #6092
2017-10-13 15:56:40 +03:00
// We can override that by setting the capability
fakeFolder.syncEngine().account()->setCapabilities({ { "dav", QVariantMap{ { "invalidFilenameRegex", "" } } } });
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
QVERIFY(fakeFolder.syncOnce());
Sync: Add capability for invalid filename regexes #6092
2017-10-13 15:56:40 +03:00
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
// Check that new servers also accept the capability
fakeFolder.syncEngine().account()->setServerVersion("10.0.0");
fakeFolder.syncEngine().account()->setCapabilities({ { "dav", QVariantMap{ { "invalidFilenameRegex", "my[fgh]ile" } } } });
fakeFolder.localModifier().insert("C/myfile.txt");
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
QVERIFY(fakeFolder.syncOnce());
Sync: Add capability for invalid filename regexes #6092
2017-10-13 15:56:40 +03:00
QVERIFY(!fakeFolder.currentRemoteState().find("C/myfile.txt"));
Rename detection: File size must be equal Comparison of file sizes for potential conflicts was added in 0eb9401c624f20a128b46f8eb1fa5a984f9ef61e, but did not extend to checking the file size in case of potential local moves. This commit adds this check and adds tests for various move+change scenarios.
2017-10-05 12:39:35 +03:00
}
Local discovery: Use db instead of filesystem We mostly trust the file watchers meaning that we don't re-scan the local tree if we have done that recently and no file watcher events have arrived. If the file watchers invalidate a subtree, we rescan only that subtree. Since we're not entirely sure the file watchers are reliable, we still do full local discoveries regularly (1h by default). There is a config file setting as well as an environment variable to control the interval.
2017-09-14 17:43:23 +03:00
// Check correct behavior when local discovery is partially drawn from the db
void testLocalDiscoveryStyle()
{
FakeFolder fakeFolder{ FileInfo::A12_B12_C12_S12() };
// More subdirectories are useful for testing
fakeFolder.localModifier().mkdir("A/X");
fakeFolder.localModifier().mkdir("A/Y");
fakeFolder.localModifier().insert("A/X/x1");
fakeFolder.localModifier().insert("A/Y/y1");
QVERIFY(fakeFolder.syncOnce());
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
// Test begins
fakeFolder.localModifier().insert("A/a3");
fakeFolder.localModifier().insert("A/X/x2");
fakeFolder.localModifier().insert("A/Y/y2");
fakeFolder.localModifier().insert("B/b3");
fakeFolder.remoteModifier().insert("C/c3");
fakeFolder.syncEngine().setLocalDiscoveryOptions(LocalDiscoveryStyle::DatabaseAndFilesystem, { "A/X" });
QVERIFY(fakeFolder.syncOnce());
QVERIFY(fakeFolder.currentRemoteState().find("A/a3"));
QVERIFY(fakeFolder.currentRemoteState().find("A/X/x2"));
QVERIFY(!fakeFolder.currentRemoteState().find("A/Y/y2"));
QVERIFY(!fakeFolder.currentRemoteState().find("B/b3"));
QVERIFY(fakeFolder.currentLocalState().find("C/c3"));
QCOMPARE(fakeFolder.syncEngine().lastLocalDiscoveryStyle(), LocalDiscoveryStyle::DatabaseAndFilesystem);
QVERIFY(fakeFolder.syncOnce());
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
QCOMPARE(fakeFolder.syncEngine().lastLocalDiscoveryStyle(), LocalDiscoveryStyle::FilesystemOnly);
}
Merge remote-tracking branch 'origin/2.4'
2017-11-20 13:54:56 +03:00
Discovery: Treat files starting with '.' as hidden #6145 This bug was introduced when strcmp(a, b) != 0 was accidentally converted to a == b.
2017-11-08 13:56:31 +03:00
void testDiscoveryHiddenFile()
{
FakeFolder fakeFolder{ FileInfo::A12_B12_C12_S12() };
QVERIFY(fakeFolder.syncOnce());
QCOMPARE(fakeFolder.currentLocalState(), fakeFolder.currentRemoteState());
// We can't depend on currentLocalState for hidden files since
// it should rightfully skip things like download temporaries
auto localFileExists = [&](QString name) {
return QFileInfo(fakeFolder.localPath() + name).exists();
};
fakeFolder.syncEngine().setIgnoreHiddenFiles(true);
fakeFolder.remoteModifier().insert("A/.hidden");
fakeFolder.localModifier().insert("B/.hidden");
QVERIFY(fakeFolder.syncOnce());
QVERIFY(!localFileExists("A/.hidden"));
QVERIFY(!fakeFolder.currentRemoteState().find("B/.hidden"));
fakeFolder.syncEngine().setIgnoreHiddenFiles(false);
fakeFolder.syncJournal().forceRemoteDiscoveryNextSync();
QVERIFY(fakeFolder.syncOnce());
QVERIFY(localFileExists("A/.hidden"));
QVERIFY(fakeFolder.currentRemoteState().find("B/.hidden"));
}
Add TestSyncEngine and TestSyncFileStatusTracker auto tests To be able to test the SyncEngine efficiently, a set of server mocking classes have been implemented on top of QNetworkAccessManager. The local disk side hasn't been mocked since this would require adding a large abstraction layer in csync. The SyncEngine is instead pointed to a different temporary dir in each test and we test by interacting with files in this directory instead. The FakeFolder object wraps the SyncEngine with those abstractions and allow controlling the local files, and the fake remote state through the FileModifier interface, using a FileInfo tree structure for the remote-side implementation as well as feeding and comparing the states on both side in tests. Tests run fast and require no setup to be run, but each server feature that we want to test on the client side needs to be implemented in this fake objects library. For example, the OC-FileId header isn't set as of this commit, and we can't test the file move logic properly without implementing it first. The TestSyncFileStatusTracker tests already contain a few QEXPECT_FAIL for what I esteem being issues that need to be fixed in order to catch up on our test coverage without making this patch too huge.
2016-08-04 15:59:46 +03:00
};
QTEST_GUILESS_MAIN(TestSyncEngine)
#include "testsyncengine.moc"
Reference in a new issue Copy permalink