Client Testing
[Native API]

The testing framework of the C++ client library is based on CPPUnit. More...

Collaboration diagram for Client Testing:

Classes

class  ClientTestFactory
 generates tests on demand based on what the client supports More...
class  TestFileSource
 This code uses the ClientTest and TestFileSyncSource to test real synchronization against a server. More...

Functions

void simplifyFilename (string &filename)
const string & getCurrentTest ()
static void printTests (CppUnit::Test *test, int indention)
int main (int argc, char *argv[])
static USE_NAMESPACE std::list
< std::string > 
listAnyItems (SyncSource *source, SyncItem *(SyncSource::*first)(), SyncItem *(SyncSource::*next)())
 utility function to iterate over different kinds of items in a sync source
static std::list< std::string > listNewItems (SyncSource *source)
static std::list< std::string > listUpdatedItems (SyncSource *source)
static std::list< std::string > listDeletedItems (SyncSource *source)
static std::list< std::string > listItems (SyncSource *source)
std::list< std::string > listItemsOfType (SyncSource *source, itemType type)
 generates list of UIDs in the specified kind of items
static int countNewItems (SyncSource *source)
static int countUpdatedItems (SyncSource *source)
static int countDeletedItems (SyncSource *source)
static int countItems (SyncSource *source)
int countItemsOfType (SyncSource *source, itemType type)
 utility function which counts items of a certain kind known to the sync source
static void importItem (SyncSource *source, std::string &data)
static std::string updateItem (CreateSource createSource, const std::string &uid, const char *data)
 deletes specific item locally via sync source
static void deleteItem (CreateSource createSource, const std::string &uid)
 deletes specific item locally via sync source
template<class T , class V >
int countEqual (const T &container, const V &value)
static USE_NAMESPACE
SyncSourceConfig
createBriefcaseDefaultConfig ()
static void setDefaults (SyncManagerConfig *config, const char *testName, const bool setClientDefaults, ArrayList *defaultSources=NULL)
 Sets the default values (from DefaultConfigFactory) in the passed SyncManagerConfig.
const StringBuffer initAdapter (const char *testName)
 Initializes the PlatformAdapter for a given test.
SyncManagerConfiggetNewSyncManagerConfig (const char *testName, const bool setClientDefaults, ArrayList *defaultSources)
 Generates and returns a NEW ALLOCATED default configuration (SyncManagerConfig).
DMTClientConfiggetNewDMTClientConfig (const char *testName, const bool setClientDefaults, ArrayList *defaultSources)
 Generates and returns a NEW ALLOCATED default configuration (DMTClientConfig).
char * loadTestFile (const char *testName, const char *fileName, bool binary)
 Loads and returns (NEW ALLOCATED) the content of a test file.
StringBuffer getTestFileFullPath (const char *testName, const char *fileName)
 Returns the full path of a given test file (concat "testcases/testName/fileName").
StringBuffer getTestDirFullPath (const char *testName)
 Returns the full path of a given test file (concat "testcases/testName/") with the final slash at the end.
virtual void LocalTests::addTests ()
 adds the supported tests to the instance itself; this is the function that a derived class can override to add additional tests
virtual std::string LocalTests::insert (CreateSource createSource, const char *data, bool relaxed=false)
 opens source and inserts the given item; can be called regardless whether the data source already contains items or not
virtual void LocalTests::update (CreateSource createSource, const char *data, bool check=true)
 assumes that exactly one element is currently inserted and updates it with the given item
virtual void LocalTests::deleteAll (CreateSource createSource)
 deletes all items locally via sync source
virtual void LocalTests::compareDatabases (const char *refFile, SyncSource &copy, bool raiseAssert=true)
 takes two databases, exports them, then compares them using synccompare
virtual int LocalTests::insertManyItems (CreateSource createSource, int startIndex=1, int numItems=0, int size=-1)
 insert artificial items, number of them determined by TEST_EVOLUTION_NUM_ITEMS unless passed explicitly
virtual void LocalTests::testOpen ()
virtual void LocalTests::testIterateTwice ()
virtual void LocalTests::testSimpleInsert ()
virtual void LocalTests::testLocalDeleteAll ()
virtual void LocalTests::testComplexInsert ()
virtual void LocalTests::testLocalUpdate ()
virtual void LocalTests::testChanges ()
virtual void LocalTests::testImport ()
virtual void LocalTests::testImportDelete ()
virtual void LocalTests::testManyChanges ()
virtual void LocalTests::testLinkedItemsParent ()
virtual void LocalTests::testLinkedItemsChild ()
virtual void LocalTests::testLinkedItemsParentChild ()
virtual void LocalTests::testLinkedItemsChildParent ()
virtual void LocalTests::testLinkedItemsChildChangesParent ()
virtual void LocalTests::testLinkedItemsRemoveParentFirst ()
virtual void LocalTests::testLinkedItemsRemoveNormal ()
virtual void LocalTests::testLinkedItemsInsertParentTwice ()
virtual void LocalTests::testLinkedItemsInsertChildTwice ()
virtual void LocalTests::testLinkedItemsParentUpdate ()
virtual void LocalTests::testLinkedItemsUpdateChild ()
virtual void LocalTests::testLinkedItemsInsertBothUpdateChild ()
virtual void LocalTests::testLinkedItemsInsertBothUpdateParent ()
 SyncTests::SyncTests (const std::string &name, ClientTest &cl, std::vector< int > sourceIndices, bool isClientA=true)
virtual void SyncTests::addTests ()
 adds the supported tests to the instance itself
virtual void SyncTests::compareDatabases ()
 compare databases of first and second client
virtual void SyncTests::deleteAll (DeleteAllMode mode=DELETE_ALL_SYNC)
 deletes all items locally and on server
virtual void SyncTests::doCopy ()
 get both clients in sync with empty server, then copy one item from client A to B
virtual void SyncTests::refreshClient ()
 replicate server database locally: same as SYNC_REFRESH_FROM_SERVER, but done with explicit local delete and then a SYNC_SLOW because some servers do no support SYNC_REFRESH_FROM_SERVER
virtual void SyncTests::testDeleteAllRefresh ()
virtual void SyncTests::testRefreshSemantic ()
virtual void SyncTests::testRefreshStatus ()
virtual void SyncTests::testUpdate ()
virtual void SyncTests::testComplexUpdate ()
virtual void SyncTests::testDelete ()
virtual void SyncTests::testMerge ()
virtual void SyncTests::testTwinning ()
virtual void SyncTests::testOneWayFromServer ()
virtual void SyncTests::testOneWayFromClient ()
virtual void SyncTests::testItems ()
virtual void SyncTests::testAddUpdate ()
virtual void SyncTests::testManyItems ()
virtual void SyncTests::doVarSizes (bool withMaxMsgSize, bool withLargeObject, const char *encoding)
 implements testMaxMsg(), testLargeObject(), testLargeObjectEncoded() using a sequence of items with varying sizes
virtual void SyncTests::sync (SyncMode syncMode, const std::string &logprefix="", CheckSyncReport checkReport=CheckSyncReport(), long maxMsgSize=0, long maxObjSize=0, bool loSupport=false, const char *encoding="")
 executes a sync with the given options, checks the result and (optionally) the sync report
virtual void ClientTest::registerTests ()
 This function registers tests using this instance of ClientTest for later use during a test run.
 ClientTest::ClientTest (int serverSleepSec=0, const std::string &serverLog="")
virtual LocalTestsClientTest::createLocalTests (const std::string &name, int sourceParam, ClientTest::Config &co)
 Creates an instance of LocalTests (default implementation) or a class derived from it.
virtual SyncTestsClientTest::createSyncTests (const std::string &name, std::vector< int > sourceIndices, bool isClientA=true)
 Creates an instance of SyncTests (default) or a class derived from it.
static int ClientTest::dump (ClientTest &client, SyncSource &source, const char *file)
 utility function for dumping items which are C strings with blank lines as separator
static int ClientTest::import (ClientTest &client, SyncSource &source, const char *file)
 utility function for importing items with blank lines as separator
static bool ClientTest::compare (ClientTest &client, const char *fileA, const char *fileB)
 utility function for comparing vCard and iCal files with the external synccompare.pl Perl script
virtual void ClientTest::postSync (int res, const std::string &logname)
 This is called after successful sync() calls (res == 0) as well as after unsuccessful ones (res != 1).
static void ClientTest::getTestData (const char *type, Config &config)
 A derived class can use this call to get default test cases, but still has to add callbacks which create sources and execute a sync session.
virtual void CheckSyncReport::check (int res, SyncReport &report) const
 checks that the sync completed as expected and throws CPPUnit exceptions if something is wrong
 TestFileSource::TestFileSource (const std::string &id)
virtual void TestFileSource::getSourceConfig (int source, Config &config)
 Called to fill the given test source config with information about a sync source identified by its index.
virtual int TestFileSource::sync (const int *activeSources, SyncMode syncMode, const CheckSyncReport &checkReport, long maxMsgSize, long maxObjSize, bool loSupport, const char *encoding=0)
 Execute a synchronization with the selected sync sources and the selected synchronization options.
SyncSourceTestFileSource::createSource (int source, const char *trackingSuffix)

Variables

ClientListener syncListener

Detailed Description

The testing framework of the C++ client library is based on CPPUnit.

It fulfills several different goals:

Some of the design goals were:

The test runner "client-test" is implemented in client-test-main.cpp. It automatically assembles all CPPUnit tests it was linked against and runs them all if started without parameters. Output is ASCII. The return code indicates success or failure of any test. "-h|--help" print some usage information and a full list of all available tests. Tests or test groups as printed in that list can be given as command line parameters to run just these tests. Because tests do not communicate directly with the command line front-end, they must be passed parameters via environment variables. See the description of TestFileSource for further information about running those end-to-end synchronization tests.

The test runner itself also understands some environment variables: CLIENT_TEST_FAILURES can be set to a comma separated list of tests which are allowed to fail without affecting the return code of the test runner.

On Linux the test runner supports setting a timeout which is triggered if a test runs for more than the number of seconds set in the environment variable CLIENT_TEST_ALARM. This relies on signals and therefore is disabled in compilations on Windows with an ifdef HAVE_SIGNALS_H.

Because of the dependency on CPPUnit the testing needs to be enabled explicitly. This can be done separately for tests embedded in the object files of the library (unit tests) and the end-to-end synchronization (integration tests) which can be used with release versions of the library.

On POSIX systems the configure --enable-integration-test and --enable-unit-tests switches enable the testing. "make check" automatically executes the unit tests.

On Windows the Visual Studio solution file build/win32/client-test.sln was prepared to build the "client-test" executable. The cppunit project is required when loading the client-test solution, so you need to retrieve it and put it under the "tools/cppunit" directory. The cppunit project can be checkout from: https://core.forge.funambol.org/svn/core/trunk/tools/cppunit

Unit tests are only found if they are pulled into the executable. On Linux this is done by searching for variables with a special name, created by the FUNAMBOL_TEST_SUITE_REGISTRATION() macro in test.h, and setting link flags on the fly. On Windows an additional C file with references to these variables is used instead. This is done by a Python script, thus python.exe must be installed and found in the search path of Visual Studio. Any suggestion how to avoid the dependency on external scripting tools without having to manually register tests in a central file is welcome...

The integration tests depend on auxiliary files (test/synccompare.pl and test/testcases) which must be found in the directory where the test is run. On Linux the makefile rules automatically copy these, but the Visual Studio project file does not. Note that test/synccompare.pl is written in Perl (and thus creates another external dependency) because a) C/C++ do not have a standard library to implement the complex regular pattern matching/replace done in that script and b) the script is also meant to be shipped with clients (SyncEvolution does that) and thus needs to run on as many systems as possible. Perl is still the most universally available scripting language for that purpose.

Adding new unit tests is best done by following the example in src/c++/common/base/util/BasicTime.cpp.

Regular testing of a server can be done by using the reference data provided together with ClientTest and synchronizing the server against simple files on the client side via a TestFileSource. This is what enabling the integration tests adds to the "client-test" binary. By keeping the client side fixed regressions in the server can be detected. Regular testing of the client library can be done by testing the current client library source against stable servers.

For testing a client based on the C++ library the client developer needs to implement the ClientTest interface. There are some additional requirements for the client's SyncSource implementations:

The TestFileSource is an example how the ClientTest interface could be implemented.


Function Documentation

void LocalTests::addTests (  )  [virtual, inherited]

void CheckSyncReport::check ( int  res,
SyncReport report 
) const [virtual, inherited]

checks that the sync completed as expected and throws CPPUnit exceptions if something is wrong

Parameters:
res return code from SyncClient::sync()
report the sync report stored in the SyncClient

References CLIENT, SyncSourceReport::getItemReportFailedCount(), SyncSourceReport::getItemReportSuccessfulCount(), SyncSourceReport::getSourceName(), SyncSourceReport::getState(), SyncReport::getSyncSourceReport(), SOURCE_INACTIVE, and SyncReport::toString().

Referenced by TestFileSource::sync().

void LocalTests::compareDatabases ( const char *  refFile,
SyncSource copy,
bool  raiseAssert = true 
) [virtual, inherited]

takes two databases, exports them, then compares them using synccompare

Parameters:
refFile existing file with source reference items, NULL uses a dump of sync source A instead
copy a sync source which contains the copied items, begin/endSync will be called
raiseAssert raise assertion if comparison yields differences (defaults to true)

References SyncSource::beginSync(), LocalTests::client, ClientTest::Config::compare, LocalTests::config, LocalTests::createSourceA, ClientTest::Config::dump, and SyncSource::endSync().

int countItemsOfType ( SyncSource source,
itemType  type 
)

utility function which counts items of a certain kind known to the sync source

Parameters:
source valid source ready to iterate; NULL triggers an assert
itemType determines which iterator functions are used
Returns:
number of valid items iterated over

References listItemsOfType().

LocalTests * ClientTest::createLocalTests ( const std::string &  name,
int  sourceParam,
ClientTest::Config co 
) [virtual, inherited]

Creates an instance of LocalTests (default implementation) or a class derived from it.

LocalTests provides tests which cover the SyncSource interface and can be executed without a SyncML server. It also contains utility functions for working with SyncSources.

A ClientTest implementation can, but doesn't have to extend these tests by instantiating a derived class here.

SyncTests * ClientTest::createSyncTests ( const std::string &  name,
std::vector< int >  sourceIndices,
bool  isClientA = true 
) [virtual, inherited]

Creates an instance of SyncTests (default) or a class derived from it.

SyncTests provides tests which cover the actual interaction with a SyncML server.

A ClientTest implementation can, but doesn't have to extend these tests by instantiating a derived class here.

DMTClientConfig* getNewDMTClientConfig ( const char *  testName,
const bool  setClientDefaults,
ArrayList *  defaultSources = NULL 
)

Generates and returns a NEW ALLOCATED default configuration (DMTClientConfig).

Parameters:
testName the name of the test which called this method (for deviceID)
setClientDefaults if true, will set the default deviceConfig and accessConfig
defaultSources ArrayList of source names (StringBuffer), to create default SyncSourceConfig. If not specified, non is created. Default is NULL.
Returns:
the (new allocated) DMTClientConfig

References initAdapter(), and setDefaults().

SyncManagerConfig* getNewSyncManagerConfig ( const char *  testName,
const bool  setClientDefaults,
ArrayList *  defaultSources = NULL 
)

Generates and returns a NEW ALLOCATED default configuration (SyncManagerConfig).

Parameters:
testName the name of the test which called this method (for deviceID)
setClientDefaults if true, will set the default deviceConfig and accessConfig
defaultSources ArrayList of source names (StringBuffer), to create default SyncSourceConfig. If not specified, non is created. Default is NULL.
Returns:
the (new allocated) SyncManagerConfig

References initAdapter(), and setDefaults().

void TestFileSource::getSourceConfig ( int  source,
Config config 
) [virtual, inherited]

Called to fill the given test source config with information about a sync source identified by its index.

It's okay to only fill in the available pieces of information and set everything else to zero.

Implements ClientTest.

References ClientTest::Config::createSourceA, ClientTest::Config::createSourceB, getTestData(), and TestFileSource::sources.

Referenced by TestFileSource::TestFileSource().

void ClientTest::getTestData ( const char *  type,
Config config 
) [static, inherited]

A derived class can use this call to get default test cases, but still has to add callbacks which create sources and execute a sync session.

Some of the test cases are compiled into the library, other depend on the auxiliary files from the "test" directory. Currently supported types:

  • vcard30 = vCard 3.0 contacts
  • vcard21 = vCard 2.1 contacts
  • ical20 = iCal 2.0 events
  • vcal10 = vCal 1.0 events
  • itodo20 = iCal 2.0 tasks

References ClientTest::Config::childItem, compare(), ClientTest::Config::compare, ClientTest::Config::complexUpdateItem, dump(), ClientTest::Config::dump, ClientTest::Config::import, ClientTest::Config::insertItem, ClientTest::Config::mergeItem1, ClientTest::Config::mergeItem2, ClientTest::Config::numItems, ClientTest::Config::parentItem, ClientTest::Config::sizeProperty, ClientTest::Config::sourceName, ClientTest::Config::templateItem, ClientTest::Config::testcases, ClientTest::Config::type, ClientTest::Config::uniqueProperties, ClientTest::Config::updateItem, and ClientTest::Config::uri.

Referenced by TestFileSource::getSourceConfig().

StringBuffer getTestDirFullPath ( const char *  testName  ) 

Returns the full path of a given test file (concat "testcases/testName/") with the final slash at the end.

Parameters:
testName the test name
Returns:
the full path of the test file

StringBuffer getTestFileFullPath ( const char *  testName,
const char *  fileName 
)

Returns the full path of a given test file (concat "testcases/testName/fileName").

Parameters:
testName the test name
fileName the file name
Returns:
the full path of the test file

Referenced by utilsTest::getFileModTimeTest(), and loadTestFile().

const StringBuffer initAdapter ( const char *  testName  ) 

Initializes the PlatformAdapter for a given test.

To avoid spamming the config folder, the root context is composed so that all the tests will work under "Funambol/client-test", then the test name.

Referenced by getNewDMTClientConfig(), getNewSyncManagerConfig(), and TestFileSource::TestFileSource().

std::string LocalTests::insert ( CreateSource  createSource,
const char *  data,
bool  relaxed = false 
) [virtual, inherited]

opens source and inserts the given item; can be called regardless whether the data source already contains items or not

The type of the item is unset; it is assumed that the source can handle that.

Parameters:
relaxed if true, then disable some of the additional checks after adding the item
Returns:
the UID of the inserted item

References LocalTests::source.

int LocalTests::insertManyItems ( CreateSource  createSource,
int  startIndex = 1,
int  numItems = 0,
int  size = -1 
) [virtual, inherited]

insert artificial items, number of them determined by TEST_EVOLUTION_NUM_ITEMS unless passed explicitly

insert artificial items, number of them determined by config.numItems unless passed explicitly

Parameters:
createSource a factory for the sync source that is to be used
startIndex IDs are generated starting with this value
numItems number of items to be inserted if non-null, otherwise TEST_EVOLUTION_NUM_ITEMS is used
size minimum size for new items
Returns:
number of items inserted
Parameters:
createSource a factory for the sync source that is to be used
startIndex IDs are generated starting with this value
numItems number of items to be inserted if non-null, otherwise config.numItems is used
size minimum size for new items
Returns:
number of items inserted

References LocalTests::config, LocalTests::createSourceA, ClientTest::Config::numItems, ClientTest::Config::sizeProperty, ClientTest::Config::templateItem, and ClientTest::Config::uniqueProperties.

char* loadTestFile ( const char *  testName,
const char *  fileName,
bool  binary = false 
)

Loads and returns (NEW ALLOCATED) the content of a test file.

The test files are located under the dir "testcases/<test name>".

Parameters:
testName the test name
fileName the file name
binary if true the file has to be opened in binary mode
Returns:
a new allocated buffer with the file content

References getTestFileFullPath().

Referenced by utilsTest::getFileModTimeTest().

USE_NAMESPACE void ClientTest::postSync ( int  res,
const std::string &  logname 
) [virtual, inherited]

This is called after successful sync() calls (res == 0) as well as after unsuccessful ones (res != 1).

The default implementation sleeps for the number of seconds specified when constructing this instance and copies the server log if one was named.

Parameters:
res result of sync()
logname base name of the current sync log (without ".client.[AB].log" suffix)

References serverLogFileName, and serverSleepSeconds.

Referenced by SyncTests::sync().

void ClientTest::registerTests (  )  [virtual, inherited]

This function registers tests using this instance of ClientTest for later use during a test run.

The instance must remain valid until after the tests were run. To run them use a separate test runner, like the one from client-test-main.cpp.

References factory.

static void setDefaults ( SyncManagerConfig config,
const char *  testName,
const bool  setClientDefaults,
ArrayList *  defaultSources = NULL 
) [static]

Sets the default values (from DefaultConfigFactory) in the passed SyncManagerConfig.

Parameters:
config the SyncManagerConfig to fill
testName the name of the test which called this method (for deviceID)
setClientDefaults if true, will set the default deviceConfig and accessConfig
defaultSources ArrayList of source names (StringBuffer), to create default SyncSourceConfig. If not specified, non is created. Default is NULL.

References SyncManagerConfig::getAccessConfig(), SyncManagerConfig::getDeviceConfig(), SyncManagerConfig::setClientDefaults(), SyncManagerConfig::setSourceDefaults(), and SyncManagerConfig::setSyncSourceConfig().

Referenced by getNewDMTClientConfig(), and getNewSyncManagerConfig().

int TestFileSource::sync ( const int *  activeSources,
SyncMode  syncMode,
const CheckSyncReport checkReport,
long  maxMsgSize,
long  maxObjSize,
bool  loSupport,
const char *  encoding = 0 
) [virtual, inherited]

Execute a synchronization with the selected sync sources and the selected synchronization options.

The log file in LOG has been set up already for the synchronization run and should not be changed by the client.

Parameters:
activeSources a -1 terminated array of sync source indices
syncMode the synchronization mode to be used
checkReport has to be called after a successful or unsuccessful sync, will dump the report and (optionally) check the result; beware, the later may throw exceptions inside CPPUNIT macros
maxMsgSize >0: enable the maximum message size, else disable it
maxObjSize same as maxMsgSize for maximum object size
loSupport if true, then the sync is expected to enable Large Object support
encoding if non-empty, then let client library transform all items into this format (guaranteed to be not NULL)
Returns:
return code of SyncClient::sync()

Implements ClientTest.

References CheckSyncReport::check(), TestFileSource::config, SyncClient::getSyncReport(), SyncSourceConfig::setEncoding(), SyncSourceConfig::setSync(), TestFileSource::sources, and SyncClient::sync().

USE_NAMESPACE TestFileSource::TestFileSource ( const std::string &  id  )  [inherited]

void LocalTests::update ( CreateSource  createSource,
const char *  data,
bool  check = true 
) [virtual, inherited]

assumes that exactly one element is currently inserted and updates it with the given item

The type of the item is cleared, as in insert() above.

Parameters:
check if true, then reopen the source and verify that the reported items are as expected
The type of the item is cleared, as in insert() above.

References CreateSource::createSource, and LocalTests::source.


Generated on Tue Apr 6 13:00:40 2010 for Funambol Outlook Plug-in Library by  doxygen 1.5.7.1