kSyncML: New Enhydra Open Source Project
SyncML in Wireless Devices

Today it’s not enough to have your personal information such as e-mail, contacts, calendar entries and appointments stored inside your laptop. I’m not even mentioning desktop computers. People need to access information everywhere, regardless of device, network and location. What’s even more important is not only viewing the information, but ability to modify and synchronize changes back to centralized storage. Not only that, such functionality should be available from devices such as PDA or better yet mobile phone. Mentioned requirements led to creating standard way to exchange and synchronize information between various devices. The independent organization SyncML consortium (www.syncml.org) was formed by industry leaders such as Ericsson, IBM, Nokia and now accounts of several hundred members to promote and oversee development of SyncML - XML-based common mobile data synchronization protocol. As stated in SyncML whitepaper (http://www.syncml.org/technology.html):

“…A data synchronization protocol defines the workflow for communication during a data synchronization session when the mobile device is connected to the network. The protocol must support naming and identification of records, common protocol commands to synchronize local and network data, and it can support identification and resolution of synchronization conflicts…”.

SyncML is transport protocol agnostic. So far standard addresses two transport mechanisms: SyncML over HTTP and SyncML over OBEX, but developers are free to implement SyncML over any transport. SyncML is also agnostic to data format itself. Standard describes synchronizing of common personal data formats such as vCard and vCalendar, but the only restriction applied to document format is ability to name and locate specific record inside document, so SyncML can be used for synchronizing text files, RDBMS tables, XML documents and almost any other existing document formats.

SyncML future looks bright. GSM Association considers SyncML along with Java and MMS as one of the key technologies for handset manufactures to support (see press release, http://www.gsmworld.com/news/press_2001/press_releases_24.html). Several phones already contain SyncML implementations. Just to name a few: Nokia 6310, 7650, Nokia Communicator 9210 and 9290, Ericsson T39m, R520m . It seems that every new mobile phone that includes some sort of personal organizer software uses SyncML to synchronize this data.

Developing custom SyncML engine

Even though SyncML is being widely adopted, developers who want to incorporate SyncML into their mobile application will face the necessity to write their own implementation of SyncML, since most existing solutions are incorporated on mobile phone firmware and do not expose API to developers. Also there aren’t any commercially available SyncML solutions for wireless devices, except maybe SyncML client from Symbian OS. Due to heavy constraints which applied by limited device capabilities, writing custom solution maybe the only choice. In this article we would like to highlight several problems that developers will face implementing custom SyncML engine. Also we will describe our own solution, and our solutions to common problems.

Common problems

XML parser. First of all, SyncML is based on XML, so you need an XML parser to read/write SyncML packets. It’s not a problem on desktop or server where variety of XML parsers are available, but in mobile development the choice is poor. Neither J2ME, BREW, PalmOS or Symbian OS have built-in XML API, so you should write your own one or use third-party library. There are few compact parsers available for J2ME such as kXML (kxml.enhydra.org), MinML (http://www.wilson.co.uk/xml/minml.htm) and NanoXML (nanoxml.sourceforge.net). But for BREW, PalmOS or SymbianOS the only choice we know about is porting James Clark’s expat parser (www.jclark.com/xml/expat.html), which is quite compact and portable. Reaxion will also release XML parser for BREW and other platforms soon. The second problem with XML parser is that DOM parser is too heavy for mobile environment, so developers should stick with SAX-based parser, which is harder to understand, but much thinner. Some parsers like kXML even sacrifice SAX compatibility and implement own lightweight API to decrease library size.

Network problems. Despite all efforts of wireless network operators networks are still very unstable and suffer from problems such as:

  • High network latency
  • Limited bandwidth
  • Low reliability of both data and connectivity.

Keeping all this restrictions in mind developer should choose a right strategy to avoid them. First of all, it probably makes sense to make an asynchronous messages exchange to avoid problems with connectivity. Next, we should assure that no changes are lost and no duplicate changes are made. We can store each change in persistent storage and re-send it every synchronization session until we receive confirmation from other side that change is applied and confirmed. But this highlights another problem – a need to have an unique identifier for each change, SyncML supplies one, but it’s unique only in session scope and we need uniqueness in wider scope to support asynchronous messaging. In end of this article we present our way to avoid this problem, but it’s still an open question. And the third problem is that all those XML languages such as SyncML are quite large, which leads to increase in network traffic. The native solution is usage of WBXML – standard for presenting any XML documents in binary form. It’s already used to pack WML pages in WAP, but it seems that there aren’t any built-in APIs to work with WBXML. One possible solution is using already mentioned kXML library which has WBXML parser, but it adds additional kilobytes to your application, which can be critical in many cases.

 Binding to document formats. As we already mentioned, SyncML is document-agnostic. But you need some document-specific code to apply changes to document. Such code maybe implemented with SyncML engine (at least for some common document formats), but in mobile development every feature which is potentially useless in some cases should be avoided, since it increases size of application which is as we mentioned again and again is critical. So, developer should choose to either incorporate document-specific code for some formats into SyncML engine or define common set of interfaces, which every document should implement to be synchronizable.

                                                   

Our own solution – Tequila kSync client.

            Here at Reaxion Corp. (www.reaxion.com) we have faced the necessity of writing SyncML engine for our product. So we developed our own solution code-named Tequila kSync. The initial version of tequila kSync is already available as open-source at ksync.enhydra.org, though it lacks some important features, described above. Next version 2.0 solves many of mentioned problems and will be available as open-source soon. Below, we describe key elements of kSync 2.0 architecture and hope that this will be helpful for other developers implementing their own SyncML engines.

What is Tequila kSync Client 2.0

  • A J2ME library that implements SyncML HTTP Transport Adapter, SyncML Client Agent and Changelog Storage for mobile devices.
  • Doesn’t enforce any restrictions on data format.
  • Supports 3 SyncML scenarios (http://www.syncml.org/technology.html): 2-Way Sync, Refresh from Server and Slow Sync.
  • Is optimized on binary file size. Obfuscated binary file size is 17k without MinML XML parser (which adds another 8k).
  • Does not contain any Sync Engine functionality. All logic that could be moved on server is not present on client.
  • Facilitates conflicts resolution on a server side where all conflict resolution logic is implemented.
  • Guaranties data integrity in event of communication failure.

Sync Server

kSync Client based application is supposed to interact with corresponding kSync Server. The usual approach in Java is to implement such server as servlet-based Java Application. Enhydra application (www.enhydra.org) server could be used as servlet engine for it. Our kSync Server 1.0 (ksync.enhydra.org) supports valid XML documents. Server is implemented as a servlet-based Java Application and runs under Enhydra Application Server.

Architecture

Tequila kSync Client 2.0 could synchronize only one document in one synchronization session.

This is done in order to make the client as THIN as possible. However this limitation does not prohibit synchronizing another document in the next synchronization session. We don’t make any restrictions on number of documents that could be synchronized.

Tequila kSync Client 2.0 has 3 open interfaces that should be implemented in for all documents that wish to be synchronized:

            IDocumentSerializer – has only one method that serializes the document to String

            IDocumentParser – effrectively a corresponding deserializer interface and has only one method that retrives the original document from String

            IDocumentChanger – has 3 methods: add, delete, refresh for performing appropriate operations on the document.

When any changes are committed on the client, application should call appropriate methods in kSync Client,

Here is the code example of working the Application with kSync Client

// Create an instance of ISyncClient.
ISyncClient syncClient = new SyncClient(“http://sync.server.com”, “IMEI:001004FF1234567”, 
“John”);   
// Make the document “testDocumentId” available for being synchronized. 

// Set instances of IDocumentSerializer, IDocumentParser, 

// IDocumentChanger interfaces  
IDocumentChanger changerFromClient = syncClient.setDocument(“testDocumentId”, 
  clParser, clSerializer, changerFromServer); 
 
//Refresh the document from Server
changerFromClient.refreshFromServer(“testDocumentId”); 
//Add item to the doc
changerFromClient.add(“itemId1”, null, “itemValue2”); 
//Replace item’s value
changerFromClient.replace(“itemId1”, null, “itemValue2”); 

//Delete the item
changerFromClient.delete(“itemId1”, null, null); 
//Synchronize the document with server
syncClient.sync2Way(“testDocumentId”); 

Implementation Issues

XML Parser

Tequila kSync Client uses MinML XML parser (http://www.wilson.co.uk/xml/minml.htm) for parsing SyncML packets. If application that uses kSync need to parse XML it is recommended for the application also use MinML XML parser to minimize application size

Stability

Sending HTTP request to server doesn’t guarantee that message will reach the server. Same is true for the HTTP response. This is especially relevant for mobile devices, since network connections are highly unstable. We need to insure that every change reaches its final reciepient, Let’s look at few possible scenarios:

1. sending message once doesn’t not guarantee anything, since message may not reach the server and even if it did, we may not get a response.
2. Another possibility is to keep sending the change until server replies with change status response. In this scenario the change could be applied several time by Server, because server won’t have any knowledge if this change was received earlier.

To avoid both of these issues we identified all client and server changes with unique IDs. This ID is unique not only for current SyncML message, as SyncML specification demands, but it is unique within session. This design decision doesn’t conflict with SyncML specification, because unique change ID is automatically unique per SyncML message.

The change ID is sent to or received from Server in <CmdID> SyncML element. On each change received from server client sends back a status change and stores it in RMS until the synchronization session is finished and server sent back a change status (confiramation). This opearion is not limited to single request/response pair, therefore ensuring the data integrity in an event of communication failure. If the connection fails during sending the change request to the server and no response was received from server, client will send change request in the next synchronization session. If client receives the status change more than once it won’t apply it the next time, because change status was already stored in RMS