Mesh4x SMS Adapter: Sync data without an Internet connection

posted on: June 24, 2008

Mesh4x has a new feature that allows you sync data between a local desktop, server or mobile device and a remote computer even if you have no Internet access, by sending and receiving little batches of text messages. Databases, spreadsheets and even maps can be kept up to date using the right adapters. Algorithmic work was done to minimize the number of text messages needed, and the result is having up-to-date information on both ends of the exchange. This data can be in turn shared further with other devices locally and synchronized again to the remote source.

Scenarios

OpenMRS (http://openmrs.org) is an open-source Medical Record Management system used extensively in africa (Tanzania, Rwanda, Malawi, Zimbabwe, Kenya…) and increasingly in the Middle East and Americas (Peru, Honduras and Haiti come to mind). OpenMRS is used to improve patient care and simplify the records management at the clinic where it’s used. It is common for these clinics to have just one computer and have no internet connection. Cell phone coverage can be present, ranging from reliable to dodgy for voice (just 1 bar of signal is typically reliable enough for SMS, but terrible for voice or data).

A rural clinic in Rwanda, photo credit Neal Lesh of the OpenMRS community

There are two sync scenarios I heard about this week talking with the OpenMRS and OpenROSA teams that Mesh4x addresses. (Note – we haven’t planned to do this work yet I’m just using these scenarios as concrete examples of how mesh4x over SMS can help in the context of medical record management)

Scenario 1: OpenMRS to OpenMRS sync

The clinic is updating patient records that need to be kept up to date with the province-level hospital. In this case the clinic has a computer under a desk with a cell phone reliably plugged into it, and periodically, it would sync with a similar setup in the province level. It could also go straight up to central and then down again to the province level, as province hospitals do tend to have connectivity.

Mr. Vanra Ieng shows a nift enclosure that makes sure the phone plugged in to the computer will be reliably working!Here you can see Vanra Ieng from the WHO/Ministry of Health in Cambodia showing a physical enclosure that makes sure his phones – used in a similar setup, as an attachment to a computer- don’t get unplugged from the PC or power, and are used for ‘intended purposes’ only (people have personal phones and other means of communication as well, and he needs to make sure it keeps running as this is for a pilot on sending disease indicators from key districts to central level).

Scenario 2: OpenMRS to mobile data gathering client

javaROSA is an open source mobile client built in Java that is used for XForms-based data collection that works on lowest-common-denominator phones as well as PDAs. You can fill in the forms and send the data via Infrared, bluetooth, http (If there is GPRS available)

If I understood the conversations at the HISA meetings,they are working on a feature to send data one-way via SMS messages (serializing objects and sending them over a set of messages). With the SMS adapter, community health workers could be taking data on their mobile devices and updating centralized computers, as well as getting the latest information on the device nd updating their local information by querying for the information of patients they hadn’t seen before but are facing now, or patients that have visited the clinic since the information was taken. In addition, they could even beam (SMS) information with a colleague directly, phone to phone.

In each scenario, though, how many text messages are we talking about? In our tests, starting with a large up-to-date dataset (a KML map) and added a "pushpin" with a relatively long description. It required a grand total of 8 text messages. This includes all the steps needed to compare versions on both sides of the communication, and send the new pushpin over (see Under the Hood for more details).

If there are more items that have changed, and the larger the items themselves, the more messages are required to transmit them, of course. But we think this is a very low baseline considering the outcome: up-do date information on both sides that can, in turn, be shared with more devices locally using even more economical means such as infrared or bluetooth.

Under the hood

So what does it take to synchronize data over text messages? 

1) We need to be able to send/receive SMS messages from a phone via a USB cable. In the code we abstract this behind a provider model, and the default implementation will be based on SMSLib. We envision in a future a server version, potentially using BT’s web21c infrastructure to do so.

2) The mesh protocol must be reduced to a bare minimum so it is efficient to use over tiny and unreliable text messages. We do so by combining exchanges that achieve the following:

  1. A collection-level check: is any sync needed?
  2. Item-level checks: which items have been added, updated, or deleted relative to the version information available locally?
  3. Item exchange – 2-way sending and receiving the changed items themselves. Originally we were zipping the data and sending that over if appropriate, now we are using a variation of the RSync algorithms which use creative hashing (math operations on the data) to send the minimal information over.

3) SMS is an unreliable transport and as such there is a layer in the code that compensates for this by managing message batches. A batch allows us to split up a large payload into text messages to reconstitute on the other side, tolerating messages coming out of order, dealing with lost messages, and timing out on operations that have taken too long to complete.

It is important to understand that the goal of this adapter is not only "sending" the data for a new item or "receiving" it – This adapter checks for which items to send/receive and also sends/receives the full versioning information for the item. That makes it possible to keep sharing it with other applications and users while maintaining the ability to reconcile updates and detect version conflicts.

A big kudos to Tondat who has been moving at warp speed with this codebase. The first checkin was on June 9th! The quality of the code is very high, and the ingenious use of gzip, Base91 and Rsync shows . Check out the source.

Next steps

  • Finish the optimization of the FeedSync protocol (which Mesh4x uses under the covers) in edge conditions (e.g. sharing conflicting payloads).
  • Implement the SMSLib adapter and test it well with a couple of appropriate phones.
  • Add this capability into the demo Java application that is used to demonstrate the KML adapter. This will let you specify a phone number in addition to the http URL and the file path in the sync endpoint box.
  • Further optimize formats, encoding and memory usage.
  • Pursue collaborations with openROSA/openMRS that resonate strongly with the community needs we are see in South East Asia. If you think of any scenarios where this could help your technology please share them here!

References

http://mesh4x.org
http://en.wikipedia.org/wiki/Rsync

Comments are closed.

View more of InSTEDD's blog posts.