We have described the general server design and threading model in our last article. Today we will go under the hood to explore the engineering details of how data is transferred between the server and the client. On the high level, instruction requests are harvested in sets and queued for processing by the worker threads. The worker threads resolves the XML instruction requests into a set of executable instructions on the server and processes them after correctly authenticating the source. The complexity of this system is based on a simple but advanced communication protocol designed for today’s needs.

Communication Protocol
The communication protocol between the server and the client is in XML. The client aggregates a list of instructions that they want executed on the server and then transmits that to the server. The sequence of the execution is determined by the ordering of the client’s XML request, which is done bottom up. The last instruction at the bottom of the XML request is the first to be executed on the server, followed by the last second and so on. The format of the protocol is not entirely XML based. The protocol divides all the data received or send into two segments. The first segment hold the XML instruction, the second segment hold raw attachments, if any. Attachment segment may hold additional file contents that are not XML based (eg. binary data). The first 8 bytes or 64 bits dictate the total data block size. The second 4 bytes or 32 bits indicates the first segment size, the remaining data belong to the second segment.

This protocol design although simple will allow us to provide a basic mechanism to transfer vast amount of differing types of data depending on the XML instruction request defined by the first segment. You can view the first segment as the instruction manual to process the request or sets of requests and the second segment as any other additional data required to fulfill the request or sets of requests. It may seem that this protocol is nothing special on first inspection but we think otherwise. Many of the protocols used today (eg. web server) are done using fixed Id values and headers which are less flexible and robust. The reason is that these older protocols mainly focuses on the format of transferring data and stopped there. Data that we communicate to each other today demands more than just being transferred but also properly processed to have any real value.

A simple example will be a file transfer using this more advanced high level protocol. The first XML segment can define a file transfer indicating how big the file size is in the attachment, the file name, destination location and etc. The protocol can also handle multiple file transfers in a single request. This will then allow the server to save the file to some predefined location. The XML segment might also define a decompression instruction that will result in the transferred file being uncompressed at the end of the entire exchange. The XML segment might also further define what is to be done with the uncompressed data and etc. This results in a more flexible and robust system that can be self descriptive and automated at a high level.

If you use our work, we ask that you remember to give us the credit that we deserve. Our hope is to develop better technology through open sharing so that future generations can benefit. Research and development for newer and better technology is still in great need as our generation moves towards the digital age, so join us in this fight towards the cutting edge of information technology.