Table of Content
- Client Overview
- Engagement Situation & Challenges
- Key Engagement Objectives
- Xoriant contributions
- Tools & Technologies
- Xoriant Major Contributions
- Client Benefits
Our client is a leading US based ISV delivering data acceleration and management solutions that enable companies and individuals to transfer large amounts of data at high speed over the internet cloud. Our client's portfolio of software solutions help organizations and individuals transfer data fast and manage data across a multitude of applications while maximizing performance, maintaining data security, privacy and minimizing capital and operating costs.
ENGAGEMENT SITUATION & CHALLENGES
Our client wanted to develop data acceleration and management solution that would enable businesses and individuals to transfer large amounts of data at high speeds, while optimizing the flow of information over the Internet and maximizing network performance.
Data transfer over the network is driven by two parameters speed and reliability. Two protocols considered for data transfer UDP and TCP has their own pros and cons whereby UDP ensures high speed of transfer but compromises on reliability while TCP ensures reliability and compromises on speed of transfer.
Another issue was there was no pre-defined data size and it could range anywhere between a couple of Gigabytes to Terabytes. Transferring bulk data over the network resulted in increased time and cost to the end user and client wanted to optimize these parameters to benefit the end customer.
In addition, a prime requirement was data security, since the internet is prone to lot of malicious content which could lead to several adverse effects to the enterprise.
The client was looking for an agile innovative partner with strong product and technical background who will understand the complexity of the transfer solution and provide competitive offerings for the "High Speed" data transfer solution. The client wanted to partner with a product engineering company who would adapt to the evolving requirements of the solution and yet keep low turnaround time.
KEY ENGAGEMENT OBJECTIVES
Ensuring the solution is available for different flavours of Windows and Linux operating systems.
Support individual users and also be able to be used in an organizational setup supporting multiple systems and multiple applications
In case of enterprise solution, data security was considered as a prime factor
Cost effective mechanism for data transfer without effecting the quality/speed of the transfer
Testing the application for various systems which is unique for different users/ organizations
Xoriant Core development team comprising of a technical architect and technical had a serious of discussions / meetings with the client to gather finer details of this engagement. A team was then defined and it was then defined and it was decided to adopt Agile methodology of project execution allowing quick client feedback / review with feature based deliveries.
Two different components were architected, one to act as a sender and another as a receiving component for data transfer. The receiving component would act as a factory and create a server to listen to each of the client requests.
The solution was designed considering two different scenarios, in one wherein data transfer speed is of most importance over reliability UDP protocol was used for data transfer. Along with the UDP a periodic connection is established which checks for the acknowledgments from the receiving end. Since, the acknowledgement is periodic the time is reduced and the packet is also received with maximum throughput.
For the other scenario when reliability of transfer is more important over speed of transfer TCP connection is used as sending data over UDP resulted in significant data loss. However sending data over TCP reduced the throughput for which a re-entrant TCP multiplexer module was implemented.
In essence, when the data is to be transferred the sender component establishes multiple connections specified in the configuration with the receiver. These multiple connections on the same port create a roadway to send-receive the data thereby accelerating the transfer. The receiver then accumulates the received packets and arranges them as per the packet header information.
An in-memory compressor feature is built in the sender component where zlib default compress is used to compress/decompress data at the sender and receiver end. This enables transferring huge amount of data in a significantly shorter time.
Solution was designed to also ensure data security as another key feature, AES-128 and AES-256 algorithms are used for providing 128 / 256 bit encryptions. mcrypt library is implemented for data encryption based on key given by user on the configuration.
Solution not only can be used for an individual user but can also be used on an enterprise level where an organization can transfer data using a single receiver component connected to the multiple sender components installed on standalone machines. This solution will not only ensure the accelerated speed of data transfer but also the encryption and compression of data.
A wizard type installer was created for the installation of the sender and receiver components using NSIS scripts. This would then be installed as a Windows service or a Linux demon on the Windows and Linux machines respectively. A rich UI was also provided to set configuration parameters like listening / sending port, encryption algorithm, to be compressed, etc. for each component. Another functionality that can be configured was to enable data transfer across the firewall after due SOCKS5 authentication.
A QA team was formed to define a comprehensive testing process for this engagement. Certification matrix was created with OS and different flavors,solution was tested against them. Tools like Filezilla was used for manual file upload and the solution was compared for the speed and reliability of data transfer while JMeter was used for load testing.
Automation scripts were written to replicate real time scenario and send random size of data to be transferred.
Wireshark was used to trace the path of the data transfer and identify if the right protocol was applied.
TOOLS AND TECHNOLOGIES
C #, Qt, NSIS scripts, Batch scripts, shell scripts
VS 2008 IDE, Qt Creator
Bugzilla, Tortoise CVS
XORIANT MAJOR CONTRIBUTIONS
Two separate code bases were created to run as Windows service or a Linux demon on the Windows and Linux machines respectively
Data reliability is ensured by sending an acknowledge module which does the sampling of the packets sent and ensures that data is sent to the receiver end for transfer using UDP
Data security is ensured by using AES-128 and AES -256 encryption algorithms
Solution is also used as an enterprise level where a single receiver component can be connected to the multiple sender components installed on standalone and the inbuilt SOCKS5 client can deliver data across the firewall after due proxy authentications.
In-memory compressor feature is built in for compression/decompression of data thereby resulting in faster transfer
Intensive testing both manual and automated was performed for different flavors of Linux and Windows OS
Provided the client ‘first mover’ advantage over its competition as there was no product exclusively for accelerated data transfer available in the market
Product could meet the targeted the enterprise customers by providing product reliability and the security feature which became the key drivers of customer growth
Achieved time to market goals and cost advantage due to offshore model of project execution
Enabled client engineering team to focus on core competency areas as Xoriant team was involved in end-to-end solution development
Product also could target huge transfer requirements for clients by improvement in speeds of data transfer to the tune of 100 Mb in 17 seconds versus the earlier 100 Mb in 6*60 seconds.