Overview of Big Data

The promise of Big Data is to drive profitability by reducing costs, improving product/service quality, decreasing time of delivery, optimizing product offerings and making smart business decisions in everything from medical treatments to social media campaigns, to customer service and marketing promotions. Big Data is a term used to describe data sets which break the capabilities of traditional IT as they demand massive storage requirements, huge transaction counts, big analytical needs, quick response times, and enormous bandwidth requirements. According to Gartner and IBM, Big Data is characterized by 4 Vs: volume (size of the data set), velocity (speed of data collection and analysis), variety (data formats), and veracity (quality of data).

The data sets can be structured data such as transaction records, customer records, call center records or semi-structured data such as emails and XML files or even unstructured data such as video, twitter feeds, customer comments, etc. And these data sets can arrive from various data sources such as sensors, data entry terminals, social media, and web interactions at varying speeds (real time, periodic, batch) and can quickly grow to terabyte (1000GB) and petabyte (1000TB) ranges. The challenges for organizations are in acquiring, storing and accessing large data sets effectively; making sure the quality of data set is high and analyzing the data accurately to realize the promise of Big Data as depicted below.

Big Data competencies

Online companies such as Facebook, Google, LinkedIn, Twitter, Amazon, and eBay have used Big Data collection and analysis to predict their users’ behaviors/needs and provided unprecedented value to their customers and consequently to their shareholders. These companies born in the digital era have been at the forefront of collecting vast quantities of digital data through web and social media interactions, and have evolved solutions, as well as many of the industry’s leading-edge technologies, to address the consequent challenges and transform them into opportunities. These companies are hence very adept in using the Big Data technologies to analyze data to drive product/service decisions. Unlike these and other online companies, the traditional enterprises in retail, healthcare, pharma, automotive, financial services, etc. while understanding the potential benefits of Big Data, are saddled with several challenges:

  • Large data sets exist in separate business units within traditional enterprises and they are not integrated due to organizational silos as well as technological challenges
  • Besides digital data such as web and social media interactions, the traditional enterprises also have conventional data derived from product transactions and interaction channels such as call centers and point-of-sale systems. Analytics has to be performed across all data in an integrated manner in order to be truly useful
  • Being set in the relational database and data warehouse world, it is not easy for traditional enterprises to move to large data technologies such as Hadoop and NoSQL
  • Again being set in the structured, schema driven DB world ,traditional enterprises are not ready for the unstructured, no-schema driven data analysis of customer comments, social media feeds and video analytics
  • Big Data implementations are complex and many components of Big Data are open source, so compatibility issues often arise when integrating them into a broader legacy infrastructure

Xoriant is in a unique position to bring its Big Data expertise of not just technology knowledge, but real-world operational experience with Big Data, to traditional enterprises. Having worked with large enterprises with established enterprise technologies (relational DB, desktop statistics and visualization packages) for the past twenty years, Xoriant understands the technological needs and organizational challenges faced by these companies with respect to Big Data and helps bridge the gap with its Big Data capabilities.

Xoriant Big Data Practice

Xoriant has strong skill sets in various Big Data technologies, platforms, and tools which are a mix of open-source and commercial offerings. For instance, Xoriant has delivered solutions using Hadoop (Cloudera, Hadapt, Hortonworks, HBase, Hive) and NoSQL DBs (MarkLogic, MongoDB, Cassandra, Solr) as well as Google BigQuery (based on the Google's Dremel architecture). In addition, Xoriant brings to market purpose-built Big Data IP such as Mezzure, which leverages techniques such as graph modeling of business data structure and relationships to implement industry-unique search capabilities on existing business data, thus leap frogging business customers to be on par with the modern online firms.

Recognizing that log data has till date been one of the most practical and leverageable examples of Big Data, Xoriant has built up expertise in technologies such as Splunk to address this booming need.

Leveraging this comprehensive expertise in Big Data technologies and client specific offerings, Xoriant can provide the following tangible solutions:

  • Assess and integrate structured and unstructured data
  • Capture, store and analyze all data, all the time as well as just in time
  • Build confidence in Big Data with the ability to integrate, understand, manage and govern data appropriately across its lifecycle
  • Using the Google Cloud Platform along with Google BigQuery, Xoriant has provided solution offerings ranging from data cleaning/structuring, storage (with GCP) to complex solutions involving analytical modeling for predictive and sentimental analysis
  • Bring the power of Hadoop to the enterprise with application accelerators, analytics, visualization, development tools, performance and security features
  • Provide domain specific analytical solutions, predictive models, and sentiment analysis
  • Provide industry-leading database performance across multiple workloads while lowering administration, storage, development and server costs

Furthermore, Xoriant takes a holistic approach to the Big Data problem and provides expertise to companies that are at various stages of Big Data implementation:

  • Early stage: At this stage enterprises have heard about the benefits of Big Data but don’t know how to proceed. Xoriant can assist by showing how it could be done, recommending technologies to leverage, and take the customer all the way through to a solution
  • Mid-stage: At this stage companies clearly know a business problem which can be solved by Big Data but are looking for help in identifying tools / technologies / mechanisms to solve this. Xoriant can help by evaluating the need and specifying the solution, including end-to-end solutions such as Mezzure that we can recommend
  • Implementation stage: At this stage Xoriant helps our client implement their specific Big Data requirements with right set of tools and technologies. Xoriant employs flexible models that range from entire project ownership to augmentation of teams both onsite and offshore with expert personnel
  • Support stage: Here Xoriant provides maintenance and support for existing Big Data solutions

Xoriant Big Data Client Engagements

  • Healthcare: Created a robust analytics solution using Google BigQuery for a Healthcare client, using which the individual enrollment and billing data running into few Terabytes are analyzed in a matter of seconds to provide an enhanced customer experience along with an informative reporting dashboard for the stakeholders
  • Hospitality: Management of customized pin tables for guests, involving both personalization of data and processes using MongoDB and Grails for a very large resort/amusement park group
  • Consumer Package Goods (CPG): Using Cassandra, developed a patent-pending IP to provide a unique intelligence-gathering mechanism provided in SaaS mode. Used large enterprise data assets from both structured and unstructured information stores, with extensive executive dashboards to tailor criteria in modern dynamic environments
  • Social Media: Developed Hive/Hadoop warehouse and interfaces with other systems/tools. Over 12,000 servers, total DB size in petabytes, with over 100TB added every day

Xoriant Big Data Services

Xoriant,combining its know-how in Big Data and traditional enterprise applications can provide the following services. Services are delivered with Xoriant’s flexible delivery methodology to suit the needs of the project and the customer.

  1. Consulting Services:
  • Strategy and roadmap definition
  • Solution evaluation and recommendation
  • Architecture evaluation
  • Visualization and analytics consulting
  • Best practices
  1. Implementation Services:
  • Design
  • Open source software customization
  • Protection to ensure availability, security and compliance of Big Data systems
  • Analytic tools implementation
  • Solution accelerators
  • Testing
  • Production rollout
  1. Sustenance Services:
  • Production Support
  • Level 1, 2, and 3 support