Big Data and Analytics

In-Memory Database Computing – A Smarter Way of Analyzing Data

In Memory Database Computing - Smarter way of data analysis

Querying data from physical disks was traditional approach, In-memory database computing (IMDBC) replaced this approach, as in IMDBC data is queried through the computer’s random access memory i.e. RAM. Upon querying data from RAM results in shorter query response time and allows analytics applications to support faster business decisions. It allows performing many scenarios in minutes as opposed to hours. [1]

The figure given below borrowed from the article referenced as [3] depicts the comparison between traditional computing approach and in-memory computing approach. As it can be clearly seen in the figure that in traditional approach data used to get stored in external device whereas in in-memory computing data is loaded into main memory.

In-Memory Database Computing - A way to get faster and smarter analysis

Figure 1, Traditional vs. In-Memory Computing Approach [3].

Importance of In-memory database computing (IMDBC)

In comparison to recent times only a few selected business intelligence users in an organisation used to have a weekly report. In contrast, today, every company is trying to implement fact-based decision making model across the organisation. However, every company knows that to implement the same, they have to buy analytical tools, which are not only expensive but also difficult to use as these tools requires extensive IT expertise, knowledge, and hands-on experience. Nowadays, business experts desire speedy access to information and easy analysis and in such case if tools aren’t that fast and easy, business experts will have no choice other than waiting for slow responsive answer to the desired query.

IMDBC not only provides win-win situation to the business users who can get self-service analysis capabilities, but also provides greater amount of agility as using in-memory tool team would spend quite less time on query analysis, model building, data preparation, data joining and performance-tuning tasks. It has also been claimed that in-memory technology can eradicate requirement of having data warehouse in place and so does cost and complexity associated with the same can be eliminated [2]. IMDBC also provides full picture of business scenarios as there is no limit to the granularity of the data view available to the user. This drill-down capability enables user to view data from different perspectives and take real-time business decisions which eventually can lead to higher business value and revenue growth of an organisation. The ‘SAP’ was having very poor end-user application support for its performance and pushing IMDBC solutions have helped SAP to quickly overcome the performance issues at very low cost and that too without re-factoring the design of the whole application as it becomes very difficult for the organisation to justify the poor performance at end-user level after, they have already spent millions of dollars on software and hardware implementations. [5]

To better understand the IMDBC, we will target business problem associated with the food retailer to learn more about how exactly IMDBC works. The BI business users of food retailer in the beginning used to stumble while trying to track and analyse data about the online traffic to discover the user behaviour and patterns. They certainly started using big data for analysis, but it was not working right. The higher authorities of BI team decided to implement QlikView (IMDBC tool). QlikView provided food retailer business user’s very good flexibility compared to the old tools for creating queries on the fly and joining information from dissimilar data sources to get answers for all business questions. Prior to IMDBC solution implementation, it used to be very tedious to build reports. However, after implementing it, the reporting became much faster and business users started spending more time in taking actions rather analysing it. [6]

Types of In-memory database computing (IMDBC) tools

IMDBC comes in different flavours and when we compare these tools we need not only to consider factors like speed of querying, reporting and analysis but also need to consider other factors such as flexibility, agility and rapid prototyping. For many other reasons, all in-memory analytics approaches are created distinctively. Below given are the options available for the users to buy in the market today;

Proprietary/ Commercial/Paid types of (IMDBC) tools

This section includes the list of proprietary/ commercial/paid types of IMDBC tools.

1.     In-memory OLAP

This in-memory classis MOLAP cube entirely gets loaded into in memory. The following named tools are available in the market for the buyers like IBM Cognos TM1, Actuate BIRT. The main advantages of using these tools are,

  • It provides faster reporting, querying and analysis as entire model and data gets loaded into memory
  • It allows users to write back
  • In-memory OLAP provides accessibility access via 3 rd party tools

On-contrary to the advantages this type of tools also has few disadvantages as mentioned below,

  • Traditional multidimensional data modelling is required to implement in-memory OLAP
  • The biggest drawback is the limited single physical memory space i.e. 3 TB , but yet only 300 GB has been achieved practically
2.     In-memory ROLAP

In this in-memory tool i.e. ROLAP, metadata gets loaded entirely in-memory. This product is only available with MicroStrategy merchant. The main advantages of using in-memory ROLAP are,

  • It speeds up reporting, querying and analysis as metadata fully gets loaded into memory
  • These tools doesn’t have limitation to the physical memory as compare to the earlier OLAP type

Apart from advantages this type too has few disadvantages such as,

  • The entire data model doesn’t get loaded into memory, only metadata does, even though tool provider i.e. MicroStrategy has ability to build complete cubes from the subset of data available in the in memory
  • Traditional multidimensional data modelling is required
3.     In-memory inverted index

In this kind of in-memory tool Index including data gets loaded into memory. Inverted Index in- memory tools are available with SAP BusinessObjects (BI Accelerator), and Endeca vendors. Following given are the advantages of inverted index products,

  • Faster reporting, querying and analysis can be achieved as entire index gets loaded into in- memory
  • These in-memory tools requires less effort to create data model as compared to OLAP based solutions

This category also has below given disadvantages,

  • Physical memory limitations
  • It required Index modelling at en extent
  • Reporting and analysis activities are restricted to the entity relationships built in index
4.     In-memory associative index

In this type, index with every attribute is correlated to every other attribute. In-memory associative index products are available with QlikView, TIBCO Spotfire, SAS JMP, and Advizor Solutions merchants. In-memory associative index products has following advantages,

  • It provides faster reporting, querying and analysis as entire index gets loaded into memory
  • Modelling effort required it less than compared to an OLAP based products
  • Without having any model constraint reporting, querying, analysis can be achieved. For example, any attribute can be instantly reused as fact or as a dimension.

Despite of having good advantages these kinds of products do have few disadvantages as given below,

  • Physical memory limitations
  • To the load the data modelling is still required
5.     In-memory Spreadsheet

In this category spreadsheets like arrays are loaded entirely into memory. This product is available in the market with only Microsoft (PowerPivot) company. This form of in-memory products have below given advantages,

  • As entire Spread-sheet gets loaded into in-memory faster reporting, querying and analysis can be easily achieved
  • Modelling is not required at all

This form has only one disadvantage as given below,

  • Physical memory limitations
Open Source/Freeware types of IMDBC tools

This section includes the list of Open Source/Freeware types of IMDBC tools.

1.     In-memory Distributed

The Pivotal has released GemFire distributed IMDBC tool, which basically can support large amount of data in the working memory of multiple servers of nodes. GemFire can also balance data across hundreds of nodes; possibly can manage terabytes of data. The Apache has developed Apache Ignite In-Memory Data Fabric integrated, high-performance , distributed in-memory platform to compute and transact on large-scale data sets in real-time.

The advantage of having GemFire and Apache Ignite In-Memory Data Fabric in place is that,

  • It can provide enterprise applications with low-latency access to datasets that are too large.
  • The major feature is fail-over capabilities, in case of single or multiple node failure system remains responsive.
2.     In-memory Hybrid

Altibase has developed HBD In-memory database with hybrid architecture.  The HBD offers high performance data processing in main memory with even having limited storage capabilities of physical disk.

Advantages of Hybrid In-memory

  • HBD has NVRAM (non-volatile memory), this has battery backup and this gets started in case of power failure and that can help in maintaining data in the event of power failure.
IMDBC Challenges

In-memory technology enhances application performance at great extent. However, there are many underlying challenges that need to be addressed prior implementing it. IMDBC has made database querying inexpensive and feasible, adoptions rate is increasing. However, due to fast read/slow write capability they are not getting adopted swiftly. [4]

The significant performance has definitely been offered by IMDBC over disk-based systems. However, disk transfer speeds and memory capacities are still the major concerns associated with in-memory solutions. Due to this reason, not every application can be run successfully in-memory and provide the same benefits.

The major challenge on IMDBC is having slow recovery times. As to recover an in-memory database, you have to refresh all that memory, and that has to be derived from the disk. However, this may soon change. [5]


IMDBC approach has been around for 30 years, but the reason it has been in headlines recent times is all because it wasn’t feasible to have 64- GB of computing ram as in the market maximum of amount of RAM available was 4-GB, which was not enough to run high-end analytical solutions or multi-user BI solutions. IMDBC has become more feasible for many businesses as the cost has been gradually declined over the period of years. Nowadays, it is possible to cache large volumes of data in a computer’s RAM (perhaps it could be an entire data warehouse or data mart — size of 512 PB (peta byte)) due to the newer 64-bit operating systems, with 1 terabyte (TB) addressable memory (potentially can be more in the future).  IMDBC approach provides way better performance, speeds up reporting, querying and analysis compared to traditional approach. IMDBC let business users take better decisions with speed and precision. If we keep these together, the improved access and response competences offered by in-memory technologies can help organisations to deliver the correct information to perfect business decision makers at the right time. [3]


[1] Cindi Howson, “Take Advantage of In-Memory”, April 2009, [Online]. Available: [Accessed: Aug.31, 2016].

[2] Margaret Rouse, “In-memory analytics”, Jan 2015, [Online]. Available: [Accessed: Oct.20, 2016].

[3] Anytersys, “System Solutions: High-performance in-memory DBMS”, [Online]. Available: [Accessed: Oct. 24, 2016].

[4] Rober L. Mitchell, Oct 2014,” 8 big trends in big data analytics”, Available: [Accessed on: Jan. 20, 2017].

[5] Sharon D’Souza, “In-memory database technology gains ground, but challenges remain.” Jan, 2012, Available: [Accessed on: Jan. 20, 2017].

[6] Beth Stackpole, “In-memory analytics tools pack potential big data punch”, May 2013, Available: [Accessed: Jan. 21, 2017]