Realizing the IoT Data Edge Platform Potential - Part 2

In the first part of this blog series, we gave a complete overview of IoT Data Edge Platform – architecture, capabilities, and implementation. In this blog, we will look at how IoT Data Edge Platform is deployed on a 3-Node VMware vSphere vSAN Cluster with Cloud Native Storage Policy configuration.

The Cloud Native Storage CSI (Container Storage Interface Plugin for Kubernetes) provides high performance, highly resilient, persistent data storage solution for various microservices pods of the IoT Data Edge Platform.

The IoT Data Edge Platform is an IoT software stack that consists of two main parts:

1. Appliance Software: Installed as a Cluster on powerful servers and deployed in a secure remote data center.

2. Edge Software: Also known as the Gateway, Edge software is installed on the standalone machine. It is deployed closer to the source sensors. One or more gateway collects data from the sensors and sends it to the appliance for further processing.

The primary objective of this blog is to give you a methodological approach of right sizing the IoT Data Edge Platform servers and Pod configurations to achieve maximum data transfer rates safely. A right sized server is important to achieve maximum ROI at an optimal system performance.

By executing performance tests in a methodological way, we can ascertain the perfect server size to deploy the IoT Data Edge Platform and find the answer to the following question:

By executing performance tests in a methodological way, we can ascertain the perfect server size to deploy the IoT Data Edge Platform and find the answer to the following question:

Setting up the Infrastructure for IoT Data Edge Platform

Before we learn the methodological approach for right sizing the IoT Data Edge Platform Servers, we should briefly understand the setup and deployment configurations.

IoT Data Edge Platform Appliances

The IoT Data Edge Platform Appliance software is deployed in a secure data centre, as microservices application, on a 3 Node Kubernetes cluster as VMware Linux Virtual Machines.

These Virtual Machines are deployed on a high-performance VMware vSAN Cluster with Cloud Native Storage policy configured. The Cloud Native Storage provides highly resilient, persistent storage to the various service pods of the appliances.

IoT Data Edge Platform Gateway

The IoT Data Edge Platform Gateway software is deployed closer to IoT sensor location (e.g. factory outlet) as Linux Virtual Machines or Physical Machines.

The IoT sensors register into the gateway and one or more gateway registers into the appliance.

Apache Ni-Fi (Sensor Simulator)

Apache NiFi is an opensource software for automating and managing the data flow between systems. As compared to other commercially available tools, Apache NiFi is recommended, as It helps us simulate the IoT sensor by sending data to the appliance in desired data format (JSON & Influx data format) and desired IoT protocols like AMQP and MQTT. We can also easily control the data transmission rate using Apache Ni-Fi.

Methodological Approach to find Right Size IoT Data Edge Platform Configurations

The high-level approach to right sizing the server is:

Step-1: Find the Critical Breakpoint Configuration

In this step, we try to ascertain the minimum possible configuration at which the IoT Data Edge Platform would work. A configuration below this point means the IoT Data Edge Platform would fail to start its services.

The way to find this by looking at the minimum resources (CPU & Memory) requirement of each pod and then summing up the values or you could do trial and error by reducing the Virtual Machines configurations. As an example, the minimum configuration to successfully start all the service Pods of the IoT Data Edge Platform is 12 CPU and 32 GB RAM.

Step-2: Do Baseline Performance Testing to Find Out Right Size Server Configuration

In this step, we select an arbitrarily huge virtual machine configuration (e.g., 48 CPU and 256 GB RAM) and using Apache Ni-Fi, start sending data to the IoT Data Edge Platform at maximum data rate. Now, observe the total CPU and Memory consumption of the Virtual Machines that hosts the IoT Data Edge Platform Appliance and IoT Data Edge Platform Gateways.

If the default CPU and Memory consumption of the Pods reaches 80% or is causing the Pods to crash, then we increase the default CPU & Memory limit configured on the POD.

Furthermore, if we observe message loss or message queuing up rapidly and there are still enough CPU & Memory resources available, then we reduce the data transmission speed from Apache Ni-Fi.

By increasing the CPU & Memory of the Pods or by reducing the data transmission speed Iteratively, you would reach a point where data could be safely handled by the IoT Edge Data Platform.

Once, you achieve a stable transmission speed, make a note of the total CPU and Memory consumption of the Virtual Machines. Thus, we find out the Right Size Server configuration after few iterations. The most important point to be noted here is that there should be no loss of transmitted messages, or the messages should not be queued up in the message broker.

Step-3: Doing Performance testing on Right Size Server Configuration

Once we find out the right size server configuration of the IoT Data Edge Platform, we scale down the Virtual Machine Node configuration to a value we found during the performance testing, that we did in step 2. We again do Performance benchmarking of the IoT Data Edge Platform just to make sure that the Scaled down configuration works as desired.

The Right Size IoT Data Edge Platform Benefit

With the help of a right sizing methodology explained above, businesses can decide the perfect server configuration that can handle the IoT message load safely and optimally. Thus, enabling your business to achieve maximum Return on Investments (ROI) at an optimal system performance.

Want to know how Xoriant can help you with IoT and Edge?

Talk to Xoriant Experts