Technology’s increasingly rapid evolution introduces fresh innovation in a short amount of time. Digital transformation is happening faster than ever and many businesses are obligated to adapt and keep up with the new trends in order to retain their old customers and attract new ones. Since digital transformation is a broad topic, this blog will be focused on integrating real-time applications and systems with IoT devices.
What is the Internet of Things (IoT)?
Internet of Things is a term that refers to expanding the internet connectivity to any non-standard physical computing devices that have embedded sensors and software which can collect, process and transmit the data over wireless networks. The concept of connecting smart devices through a network is not quite new. There are many implementations and network communication protocols for connecting, integrating and interacting with smart devices, but the rapid increase in the high demand and usage of IoT devices drove many industries to upgrade their systems architecture and networks in order to support fast and reliable communication between devices in the last year. According to Juniper Research, there will be 46 billion IoT devices by the end of 2021, which is a 200% increase from 2016. Telecommunication companies are making huge progress towards integrating and enabling communication among (IoT)smart devices, especially with 5G network’s new service categories URLLC (Ultra-Reliable Low Latency Communication) and eMB (Enhanced Mobile Broadband). The new 5G network architecture that includes small cells, millimetre waves, and gigabit speeds for uplink and downlink, enables the faster and more reliable establishment of connections and data transmission for real-time critical traffic.
Architecture and systems design for real-time IoT system
When designing real-time critical systems there are several things that need to be considered:
First, there isn’t only one standard architecture that can fit all types of IoT devices. Secondly, there are common features that are considered essential for any IoT system but the importance and criticality of each as it relates to the application’s and device’s requirements will determine the final and most suitable architecture for a given type of IoT system. The following features are essential to consider when designing the system: reconfigurability, scalability, low power consumption, distributed functionality and modular architecture design, interoperability, flexibility and tunable performance.
There are several architecture guidelines that are suggested for an IoT system:
- Three-layer architecture
- Five-layer architecture
- Cloud-based architecture
- Fog-based architecture
- Edge-based architecture
- Mist-based architecture
- Big data architecture
- QOS-based architecture
- Mobility first-based architecture
- Cloud things-based architecture
Three-Layer Architecture
The three-layer architecture consists of the device, network and application layers. Let’s take a deeper look into each:
Device layer; A layer that consists of physical IoT devices with integrated sensors and/or actuators, also called “edge devices”. IoT devices have to be connected directly or indirectly to the Internet. Arduino with Arduino Ethernet connection, Arduino Yun with WiFi connection and Raspberry Pi connected via Ethernet or WiFi are only a few examples of IoT devices with direct Internet communication. Each device needs to have: a unique identifier(UUID), Bluetooth identifier, Wi-Fi MAC address, OAuth2 Token.
Network layer; The smart devices transmit their data over gateways or routers. In this stage data is collected and distributed. In most cases when working with sensors or actuators, data should be converted from analogue to digital format. This data is raw and the volume of the data is at its maximum so additional filtering and compression are applied to optimize the size of the data for transmission. This happens in large IoT systems where a multitude of gateways serve high volumes of data and provide computing and peripheral functionality such as pre-processing, AI, filtering raw data, multiple protocol translation as well as implementing data encryption in the nodes for preventing malicious attacks.
Application layer; This layer is the cloud or data server where pre-processed and filtered data is stored and ready to be used for additional processing, analytics, big data processing, business logic, alerts, etc.
Application layer
This layer has the key components when creating server-side applications for processing data from IoT devices. When designing the application layer, commonly used components are: message brokers, consumers (can be a complex set of multiple components that handle the processing part), producers (IoT devices/data sources), and storage (non-relational or relational databases) depending on system requirements.
The flow:
Multiple data sources (IoT devices) publish data to the IoT Messaging Broker in a queue or topic depending on the use case. The consumer connects to the queue or topic and starts accepting its messages. Assuming that all of the data is correct, the consumer does data validation, error detection, error handling, processes and encrypts the data. In the processing part, aggregated data is not only being processed, compressed, filtered and validated but could also be analyzed and encrypted. After the processing part is completed, processed data can be stored on-premises or with any cloud provider and can be used for additional analysis for business purposes.
IoT Messaging Broker
Messaging brokers validate, route, store and deliver messages to specific consumers/destinations. Using a messaging broker in IoT or any other systems will tremendously ease the developer’s job. Its functions related to the standardized means of handling data flow in applications, services and systems communication, are accomplished by translating messages between formal messaging protocols. A great advantage of using messaging brokers when the system is intended to serve big volumes of data in a reliable way is asynchronous messaging and enabled acknowledgements. Messages from the queues or topics will be stored in the broker until the consumer sends an acknowledgement to the broker. Depending on the usage, there could be topics or queues, and point-to-point messaging vs. pub/sub messaging model. For example, when an IoT device publishes a message to a topic, that message can be received by multiple consumers while on the other hand, when a message is sent to a queue, it can be received by only one consumer.
Consumer Architecture
Consumer architecture can vary from a simple to complex set of multiple components that handle the processing part. The complexity of the system depends upon the type of the data that needs to be processed, the type of calculations/processing that must be implemented and if necessary, the addition of encryption mechanisms and the application of error detection/handling in generated and processed data. Key operations of the processing part are data acquisition/aggregation, validation, querying, processing, error detection and handling, optionally encryption.
- Data acquisition: In this step data is received from the message broker and, if necessary, could be stored for future calculations or processing when implementing ML or AI. Optionally, aggregated data can be compressed and filtered, allowing systems to work only with valuable information. This will enable efficient communication in times of increased data streaming rates and limited bandwidth within the internal system.
- Data validation: One of the key functionalities that data-processing components must have is the data validation process. When working with real-time data and different types of IoT devices it is not necessary that the generated data will always be accurate and reliable. Applying certain rules, logic, semantic and validation checks before processing and storing processed data is key to having a system with a 99% data validation rate.
- Querying: Aggregated data used for applying calculations and processing to the real-time data based on previously calculated data must be accessible and available with read-only permissions for calculations access and read and write multiple access for updating the calculations. In this case, it is best to use an in-memory data structure store for faster retrieval.
- Processing: The processing part is dependent upon what type of calculations or processing needs to be done on the aggregated data. At its core, the processing part is related to transforming generated data from the IoT devices into valuable information that has been collected and processed over a period of time and used by the companies/service providers for business purposes. Raw aggregated data that hasn’t been processed, filtered or subjected to any analysis, have no specific use or meaning for businesses.
Error detection and handling mechanisms
When designing a system that is fully automated with 0-1% dumped corrupted data rates, it is important to implement a component that has not only data validation mechanisms but also error detection and handling mechanisms for fixing the corrupted data. Having a component that handles the corrupted data will increase system efficiency in terms of data processing rates since corrupted data that has been dumped in the data validation step will be retrieved in the system to the processing component when fixed. Depending on the data and which information of the data is critical for the processing and validation part, error handling components can vary from very simple to complex implementations. Both solutions (complex and simple) for developing error handling mechanisms are valid until the valuable information used in the processing part can be detected, fixed and retrieved in the system to proceed with the processing. The most common error detection mechanisms are fixing data formats, types, structures, checking data sources, adding probabilities at different levels to reduce uncertainty when passing the validation.
Data visualization/Analytics
When working with IoT systems and devices, organizations are dealing with huge amounts of data that come from a vast array of devices that can be sensors, smart devices and integrated platforms. As mentioned in the processing part, raw, unprocessed and unfiltered data is essentially unusable for organizations. Data visualization tools are game-changers for presenting relevant information in a user-friendly and, most importantly, understandable format for the public. There are many applications for monitoring systems, aggregating system status and application logs, and presenting the aggregated data in dashboards, graphs, charts and tables. A common architecture for log aggregation is the UI component for data visualization, log aggregator and log agent that is deployed on each machine, and the same applies for monitoring architecture. Many data visualization tools have integrated notifications and alerts, which enables users or IT staff to be continuously updated and notified if a certain threshold is reached.
Final thoughts
Device technologies like wearables, mobile applications that use sensors, IoT devices, and sensor tags are becoming increasingly popular and used in every area. Maximizing the benefits of the Internet of things will bring social and economic improvements in many economies, new business opportunities for stakeholders, and new challenges for IT staff.
