IoT Devices Testing: Challenges and Recommendations

Jean Baptiste MINANI -

Introduction

The Internet of Things (IoT) has rapidly transformed the way we live and work, connecting diverse devices and systems in different domains such as health, transportation, agriculture, manufacturing, smart cities, and smart homes. Any IoT system will consist of at least the following four layers, as shown in Figure 1.

Figure 1: IoT System Layers

Device Layer
This includes the sensors, actuators, and other smart devices and connected devices that constitute the physical layer and device layer. These smart devices capture data (sensors) from their surroundings, take action (actuators), or, sometimes, both.

Network Layer
The second layer is a network layer that transfers the data from the device layer to the cloud layer via cellular connectivity and vice versa. This layer may do some processing and decision-making, including authentication and encryption.

  1. The most popular protocols used are:
  2. HTTP/HTTPS (and RESTful approaches on those procols)
  3. Constrained application protocol (CoAP)
  4. MQTT 3.1/3.1.
  5. WebSocket

Cloud Layer
The cloud layer stores, manages, and processes data from multiple devices/sensors with different data formats, and protocols, and provides a unified view of the data to the application layer.

Application Layer
The application layer uses the data to deliver solutions like analytics and reporting to end users. In other words, the application layer provides the business value from the IoT infrastructure.

The widespread adoption of IoT comes with its own set of challenges [3], particularly in the testing phase. In this blog, we will explore the key challenges faced during IoT device testing and provide some recommendations to both practitioners and device manufacturers.

Key Challenges for Testing IoT Devices and Recommendations

Lack of standards
Many IoT devices do not adhere to established standards or specifications set by regulatory bodies, industry organizations, or technical standards bodies. As a result, non-standard compliant IoT devices may exhibit inconsistencies in performance, compatibility (interoperability) issues with other devices, security vulnerabilities, lack of support and documentation, etc. Since there are no common standards for IoT devices, testing IoT devices involves an extensive analysis process for each device, making the testing process more complex and extensive. One of the recommendations to overcome this challenge is to promote collaboration among IoT device manufacturers to establish common standards for IoT devices.

Resource constraints

Many IoT devices have limited resources such as processing power, memory, energy, and bandwidth.

  1. Processing Power: IoT devices often have limited processing capabilities compared to traditional computing devices.
  2. Memory: IoT devices typically have limited memory capacity for storing program code, data, and configuration settings. This limitation may require developers to optimize code and use efficient data structures to minimize memory usage.
  3. Energy: Many IoT devices are battery-powered or operate in remote locations where access to power sources is limited. As a result, energy efficiency is critical, and devices must be designed to minimize power consumption to maximize battery life or operate using alternative energy sources such as solar energy.
    Bandwidth: IoT devices often rely on wireless communication technologies such as Wi-Fi, Bluetooth, or cellular networks to transmit data to other devices or the cloud. These devices may have limited bandwidth or intermittent connectivity, requiring developers to implement data compression techniques, prioritize data transmission, or use protocols optimized for low-bandwidth environments.
  4. Storage: IoT devices may have limited onboard storage capacity for storing data logs, firmware updates, or configuration files. Developers must carefully manage storage space and implement strategies such as data compression, offloading data to the cloud, or using external storage solutions to accommodate storage requirements.
  5. Security: Constrained resources can also impact the implementation of security features on IoT devices.

Developers should optimize testing processes for minimal resource usage by adopting mixed environments, as illustrated in Figure 2.

Figure 2: Testing Using Real IoT Device Should be Optimized [1]

Device Connectivity

Device connectivity is the ability IoT devices to establish and maintain reliable communication with other devices within an IoT ecosystem. Verifying that IoT devices can effectively exchange data, commands, and status updates with other components of the system, including sensors, actuators, gateways, cloud platforms, and mobile applications is a challenging task.

Different Data Format

IoT data has 3 characteristics also known as the 3 V’s (volume, velocity, and variety) [2]. Data collected by IoT devices comes in a different format, and testing the validity and integrity of this data can be challenging. To address this challenge, one of the recommendations could be to consider adding a data abstraction layer to manage and process data from multiple devices/sensors with different data formats, and protocols, and provide a unified view of the data to the application layer.

Diverse Ecosystems and Standards

One of the primary challenges in IoT device testing is the vast diversity in ecosystems and standards. IoT devices come from various manufacturers, each employing different communication protocols and standards. This diversity complicates the testing process for the developers, as it may require vast knowledge of many technologies, making it difficult for them to test those devices. The implementation of APIs offers a standardized interface, facilitating seamless communication and interaction among heterogeneous devices within the Internet of Things (IoT) ecosystem. These APIs define well-defined protocols and data formats, thereby establishing a uniform framework for inter-device communication.

Testing in real-world scenarios

Many IoT devices undergo testing in simulated environments, which may not accurately replicate the behavior of IoT systems in their intended operational environments. Some companies have already begun offering services such as Device as a Service (DaaS) or Environment as a Service (EaaS), also known as Testbed as a Service. This option could be considered by developers who wish to test IoT systems in real environments (using actual devices) rather than purchasing the devices themselves.

Conclusion

Testing IoT devices is a complex yet essential process to ensure that the device is ready to be deployed in real life. With many devices deployed, it is more complex to determine which device is causing the problem or if there are issues related to the system, connectivity. Embracing automation, collaboration, and adoption of device as a service (DaaS) or testing environment as a service (EaaS) will be key for effective testing of IoT devices.

References:

[1]. Bosmans, Stig, et al. "Testing IoT systems using a hybrid simulation based testing approach." Computing 101 (2019): 857-872.

[2]. Leotta, Maurizio, et al. "Towards an acceptance testing approach for Internet of Things systems." Current Trends in Web Engineering: ICWE 2017 International Workshops, Liquid Multi-Device Software and EnWoT, practi-O-web, NLPIT, SoWeMine, Rome, Italy, June 5-8, 2017, Revised Selected Papers 17. Springer International Publishing, 2018.

[3]. Bures, M., Cerny, T., Ahmed, B.S. (2019). Internet of Things: Current Challenges in the Quality Assurance and Testing Methods. In: Kim, K., Baek, N. (eds) Information Science and Applications 2018. ICISA 2018. Lecture Notes in Electrical Engineering, vol 514. Springer, Singapore.