IEEE 2668 Internet of Things Maturity Index (IDex)
By City University of Hong Kong and the Electrical and Mechanical Services Department
Smart city construction is a global trend in urban development. As one of the most competitive and international cities in the world, Hong Kong has dedicated itself to developing and promoting innovations to facilitate smart city development. In December 2017, the HKSAR Government published the Smart City Blueprint for Hong Kong, aiming to achieve “6S” for Hong Kong (namely, “Smart Mobility”, “Smart Living”, “Smart Environment”, “Smart People”, “Smart Government” and “Smart Economy”)1. Based on the achievements of the first Blueprint, the Smart City Blueprint for Hong Kong 2.0 was released by the government in December 2020 as the second edition. By the time this article was written, over 100 smart city initiatives had been completed and had succeeded in addressing daily life issues through the most advanced Internet of Things (IoT) technologies1.
The popularisation of IoT has tremendously proliferated in various fields of smart city applications: smart transportation, smart agriculture, smart building, smart grid—to name just a few. Forecasts pinpoint 80 billion IoT objects by 2025, which translates into billions of IoT connections. These IoT objects refer to various connected things, including devices, networks, algorithms, management processes, systems, applications, services, infrastructures. The IoT objects may have different specifications, applications and requirements, making the evaluation of IoT objects difficult. Unstandardised IoT design and evaluation may result in interoperability issues and underlying cyber threats. IoT objects are connected through various emerging IoT wireless protocols, namely, Narrowband IoT (NB-IoT), Long Range (LoRa), Sigfox, and 5G protocols, etc. The massive unregulated IoT connectivity potentially incurs interference, thereby undermining the performance of IoT applications. The IoT development is currently encountering challenges in a number of aspects2: (1) the design and implementation of IoT infrastructure; (2) the interoperability between different IoT solutions and/or infrastructure; (3) the objective and standardised evaluation method for IoT objects.
The Internet of Things Maturity Index (IDex) is first proposed by IEEE 2668 Standard Working Group to address the challenges already mentioned. IEEE 2668 standard defines the mechanism and criteria for the evaluation of IoT objects pertinent to their applications and performance2. The evaluation result is quantified as an indicator value, namely IDex. IDex grades and ranks the maturity of the IoT objects on a five-level scale, ranging from one (the lowest maturity) to five (the highest maturity), meeting the IoT stakeholders’ demand for explicit indication and comparison of IoT objects2. In addition, IDex can be applied to the prediction of the performance changes under different operating environments, and it aligns with the guidelines for implementing and improving the performance of IoT objects. The compliance with the IEEE 2668 standard will increase the efficiency of deploying IoT objects and facilitate the future integration of various IoT objects into the growing IoT environment, occasioning a harmonised IoT ecosystem. Given the characteristics of unified evaluation, IDex is also generally regarded as “Michelin of IoT” in the industrial field.
Figure 1: IDex is recognised as “Michelin of IoT”
IEEE 2668 IDex development journey
The IEEE 2668 IDex was initiated in 2018 by Dr Tsang Kim Fung*, who was Associate Professor of City University of Hong Kong in 2018. On 26 August 2019, IDex was officially inaugurated and elaborated in the conference entitled “Sensors and IoT (SIoT) Standard for Smart City & Inauguration of IEEE 2668 IoT Maturity Index” at the Hong Kong Science and Technology Park (HKSTP). Dr Tsang Kim Fung (Chairman of IEEE 2668 Standard Working Group), Ir Peter T C Yeung (Head of ICT, Smart City & Green Technology, Hong Kong Science Park), Dr Victor Huang (IEEE Industrial Electronics Society Technical Committee Chair of Standards at the time), and Dr Stefan Mozar (President of IEEE Product Safety Engineering Society at the time) attended the SIoT conference and announced the official launch of the IEEE 2668 IDex. This conference also attracted over 300 experts and scholars to discuss and exchange their views. In addition, more than a dozen organisations and government departments have established cooperation relationships with IEEE 2668 IDex, including the Electrical and Mechanical Services Department (EMSD), Hong Kong Science Park Companies, Cyberport companies, Hong Kong Productivity Council, GS1 Hong Kong, Huizhou Tonghu Technology Town, local startups and multinational companies.
SIoT conference
To further boost development progress of IEEE 2668 standard, a Memorandum of Understanding (MoU) on “IEEE 2668 IDex” was signed on 21 September 2021 between HKSTP and the IEEE Hong Kong Section at HKSTP’s Building 17W. The HKSTP Chief Executive Officer, Mr Albert Wong and the IEEE Hong Kong Section Chair (2020 and 2021), Dr Paulina Yenbic Chan, signed the MoU and announced a formal cooperation on the IEEE 2668 IDex standard. Under the agreement, the two parties actively encourage all industries to adopt the IEEE 2668 standard to assist IoT developers and potential users to establish unified evaluation standards on the performance of IoT sensors, devices, networks, systems, infrastructures, applications, etc. to achieve IoT best practices.
Through the great effort of IEEE 2668 Working Group and cooperated enterprises and government departments, IEEE 2668 standard development work has been completed and the formal IEEE 2668 standard was finally approved on 3 December 20222.
HKSTP and IEEE MoU signing ceremony
IDex definition
IDex provides an objective, distinct, visible, quantitative, and reliable indicator of the maturity of an IoT object2. The term “IoT” refers to any devices that are network-connected and/or created with certain functions in mind, such as data collecting, actuation, communication, analytics, processing, and visualisation. A hybrid network with or without an internet connection or a public, or private network could be used. IDex can assess every object within the IoT ecosystem and/ or outside of the IoT environment. An IoT object can often be classified by the IDex into five levels between 1 and 5. IDex 1 represents the lowest performance of an IoT object, while IDex 5 represents the best performance2.
The IEEE 2668 Global Standard gives definition to IDex, which is an objective, knowledge-based, distinct, visible, quantifiable, and reliable indicator of the maturity of an IoT object.
In addition to the discrete representation, IDex may express the maturity of an IoT object in a real number within the range [0, 5]. It must be kept in mind that the IDex level can be calculated from the IDex score (IDex level 1 to IDex level 5)2.
IDex may be customised for a variety of infrastructure fields, namely infrastructure-oriented IDex, to indicate the maturity of an IoT object pertinent to its infrastructure field. In this case, subscript shall be used to denote the infrastructure field in the infrastructure-oriented IDex, namely, IDexinfrastructure field. For instance, when evaluating an IoT system at the device level, the infrastructure-oriented IDex shall be represented as IDexdevice.
Figure 2: Discrete IDex level2
IDex evaluation mechanism
A universal IDex evaluation mechanism for IoT objects is defined by the IEEE 2668 standard. The core of the entire evaluation procedure is the IDex evaluation model. It gets the output from the object input module, which consists of the IoT object's evaluation matrix and identification number. The data of specifications and evaluation parameters, services, and infrastructure field may be included in the evaluation matrixes. Based on the chosen services and infrastructure fields, the IDex model database selector is used to choose the suitable IDex evaluation model for assessing the IoT object. The IDex evaluation model then receives input from the evaluation matrix. After that, the decision-making process is performed based on the IDex evaluation model, which yields the evaluation result in terms of the IDex levels for the chosen services and infrastructural fields. The identification numbers, evaluation matrix, and output IDex levels of the IoT object are outputted and stored in the IDex database.
Figure 3: IDex evaluation mechanism2
IEEE 2668 standardised IoT architecture
Up to this point, the IoT architecture has been defined in a variety of ways, including three-layer, four-layer, seven-layer architecture, and other variations. The IDex evaluation may not be feasible for such inconsistent IoT architectures. In order to make the IDex evaluation easier, a standardised IoT architecture is needed. The interoperability of the IoT solutions in compliance with IEEE 2668 will also be improved by the standardised IoT architecture. In addition, based on the standardised IoT architecture, the IDex evaluation results can also be embedded into IoT solutions through standard interface. Besides, standard services, including data verification and certification processes, can also be performed on the standardised IoT architecture.
In IEEE 2668, a three-layer IoT architecture is adopted, including device layer, network layer, and application layer. The device layer mainly consists of sensors/actuators, processing unit and communication module. The sensors/ actuators refer to devices that detect and transfer energy forms between traditional physical energies (e.g., light, sound wave, mechanical energy) and electrical signals (e.g., analog impulse, digital signal). Theses sensors/actuators are mainly responsible for data collection. The common sensors/actuators include temperature sensors, light intensity sensors, pressure sensors, motors, thermal actuators, and magnetic actuators. The processing unit refers to the computational components, such as a microcontroller (MCU) and system logic. The main tasks of the processing unit are driving sensors/actuators and conveying the collected data to the communication module. The communication module performs data communication between device layer and network layer through wired (e.g., cable, optical fiber, power line) and/or wireless (e.g., Wi-Fi, Bluetooth, LoRa, Sigfox, NB-IoT, 5G) communication technologies.
The network layer performs data aggregation of the collected data in the device layer and data forwarding tasks to the application layer. The typical components of the network layer include radio middleware and network platform. The radio middleware is responsible for establishing a connection between IoT devices and IoT networks based on the same communication technologies used in the communication module. The typical radio middleware includes Wi-Fi routers, LoRa gateways, Sigfox gateways, NB-IoT base stations, and 5G base stations. The network platform (e.g., internet core network, private network, public network, and cloud network) is responsible for data processing and providing application interfaces for user-defined applications.
The application layer mainly consists of various user-defined clients, such as websites, dashboards, and software. These clients perform smart control and decision-making to meet users’ requirements. The communication between network layer and application layer is based on network protocols, such as Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Message Queuing Telemetry (MQTT), Hypertext Transfer Protocol (HTTP), and Constrained Application Protocol (CoAP).
Based on the above three-layer IoT architecture, the IEEE 2668 added three essential elements to ensure system interoperability, namely Digital Information Specification (DIS), Platform Interoperable Device (PID), and Data Processing Middleware (DPM).
Specifically, DIS is a data structure with a standard-defined format that stores all necessary information, such as IDex level of the IoT solution, device specifications, communication configurations, and other operational parameters. By parsing DIS, IoT stakeholders can design and integrate multiple IoT solutions using the standardised IoT architecture. With DIS, stakeholders can also validate the IDex level of IoT systems. The DIS enables the PID and DPM to recognise, calibrate, and register new IoT devices on the existing IoT infrastructure2.
The PID is specified to conduct sensing and actuation functions in addition to communicating with the network layer. The PID is capable of parsing DIS and includes a secure interface for embedding DIS within the PID.
The DPM is defined on the network layer to connect the PID and user-defined applications on the application layer and to facilitate data interchange between the device and application layers. The DPM is able to parse the DIS and includes an interface for conducting typical services, such as IDex evaluation, IDex verification, and data accuracy verification.
Figure 4: IEEE 2668 standardised IoT architecture2
Cooperation of IEEE 2668 IDex and Government-Wide IoT Network (GWIN)
GWIN is a government network of wireless sensors installed across Hong Kong to support various smart applications. GWIN aids the digitalisation of Electrical and Mechanical (E&M) equipment and improves the quality of public services. GWIN enables customers to monitor assets remotely and effectively, evaluate operational data intelligently, and execute predictive maintenance and optimisation. Numerous applications, such as environmental monitoring, activity detection, E&M monitoring, and smart metering, have been implemented in GWIN.
Instead of conventional 4G or short-range wireless technologies (e.g., Wi-Fi and Bluetooth), sensors in GWIN are connected via low-power and private LoRa network. The network offers a large-scale coverage with a smaller number of gateways and also reduces the cost and complexity of sensor deployment. Besides, private LoRa network enhances the security of GWIN and data without requiring the use of a third-party network.
Given the above advantages of LoRa, a variety of smart applications based on it have been developed. However, as the number of LoRa applications increases, serious harmonisation issues have been exposed. It is because LoRa is based on unlicensed frequency band of 923-925 MHz in Hong Kong, which allows free private network deployment by users without authorisation. Apart from LoRa, the unlicensed band (namely, 920-925 MHz) has also been widely used by Sigfox and partial Radio Frequency Identification (RFID) technologies in Hong Kong. The potential rise of unlicensedbased protocols would increase the overheads of shared spectrum, which incurs interferences. In addition, as networks scale up, the performance will be (1) subjected to challenges as a result of network expansions; and (2) exposed to hostile influences due to other non-harmonised networks. In general, the network performance will be potentially degraded (in terms of reliability, security, etc.) due to the ever up-scaling of networks or incorrect system designs. Moreover, the critical applications (such as fire alarms and lift safety) would suffer severe safety issues due to ignorance of alerts in an environment of increasing interference. Therefore, harmonisation efforts among unlicensed-band networks are desperately needed.
In view of this situation, GWIN introduces the IEEE 2668 standard and cooperates with major organisations in Hong Kong, including the MTR Corporation, Hongkong Electric Company, and CLP Power Hong Kong, to establish best practices and guidelines to attain harmony of multiple networks in the wireless 920 to 925 MHz frequency band.
Figure 5: IoT harmonisation networks
IEEE 2668 compliance design for GWIN
To facilitate harmony of multiple networks in the 920-925 MHz unlicensed band, IEEE 2668 compliance design is applied in GWIN to optimise the network architecture and improve system interoperability.
In the IEEE 2668 compliance design, a DIS is customised in the GStack management platform. GStack is a platform that manages all end devices and gateways connected in GWIN. GStack provides an interface for storing information, such as gateway installation date and installation status in CSV format. To comply with IEEE 2668 standard and enable plug and play, the information of end devices is regulated in DIS standardised format.
Figure 6: GStack management platform3
There are two types of customised DIS in GWIN, including Device DIS and PHY DIS. The Device DIS contains the data collection features and control instructions of a device. The PHY DIS contains the communication information between the device layer and network layer. Details of the two DIS could be supplemented from GWIN team if necessary.
Once DIS is stored in GStack successfully, GWIN management would be convenient and efficient. Only those end devices following DIS standards are allowed to join GWIN, which improves the management efficiency. Besides, standard Once DIS is stored in GStack successfully, GWIN management would be convenient and efficient. Only those end devices following DIS standards are allowed to join GWIN, which improves the management efficiency. Besides, standard services and smart services can also be implemented for DIS, such as harmonisation evaluation. The harmonisation evaluation result, namely IDexharmonisation score, can be embedded into DIS for the GWIN managers’ evaluation.
IoT harmonisation evaluation
Based on the IEEE 2668 compliance design in GWIN, IDex evaluation trial on IoT harmonisation is performed to formulate a technical guideline on IoT harmonisation in 920-925 MHz in Hong Kong.
An IDex compliance design has been completed in GWIN to formulate a technical guideline on IoT harmonisation and best practices.
As is well-known, different smart services are assigned different criticalities in IoT networks. For instance, smart fire alarms require immediate response to maintain safety, which has the highest criticality level. On the other hand, smart environment monitoring aims to measure room temperature and humidity automatically, which has a relatively low criticality level. Thus, in consideration of the different criticalities of services, IoT applications are classified into three main categories: safety, control, and monitoring4. IoT services in each category have their own functionalities and criticalities.
- The safety category refers to emergency messages when the alarms are triggered, which usually require an immediate response.
- The control category encompasses command messages sent from operators to end devices within a short time.
- The monitoring category usually represents the periodic sensor measurements from end devices to the network server.
In most real-life scenarios, the three categories’ services coexist and collaborate with one another. In general, different types of services have different quality of service (QoS) requirements—packet loss rate (PLR), latency, throughput and so on—due to their different criticality levels. For instance, monitoring services are usually tolerant of about 10% PLR and 10 s latency, while safety services require strict latency constraint (e.g., < 3 s) and PLR (e.g., <1%). This diversity of QoS requirements makes it difficult to evaluate application services comprehensively and uniformly. In addition, the conventional average QoS of the network cannot represent the actual harmonisation performance with multiple application services. For example, a network with a high average network QoS, but low QoS in its safety services, is usually not accepted by users and obviously does not meet harmonisation demands.
Therefore, Harmonisation IDex (HDex), which is based on IEEE 2668, is proposed to evaluate the harmonisation performance of multiple coexisting application services. HDex is established using the discrepancy between user requirements for each type of service and its QoS performance (namely, latency, PDR, and throughput)5. A final HDex score is provided to present the harmonisation degree of all LoRaWAN services in five simple levels: Bad = 1, Poor = 2, Fair = 3, Good = 4, and Excellent = 5. The user requirements for each type of service are derived from the DIS information (namely, expected PLR, expected latency, expected throughput). The final HDex score will also be embedded into DIS to provide technical guidance for developers.
To ensure that all services in the network reach their expected performances and that IoT harmonisation of multiple applications is achieved, the final average HDex score and the HDex level for each of the services should be above 3. When the HDex score is below 3, improvement measures (for example, reducing the number of end devices, decreasing duty cycle) should be taken by the developers.
Figure 7: HDex evaluation of different solutions
At present, EMSD has established a workgroup to promote IoT harmonisation based on IEEE 2668. Multiple enterprises and organisations in Hong Kong have been involved, including CLP Power, HKSTP, HK Electric, Water Supplies Department, and MTR Corporation. Through strong collaboration, IEEE 2668 will facilitate a more harmonised IoT environment in Hong Kong.
Conclusion and future work
The IEEE 2668 Internet of Things Maturity Index (IDex) is an international IoT standard for grading and ranking all IoT objects. The standard provides the criteria for quantitatively evaluating the performance of IoT objects and delivers guidelines and regulations for implementing the IoT applications. The compliance with the IEEE 2668 standard will increase the efficiency of deploying IoT objects and promote the future integration of IoT objects into the growing IoT environment. By incorporating the IEEE 2668 standard, the Government-Wide IoT Network (GWIN) will coordinate and co-exist with other IoT networks in Hong Kong with full harmonisation to facilitate network performance and efficiency.
Acknowledgements
This article was prepared by Ir Lee Che Kit, Ir Herman Ma Charn Hing, Ir Nicholas Lee, Ir Dr Tsang Kim Fung, Dr Wei Yang, and Dr Wang Hao. Ir Lee Che Kit, Ir Herman Ma Charn Hing, and Ir Nicholas Lee are engineers responsible for the design and development of the Government-Wide IoT Network of the Electrical and Mechanical Services Department of the HKSAR Government; Ir Dr Tsang Kim Fung is the Chairman of IEEE 2668 Working Group and also the Chairman of Standards Committee, IEEE Consumer Electronics Society; Dr Wei Yang and Dr Wang Hao are members of the IEEE 2668 Working Group.
References
- The HKSAR Government (2020). Government releases Smart City Blueprint for Hong Kong 2.0. Available at: https://www.info.gov.hk/gia/general/202012/10/P2020121000626.htm [Accessed on 10 March 2023]
- IEEE Standards Association (2022). IEEE 2668-2022: IEEE Approved Draft Standard for Maturity Index of Internet-of-things: Evaluation, Grading and Ranking. Available at: https://standards.ieee.org/ieee/2668/7254/
- GWIN’s GStack. Available at: https://gstack.gwin.emsd.gov.hk/ [Accessed on 20 March 2023]
- Carvalho D F, Ferrari P, Sisinni E, and Flammini A (2020). ‘Improving Redundancy in LoRaWAN for Mixed-Criticality Scenarios,’ IEEE Systems Journal, 15(3), 3682-3691. doi: https://doi.org/10.1109/JSYST.2020.3015274.
- Wei Y, Tsang K F, Wang W, Zhou M M (2023). ‘Priority-Based Resource Allocation Optimization for Multi-Service LoRaWAN Harmonization in Compliance with IEEE 2668,’ Sensors, 23(5), 2660. doi: https://doi.org/10.3390/s23052660.