Similar to other global cities, Hong Kong has many ageing buildings that need regular inspection and maintenance to ensure safety. The conventional manual approach to inspecting tall buildings is unsafe, time-consuming and costly. This article presents a novel concept based on an automated drone-based inspection approach.

In this quantitative approach, unlike the traditional qualitative method, a large number of high-resolution visual images are used to map the building envelope in detail. Using cloud-based artificial intelligence (AI) models is imperative to store, process, and detect defects and modifications based on temporal data of the facade. Proposed is a novel concept of the RaSpect Building Fingerprint System (RBFS) as a form of unique identification of the building (Building ID). RBFS comprises a 3D digital model of the building generated using the geo-tagged visual image data. It is combined with general building data such as name, address, owner, year of construction, management company and construction system.

The defects detected and labelled by a trained AI model also form part of the building fingerprint. Such a system is designed to help manage the ageing building stock of a city such as Hong Kong, where one-third of the buildings are predicted to be older than 65 years by 2040. For example, once RBFS is established, the same can be used as a baseline reference to compare with subsequent inspection results automatically to diagnose changes in defects, additions/alterations, repair works, unauthorised building works (UBW), etc. This system can also help municipal governments map any city to locate the ageing buildings and classify them into safety-deficient, functionally-deficient, aesthetically-deficient or no-deficiency categories. With growing urbanisation worldwide, the implementation of such AI-powered digital technology will make built infrastructure safer, greener and smarter. While RBFS is designed mainly for the existing building stock, it has an important role in the case of newly built infrastructure as well. For a newly constructed project, handover from the builder to the owner is an important milestone in the building life cycle. It requires detailed documentation of the building and comparing as-built versus as-designed projects. The facility management team must have a baseline to plan maintenance and building operations. The proposed building fingerprint system will be an important framework to record quality status continuously.

Building Fingerprint System (BFS)

Unique identification for a building can have many applications and benefits. This is analogous to the use of photo identification for a passport which uniquely identifies a person for various purposes. The Building ID, which is designated as the RaSpect Building Fingerprint System (RBFS), can be primarily used to document the condition of a built infrastructure or a particular building at a given point in time. Thus, it helps establish a benchmark of information regarding all aspects of the said property or asset that can be of value to different stakeholders such as its owner, those who may lease or rent it, and regulatory authorities such as the government or local bodies. Professionals such as architects, engineers, real estate agents and building surveyors also provide various services for the building. All information from the subsequent inspection can be added to RBFS and compared to the original data to estimate changes indicating damage, degradation or modifications.

Figure 1 shows a high-level software architecture for the system. It offers several sources of data collection, namely, Unmanned Aerial Systems (UAS) or drones, handheld cameras or mobile apps, Internet of Things (IoT) sensors and building records, layout plans and repair records. The data is uploaded to a building fingerprint cloud platform. The cloud platform can generate 3D models, localise building components, defects, etc., and map out locations. It also stores summary of time-series data gathered from IoT sensors in a highly secured database at specific time points. The platform provides for user management, service subscription and billing management. The system includes inspection services, monitoring services, navigation services, AI analytics services and third party services through open platform pluggable architecture. These are linked to the cloud platform through an API gateway and can be triggered by life cycle events to generate alerts and reports. The system also has Building Information Management (BIM) import/export capability.

The system as envisaged may be initially limited by the capability of the drone inspection system to capture all the features of the facade, such as UBW, and protruding building elements by accommodating images taken from multiple angles. Also, the system may not initially have the capability to incorporate inspection data as per the Hong Kong Mandatory Building Inspection Scheme (2012)¹ which includes a detailed inspection of the building interior. However, the same can be incorporated via images taken by handheld cameras, mobile phones using an app and other suitable means. The AI model's accuracy in detecting building components and defects could also take some time to achieve a reasonable standard, say 75% for example.

Figure 1: High-level software architecture for Building Fingerprint System

Figure 2 below shows a typical schema that may be used to capture important information about the building to be combined with visual inspection information in the form of images of both the exterior and interior of the building.

Figure 2: Typical schema for Building Fingerprint System

Such a schema can gather information about all buildings in a given jurisdiction and continuously update at regular intervals to keep the information current. The data can be stored, and its immutability can be ensured using cryptographic encryption with a blockchain-based system.

AI-powered BFS

Hong Kong introduced the Mandatory Building Inspection Scheme (MBIS) in 2012 to ensure regular and periodic inspection of buildings once they are older than 30 years. This was intended to ensure that safety risks are minimised and building failures or collapses are prevented. Hong Kong has 42,000 buildings, including 8,000 high-rises and skyscrapers, which present a challenge for building inspectors and surveyors to conduct condition assessments of building facades.

In recent years, RaSpect AI has introduced the use of drones as an alternative to manual inspection. It has demonstrated the efficacy of drone inspection, which covers every square inch of the building facade and takes both visual and thermal images of high resolution that can be used to detect all irregularities, defects and deficiencies. Since many images must be analysed, machine learning becomes imperative to automate the detection of defects such as cracks, spalling and delamination. The proprietary AI models (patent pending) developed by RaSpect have proven to reduce the manual effort required in data analysis and defect detection. The AI is built as a sequential combination of Computer Vision algorithms for Semantic Segmentation, object detection and image classification. The detected defects are systematically mapped on the building façade so that a comprehensive evaluation of the current asset status can be performed.

In addition, the geo-tagged images of the building façade can be used to create a 3D model using photogrammetry. This can be an accurate record of the condition of the building at the time of inspection and forms the core of the proposed building fingerprint system. An example of a 3D model built using drone inspection data is shown in Figure 3 below. The results of the inspection are part of the proposed system, as further illustrated in Figure 4. These results include the identified building defects and their severity, classified as follows: (1) critical defects that represent an immediate threat to life from a falling object risk, (2) major defects that represent a safety risk, (3) moderate defects that represent risks to the functionality or serviceability of the building and (4) minor risks that have the potential to become moderate or major risks unless repaired.

The model also identifies cosmetic risks that only affect the aesthetics of the building, such as stains or discolouration. These defects can be mapped onto the building façade either in 2D elevations or 3D models. Such information is important for planning the repair and maintenance of the building, including estimating the quantity and cost of repair.

Figure 3: A 3D photogrammetry model developed using the geo-tagged drone inspection data

Figure 4: Results of drone inspection of building facade in terms of identified defects and their severity

Application in building maintenance, safety and sustainability

Various proposed applications of the BFS and use cases can be divided into three broad categories of maintenance, safety and sustainability. These are described with examples in the following sections.

Application in building maintenance

Building maintenance can be divided into corrective or breakdown maintenance, preventive or scheduled maintenance, and predictive or prescriptive maintenance. In the first case, repair and maintenance are carried out as a reaction in the event of failure. Examples are a blocked drainage system and a de-bonded tile or cladding from the facade falling off that causes injury or damage below. Preventive maintenance takes place when the condition of the building is systematically assessed at regular intervals and repair is undertaken proactively to prevent undesirable incidents from occurring. E&M equipment such as lifts or pumps are often replaced at pre-determined intervals based on their estimated service life. In the case of predictive maintenance, repair action is undertaken  based on continuous monitoring of building conditions that help generate data which can predict breakdowns and incidents. Consequently, preventive action can be taken by facility management or the asset owner.

The most advanced maintenance strategy is predictive maintenance, also known as Condition Based Maintenance (CBM). The continuous monitoring of the condition of the assets using IoT sensors and real-time collection of data facilitates analytics that can predict potential failure, prompting preventive maintenance action. The analytics may even prescribe the required maintenance in prescriptive maintenance, such as repair, replacement of parts or a complete overhaul. In a smart building management system, the BFS can be an integral part that supplements real-time monitoring data with periodic routine inspection and less frequent major survey results (as in the case of an MBIS survey or energy audit). Such integration of different data including equipment changes, retrofits and maintenance records would enable facility managers and asset owners to plan and execute the most appropriate and advanced maintenance programme to ensure functionality and long service life of buildings.

Application to enhance building safety

In the dense urban setting of Hong Kong, the risk of falling objects from the facades of tall buildings is ever-present. Thankfully this has happened infrequently to date and only occasionally has caused injury or fatalities. The latest such incident took place on the 4 January when a piece of glass panel fell from the curtain wall of Manulife Financial Centre, a well-known city landmark. Such incidents are a constant threat to all modern global cities with an increasing number of tall buildings. In December 2021, a 35-year-old construction worker was killed when a piece of concrete fell on him while working on State Theatre Building renovation project, a grade 1 historic building.² In January 2019, a 24-year-old woman was killed by a falling window in the Sham Shui Po area of Hong Kong.³ Many other less significant incidents are reported at frequent intervals. Given Hong Kong's tall building stock of over 42,000 buildings, of which almost 21,000 are over 30 years old, this risk will only worsen with time. Given such fatal outcomes, the safety risks from falling objects cannot be underestimated or ignored.

Using the BFS, it will be possible to record the condition of the building, including its defects as revealed in the drone survey of the exterior facade and condition survey of the interior using the mobile app or handheld camera as per MBIS. However, the system may be limited in covering all potential safety hazards including windows under the Mandatory Window Inspection Scheme (MWIS) by factors such as resolution, angle of image and range. Automated time comparison of inspection results from different surveys would enable the prediction of likely failures that would need attention. Critical and major category defects that may lead to accidents can be prevented by timely repair and maintenance action.

Application to improve building sustainability

Buildings are one of the significant sources of Green House Gas (GHG) emissions because of their operational energy consumption. They account for more than 40% of total energy consumption in the world.⁴ In Hong Kong, buildings account for more than 90% of electricity consumption (which is 56% of the total energy end-use) and 60% of its carbon emissions.^5,6 As the world seeks solutions to achieve the net-zero carbon emission target set for 2050, buildings will be increasingly subject to energy audits to minimise their operational energy consumption and related carbon footprint. The building fingerprint, a repository of all relevant information, can be a starting point for any energy audit process. The inspection data, which would include both visual and infrared images, can be used to identify thermal bridges indicative of energy leakage.

Table 1: Building condition category as per BFS record

Implementation for new built

While the BFS is designed mainly for existing building stock, it has an important role in the case of newly built. Handover of a newly constructed project from the builder to the owner is an important milestone in the building’s life cycle. It requires detailed documentation of the building and a comparison between as-built versus as-designed projects. Identifying all construction defects that may require the contractor to repair beforehand is also an important part of the process. The facility management team needs to have a baseline to design its maintenance and building operations plan. The proposed BFS can capture building quality over its entire life cycle and will be an important framework to support all of the above-mentioned use cases.

Citywide building stock management

The building stock of any city can be better managed with the proposed system, as illustrated in Figure 5. With the condition of the buildings categories, that is, safety-deficient, functionally-deficient, aesthetically-deficient or no-deficiency categories in terms of their BFS, then repair, rehabilitation or renovation can be prioritised accordingly.

Figure 5: Example of a block of buildings categorised as per building fingerprint information

Summary and conclusion

This article presented the concept of a building fingerprint system comprising building metadata, inspection report, 3D photogrammetry model and other relevant related information. The components of the proposed system, such as drone inspection, defect and component identification using AI models and 3D visualisation have already beenoffered by RaSpect in over 100 projects since inception in 2017. The proposed system conceptually brings together these and other components such as a mobile inspection app and blockchain secured immutable database to create the unique building identification. Such a system can create a benchmark for any built infrastructure asset in urban areas to be maintained to ensure safety, functionality and sustainability. While it is primarily geared towards buildings, the same can also be extended to other structures such as bridges, tunnels and highways. Several examples of potential applications have been cited in the article to illustrate the system's utility when implemented at the individual building level or on a citywide scale.

Acknowledgments

RaSpect Intelligence Inspection Limited would like to thank all its investors, especially Hong Kong Science and Technology Parks (HKSTP), for incubation support and all its clients for their faith in the pioneering building inspection solutions developed by the company.

This article was prepared by Dr Dhanada K Mishra, Dr Edward Chan, Ir Dr Dennis Lee, Ir Nigel Cheung, Prof Mathew Yuen and Mr Harris Sun of RaSpect Intelligence Inspection Limited.

References

1. Mandatory Building Inspection Scheme – Building Department (2012). Mandatory Building Inspection Scheme. [online]. Available at: <https://www.bd.gov.hk/en/safety-inspection/mbis/index.html#:~:text=Under the Mandatory Building Inspection,supervise the prescribed repair works>. [Accessed on 25 May 2022].
2. Clifford L (2021). Hong Kong construction worker killed by falling concrete slab at site of State Theatre Building renovation project. South China Morning Post. 6 December 2021. Available at: <https://www.scmp.com/news/hong-kong/society/article/3158604/hong-kong-construction-worker-killed-falling-concrete-slab>. [Accessed on 26 May 2022].
3. Clifford L (2019). Killer falling window at The Mira Hong Kong was no freak accident, government reveals. South China Morning Post. 28 February 2019. Available at: <https://www.scmp.com/news/hongkong/society/article/2187976/killer-falling-window-mira-hong-kongwas-no-freak-accident>. [Accessed on 26 May 2022].
4. World Energy Outlook (2008). IEA. International Energy Agency: Paris, France.
5. How high-density, high-rise Hong Kong uses green buildings to help fight climate change (2019). South China Morning Post. 13 November 2019. Available at: <https://www.scmp.com/presented/lifestyle/topics/building-our-green-future/article/3037555/how-high-densityhigh-rise>. [Accessed on 25 May 2022].
6. Hong Kong Energy End-use Data 2021. EMSD : Set of Data (762). Electrical and Mechanical Services Department, Government of the HKSAR. pp83.