Introduction

Traditional tower crane operation relies heavily on human operators, and faces challenges of operator fatigue, human error, visual blind spots and environmental risks. The recent collapse of a tower crane at a construction site in Sau Mau Ping in September 2022 further motivated the development of an AI Tower Crane by the Hong Kong Housing Authority (HA) and the Hong Kong Centre for Construction Robotics (HKCRC). Therefore, HA signed the Memorandum of Understanding (MoU) with HKCRC on 16 January 2024 to develop a safety-focused, multifunctional AI system for tower cranes. The advent of AI Tower Cranes can mitigate the above issues, by allowing operators to control cranes remotely from ground level without the need to climb into the cabin. In addition, both safety and efficiency can be improved by incorporating advanced features into the AI Tower Crane, such as Artificial Intelligence (AI)-based safety risk detection, automated route planning and lifting, and anti-swing control. Furthermore, Hong Kong is experiencing a declining construction workforce due to the ageing population and challenging working conditions. This innovative technology can help attract young people to the construction industry and alleviate labour shortages.

System overview

The AI Tower Crane system developed by HA and HKCRC integrates the hardware and software together. This article provides a comprehensive overview of the system's four main development stages: tower crane robotisation, AIbased safety monitoring, remote and driver-assistance control, and autonomous driving.

Firstly, robotisation of tower cranes includes sensor installation, electrical modification, communication setup, and remote cockpit development which is based on a highly adaptive design concept for easy application to most flathead tower cranes. Secondly, a smart site safety monitoring system is developed using cutting-edge AI technologies. Thirdly, the remote-control function of tower cranes, which is the core feature of this system, integrates advanced technologies such as mapping, hook and load perception, driver-assistance auto-lifting control and anti-swing control to significantly enhance the safety and efficiency of the tower crane. Finally, a Level 3 autonomous driving approach for tower crane is introduced, which means responsibility for the driving task is assigned to the automated control system, while the operator is only responsible for monitoring and managing fault situations.

Fig. 2 - The rendering images of automatic charging system

Robotisation of tower cranes

First, the sensor installation solution for tower cranes is proposed. As shown in Fig. 1, Light Detection and Ranging (LiDAR) sensors, cameras and encoders are strategically installed. Encoders for the slewing, trolleying and hoisting motors are used to obtain the slewing angle, trolley position, and hook height, respectively. The trolley top-down view LiDAR and camera are used for mapping, hook and load perception, and safety monitoring. The cockpit front view LiDAR and camera are mainly used for AI-based safety monitoring. The rest of the cameras are primarily used to supplement the field of view for remote control operation and safety monitoring.

To accommodate the new functionalities, the crane's electrical system undergoes several upgrades and modifications. Firstly, as shown in Fig. 2, the CraneVision charging system (an integrated component comprising a battery and trolley top-down view LiDAR and camera), which enables automatic charging function. To ensure safety, the system includes an intelligent recognition feature based on Radio-frequency Identification (RFID) that allows charging to commence only when the mechanical systems are fully closed and prepared. The system also features leakage protection, automatically cutting off the power supply when leakage exceeds 30 mA. Secondly, an external programmable logic controller (PLC) is integrated with the tower crane's built-in PLC via a switch. Remote-control signals are received by the cockpit industrial computer, and then sent to the built-in PLC through the switch and external PLC. In this way, only an external switch is added to the builtin PLC without changing the original crane control system.

A robust communication network is essential for the remote operation of tower crane. The network employs highspeed, low-latency communication protocols to ensure seamless data transmission between the trolley, cockpit, and remote cockpit, incorporating both wired and wireless communication channels. The communication infrastructure is designed to handle large volumes of data, including highresolution video and point cloud feeds, sensor data, and control commands. As shown in Fig. 3, for the tower crane trolley, the top-down view LiDAR and camera, encoders, a network bridge, and an industrial computer (for mapping and perception) are connected through a switchboard. In the cockpit, a network bridge, sensors, an external PLC and an industrial computer (for control) are connected through a switchboard. In the remote cockpit, the joystick, displaying devices and the high-performance computer are connected together through a switchboard. The trolley and the cockpit communicate via wireless network bridges, which provide stable data transmission with low latency. Communication between the cockpit and the remote cockpit require higher real-time capability and greater bandwidth due to the transmission of multiple video streams, making fiber-optic communication the preferred solution. However, some construction sites are located in remote areas with limited fiber-optic coverage or poor signal transmission. To address this challenge, HKCRC and HA are collaborating with the Hong Kong Applied Science and Technology Research Institute (ASTRI) to explore enhanced 5G wireless networks tailored for on-site coverage.

Fig. 3 - The communication network tower crane

The remote cockpit is designed to be ergonomic and user-friendly, providing operators with an intuitive and comprehensive interface for crane control. As shown in Fig. 4, multiple versions of the remote cockpit have been deployed and tested across various sites, which include features such as real-time video feeds, AI-assisted safety monitoring and alarming, and driver-assistance control functions.

Fig. 4 - Different versions of the AI remote cockpit

Fig. 5 -: The AI safety monitoring subsystem based on cockpit front-view LiDAR and camera

(Left) subfigure: MiC components are detected and segmented while humans are detected. The MiC components and personnel at risk are marked in red when safety risk is present

(Right) subfigure: MiC components and humans are detected and marked in green when no safety risk is present

AI-based safety monitoring

Several safety-related policies have been introduced and implemented by the government. For example, since 2019, HA has been committed to promoting the widespread adoption of smart site safety monitoring equipment within the construction industry to provide a safe working environment for site personnel. The proposed system contains two AI-based solutions to enhance safety monitoring, which include the cockpit-front-view-LiDAR-and-camerabased system and trolley-top-down-view camera-based system. The former, shown in Fig. 5, adopts sensor fusion technology for 3D object detection and segmentation. It enables the detection of modular integrated construction (MiC) components and humans from a bird's-eye view, and issues warnings to the crane operator when safety risks are identified. The latter, shown in Fig. 6, provides a top-down view of the 2D detection of hooks and personnel, triggering an alarm if a worker enters the lifting danger zone. Based on these alerts, the operator is informed of potential safety risks and can then plan a lifting path with minimal risk.

Fig. 6 - The AI safety monitoring subsystem based on trolley top-down view camera

Fig. 7 - The hook and load perception under various scenarios

Remote and driver-assisted control system

Hook and load perception

Real-time information on the hook and load is essential for driver-assisted path planning and obstacle avoidance. Furthermore, the proposed anti-swing algorithm requires accurate hook or load’s positional feedback. Fig. 7 shows the perception algorithm demonstrate stable and reliable performance in detecting hooks and loads in different operational scenarios.

Automatic lifting and anti-swing control

The automatic lifting algorithm mainly includes mapping, route planning and control. Using data from the top-down view LiDAR and camera, the mapping system creates a detailed 3D point cloud of the construction site. Route planning is the core algorithm of auto-lifting control, as it determines the optimal movement path for the crane. Such technology can dynamically adjust the planned path in response to changes in the environment, ensuring smooth and collision-free operations.

For anti-swing control algorithms, based on the perceived position of the hook, a “hook-chasing” control strategy is adopted. This approach means that when the hook begins to swing, the trolley’s motion is controlled to align with the direction of the hook’s movement. Imagine holding a glass of water while walking—if you move too quickly, the water sloshes and begins to swing. To prevent spillage, you instinctively adjust the glass in opposition to the water’s movement to stabilise it (anti-swing). Fig. 8 shows this control system can effectively minimise hook swing under both the load and unload scenarios.

Meanwhile, a large volume of operational data is collected from experienced tower crane operators to further enhance the performance of the anti-swing and automated lifting algorithms. In addition, these operation data can be used to establish a simulation system that helps new operators quickly acquire operational skills, effectively using AI to assist human operators and reduce the learning curve associated with tower crane operation.

Fig. 8 - The comparison of the hook sway positions in the boom direction with and without anti-swaying feature

(Left) subfigure: no load beneath the hook

(Right) subfigure: load beneath the hook

Autonomous tower cranes

In addition to the remote-control function of the tower crane, HA and HKCRC are also developing Level 3 autonomous driving capabilities for tower cranes to ensure safe and collision-free operations. Unlike remote and driverassistance- control tower cranes, which still rely on human control, autonomous tower cranes can minimise human involvement in operational control. The development of autonomous tower cranes requires a comprehensive legal, regulatory, and industry management framework that must be carefully established, enhanced, and continuously updated. As autonomous technologies advance, it is imperative that regulatory bodies to introduce robust guidelines to ensure safety, efficiency, and accountability in the use of these machines. This includes not only the creation of new policies that address the unique challenges of autonomous operation, but also the regular revision of existing regulations to incorporate emerging best practices and technological innovations. Furthermore, collaboration among industry stakeholders, including government authorities, professional institutions, machinery manufacturers, construction firms, and regulatory agencies, is essential to establish standards that support the safe integration of autonomous tower cranes into construction workflows. Ultimately, a dynamic and responsive regulatory environment will be critical in fostering technological adoption while safeguarding public safety and maintaining operational integrity within the construction sector.

Conclusion

Motivated by recent tower crane collapse incidents in recent years, HA and HKCRC developed a safety-focused multifunctional AI system for AI Tower Cranes, demonstrating both research and development excellence and the practical application of innovative technologies within the construction industry. The proposed system enables operators to control tower cranes remotely from groundlevel cabins, creating a safer and more comfortable working environment for crane operators. In addition, the developed remote and driver-assistance control system incorporates advanced features, such as AI-based safety risk detection, automatic lifting, and anti-swing control. These features not only enhance safety, but also improve operational efficiency. As shown in Fig. 9, the system has already been developed, tested, and deployed at various sites in Hong Kong, representing a significant advancement in construction technology.

The research and application of any new technologies or innovation inevitably face challenges and difficulties, requiring joint efforts from government bodies, industry stakeholders, academic and research institutions. In the future, HA, HKCRC and ASTRI will continue to seek further cooperation, aiming to make crane operations safer, simpler and more efficient.

Fig. 9 - The proposed system is developed, tested, and used on two construction sites in Hong Kong

Acknowledgement

During the implementation of this project, we have encountered many challenges and difficulties. HA would like to express its sincere gratitude to HKCRC. Through the joint efforts and collaboration of all parties, we have leveraged the respective strengths and capabilities of each organisation to promote the application of the AI Tower Crane research project in Hong Kong.

This article was jointly prepared by Ir Daniel H W Leung, Ir Rayson W H Wong, Ir Romeo F H Yiu, Hong Kong Housing Authority and Dr Haobo Liang, Wenshan Xue, Hong Kong Center for Construction Robotics.