The power of virtual reality in modern engineering
By Ir Dr Henry LAUDecades back, when we engineers looked at virtual reality (VR), it all seemed that these are merely gimmicks for gamers and futuristic sci-fi artifices. Nowadays, with the rapid development of technology in computers and information, sensors and devices, complemented by new materials and modern engineering processes, VR has become a reality in our everyday life.
The deployment of VR technology has emerged as one of the very attractive and, in some cases, indispensable means for 'serious undertakings' including engineering design, system analysis, process simulation, field operations training, information visualisation, human skill qualification, etc. VR systems ranging from super-high-end multi-screen Cave Automatic Virtual Environment or CAVE-based systems to consumer-grade head-mounted devices are readily available even on high street shops. VR not only appears in national labs or the silver screen, but in the hands of professionals going about their everyday jobs and even school kids learning something new.
In fact, the idea of VR is not new. Medieval artists and early filmmakers have long dreamt of using the concept to show their audiences what exists in their mind or things that happen some distance away. The amazing features of telepresence and immersion that underpin the fundamentals of VR were developed with the use of panoramic images, perspective drawings, geometric surfaces, binocular viewing devices, etc, to create a sense of depth and space, thereby re-creating an environment that simulates physical presence in the real or imaginary worlds, and perhaps even let users interact in that world. Today some VR systems go as far as vividly creating a host of sensory experiences including 3D images, surround sound, haptic touch, smell, taste, motion and other interactivities that occur in our everyday life.
The boom of VR is also boosted by investors and financiers who identify VR as the 'next mega tech theme' with a projected global annual market of over US$10 billion¹. With VR technology providing users with a radically new technology experience, new markets will be opened in a wide spectrum of domains including science and engineering, creative arts, education, social media and many more.
What is VR?
So what is VR then? Generally speaking, anything that enables human users to perceive and experience an artificially created scenery or situation can be called VR. Looked at holistically, VR falls at one end of the spectrum of mixed reality (Figure 1). When we expand the notion of reality and virtuality, a world that is created using virtual or computer-generated objects representing certain real-life scenarios is often referred to as VR. At the other end of the spectrum, where everything is real and physical, we have the real world. Somewhere in between these two extremes, where both real and virtual objects coexist and may be interacting with one another, is the world we call 'augmented reality' (AR) or mixed reality. In fact, quite a number of today's realisations of so-called 'VR' applications, such as the projection of images onto real objects and the superposition of computer-generated objects onto real scenes; are AR implementations.
Technically speaking, a 'true' VR system has two very important features; namely, the system should provide the sense of immersion and telepresence to a human user. In this context, if one cannot distinguish one's experience in a computer-generated world from that in a real world, then this computer-generated world is an immersive environment and this sensation is called immersion. Telepresence on the other hand is the ability to place one into a space that is some distance away, physically non-collocated or even non-existent.
Under these two fundamental characteristics, other corresponding features include interactivity with virtual objects, real-time motion tracking, and stereoscopic display that are all natural in our everyday world and enhance the quality of the VR system. Under this definition, I would hesitate to consider 3D movies or computer games as VR.
VR and engineering
Back to our bread and butter: how can VR help engineers do our job better? Considering the attractive features of VR, the ability to visualise and interact with lifelike computer-generated objects that are crafted by our imagination, images that are so realistic that it is hard to tell whether they are real or virtual, scenarios that are created without the boundaries of space and time, etc; all these have inspired professional engineers to dream and innovate novel approaches to solve real-world problems that we are facing every day, in a better, more efficient and effective manner, both in cost and time.
The ability of VR technology to rapidly build models of complex systems with realistic dynamics and create scenarios that users can interact with has attracted plenty of interest within the engineering community. Proposals such as product design with interactive VR, setting up and evaluation of production lines in virtual factories, VR-empowered surveying, construction project management in virtual construction sites, power plant maintenance planning with AR, etc, are expected to vastly enhance operation efficiency as well as the quality of work when compared with traditional practices. These sound very much feasible and welcoming, but to what extend are they happening?
An immersive virtual environment
At the University of Hong Kong's Department of Industrial & Manufacturing Systems Engineering, academics and professional engineers with expertise in cyber physical systems have been working on VR technology research and development in the past decade. Our innovation is the VR environment 'imseCAVE'² and a handful of content development tools for the creation of immersive virtual environments and interactive VR and AR content for various engineering applications (Figure 2).
imseCAVE is a fully immersive and interactive VR environment that provides extremely vivid stereoscopic views of 3D sceneries and allows users to interact with virtual objects with intuitive user interfaces. Designed based on the Cave Automatic Virtual Environment or CAVE, the system is a low-cost, high-performance VR system that is capable of creating virtual environment through the display of computer-generated stereographic images onto a number of wall and floor screens. Through continual tracking of the user's position with a well-integrated real-time motion tracking system, the computer-generated images are updated according to the perspective of the user. In addition to immersive visual content, the system incorporates surround sound effect and natural 3D user interfaces - the MagicPad and MaginPen³ - that mimic the natural pen and paper interface. All these come together to deliver a fully immersive sensation. This versatile VR platform greatly facilitates the design of products and processes, analysis of complex engineering systems and operations, quality-assurance of structures and buildings, and even online e-training and e-learning for engineering students.
VR in action
Over the years, the imseCAVE VR system and corresponding VR/AR technology have been deployed in the training of skilled operators and professionals in different engineering disciplines and the simulation of very large and complex logistics systems including automated air cargo terminals and port container terminals in collaboration with industries. These virtual training systems and simulation systems not only allow engineers to carrying out their jobs more effectively, but the system also provides a means for management and other stakeholders to visualise an operation in a more transparent and direct manner. Moreover, through the use of VR, systems and operations can be studied without the need to disturb normal operation. More importantly, experiments can be performed on virtual systems and environments where the target operations and design parameters can be investigated in an authentic, safe and controlled environment. We highlight some of these successful cases here.
Field operations training
A VR-enabled interactive e-training system developed for the aviation logistics industry that is designed to train supervisors and operators in ramp operation where air cargo are loaded and unloaded from freight aircraft. The system is developed using imseCAVE to recreate scenarios in air cargo handling operations where operators use various user interfaces including hand-held devices and mock-up freight aircraft cargo door control panels to manipulate the virtual air cargo. In addition to facilitating effective operator's skill capturing and assessment, the system enables offline planning of cargo operation and performance evaluation (Figures 3).
Equipment and machine simulation
In the construction industry, the safe operation of a tower crane, which is one of the key equipment on a construction site, requires special skills and experience. A tower crane simulator is developed using VR technology to recreate a realistic control cabin and a virtual construction site to provide an immersive training environment for training and skill qualification of tower crane operators. The system allows users to 'operate' the virtual tower crane through bespoke control interface to practise the skill of operating a tower crane according to defined industrial standards and safety requirements (Figure 4).
A VR quay crane operating environment was created using imseCAVE. The system provides a vivid experience of container handling with port container terminal equipment such as quay cranes, tractors and container vessels. A pair of physical joysticks that mimic the actual operator's interface is integrated with the virtual environment. The system allows crane operators to be trained offline manner, where different operating conditions can be simulated in a controlled environment (Figure 5).
System analysis and evaluation
Using VR technology, complex systems and information can be effectively displayed and visualised by system engineers carrying out system analysis and design. Through appropriate deployment of VR technology, the operation of an automated material handling system for handling express cargo is displayed in the 3D space through a projection-based CAVE automatic virtual environment so that engineers can directly visualise cargo flow and the performance of the conveyor-based cargo sorting system (Figures 6).
Facility and product design
VR is also used for facility and product development. Building information modelling information can be visualised in a 3D space to help architects and engineers study and evaluate the design of buildings prior to their actual construction (Figure 7). In product development, VR complements 3D printing and rapid prototyping to enable designers to visualise their designs in a VR environment where the virtual product is displayed in 3D and can be manipulated virtually (Figure 8).
While we are currently seeing a lot of hype surrounding VR with keen interest within the community and many people believing in the superpower of VR, we professional engineers need to manage our expectations carefully and not become over-enthusiastic about what VR can do for us. The time will come as the development of software and hardware such as game engines, sensing and actuation devices, projection and computing systems are more up to users' expectations so that the mass deployment of VR by the engineering community could become a reality.
1. "Next mega tech theme is virtual reality", PiperJaffray Investment Research, May 2015.
2. H Y K Lau and L K Y Chan: :"A virtual reality-based system for visual analysis of complex industrial facilities and operations", Proceedings of the 8th International Conference on Manufacturing and Management, 2004.
3. L K Y Chan and H Y K Lau: (2010) "The MagicPad: a spatial augmented reality-based user interface". Proceedings of the IDMME Virtual Concept 2010, Bordeaux, France, 20-22 October 2010.
About the author: Ir Dr Henry Lau is an associate professor in the Department of Industrial & Manufacturing Systems Engineering, the University of Hong Kong, and a committee member of MI Division, Session 2016/2017.