A UAV for bridge inspection: Visual servoing control law with orientation limits

doi:10.1016/j.autcon.2006.12.010

Automation in Construction

Volume 17, Issue 1, November 2007, Pages 3-10

https://doi.org/10.1016/j.autcon.2006.12.010 Get rights and content

Abstract

This paper describes the dynamics of an unmanned aerial vehicle (UAV) for monitoring of structures and maintenance of bridges. It presents a novel control law based on computer vision for quasi-stationary flights above a planar target. The first part of the UAV's mission is the navigation from an initial position to a final position in an unknown 3D environment. The new control law uses the homography matrix computed from the information obtained from the vision system. The control scheme will be derived with backstepping techniques. In order to keep the camera's field of view, the control law uses saturation functions for bounding the UAV orientation and limiting it to very small values.

Introduction

Rapid advances in control theories, computing abilities, communications, and sensing technology offer a great tool for the unmanned aerial vehicles technology. In the last two decades a great interest in the UAV technology has risen in military applications, and many projects have been studied and applied.

In LCPC-Paris (Laboratoire Central des Ponts et Chaussées), we have started a project pertaining to civil applications of a UAV: bridge inspection and traffic surveillance. This project for bridges’ inspection is called PMI (Plate-forme Mobile d'Instrumentation) which is a UAV capable of quasi-stationary flights whose mission is the inspection of bridges and location of defects and cracks.

Ageing infrastructure has become a major concern in western European countries. In France, roughly half of the bridge life cost is due to repairing and maintenance. Since many bridges were built in the sixties, health diagnostics and assessment of residual life already proves very important and will become increasingly crucial in the following decades. To this end, systematic bridge inspection has been organized for a long time and well-defined visual observation and reporting task are periodically carried out on bridges [1], [2], this inspection must be done at least once every 6 years to control the evolution of defects and cracks. Current visual inspection involves a rather heavy logistics. A large platform mounted on a heavy truck is operated by a team of highly specialized inspectors, who work in extremely difficult and risky conditions, looking for small defects or damages in sometimes hardly accessible components of the structure (i.e. cables). Of course, the bridge under inspection is closed to traffic. This gives rise to a classical footbridge. Fig. 1 shows an example of a classical footbridge.

Progresses in visual bridge inspection look possible in the direction of remote sensing and process automating. The use of unmanned aerial vehicles could be of a particular interest as camera carriers and image transmitters. The UAV could follow a predetermined path or could move by visual servoing and detects by the means of image treatment the size and location of defects and cracks. The use of a drone for inspection will give us multiple advantages over traditional inspection method:

•
Reducing work accident risk.
•
Budget reduction: less logistics and less working hours.
•
The bridge will ultimately not be closed for traffic.
•
The possibility of using nondestructive techniques (infrared, shearography, …) for crack's detection.

For the sake of this application, a novel control law based on visual servoing is derived to control the UAV. Almost all control theories for UAV's are built around a vision system, using visual servoing as a control method. A typical vision system will include an off-the-shelf camera, Inertial Navigation System (INS) and in some cases a Global Positioning System (GPS).

In this paper, we consider a general mechanical model of a flying robot capable of quasi-stationary maneuvers. Then we derive a control law from classical backstepping techniques [3] of autonomous hovering system based on operating the translational from the rotational rigid body (airframe) dynamics. A novel approach is also presented; it will limit the orientation of the UAV. Limiting the orientation will ensure that the object will remain in the camera's field of view. We will prove the stability of such a strategy based on saturation functions. Lastly, we present some simulation results of the new control law and the feasibility trial executed on the viaduct of Saint Cloud in Paris.

Section snippets

Related work

The principal question that naturally arises while using vision in control application is:

How should the information from vision sensors be used for robotic control purposes?

There exist three different methods of visual servoing: 3D, 2D and 2½D. 3D visual servoing techniques that involve reconstruction of the target pose with respect to the camera are called: position based visual servoing (3D visual servoing). This kind of techniques leads to a Cartesian motion planning problem. Its main

A general UAV dynamic model

To derive a general dynamic model for a UAV is not an easy task because each model has its own capabilities and aerodynamical properties. In this section, we will derive mechanical equations for UAV's in hover or quasi-stationary conditions.

Let F^⁎ = {E_x, E_y, E_z} denote a right-hand inertial or world frame, such that E_z denotes the vertical direction downwards into the earth. Let ξ = (x, y, z) denote the position of the center of mass of the object in the frame F^⁎ relative to a fixed origin in F^⁎

Camera modeling and visual servoing method

In this connection we will present a brief discussion of the camera projection model and then introduce the homography matrix to use the 2½D visual servoing method.

For the control law we will derive the error dynamics equations from Eqs.(1), (2), (3), (4), they will be based on a defined visual error. Using the Lyapunov control design, we will find the desired force u and the desired orientation R_d to converge the UAV to a position described by a desired image.

Limiting the UAV orientation

In the theoretical developments based on the backstepping [3], the proposed control law assures an exponential convergence towards the desired position. However, this type of convergence is not recommended when the vehicle is initially far from the desired position. Indeed, the dynamic model based on quasi-stationary conditions (hovering conditions) is not valid anymore, because the dynamics of such a convergence will provoke a different flight mode. Moreover, the target image may leave the

Attitude dynamics control

The next step of the control design involves the control of the attitude dynamics such that the error R˜ − I converges exponentially to zero. We will use a quaternion representation of the rotation to obtain a smooth control for R˜. The attitude deviation R˜ is parameterized by a rotation γ˜ around the unit vector k˜. Using Rodrigues’ formula [11] one has $\tilde{R} = I + sin (\tilde{γ}) sk (\tilde{k}) + (1 - cos (\tilde{γ})) sk {(\tilde{k})}^{2} .$ The quaternion representation describing the deviation R˜ is given by [13] $\tilde{η} : = sin \frac{\tilde{γ}}{2} \tilde{k}, {\tilde{η}}_{d} = cos \frac{\tilde{γ}}{2}; with | | \tilde{η} |$

Simulation results

In order to evaluate the efficiency of the proposed servoing technique with orientation limits, simulation results for a hovering robot are presented. The experiment considers a basic stabilization mission. The target is composed of five points: four on the vertices of a planar square and one on its center. The available signals are the pixel coordinates of the five points observed by the camera.

For this experiment, it is assumed that the desired plane is perpendicular to the line of sight. The

Conclusion

This paper reflects a control law of a UAV under some practical restrictions. It presents a control strategy for the autonomous flight with orientation limits in order to keep the object in the camera's view field. This strategy only requires that the system is able to measure, with a video camera, the image plane mapping features on a planar surface. A control law based on separating translation and orientation dynamics is exploited. It also limits the UAV's orientation to small values of

References (14)

A. Teel
Global stabilization and restricted tracking for multiple integrators with bounded controls
Syst. Control. Lett.
(1992)
B. Godart
Instruction technique pour la surveillance et l'entretien des ouvrages d'art, Documents edited and diffused by the lcpc and setra
(1991)
T. Hamel et al.
Visual servoing of under-actuated dynamic rigid body system: An image space approach
S. Hutchinson et al.
A tutorial on visual servocontrol
IEEE Trans. Robot. Autom.
(1996)
E. Malis, Contribution à la modélisation et à la commande en asservissement visuel, Ph.D. thesis, Université de Rennes...
D. Suter et al.
Visual Servoing Based on Homography Estimation for the Stabilization of an X4-flyer
(2002)

There are more references available in the full text version of this article.

Cited by (396)

A lightweight convolutional neural network for automated crack inspection
2024, Construction and Building Materials
Convolutional neural networks (CNN) have been increasingly embraced in building inspection and maintenance to realize automated crack inspection. However, CNNs are still less frequently utilized in the construction industry because most of them are designed with complex architectures that need to be executed by high-powered and expensive GPUs. To improve its practical applicability, this study aims to design a CNN model with a lightweight architecture for crack inspection. The lightweight design is expected to be feasible on a variety of electronic devices, such as mobile phones. To do so, layer pruning and parameter reduction methods were employed to design model depth and weight, respectively. Specifically, the CNN depth was first determined by training and testing 24 experimental groups that contain sequentially increased convolutional and pooling blocks. Secondly, 4 × 4 convolutional filters with 2 strides were inserted between adjacent convolutional layers to reduce model weight and memory. As a result, only 6.82 million weights and 24 M of memory were remined, compared to 138 million weights and 421 M of memory in the widely utilized model VGG-16. During an on-site validation, the proposed lightweight CNN was successfully executed on a mobile robot powered by a CPU-driven robotic control board. Both cracked and non-cracked surfaces could be detected automatically, remotely, and in real-time. Outstanding performance was also demonstrated, with F1-scores in the training and testing datasets of 96.8% and 92.4%, respectively. Overall, the designed lightweight CNN model is validated as more practicable for on-site inspection devices, which facilitate the wide spread of computer vision-based technologies in the construction industry. Meanwhile, the findings provide a potential solution for balancing the weight and accuracy of CNNs.
Co-design of a multirotor UAV with robust control considering handling qualities and motor failure
2024, Aerospace Science and Technology
Over the last few decades, multirotor UAVs have become significantly popular because of their various applications. However, multirotors suffer from poor endurance performance as compared to other aircraft configurations with lifting surfaces. This limitation can be addressed by improving the technology of components or by developing design optimization approaches at the vehicle level. This paper presents a co-design approach for the multirotor structure and propulsive system to improve the vehicle's energetic efficiency while respecting dynamic performance requirements. For this purpose, the approach uses a bi-level formulation to optimize the design and control concurrently. The methodology offers the possibility to activate or not a passive fault tolerant strategy in the controller synthesis to be tolerant to single rotor failure. In contrast to conventional design approaches that sequentially design each part, the co-design approach leads to a better optimal solution by considering the interactions and couplings between design and control. With the proposed co-design approach, the results show the soundness of the proposed design methodology and a significant reduction in the consumed energy compared to a reference non-optimal design while maintaining the same requirements in terms of dynamic performance.
As-built BIM reconstruction of piping systems using smartphone videogrammetry and terrestrial laser scanning
2023, Automation in Construction
The increasing adoption of Building Information Modeling (BIM) in the construction industry has created a demand for methods to update existing facilities with accurate as-built BIM models. While terrestrial laser scanners (TLS) are commonly used for indoor surveys due to their speed and precision, they face limitations in areas with physical constraints like confined spaces and occlusions, with pipes being particularly affected. This paper presents a smartphone videogrammetry approach designed to enhance the accuracy and completeness of terrestrial laser scanner (TLS) point clouds for as-built BIM reconstruction. The method involves capturing missing areas within the TLS point cloud with smartphones and then seamlessly integrating these data into a more comprehensive multi-modal point cloud. This approach is validated using real scan data and offers convenience through the use of readily available and cost-effective smartphones, while its iterative nature facilitates progressive BIM reconstruction of pipe structures. Additionally, a compatible progressive BIM reconstruction framework is outlined, aiming to modernize scan-to-BIM practices in the construction industry, with a case study demonstrating its feasibility.
AI-powered health monitoring of anode baking furnace pits in aluminum production using autonomous drones
2023, Engineering Applications of Artificial Intelligence
Industrial health monitoring in factories is essential for quality assurance, energy and cost reduction, and health and safety. In aluminum factories, anode furnace pits’ flue walls deform over time due to cyclic heating and cooling. They are inspected and classified using manually acquired measurements in a process that takes several hours and is done under high temperatures using specialized equipment. We propose an end-to-end AI-powered system for automated inspection using drones. We fly a drone carrying color and depth cameras to film and navigate a 50-pit furnace floor autonomously in a GPS-denied environment. We then mosaic the recorded videos to produce color and depth mosaics using frame-to-frame motion parameters estimated using the color videos and applied to both. We finish the mosaics using depth-to-color mosaic registration based on maximizing mutual information on gradients. We extract pit images from the mosaics using a YOLOv5 object detection initially trained using a physical floor model with a proposed data augmentation scheme and then fine-tuned for the on-site environment. We achieve a mean average precision of 94.7%. Once pit images are detected and initially classified, we propose a dual-stage segmentation algorithm using the Hough transform and a semantic segmentation network trained using probabilistic feature images with a precision of 96.67%. Segmented pits with depth information allow us to produce 3D models of pits to aid in temporal monitoring and diagnosis confirmation. Our system is cost-effective and reduces inspection downtime by 87%, eliminating the need for human intervention.
Automated crack detection and crack depth prediction for reinforced concrete structures using deep learning
2023, Construction and Building Materials
Automatic inspection for crack detection and estimation of the crack depth is critical in assessing the damage and determining the appropriate method of repair in concrete structures. Most of the studies which have employed deep learning models for automatic inspection are limited to the detection and estimation of the width, length, area, and direction of cracks. The innovation of this study lies in developing a comprehensive automated crack detection and crack depth evaluation framework for concrete structures using images taken from portable devices. Firstly, a binary-class Convolutional Neural Network (CNN) model was developed to automatically detect the cracks on a concrete surface. Secondly, an integrated CNN model combining the convolutional feature extraction layers and regression models (RF and XGBoost) was developed to automatically predict the depth of the cracks. The proposed framework has been validated on a reinforced concrete (RC) slab. Results indicate the models are accurate and reliable for automated inspection of the cracks which could help in evaluating the condition of a concrete structure and choosing suitable repair methods.
Survey of robotics technologies for civil infrastructure inspection
2023, Journal of Infrastructure Intelligence and Resilience
The demands for infrastructure inspection using autonomous robots have noticeably increased, and the market is expected to grow accordingly. One of the advantages is that autonomous robots can navigate the environment and interact with humans because an inspection of a high-rise building, for instance, is considered an extremely challenging task for a human. Inspection robot systems can be classified as ground, aerial, underwater robots, or types of sensors used for inspection, such as visual or non-visual sensors. Users can choose a specific robot platform for their target and environment among them. This paper reviews various inspection robots and categorizes them according to their automated inspection system to aid the user in a good choice. Especially, unmanned aerial vehicles (UAVs) are preferred among the various robot platforms due to their high manoeuvrability. Thus, two types of aerial inspection robot platforms, such as climbing aerial robots and autonomous drone navigation systems, are introduced in detail.

View all citing articles on Scopus

View full text

A UAV for bridge inspection: Visual servoing control law with orientation limits

Abstract

Introduction

Section snippets

Related work

A general UAV dynamic model

Camera modeling and visual servoing method

Limiting the UAV orientation

Attitude dynamics control

Simulation results

Conclusion

Syst. Control. Lett.

Instruction technique pour la surveillance et l'entretien des ouvrages d'art, Documents edited and diffused by the lcpc and setra

Visual servoing of under-actuated dynamic rigid body system: An image space approach

A tutorial on visual servocontrol

IEEE Trans. Robot. Autom.

Visual Servoing Based on Homography Estimation for the Stabilization of an X4-flyer