4D modeling of soil surface during excavation using a solid-state 2D profilometer mounted on the arm of an excavator
Graphical abstract
Introduction
Over the past few years, construction machinery, mining and forest industry have experienced a vigorous development in the field of work machines, directly integrated digital processes and automation [1]. The control of earthmoving machinery is based on information models that provide the necessary information for the control systems of machines [2]. The control systems of machines offer the opportunity to improve machine utilisation, savings in fuel costs, improvements in machine maintenance, improvements in health and construction site safely [[3], [4], [5]]. Nowadays, the application of 3D technology is widely spread in different fields of the industry, including those deployed for robotic guidance and product profiling [6,7]. This achievement is due to low-cost and reliable optoelectronic technology and mass-production techniques, such as semiconductor lasers, miniature optics, precision electronics, signal processing and signal processing technologies [8].
The goal of the measurement of 3D surface profiles is to realise work quality, reduce material waste and ensure more efficient and productive construction sites by connecting information. Traditionally, a surveyor prepares data for measuring an equipment and performs surveying in a construction site. In the new method, all measurements are performed by an automated machine guidance that uses positioning devices, such as the Global Positioning Systems (GPS). The use of 3D applications in the worksite can increase the performance of an individual task by typically 15% [9]. Currently, the most commonly used 3D navigation system of excavators is the inertial measurement unit (IMU) sensors embedded in the GPS or global navigation satellite system (GNSS). In the system, the position (XYZ) of the bucket of an excavator can be used to estimate the profile of a surface [10]. Bradley and Seward reported the first implementation of a measurement system that was equipped with GPS and pressure and displacement sensors in 1998 [3]. Trimble Inc. developed 3D visualize tools (Trimple Siteworks Positioning Systems) in the worksite, which is based on the GNSS antenna. Novatron Ltd. offers solutions for digitalizing earthmoving job sites and robotizing earthmoving machinery and also guides the XSitePro system. The XSitePro system is integrated into the open building information modeling (BIM) process (Open Infra. Unfortunately, this method gives coordinates of the individual points from the surface. Only a few papers in the literature report other techniques to measure the profile of a surface from an excavator. Komatsu introduced a measurement system based on stereo cameras, GPS and drones. Large worksites can be quickly surveyed in 3D with the use of a drone equipped with a stereo camera. For example, 3D data help estimate stockpile volumes of materials and trench depths for real-time decision making. Fukui et al. developed a stereo camera-based method for autonomous rock pile excavation from an excavator [11]. The goal of stereo matching is to find out the corresponding matching points in the left and right images after calibration. However, matching these images with high accuracy presents challenges and needs an implementation of specific procedures [12]. Stereo matching offers good target separation, high resolution, density, repeatability, accuracy with no moving parts and simple setups at a very low cost. The measurement accuracy of the method varies depending upon the object to be measured and if the object does not present a rich surface texture [13,14]. Other limiting factors include low reliability in bad light conditions and calibration difficulties. Yamamoto et al. developed a 3D visualisation method in a worksite based on stereo cameras and light detection and ranging (LIDAR) [15]. The 3D image was obtained by combining the vertical plane of the 2D LIDAR to the rotation movement of the excavator. LIDAR systems base their operation on the measurement of the time of flight (TOF) of a pulsed light emitted from a laser diode until it is received by an emitter. Therefore, the measurement requires high-precision electronics to measure at the speed of light. The emission lines are in infrared ranges (905 or 1550 nm). The LIDAR can be classified according to their construction as rotary or solid state [16]. The advantages of LIDAR are real-time distance estimation, quick data collection, use in day and night time and good measurement accuracy [17]. The common limitations for all-optical techniques are extreme weather conditions, such as heavy rain, snow and fog, or risk of mechanical durability. Moreover, the signal quality and maximum range can be reduced due to the contamination of lenses, and the camera methods are sensitive to low or high illumination.
Our goal is to develop an accurate surface profile measurement for an excavator. Method can enable us to prevent overly digging and eliminating excess material use and additional transportation costs at an earth construction site. A new kind of pulsed TOF 2D LIDAR profilometer to the 4D visualisation of the ground surface was used in this work. The profilometer enables simultaneous measurements to 256 directions in an angle range of ±20° and a frame rate of 25 frames per second. Our flash 3D LIDAR sensor with no moving parts provides high precision with a sufficient frame rate and resolution especially in sunny conditions [18,19]. The profilometer was mounted in a plastic case on the Bobcat E85 excavator, which was equipped with Novatron's control system. The solutions suggested in this work are to use a solid-state 2D profilometer to measure the profile of a surface in the horizontal view and to use the movement of the excavator boom to inherently scan the view vertically to provide a 3D surface profile. The data analysis of a terrain is based on a 4D map consisting of 3D (XYZ) and intensity data. The XYZ coordinates determine the location of an object, and intensity information can be used to roughly identify material and surface markings.
Section snippets
Theory (excavator kinematics)
In our system, a profilometer is attached to the arm of excavators. The arm is rotated to obtain 3D scans of the object. Coordinates for each frame of the profilometer is obtained by including its orientation and coordinates and distances it measured to the transformation matrix introduced in this section. Our approach is quaternion algebra.
Fig. 1 shows the coordinate transformation frames in our profilometer system. The position of vector P with respect to the world coordinate system can be
Excavator with a control system
The commercial excavator selected for the research was Bobcat E85 (8.5 tons), which has been modified for automation purposes. Fig. 2 shows a view of the system. An excavator is comprised of three planar implements connected through revolute joints, known as the boom, tipper and bucket. The boom, arm and bucket are controlled via Novatron's control system. The hydraulic system of the machine valves was retrofitted for a precise electrical control. The excavator was outfitted with
Experiments and results
In this work, first, the accuracy of the system was tested by measuring the dimensions of the rectangular box using the robot. Second, a 4D measurement system was demonstrated that is able to map the terrain from the excavator.
Discussion
In this study, we implemented a low-cost 2D laser profilometer for capturing the 4D point cloud information from a terrain. The profilometer was integrated into the control of the excavator system, which enabled automatic data collection. In the construction site, the profilometer quickly gives results, which could be checked on the spot right away. In this study, the detection accuracy of the profilometer was estimated to be better than 10 mm in an indoor condition. The surface reflectivity
Conclusions
The obtained preliminary field test results show that the profilometer allows the inspection and 4D visualisation of the surface from the excavator. The working profilometer proved its feasibility and robustness, and the method offers enough measurement accuracy in relation to the measurement range in the worksite. Two main advantages of the method are identified to be robustness followed from the fact that no moving parts are used in the solid scanner and the relatively low price of the
Declaration of competing interest
There are no conflicts to declare.
Acknowledgment
We gratefully acknowledge funding from the Business Finland, (4294/31/2018).
References (28)
- et al.
Advances in 3D measurement data management for industry 4.0
Procedia Manufacturing
(2017) - et al.
An autonomous excavator with vision-based track-slippage control
IEEE Trans. Control Syst. Technol.
(2005) - et al.
Development of an earthmoving machinery autonomous excavator platform
- et al.
The development, control and operation of an autonomous robotic excavator
J. Intell. Robot. Syst.
(1998) - D. Schmidt, K. Berns, An autonomous excavator for landscaping tasks, ISR 2010 (41st International Symposium on...
- et al.
Robot guidance using machine vision techniques in industrial environments: a comparative review
Sensors
(2016) - et al.
Applied Microphotonics
(2006) Dense 3-D surface acquisition by structured light using off-the-shelf components
- V. Hokkanen, 3D excavator application in a future jobsite, 1st international Mobile Machine Control (MMC) conference,...
- et al.
Estimation with applications to dynamic status of an excavator without renovation
Imitation-based control of automated ore excavator: improvement of autonomous excavation database quality using clustering and association analysis processes
Journal Advanced Robotics
Stereo vision sensing: review of existing systems
12th International Conference on Sensing Technology (ICST), IEEE
3D terrain reconstruction of construction sites using a stereo camera
Autom. Constr.
Applying sensor-based technology to improve construction safety management
Sensors
Cited by (16)
Open InfraBIM for remote and autonomous excavation
2023, Automation in ConstructionEnd-to-end deep learning for reverse driving trajectory of autonomous bulldozer
2022, Knowledge-Based SystemsCitation Excerpt :Sun et al. [19] realized dozer blade pose estimation based on fusion-based data for construction guidance; it can improve work accuracy and efficiency. Niskanen et al. [20] built a 4D model of soil through laser light detection; it can identify the marks on the material surface. Kim et al. [21] developed a path-planning algorithm to facilitate the application and development of autonomous construction equipment on construction sites.
Sensing, perception, decision, planning and action of autonomous excavators
2022, Automation in ConstructionReal-time mixed reality-based visual warning for construction workforce safety
2022, Automation in ConstructionCitation Excerpt :Unlike 2D image processing, 3D image processing contains an additional depth map to record the distance from a camera to an object. The depth map can let a computer understand the spatial relationship between tracked objects, but getting a depth map requires extra technologies, such as structured light and time of flight [65–67], and these requirements make 3D image processing more difficult to achieve hazard identification on dynamic and complex construction sites. To this end, some researchers are trying to analyse spatial relationships between tracked objects based on 2D images.
Time-of-flight sensor for getting shape model of automobiles toward digital 3D imaging approach of autonomous driving
2021, Automation in ConstructionCitation Excerpt :Moving vehicles need to be realistically digitized for interfacing with highway and transportation infrastructure modeling and BIM for the optimization of traffic flows at different times of the day. Our previous work studied the measuring of a soil surface profile from an excavator using a TOF laser 2D profilometer [11]; a 4D surface map was created by combining 2D profilometer 3D (XYZ) data and intensity information. The results indicated encouraging profilometer performance for the visualization of a ground surface.
On-site autonomous construction robots: Towards unsupervised building
2020, Automation in ConstructionCitation Excerpt :The demonstration featured adaptive control based on laser range finding used to avoid obstacles, detect the truck, and measure the volume of soil contained inside it. More recent advancements include achieving reliable pose and position estimation using sensor fusion of IMUs and differential GPS [38,39], improved real-time modeling of soil parameters [18,40], and advancements in path planning algorithms [41]. Equipment manufacturers such as Caterpillar and Komatsu have been incrementally adding autonomy such as grade-assist systems to some of their existing line of equipment.