4D modeling of soil surface during excavation using a solid-state 2D profilometer mounted on the arm of an excavator

https://doi.org/10.1016/j.autcon.2020.103112Get rights and content

Highlights

  • The method creates 4D surface maps from an excavator.

  • Based on a time-of-flight (TOF) laser profilometer and IMU sensors

  • The 4D map was created by combining 3D (XYZ) data with intensity data.

  • The 4D visualisation of the surface from an excavator was demonstrated.

Abstract

The aim of this research is to create a 4D point cloud map from a trench through a solid-state 2D profilometer. The profilometer is integrated with an 8.5-ton medium-size excavator's machine control system with pose calculation. Accordingly, the point cloud was transformed to a work coordinate system. We used a recently developed pulsed time-of-flight laser light detection and ranging profilometer, which makes possible simultaneous depth measurements to 256 directions in an angle range of ±20° and measurement range from 1 m to 8 m with a frame rate of 25 frames per second under sunlight conditions (≈50 Klux). The analysis is based on a 4D map, which consists of 3D data (XYZ) and intensity information. The XYZ coordinates give the position of an object, and the intensity data can be used to roughly identify materials and recognise surface markings, such as texts. An analysis of the results shows that the detection accuracy of the profilometer is better than ±10 mm. The main advantages of our method are accuracy, high update rate, compact size, real-time measurement and a construction without moving parts. Our technique has a great potential in construction applications, where accurate measurements of a surface shape are needed.

Introduction

Over the past few years, construction machinery, mining and forest industry have experienced a vigorous development in the field of work machines, directly integrated digital processes and automation [1]. The control of earthmoving machinery is based on information models that provide the necessary information for the control systems of machines [2]. The control systems of machines offer the opportunity to improve machine utilisation, savings in fuel costs, improvements in machine maintenance, improvements in health and construction site safely [[3], [4], [5]]. Nowadays, the application of 3D technology is widely spread in different fields of the industry, including those deployed for robotic guidance and product profiling [6,7]. This achievement is due to low-cost and reliable optoelectronic technology and mass-production techniques, such as semiconductor lasers, miniature optics, precision electronics, signal processing and signal processing technologies [8].

The goal of the measurement of 3D surface profiles is to realise work quality, reduce material waste and ensure more efficient and productive construction sites by connecting information. Traditionally, a surveyor prepares data for measuring an equipment and performs surveying in a construction site. In the new method, all measurements are performed by an automated machine guidance that uses positioning devices, such as the Global Positioning Systems (GPS). The use of 3D applications in the worksite can increase the performance of an individual task by typically 15% [9]. Currently, the most commonly used 3D navigation system of excavators is the inertial measurement unit (IMU) sensors embedded in the GPS or global navigation satellite system (GNSS). In the system, the position (XYZ) of the bucket of an excavator can be used to estimate the profile of a surface [10]. Bradley and Seward reported the first implementation of a measurement system that was equipped with GPS and pressure and displacement sensors in 1998 [3]. Trimble Inc. developed 3D visualize tools (Trimple Siteworks Positioning Systems) in the worksite, which is based on the GNSS antenna. Novatron Ltd. offers solutions for digitalizing earthmoving job sites and robotizing earthmoving machinery and also guides the XSitePro system. The XSitePro system is integrated into the open building information modeling (BIM) process (Open Infra. Unfortunately, this method gives coordinates of the individual points from the surface. Only a few papers in the literature report other techniques to measure the profile of a surface from an excavator. Komatsu introduced a measurement system based on stereo cameras, GPS and drones. Large worksites can be quickly surveyed in 3D with the use of a drone equipped with a stereo camera. For example, 3D data help estimate stockpile volumes of materials and trench depths for real-time decision making. Fukui et al. developed a stereo camera-based method for autonomous rock pile excavation from an excavator [11]. The goal of stereo matching is to find out the corresponding matching points in the left and right images after calibration. However, matching these images with high accuracy presents challenges and needs an implementation of specific procedures [12]. Stereo matching offers good target separation, high resolution, density, repeatability, accuracy with no moving parts and simple setups at a very low cost. The measurement accuracy of the method varies depending upon the object to be measured and if the object does not present a rich surface texture [13,14]. Other limiting factors include low reliability in bad light conditions and calibration difficulties. Yamamoto et al. developed a 3D visualisation method in a worksite based on stereo cameras and light detection and ranging (LIDAR) [15]. The 3D image was obtained by combining the vertical plane of the 2D LIDAR to the rotation movement of the excavator. LIDAR systems base their operation on the measurement of the time of flight (TOF) of a pulsed light emitted from a laser diode until it is received by an emitter. Therefore, the measurement requires high-precision electronics to measure at the speed of light. The emission lines are in infrared ranges (905 or 1550 nm). The LIDAR can be classified according to their construction as rotary or solid state [16]. The advantages of LIDAR are real-time distance estimation, quick data collection, use in day and night time and good measurement accuracy [17]. The common limitations for all-optical techniques are extreme weather conditions, such as heavy rain, snow and fog, or risk of mechanical durability. Moreover, the signal quality and maximum range can be reduced due to the contamination of lenses, and the camera methods are sensitive to low or high illumination.

Our goal is to develop an accurate surface profile measurement for an excavator. Method can enable us to prevent overly digging and eliminating excess material use and additional transportation costs at an earth construction site. A new kind of pulsed TOF 2D LIDAR profilometer to the 4D visualisation of the ground surface was used in this work. The profilometer enables simultaneous measurements to 256 directions in an angle range of ±20° and a frame rate of 25 frames per second. Our flash 3D LIDAR sensor with no moving parts provides high precision with a sufficient frame rate and resolution especially in sunny conditions [18,19]. The profilometer was mounted in a plastic case on the Bobcat E85 excavator, which was equipped with Novatron's control system. The solutions suggested in this work are to use a solid-state 2D profilometer to measure the profile of a surface in the horizontal view and to use the movement of the excavator boom to inherently scan the view vertically to provide a 3D surface profile. The data analysis of a terrain is based on a 4D map consisting of 3D (XYZ) and intensity data. The XYZ coordinates determine the location of an object, and intensity information can be used to roughly identify material and surface markings.

Section snippets

Theory (excavator kinematics)

In our system, a profilometer is attached to the arm of excavators. The arm is rotated to obtain 3D scans of the object. Coordinates for each frame of the profilometer is obtained by including its orientation and coordinates and distances it measured to the transformation matrix introduced in this section. Our approach is quaternion algebra.

Fig. 1 shows the coordinate transformation frames in our profilometer system. The position of vector P with respect to the world coordinate system can be

Excavator with a control system

The commercial excavator selected for the research was Bobcat E85 (8.5 tons), which has been modified for automation purposes. Fig. 2 shows a view of the system. An excavator is comprised of three planar implements connected through revolute joints, known as the boom, tipper and bucket. The boom, arm and bucket are controlled via Novatron's control system. The hydraulic system of the machine valves was retrofitted for a precise electrical control. The excavator was outfitted with

Experiments and results

In this work, first, the accuracy of the system was tested by measuring the dimensions of the rectangular box using the robot. Second, a 4D measurement system was demonstrated that is able to map the terrain from the excavator.

Discussion

In this study, we implemented a low-cost 2D laser profilometer for capturing the 4D point cloud information from a terrain. The profilometer was integrated into the control of the excavator system, which enabled automatic data collection. In the construction site, the profilometer quickly gives results, which could be checked on the spot right away. In this study, the detection accuracy of the profilometer was estimated to be better than 10 mm in an indoor condition. The surface reflectivity

Conclusions

The obtained preliminary field test results show that the profilometer allows the inspection and 4D visualisation of the surface from the excavator. The working profilometer proved its feasibility and robustness, and the method offers enough measurement accuracy in relation to the measurement range in the worksite. Two main advantages of the method are identified to be robustness followed from the fact that no moving parts are used in the solid scanner and the relatively low price of the

Declaration of competing interest

There are no conflicts to declare.

Acknowledgment

We gratefully acknowledge funding from the Business Finland, (4294/31/2018).

References (28)

  • C. Emmer et al.

    Advances in 3D measurement data management for industry 4.0

    Procedia Manufacturing

    (2017)
  • P. Saeedi et al.

    An autonomous excavator with vision-based track-slippage control

    IEEE Trans. Control Syst. Technol.

    (2005)
  • R. Heikkilä et al.

    Development of an earthmoving machinery autonomous excavator platform

  • D.A. Bradley et al.

    The development, control and operation of an autonomous robotic excavator

    J. Intell. Robot. Syst.

    (1998)
  • D. Schmidt, K. Berns, An autonomous excavator for landscaping tasks, ISR 2010 (41st International Symposium on...
  • L. Pérez et al.

    Robot guidance using machine vision techniques in industrial environments: a comparative review

    Sensors

    (2016)
  • W.R. Jamroz et al.

    Applied Microphotonics

    (2006)
  • J. Guehring

    Dense 3-D surface acquisition by structured light using off-the-shelf components

  • V. Hokkanen, 3D excavator application in a future jobsite, 1st international Mobile Machine Control (MMC) conference,...
  • S. Lee et al.

    Estimation with applications to dynamic status of an excavator without renovation

  • R. Fukui et al.

    Imitation-based control of automated ore excavator: improvement of autonomous excavation database quality using clustering and association analysis processes

    Journal Advanced Robotics

    (2017)
  • A. O’Riordan et al.

    Stereo vision sensing: review of existing systems

    12th International Conference on Sensing Technology (ICST), IEEE

    (2018)
  • C.S. Pan et al.

    3D terrain reconstruction of construction sites using a stereo camera

    Autom. Constr.

    (2016)
  • M. Zhang et al.

    Applying sensor-based technology to improve construction safety management

    Sensors

    (2017)
  • Cited by (16)

    • End-to-end deep learning for reverse driving trajectory of autonomous bulldozer

      2022, Knowledge-Based Systems
      Citation Excerpt :

      Sun et al. [19] realized dozer blade pose estimation based on fusion-based data for construction guidance; it can improve work accuracy and efficiency. Niskanen et al. [20] built a 4D model of soil through laser light detection; it can identify the marks on the material surface. Kim et al. [21] developed a path-planning algorithm to facilitate the application and development of autonomous construction equipment on construction sites.

    • Real-time mixed reality-based visual warning for construction workforce safety

      2022, Automation in Construction
      Citation Excerpt :

      Unlike 2D image processing, 3D image processing contains an additional depth map to record the distance from a camera to an object. The depth map can let a computer understand the spatial relationship between tracked objects, but getting a depth map requires extra technologies, such as structured light and time of flight [65–67], and these requirements make 3D image processing more difficult to achieve hazard identification on dynamic and complex construction sites. To this end, some researchers are trying to analyse spatial relationships between tracked objects based on 2D images.

    • Time-of-flight sensor for getting shape model of automobiles toward digital 3D imaging approach of autonomous driving

      2021, Automation in Construction
      Citation Excerpt :

      Moving vehicles need to be realistically digitized for interfacing with highway and transportation infrastructure modeling and BIM for the optimization of traffic flows at different times of the day. Our previous work studied the measuring of a soil surface profile from an excavator using a TOF laser 2D profilometer [11]; a 4D surface map was created by combining 2D profilometer 3D (XYZ) data and intensity information. The results indicated encouraging profilometer performance for the visualization of a ground surface.

    • On-site autonomous construction robots: Towards unsupervised building

      2020, Automation in Construction
      Citation Excerpt :

      The demonstration featured adaptive control based on laser range finding used to avoid obstacles, detect the truck, and measure the volume of soil contained inside it. More recent advancements include achieving reliable pose and position estimation using sensor fusion of IMUs and differential GPS [38,39], improved real-time modeling of soil parameters [18,40], and advancements in path planning algorithms [41]. Equipment manufacturers such as Caterpillar and Komatsu have been incrementally adding autonomy such as grade-assist systems to some of their existing line of equipment.

    View all citing articles on Scopus
    View full text