Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Computer-vision-guided semi-autonomous concrete crack repair for infrastructure maintenance using a robotic arm

Computer-vision-guided semi-autonomous concrete crack repair for infrastructure maintenance using... Engineering inspection and maintenance technologies play an important role in safety, operation, maintenance and management of buildings. In project construction control, supervision of engineering quality is a difficult task. To address such inspection and maintenance issues, this study presents a computer‑ vision‑ guided semi‑autonomous robotic system for identification and repair of concrete cracks, and humans can make repair plans for this system. Con‑ crete cracks are characterized through computer vision, and a crack feature database is established. Furthermore, a trajectory generation and coordinate transformation method is designed to determine the robotic execution coordi‑ nates. In addition, a knowledge base repair method is examined to make appropriate decisions on repair technology for concrete cracks, and a robotic arm is designed for crack repair. Finally, simulations and experiments are conducted, proving the feasibility of the repair method proposed. The result of this study can potentially improve the perfor‑ mance of on‑site automatic concrete crack repair, while addressing such issues as high accident rate, low efficiency, and big loss of skilled workers. Keywords: Computer vision, Concrete crack repair, Robotic construction, Semi‑autonomous, Knowledge base system, Human decision making recognition of concrete cracks (Dan & Dan, 2021). 1 Introduction Surface condition deficiencies are often evaluated by Cracks on the surfaces of concrete engineering structures combining computer-vision detection and surveying are among the earliest indicators of structural deteriora- equipment (Shamsabadi, et  al., 2022). As a result, com- tion. Structures suffer from fatigue stress and cyclic load - puter-vision-based concrete-crack detection is becoming ing (Tedeschi & Benedetto, 2017). As a result of external a type of non-destructive testing technique (Kim et  al., loads, minute cracks on concrete surfaces may produce 2022), with many methodologies used to determine the interconnected passageways, which will worsen the safety existence and location of cracks. Although some stud- of structures (Algaifi et  al., 2018). Thus, civil engineers ies have focused on extracting such basic information as face the challenge of reducing the harm caused by dete- length, width, and depth (Cha et al., 2017), such informa- riorating structures. In this regard, intelligent technology tion is not enough in making decisions on crack repair for unmanned detection and repair is necessary. behavior. As one of the potentially useful technologies, com- Conventional repair materials are classified under puter vision is increasingly implemented in automated various criteria (Tsiatas & Robinson, 1795). Crack epoxy injection is one of the common methods to reconstruct *Correspondence: charleschou@163.com the lost strength properties (Ahmad et  al., 2013). Based National Center of Technology Innovation for Digital Construction, on the properties of the epoxy injected, this type of Huazhong University of Science and Technology, Wuhan 430074, China repair proves effective on new cracks that arise outside Full list of author information is available at the end of the article © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. Chen et al. AI in Civil Engineering (2022) 1:9 Page 2 of 16 the region of previously repaired cracks. With regard to morphological approach, percolation-based method, and repair techniques for concrete cracks, such information practical technique (Wang et  al., 2010), most of which as length, width, and depth is not enough crack remedy. are used to determine whether cracks exist and where Therefore, researchers have started to pay attention to they are located. Spatial wavelet transformation has also the use of bacteria to repair cracks in concrete (Maedeh been proposed to detect and localize cracks by amplify- & Mehdi et al., 2020; Zhou et al., 2019). A review of the ing weak perturbation signal at crack locations (Mar- existing researches reveals that most of the research on dasi et  al., 2018). Liu et  al. (2019) have used U-net fully concrete-crack repair techniques is mainly on materials, convolutional networks to detect concrete cracks based and few studies have been conducted by using robots. on computer vision. Liu et  al. (2021) have adopted the A robot system for construction quality assessment has integration of Convolutional Neural Network and Active been used to optimize the autonomous visual inspection Contour Model to perform crack segmentation. With function, so as to cut labor cost and improve accuracy deep learning in frequency domain, Zhang et  al. (2020) (Liu et  al., 2017). An automated integration system has try to detect cracks on concrete bridge decks in real time. been developed for remote inspection and repair without direct human intervention. In addition, a semi-autono- mous robotic system has been proposed for inspection 2.2 Repair techniques for concrete cracks and repair of pavements and bridges, while improving the Many existing researches are focused on restora- security of properties and inspectors (Sutter et al., 2018). tive materials. For example, viscosity and mechanical However, these repair platforms are semi-autonomous strength of epoxy materials are explored for repairing and pre-programmed. In contrast, this study will design concrete cracks under low-temperature construction in a computer-vision-guided semi-autonomous robotic sys- winter (Dana et  al., 2021). Given that the quality moni- tem for concrete crack inspection and repair, with the toring of crack fullness and solidification is important help of human decisions. (Bykov et al., 2017), high efficiency of injection processes is pursued by strict observation of the process norms, particularly about the qualitative fullness of crack cavity 2 Related researches and full solidification of injected materials. However, no 2.1 C omputervision‑ ‑ guided recognition of concrete unified test method is available both at home and abroad. cracks u Th s, quality control in crack repair is difficult to imple - Cracks on concrete surfaces of engineering structures ment (Wang et al., 2015; Zhu et al., 2015). are among the earliest indicators of structural deterio- Traditional methods are based on manual repair. How- ration as a result of fatigue and other negative factors ever, computational and autonomous properties play (Mohan & Poobal, 2018), leading to weakened mate- an increasingly important role in improving the knowl- rial integrity. Therefore, observing concrete cracking is edge about the effect of most concrete crack-repair crucial for characterizing the safety of structures. Most techniques on the mechanical performance of repaired existing approaches to crack detection are empirical, components (Marazani et  al., 2017). For self-healing with human’s visual observation becoming the common agents in concrete, cement stone samples can deposit method of crack detection. However, this and other simi- a new layer of calcium carbonate minerals on the sur- lar conventional methods to characterize and inspect face to cover cracks (Boumaaza et  al., 2017; Choi et  al., cracks are time-consuming and error-prone. So an inno- 2017). The crack healing potentials of bacteria have been vative method is necessary. Recently, the developments compared with traditional repair techniques through in digital facility and methods have widened the initial experimental observation (Maedeh et al., 2020). Further- field of concrete crack recognition (Valença et al., 2012). more, researchers in the process optimization field have The purpose of crack inspection may vary, depending invented new technologies and optimized some repair on parameters to be inspected. Crack detection may be operations. In some researches, a piezoelectric patch was delivered based on length, width, depth, and direction bonded on a beam through an external voltage to affect of cracks (Pantoja-Rosero et al., 2022). The major advan - the closure of concrete cracks (Riccardo et  al., 2020). tage of the computer vision technique lies in that it can These techniques can also be used to control the fail - provide more accurate results than traditional manual ure modes and stress distribution around beam chords methods (Shanaka et  al., 2022). Some of the conun- (Osman et  al., 2017). In a systematic method, Kim et  al. drums in computer vision recognition are related with (2019) have evaluated the performance of crack repair different shapes, irregular cracks, and various noises. materials by using PZT-based EMI technology, which By virtue of superior performance of computer vision, can reflect the structural characteristics in evaluating many types of image processing and recognition meth- the repair efficiency over time. Ramesh et al. (2021) have ods have been proposed, such as integrated algorithm, Chen  et al. AI in Civil Engineering (2022) 1:9 Page 3 of 16 deployed non-destructive testing methods to repair and the experience of workers, but also on the level of their refurbish reinforced concrete structures. fatigue. Therefore, this system to be designed shall ensure To detect concrete cracks in engineering, personnel the safety and suitability of the control mode. Being may have to enter hazardous environments. Automatic semi-autonomous, this system can improve the inspec- approaches are the only effective way to support human tion efficiency and accuracy by automatically identify - exploring extreme environments. A review of the existing ing concrete cracks. Tele-operation of robots should be researches on concrete-crack repair techniques finds that considered for the operation process. In addition, a semi- they generally pay attention to materials. supervised computer vision system has been developed in ROBO-SPECT European FP7 project to detect tunnel 2.3 I ntegration of automated recognition and repair diseases (Menendez et al., 2018), and Harsh et al. (2020) for concrete cracks have used robots and computer vision to detect and Several previous studies have attempted to optimize quantify defects in dam spillways. As these repair plat- the crack recognition process for concrete values. For forms are semi-autonomous and pre-programmed, most example, a computer vision system for a train inspec- of the current crack inspection and repair platforms are tion monorail was proposed and installed in the Large focused only on detection. To address this limitation, this Hadron Collider to gather data from various sensors study introduces a computer-vision-guided semi-auton- and capture images by the European Organization for omous robotic system, which is dedicated to concrete Nuclear Research, only purposed for recording data and crack inspection and repair projects that involve decision reducing personnel intervention (Attard et al., 2018). The making by humans. recognition process of engineering concrete cracks has been automated to a certain extent based on deep learn-3 Methodology ing methods (Chheng & Likitlersuang, 2018), including As shown in Fig.  1, four main steps are involved in the Convolutional Neural Network (CNN), Recurrent Neu- computer-vision-guided semi-autonomous concrete ral Network (RNN), and Transfer Learning (2017b; Cha crack repair process using robotic arms. The first step et  al., 2018; Huang et  al., 2018; Xue & Li, 2018; Zhang includes feature acquisition and trajectory extraction, et al., 2017a). Overall, the structural crack repair process purposed to recognize concrete cracks. Feature acquisi- is slow, labor intensive, and subjective so far. To over- tion is performed to determine the length, width, depth, come these working limitations, automatic repair sys- and other measures of cracks through a computer vision tems have to be developed (Kovačević et al. 2021). process, while trajectory coordinates are calculated via An urgent need is to design a fully automated integra- hand-eye calibration. After a knowledge base is created tion system for inspection and reparation, which shall to determine appropriate crack features for the repair enable remote operations, without any need for direct method, the overall repair process will be simulated by human intervention. One selection is to improve the code programming and software operation. The decision automatic behaviors of robots. It has been shown that made by humans based on the knowledge base includes the quality of manual operations depends not only on the establishment of relevant standards and specification Fig. 1 Architecture of the crack repair system Chen et al. AI in Civil Engineering (2022) 1:9 Page 4 of 16 databases, e.g., a crack feature database. Once the simula- position, crack position, crack material, crack properties, tion process is determined, the crack repair process will crack width, crack length, crack depth, and crack direc- be launched, including the design of robotic arms, the tion, as shown in Table 1. execution of repair operation, and the evaluation in the Various features are detected with different tech - next step. Finally, the semi-autonomous concrete-crack niques. For instance, some crack features are acquired repair process is tested and verified under laboratory in infrared, laser, ultrasonic, and various other com- conditions based on computer vision. Each aspect of the puter-vision-based imaging methods. With regard to the repair process is described in detail in Fig. 1. infrastructure for the crack feature acquisition using the computer vision technique, a general workflow of such 3.1 A cquisition of concrete crack features using computer acquisition is shown in Fig. 2. vision The computer-vision-based crack-feature acquisition 3.2 Trajectory extraction and hand–eye calibration involves: detecting cracks, determining locations of crack Extraction of the coordinates of cracks involves image components, and measuring the length, width, and depth pre-processing, denoising, and edge-region processing of cracks. A project requires a large amount of data, for crack trajectory extraction. Region points of a crack which have to be recorded and organized through vari- trajectory are used in thresholding value selection cre- ous methods. A database is needed to store the data on ated from edge points on the basis of human decision the majority of concrete crack repairs. In general, manag- making. A region of interest (ROI) needs to be set to ing the raw data involves an independent database, which insulate the background and the crack trajectory region, can be built with an electronic spreadsheet application. and the minimum distance from the region to the crack Crack features can be divided into eight categories based is not less than 1  cm but not more than 2  cm. A filter - on the following engineering feature values: component ing method should be used to eliminate the useless Table 1 Features and values of crack feature database No. Crack feature Edit mode Feature value 1 Component position Single election Columns, beam, plate, wall, roof frame, cantilevered member, and floor 2 Crack location Single election Cushion layer, surface layer, structural layer, plastering layer, coating, joint, middle and lower part, upper part of support, end, periphery of plate, and bottom of plate 3 Crack material Single election Concrete, masonry, plastering, cement, and asphalt 4 Crack property Single election Splitting, vertical, penetrating, cracking, weathering, axial compression, eccentric compression, and bending 5 Crack width Single election < 0.2 mm, 0.2–0.3 mm, 0.3–0.4 mm, 0.4–0.5 mm, 0.5–1 mm, 1–1.5 mm, and > 0.5 mm 6 Crack length Direct import Text type 7 Crack depth Direct import Text type 8 Crack direction Single election Vertical, horizontal, and oblique Fig. 2 Workflow of crack feature acquisition Chen  et al. AI in Civil Engineering (2022) 1:9 Page 5 of 16 information. To transform a gray image into a binary one, this study implements the thresholding technique, with which a picture can be compartmentalized by using the local threshold. The points of the trajectory are depicted from the points of the area outline. In the marginal area, the sub-pixel type of data contour is gen- erated by utilizing the marginal form. The outer margin of the pixels is utilized as contour points, as shown in Fig. 3. 3.2.1 Camera coordinate calculation Wiping off useless pixels through the filtering technology is crucial to identifying the important data. The thresh - Fig. 4 Pinhole camera model olding measure, in which an image is divided into mul- tiple local thresholds, is implemented in this study to transform a gray image into a binary one. The measure - in Fig. 5. T is the hand–eye relationship and T is the kin- x . 6 ment can be carried out under any normal indoor light ematics positive value. conditions. Camera coordinates are calculated in the pin- hole camera model (Eq.  1), as shown in Fig.  4, where K 0 T = T · T · T , (2) G . 6 x C represents camera intrinsics. The relationship between the polar coordinates of the x u manipulator and the coordinate system of the target object −1 y v = K z , c c (1) (calibration board) T is a fixed value. T represents the G 6 z 1 transformation relationship between the coordinate sys- tem of the 6 degrees of freedom manipulator and the end (forward kinematics) of the manipulator. T represents the 3.2.2 Robot coordinate calculation coordinate transformation relationship between the end of Parameter A can be calculated with the form, and the cal- the robotic arm and the camera. T represents the trans- culation can be completed in the Matlab arm calibration formation of the camera to the coordinate system of the toolbox. Parameter B can be solved by using the Matlab target object. camera calibration toolbox and Zhang Zhengyou Camera Calibration Method (Zhang, 2000). The relative positional T · T · T = T · T · T , 6 x C 6 x C (3) 1 1 2 2 relationship between a camera and the sixth axis of the manipulator must be changed by three positions, as shown Fig. 3 Workflow of trajectory extraction and hand ‑ eye calibration Chen et al. AI in Civil Engineering (2022) 1:9 Page 6 of 16 T indicates the hand-eye relationship and T indicates x . 6 the kinematics positive solution of the manipulator. 3.3 Design of knowledge base and simulation of crack repair A knowledge base is designed for the crack repair method driven by meta-knowledge. This inference process will screen data from the crack feature, repair standard, and robot code databases and contribute to the data ware- house system, as presented in Table  2. Furthermore, the expression of the knowledge base is constantly adjusted on the basis of the robot execution characteristics, so as to accurately select the alternatives containing the tools, materials, standards, and related information. The process of requirement analysis, database design, data coding, item importing, data verification and Fig. 5 Schematic of T value solution screening, back-up, and exporting for analysis is illus- trated in Fig.  6. With the RobotStudio design module, a virtual simulation environment is designed, where a robot arm and concrete cracks can be displayed. First, the −1 −1 T · T · T = T · T · T , 6 x x C (4) 1 2 6 C 2 1 robot or robotic platform to repair the concrete cracks can be deployed. Second, the execution process can be Equation  (4) can also be expressed as A · X  =  X · B , simulated to detect the movement conflict by using a −1 where A = T · T represents the relative relationship 6 1 2 collision detection module and code compilation. The −1 between the two position postures, and B = T · T 2 C 1 simulation enables the robot to hold back human beings represents the relative relationship between the two posi- from the detriment in uncomplicated testing scenarios tion pose cameras. Parameters A and B can be obtained and improve the efficiency and feasibility of the repair through the forward kinematics relationship of the process. manipulator and the external parameter matrix of the camera. 3.4 Design of robotic arm for crack repair     A semi-autonomous concrete-crack repair robot is x x 6 c −1     y = y ∗ T , implemented to support repairing a wide range of cracks. 6 c (5) z z 6 c A human intelligence-based approach is used to design a convertible terminal tool, which is identified as ideal,     because it refers to most of the repair methods for con- x x 0 6 −1 crete cracks, including the operations, tools, and materi-     y = y ∗ T , 0 6 (6) . 6 als used in the repair process, as shown in Table  3. This z z 0 6 study also investigates a robotic arm control method based on Asea Brown Boveri Ltd. (ABB) robot language After the T value is obtained, the motion coordinates for applications with related execution instructions. of the manipulator can be calculated. In Eqs. (5) and (6), Table 2 Code for repair of concrete cracks No. Standard number Standard name 1 GB 500100‑2010 Code for design of concrete structures 2 GB/T 23660‑2009 Building structure crack split‑stopping tape 3 JC/T 1041‑2007 Epoxy grouting resin for concrete cracks 4 JJF 1334‑2012 Calibration specifications for concrete cracks’ width and depth measuring instruments 5 JGJ 369‑2016 Code for design of pre‑stressed concrete structures 6 JTS 151‑2011 Code for design of concrete structures of port and waterway engineering 7 TB 10092‑2017 Code for design of concrete structures of railway bridge and culvert Chen  et al. AI in Civil Engineering (2022) 1:9 Page 7 of 16 Fig. 6 Design of knowledge base for crack repair Table 3 Functional requirements for repairing concrete cracks No. Operation Tool Material Function 1 Slotting Slotting machine, and polisher / Surface closure Filling sealing 2 Surface cleaning Polisher / Coating sealing 3 Internal cleaning Air blowing device / Pressure grouting 4 Crack closure Scraper Sealant 5 Brush modification Brush roller Coating 6 Paste grouting nozzle Grout mouth Sealant 7 Grouting Seam device Epoxy resin 8 Press polymer mortar Spatula Polymer cement mortar 4 Case study: laboratory and experiments The data are collated in Excel to perform detection and 4.1 C rack feature database high-performance transformation, and then converted to A decision support system for crack repair is developed, a record for the crack feature database. Table 4 shows the which includes a crack feature database and a crack- attribute table structure of the crack features (contain- repair method knowledge base. Table  1 lists some of the ing 200 entries of data). The crack feature data for every keywords used in the crack feature database. The data interface are transformed into the database, as shown in requirements imposed by some recent researches are Fig. 8. summarized into 10 categories and 7 aspects. The char - acteristics of the crack feature database are installed in 8 major ports, used to describe the crack characteristics. 4.2 Knowledge base of the repair methods Text Type (2) consists of the length and depth, compo- The database framework of the knowledge base is estab - nents, location, material, property, width, and direction lished, as presented in Table  2. The repair methods’ data of cracks, defined as Enumeration Type (6). The number requirements can be divided into 10 types and 8 aspects. is in Int Type (1). Each data picture is listed in the table The characteristics of the repair methods are installed in using Blob Type (1). Before the image processing, the 6 major interfaces. Text Type (7) is composed of technol- pixel resolution and image capture should be done. After ogy number, name, range, material, tool, process, and ref- preprocessing and enhancement of grayscale images and erence. Steps and robotic execution code are defined as others images, they can capture the pictures of cracks, as Blob Type (2). The number is Int Type (1). Table  5 shows shown in Fig.  7. The threshold method of segmentation the attribute table structure of the crack repair methods. is used after smoothening the images’ spatial filtering. The repair technology data for each port are converted to And the length and width of cracks are also calculated to the established database. The Excel data table converted evaluate the parameters of images of cracks by analogy. for the database is shown in Fig. 9, which now consists of Chen et al. AI in Civil Engineering (2022) 1:9 Page 8 of 16 Fig. 7 Processing of images: a image acquisition, b binarization and filtering; c noise reduction and edge detection; and d ROI crack line tagging Table 4 Data structure and SQL statements of crack features also be explicitly executable. By using the RobotStu- dio design module, a virtual simulation environment is Column content Column name Data type SQL statement designed, where a robot arm and concrete cracks can be ID No Int CREATE TABLE displayed. First, the robot or robotic platform for repair- FEATURE Crack picture Picture Blob ing concrete cracks can be arranged. Next, the execution (No. INT NOT NULL Component posi‑ Component Enum process can be simulated to detect the movement conflict Picture BLOB NOT tion NULL by using a collision detection module and code compila- Crack location Location Enum Component ENUM tion. It is demonstrated that the simulation should enable (20) Crack material Material Enum the robot to prevent human from being harmed in simple Location ENUM (20) Crack property Feature Enum Material ENUM (20) test scenarios and make the repair process more efficient Crack width Width Enum Feature ENUM (20) and feasible. The semi-autonomous robot arm provides Width ENUM (20) Crack length Length Text software for offline and online programming of robots. It Length CHAR (20) Crack depth Depth Text Depth CHAR (20) implements a methodology to create a BIM model of an Trend ENUM (20)) Crack direction Trend Enum existing physical robot, which is described by taking the example of 6-axis robotic manipulator (ABB IRB 6700- 235). Later, the crack trajectory parameters extracted six types of process data, which will continue to expand with computer vision are compiled to execute the code. in the future. With the locus coordinate parameters, action simulation is exported. The simulation is developed with RobotStu - dio, which can connect to Visual Studio to execute the 4.3 Simula tion system and execution device robot motion and collision detection. The application is Given the expanding ability of robots to take semi- finally integrated with the robotic manipulator, as shown autonomous concrete crack repair, it is imperative that in Fig. 10. mechanisms are put in place to guarantee the safety of Although most multifunction tools of the semi-auton- their behavior and process simulation. Moreover, semi- omous robots available now have a circular flange plate, autonomous robots should be safer; arguably, they should Chen  et al. AI in Civil Engineering (2022) 1:9 Page 9 of 16 Fig. 8 Interface of the crack‑feature database system Table 5 Data structure and SQL statements of the knowledge base for the repair methods Column content Column name Data type SQL statement ID No Int CREATE TABLE TECHNOLOGY (No. INT NOT NULL Technology ID Tech. No Text Tech. No. CHAR (20) Method name Name Text Name CHAR (20) Range CHAR (20) Scope of application Range Text Material CHAR (20) Repair material Material Text Tools CHAR (20) Repair Tools Tools Text Process CHAR (20) Steps BLOB NOT NULL Repair process Process Text Code BLOB NOT NULL Repair steps Steps Blob Reference CHAR (20) Robotic execution code Code Blob Reference Reference Text this study extends the flange plate from different repair simulated experimentally in RobotStudio, finding that tools for repairing concrete cracks, as shown in Fig.  11. the error rate is within 6% of the surface for a number of The repair tools presented are mainly composed of seal - actuation levels. ing, grabbing, blowing, and seaming devices, which use grout nipples and different sealants; and their circular 5 Repair process and discussion flange plate and switch can be actively controlled through The semi-autonomous robotic platform is implemented the pneumatic soft bending actuators embedded along in a laboratory, which includes industrial robot integrat- the edges. Tools are converted according to the require- ing multifunctional tools, workbench, and repairing com- ments of corresponding steps. The multifunctional ponents, as shown in Fig. 12. In the course of experiment, and convertible repair device model is validated and each step has its code to command the robot, including: Chen et al. AI in Civil Engineering (2022) 1:9 Page 10 of 16 Fig. 9 Interface of knowledge base system for the repair methods Fig. 10 Simulation interface for robotic crack repair Chen  et al. AI in Civil Engineering (2022) 1:9 Page 11 of 16 Fig. 11 Multifunctional and convertible repair device sealant extrusion and smearing (Fig.  12a), grout nipple re-decoded using their fitness values. Then, the cor - grabbing (Fig. 12b), grout nipple pasting (Fig. 12c), clean- responding repair operations are implemented by the ing and injection (Fig. 12d), plug grabbing (Fig. 12e), and industrial robot, as shown in Fig. 14. plug installation and curing (Fig.  12f). Functional fix - All the procedures of the crack repair process are ture for grabbing the grout nipple and plug is driven by debugged with RobotStudio software. The main program a MHL2-40D cylinder. Both the motion path and speed of the repair procedure and simulation environment is of each operation of the robot are controlled through the illustrated in the ABB programming language, as shown control procedures, so as to make the pose and the grab- in Fig.  15. At the beginning of the procedure, the code bing speed of the construction adjustable. In addition, script InitAll is defined to initialize all parameters and the coordinates of these moving points that have been empty storage spaces. CheckHomePos is presented to calculated before will be compared with those acquired find appropriate positions of the concrete crack photo from the process simulation of the concrete crack repair pose, while triggering do08 and di00 to send the photo process. The data are utilized when the teaching appa - commands and perform images for analysis. Robot allo- ratus is deployed to control the accuracy of the process. cation is implemented to blow up and clean the cracks, And all the operations are implemented in the ABB pro- stick the nozzle, install the plug, and apply the sealant. gramming language, as elaborated below. It is important All the operations correspond to a di signal, as indicated to control the accuracy during the construction process, in Fig.  16. If di0x (x = 1, 2, 3, 4, 5) = 1, then relevant because the accuracy can even affect the quality of the operations are triggered and executed. component repairing results. Eight points are selected in the trajectory of crack The motion execution encoding of Fig.  13 is explained extraction based on computer vision. Each point of the in Fig.  14. The robot’s operation and execution proce - cracks is measured by the actual coordinates of the robot dures are allocated to each step of the repair method. operation, and the data set so obtained is used sequen- During the process, the robot allocation is decided to tially in calculation. All the points of simulation and real build a new workstation, which is divided into many measurements are listed in Table  6. As presented in the possible tasks in the former decision. In the pro- experiment, the accuracy reaches the last two decimal posed approach, the original task time is evaluated to a points. In this task, Cosine Similarity analysis is imple- greater value. Once the new best task and waiting time mented to estimate the similarity between the two sets of are achieved, all the individuals in the solution will be data, which are calculated in Formula (7). The results are Chen et al. AI in Civil Engineering (2022) 1:9 Page 12 of 16 Fig. 12 Repair process of cracked concrete: a sealant extrusion and smearing; b grout nipple grabbing; c grout nipple pasting; d cleaning and injection; e plug grabbing; and f plug installation and curing Fig. 13 Contrast of photos of concrete crack repair: a before and b after the repair Chen  et al. AI in Civil Engineering (2022) 1:9 Page 13 of 16 presented through the Average Cosine Similarity. As can be seen, its value is very close to 1, showing that the two sets of data are highly correlated with each other. There - fore, it is concluded that the crack trajectory recogni- tion based on computer vision is similar to that of crack repair, proving high recognition accuracy. A · B Similarity = cos(θ ) = ||A||||B|| (7) A × B i i i=1 = . n 2 n 2 (A ) × (B ) i i i=1 i=1 Network planning is conducted to plan and control projects and to identify the best solution by looking for key jobs and key chains. The activity on the edge network of the restoration project is drawn, as shown in Fig.  17, where the key path is marked in red. The time-consuming factors of the entire project are indicated for the nodes in the critical path. All the time values are calculated in the simulation software. However, some of the steps take an excessive amount of time. Therefore, the simulation can be started with the nodes in the critical path, so as to reduce the time consumed. For example, the algorithm can be optimized to identify the cracks with higher effi - ciency and calculate the relevant coordinates. In addi- tion, the algorithm can be greatly improved to reduce project execution time by boosting the technology and Fig. 14 Procedure flow of crack repair process execution steps. Fig. 15 Main program of the crack repair procedure Chen et al. AI in Civil Engineering (2022) 1:9 Page 14 of 16 Fig. 16 Program and IO signal triggering interface Table 6 Cosine similarity analysis of fracture coordinates Model 1 2 3 4 5 6 7 8 Simulation x, mm 71.32 114.31 153.89 265.89 289.76 378.90 400.11 423.56 y, mm 167.11 151.98 149.96 190.23 191.68 152.11 156.11 158.11 z, mm 150.00 150.00 150.00 150.00 150.00 150.00 150.00 150.00 x, mm 70.18 110.71 155.94 134.94 304.58 380.28 405.47 420.79 y, mm 180.73 135.10 132.67 182.18 195.38 143.65 180.34 134.67 z, mm 150.11 151.90 152.11 160.55 148.11 172.75 170.04 144.93 Cosine similarity 0.9992 0.9983 0.9981 0.9469 0.9997 0.9986 0.9986 0.9990 Mean cosine Similarity 0.9923 Fig. 17 Network‑ critical path of repair process robotic arms. The extraction of the coordinates of cracks 6 Conclusions proves to be an efficient way to acquire the trajectory This study explores a computer-vision-guided semi- of cracks. Furthermore, after the feature extraction for autonomous concrete-crack repair method by using Chen  et al. AI in Civil Engineering (2022) 1:9 Page 15 of 16 Baduge, S. K., Thilakarathna, S., & Perera, J. S. (2022). Artificial intelligence and concrete cracks, a crack feature database and a repair smart vision for building and construction 4.0: Machine and deep learn‑ method knowledge base are applied in determining the ing methods and applications. Automation in Construction, 141, 104440. repair process based on human intervention. The method Boumaaza, M., Bezazi, A., Bouchelaghem, H., Benzennache, N., Amziane, S., & Scarpa, F. (2017). Behavior of pre‑ cracked deep beams with composite proposed in this study can be optimized to save pro- materials repairs. Structural Engineering and Mechanics., 63(5), 575–583. ject execution time. One of the major contributions of https:// doi. org/ 10. 12989/ sem. 2017. 63.5. 575 this study is that it can make the system fully autono- Bykov, A., Matveenko, V., Shardakov, I., & Shestakov, A. (2017). Shock wave method for monitoring crack repair processes in reinforced concrete mous. Simulations and experiments demonstrate that structures. Mechanics of Solids., 52(4), 378–383. https:// doi. org/ 10. 3103/ the proposed method can improve the crack detection s0025 65441 70400 33 for concrete structures and enhance the scheme deci- Cha, Y. J., Choi, W., & Büyüköztürk, O. (2017). Deep learning‑based crack dam‑ age detection using convolutional neural networks. Computer-Aided Civil sion-making and construction automation in the repair and Infrastructure Engineering., 32(5), 361–378. https:// doi. org/ 10. 1111/ process. mice. 12263 For future improvement, this system shall be maintain- Cha, Y. J., Choi, W., Suh, G., Mahmoudkhani, S., & Büyüköztürk, O. (2018). Auton‑ omous structural visual inspection using region‑based deep learning for able and checkable, while minimizing or even eliminating detecting multiple damage types. Computer-Aided Civil and Infrastructure human intervention or supervision. Further research is Engineering., 33(9), 731–747. https:// doi. org/ 10. 1111/ mice. 12334 ongoing to design an improved system, so as to perform Chheng, C., & Likitlersuang, S. (2018). Underground excavation behaviour in Bangkok using three‑ dimensional finite element method. Computers and accurate and cost-effective inspection, maintenance, Geotechnics., 95, 68–81. https:// doi. org/ 10. 1016/j. compg eo. 2017. 09. 016 and evaluation of civil structures that can restored and Choi, S.‑ G., Wang, K., Wen, Z., & Chu, J. (2017). Mortar crack repair using micro‑ repaired in a safer, less costly manner. bial induced calcite precipitation method. Cement and Concrete Compos- ites., 83, 209–221. https:// doi. org/ 10. 1016/j. cemco ncomp. 2017. 07. 013 Dan, D. H, & Dan, Q. (2021). Automatic recognition of surface cracks in bridges based on 2D‑APES and mobile machine vision. Measurement. 2021. Author contributions Dana, D., Karl, D., & Agathe, R. (2021). Eec ff t of casting and curing temperature Conceptualization, CZ; Methodology, CZ and RC; Writing‑ original draft on the interfacial bond strength of epoxy bonded concretes. Construction preparation, CZ and RC; Writing‑review and editing, CZ and RC; Supervision and Building Materials, 307, 124328. CZ and L‑LC; Funding acquisition, CZ. All authors have read and agreed to the Harsh, R, Chris, B, & Ali, M. (2020). Defects detection and quantifica‑ published version of the manuscript. tion in dam spillways using robotics and computer vision. 573–582. Huang, H.‑ W., Li, Q.‑ T., & Zhang, D.‑M. (2018). Deep learning based image Funding recognition for crack and leakage defects of metro shield tunnel. Tunnel- This research is partly supported by the National Natural Science Foundation ling and Underground Space Technology., 77, 166–176. https:// doi. org/ 10. of China (Grants nos.71732001 and 71821001) and the major science and 1016/j. tust. 2018. 04. 002 technology project in Hubei Province, China: Key Technologies and Applica‑ Jafarnia, M. S., Saryazdi, M. K., & Moshtaghioun, S. M. (2020). Use of bacteria tions of Intelligent Construction (2020ACA006). for repairing cracks and improving properties of concrete containing limestone powder and natural zeolite. Construction and Building Materials, Availability of data and materials 242, 118059. Some or all data, models, or code that support the findings of this study are Kim, H., Liu, X. M., Ahn, E., Shin, M., Shin, S. W., & Sim, S. (2019). Performance available from the corresponding author upon reasonable request. assessment method for crack repair in concrete using PZT‑based electro ‑ mechanical impedance technique. NDT and E International., 104, 90–97. Declarations https:// doi. org/ 10. 1016/j. ndtei nt Kim, H., Sim, S.‑H., & Spencer, B. F. (2022). Automated concrete crack evalua‑ Competing interests tion using stereo vision with two different focal lengths. Automation in The authors declare no competing interests. Construction, 135, 104136. Kovačević, M. S., Bačić, M., Vukomanović, M., & Cerić, A. (2021). A framework for Author details automatic calculation of life‑ cycle remediation costs of secondary lining National Center of Technology Innovation for Digital Construction, Huazhong cracks. Automation in Construction, 129, 103714. University of Science and Technology, Wuhan 430074, China. School of Civil Liu, L., Yan, R.‑ J., Maruvanchery, V., Kayacan, E., Chen, I.‑M., & Tiong, L. K. (2017). and Hydraulic Engineering, Huazhong University of Science and Technology, Transfer learning on convolutional activation feature as applied to a Wuhan 430074, China. building quality assessment robot. International Journal of Advanced Robotic Systems., 14(3), 1729881417712620. https:// doi. org/ 10. 1177/ Received: 25 April 2022 Revised: 15 July 2022 Accepted: 8 November 17298 81417 712620 Liu, Y. Q., & Yeoh, J. (2021). Robust pixel‑ wise concrete crack segmentation and properties retrieval using image patches. Automation in Construction., 123, 103535. https:// doi. org/ 10. 1016/j. autcon Liu, Z. Q., Cao, Y. W., Wang, Y. Z., & Wang, W. (2019). Computer vision‑based concrete crack detection using U‑net fully convolutional networks. Auto - References mation in Construction., 104, 129–139. https:// doi. org/ 10. 1016/j. autcon Ahmad, S., Elahi, A., Barbhuiya, S., & Farooqi, Y. (2013). Repair of cracks in simply Marazani, T., Madyira, D. M., & Akinlabi, E. T. (2017). Repair of cracks in metals: supported beams using epoxy injection technique. Materials and Struc- A review. Procedia Manufacturing., 8, 673–679. https:// doi. org/ 10. 1016/j. tures., 46(9), 1547–1559. https:// doi. org/ 10. 1617/ s11527‑ 012‑ 9996‑xpromfg. 2017. 02. 086 Algaifi, H. A., Bakar, S. A., Sam, A. R. M., Abidin, A. R. Z., Shahir, S., & AL ‑ Towayti, Mardasi, A. G., Wu, N., & Wu, C. (2018). Experimental study on the crack detec‑ W. A. H. (2018). Numerical modeling for crack self‑healing concrete by tion with optimized spatial wavelet analysis and windowing. Mechanical microbial calcium carbonate. Construction and Building Materials., 189, Systems and Signal Processing, 104, 619–630. 816–824. https:// doi. org/ 10. 1016/j. conbu ildmat. 2018. 08. 218 Menendez, E., Victores, J. G., Montero, R., Martínez, S., & Balaguer, C. (2018). Attard, L., Debono, C. J., Valentino, G., & Di Castro, M. (2018). Vision‑based Tunnel structural inspection and assessment using an autonomous change detection for inspection of tunnel liners. Automation in Construc- robotic system. Automation in Construction, 87, 117–126. https:// doi. org/ tion, 91, 142–154. https:// doi. org/ 10. 1016/j. autcon. 2018. 03. 02010. 1016/j. autcon. 2017. 12. 001 Chen et al. AI in Civil Engineering (2022) 1:9 Page 16 of 16 Mohan, A., & Poobal, S. (2018). Crack detection using image processing: A criti‑ Engineering Management., 2(4), 351–361. https:// doi. org/ 10. 15302/J‑ FEM‑ cal review and analysis. Alexandria Engineering Journal., 57(2), 787–798. 20150 60 https:// doi. org/ 10. 1016/j. aej. 2017. 01. 020 Osman, B. H., Wu, E., Bohai, J., & Abdallah, M. (2017). Repair technique of Publisher’s Note pre‑ cracked reinforced concrete (RC) beams with transverse openings Springer Nature remains neutral with regard to jurisdictional claims in pub‑ strengthened with steel plate under sustained load. Journal of Adhesion lished maps and institutional affiliations. Science and Technology., 31(21), 2360–2379. https:// doi. org/ 10. 1080/ 01694 243. 2017. 13010 73 Pantoja‑Rosero, B. G., Oner, D., & Kozinski, M. (2022). TOPO ‑Loss for continuity‑ preserving crack detection using deep learning. Construction and Building Materials, 344(15), 128264. Ramesh, G., Srinath, D., Ramya, D., & Krishna, B. V. (2021). Repair, rehabilitation and retrofitting of reinforced concrete structures by using non‑ destruc‑ tive testing methods. Materials Today: Proceedings. https:// doi. org/ 10. 1016/j. matpr Riccardo, M., Lorenzo, B., Brunella, B., Cristina, T., John, S., & Iulia, M. (2020). A crack closure system for cementitious composite materials using knotted shape memory polymer (k‑SMP) fibres. Cement and Concrete Composites, 114, 103757. Shamsabadi, E. A., Xu, C., & Rao, A. S. (2022). Vision transformer‑based autono ‑ mous crack detection on asphalt and concrete surfaces. Automation in Construction, 140, 104316. Sutter, B., Lelevé, A., Pham, M. T., Gouin, O., Jupille, N., Kuhn, M., Lulé, P., Michaud, P., & Rémy, P. (2018). A semi‑autonomous mobile robot for bridge inspection. Automation in Construction., 91, 111–119. https:// doi. org/ 10. 1016/j. autcon. 2018. 02. 013 Tedeschi, A., & Benedetto, F. (2017). A real‑time automatic pavement crack and pothole recognition system for mobile Android‑based devices. Advanced Engineering Informatics., 32, 11–25. https:// doi. org/ 10. 1016/j. aei. 2016. 12. Tsiatas, G., & Robinson, J. (2002). Durability evaluation of concrete crack repair systems. Transportation Research Record Journal of the Transportation Research Board., 1795, 82–87. https:// doi. org/ 10. 3141/ 1795‑ 11 Valença, J., Dias‑ da‑ Costa, D., & Júlio, E. (2012). Characterisation of concrete cracking during laboratorial tests using image processing. Construction and Building Materials., 28(1), 607–615. https:// doi. org/ 10. 1016/j. conbu ildmat. 2011. 08. 082 Wang, P., & Huang, H. (2010). Comparison analysis on present image‑based crack detection methods in concrete structures. In Proc., Image and Signal Processing (CISP), 2010 3rd International Congress on, IEEE. 2530–2533. https:// doi. org/ 10. 1109/ CISP. 2010. 56474 96 Wang, S. H. M., Williams, P., Shi, J., & Yang, H. (2015). From green to sustainabil‑ ity—trends in the assessment methods of green buildings. Frontiers of Engineering Management., 2(2), 114–121. https:// doi. org/ 10. 15302/J‑ FEM‑ 20150 18 Xue, Y., & Li, Y. (2018). A fast detection method via region‑based fully convolu‑ tional neural networks for shield tunnel lining defects. Computer-Aided Civil and Infrastructure Engineering., 33(8), 638–654. https:// doi. org/ 10. 1111/ mice. 12367 Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transac‑ tions on pattern analysis and machine intelligence. 22. https:// dwz. cn/ m0obP tcb Zhang, A., Wang, K. C., Fei, Y., Liu, Y., Chen, C., Yang, G., Li, J. Q., Yang, E., & Qiu, S. (2017a). Automated pixel‑level pavement crack detection on 3D Asphalt surfaces with a recurrent neural network. Computer-Aided Civil and Infra- structure Engineering. https:// doi. org/ 10. 1111/ mice. 12409 Zhang, A., Wang, K. C., Li, B., Yang, E., Dai, X., Peng, Y., Fei, Y., Liu, Y., Li, J. Q., & Chen, C. (2017b). Automated pixel‑level pavement crack detection on 3D asphalt surfaces using a deep‑learning network. Computer-Aided Civil and Infrastructure Engineering., 32(10), 805–819. https:// doi. org/ 10. 1111/ mice. 12297 Zhang, Q. Y., Barri, K., Babanajad, S., & Alavi, A. (2020). Real‑time detection of cracks on concrete bridge decks using deep learning in the frequency domain. Journal Pre-Proofs. https:// doi. org/ 10. 1016/j. eng. 2020. 07. 026 Zhou, C., Ding, L., Zhou, Y., & Skibniewski, M. J. (2019). Visibility graph analysis on time series of shield tunneling parameters based on complex network theory. Tunnelling and Underground Space Technology., 89, 10–24. https:// doi. org/ 10. 1016/j. tust. 2019. 03. 019 Zhu, J.‑ W., Zhou, L.‑N., Yu, M.‑ Y., & Zhai, Z. (2015). The development and com‑ parative analysis of engineering project management modes. Frontiers of http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png AI in Civil Engineering Springer Journals

Computer-vision-guided semi-autonomous concrete crack repair for infrastructure maintenance using a robotic arm

Chen, Rui; Zhou, Cheng; Cheng, Li-li
AI in Civil Engineering , Volume 1 (1) – Dec 30, 2022
Download PDF
16 pages

Loading next page...
 
/lp/springer-journals/computer-vision-guided-semi-autonomous-concrete-crack-repair-for-X50MwYnCAM

References (48)

Publisher
Springer Journals
Copyright
Copyright © The Author(s) 2022
eISSN
2730-5392
DOI
10.1007/s43503-022-00007-7
Publisher site
See Article on Publisher Site

Abstract

Engineering inspection and maintenance technologies play an important role in safety, operation, maintenance and management of buildings. In project construction control, supervision of engineering quality is a difficult task. To address such inspection and maintenance issues, this study presents a computer‑ vision‑ guided semi‑autonomous robotic system for identification and repair of concrete cracks, and humans can make repair plans for this system. Con‑ crete cracks are characterized through computer vision, and a crack feature database is established. Furthermore, a trajectory generation and coordinate transformation method is designed to determine the robotic execution coordi‑ nates. In addition, a knowledge base repair method is examined to make appropriate decisions on repair technology for concrete cracks, and a robotic arm is designed for crack repair. Finally, simulations and experiments are conducted, proving the feasibility of the repair method proposed. The result of this study can potentially improve the perfor‑ mance of on‑site automatic concrete crack repair, while addressing such issues as high accident rate, low efficiency, and big loss of skilled workers. Keywords: Computer vision, Concrete crack repair, Robotic construction, Semi‑autonomous, Knowledge base system, Human decision making recognition of concrete cracks (Dan & Dan, 2021). 1 Introduction Surface condition deficiencies are often evaluated by Cracks on the surfaces of concrete engineering structures combining computer-vision detection and surveying are among the earliest indicators of structural deteriora- equipment (Shamsabadi, et  al., 2022). As a result, com- tion. Structures suffer from fatigue stress and cyclic load - puter-vision-based concrete-crack detection is becoming ing (Tedeschi & Benedetto, 2017). As a result of external a type of non-destructive testing technique (Kim et  al., loads, minute cracks on concrete surfaces may produce 2022), with many methodologies used to determine the interconnected passageways, which will worsen the safety existence and location of cracks. Although some stud- of structures (Algaifi et  al., 2018). Thus, civil engineers ies have focused on extracting such basic information as face the challenge of reducing the harm caused by dete- length, width, and depth (Cha et al., 2017), such informa- riorating structures. In this regard, intelligent technology tion is not enough in making decisions on crack repair for unmanned detection and repair is necessary. behavior. As one of the potentially useful technologies, com- Conventional repair materials are classified under puter vision is increasingly implemented in automated various criteria (Tsiatas & Robinson, 1795). Crack epoxy injection is one of the common methods to reconstruct *Correspondence: charleschou@163.com the lost strength properties (Ahmad et  al., 2013). Based National Center of Technology Innovation for Digital Construction, on the properties of the epoxy injected, this type of Huazhong University of Science and Technology, Wuhan 430074, China repair proves effective on new cracks that arise outside Full list of author information is available at the end of the article © The Author(s) 2022. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. Chen et al. AI in Civil Engineering (2022) 1:9 Page 2 of 16 the region of previously repaired cracks. With regard to morphological approach, percolation-based method, and repair techniques for concrete cracks, such information practical technique (Wang et  al., 2010), most of which as length, width, and depth is not enough crack remedy. are used to determine whether cracks exist and where Therefore, researchers have started to pay attention to they are located. Spatial wavelet transformation has also the use of bacteria to repair cracks in concrete (Maedeh been proposed to detect and localize cracks by amplify- & Mehdi et al., 2020; Zhou et al., 2019). A review of the ing weak perturbation signal at crack locations (Mar- existing researches reveals that most of the research on dasi et  al., 2018). Liu et  al. (2019) have used U-net fully concrete-crack repair techniques is mainly on materials, convolutional networks to detect concrete cracks based and few studies have been conducted by using robots. on computer vision. Liu et  al. (2021) have adopted the A robot system for construction quality assessment has integration of Convolutional Neural Network and Active been used to optimize the autonomous visual inspection Contour Model to perform crack segmentation. With function, so as to cut labor cost and improve accuracy deep learning in frequency domain, Zhang et  al. (2020) (Liu et  al., 2017). An automated integration system has try to detect cracks on concrete bridge decks in real time. been developed for remote inspection and repair without direct human intervention. In addition, a semi-autono- mous robotic system has been proposed for inspection 2.2 Repair techniques for concrete cracks and repair of pavements and bridges, while improving the Many existing researches are focused on restora- security of properties and inspectors (Sutter et al., 2018). tive materials. For example, viscosity and mechanical However, these repair platforms are semi-autonomous strength of epoxy materials are explored for repairing and pre-programmed. In contrast, this study will design concrete cracks under low-temperature construction in a computer-vision-guided semi-autonomous robotic sys- winter (Dana et  al., 2021). Given that the quality moni- tem for concrete crack inspection and repair, with the toring of crack fullness and solidification is important help of human decisions. (Bykov et al., 2017), high efficiency of injection processes is pursued by strict observation of the process norms, particularly about the qualitative fullness of crack cavity 2 Related researches and full solidification of injected materials. However, no 2.1 C omputervision‑ ‑ guided recognition of concrete unified test method is available both at home and abroad. cracks u Th s, quality control in crack repair is difficult to imple - Cracks on concrete surfaces of engineering structures ment (Wang et al., 2015; Zhu et al., 2015). are among the earliest indicators of structural deterio- Traditional methods are based on manual repair. How- ration as a result of fatigue and other negative factors ever, computational and autonomous properties play (Mohan & Poobal, 2018), leading to weakened mate- an increasingly important role in improving the knowl- rial integrity. Therefore, observing concrete cracking is edge about the effect of most concrete crack-repair crucial for characterizing the safety of structures. Most techniques on the mechanical performance of repaired existing approaches to crack detection are empirical, components (Marazani et  al., 2017). For self-healing with human’s visual observation becoming the common agents in concrete, cement stone samples can deposit method of crack detection. However, this and other simi- a new layer of calcium carbonate minerals on the sur- lar conventional methods to characterize and inspect face to cover cracks (Boumaaza et  al., 2017; Choi et  al., cracks are time-consuming and error-prone. So an inno- 2017). The crack healing potentials of bacteria have been vative method is necessary. Recently, the developments compared with traditional repair techniques through in digital facility and methods have widened the initial experimental observation (Maedeh et al., 2020). Further- field of concrete crack recognition (Valença et al., 2012). more, researchers in the process optimization field have The purpose of crack inspection may vary, depending invented new technologies and optimized some repair on parameters to be inspected. Crack detection may be operations. In some researches, a piezoelectric patch was delivered based on length, width, depth, and direction bonded on a beam through an external voltage to affect of cracks (Pantoja-Rosero et al., 2022). The major advan - the closure of concrete cracks (Riccardo et  al., 2020). tage of the computer vision technique lies in that it can These techniques can also be used to control the fail - provide more accurate results than traditional manual ure modes and stress distribution around beam chords methods (Shanaka et  al., 2022). Some of the conun- (Osman et  al., 2017). In a systematic method, Kim et  al. drums in computer vision recognition are related with (2019) have evaluated the performance of crack repair different shapes, irregular cracks, and various noises. materials by using PZT-based EMI technology, which By virtue of superior performance of computer vision, can reflect the structural characteristics in evaluating many types of image processing and recognition meth- the repair efficiency over time. Ramesh et al. (2021) have ods have been proposed, such as integrated algorithm, Chen  et al. AI in Civil Engineering (2022) 1:9 Page 3 of 16 deployed non-destructive testing methods to repair and the experience of workers, but also on the level of their refurbish reinforced concrete structures. fatigue. Therefore, this system to be designed shall ensure To detect concrete cracks in engineering, personnel the safety and suitability of the control mode. Being may have to enter hazardous environments. Automatic semi-autonomous, this system can improve the inspec- approaches are the only effective way to support human tion efficiency and accuracy by automatically identify - exploring extreme environments. A review of the existing ing concrete cracks. Tele-operation of robots should be researches on concrete-crack repair techniques finds that considered for the operation process. In addition, a semi- they generally pay attention to materials. supervised computer vision system has been developed in ROBO-SPECT European FP7 project to detect tunnel 2.3 I ntegration of automated recognition and repair diseases (Menendez et al., 2018), and Harsh et al. (2020) for concrete cracks have used robots and computer vision to detect and Several previous studies have attempted to optimize quantify defects in dam spillways. As these repair plat- the crack recognition process for concrete values. For forms are semi-autonomous and pre-programmed, most example, a computer vision system for a train inspec- of the current crack inspection and repair platforms are tion monorail was proposed and installed in the Large focused only on detection. To address this limitation, this Hadron Collider to gather data from various sensors study introduces a computer-vision-guided semi-auton- and capture images by the European Organization for omous robotic system, which is dedicated to concrete Nuclear Research, only purposed for recording data and crack inspection and repair projects that involve decision reducing personnel intervention (Attard et al., 2018). The making by humans. recognition process of engineering concrete cracks has been automated to a certain extent based on deep learn-3 Methodology ing methods (Chheng & Likitlersuang, 2018), including As shown in Fig.  1, four main steps are involved in the Convolutional Neural Network (CNN), Recurrent Neu- computer-vision-guided semi-autonomous concrete ral Network (RNN), and Transfer Learning (2017b; Cha crack repair process using robotic arms. The first step et  al., 2018; Huang et  al., 2018; Xue & Li, 2018; Zhang includes feature acquisition and trajectory extraction, et al., 2017a). Overall, the structural crack repair process purposed to recognize concrete cracks. Feature acquisi- is slow, labor intensive, and subjective so far. To over- tion is performed to determine the length, width, depth, come these working limitations, automatic repair sys- and other measures of cracks through a computer vision tems have to be developed (Kovačević et al. 2021). process, while trajectory coordinates are calculated via An urgent need is to design a fully automated integra- hand-eye calibration. After a knowledge base is created tion system for inspection and reparation, which shall to determine appropriate crack features for the repair enable remote operations, without any need for direct method, the overall repair process will be simulated by human intervention. One selection is to improve the code programming and software operation. The decision automatic behaviors of robots. It has been shown that made by humans based on the knowledge base includes the quality of manual operations depends not only on the establishment of relevant standards and specification Fig. 1 Architecture of the crack repair system Chen et al. AI in Civil Engineering (2022) 1:9 Page 4 of 16 databases, e.g., a crack feature database. Once the simula- position, crack position, crack material, crack properties, tion process is determined, the crack repair process will crack width, crack length, crack depth, and crack direc- be launched, including the design of robotic arms, the tion, as shown in Table 1. execution of repair operation, and the evaluation in the Various features are detected with different tech - next step. Finally, the semi-autonomous concrete-crack niques. For instance, some crack features are acquired repair process is tested and verified under laboratory in infrared, laser, ultrasonic, and various other com- conditions based on computer vision. Each aspect of the puter-vision-based imaging methods. With regard to the repair process is described in detail in Fig. 1. infrastructure for the crack feature acquisition using the computer vision technique, a general workflow of such 3.1 A cquisition of concrete crack features using computer acquisition is shown in Fig. 2. vision The computer-vision-based crack-feature acquisition 3.2 Trajectory extraction and hand–eye calibration involves: detecting cracks, determining locations of crack Extraction of the coordinates of cracks involves image components, and measuring the length, width, and depth pre-processing, denoising, and edge-region processing of cracks. A project requires a large amount of data, for crack trajectory extraction. Region points of a crack which have to be recorded and organized through vari- trajectory are used in thresholding value selection cre- ous methods. A database is needed to store the data on ated from edge points on the basis of human decision the majority of concrete crack repairs. In general, manag- making. A region of interest (ROI) needs to be set to ing the raw data involves an independent database, which insulate the background and the crack trajectory region, can be built with an electronic spreadsheet application. and the minimum distance from the region to the crack Crack features can be divided into eight categories based is not less than 1  cm but not more than 2  cm. A filter - on the following engineering feature values: component ing method should be used to eliminate the useless Table 1 Features and values of crack feature database No. Crack feature Edit mode Feature value 1 Component position Single election Columns, beam, plate, wall, roof frame, cantilevered member, and floor 2 Crack location Single election Cushion layer, surface layer, structural layer, plastering layer, coating, joint, middle and lower part, upper part of support, end, periphery of plate, and bottom of plate 3 Crack material Single election Concrete, masonry, plastering, cement, and asphalt 4 Crack property Single election Splitting, vertical, penetrating, cracking, weathering, axial compression, eccentric compression, and bending 5 Crack width Single election < 0.2 mm, 0.2–0.3 mm, 0.3–0.4 mm, 0.4–0.5 mm, 0.5–1 mm, 1–1.5 mm, and > 0.5 mm 6 Crack length Direct import Text type 7 Crack depth Direct import Text type 8 Crack direction Single election Vertical, horizontal, and oblique Fig. 2 Workflow of crack feature acquisition Chen  et al. AI in Civil Engineering (2022) 1:9 Page 5 of 16 information. To transform a gray image into a binary one, this study implements the thresholding technique, with which a picture can be compartmentalized by using the local threshold. The points of the trajectory are depicted from the points of the area outline. In the marginal area, the sub-pixel type of data contour is gen- erated by utilizing the marginal form. The outer margin of the pixels is utilized as contour points, as shown in Fig. 3. 3.2.1 Camera coordinate calculation Wiping off useless pixels through the filtering technology is crucial to identifying the important data. The thresh - Fig. 4 Pinhole camera model olding measure, in which an image is divided into mul- tiple local thresholds, is implemented in this study to transform a gray image into a binary one. The measure - in Fig. 5. T is the hand–eye relationship and T is the kin- x . 6 ment can be carried out under any normal indoor light ematics positive value. conditions. Camera coordinates are calculated in the pin- hole camera model (Eq.  1), as shown in Fig.  4, where K 0 T = T · T · T , (2) G . 6 x C represents camera intrinsics. The relationship between the polar coordinates of the x u manipulator and the coordinate system of the target object −1 y v = K z , c c (1) (calibration board) T is a fixed value. T represents the G 6 z 1 transformation relationship between the coordinate sys- tem of the 6 degrees of freedom manipulator and the end (forward kinematics) of the manipulator. T represents the 3.2.2 Robot coordinate calculation coordinate transformation relationship between the end of Parameter A can be calculated with the form, and the cal- the robotic arm and the camera. T represents the trans- culation can be completed in the Matlab arm calibration formation of the camera to the coordinate system of the toolbox. Parameter B can be solved by using the Matlab target object. camera calibration toolbox and Zhang Zhengyou Camera Calibration Method (Zhang, 2000). The relative positional T · T · T = T · T · T , 6 x C 6 x C (3) 1 1 2 2 relationship between a camera and the sixth axis of the manipulator must be changed by three positions, as shown Fig. 3 Workflow of trajectory extraction and hand ‑ eye calibration Chen et al. AI in Civil Engineering (2022) 1:9 Page 6 of 16 T indicates the hand-eye relationship and T indicates x . 6 the kinematics positive solution of the manipulator. 3.3 Design of knowledge base and simulation of crack repair A knowledge base is designed for the crack repair method driven by meta-knowledge. This inference process will screen data from the crack feature, repair standard, and robot code databases and contribute to the data ware- house system, as presented in Table  2. Furthermore, the expression of the knowledge base is constantly adjusted on the basis of the robot execution characteristics, so as to accurately select the alternatives containing the tools, materials, standards, and related information. The process of requirement analysis, database design, data coding, item importing, data verification and Fig. 5 Schematic of T value solution screening, back-up, and exporting for analysis is illus- trated in Fig.  6. With the RobotStudio design module, a virtual simulation environment is designed, where a robot arm and concrete cracks can be displayed. First, the −1 −1 T · T · T = T · T · T , 6 x x C (4) 1 2 6 C 2 1 robot or robotic platform to repair the concrete cracks can be deployed. Second, the execution process can be Equation  (4) can also be expressed as A · X  =  X · B , simulated to detect the movement conflict by using a −1 where A = T · T represents the relative relationship 6 1 2 collision detection module and code compilation. The −1 between the two position postures, and B = T · T 2 C 1 simulation enables the robot to hold back human beings represents the relative relationship between the two posi- from the detriment in uncomplicated testing scenarios tion pose cameras. Parameters A and B can be obtained and improve the efficiency and feasibility of the repair through the forward kinematics relationship of the process. manipulator and the external parameter matrix of the camera. 3.4 Design of robotic arm for crack repair     A semi-autonomous concrete-crack repair robot is x x 6 c −1     y = y ∗ T , implemented to support repairing a wide range of cracks. 6 c (5) z z 6 c A human intelligence-based approach is used to design a convertible terminal tool, which is identified as ideal,     because it refers to most of the repair methods for con- x x 0 6 −1 crete cracks, including the operations, tools, and materi-     y = y ∗ T , 0 6 (6) . 6 als used in the repair process, as shown in Table  3. This z z 0 6 study also investigates a robotic arm control method based on Asea Brown Boveri Ltd. (ABB) robot language After the T value is obtained, the motion coordinates for applications with related execution instructions. of the manipulator can be calculated. In Eqs. (5) and (6), Table 2 Code for repair of concrete cracks No. Standard number Standard name 1 GB 500100‑2010 Code for design of concrete structures 2 GB/T 23660‑2009 Building structure crack split‑stopping tape 3 JC/T 1041‑2007 Epoxy grouting resin for concrete cracks 4 JJF 1334‑2012 Calibration specifications for concrete cracks’ width and depth measuring instruments 5 JGJ 369‑2016 Code for design of pre‑stressed concrete structures 6 JTS 151‑2011 Code for design of concrete structures of port and waterway engineering 7 TB 10092‑2017 Code for design of concrete structures of railway bridge and culvert Chen  et al. AI in Civil Engineering (2022) 1:9 Page 7 of 16 Fig. 6 Design of knowledge base for crack repair Table 3 Functional requirements for repairing concrete cracks No. Operation Tool Material Function 1 Slotting Slotting machine, and polisher / Surface closure Filling sealing 2 Surface cleaning Polisher / Coating sealing 3 Internal cleaning Air blowing device / Pressure grouting 4 Crack closure Scraper Sealant 5 Brush modification Brush roller Coating 6 Paste grouting nozzle Grout mouth Sealant 7 Grouting Seam device Epoxy resin 8 Press polymer mortar Spatula Polymer cement mortar 4 Case study: laboratory and experiments The data are collated in Excel to perform detection and 4.1 C rack feature database high-performance transformation, and then converted to A decision support system for crack repair is developed, a record for the crack feature database. Table 4 shows the which includes a crack feature database and a crack- attribute table structure of the crack features (contain- repair method knowledge base. Table  1 lists some of the ing 200 entries of data). The crack feature data for every keywords used in the crack feature database. The data interface are transformed into the database, as shown in requirements imposed by some recent researches are Fig. 8. summarized into 10 categories and 7 aspects. The char - acteristics of the crack feature database are installed in 8 major ports, used to describe the crack characteristics. 4.2 Knowledge base of the repair methods Text Type (2) consists of the length and depth, compo- The database framework of the knowledge base is estab - nents, location, material, property, width, and direction lished, as presented in Table  2. The repair methods’ data of cracks, defined as Enumeration Type (6). The number requirements can be divided into 10 types and 8 aspects. is in Int Type (1). Each data picture is listed in the table The characteristics of the repair methods are installed in using Blob Type (1). Before the image processing, the 6 major interfaces. Text Type (7) is composed of technol- pixel resolution and image capture should be done. After ogy number, name, range, material, tool, process, and ref- preprocessing and enhancement of grayscale images and erence. Steps and robotic execution code are defined as others images, they can capture the pictures of cracks, as Blob Type (2). The number is Int Type (1). Table  5 shows shown in Fig.  7. The threshold method of segmentation the attribute table structure of the crack repair methods. is used after smoothening the images’ spatial filtering. The repair technology data for each port are converted to And the length and width of cracks are also calculated to the established database. The Excel data table converted evaluate the parameters of images of cracks by analogy. for the database is shown in Fig. 9, which now consists of Chen et al. AI in Civil Engineering (2022) 1:9 Page 8 of 16 Fig. 7 Processing of images: a image acquisition, b binarization and filtering; c noise reduction and edge detection; and d ROI crack line tagging Table 4 Data structure and SQL statements of crack features also be explicitly executable. By using the RobotStu- dio design module, a virtual simulation environment is Column content Column name Data type SQL statement designed, where a robot arm and concrete cracks can be ID No Int CREATE TABLE displayed. First, the robot or robotic platform for repair- FEATURE Crack picture Picture Blob ing concrete cracks can be arranged. Next, the execution (No. INT NOT NULL Component posi‑ Component Enum process can be simulated to detect the movement conflict Picture BLOB NOT tion NULL by using a collision detection module and code compila- Crack location Location Enum Component ENUM tion. It is demonstrated that the simulation should enable (20) Crack material Material Enum the robot to prevent human from being harmed in simple Location ENUM (20) Crack property Feature Enum Material ENUM (20) test scenarios and make the repair process more efficient Crack width Width Enum Feature ENUM (20) and feasible. The semi-autonomous robot arm provides Width ENUM (20) Crack length Length Text software for offline and online programming of robots. It Length CHAR (20) Crack depth Depth Text Depth CHAR (20) implements a methodology to create a BIM model of an Trend ENUM (20)) Crack direction Trend Enum existing physical robot, which is described by taking the example of 6-axis robotic manipulator (ABB IRB 6700- 235). Later, the crack trajectory parameters extracted six types of process data, which will continue to expand with computer vision are compiled to execute the code. in the future. With the locus coordinate parameters, action simulation is exported. The simulation is developed with RobotStu - dio, which can connect to Visual Studio to execute the 4.3 Simula tion system and execution device robot motion and collision detection. The application is Given the expanding ability of robots to take semi- finally integrated with the robotic manipulator, as shown autonomous concrete crack repair, it is imperative that in Fig. 10. mechanisms are put in place to guarantee the safety of Although most multifunction tools of the semi-auton- their behavior and process simulation. Moreover, semi- omous robots available now have a circular flange plate, autonomous robots should be safer; arguably, they should Chen  et al. AI in Civil Engineering (2022) 1:9 Page 9 of 16 Fig. 8 Interface of the crack‑feature database system Table 5 Data structure and SQL statements of the knowledge base for the repair methods Column content Column name Data type SQL statement ID No Int CREATE TABLE TECHNOLOGY (No. INT NOT NULL Technology ID Tech. No Text Tech. No. CHAR (20) Method name Name Text Name CHAR (20) Range CHAR (20) Scope of application Range Text Material CHAR (20) Repair material Material Text Tools CHAR (20) Repair Tools Tools Text Process CHAR (20) Steps BLOB NOT NULL Repair process Process Text Code BLOB NOT NULL Repair steps Steps Blob Reference CHAR (20) Robotic execution code Code Blob Reference Reference Text this study extends the flange plate from different repair simulated experimentally in RobotStudio, finding that tools for repairing concrete cracks, as shown in Fig.  11. the error rate is within 6% of the surface for a number of The repair tools presented are mainly composed of seal - actuation levels. ing, grabbing, blowing, and seaming devices, which use grout nipples and different sealants; and their circular 5 Repair process and discussion flange plate and switch can be actively controlled through The semi-autonomous robotic platform is implemented the pneumatic soft bending actuators embedded along in a laboratory, which includes industrial robot integrat- the edges. Tools are converted according to the require- ing multifunctional tools, workbench, and repairing com- ments of corresponding steps. The multifunctional ponents, as shown in Fig. 12. In the course of experiment, and convertible repair device model is validated and each step has its code to command the robot, including: Chen et al. AI in Civil Engineering (2022) 1:9 Page 10 of 16 Fig. 9 Interface of knowledge base system for the repair methods Fig. 10 Simulation interface for robotic crack repair Chen  et al. AI in Civil Engineering (2022) 1:9 Page 11 of 16 Fig. 11 Multifunctional and convertible repair device sealant extrusion and smearing (Fig.  12a), grout nipple re-decoded using their fitness values. Then, the cor - grabbing (Fig. 12b), grout nipple pasting (Fig. 12c), clean- responding repair operations are implemented by the ing and injection (Fig. 12d), plug grabbing (Fig. 12e), and industrial robot, as shown in Fig. 14. plug installation and curing (Fig.  12f). Functional fix - All the procedures of the crack repair process are ture for grabbing the grout nipple and plug is driven by debugged with RobotStudio software. The main program a MHL2-40D cylinder. Both the motion path and speed of the repair procedure and simulation environment is of each operation of the robot are controlled through the illustrated in the ABB programming language, as shown control procedures, so as to make the pose and the grab- in Fig.  15. At the beginning of the procedure, the code bing speed of the construction adjustable. In addition, script InitAll is defined to initialize all parameters and the coordinates of these moving points that have been empty storage spaces. CheckHomePos is presented to calculated before will be compared with those acquired find appropriate positions of the concrete crack photo from the process simulation of the concrete crack repair pose, while triggering do08 and di00 to send the photo process. The data are utilized when the teaching appa - commands and perform images for analysis. Robot allo- ratus is deployed to control the accuracy of the process. cation is implemented to blow up and clean the cracks, And all the operations are implemented in the ABB pro- stick the nozzle, install the plug, and apply the sealant. gramming language, as elaborated below. It is important All the operations correspond to a di signal, as indicated to control the accuracy during the construction process, in Fig.  16. If di0x (x = 1, 2, 3, 4, 5) = 1, then relevant because the accuracy can even affect the quality of the operations are triggered and executed. component repairing results. Eight points are selected in the trajectory of crack The motion execution encoding of Fig.  13 is explained extraction based on computer vision. Each point of the in Fig.  14. The robot’s operation and execution proce - cracks is measured by the actual coordinates of the robot dures are allocated to each step of the repair method. operation, and the data set so obtained is used sequen- During the process, the robot allocation is decided to tially in calculation. All the points of simulation and real build a new workstation, which is divided into many measurements are listed in Table  6. As presented in the possible tasks in the former decision. In the pro- experiment, the accuracy reaches the last two decimal posed approach, the original task time is evaluated to a points. In this task, Cosine Similarity analysis is imple- greater value. Once the new best task and waiting time mented to estimate the similarity between the two sets of are achieved, all the individuals in the solution will be data, which are calculated in Formula (7). The results are Chen et al. AI in Civil Engineering (2022) 1:9 Page 12 of 16 Fig. 12 Repair process of cracked concrete: a sealant extrusion and smearing; b grout nipple grabbing; c grout nipple pasting; d cleaning and injection; e plug grabbing; and f plug installation and curing Fig. 13 Contrast of photos of concrete crack repair: a before and b after the repair Chen  et al. AI in Civil Engineering (2022) 1:9 Page 13 of 16 presented through the Average Cosine Similarity. As can be seen, its value is very close to 1, showing that the two sets of data are highly correlated with each other. There - fore, it is concluded that the crack trajectory recogni- tion based on computer vision is similar to that of crack repair, proving high recognition accuracy. A · B Similarity = cos(θ ) = ||A||||B|| (7) A × B i i i=1 = . n 2 n 2 (A ) × (B ) i i i=1 i=1 Network planning is conducted to plan and control projects and to identify the best solution by looking for key jobs and key chains. The activity on the edge network of the restoration project is drawn, as shown in Fig.  17, where the key path is marked in red. The time-consuming factors of the entire project are indicated for the nodes in the critical path. All the time values are calculated in the simulation software. However, some of the steps take an excessive amount of time. Therefore, the simulation can be started with the nodes in the critical path, so as to reduce the time consumed. For example, the algorithm can be optimized to identify the cracks with higher effi - ciency and calculate the relevant coordinates. In addi- tion, the algorithm can be greatly improved to reduce project execution time by boosting the technology and Fig. 14 Procedure flow of crack repair process execution steps. Fig. 15 Main program of the crack repair procedure Chen et al. AI in Civil Engineering (2022) 1:9 Page 14 of 16 Fig. 16 Program and IO signal triggering interface Table 6 Cosine similarity analysis of fracture coordinates Model 1 2 3 4 5 6 7 8 Simulation x, mm 71.32 114.31 153.89 265.89 289.76 378.90 400.11 423.56 y, mm 167.11 151.98 149.96 190.23 191.68 152.11 156.11 158.11 z, mm 150.00 150.00 150.00 150.00 150.00 150.00 150.00 150.00 x, mm 70.18 110.71 155.94 134.94 304.58 380.28 405.47 420.79 y, mm 180.73 135.10 132.67 182.18 195.38 143.65 180.34 134.67 z, mm 150.11 151.90 152.11 160.55 148.11 172.75 170.04 144.93 Cosine similarity 0.9992 0.9983 0.9981 0.9469 0.9997 0.9986 0.9986 0.9990 Mean cosine Similarity 0.9923 Fig. 17 Network‑ critical path of repair process robotic arms. The extraction of the coordinates of cracks 6 Conclusions proves to be an efficient way to acquire the trajectory This study explores a computer-vision-guided semi- of cracks. Furthermore, after the feature extraction for autonomous concrete-crack repair method by using Chen  et al. AI in Civil Engineering (2022) 1:9 Page 15 of 16 Baduge, S. K., Thilakarathna, S., & Perera, J. S. (2022). Artificial intelligence and concrete cracks, a crack feature database and a repair smart vision for building and construction 4.0: Machine and deep learn‑ method knowledge base are applied in determining the ing methods and applications. Automation in Construction, 141, 104440. repair process based on human intervention. The method Boumaaza, M., Bezazi, A., Bouchelaghem, H., Benzennache, N., Amziane, S., & Scarpa, F. (2017). Behavior of pre‑ cracked deep beams with composite proposed in this study can be optimized to save pro- materials repairs. Structural Engineering and Mechanics., 63(5), 575–583. ject execution time. One of the major contributions of https:// doi. org/ 10. 12989/ sem. 2017. 63.5. 575 this study is that it can make the system fully autono- Bykov, A., Matveenko, V., Shardakov, I., & Shestakov, A. (2017). Shock wave method for monitoring crack repair processes in reinforced concrete mous. Simulations and experiments demonstrate that structures. Mechanics of Solids., 52(4), 378–383. https:// doi. org/ 10. 3103/ the proposed method can improve the crack detection s0025 65441 70400 33 for concrete structures and enhance the scheme deci- Cha, Y. J., Choi, W., & Büyüköztürk, O. (2017). Deep learning‑based crack dam‑ age detection using convolutional neural networks. Computer-Aided Civil sion-making and construction automation in the repair and Infrastructure Engineering., 32(5), 361–378. https:// doi. org/ 10. 1111/ process. mice. 12263 For future improvement, this system shall be maintain- Cha, Y. J., Choi, W., Suh, G., Mahmoudkhani, S., & Büyüköztürk, O. (2018). Auton‑ omous structural visual inspection using region‑based deep learning for able and checkable, while minimizing or even eliminating detecting multiple damage types. Computer-Aided Civil and Infrastructure human intervention or supervision. Further research is Engineering., 33(9), 731–747. https:// doi. org/ 10. 1111/ mice. 12334 ongoing to design an improved system, so as to perform Chheng, C., & Likitlersuang, S. (2018). Underground excavation behaviour in Bangkok using three‑ dimensional finite element method. Computers and accurate and cost-effective inspection, maintenance, Geotechnics., 95, 68–81. https:// doi. org/ 10. 1016/j. compg eo. 2017. 09. 016 and evaluation of civil structures that can restored and Choi, S.‑ G., Wang, K., Wen, Z., & Chu, J. (2017). Mortar crack repair using micro‑ repaired in a safer, less costly manner. bial induced calcite precipitation method. Cement and Concrete Compos- ites., 83, 209–221. https:// doi. org/ 10. 1016/j. cemco ncomp. 2017. 07. 013 Dan, D. H, & Dan, Q. (2021). Automatic recognition of surface cracks in bridges based on 2D‑APES and mobile machine vision. Measurement. 2021. Author contributions Dana, D., Karl, D., & Agathe, R. (2021). Eec ff t of casting and curing temperature Conceptualization, CZ; Methodology, CZ and RC; Writing‑ original draft on the interfacial bond strength of epoxy bonded concretes. Construction preparation, CZ and RC; Writing‑review and editing, CZ and RC; Supervision and Building Materials, 307, 124328. CZ and L‑LC; Funding acquisition, CZ. All authors have read and agreed to the Harsh, R, Chris, B, & Ali, M. (2020). Defects detection and quantifica‑ published version of the manuscript. tion in dam spillways using robotics and computer vision. 573–582. Huang, H.‑ W., Li, Q.‑ T., & Zhang, D.‑M. (2018). Deep learning based image Funding recognition for crack and leakage defects of metro shield tunnel. Tunnel- This research is partly supported by the National Natural Science Foundation ling and Underground Space Technology., 77, 166–176. https:// doi. org/ 10. of China (Grants nos.71732001 and 71821001) and the major science and 1016/j. tust. 2018. 04. 002 technology project in Hubei Province, China: Key Technologies and Applica‑ Jafarnia, M. S., Saryazdi, M. K., & Moshtaghioun, S. M. (2020). Use of bacteria tions of Intelligent Construction (2020ACA006). for repairing cracks and improving properties of concrete containing limestone powder and natural zeolite. Construction and Building Materials, Availability of data and materials 242, 118059. Some or all data, models, or code that support the findings of this study are Kim, H., Liu, X. M., Ahn, E., Shin, M., Shin, S. W., & Sim, S. (2019). Performance available from the corresponding author upon reasonable request. assessment method for crack repair in concrete using PZT‑based electro ‑ mechanical impedance technique. NDT and E International., 104, 90–97. Declarations https:// doi. org/ 10. 1016/j. ndtei nt Kim, H., Sim, S.‑H., & Spencer, B. F. (2022). Automated concrete crack evalua‑ Competing interests tion using stereo vision with two different focal lengths. Automation in The authors declare no competing interests. Construction, 135, 104136. Kovačević, M. S., Bačić, M., Vukomanović, M., & Cerić, A. (2021). A framework for Author details automatic calculation of life‑ cycle remediation costs of secondary lining National Center of Technology Innovation for Digital Construction, Huazhong cracks. Automation in Construction, 129, 103714. University of Science and Technology, Wuhan 430074, China. School of Civil Liu, L., Yan, R.‑ J., Maruvanchery, V., Kayacan, E., Chen, I.‑M., & Tiong, L. K. (2017). and Hydraulic Engineering, Huazhong University of Science and Technology, Transfer learning on convolutional activation feature as applied to a Wuhan 430074, China. building quality assessment robot. International Journal of Advanced Robotic Systems., 14(3), 1729881417712620. https:// doi. org/ 10. 1177/ Received: 25 April 2022 Revised: 15 July 2022 Accepted: 8 November 17298 81417 712620 Liu, Y. Q., & Yeoh, J. (2021). Robust pixel‑ wise concrete crack segmentation and properties retrieval using image patches. Automation in Construction., 123, 103535. https:// doi. org/ 10. 1016/j. autcon Liu, Z. Q., Cao, Y. W., Wang, Y. Z., & Wang, W. (2019). Computer vision‑based concrete crack detection using U‑net fully convolutional networks. Auto - References mation in Construction., 104, 129–139. https:// doi. org/ 10. 1016/j. autcon Ahmad, S., Elahi, A., Barbhuiya, S., & Farooqi, Y. (2013). Repair of cracks in simply Marazani, T., Madyira, D. M., & Akinlabi, E. T. (2017). Repair of cracks in metals: supported beams using epoxy injection technique. Materials and Struc- A review. Procedia Manufacturing., 8, 673–679. https:// doi. org/ 10. 1016/j. tures., 46(9), 1547–1559. https:// doi. org/ 10. 1617/ s11527‑ 012‑ 9996‑xpromfg. 2017. 02. 086 Algaifi, H. A., Bakar, S. A., Sam, A. R. M., Abidin, A. R. Z., Shahir, S., & AL ‑ Towayti, Mardasi, A. G., Wu, N., & Wu, C. (2018). Experimental study on the crack detec‑ W. A. H. (2018). Numerical modeling for crack self‑healing concrete by tion with optimized spatial wavelet analysis and windowing. Mechanical microbial calcium carbonate. Construction and Building Materials., 189, Systems and Signal Processing, 104, 619–630. 816–824. https:// doi. org/ 10. 1016/j. conbu ildmat. 2018. 08. 218 Menendez, E., Victores, J. G., Montero, R., Martínez, S., & Balaguer, C. (2018). Attard, L., Debono, C. J., Valentino, G., & Di Castro, M. (2018). Vision‑based Tunnel structural inspection and assessment using an autonomous change detection for inspection of tunnel liners. Automation in Construc- robotic system. Automation in Construction, 87, 117–126. https:// doi. org/ tion, 91, 142–154. https:// doi. org/ 10. 1016/j. autcon. 2018. 03. 02010. 1016/j. autcon. 2017. 12. 001 Chen et al. AI in Civil Engineering (2022) 1:9 Page 16 of 16 Mohan, A., & Poobal, S. (2018). Crack detection using image processing: A criti‑ Engineering Management., 2(4), 351–361. https:// doi. org/ 10. 15302/J‑ FEM‑ cal review and analysis. Alexandria Engineering Journal., 57(2), 787–798. 20150 60 https:// doi. org/ 10. 1016/j. aej. 2017. 01. 020 Osman, B. H., Wu, E., Bohai, J., & Abdallah, M. (2017). Repair technique of Publisher’s Note pre‑ cracked reinforced concrete (RC) beams with transverse openings Springer Nature remains neutral with regard to jurisdictional claims in pub‑ strengthened with steel plate under sustained load. Journal of Adhesion lished maps and institutional affiliations. Science and Technology., 31(21), 2360–2379. https:// doi. org/ 10. 1080/ 01694 243. 2017. 13010 73 Pantoja‑Rosero, B. G., Oner, D., & Kozinski, M. (2022). TOPO ‑Loss for continuity‑ preserving crack detection using deep learning. Construction and Building Materials, 344(15), 128264. Ramesh, G., Srinath, D., Ramya, D., & Krishna, B. V. (2021). Repair, rehabilitation and retrofitting of reinforced concrete structures by using non‑ destruc‑ tive testing methods. Materials Today: Proceedings. https:// doi. org/ 10. 1016/j. matpr Riccardo, M., Lorenzo, B., Brunella, B., Cristina, T., John, S., & Iulia, M. (2020). A crack closure system for cementitious composite materials using knotted shape memory polymer (k‑SMP) fibres. Cement and Concrete Composites, 114, 103757. Shamsabadi, E. A., Xu, C., & Rao, A. S. (2022). Vision transformer‑based autono ‑ mous crack detection on asphalt and concrete surfaces. Automation in Construction, 140, 104316. Sutter, B., Lelevé, A., Pham, M. T., Gouin, O., Jupille, N., Kuhn, M., Lulé, P., Michaud, P., & Rémy, P. (2018). A semi‑autonomous mobile robot for bridge inspection. Automation in Construction., 91, 111–119. https:// doi. org/ 10. 1016/j. autcon. 2018. 02. 013 Tedeschi, A., & Benedetto, F. (2017). A real‑time automatic pavement crack and pothole recognition system for mobile Android‑based devices. Advanced Engineering Informatics., 32, 11–25. https:// doi. org/ 10. 1016/j. aei. 2016. 12. Tsiatas, G., & Robinson, J. (2002). Durability evaluation of concrete crack repair systems. Transportation Research Record Journal of the Transportation Research Board., 1795, 82–87. https:// doi. org/ 10. 3141/ 1795‑ 11 Valença, J., Dias‑ da‑ Costa, D., & Júlio, E. (2012). Characterisation of concrete cracking during laboratorial tests using image processing. Construction and Building Materials., 28(1), 607–615. https:// doi. org/ 10. 1016/j. conbu ildmat. 2011. 08. 082 Wang, P., & Huang, H. (2010). Comparison analysis on present image‑based crack detection methods in concrete structures. In Proc., Image and Signal Processing (CISP), 2010 3rd International Congress on, IEEE. 2530–2533. https:// doi. org/ 10. 1109/ CISP. 2010. 56474 96 Wang, S. H. M., Williams, P., Shi, J., & Yang, H. (2015). From green to sustainabil‑ ity—trends in the assessment methods of green buildings. Frontiers of Engineering Management., 2(2), 114–121. https:// doi. org/ 10. 15302/J‑ FEM‑ 20150 18 Xue, Y., & Li, Y. (2018). A fast detection method via region‑based fully convolu‑ tional neural networks for shield tunnel lining defects. Computer-Aided Civil and Infrastructure Engineering., 33(8), 638–654. https:// doi. org/ 10. 1111/ mice. 12367 Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transac‑ tions on pattern analysis and machine intelligence. 22. https:// dwz. cn/ m0obP tcb Zhang, A., Wang, K. C., Fei, Y., Liu, Y., Chen, C., Yang, G., Li, J. Q., Yang, E., & Qiu, S. (2017a). Automated pixel‑level pavement crack detection on 3D Asphalt surfaces with a recurrent neural network. Computer-Aided Civil and Infra- structure Engineering. https:// doi. org/ 10. 1111/ mice. 12409 Zhang, A., Wang, K. C., Li, B., Yang, E., Dai, X., Peng, Y., Fei, Y., Liu, Y., Li, J. Q., & Chen, C. (2017b). Automated pixel‑level pavement crack detection on 3D asphalt surfaces using a deep‑learning network. Computer-Aided Civil and Infrastructure Engineering., 32(10), 805–819. https:// doi. org/ 10. 1111/ mice. 12297 Zhang, Q. Y., Barri, K., Babanajad, S., & Alavi, A. (2020). Real‑time detection of cracks on concrete bridge decks using deep learning in the frequency domain. Journal Pre-Proofs. https:// doi. org/ 10. 1016/j. eng. 2020. 07. 026 Zhou, C., Ding, L., Zhou, Y., & Skibniewski, M. J. (2019). Visibility graph analysis on time series of shield tunneling parameters based on complex network theory. Tunnelling and Underground Space Technology., 89, 10–24. https:// doi. org/ 10. 1016/j. tust. 2019. 03. 019 Zhu, J.‑ W., Zhou, L.‑N., Yu, M.‑ Y., & Zhai, Z. (2015). The development and com‑ parative analysis of engineering project management modes. Frontiers of

Journal

AI in Civil Engineering – Springer Journals

Published: Dec 30, 2022

Keywords: Computer vision; Concrete crack repair; Robotic construction; Semi-autonomous; Knowledge base system; Human decision making

There are no references for this article.