[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Wheel Loader Scooping Controller Using Deep Reinforcement Learning; DOI: 10.1109/access.2021.3056625; license: cc-by-sa.

[Assistant]
Wang, ‘‘Intelligent excavator control system for lunar mining system,’’ J. Aerosp. Eng., vol. 8, no. 1, pp. 16–24, Jan. 1995. [19] A. A. Dobson, J. A. Marshall, and J. Larsson, ‘‘Admittance control for robotic loading: Underground ﬁeld trials with an LHD,’’ in Field and Service Robotics. Toronto, ON, Canada, 2015. [20] D. Jud, G. Hottiger, P. Leemann, and M. Hutter, ‘‘Planning and control for autonomous excavation,’’ IEEE Robot. Autom. Lett., vol. 2, no. 4, pp. 2151–2158, Oct. 2017. [21] G. J. Maeda, I. R. Manchester, and D. C. Rye, ‘‘Combined ILC and disturbance observer for the rejection of near-repetitive disturbances, with application to excavation,’’ IEEE Trans. Control Syst. Technol., vol. 23, no. 5, pp. 1754–1769, Sep. 2015. VOLUME 9, 2021 24153
[page 10] O. Azulay, A. Shapiro: Wheel Loader Scooping Controller Using Deep Reinforcement Learning [22] S. Singh and H. Cannon, ‘‘Multi-resolution planning for earthmoving,’’ in Proc. IEEE Int. Conf. Robot. Automat., May 1998, pp. 121–126. [23] O. Kanai, H. Osumi, S. Sarata, and M. Kurisu, ‘‘Autonomous scooping of a rock pile by a wheel loader using disturbance observer,’’ in Proc. 23rd Int. Symp. Autom. Robot. Construct., Oct. 2006, pp. 3–5. [24] S. Dadhich, U. Bodin, F. Sandin, and U. Andersson, ‘‘Machine learning approach to automatic bucket loading,’’ in Proc. 24th Medit. Conf. Control Autom. (MED), Jun. 2016, pp. 1260–1265. [25] E. Halbach, J. Kamarainen, and R. Ghabcheloo, ‘‘Neural network pile loading controller trained by demonstration,’’ in Proc. Int. Conf. Robot. Autom. (ICRA), May 2019, pp. 980–986. [26] R. Jonschkowski and O. Brock, ‘‘Learning state representations with robotic priors,’’ Auto. Robots, vol. 39, no. 3, pp. 407–428, Oct. 2015. [27] T. Lesort, N. Díaz-Rodríguez, J.-F. Goudou, and D. Filliat, ‘‘State rep- resentation learning for control: An overview,’’ Neural Netw., vol. 108, pp. 379–392, Dec. 2018. OSHER AZULAY received the B.Sc. and M.Sc. degrees (Hons.) from the Department of Mechan- ical Engineering, Ben-Gurion University of the Negev, Israel, in 2019 and 2020, respectively. He is currently pursuing the Ph.D. degree with the School of Mechanical Engineering, Tel Aviv University. His current research interests include robotic manipulators and reinforcement learn- ing. He had been awarded numerous prizes for academic and research excellence, including certiﬁcates of achievement 2017–2018 and 2018–2019 and an Excellence Scholarship from the Mechanical Engineering Department, Ben-Gurion Uni- versity of the Negev. AMIR SHAPIRO (Member, IEEE) received the B.Sc., M.Sc., and Ph.D. degrees in mechanical engineering from the Technion, Israel Institute of Technology, Haifa, in 1997, 2000, and 2004, respectively. From 2005 to 2006, he was a Postdoc- toral Fellow with the Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA. He is cur- rently an Associate Professor and the Director of the Robotics Laboratory, Department of Mechan- ical Engineering, Ben-Gurion University of the Negev, Be’er Sheva, Israel. His research interests include locomotion of multi-limbed mechanisms in unstructured complex environments, motion planning algorithms for multi-limbed robots, robot grasping-design, control, and stability analysis, climbing robots, snake-like robots, multi-robot on-line motion planning, and agricultural robotics. 24154 VOLUME 9, 2021


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Intelligent Classification of Surrounding Rock of Tunnel Based on 10 Machine Learning Algorithms; DOI: 10.3390/app12052656; license: cc-by-sa.

[Assistant]
Field collection of the drilling parameters. (a) Blasting hole drilling in the tunnel face; (b) the cemented carbide bit. The area of the tunnel face is about 150 m2, containing about 200~300 blasting holes. The value of each drilling parameter for each sample is the average of all the drilling holes in the tunnel face. The typical layout of the blasting hole is shown in Figure 5. The grade of the surrounding rock in this study is speciﬁed in the current Code for Design of Railway Tunnel (TB10003-2016). According to the degree of hardness, integrity, groundwater state, crustal stress state, and major weak structural surface, the surrounding rocks are divided into Grades I~VI according to the quality. The approximate correspondence between the surrounding rock grade in this study and the indices of other classiﬁcation methods, such as the Q method, RMR method, and BQ method, is shown in Table 1 [55]. Table 1. Approximate correspondence between the surrounding rock grade in this study and other indices (Q, RMR, and BQ). Method Surrounding Rock Grade I II III IV V Q value (10,000,40) (40,10) (10,1) (1,0.1) (0.1,0.001) RMR value (100,80) (80,60) (60,40) (40,20) (20,0) BQ value (700,550) (550,450) (450,350) (350,250) (250,0) Note: Grade-VI generally refers to the special geology, such as the powdery fault fracture zone, aeolian sand, and seriously collapsible loess.
[page 6] Appl. Sci. 2022, 12, 2656 6 of 20 Figure 5. The typical layout of the blasting hole (mm). The geological sketch method is usually adopted to determine the grade of the sur- rounding rock in the tunnel, which is analyzed and determined by professional geological engineers. The geological sketch record card of the tunnel face is shown in Table 2. Table 2. Geological sketch record card of the tunnel face. The Geological Sketch Record Card of Tunnel Face Project name: XXX Mileage: XXX Date: XXX Construction unit: XXX No. Item State description 1 Tunnel face and support type Width (m) Area (m2) Excavation method Depth (m) Advanced support Primary support Secondary lining 2 Lithology 3 Weathering degree Slightly Weakly Strongly Totally 4 Degree of hardness (Mpa) Extremely hard (>60) Hard (30~60) Relatively soft (15~30) Soft (5~15) Extremely soft (<5) 5 Number of structural surfaces 1 2 3 >3 6 Average spacing of the structural surface (m) >1 0.4~1 0.2~0.4 0~0.2 7 Degree of structural surface development Not developed Relatively developed Developed Extremely developed 8 Crack width (mm) 0~1 1~3 >3 9 Crack ﬁlling None Siliceous Calcium Argillaceous Rock debris Mud clamps rock debris
[page 7] Appl. Sci. 2022, 12, 2656 7 of 20 Table 2. Cont. The Geological Sketch Record Card of Tunnel Face 10 Degree of structural surface bonding Good Fair Poor Extremely poor 11 Integrity Integrated Relatively integrated Relatively broken Broken Extremely broken 12 Groundwater state None Drip outﬂow Linear outﬂow Inrush outﬂow 13 The major weak structural surface Occurrence Angle with tunnel axis 14 Crustal stress state Low High Extremely high 15 Surrounding rock grade I II III IV V VI Tabulator: XXX Reviewer: XXX By the method discussed above, 912 intelligent surrounding rock classiﬁcation samples were collected in ﬁve tunnels of the Zhengzhou–Wanzhou high-speed railway. These samples cover three surrounding rock grades (grade-III, grade-IV, and grade-V) and ﬁve main lithologies (dolomite, sandstone, limestone, shale, and mudstone). More details are shown in Table 3. Table 3. Intelligent surrounding rock classiﬁcation sample statistics. Surrounding Rock Grade Tunnel Lithology Sample Size Total III Luojiashan Tunnel Dolomite 110 325 Chufeng Tunnel Dolomite 28 Xinhua Tunnel Sandstone 31 Xiangjiawang Tunnel Limestone 156 IV Gaojiaping Tunnel Shale/Sandstone 105 420 Luojiashan Tunnel Dolomite 75 Chufeng Tunnel Dolomite 44 Xinhua Tunnel Sandstone 125 Xiangjiawang Tunnel Limestone 30 Xiangluping Tunnel Mudstone/Sandstone 41 V Gaojiaping Tunnel Shale 84 167 Luojiashan Tunnel Dolomite 62 Chufeng Tunnel Dolomite 4 Xinhua Tunnel Dolomite 15 Xiangjiawan Tunnel Limestone 2 Total 912 2.3. Sample Data Cleaning To analyze the sample data outliers, four box diagrams are shown in Figure 6 according to the drilling parameter class and surrounding rock grade.
[page 8] Appl. Sci. 2022, 12, 2656 8 of 20 (a) (b) (c) (d) Figure 6.

Intelligent Classification of Surrounding Rock of Tunnel Based on 10 Machine Learning Algorithms


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Wheel Loader Scooping Controller Using Deep Reinforcement Learning; DOI: 10.1109/access.2021.3056625; license: cc-by-sa.

[Assistant]
In the next phase, the robot changes the lift, tilt, and velocity actions simultaneously to navigate the bucket tip through the earth pile. Lift and tilt angles are increased and velocity decreases while the robot excavates deeper. In the last phase, the robot drives backward away from the pile, while lifting the arm to its maximum height to maintain the soil inside the bucket without loss of the material. B. REAL ENVIRONMENT EVALUATION To investigate the performance and adaptability of our scoop- ing controller in real-life machinery, we conducted several experiments with three different mean inclination angles of the pile. We explored the learned policies trained in simulation in the real world without tuning the network weights. To derive the amount of excavated soil in the bucket, we attached a load sensor to the bucket joint that measured the load on the linear actuator. Under the assumption that acceleration and velocity are negligible during excavation, the scooping dynamics in joint space can be reduced to a simple static equilibrium: τ = G + JT f (8) where f is a vector representing the contact forces, G is a vector containing the gravity terms, τ is a vector used to denote the torque at each joint and J is the Jacobian matrix of the arm. The masses of the links of the arm mechanism are known, as well as the Jacobian matrix, so J and G are obtained through direct computation using the joint posi- tions. f is obtained through the load cell measurements since we can derive a simple quasi-static analysis that describes the moment’s equilibrium at the bucket joint. Although we approximated the accumulated mass using the load cell, as described in Section III, most of the inconsistency between the simulation and real-life occurred during the ﬁll phase, as can be seen in Fig. 10, because the robot basically trans- lated this to additional scooped earth, resulting in an earlier exit phase. FIGURE 10. Example of comparisons between the normalized extracted mass inputs in the simulated (100 episodes) and real-life experiments. The data points are normalized by the maximum measured values of each set. In our experiments, we deﬁned two environment setups to be constant: (1) the position of the pile and (2) the starting position of the robot (1[m] in front of the pile). We set these conditions to keep the problem’s complexity manageable and because these setups are common to various types of local- ization methods and feature detection methods. The bucket scooping trajectories shown in this section were obtained using direct kinematics of the arm mechanism feedback and odometry from the wheel encoders. Computation of the total excavated earth in the experiments was carried out based on the applied torque, where the bucket exits the pile of gravel. To compare the planner’s performance to human operators, the ﬁrst step in the evaluation consisted of collecting data from manually driven scooping cycles using manual control. The ﬁrst author collected the data while attempting to man- ually imitate the three phase scoop strategy. Although the goal was to achieve continuous actions, the motions were less skilled than that of an expert operator. In total, 20 trials were conducted, which were less smooth during the ﬁrst attempts than in the later ones. We evaluated the learned scooping policy over 5 consecutive scooping cycles, where the earth’s inclination angle set to 32 degrees. Fig. 11 and Fig. 12 show the results, where each phase is labelled. In most of the approach phases, the robot prepared to interact with the soil by adjusting the bucket to a horizontal angle and decreasing the lift angle. Each contact with the pile triggered the ﬁll phase, where the robot penetrates the pile with a ‘‘slicing motion’’. During that motion, the applied torque increased due to higher pile resistance. The controller continued the ﬁlling phase until a combination of depth and torque thresholds were reached, leading to a low variation exit phase initial- ization. Although we initialized each trial with the same setup, the controller executed different trajectories during the approach phase, due to changes in the soil-tool interaction, VOLUME 9, 2021 24151
[page 8] O. Azulay, A. Shapiro: Wheel Loader Scooping Controller Using Deep Reinforcement Learning FIGURE 11. Comparisons of 5 consecutive scooping attempts from the pile with an average of 32 degrees of inclination angle. Bucket trajectory (A) applied torque at the bucket joint (B).

Wheel Loader Scooping Controller Using Deep Reinforcement Learning


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Automated Excavator Based on Reinforcement Learning and Multibody System Dynamics; DOI: 10.1109/access.2020.3040246; license: cc-by-sa.

[Assistant]
The step method is called for each time step of the simulation. It sends the observation vector, receives the action vector and assigns Algorithm 1 PPO-CMA (Server Side) 1: call function make: 2 for iteration = 1, 2, ...do 3: while iteration simulation budget N not exceeded do 4: call function reset 5: for T timesteps run agent on current policy or until a terminal state 6: call function step 7: end while 8: Train critic network and policy 9: Save (serialize) agent 10: end for Algorithm 2 Step Function in Simulation Environment (Client Side) function step (simulation parameters list): 1: Receive action vector from agent (server) 2: Set actions to simulation model inputs 3: for action 4: Set action value to the hydraulic system input 5: end for 6: for observation parameter 7: Get simulation variables from Solver 8: Compose observation vector 9: Calculate reward 10: Encode reward and observation to a single message 11: end for 12: Send encoded observation vector, reward and done to the agent end function it to the corresponding inputs. The reset method is used to restart the simulation when the simulation budget is exceeded or the termination state has been reached. The Parameters class is used for transfer of simulation-related parameters to the RL algorithm. The State class obtains the observation vector from the simulation and calculates the reward value of the provided action. III. PROBLEM STATEMENT The above procedure was applied to an excavator model with a complicated environment with deformable ground. Fig.4 shows the structure of the model. The model is a system with closed loops and has 4 Degrees of Freedom (DOF). The model consists of a total of nine rigid bodies interconnected by 10 joints. The model is equipped with the hydraulic circuit shown in Fig.5. The excavator model has 4 inputs controlling inputs rotation of the upper carriage, boom lift, boom tilt and bucket. The resulting action vector takes the form: a = [uuc ulb uda ub]T (19) where uuc is the upper carriage input, ulb the lift boom input, uda the dipper arm input and ub the bucket rotation input. The input uuc controls the hydraulic motor slew, which rotates the 214002 VOLUME 8, 2020
[page 6] I. Kurinov et al.: Automated Excavator Based on RL and Multibody System Dynamics FIGURE 4. Topological structure of the excavator model with hydraulic actuators. FIGURE 5. Hydraulic circuit schematics of the excavator. upper carriage around Y-axis relative to the under carriage. The lift boom input ulb controls the boom lift hydraulic cylin- ders, which rotates the lift boom around Z-axis relative to the upper carriage. The dipper arm input uda controls the dipper arm hydraulic cylinder, which rotates the dipper arm around Z-axis relative to the lift boom. The bucket input ub controls the hydraulic cylinder, which rotates the bucket around Z-axis relative to the dipper arm, please see Figure 4. For evaluation of the performed actions, the agent must access the observation vector and reward function. The observation vector of the excavator model is different for each reward function type but contains the same parameters groups, which are set as bodies, task and goal position. The ﬁrst group consists of the global coordinates of the bodies in meters and the rotations of the bodies in Euler parameters. In the case of an excavator, this group includes the upper carriage, lift boom, tilt boom and bucket bodies. Task-related observations are represented by values from sensors. Normal- ized mass in the hopper and bucket, a number of collisions of the bucket with the hopper are used as the goal group. Com- bination of these groups results to the following observation vector: s =  t mh mb lt−1 lt 1l rT rT g nc ng T (20) where, t is elapsed time, mh is normalized mass in the hopper, mb is normalized mass in the bucket, lt−1 is the distance to current goal on the previous time step, lt is the distance to the goal, 1l is the difference between lt and lt−1, r is position vector of bodies, rg is position vector of goals, nc is the number of the collisions during one episode and ng is the number of the goals already reached. The r vector has a size of 28, which is equal to the number of positions and rotations of the 4 main moving bodies. It consists of: r =  rT b1 rT b2 rT b3 rT b4 T (21) where, rbn is the vector of the body n positions.

Automated Excavator Based on Reinforcement Learning and Multibody System Dynamics


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Automated Excavator Based on Reinforcement Learning and Multibody System Dynamics; DOI: 10.1109/access.2020.3040246; license: cc-by-sa.

[Assistant]
The reinforcement learning method applies Proximal Policy Optimization with Covari- ance Matrix Adaptation (PPO-CMA) algorithm [15]. The agent is operating the excavator model that was based on the use of the multibody system dynamics. The current study employs the semi-recursive multibody approach, in which relative joint coordinates are used to minimize the number of differential equations [16]–[19]. The excavator is actuated by hydraulics. In this study, the hydraulic systems of the exca- vator are modeled using lumped ﬂuid theory, in which the hydraulic system is discretized to volumes [20], [21]. To sim- ulate real-life tasks of the excavator, deformable ground model based on a combination of the cellular automata and particle method is implemented [22]–[25]. The agent and the multibody solver are connected by ZeroMQ [26] asyn- chronous messaging library, where reinforcement learning algorithm is the server side and simulation environment is the client side. The objective of this study is to computationally combine efﬁcient multibody simulations with the reinforcement learn- ing algorithm. This combination, in turn, enables the creation and training of an autonomous agent. In this way autonomous agents can be designed effectively without endangering per- sonnel or machinery. II. METHODS This study combines ﬁve elements. The ﬁrst element is reinforcement learning implemented using the PPO-CMA algorithm [15]. The second element is a multibody system dynamics solver that uses a semi-recursive method [16]–[19]. The third method is the hydraulics simulation using lumped ﬂuid theory [20], [21]. The fourth method is a deformable ground implemented in the multibody simulation by combin- ing cellular automata and particle methods [22]–[25]. The last element focuses on connecting the RL algorithm with the multibody simulation via a custom application program interface (API) in Python based on ZeroMQ [26]. VOLUME 8, 2020 213999
[page 3] I. Kurinov et al.: Automated Excavator Based on RL and Multibody System Dynamics A. REINFORCEMENT LEARNING AND PROXIMAL POLICY OPTIMIZATION WITH COVARIANCE MATRIX ADAPTATION ALGORITHM Reinforcement learning is a machine learning paradigm in which learning is realized by a trial and error process. Unlike the other two main machine learning techniques, supervised learning and unsupervised learning, RL does not require a predeﬁned dataset and uses the environment directly for learning. The environment and agent are usually simulation models that enable observation – a vector of data about the environment and reward – which is the score of a performed action. The agent is the part of the simulation capable of taking action. The action is a set of inputs for interaction with the environment. The agent interacts with an environment by providing an action vector, getting an observation vector and receiving evaluation via a reward function. Reinforcement learning algorithms can be divided into model-based and model-free algorithms [27]. The differ- ence between model-based and model-free approaches is the knowledge of the agent about the reward function and transition matrix which maps observations to actions. In model-based approaches, the transition matrix and reward function deﬁne the model of the world. On the other hand, in model-free methods, the agent does not have knowledge about the transition matrix and reward function and, thus, the agent must estimate state-action pairs according to pre- vious experience. This research focuses on application of a model-free policy optimization method. The policy deﬁnes the behavior of the agent. In reinforcement learning, the policy is parametrized, e.g., by weights and biases of the neural network, which can be adjusted to change the behavior of the agent. A common policy optimization method is Proximal Policy Optimization (PPO), which has been shown to be able to operate in complex environments [28]. The basic idea of PPO is to optimize the policy with a gradient and limit changes to the area of sampled actions [28]. Nevertheless, PPO has a tendency to slow down or get stuck at local optima, because of prema- ture shrinkage of the exploration area [15]. The algorithm called Proximal Policy Optimization with Covariance Matrix Adaptation was designed to alleviate this problem [15]. The PPO-CMA algorithm is presented in the PPO-CMA study [1]. B.


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: How to evaluate performance of prediction methods? Measures and their interpretation in variation effect analysis; DOI: 10.1186/1471-2164-13-s4-s2; license: cc-by.

[Assistant] Further analyses were made about variations located in different protein secondary structural elements, on the surface or in the core of a protein, and according to protein structure type. The conclusion was that even at best, the predictions were only moderately accurate (~60%) and significant improvements would be needed. The correlation of the methods was poor. In another study six programs includeing CC/PBSA, EGAD, FoldX, I-Mutant2.0, Rosetta, and Hunter were compared [3]. The dataset contained 2156 single varia- tions from ProTherm. The goal of the study was to com- pare the performance of the methods in ΔΔG prediction. Thus, they did not directly predict the effect on protein function, just the extent of free energy change. The only measure used was correlation between the experimental and predicted ΔΔG values. The ability of Dmutant, two versions of I-Mutant 2.0, MUpro, and PoPMuSiC to detect folding nuclei affected by variations has been evaluated [34]. The dataset con- tained 1409 variations from the ProTherm and some methods were tested with the same data which they had been trained. They used only correlation coefficients as quality measures. The best being in the range of ~0.5. The performance of structure-based stability preditors, Dmutant, FoldX, and I-Mutant 2.0, were investigated with data for two proteins. There were 279 rhodopsin and 54 bacteriorhodopsin variations [35]. The best prediction accuracy for the rhodopsin dataset was <0.60, while it was somewhat greater for the bacteriorhodopsin dataset. Splice site predictors mRNA maturation is a complex process, which may be affected by variations in many steps. Prediction beha- viour of nine systems, GenScan, GeneSplicer, Human Splicing Finder (HSF), MaxEntScan, NNSplice, Splice- Port, SplicePredictor, SpliceView and Sroogle was tested [4]. The test dataset contained altogether 623 variations. The first dataset contained 72 variations that affect the four invariant positions of 5’ and 3’ splice sites. The second one included 178 variations either localized at splice sites in non-canonical positions, distant intronic variations, and short distance variations. The third set of 288 exonic varia- tions included 10 exonic substitutions that activate a cryp- tic splice site. In the fourth dataset were negative controls, altogether 85 variations without effect on splicing. The results contain just the numbers of predicted cases and the percentage of correct ones, thus detailed analysis of the merits of the methods cannot be made. The authors recommended some programs but stated that the in silico predictions need to be validated in vitro. Checklist for method developers and users This checklist is provided to help when comparing and measuring performance of predictors and when selecting a suitable one. These are items that method developers should include in articles, or as supplement to articles, as they enable effective comparison and evaluation of the performance of predictors. Items to check when estimating method performance and comparing performance of different methods: - Is the method described in detail? - Have the developers used established databases and benchmarks for training and testing (if available)? - If not, are the datasets available? - Is the version of the method mentioned (if several versions exist)? - Is the contingency table available? - Have the developers reported all the six performance measures: sensitivity, specificity, positive predictive value, negative predictive value, accuracy and Matthews correlation coefficient. If not, can they be calculated from figures provided by developers? - Has cross validation or some other partitioning method been used in method testing? - Are the training and test sets disjoint? - Are the results in balance e.g. between sensitivity and specificity? - Has the ROC curve been drawn based on the entire test set? - Inspect the ROC curve and AUC. - How does the method compare to others in all the measures? - Does the method provide probabilities for predictions? Acknowledgements This work was supported by the Sigrid Jusélius Foundation, Biocenter Finland and the Competitive Research Funding of Tampere University Hospital.

How to evaluate performance of prediction methods? Measures and their interpretation in variation effect analysis


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Wheel Loader Scooping Controller Using Deep Reinforcement Learning; DOI: 10.1109/access.2021.3056625; license: cc-by-sa.

[Assistant]
The sequence of state-action pairs deﬁnes the trajectory τ = (s0, a0, s1, a1 . . . . . . sH, aH) of length H. The return Rγ t is the total discounted reward for the initial state over the trajectory Rγ t = PH i=t γ i−tr (si, ai). The goal in reinforcement learning is to learn a policy which maximizes the expected return from the initial distribution J = E  Rγ 1 |π  . The state-value function is deﬁned as the expectation value of the return over all allowed trajectories from a speciﬁc state s, V π (st) = Eπ  Rγ t |st  . While the action-value function describes the expected return after tak- ing an action at in state st, Qπ (st, at) = Eπ  Rγ t |st, at  . DDPG is an actor-critic off-policy gradient algorithm that implements a stochastic behavior policy and estimates a deterministic target policy [16]. A schematic of the pro- cedure can be seen in Fig. 2. DDPG primarily uses two neural networks in its learning phase, one for the actor (policy network) and one for the critic (value network), with weights θµ and θQ, respectively. The actor network is used to approximate the optimal policy whereas the critic utilizes the value-based approach to estimate the value of state-action pairs. The actor function µ (st|θµ) deterministically maps states for speciﬁc actions. The critic’s output is the estimated Q-value of the current state and action provided by the actor Q st, at|θQ . These networks compute action predictions for the current state and generate the Temporal-Difference (TD) error at each time step while using a set of target networks µ′ (st) , Q′ (st, at), with weights θµ′ and θQ′ respectively. The critic’s loss function L, and target yt are computed from the sum of the immediate reward and the outputs of the target actor and critic networks: yi = ri + γ Q′ h si+1, µ′  si+1|θµ′ |θQ′i (1) L = 1 N X i  yi −Q  si, ai|θQ2 (2) VOLUME 9, 2021 24147
[page 4] O. Azulay, A. Shapiro: Wheel Loader Scooping Controller Using Deep Reinforcement Learning FIGURE 3. Architecture of the Komodo robot with our arm mechanism. The critic is updated by minimizing the loss and the actor is updated by applying the chain rule using the sampled policy gradient: ∇θµJ ≈1 N X i × h ∇aQ  s, a|θQ |s=si,a=µ(si|θµ)∇θµµ s|θµ |s=si i (3) IV. SYSTEM DESIGN To investigate bucket-scooping processes, we developed an arm mechanism that we attached to a Komodo robot (see Fig. 3). The robot is a 40kg weight fully Robot Operating System (ROS) supported skid-steer robot that includes an Intel NUC i7 CPU, SICK LMS1 range laser scanner, an Asus Xtion pro depth camera, and 6-axis IMU. To control the robot motion, we used the Komodo built-in Roboteq differential driver for the motor controllers. For the loading mechanism, earth-moving mechanisms typically consist of a robotic arm controlled by hydraulic pistons and a bucket. In this work, we developed an arm mechanism that resembles to an indus- trial skid-steer loader that consists of three main parts: 1) a base link, 2) an arm assembly, and 3) a bucket, as shown in Fig. 3. The mechanism is constructed primarily from strong lightweight 6061 aluminum parts that can withstand a bucket load of up to 12[Kg]. The mechanism is controlled by two parallel systems of linear actuators with a built-in position feedback potentiometer. The arm’s linear actuators are responsible for the lift motion, whereas the bucket’s linear actuators generate its tilting motion. To control tilt and lift motion, we used a Proportional Integral Derivative controller (PID), employing feedback from the linear actuator’s poten- tiometer. The system is operated using the Robot Operation System (ROS) interface. The electrical system is depicted in Fig. 4, and the com- ponents are listed in Table 1. We built the electric cir- cuit to be powered by the robot’s power source, whereas the components with other power requirements utilized FIGURE 4. The electrical scheme of the arm mechanism. TABLE 1. Arm mechanism components. voltage converters. We divided the control system into two levels. The higher level was composed of the PID application on the tilt and lift motion, communication initializing, and syncing between the various channels using ROS. The lower- level controller was composed of two slave nodes that sub- scribe to the data published by the main controller channels.


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Detection of Ionospheric Scintillation Based on XGBoost Model Improved by SMOTE-ENN Technique; DOI: 10.3390/rs13132577; license: cc-by-sa.

[Assistant]
The wideband antenna was split into several ports that were connected to a commercial ISM receiver, Septentrio PolaRx, and SDR-based RF front ends, respectively. Scintillation-related mea- surements including channel correlation values were continuously collected by the ISM receiver, and relevant scintillation indices and event indicators were calculated simulta- neously [10]. Furthermore, the indicators were to be compared with the threshold values preset before data collection to trigger the data server to record raw IF samples generated by the SDR front ends [38,39]. It should be mentioned that only data affected by the natural ionospheric scintillation phenomenon were to be recorded, to save a large amount of memory. These data could be utilized in research on optimal algorithms with respect to the ISM receiver, as well as providing a database for the analysis of strong scintillation characteristics. It is worth mentioning that, in the process of the acquisition and tracking, the DLL and PLL pull-in noise bandwidths were set as 2 Hz and 25 Hz, respectively, while the pull-in time was set as 500 ms. Once the signals were locked and the tracking loop was kept stable, the DLL and PLL pull-in noise bandwidths were reset to 1 Hz and 10 Hz, respectively, to reduce the inﬂuence of noise and other sources of interference. Natural Ionospheric Scintillation Events Wideband Antenna Splitter Commercial Scintillation Receiver(Septentrio PolaRx) RF Front End 1 RF Front End 2 RF Front End N · · · SDR Front-ends Circular Buffer 1 Circular Buffer 2 Circular Buffer N · · · Event Data Buffers Autonomous Scintillation Monitoring and Event Trigger Data Collection and Control Server Data Storage Figure 2. Architecture of the scintillation event-driven data collection system developed in the equatorial region. The commercial ISM Septentrio PolaRx is used to collect relevant navigation data and trigger SDR-based RF front ends when natural scintillation events occur.
[page 6] Remote Sens. 2021, 13, 2577 6 of 22 A dataset of GPS L1 C/A signals was selected for the study. It was recorded from the GTEC free front ends, which were conﬁgured to collect zero-frequency data with 8-bit- resolution samples at a 20 MHz complex sampling rate. The collection period lasted from March 2013 to February 2015, covering the latest peak period for solar activity. Furthermore, these available data were processed by the SDR receiver advanced by a combination coherent/con-coherent integration acquisition algorithm to enhance the acquisition and tracking performance under the strong ionospheric scintillation environment. Overall, 45 segments of one-hour data were used in the following research, with two or three satellites’ data selected in each segment. The I/Q values were sampled at 1000 Hz, while the sample rate was set to 1 Hz according to the size of the shifting window. Based on three levels of scintillation intensity, Table 1 presents the information of the training dataset and testing dataset used in the following research. Table 1. The distribution information of training dataset and testing dataset on three classes. Dataset Weak (Class 0) Moderate (Class 1) Strong (Class 2) Total Ratio Training 155,892 136,343 38,515 349,559 4.05:3.54:1 Testing 18,680 14,496 5665 38,841 3.30:2.56:1 2.3. Feature Extraction The features play a decisive role in the classiﬁcation model. There are different sets of features producing different results for the detection accuracies, while the set of signal- based features as well as corresponding combinations are proved to be optimal with the location-independent technique [16]. The raw GNSS signal measurements in phase (I) and quadrature-phase (Q) correlators were utilized as features, which were extracted from the software GNSS receiver at the end of the tracking process. In recent research [16], the feature set of ⟨I⟩, ⟨Q⟩, I2 , Q2 , ⟨SI⟩, SI2 —namely, the average values of I and Q correlator outputs, I2, Q2, signal intensity SI and SI2—has been proved to show excellent performance for ionospheric scintillation detection by machine learning algorithms.

Detection of Ionospheric Scintillation Based on XGBoost Model Improved by SMOTE-ENN Technique


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Automated Excavator Based on Reinforcement Learning and Multibody System Dynamics; DOI: 10.1109/access.2020.3040246; license: cc-by-sa.

[Assistant] [page 1] Received October 16, 2020, accepted November 15, 2020, date of publication November 24, 2020, date of current version December 10, 2020. Digital Object Identifier 10.1109/ACCESS.2020.3040246 Automated Excavator Based on Reinforcement Learning and Multibody System Dynamics ILYA KURINOV 1, GRZEGORZ ORZECHOWSKI 1,2, PERTTU HÄMÄLÄINEN 3, AND AKI MIKKOLA 1 1Department of Mechanical Engineering, LUT University, 53850 Lappeenranta, Finland 2Mevea Ltd., 53850 Lappeenranta, Finland 3Department of Computer Science, Aalto University, 02150 Espoo, Finland Corresponding author: Ilya Kurinov (ilya.kurinov@lut.ﬁ) This work was supported in part by the European Union’s Horizon 2020 Research and Innovation Programme through the Marie Sklodowska-Curie Project under Grant 845600 (RealFlex) in part with Academy of Finland #316106. ABSTRACT Fully autonomous earth-moving heavy equipment able to operate without human intervention can be seen as the primary goal of automated earth construction. To achieve this objective requires that the machines have the ability to adapt autonomously to complex and changing environments. Recent developments in automation have focused on the application of different machine learning approaches, of which the use of reinforcement learning algorithms is considered the most promising. The key advantage of reinforcement learning is the ability of the system to learn, adapt and work independently in a dynamic environment. This article investigates an application of reinforcement learning algorithm for heavy mining machinery automation. To this end, the training associated with reinforcement learning is done using the multibody approach. The procedure used combines a multibody approach and proximal policy optimization with a covariance matrix adaptation learning algorithm to simulate an autonomous excavator. The multibody model includes a representation of the hydraulic system, multiple sensors observing the state of the excavator and deformable ground. The task of loading a hopper with soil taken from a chosen point on the ground is simulated. The excavator is trained to load the hopper effectively within a given time while avoiding collisions with the ground and the hopper. The proposed system demonstrates the desired behavior after short training times. INDEX TERMS Autonomous agents, discrete event dynamic automation systems, learning and adaptive systems, real-time simulation, multibody system dynamics, reinforcement learning, PPO-CMA. I. INTRODUCTION Mobile earth-moving machinery is widely used in the con- struction, forestry and mining industries. The operation of such machinery is challenging and often hazardous, because the machines are used in dangerous environments like open-pit mines and large infrastructure projects. From the performance perspective, machine operators are often the weakest link, as they introduce delays in the form of statutory rest periods, transportation to and from the worksite, and fatigue-induced errors. Consequently, many manufacturers of heavy machinery have begun to explore greater automation of machine operations. Tele-remote operation [1] is nowa- days possible, which eliminates the aspect of the dangerous environment, but other issues remain. To fully address the The associate editor coordinating the review of this manuscript and approving it for publication was Wei Zhang. challenges associated with heavy earth-moving operations, there is a need for fully automated and operator-independent machines. Use of autonomous earth-moving machinery can increase safety and productivity, but production and development of such equipment is a formidable task. The main chal- lenge facing earth-moving machinery automation is the com- plexity of the working environment. In mining and other earth-moving operations, the environment is characterized by diversity, variable geometry, the effects of the machine or other machines on the environment itself, the properties of the soil or media to be excavated, and external factors such as ambient temperature and climatic conditions. Control of machine operations using conventional methods is therefore a challenging enterprise [2]. The autonomous machine should be able to adapt to the sum of external factors and make decisions based on the task and state of the environment. 213998 This work is licensed under a Creative Commons Attribution 4.0 License.


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Detection of Ionospheric Scintillation Based on XGBoost Model Improved by SMOTE-ENN Technique; DOI: 10.3390/rs13132577; license: cc-by-sa.

[Assistant]
Feature matrix Training data (scintillation data) Feature extraction Manual visual inspection Feature matrix Label vector SMOTE-ENN resampling XGBoost algorithm Trained model Cross validation Testing data (scintillation data) Feature extraction Predictable model Predicted labels Manual visual inspection Manual labels Performance evaluation New dataset Resampling Training Predicting Raw dataset Figure 5. The process ﬂow of XGBoost detection model improved by SMOTE-ENN resampling technique. 4. Results A series of experiments were conducted to evaluate the performance of the proposed method and compare it with that of the decision tree and random forest algorithms, which have shown high validation accuracy in previous research [16]. The evaluation criterion with which the results were quantitatively analyzed is introduced. Based on the validation
[page 11] Remote Sens. 2021, 13, 2577 11 of 22 phase, the overall accuracy of three different machine learning methods in the condition of different tree depths was compared, determining the complexity, computational efﬁciency and performance of these models to a large extent. Next, different resampling methods were implemented on the training dataset, comparing their performance in improving the detection of the minority class. Finally, experiments on training sets with different ratios of the three classes were carried out to evaluate the inﬂuence of imbalanced data on scintillation detection, while previous similar experiments were conducted on testing sets. 4.1. Evaluation Criterion As one of the statistical tools most commonly used to evaluate the performance of detection algorithms, the confusion matrix, aimed at detection for three classes, was introduced. According to that, the corresponding metrics of overall accuracy, precision, recall and F1 score could be calculated to assess the performance of the detection models. Table 3 shows the example of the confusion matrix on three classes. Table 3. The confusion matrix on three classes. Class Prediction 0 1 2 Truth 0 N00 N01 N02 1 N10 N11 N12 2 N20 N21 N22 In Table 3, Nij (i, j = 0, 1, 2) refers to the number of samples corresponding to the truth class i and prediction class j. According to that, the relative evaluation indicators on three classes can be described as follows: accuracy = N00 + N11 + N22 N00 + N10 + N20 + N01 + N11 + N21 + N02 + N12 + N22 (11)      precision0 = N00 N00+N10+N20 precision1 = N01 N01+N11+N21 precision2 = N02 N02+N12+N22 (12)      recall0 = N00 N00+N01+N02 recall1 = N10 N10+N11+N12 recall2 = N02 N20+N21+N22 (13)        F1 −score0 = 2 ∗precision0 ∗recall0 precision0+recall0 F1 −score1 = 2 ∗precision1 ∗recall1 precision1+recall1 F1 −score1 = 2 ∗precision1 ∗recall1 precision1+recall1 (14) where the precision value represents the correct predicted positive ratio, and the recall value refers to the percentage of correctly predicted positive events among real positive events. Based on that, the weighted average of the precision and recall F1 score is defined as the F1 score. 4.2. Accuracy Evaluation on Cross Validation As an important parameter for algorithms based on the decision tree, random forest and XGBoost, the tree depth determines the complexity and performance of models. Excessive depth might increase the validation accuracy, but can also lead to increased model complexity and greater computational loads, causing overﬁtting. Therefore, an appropriate tree depth is vital for the training model. Based on cross-validation, Figure 6 shows the mean accuracies across different tree depths and detection methods. Of the overall 349,559 points of the training set, 100,000 entries were randomly selected and used
[page 12] Remote Sens. 2021, 13, 2577 12 of 22 in the validation experiment to reduce the running time. The accuracies of XGBoost are marked, ranging from 1 to 30, and compared with those of the decision tree and random forest, which presented high validation accuracies in previous research [16]. 0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 Tree depth 76 80 84 88 92 96 100 XGBoost Random Forest Decision Tree Figure 6. The mean accuracy of 10-fold cross validation based on different tree depths and detection methods. In the overall training dataset mentioned in Table 1, 100,000 samples were selected randomly and used in the validation process, and the remaining data were used as training samples.


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Spectral Angle Mapping and AI Methods Applied in Automatic Identification of Placer Deposit Magnetite Using Multispectral Camera Mounted on UAV; DOI: 10.3390/min12020268; license: cc-by-sa.

[Assistant] Rauhala, A.; Tuomela, A.; Davids, C.; Rossi, P. UAV Remote Sensing Surveillance of a Mine Tailings Impoundment in Sub-Arctic Conditions. Remote Sens. 2017, 9, 1318. [CrossRef] 14. Martelet, G.; Gloaguen, E.; Døssing, A.; Lima Simoes da Silva, E.; Linde, J.; Rasmussen, T.M. Airborne/UAV Multisensor Surveys Enhance the Geological Mapping and 3D Model of a Pseudo-Skarn Deposit in Ploumanac’h, French Brittany. Minerals 2021, 11, 1259. [CrossRef] 15. Saha, D.; Annamalai, M. Machine Learning Techniques for Analysis of Hyperspectral Images to Determine Quality of Food Products: A Review. Curr. Res. Food Sci. 2021, 4, 28–44. [CrossRef] [PubMed] 16. Girouard, G.; Bannari, A.; Harti, A.E.; Desrochers, A. Validated Spectral Angle Mapper Algorithm for Geological Mapping: Comparative Study between Quickbird and Landsat-TM. 6. 2014. Available online: https://www.researchgate.net/publication/ 228799788_Validated_spectral_angle_mapper_algorithm_for_geological_mapping_comparative_study_between_QuickBird_ and_Landsat-TM (accessed on 4 July 2021). 17. Shafri, H.Z.M.; Suhaili, A.; Mansor, S. The Performance of Maximum Likelihood, Spectral Angle Mapper, Neural Network and Decision Tree Classiﬁers in Hyperspectral Image Analysis. J. Comput. Sci. 2007, 3, 419–423. [CrossRef] 18. den Hartog, D.; Harlaar, J.; Smit, G. The Stumblemeter: Design and Validation of a System That Detects and Classiﬁes Stumbles during Gait. Sensors 2021, 21, 6636. [CrossRef] [PubMed] 19. Chauhan, D.; Anyanwu, E.; Goes, J.; Besser, S.A.; Anand, S.; Madduri, R.; Getty, N.; Kelle, S.; Kawaji, K.; Mor-Avi, V.; et al. Comparison of Machine Learning and Deep Learning for View Identiﬁcation from Cardiac Magnetic Resonance Images. Clin. Imaging 2022, 82, 121–126. [CrossRef] [PubMed] 20. Kobayashi, S.; Ikuta, K.; Sugimoto, R.; Honda, H.; Yamada, M.; Tominaga, O.; Shoji, J.; Taniguchi, M. Estimation of submarine groundwater discharge and its impact on the nutrient environment at Kamaiso beach, Yamagata, Japan. Nippon. Suisan Gakkaishi 2019, 85, 30–39. [CrossRef] 21. Nguyen, H.H.; Carter, A.; Hoang, L.V.; Vu, S.T. Provenance, Routing and Weathering History of Heavy Minerals from Coastal Placer Deposits of Southern Vietnam. Sediment. Geol. 2018, 373, 228–238. [CrossRef] 22. Beretta, F.; Rodrigues, A.L.; Peroni, R.L.; Costa, J.F.C.L. Automated Lithological Classiﬁcation Using UAV and Machine Learning on an Open Cast Mine. Appl. Earth Sci. 2019, 128, 79–88. [CrossRef] 23. Ono, M. Parallax Error Correction Techniques by Image Matching for ASTER/SWIR Band-to-Band Registration. In Proceedings of the Platforms and System, Rome, Italy, 9 January 1995; pp. 18–27. [CrossRef] 24. Laurence, S.J. On Tracking the Motion of Rigid Bodies through Edge Detection and Least-Squares Fitting. Exp. Fluids 2012, 52, 387–401. [CrossRef] 25. Singh Kushwah, J.; Kumar, A.; Patel, S.; Soni, R.; Gawande, A.; Gupta, S. Comparative Study of Regressor and Classiﬁer with Decision Tree Using Modern Tools. Mater. Today Proc. 2021, S2214785321076574. [CrossRef] 26. Mohammadi, M.; Rezaei, J. Ensemble Ranking: Aggregation of Rankings Produced by Different Multi-Criteria Decision-Making Methods. Omega 2020, 96, 102254. [CrossRef] 27. Haixiang, G.; Yijing, L.; Shang, J.; Mingyun, G.; Yuanyue, H.; Bing, G. Learning from Class-Imbalanced Data: Review of Methods and Applications. Expert Syst. Appl. 2017, 73, 220–239. [CrossRef] 28. Gholami, R.; Moradzadeh, A.; Youseﬁ, M. Assessing the Performance of Independent Component Analysis in Remote Sensing Data Processing. J. Indian Soc. Remote Sens. 2012, 40, 577–588. [CrossRef]

Spectral Angle Mapping and AI Methods Applied in Automatic Identification of Placer Deposit Magnetite Using Multispectral Camera Mounted on UAV


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Reinforcement learning based process optimization and strategy development in conventional tunneling; DOI: 10.1016/j.autcon.2021.103701; license: cc-by.

[Assistant]
11 it can be seen that within individual sequences of top heading or bench excavation, the agent alternates between supported and unsupported blasts which shows that it has successfully learned to avoid excessive use of support ahead of the face. 6. Conclusion and implications for digital tunneling Before drawing conclusions from the experiments, it should be pointed out that in this initial study, the agent’s possibilities to find creative and not yet imagined solutions to real world tunneling problems are confined to the given set of actions and the taken simplifications (see Section 2.1). We therefore see it as a success that the agent optimizes the given scenario and finds strategies that are comparable to current practices in tunneling. As given in the introduction, the current study should serve as a base for future developments of RL in tunneling. Consequently, optimizations that improve the current practices of “real life” tunneling are to be expected from future studies. In the experiments of the previous chapter, we can observe that the agents have found policies that minimize the overall amount of neces sary blasts, minimize changes between top heading and bench excava tion, favor long over short advance lengths and minimize the use of face support. These strategies show that the agents have learned to work in an efficient and economically optimized way. We see similarities in this RL-based tunneling strategy to real world tunneling paradigms like the NATM [49] which uses partial excavation to minimize the necessary support. Minimizing the number of changes between top heading or bench excavation while at the same time not exceeding a too long dis tance between them is also part of NATM tunneling as this optimizes construction site logistics on the one hand and safety requirements on the other. The found strategies that rely on support ahead of the face show similarities to the “Adeco” method [25] which uses heavy support installation and long advance lengths to deal with the encountered rockmass conditions. While adhering to safety requirements is impera tive, most “real life” tunneling methods work towards minimizing the number of necessary blasts which is a policy that was found by all the agents. TunnRL (see Section 2) has shown that it is not only a functioning environment/simulation of conventional tunneling, but also that a RL agent can successfully interact with it and learn optimized and inno vative strategies that seem realistic compared to real world tunnel excavation. At the same time, we see the challenges of computational instabilities and trial and error approach in the process of developing well-functioning models, highlighting this early stage in RL for tunneling. Clearly both the reward system, the rockmass-environment and the action-system has room for improvement. Still we see a signif icant potential in the TunnRL-concept: firstly, for an on-face decision support system in a further developed and more realistic version, and secondly as a first step to more advanced automation in underground Table 5 Statistics of the training runs presented in Fig. 7. Values were computed from the 500-episode sliding window average and the episode of the respective value is given in parenthesis behind it. Row-wise best performances are highlighted in grey. G.H. Erharter et al.
[page 10] Automation in Construction 127 (2021) 103701 10 construction. Where many of the developments of tunnel processes today address the automation of small-scale processes in the tunnel or at the excavation face, a further developed TunnRL could be part of the main controlling mechanism that operates the overall tunnel construc tion site. Albeit the agents in this simulation were not able to find new and undiscovered strategies of tunnel excavation (see previous chapter), we see the fact that completely untrained agents are able to find tunneling strategies which are comparable to “real world tunneling” as a proof of concept that RL is successfully applicable to this kind of prob lem. The main goal of the study is therefore fulfilled, and future studies will work towards giving the agent more capabilities and increasing the environment’s realism. This will ultimately pave the way for an opti mized decision finding process in sequential tunneling. 7. Outlook Albeit we designed TunnRL in a practice related context, there are numerous improvements to make the framework more realistic and more robust.

Reinforcement learning based process optimization and strategy development in conventional tunneling


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Reinforcement learning based process optimization and strategy development in conventional tunneling; DOI: 10.1016/j.autcon.2021.103701; license: cc-by.

[Assistant]
A copy of the agent is saved after every 1000 episodes. To observe how big the differences are between individual training runs, we trained several identical DQN agents in the above described environment. After every episode, 21 parameters are saved to monitor the training progress. Below, the parameters that are mentioned in the paper are given (see the code “A_utilities.py” in Appendix 1 for a list of all recorded parameters): ▪ number of the current episode ▪ cumulative reward of the episode ▪ current value of ε ▪ the number of face instabilities of the whole episode ▪ average loss of the DQN agent throughout the episode ▪ number of moves/blasts that were required to finish the episode/to reach a terminal state ▪ 8 counters for how many times each of the actions of Table 3 were used; the goal of these counters is to see if the agent favors some actions over others and to detect “strategy changes” 5. Experiments In this section we present five training paths of exemplary agents and discuss the different strategies they found to deal with the given task. In Fig. 7, recordings of the agents’ training are given which shows that each agent has found a unique solution, and all training paths are substan tially different from one another. Comparing the five agents to one another, all of them were able to increase the cumulative reward per episode above 200 within 10,000 episodes (Fig. 7 first row). After around 30,000 episodes, first differ ences arise where first the cumulative reward of agent 2 and then of agent 3 started to stagnate. The reward of agent 1 stagnates at around 80,000 episodes. The cumulative rewards of the agents 4 and 5 kept on increasing, whereas agent 5 became instable after around 75,000 epi sodes and agent 4 reached the maximum reward at the end of the 120,000 training episodes. Right from the start, all agents started to use long advance lengths as a means to decrease the number of blasts/moves per episode as this is an effective way to maximize the achievable reward (Fig. 7 second row). Where the agents 1, 4 and 5 all took a similar strategy that aims at continuously minimizing the blasts/moves per episode throughout the whole training, agents 2 and 3 reached a minimum of around 120 blasts/ moves per episode after around 30,000 episodes. This correlates well with the stagnating rewards after 30,000 episodes of agents 2 and 3 as described above. The biggest differences in training paths can be observed with Table 4 Rewards that the agent receives from the environment in response to its actions. Reward (points) Description Condition tl * 3 reward for achieving breakthrough posth ≥tl and posbi ≥tl tl * 3 * -1 penalty for a timeout if number of moves in current episode is >200 −6 penalty for using the wrong excavation sequence, i.e. the bench is driven further ahead than the top heading posbi > posth −5 penalty for unstable tunnel face conditions determined from face pressure pf as evaluated by Eq. (3) (see below) pf ≥0 −4 penalty for changing from top heading to bench excavation or vice versa, as this usually involves a delay of the excavation due to logistics if current a ≥200 and prev. a < 200 or if current a < 200 and prev. a ≥200 −3 penalty for a too far distance between top heading and bench (see Section 3.1) if posth – posbi > distmax (initially set to 50 m) −2 penalty for using face support as this consumes additional time and resources if a = 112; a = 152; a = 202; a = 222 −1 penalty for every other move that does not meet any of the above conditions no other condition is fulfilled G.H. Erharter et al.
[page 8] Automation in Construction 127 (2021) 103701 8 respect to face instabilities, where only the agents 2 and 3 started to actively decrease the number of face instabilities (Fig. 7 third row) early in the training process (within 10,000 episodes). In contrast to that, the number of face instabilities increased within the first 10,000 episodes for the other agents and then only decreased slowly throughout the rest of the training. The best performing agent 4 shows a remarkable trend of stagnating face instabilities until around 80,000 episodes, followed by a decrease towards the end of training. While the goal was to let all agents train for 120,000 episodes, the training process of the agents 2 and 5 became unstable after around 75,000 and 85,000 episodes respectively, which led to the abortion of training after it could be observed that the agent would not recover from this.


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Wheel Loader Scooping Controller Using Deep Reinforcement Learning; DOI: 10.1109/access.2021.3056625; license: cc-by-sa.

[Assistant] Another drawback is that the majority of work done to automate scooping processes is based on data recorded by expert operators, which limits the efﬁciency to the level of the operator’s skills [1]. Since the scooping cycle is a repetitive and dynamic task involv- ing intricate interactions with the environment, here a new strategy is put forward to solve the problem of autonomous bucket scooping automation for wheel loaders using Rein- forcement Learning (RL) methods. RL is a machine learning technique that has the potential to enable robots to learn large repertoires of behavioral skills with minimal human intervention through trial and error. The main advantage of RL is that it does not require a predeﬁned control structure and can explore the environment to ﬁnd a good policy to follow, thus mitigating the shortcoming of having to explicitly derive the interconnection with the environment. Deep Rein- forcement Learning (DRL) methods have been successfully applied in many complex robotic tasks from manipulation [11] to autonomous vehicles [12], and aerial applications [13]. However, practical real-world applications of RL are rel- atively rare since they often require unrealistic learning times, high sample complexity, and can potentially damage the robot [14], [15]. This article focuses on developing a scooping motion plan- ner that learns transferable scooping policies solely under simulation. We show that an earthmoving machine can be fully controlled using DRL methods trained solely in simula- tion and then deployed on a real Unmanned Ground Vehi- cle (UGV). The complete system architecture is depicted in Fig. 1. To further increase the robustness of the sys- tem and narrow the reality gap, we deﬁned a compact design of the observation space and performed detailed system identiﬁcation. The controller is trained using the model-free, actor-critic Deep Deterministic Policy Gradi- ent (DDPG) algorithm [16], which successfully completes scooping cycles without any user-provided demonstrations. We developed a ROS-Gazebo based open-source training environment (available at [17]) for the agent to learn scooping motions. To verify the learning algorithm and simulation viability, we deployed the controller in a real-world scenario. We also compared the performance of our DRL based con- troller against manual human control for an earthmoving cycle. The primary contribution of this paper is its novel end- to-end Neural Network (NN) based controller for a robotic wheel loader that autonomously scoops earth from a pile, with three degrees of freedom (DoF). The controller is trained solely via simulation without any prior knowledge using DRL. To the best of our knowledge, this is the ﬁrst 3-DoF DRL scooping controller to learn from simulation without previous knowledge and be deployed on a real-life excavation machine. In addition, the development of an open source [17] training environment provides the research community with a tool to collect data and observe progress performance for excavation purposes. The remainder of this paper is structured as follows. Section II further discusses related work in this area, fol- lowed by a background review in Section III. Section IV describes the system design and Section V introduces the proposed scooping controller implementation. We demon- strate our simulation and report the experimental results in Section VI, followed by conclusions and suggested future research in Section VII. II. RELATED WORKS Most previous research on automating bucket-ﬁlling pro- cesses are based on data recorded by expert operators and implement a control system that can be generally divided into three main categories. The ﬁrst is made up of position control algorithms for bucket motion trajectories, which are intended to maximize the volume scooped by the bucket [4], [5], [18], [19]. These approaches have been successful, but they rely on expert trajectories and do not generalize to different machine-pile environments. The second category is composed of compliance control algorithms, such as soil estimation based methods [6], [7] and force/torque based methods [20], [21]. Soil estimation methods predict the soil- tool interaction force and apply a heuristic-based motion by modifying the soil parameters in the program.


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Wheel Loader Scooping Controller Using Deep Reinforcement Learning; DOI: 10.1109/access.2021.3056625; license: cc-by-sa.

[Assistant]
The drawbacks of soil estimation methods include the fact that they lack precision, fail to capture the rapidly changing properties in the environments or are non-real-time capable and thus not feasible/implementable [20]. Others combine these methods with a higher-level planner. For instance, a force-control tra- jectory controller with prioritized tasks [20], a coarse and ﬁne planner that ensures equal performance over a large number of digs [22], or adding a disturbance observer to compensate for the difference between prediction and actual performance [21], [23]. These compliance control methods do not follow a desired trajectory but rather apply speciﬁc forces to the pile during the scooping motion. The third category is made up of control algorithms that employ a behavior-based approach for motion control, such as a rule-based algorithm that depends on the current phase and acts dynamically [8], [9]. Thus, most previous solutions to automate the scooping task (1) do not generalize to different machines or pile environ- ments (2) rely on prior knowledge of an expert operator and (3) require accurate models of the machine and therefore are susceptible to failure in the presence of modeling errors, wear and tear, and changing conditions [10]. This underscores the need for a generic automatic scooping solution that can be adapted to different scenarios. Few deep Learning methods have been implemented in earthmoving operations, especially in bucket-ﬁlling pro- cesses, mainly due to the task complexity. The chal- lenges include various problems such as simulating the environment, data collection, and soil-tool dynamics. Dadhich et al. [10], [24] demonstrated the implementation of a NN approach to control bucket motion during real-life 24146 VOLUME 9, 2021
[page 3] O. Azulay, A. Shapiro: Wheel Loader Scooping Controller Using Deep Reinforcement Learning FIGURE 2. DDPG algorithm schematic. loading processes. They used data collected from an expert operator to train a time-delayed NN to perform a bucket- ﬁlling motion with two degrees of freedom (static wheels). Halbach et al. [25] implemented a shallow three-DoF NN controller for automated pile loading with a robotic wheel loader that used learning from the demonstration of sensor recordings. Both of these approaches were successful in terms of the bucket ﬁlling cycle, but nevertheless rely on the prior knowledge of expert operator data, which limits the efﬁciency to the operator’s skills which are not always available. III. PRELIMINARIES A. EARTHMOVING CYCLE A wide range of earth-moving machines are currently used in industry, the most common of which are wheel loaders and excavators. Excavators are used to dig into the ground whereas wheel loaders are used to load and transport exca- vated material. An earthmoving cycle is comprised of several steps: 1) navigating towards the pile; 2) scooping up the earth; 3) navigating to the dump location; and 4) dumping. This paper focuses on automating the bucket-ﬁlling process, which is the most complex phase. The ﬁlling of a bucket is a complex granular ﬂow problem that can be divided into three general phases: approach, ﬁll, and exit [3]. Each phase operates under different conditions in terms of vehicle feedback, terrain properties, and duration. In the ﬁrst phase, approach, the front-loader moves towards the pile of earth, and the operator chooses the height and penetration angle of the front loader’s bucket. In the second phase, ﬁll, there is a simultaneous change in the lift, tilt, and throttle actions to navigate the bucket tip through the earth pile while avoiding wheel slip and piston stall. The last phase, exit, involves tilting the bucket until the breakout, while the machine moves in reverse from the pile. B. REINFORCEMENT LEARNING In RL, the problem is typically characterized as a Markov Decision Processes (MDP), which deﬁned by the tuple ⟨S, A, r, P, γ ⟩. S represents the state-space, and A is the action-space. Transitions between states are performed with transition probability p (st|st, at) ∈P, reward r (st, at) and a discount factor γ ∈[0, 1]. At each time step, the agent observes the current state st ∈S and takes action at ∈A according to the policy πθ, which can be stochastic or deter- ministic. While interacting with the environment, the system transitions to a new state st+1 ∈S and the agent receives a reward rt (st, at, st+1).


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Cutterhead and Cutting Tools Configurations in Coarse Grain Soils; DOI: 10.2174/1874836801711010182; license: cc-by.

[Assistant] Zhu, "The design and construction of Komatsu combine shield used in sand-pebble stratum", Constr. Mechanization, vol. 32, no. 11, pp. 67-69, 2011. [In Chinese]. [http://dx.doi.org/10.3969/j.issn.1001-1366.2011.11.025] [32] F. He, "Analysis on adaptability of slurry shield cutterhead and cutters in Beijing railway underground connecting line project", Tunnel Constr., vol. 31, no. 4, pp. 416-425, 2011. [33] W. Burger, "Interventions and chamber access in pressurized face TBMs", In: Rapid Excavation and Tunneling Conference Proceedings, 2011, pp. 1036-1047. [34] W. Burger, "Hard rock cutterhead design", In: Proceedings and Monographs in Engineering, Water and Earth Science, 2006, pp. 257-263. [35] F. Enrique, L.M. Juan, and S. Alejandro, "Technical approach on big preparation to succeed on the Alaskan way project", In: Rapid Excavation and Tunneling Conference Proceedings, 2011, pp. 102-118. © 2017 Li et al. This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International Public License (CC-BY 4.0), a copy of which is available at: https://creativecommons.org/licenses/by/4.0/legalcode. This license permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. View publication stats

Cutterhead and Cutting Tools Configurations in Coarse Grain Soils


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Detection of Ionospheric Scintillation Based on XGBoost Model Improved by SMOTE-ENN Technique; DOI: 10.3390/rs13132577; license: cc-by-sa.

[Assistant]
2020, 125, 2019JA027487. 3. Karan, D.K.; Daniell, R.E.; England, S.L.; Martinis, C.R.; Eastes, R.W.; Burns, A.G.; Mcclintock, W.E. First zonal drift velocity measurement of Equatorial Plasma Bubbles (EPBs) from a geostationary orbit using GOLD data. J. Geophys. Res. Space Phys. 2020, 125, e2020JA028173. [CrossRef] 4. Martinis, C.; Daniell, R.; Eastes, R.; Norrell, J.; Smith, J.; Klenzing, J.; Solomon, S.; Burns, A. Longitudinal variation of post-sunset plasma depletions from the Global-scale Observations of the Limb and Disk (GOLD) mission. J. Geophys. Res. Space Phys. 2020, 126, 2020JA028510. 5. Liu, J.; Wang, W.; Burns, A.; Solomon, S.C.; Zhang, S.; Zhang, Y.; Huang, C. Relative importance of horizontal and vertical transports to the formation of ionospheric storm-enhanced density and polar tongue of ionization. J. Geophys. Res. Space Phys. 2016, 121, 8121–8133. [CrossRef] 6. Basu, S.; MacKenzie, E.; Basu, S. Ionospheric constraints on VHF/UHF communications links during solar maximum and minimum periods. Radio Sci. 1988, 23, 363–378. [CrossRef] 7. Sreeja, V.; Aquino, M.; Elmas, Z.G. Impact of ionospheric scintillation on GNSS receiver tracking performance over Latin America: Introducing the concept of tracking jitter variance maps. Space Weather 2011, 9, S10002. [CrossRef] 8. Aarons, J.; Whitney, H.E.; Allen, R.S. Global morphology of ionospheric scintillations. Proc. IEEE 1971, 59, 159–172. [CrossRef] 9. Knepp, D.L. Radar measurement of ionospheric scintillation in the polar region. Radio Sci. 2015, 50, 968–982. [CrossRef] 10. Jiao, Y.; Morton, Y.T. Comparison of the effect of high-latitude and equatorial ionospheric scintillation on GPS signals during the maximum of solar cycle 24. Radio Sci. 2015, 50, 886–903. [CrossRef] 11. Prasad, S.N.V.S.; Rama Rao, P.V.S.; Prasad, D.S.V.V.D.; Venkatesh, K.; Niranjan, K. Morphological studies on ionospheric VHF scintillations over an Indian low latitude station during a solar cycle period (2001–2010). Adv. Space Res. 2012, 50, 56–69. [CrossRef] 12. Banville, S.; Langley, R.B.; Saito, S.; Yoshihara, T. Handling cycle slips in GPS data during ionospheric plasma bubble events. Radio Sci. 2010, 45, 1–14. [CrossRef] 13. Ji, S.; Chen, W.; Weng, D.; Wang, Z.; Ding, X. A study on cycle slip detection and correction in case of ionospheric scintillation. Adv. Space Res. 2013, 51, 742–753. [CrossRef] 14. Taylor, S.; Morton, Y.; Jiao, Y.; Triplett, J.; Pelgrum, W. An improved ionosphere scintillation event detection and automatic trigger for a GNSS data collection system. In Proceedings of the Institute of Navigation International Technical Meeting 2012, Newport Beach, CA, USA, 30 January–1 February 2012; pp. 1563–1569. 15. Dubey, S.; Wahi, R.; Gwal, A.K. Ionospheric effects on GPS positioning. Adv. Space Res. 2006, 38, 2478–2484. [CrossRef] 16. Linty, N.; Farasin, A.; Favenza, A.; Dovis, F. Detection of GNSS ionospheric scintillations based on machine learning decision tree. IEEE Trans. Aerosp. Electron. Syst. 2019, 55, 303–317. [CrossRef]
[page 21] Remote Sens. 2021, 13, 2577 21 of 22 17. Mushini, S.C.; Jayachandran, P.T.; Langley, R.B.; MacDougall, J.W.; Pokhotelov, D. Improved amplitude- and phase-scintillation indices derived from wavelet detrended high-latitude GPS data. GPS Solut. 2012, 16, 363–373. [CrossRef] 18. Ouassou, M.; Kristiansen, O.; Gjevestad, J.G.O.; Jacobsen, K.S.; Andalsvik, Y.L. Estimation of scintillation indices: A novel approach based on local kernel regression methods. Int. J. Navig. Obs. 2016. [CrossRef] 19. Jiao, Y.; Hall, J.; Morton, Y. Performance evaluations of an equatorial GPS amplitude scintillation detector using a machine learning algorithm. In Proceedings of the 29th International Technical Meeting of the Satellite Division of the Institute of Navigation, Portland, OR, USA, 12–16 September 2016; pp. 195–199. 20. Jiao, Y.; Hall, J.J.; Morton, Y.T. Automatic equatorial GPS amplitude scintillation detection using a machine learning algorithm. IEEE Trans. Aerosp. Electron. Syst. 2017, 53, 405–418. [CrossRef] 21. Jiao, Y.; Hall, J.; Morton, Y. Automatic GPS phase scintillation detector using a machine learning algorithm. In Proceedings of the 2017 International Technical Meeting of The Institute of Navigation, Monterey, CA, USA, 30 January–2 February 2017; pp. 1160–1172. 22. Jiao, Y.; Hall, J.J.; Morton, Y.T. Performance Evaluation of an automatic GPS ionospheric phase scintillation detector using a machine-learning algorithm. Navig. J. Inst. Navig.


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Two-phase multi-model automatic brain tumour diagnosis system from magnetic resonance images using convolutional neural networks; DOI: 10.1186/s13640-018-0332-4; license: cc-by-sa.

[Assistant] Ardeshir, in 2003 International Conference on Visual Information Engineering, VIE 2003. Boosted ECOC ensembles for face recognition (IET, Guildford, 2003), pp. 165–168 41. J. Kittler, R. Ghaderi, T. Windeatt, J. Matas, in Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR 2001, vol 1. Face verification using error correcting output codes, (2001), pp. 755–760 42. R. Ghani, in Proceedings of the 2001 IEEE International Conference on Data Mining. Combining labeled and unlabeled data for text classification with a large number of categories, (2001), pp. 597–598 43. J. Zhou, C. Y. Suen, in Eighth International Conference on Document Analysis and Recognition, ICDAR’05, vol 1. Unconstrained numeral pair recognition using enhanced error correcting output coding: a holistic approach, (2005), pp. 484–488 44. F. Deng, S. Guo, R. Zhou, J. Chen, Sensor multifault diagnosis with improved support vector machines. IEEE Trans. Autom. Sci. Eng. PP(99), 1–11 (2015) 45. R. Girshick, Fast R-CNN. International Conference on Computer Vision (ICCV), Santiago, Chile, 11–18 December, 2015. (IEEE, Santiago, 2015), pp. 1440–1448 46. P. Reverdy, N. E. Leonard, Parameter estimation in softmax decision-making models with linear objective functions. IEEE Trans. Autom. Sci. Eng. 13(1), 54–67 (2016) 47. C. M. Bishop, Pattern recognition and machine learning. (Springer, 2006) 48. B. Yuan, in 2016 29th IEEE International System-on-Chip Conference (SOCC). Efficient hardware architecture of softmax layer in deep neural network (IEEE, Seattle, 2016), pp. 323–326

Two-phase multi-model automatic brain tumour diagnosis system from magnetic resonance images using convolutional neural networks


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Automated Excavator Based on Reinforcement Learning and Multibody System Dynamics; DOI: 10.1109/access.2020.3040246; license: cc-by-sa.

[Assistant]
The rbn vector has the following form: rbn =  x y z e0 e1 e2 e3 T (22) where x, y and z are XYZ-axes positions of the body, e0, e1, e2 and e3 are the Euler parameters of the body. In rbn vectors stored locations and rotations of the Upper Carriage, Lift Boom, Dipper Arm and Bucket bodies. The rg stores the positions of the goals and have a size of 6. It has the following form: rg =  xg1 yg1 zg1 xg2 yg2 zg2 T (23) where xg1, yg1, zg1 are positions of Goal To Soil and xg2, yg2, zg2 are positions of Goal To Hopper markers. In total, the observation vector consists of 42 parameters. Before development of the reward function, it is important to understand the task given to the agent. Multiple excavator operations exist in real life. For instance, in construction of underground pipelines, the excavator must create a trench, which requires so-called bucket leveling. On the other hand, mining work requires movement of soil from the ground into a truck without hitting the truck. Clearly, each operation requires a different approach, but they all involve movement of soil from one point to another. Therefore, movement of soil was chosen as the basis of the experiment. The selected task consists of three steps: loading the bucket near the loading point, moving to the unloading point, and unloading the bucket. In the ﬁrst step, the agent moves the bucket to the loading position and grabs the soil. During this step, it is important to move the bucket quickly and to grab the maximum amount of soil. After loading of the bucket, the soil must be moved to the unloading point over the hopper. This step requires the agent to move the bucket with little or no loss of soil. In the ﬁnal step, the agent must unload the maximum amount of soil without hitting the hopper. The setup of the simulation is shown in Fig.6. The environ- ment consists of the soil model, points of interest for the task, VOLUME 8, 2020 214003
[page 7] I. Kurinov et al.: Automated Excavator Based on RL and Multibody System Dynamics FIGURE 6. Simulation setup. The excavator is located uphill of the hopper and the unloading hopper is positioned to the right of the excavator. and a hopper. The soil model allows collection and transfer of soil, which is essential information for training. Points of interest provide the agent with information about soil grabbing and unloading positions. The hopper is equipped with two sensors. The ﬁrst sensor collects data about the mass in the hopper, and the second sensor tracks the number of collisions with the bucket. Using this setup, it is possible to track all the variables needed for formulation of a reward function. The reward function of the excavator model utilizes three components. The ﬁrst component is a reward for reaching the point where the bucket should be loaded. This point is marked as a blue dot in Fig.6. The second component is a reward for the mass collected in the bucket during loading. For preventing abuse of soil mass reward, it is inverse propor- tional to episode time. The third component utilizes the mass in the hopper, which is penalized by the number of collisions between the hopper and bucket. Thus, the reward function can be written as: r (t) = e−l + kmbntn + 1 nc mhn (24) where r(t) is a reward at the time t, k is the mass reward coef- ﬁcient, l is the distance to the current point of interest, mbn is the normalized mass in the bucket, mhn is the normalized mass in the hopper, and nc is the number of collisions with the hopper. IV. RESULTS The excavator model is highly nonlinear and includes mechanics, hydraulics, sensor measurements, a chang- ing environment and process visualization. Consequently, the training process is challenging. In the simulation, the agent learnt acceptable performance after 105 steps, which took around two days of continuous training. Comput- ing efﬁciency was affected by the deformable ground sub- model, which could not be updated without reloading the simulation. Therefore, a signiﬁcant amount of training time was spent on reloading the simulation ﬁle. FIGURE 7. Result of the training process. Fig.7 shows the result of the training process. As can be seen from the ﬁgure, the excavator grabs soil at the desired loading position.


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Spectral Angle Mapping and AI Methods Applied in Automatic Identification of Placer Deposit Magnetite Using Multispectral Camera Mounted on UAV; DOI: 10.3390/min12020268; license: cc-by-sa.

[Assistant]
Thereafter, the user pinpoints the area of signiﬁcance, which in this case was any area within the multispectral image scene known to be occupied by magnetite iron sand. Lastly, the algorithm performs a cosine similarity calculation for every pixel in each image scene to determine whether or not there are similarities between the reference and subsequent subjects [11]. To determine the cosine similarity threshold, Equation (1) is used. a = cos−1  t × t0 ∥t∥× ∥t0∥  (1) where a is the threshold for variables related to the reference, t0 is the reference spectrum, and t is the subject’s true spectrum. Having successfully stacked the 6 bands TIFF and JPG qualitative images, as well as performing SAM analysis, the images are then converted to csv format. This data conver- sion provides quantitative values pertaining to magnetite iron sand, and non-magnetite beach sand spectral characteristics, across the 6 multispectral bands. This data structure is useful in training AI algorithms. 2.4. Beneﬁts of Employing AI Methods in Magnetite Identiﬁcation In order to determine regions of an image where magnetite iron sands exist post SAM analysis, ML and DL models are an upheld method by which this task can be performed quantitatively [15]. This step involves pre-processing the data before training the AI algorithms. Pre-processing is important as it cleans the data and makes it easier for the algorithms to interpret it; for this study, the pre-processing step involved the extraction of the SAM segmentation maps per image. This simply means magnetite and non-magnetite post SAM spectral variables, given their cosine similarity thresholds, are assigned labels so as to take advantage of supervised AI algorithms. Hence, this process is referred to as image segmentation, and is simply deﬁned as the labelling of each pixel in an image scene [15]. Pixels within and outside the threshold are assigned magnetite and non-magnetite labels, respectively. Since SAM is mostly a visual interpretation, ground truthing is needed in order to employ AI algorithms. For this reason, the SAM segmentation map is extracted together with two ground truths for magnetite and non-magnetite pixels. Ground truths are image pixels known to be 100% occupied by magnetite and non-magnetite pixels per image. Since each image is 6 bands deep, the result is a total of 12 ground truths. Ground truth for magnetite is deﬁned as the target, whereas non-magnetite ground truth is deﬁned as noise. Labelled datasets guide and allow algorithms to take advantage of both the global and local structures of the data, which in turn results in quicker and more direct classiﬁca- tions [10]. This in essence means that for each image, the csv ﬁle contains labelled pixels for both magnetite iron sands (together with target ground truth), as well as non-magnetite beach sand (together with noise ground truth). Using this data as input, the algorithms are subsequently trained and outputs pertaining to the classiﬁcation capabilities of the models in deducing magnetite iron sands and non-magnetite beach sand are presented. 2.4.1. How Machine Learning Algorithms Operate DL is a subset of ML in the broader subject of AI [10]. Though these algorithms are useful in multitudes of data-related tasks such as the classiﬁcation problem being investigated in this study, they differ in several ways. In order to infer the top performing model at each UAV ﬂight height, both ML and DL AI algorithms are trained. ML algorithms are mathematical engines of AI, which means these algorithms attribute their classiﬁcation abilities to perceived mathematical relationships present within the data [19]. In simple terms, ML algorithms try to ﬁt the data within a particular pattern,
[page 7] Minerals 2022, 12, 268 7 of 19 which can be described using mathematical functions. Since these algorithms are relatively easy to employ, several ML algorithms were trained with regard to the ground truths for target and noise pixel variables. These algorithms were Decision Trees (often abbreviated as Trees) and Ensemble algorithms. Decision Tree algorithms were employed as they are non-parametric supervised learning methods, used for classiﬁcation where the goal is to make predictions by learning simple piecewise constant approximation decision rules from data features [25]. Therefore, deeper decision rules are referred to as ﬁne, whereas shallow and less complex decision rules are referred to as course.


[System] You are Gemma4Tunnel, a careful tunnel boring machine and tunnelling engineering assistant. Use engineering terminology precisely, state assumptions, and avoid inventing source claims.

[User]
Extract reusable tunnelling/TBM engineering knowledge from this allowed source excerpt. Preserve important parameters, mechanisms, risks, and caveats. Do not add unsupported claims.

Source: Reinforcement learning based process optimization and strategy development in conventional tunneling; DOI: 10.1016/j.autcon.2021.103701; license: cc-by.

[Assistant]
Ground type Specific weight [kN/m3] Cohesion [kPa] Friction angle [◦] Permeability [m/s] gt1 24 23 20 10−5 gt2 25 40 30 10−5 G.H. Erharter et al.
[page 5] Automation in Construction 127 (2021) 103701 5 The output of the third convolutional layer is flattened/vectorized before it is fed into the dense layer and we used the mean squared error as a loss function. 3.2.1. Actions The agent can choose from 8 different actions (Table 3). Possible actions are either top heading – or bench excavation with advance lengths of either 2 or 4 m. In each step the simulation carries out the following operations: ▪ the agent chooses and executes one of the 8 actions of Table 3 and the respective length of the generated geological section is revealed ▪ it is calculated/checked if the new position of the excavation face is within a stable area or not (details in Section 3.3.2) As given in Section 2.1, the support ahead of the face has the effect on the simulation that, there cannot be unstable conditions as long as the excavation face is within the supported area. Because of technical rea sons - mostly related to the reward system (see Table 4) - each action is assigned an “action code” (a). An a < 200 denotes top heading exca vation and a ≥200 bench excavation. The choice of actions of Table 3 is based on practical engineering experience on the one hand and technical limitations of the RL agent on the other hand. On the practical side, especially the advance length is Fig. 4. Top row: an exemplary unique geological section, where brown indicates weak (gt1) and blue stronger rock (gt2). The positions of the top heading and bench are at 165.0 m and 125.0 m respectively. Bottom row: the random walk that is used to generate the geological section. Values above 0.5 are converted to gt2 and below to gt1. Note that the x-axis is the tunnel length in decimeters which corresponds to the number of datapoints of the random walk. Table 2 Used hyperparameters for the DQN agent. Except for the exploration decay (see Section 3.2.1) all are identical in their meaning as the extended data Table 1 in [14]. Hyperparameter Value Replay memory size 100,000 Replay start size 1000 Minibatch size 64 Discount (γ) 0.99 Target network update frequency 10 Initial exploration (ε) 1 Final exploration (ε) 0.05 Learning rate 0.00025 Gradient momentum 0.95 Exploration decay (εd) 0.99997 Fig. 5. Schematic representation of the DQN agent’s ANN architecture. Note the visualization of rockmass-matrix and the support-matrix to the left. The numbers below each layer are the respective shape of the layer’s weights. Dashed connection lines between layers are only for illustrational purposes. Symbols at the output layer represent the eight possible actions (ordered as in Table 3) that are chosen via Q-values by the agent. Table 3 The eight possible actions the agent can choose from. Action code (a) Excavation Advance length [m] Face support 110 top heading 2 no 112 top heading 2 yes 150 top heading 4 no 152 top heading 4 yes 200 bench 2 no 202 bench 2 yes 220 bench 4 no 222 bench 4 yes G.H. Erharter et al.
[page 6] Automation in Construction 127 (2021) 103701 6 highly influential on the stability of the excavation with longer advance lengths being more prone to failures than short ones (e.g. advance lengths in the Austrian standard ¨ONORM B2203–1 [46]: 1.0, 1.3, 1.7, 2.2, 3.0 and 4.0 m). After experimenting with advances lengths based on the standards in the beginning, we experienced that a big number of actions leads to a deterioration of the performance/confuses the agent in this model setup. This is also in accordance with other studies which have found that special measures/adaptations of the agent are necessary if the action space becomes increasingly complex [47,48]. The final set of actions in Table 3 therefore aims at giving the agent realistic options to choose from while also keeping the number of actions small (see the outlook in Section 7 for a discussion on increasing the number of actions). During an episode, the actions are chosen based on an “ε-greedy action selection process”, (see also “exploration vs. exploitation trade off”, e.g. [26]). ε is the exploration rate and is initially set to 1 (see “initial exploration in Table 2).