Generalization through Simulation: Integrating Simulated and Real Data into Deep Reinforcement Learning for Vision-Based Autonomous Flight

Abstract: Deep reinforcement learning provides a promising approach for vision-based control of real-world robots. However, the generalization of such models depends critically on the quantity and variety of data available for training. This data can be difficult to obtain for some types of robotic systems, such as fragile, small-scale quadrotors. Simulated rendering and physics can provide for much larger datasets, but such data is inherently of lower quality: many of the phenomena that make the real-world autonomous flight problem challenging, such as complex physics and air currents, are modeled poorly or not at all, and the systematic differences between simulation and the real world are typically impossible to eliminate. In this work, we investigate how data from both simulation and the real world can be combined in a hybrid deep reinforcement learning algorithm. Our method uses real-world data to learn about the dynamics of the system, and simulated data to learn a generalizable perception system that can enable the robot to avoid collisions using only a monocular camera. We demonstrate our approach on a real-world nano aerial vehicle collision avoidance task, showing that with only an hour of real-world data, the quadrotor can avoid collisions in new environments with various lighting conditions and geometry. Code, instructions for building the aerial vehicles, and videos of the experiments can be found at github.com/gkahn13/GtS

2019 International Conference on Robotics and Automation (ICRA)


Adaptive low-level control of autonomous underwater vehicles using deep reinforcement learning

Abstract: Abstract Low-level control of autonomous underwater vehicles (AUVs) has been extensively addressed by classical control techniques. However, the variable operating conditions and hostile environments faced by AUVs have driven researchers towards the formulation of adaptive control approaches. The reinforcement learning (RL) paradigm is a powerful framework which has been applied in different formulations of adaptive control strategies for AUVs. However, the limitations of RL approaches have lead towards the emergence of deep reinforcement learning which has become an attractive and promising framework for developing real adaptive control strategies to solve complex control problems for autonomous systems. However, most of the existing applications of deep RL use video images to train the decision making artificial agent but obtaining camera images only for an AUV control purpose could be costly in terms of energy consumption. Moreover, the rewards are not easily obtained directly from the video frames. In this work we develop a deep RL framework for adaptive control applications of AUVs based on an actor-critic goal-oriented deep RL architecture, which takes the available raw sensory information as input and as output the continuous control actions which are the low-level commands for the AUV’s thrusters. Experiments on a real AUV demonstrate the applicability of the stated deep RL approach for an autonomous robot control problem.


Model predictive control of autonomous mobility-on-demand systems

Abstract: In this paper we present a model predictive control (MPC) approach to optimize vehicle scheduling and routing in an autonomous mobility-on-demand (AMoD) system. In AMoD systems, robotic, self-driving vehicles transport customers within an urban environment and are coordinated to optimize service throughout the entire network. Specifically, we first propose a novel discrete-time model of an AMoD system and we show that this formulation allows the easy integration of a number of real-world constraints, e.g., electric vehicle charging constraints. Second, leveraging our model, we design a model predictive control algorithm for the optimal coordination of an AMoD system and prove its stability in the sense of Lyapunov. At each optimization step, the vehicle scheduling and routing problem is solved as a mixed integer linear program (MILP) where the decision variables are binary variables representing whether a vehicle will 1) wait at a station, 2) service a customer, or 3) rebalance to another station. Finally, by using real-world data, we show that the MPC algorithm can be run in real-time for moderately-sized systems and outperforms previous control strategies for AMoD systems.

2016 IEEE International Conference on Robotics and Automation (ICRA)


A Dexterous Robotic System for Autonomous Debridement of Osteolytic Bone Lesions in Confined Spaces: Human Cadaver Studies

Abstract: This article presents a dexterous robotic system for autonomous debridement of osteolytic bone lesions in confined spaces. The proposed system is distinguished from the state-of-the-art orthopedics systems because it combines a rigid-link robot with a continuum manipulator (CM) that enhances reach in difficult-to-access spaces often encountered in surgery. The CM is equipped with flexible debriding instruments and fiber Bragg grating sensors. The surgeon plans on the patient’s preoperative computed tomography and the robotic system performs the task autonomously under the surgeon’s supervision. An optimization-based controller generates control commands on the fly to execute the task while satisfying physical and safety constraints. The system design and controller are discussed and extensive simulation, phantom and human cadaver experiments are carried out to evaluate the performance, workspace, and dexterity in confined spaces. Mean and standard deviation of target placement are 0.5 and 0.18 mm, and the robotic system covers 91% of the workspace behind an acetabular implant in treatment of hip osteolysis, compared to the 54% that is achieved by conventional rigid tools.

IEEE transactions on robotics : a publication of the IEEE Robotics and Automation Society


Image processing using Pearson's correlation coefficient: Applications on autonomous robotics

2013 13th International Conference on Autonomous Robot Systems


WareVR: Virtual Reality Interface for Supervision of Autonomous Robotic System Aimed at Warehouse Stocktaking

Abstract: WareVR is a novel human-robot interface based on a virtual reality (VR) application to interact with a heterogeneous robotic system for automated inventory management. We have created an interface to supervise an autonomous robot remotely from a secluded workstation in a warehouse that could benefit during the current pandemic COVID-19 since the stocktaking is a necessary and regular process in warehouses, which involves a group of people. The proposed interface allows regular warehouse workers without experience in robotics to control the heterogeneous robotic system consisting of an unmanned ground vehicle (UGV) and unmanned aerial vehicle (UAV). WareVR provides visualization of the robotic system in a digital twin of the warehouse, which is accompanied by a real-time video stream from the real environment through an onboard UAV camera. Using the WareVR interface, the operator can conduct different levels of stocktaking, monitor the inventory process remotely, and teleoperate the drone for a more detailed inspection. Besides, the developed interface includes remote control of the UAV for intuitive and straightforward human interaction with the autonomous robot for stocktaking. The effectiveness of the VR-based interface was evaluated through the user study in a “visual inspection” scenario.

IEEE International Conference on Systems, Man and Cybernetics

2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC)


Towards an Autonomous Robot-Assistant for Laparoscopy Using Exteroceptive Sensors: Feasibility Study and Implementation


Scalable Multi-Task Imitation Learning with Autonomous Improvement

Abstract: While robot learning has demonstrated promising results for enabling robots to automatically acquire new skills, a critical challenge in deploying learning-based systems is scale: acquiring enough data for the robot to effectively generalize broadly. Imitation learning, in particular, has remained a stable and powerful approach for robot learning, but critically relies on expert operators for data collection. In this work, we target this challenge, aiming to build an imitation learning system that can continuously improve through autonomous data collection, while simultaneously avoiding the explicit use of reinforcement learning, to maintain the stability, simplicity, and scalability of supervised imitation. To accomplish this, we cast the problem of imitation with autonomous improvement into a multi-task setting. We utilize the insight that, in a multi-task setting, a failed attempt at one task might represent a successful attempt at another task. This allows us to leverage the robot’s own trials as demonstrations for tasks other than the one that the robot actually attempted. Using an initial dataset of multitask demonstration data, the robot autonomously collects trials which are only sparsely labeled with a binary indication of whether the trial accomplished any useful task or not. We then embed the trials into a learned latent space of tasks, trained using only the initial demonstration dataset, to draw similarities between various trials, enabling the robot to achieve one-shot generalization to new tasks. In contrast to prior imitation learning approaches, our method can autonomously collect data with sparse supervision for continuous improvement, and in contrast to reinforcement learning algorithms, our method can effectively improve from sparse, task-agnostic reward signals.

2020 IEEE International Conference on Robotics and Automation (ICRA)


A Multi-task Convolutional Neural Network for Autonomous Robotic Grasping in Object Stacking Scenes

Abstract: Autonomous robotic grasping plays an important role in intelligent robotics. However, how to help the robot grasp specific objects in object stacking scenes is still an open problem, because there are two main challenges for autonomous robots: (1) it is a comprehensive task to know what and how to grasp; (2) it is hard to deal with the situations in which the target is hidden or covered by other objects. In this paper, we propose a multi-task convolutional neural network for autonomous robotic grasping, which can help the robot find the target, make the plan for grasping and finally grasp the target step by step in object stacking scenes. We integrate vision-based robotic grasping detection and visual manipulation relationship reasoning in one single deep network and build the autonomous robotic grasping system. Experimental results demonstrate that with our model, Baxter robot can autonomously grasp the target with a success rate of 90.6%, 71.9% and 59.4% in object cluttered scenes, familiar stacking scenes and complex stacking scenes respectively.

IEEE/RJS International Conference on Intelligent RObots and Systems

2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)


Towards autonomous inspection of concrete deterioration in sewers with legged robots

Abstract: The regular inspection of sewer systems is essential to assess the level of degradation and to plan maintenance work. Currently, human inspectors must walk through sewers and use their sense of touch to inspect the roughness of the floor and check for cracks. The sense of touch is used since the floor is often covered by (waste) water and biofilm, which renders visual inspection very challenging. In this paper, we demonstrate a robotic inspection system which evaluates concrete deterioration using tactile interaction. We deployed the quadruped robot ANYmal in the sewers of Zurich and commanded it using shared autonomy for several such missions. The inspection itself is realized via a well‐defined scratching motion using one of the limbs on the sewer floor. Inertial and force/torque sensors embedded within specially designed feet captured the resulting vibrations. A pretrained support vector machine (SVM) is evaluated to assess the state of the concrete. The results of the classification are then displayed in a three‐dimensional map recorded by the robot for easy visualization and assessment. To train the SVM we recorded 625 samples with ground truth labels provided by professional sewer inspectors. We make this data set publicly available. We achieved deterioration level estimates within three classes of more than 92% accuracy. During the four deployment missions, we covered a total distance of 300 m and acquired 130 inspection samples.


An ant-inspired celestial compass applied to autonomous outdoor robot navigation

Abstract: Abstract Desert ants use the polarization of skylight and a combination of stride and ventral optic flow integration processes to track the nest and food positions when traveling, achieving outstanding performances. Navigation sensors such as global positioning systems and inertial measurement units still have disadvantages such as their low resolution and drift. Taking our inspiration from ants, we developed a 2-pixel celestial compass which computes the heading angle of a mobile robot in the ultraviolet range. The output signals obtained with this optical compass were investigated under various weather and ultraviolet conditions and compared with those obtained on a magnetometer in the vicinity of our laboratory. After being embedded on-board the robot, the sensor was first used to compensate for random yaw disturbances. We then used the compass to keep the Hexabot robot’s heading angle constant in a straight forward walking task over a flat terrain while its walking movements were imposing yaw disturbances. Experiments performed under various meteorological conditions showed the occurrence of steady state heading angle errors ranging from 0 . 3 ∘ (with a clear sky) to 2 . 9 ∘ (under changeable sky conditions). The compass was also tested under canopies and showed a strong ability to determine the robot’s heading while most of the sky was hidden by the foliage. Lastly, a waterproof, mono-pixel version of the sensor was designed and successfully tested in a preliminary underwater benchmark test. These results suggest this new optical compass shows great precision and reliability in a wide range of outdoor conditions, which makes it highly suitable for autonomous robotic outdoor navigation tasks. A celestial compass and a minimalistic optic flow sensor called M 2 APix (based on Michaelis–Menten Auto-adaptive Pixels) were therefore embedded on-board our latest insectoid robot called AntBot, to complete the previously mentioned ant-like homing navigation processes. First the robot was displaced manually and made to return to its starting-point on the basis of its absolute knowledge of the coordinates of this point. Lastly, AntBot was tested in fully autonomous navigation experiments, in which it explored its environment and then returned to base using the same sensory modes as those on which desert ants rely. AntBot produced robust, precise localization performances with a homing error as small as 0.7% of the entire trajectory.


Evolutionary Modular Robotics: Survey and Analysis

Abstract: This paper surveys various applications of artificial evolution in the field of modular robots. Evolutionary robotics aims to design autonomous adaptive robots automatically that can evolve to accomplish a specific task while adapting to environmental changes. A number of studies have demonstrated the feasibility of evolutionary algorithms for generating robotic control and morphology. However, a huge challenge faced was how to manufacture these robots. Therefore, modular robots were employed to simplify robotic evolution and their implementation in real hardware. Consequently, more research work has emerged on using evolutionary computation to design modular robots rather than using traditional hand design approaches in order to avoid cognition bias. These techniques have the potential of developing adaptive robots that can achieve tasks not fully understood by human designers. Furthermore, evolutionary algorithms were studied to generate global modular robotic behaviors including; self-assembly, self-reconfiguration, self-repair, and self-reproduction. These characteristics allow modular robots to explore unstructured and hazardous environments. In order to accomplish the aforementioned evolutionary modular robotic promises, this paper reviews current research on evolutionary robotics and modular robots. The motivation behind this work is to identify the most promising methods that can lead to developing autonomous adaptive robotic systems that require the minimum task related knowledge on the designer side.

Journal of Intelligent and Robotic Systems

Journal of Intelligent & Robotic Systems


Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search

Abstract: Model predictive control (MPC) is an effective method for controlling robotic systems, particularly autonomous aerial vehicles such as quadcopters. However, application of MPC can be computationally demanding, and typically requires estimating the state of the system, which can be challenging in complex, unstructured environments. Reinforcement learning can in principle forego the need for explicit state estimation and acquire a policy that directly maps sensor readings to actions, but is difficult to apply to unstable systems that are liable to fail catastrophically during training before an effective policy has been found. We propose to combine MPC with reinforcement learning in the framework of guided policy search, where MPC is used to generate data at training time, under full state observations provided by an instrumented training environment. This data is used to train a deep neural network policy, which is allowed to access only the raw observations from the vehicle's onboard sensors. After training, the neural network policy can successfully control the robot without knowledge of the full state, and at a fraction of the computational cost of MPC. We evaluate our method by learning obstacle avoidance policies for a simulated quadrotor, using simulated onboard sensors and no explicit state estimation at test time.


Gridbot: An autonomous robot controlled by a Spiking Neural Network mimicking the brain's navigational system

Abstract: It is true that the "best" neural network is not necessarily the one with the most "brain-like" behavior. Understanding biological intelligence, however, is a fundamental goal for several distinct disciplines. Translating our understanding of intelligence to machines is a fundamental problem in robotics. Propelled by new advancements in Neuroscience, we developed a spiking neural network (SNN) that draws from mounting experimental evidence that a number of individual neurons is associated with spatial navigation. By following the brain's structure, our model assumes no initial all-to-all connectivity, which could inhibit its translation to a neuromorphic hardware, and learns an uncharted territory by mapping its identified components into a limited number of neural representations, through spike-timing dependent plasticity (STDP). In our ongoing effort to employ a bioinspired SNN-controlled robot to real-world spatial mapping applications, we demonstrate here how an SNN may robustly control an autonomous robot in mapping and exploring an unknown environment, while compensating for its own intrinsic hardware imperfections, such as partial or total loss of visual input.

Proceedings of the International Conference on Neuromorphic Systems


A Robot to Monitor Nuclear Facilities: Using Autonomous Radiation-Monitoring Assistance to Reduce Risk and Cost

Abstract: Nuclear facilities often require continuous monitoring to ensure there is no contamination of radioactive materials that might lead to safety or environmental issues. The current approach to radiological monitoring is to use human operators, which is both time consuming and cost inefficient. As with many repetitive, routine tasks, there are considerable opportunities for the process to be improved using autonomous robotic systems.


A survey of autonomous self-reconfiguration methods for robot-based programmable matter

Abstract: Abstract While researchers envision exciting applications for metamorphic systems like programmable matter, current solutions to the shape formation problem are still a long way from meeting their requirements. To dive deeper into this issue, we propose an extensive survey of the current state of the art of self/reconfiguration algorithms and underlying models in modular robotic and self-organizing particle systems. We identify three approaches for solving this problem and we compare the different solutions using a synoptic graphical representation. We then close this survey by confronting existing methods to our vision of programmable matter, and by discussing a number of future research directions that would bring us closer to making it a reality.


Supervised Autonomous Locomotion and Manipulation for Disaster Response with a Centaur-Like Robot

Abstract: Mobile manipulation tasks are one of the key challenges in the field of search and rescue (SAR) robotics requiring robots with flexible locomotion and manipulation abilities. Since the tasks are mostly unknown in advance, the robot has to adapt to a wide variety of terrains and workspaces during a mission. The centaur-like robot Centauro has a hybrid legged-wheeled base and an anthropomorphic upper body to carry out complex tasks in environments too dangerous for humans. Due to its high number of degrees of freedom, controlling the robot with direct teleoperation approaches is challenging and exhausting. Supervised autonomy approaches are promising to increase quality and speed of control while keeping the flexibility to solve unknown tasks. We developed a set of operator assistance functionalities with different levels of autonomy to control the robot for challenging locomotion and manipulation tasks. The integrated system was evaluated in disaster response scenarios and showed promising performance.

2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)


Toward a Fully Autonomous UAV: Research Platform for Indoor and Outdoor Urban Search and Rescue


Autonomous Learning of Object Models on a Mobile Robot


Intelligent Autonomous Systems


Online Incremental Learning of the Terrain Traversal Cost in Autonomous Exploration

Abstract: (cid:151)In this paper, we address motion ef(cid:2)ciency in autonomous robot exploration with multi-legged walking robots that can traverse rough terrains at the cost of lower ef(cid:2)ciency and greater body vibration. We propose a robotic system for online and incremental learning of the terrain traversal cost that is immediately utilized to reason about next navigational goals in building spatial model of the robot surrounding. The traversal cost experienced by the robot is characterized by incrementally constructed Gaussian Processes using Bayesian Committee Machine. During the exploration, the robot builds the spatial terrain model, marks untraversable areas, and leverages the Gaussian Process predictive variance to decide whether to improve the spatial model or decrease the uncertainty of the terrain traversal cost. The feasibility of the proposed approach has been experimentally veri(cid:2)ed in a fully autonomous deployment with the hexapod walking robot.

Robotics: Science and Systems Conference


Enabling autonomous capabilities in underwater robotics

2008 IEEE/RSJ International Conference on Intelligent Robots and Systems


Autonomous Battery Exchange of UAVs with a Mobile Ground Base

2018 IEEE International Conference on Robotics and Automation (ICRA)


Learning from demonstration: repetitive movements for autonomous service robotics

2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566)


Autonomous Adaptive Modification of Unstructured Environments

Abstract: We present and validate a property-driven autonomous system that modifies its environment to achieve and maintain navigability over a highly irregular 3-dimensional terrain. In our approach we use decision procedures that tie building actions to the terrain model, giving rise to adaptive and robust building behavior. The building algorithm is driven by continuous evaluation and reaction to terrain properties, rather than relying on a structure blueprint. This capability is essential in robotic systems that operate in unstructured outdoor or remote environments, either on their own or as part of a team. We demonstrate the effectiveness of our approach by running a lowcost robot system that can build with compliant bags in a variety of irregular terrains.


Perception-Informed Autonomous Environment Augmentation with Modular Robots

Abstract: We present a system enabling a modular robot to autonomously build structures in order to accomplish high-level tasks. Building structures allows the robot to surmount large obstacles, expanding the set of tasks it can perform. This addresses a common weakness of modular robot systems, which often struggle to traverse large obstacles. This paper presents the hardware, perception, and planning tools that comprise our system. An environment characterization algorithm identifies features in the environment that can be augmented to create a path between two disconnected regions of the environment. Specially-designed building blocks enable the robot to create structures that can augment the environment to make obstacles traversable. A high-level planner reasons about the task, robot locomotion capabilities, and environment to decide if and where to augment the environment in order to perform the desired task. We validate our system in hardware experiments.


The LRU Rover for Autonomous Planetary Exploration and Its Success in the SpaceBotCamp Challenge

2016 International Conference on Autonomous Robot Systems and Competitions (ICARSC)