A Comprehensive Survey of Deep Reinforcement Learning Techniques for Soft Mobile Robots

Although several surveys have reviewed soft robotics [1,2] and others have summarized reinforcement learning in robotics [3,4], there is still no focused comparative survey specifically dedicated to the integration of Deep Reinforcement Learning (DRL) in soft mobile robots. Existing works either emphasize biology-inspired soft material technologies, or they analyze DRL algorithms primarily in rigid robotic platforms. This gap motivates the present survey, which consolidates current efforts, compares methodologies, and identifies the key research trends for DRL-driven soft mobile locomotion. Ijspeert and colleagues in 2007, in [5], presented a spinal cord model and its implementation in an amphibious salamander robot. They demonstrated how a primitive neural circuit for swimming can be extended using new physiogenetic limb oscillatory centers to explain the salamander’s ability to switch between swimming and walking, and vice versa. This model proposes neural mechanisms for adjusting speed, direction, and gait type, which are suitable for all quadrupeds. The model also predicts that the limb oscillatory centers have a lower intrinsic frequency than the body oscillatory centers, a prediction that is supported by the presented biological data. CPG-based (Central Pattern Generator) control helps reduce the dimensionality of the motion control problem while maintaining high flexibility to continuously adjust speed, direction, and gait type according to the environmental context (Figure 1). In 2008, he reviewed the research conducted on locomotor central pattern generators (CPGs), meaning neural circuits capable of generating coordinated patterns of rhythmic output signals with high dimensionality while receiving only simple, low-dimensional input signals. His review initially includes neurobiological observations related to locomotor CPGs and their numerical modeling, focusing on vertebrates. Then, the use of CPG models for controlling the locomotion of articulated robots and specific motion states is examined. He has analyzed the advantages and disadvantages of using CPGs with robots [6]. Ijspeert et al. in 2013, in [7], presented and reviewed the assumptions of Dynamic Movement Primitives (DMPs), which are a series of studies focused on modeling the attractor behaviors of autonomous nonlinear dynamical systems using statistical learning techniques. This general design principle is especially useful for modeling motor behaviors in robotics, as well as for modeling biological phenomena. The nature of their approach is such that it starts with a simple dynamical system, such as a set of linear differential equations, and transforms them into a weakly nonlinear system with prescribed attractor dynamics by means of a learnable forcing term. The nonlinear forcing term can be expressed as an independent coupling term that can be learned using standard machine learning techniques that are linear in the parameters. They explain the design principle of their approach and evaluate its properties in several practical examples in motor control and robotics. In Figure 1, the configuration of the CPG model and the salamander robot is shown. Jens Kober and his colleagues, in [3], attempted to strengthen the ties between the two research communities by reviewing the work on deep learning for behavior generation in robots. They highlighted both the key challenges in robot reinforcement learning and the notable successes. Furthermore, they studied the role of algorithms, representations, and prior knowledge in achieving these successes. As a result, the special focus of this research is on the selection between model-based and model-free methods, as well as between value function-based and policy search methods. By analyzing a simple problem in detail, they showed how reinforcement learning approaches can be profitably applied. Daniela Rus and Michael T. Tolley, in [1], discussed recent developments in the field of soft robotics. They also reviewed the design and construction of robots, actuators, stretchable electronic components, sensing systems, power sources, computation and control, modeling and kinematics, and applications of soft robots. In Figure 2, examples of mobile systems inspired by the motion of soft-bodied creatures are shown. Wang and colleagues, in [8], proposed learning from demonstration (LfD) for motion planning of arms based on ionic polymer-metal composite (IPMC) actuators using statistical machine learning algorithms. To encode demonstrated trajectories and estimate suitable paths for the arms to reproduce tasks, models are built based on Gaussian Mixture Models (GMM) and Gaussian Mixture Regression (GMR), respectively. In this study, forward and inverse kinematic models of the soft robotic arm based on IPMC were obtained for motion control. A flexible and soft robotic arm (Figure 3) with six IPMC segments was employed and validated the learned trajectories by successfully navigating through a narrow keyhole. This work proposes the introduction of the learning from demonstration (LfD) method for motion planning of a flexible and soft robotic arm based on IPMC for tasks such as passing through a hole. A skillful flexible IPMC arm with six segments was implemented to validate the learned paths from demonstration. In experiments, this arm was able to autonomously pass through a narrow opening (5 mm). With the current results, the IPMC arm could be used in complex tasks such as grasping tissue while navigating through a body cavity. Bern and colleagues, in [9], presented a class of tendon-actuated soft robots and ensured their low-cost and usability for non-expert users. The fabrication techniques they introduced are mainly based on traditional methods used for making plush toys (Figure 4). Movement of a plush robot is achieved by driving internal winches that pull (or push) tendons routed beneath the skin. They provided a forward simulation model to predict the deformation behavior of a plush robot given certain contractions of internal winches and used this in an interactive control scheme. They then explained how to invert this forward model to be used in an interactive control design. They also built a hexapod and a gripper to demonstrate the applicability of their system. In this design, the user selects a target position for the robot, and the optimal contractions of the robot’s winches are automatically computed in real-time. Zhang and colleagues, in [10], introduced an approach for soft robot control based on a joint effort to design an abstract representation of the soft robot’s state and to develop a reinforcement learning method for acquiring effective control policies (Figure 5). Since the reinforcement learning process can be trained rapidly by ignoring the specific materials and structural features of the soft robot, their approach was applied to a hexagonal PneuNet soft robot. The effectiveness of the training method and its ability to produce good control policies under varying conditions was demonstrated. Using model-free reinforcement learning, they ultimately obtained an action policy for a specific task and employed a simulator to overcome hardware limitations of the soft robot. Nakajima and colleagues, in [11], explored the use of diverse dynamics generated by stimulating soft materials, believing that such dynamics can be effectively utilized for machine learning purposes. To demonstrate this, they used a soft silicone robotic arm through a multiplexing technique. The results showed that the soft arm performs well with conventional machine learning techniques under various conditions. Their approach, on one hand, represents a fundamental departure from traditional computational methods, while on the other hand, aligns well with a broader perspective of computation by leveraging the physical properties of real-world materials. Figure 6 illustrates the soft silicone robotic arm immersed in water. Yang et al., in [12], presented a soft artificial muscle robot inspired by the cuttlefish, featuring a fully integrated onboard system, including a power source and wireless communication module (Figure 7). The cuttlefish robot’s movements are generated without any motors, relying solely on dielectric elastomer actuators that demonstrate muscle-like properties, such as large deformation and high energy density. Instead of manually tuning the controller, they utilized reinforcement learning to optimize the control approach for the cuttlefish robot. Initially, reinforcement learning improved the robot’s swimming speed by 91%, achieving 21 mm/s (0.38 body lengths per second). While the robot cannot yet change its direction, it is anticipated that directional control could be achieved using an additional soft actuator. Katzschmann et al., in [13], presented the design, fabrication, control, and ocean testing of a soft robotic fish capable of three-dimensional swimming and continuous aquatic tracking or engagement (Figure 8). This robot, powered by soft hydraulic actuators, demonstrates long-term continuous operation (several hundred meters over 40 min) with remote control underwater. Divers can guide the robotic fish using a miniature acoustic communication module by sending commands such as speed, turning angle, and dynamic diving depth. Experimental results from coral reef trials in the Pacific Ocean showed that the robotic fish can successfully navigate among aquatic life at depths ranging from 0 to 18 m. Furthermore, the presented robotic fish exhibits a wave-like tail motion driven by a soft robotic actuator design. Thuruthel et al., in [14], proposed a model-based policy learning algorithm for closed-loop predictive control of a soft robotic arm. The forward dynamics model is represented using a recurrent neural network, and the closed-loop policy is derived through trajectory optimization and supervised learning. The approach is experimentally demonstrated on a pneumatically actuated soft arm, showcasing the capability to learn a feedback control policy that accommodates varying actuation frequencies and unmodeled external loads (Figure 9). This work highlights how model-based reinforcement learning can be leveraged for robust, adaptive control in soft continuum manipulators. Homberg et al., in [15], introduced a soft robotic hand capable of robustly grasping objects using internal state measurements and a hybrid system that autonomously performs grasping and object classification based on flex sensor data. They designed a modular finger that incorporates resistive force sensors at the fingertip for contact detection and resistive flex sensors for measuring finger curvature characteristics (Figure 10). Without the need for complex programming, the hand can grasp a variety of objects. Their findings show that using a single data point from each finger, it is possible to identify the grasped object. Furthermore, a clustering algorithm was proposed to match objects between multiple grasping events, enhancing object recognition capabilities in soft robotic manipulation. Bhagat et al., in [4] present a comprehensive overview of various reinforcement learning algorithms, focusing particularly on those that have been successfully applied in real-world robotic control. They describe multiple learning paradigms and techniques, followed by their implementation examples in real-world robotic tasks, highlighting progress at the cutting edge of the field. This survey serves as a bridge between theoretical developments and practical implementations in robotic reinforcement learning. Haarnoja et al., in [16] propose a sample-efficient deep reinforcement learning algorithm based on maximum entropy RL. The method requires minimal hyperparameter tuning and few trials to learn neural network policies. They demonstrate its application to a real-world miniature robot for learning to walk (Figure 11). The robot achieves stable walking behavior in about two hours, without reliance on models or simulation. Despite its efficiency, the method lacks a safety layer, limiting its applicability to larger robots, and it requires manual resetting between episodes. Ishige et al., in [17], introduce a novel method called SenseCPG-PGPE, designed for caterpillar-like soft robots (Figure 12). This method integrates sensory mechanical feedback into a Central Pattern Generator (CPG)-based controller and optimizes it using Policy Gradient with Parameter-Based Exploration (PGPE). They show that PGPE can effectively optimize a CPG-based controller for soft robots exhibiting viscoelasticity and large deformations, where other conventional policy gradient methods such as Trust Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) fail to do so. Although the proposed SenseCPG-PGPE leverages a CPG-based controller, it is capable of achieving unstable locomotion behaviors such as climbing stairs in a simulated robot. This finding indicates that the method enables soft-bodied robots to autonomously discover a wide range of behaviors. The study also compares various reinforcement learning techniques, including Deep Deterministic Policy Gradient (DDPG), TRPO, PPO, and PGPE, concluding that only PGPE effectively enables desired behaviors in a worm-like soft robot due to its episodic parameter updates. Sathishbabu et al., in [18], propose a model-free deep reinforcement learning method for open-loop position control of a spatially continuous soft arm (Figure 13). The arm is pneumatically actuated and achieves spatial workspace coverage via combined one-sided bending and two-sided torsional deformation. Deep Q-learning with experience replay is used to train the system in simulation. The learned control policy is validated in both simulation and a physical prototype under various external loads. Notably, the system demonstrates robustness to external disturbances and self-weight, as well as the ability to reduce the number of transitions via efficient action selection. Park and Cha, in [19], introduce a soft-legged mobile robot composed of a bistable piezoelectric body and pre-curved piezoelectric legs, made of PVDF (Polyvinylidene Fluoride) films (Figure 14). The robot achieves locomotion speeds up to 70 percent of its body length per second. Numerical simulation is used to investigate the physical mechanisms underlying the locomotion. The study confirms that asymmetry in leg amplitude, dependent on actuation conditions, can influence performance. Moreover, due to its flexible structure, the soft-legged robot maintains locomotion even under impulsive shocks. Trobay et al., in [20], utilize embedded 3D printing to fabricate soft robotic fingers with discrete actuation modes and integrated ionogel soft sensors that provide tactile feedback corresponding to each degree of freedom. The sensory feedback of these fingers is evaluated through experiments such as free and blocked displacement, using readout electronics that simplify resistance measurement. By combining three such sensitized fingers, a soft robotic arm capable of different grasping configurations is constructed. The techniques presented in this work offer a promising fabrication platform for building multi-DoF soft arms requiring sensory feedback for closed-loop control and machine learning algorithms. Algorithm 1 illustrates the closed-loop object grasping algorithm used in this study. Berndt et al., in [21], argue that while physics-based modeling techniques such as the finite element method are promising, they are inherently constrained by physical assumptions. Soft robots in the real world are often constructed from unpredictable materials using imprecise fabrication methods, leading to significant modeling uncertainty. As a response, the authors present a novel machine learning-based approach for controlling soft robots. They use gradient-based optimization to learn a variable quasi-static physical model of a soft robot and determine optimal open-loop control inputs. The learned model captures phenomena absent in ideal physics-based simulations. Furthermore, the paper discusses the impact of network complexity on model accuracy and offers practical techniques for capturing high-quality motion data. Li et al., in [22], present a data-driven control framework that employs deep reinforcement learning (DRL) to address the locomotion control problem of a soft underwater robot (Figure 15). The robot is constructed with dielectric elastomer actuators (DEAs), enabling it to swim. Initially, a simulation model of the soft robot is developed for training the neural network controller. The trained network is then transferred to a physical prototype to evaluate real-world performance. The soft robot successfully achieves straight-line locomotion in turbulent water environments. Experimental results validate the potential of DRL in enhancing the locomotion capabilities of soft mobile robots. Reisinejad et al., in [23], propose two frameworks to optimize the mechanical performance of a pneumatically-driven soft actuator with multiple chambers. The geometric configuration and the air chamber layout are optimized separately using a firefly algorithm and deep reinforcement learning (DRL), respectively, in both model-based and finite element analysis (FEA)-based environments. The DRL-based approach is integrated with both modeling frameworks to fully explore the design space and enable cross-validation. Comparative results show that DRL-generated designs achieve the desired displacement while effectively decoupling vertical and horizontal deformations, making them suitable for practical soft robotic applications. Yin and Mueller [24] argue that one strategy to overcome current sensory input limitations is the integration of complex soft sensors with deep learning. They demonstrate this through a bio-inspired soft robotic sensor mimicking the rapid, non-rigid deformations of bat ears. Unlike traditional methods requiring multiple frequencies or pressure sensors, their approach works effectively using a single sensor and single frequency, showcasing the potential of deep learning in extracting rich features from minimal sensory inputs. Thanabalan [25] explores the modeling of a soft robotic actuator using a combination of finite element modeling (FEM) and deep neural networks for predicting bending and end-effector control. Using 1000 data samples generated from ANSYS simulations, the study finds that deep networks can accurately reproduce actuator behavior, suggesting DNNs as ideal tools for predictive modeling in soft robotics. Lin et al. [26] present a mechanically undulating control system for a fully soft robot with fish-like motion. The robot uses fast, bidirectional PneuNet actuators integrated with ultrasoft eGaIn pressure sensors in silicone channels (Figure 16). A simple, data-driven parametric control model, tuned via genetic algorithms, allows for lightweight and accurate simulation across a range of actuation frequencies and pressures, capturing key soft actuator dynamics such as bidirectional contraction. Kim et al. [2] provide a comprehensive review of machine learning techniques in soft robotics. Their work categorizes ML implementations into soft sensors, actuators, and wearable robots, and analyzes trends and application- specific strategies. The paper offers a concise summary of available ML methods and their relevance to different soft robotic domains. Wang et al. [27] synthesize the characteristics of soft arms and examine reinforcement learning (RL) applications in soft robotic arms. The paper discusses nature-inspired modeling and real-robot deployment of RL-based controllers, and provides a future outlook with suggestions for using various RL algorithms (Figure 17) in practical scenarios, based on their characteristics and suitability for soft robotic control. In Table 1 a brief abstract of the data in reviewed references is collected.