FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills

Read original: arXiv:2404.19217 - Published 5/2/2024 by Yongqiang Zhao, Kun Qian, Boyi Duan, Shan Luo

🤿

Overview

Simulation is a widely used tool in robotics to reduce hardware costs and collect large-scale data.
Despite previous efforts to simulate optical tactile sensors, there are still challenges in efficiently synthesizing images and replicating marker motion under different contact loads.
This paper proposes a fast optical tactile simulator called FOTS to simulate optical tactile sensors.

Plain English Explanation

Robots need to be able to sense touch and pressure in order to interact with the world around them. One way to simulate this is through optical tactile sensors. These sensors use cameras to detect changes in light patterns when the sensor surface is touched or deformed.

However, it can be difficult to accurately simulate the behavior of these sensors in software. The researchers in this paper developed a new simulation system called FOTS that can quickly and realistically generate images and track the movement of markers on the sensor surface when different amounts of force are applied.

The key innovations in FOTS are:

Using multi-layer perceptron mapping and planar shadow generation to simulate the optical response of the sensor.
Employing marker distribution approximation to simulate how the markers on the sensor surface move when the underlying elastomer material deforms.

Compared to previous approaches, the researchers show that FOTS can generate higher quality images at a faster rate, running at 28.6 frames per second for the optical simulation and 326.1 frames per second on a single CPU without a GPU. This allows robots to practice touch-based tasks purely in simulation before trying them in the real world, a technique known as Sim2Real learning.

Technical Explanation

The FOTS system uses a combination of techniques to efficiently simulate the behavior of optical tactile sensors:

Multi-layer perceptron mapping: The researchers trained a neural network to map the pressure distribution on the sensor surface to the corresponding appearance of the sensor image. This allows FOTS to quickly generate realistic sensor images without having to simulate the full optical physics.
Planar shadow generation: To capture the effects of light interacting with the sensor surface, FOTS generates realistic shadow patterns using a planar projection model.
Marker distribution approximation: When the sensor surface deforms under contact, the markers embedded in the elastomer material move. FOTS approximates this marker motion by modeling the distribution of the markers and how they shift in response to the deformation.

Experiments show that FOTS outperforms previous optical tactile sensor simulation methods in terms of both image quality and rendering speed. The system can generate sensor images at 28.6 frames per second and track marker motion at 326.1 frames per second on a single CPU. This high performance enables FOTS to be integrated with physical simulators like MuJoCo for Sim2Real learning of tactile-based robot manipulation skills.

Critical Analysis

The authors acknowledge that while FOTS represents a significant advance in optical tactile sensor simulation, there are still some limitations to the approach:

The multi-layer perceptron mapping used to generate sensor images relies on having a large dataset of training examples, which may not always be available.
The marker motion approximation, while computationally efficient, may not capture all the nuances of how markers actually move in response to complex deformations.
The current implementation of FOTS is limited to simulating a single sensor, and extending it to handle multiple sensors or more complex sensor geometries may require further research.

Additionally, while the paper demonstrates the effectiveness of FOTS for a peg-in-hole task, it would be interesting to see how the simulator performs on a wider variety of tactile-based robotic manipulation challenges. Exploring these broader applications could help further validate the utility of the FOTS approach.

Conclusion

The FOTS simulator represents a significant advancement in the field of optical tactile sensor simulation. By combining efficient neural network-based image generation with a marker motion approximation model, the system can quickly and accurately simulate the behavior of these sensors. This enables robots to practice touch-based tasks purely in simulation, reducing the need for physical hardware and accelerating the development of tactile-based manipulation skills. While the current implementation has some limitations, the core ideas behind FOTS could have broad applicability in the field of robotic perception and control.

This summary was produced with help from an AI and may contain inaccuracies - check out the links to read the original source documents!

Follow @aimodelsfyi on 𝕏 →

Related Papers

🤿

FOTS: A Fast Optical Tactile Simulator for Sim2Real Learning of Tactile-motor Robot Manipulation Skills

Yongqiang Zhao, Kun Qian, Boyi Duan, Shan Luo

Simulation is a widely used tool in robotics to reduce hardware consumption and gather large-scale data. Despite previous efforts to simulate optical tactile sensors, there remain challenges in efficiently synthesizing images and replicating marker motion under different contact loads. In this work, we propose a fast optical tactile simulator, named FOTS, for simulating optical tactile sensors. We utilize multi-layer perceptron mapping and planar shadow generation to simulate the optical response, while employing marker distribution approximation to simulate the motion of surface markers caused by the elastomer deformation. Experimental results demonstrate that FOTS outperforms other methods in terms of image generation quality and rendering speed, achieving 28.6 fps for optical simulation and 326.1 fps for marker motion simulation on a single CPU without GPU acceleration. In addition, we integrate the FOTS simulation model with physical engines like MuJoCo, and the peg-in-hole task demonstrates the effectiveness of our method in achieving zero-shot Sim2Real learning of tactile-motor robot manipulation skills. Our code is available at https://github.com/Rancho-zhao/FOTS.

5/2/2024

📈

Simulation of Optical Tactile Sensors Supporting Slip and Rotation using Path Tracing and IMPM

Zirong Shen, Yuhao Sun, Shixin Zhang, Zixi Chen, Heyi Sun, Fuchun Sun, Bin Fang

Optical tactile sensors are extensively utilized in intelligent robot manipulation due to their ability to acquire high-resolution tactile information at a lower cost. However, achieving adequate reality and versatility in simulating optical tactile sensors is challenging. In this paper, we propose a simulation method and validate its effectiveness through experiments. We utilize path tracing for image rendering, achieving higher similarity to real data than the baseline method in simulating pressing scenarios. Additionally, we apply the improved Material Point Method(IMPM) algorithm to simulate the relative rest between the object and the elastomer surface when the object is in motion, enabling more accurate simulation of complex manipulations such as slip and rotation.

5/7/2024

TacSL: A Library for Visuotactile Sensor Simulation and Learning

Iretiayo Akinola, Jie Xu, Jan Carius, Dieter Fox, Yashraj Narang

For both humans and robots, the sense of touch, known as tactile sensing, is critical for performing contact-rich manipulation tasks. Three key challenges in robotic tactile sensing are 1) interpreting sensor signals, 2) generating sensor signals in novel scenarios, and 3) learning sensor-based policies. For visuotactile sensors, interpretation has been facilitated by their close relationship with vision sensors (e.g., RGB cameras). However, generation is still difficult, as visuotactile sensors typically involve contact, deformation, illumination, and imaging, all of which are expensive to simulate; in turn, policy learning has been challenging, as simulation cannot be leveraged for large-scale data collection. We present textbf{TacSL} (textit{taxel}), a library for GPU-based visuotactile sensor simulation and learning. textbf{TacSL} can be used to simulate visuotactile images and extract contact-force distributions over $200times$ faster than the prior state-of-the-art, all within the widely-used Isaac Gym simulator. Furthermore, textbf{TacSL} provides a learning toolkit containing multiple sensor models, contact-intensive training environments, and online/offline algorithms that can facilitate policy learning for sim-to-real applications. On the algorithmic side, we introduce a novel online reinforcement-learning algorithm called asymmetric actor-critic distillation (sysName), designed to effectively and efficiently learn tactile-based policies in simulation that can transfer to the real world. Finally, we demonstrate the utility of our library and algorithms by evaluating the benefits of distillation and multimodal sensing for contact-rich manip ulation tasks, and most critically, performing sim-to-real transfer. Supplementary videos and results are at url{https://iakinola23.github.io/tacsl/}.

8/14/2024

Soft Contact Simulation and Manipulation Learning of Deformable Objects with Vision-based Tactile Sensor

Jianhua Shan, Yuhao Sun, Shixin Zhang, Fuchun Sun, Zixi Chen, Zirong Shen, Cesare Stefanini, Yiyong Yang, Shan Luo, Bin Fang

Deformable object manipulation is a classical and challenging research area in robotics. Compared with rigid object manipulation, this problem is more complex due to the deformation properties including elastic, plastic, and elastoplastic deformation. In this paper, we describe a new deformable object manipulation method including soft contact simulation, manipulation learning, and sim-to-real transfer. We propose a novel approach utilizing Vision-Based Tactile Sensors (VBTSs) as the end-effector in simulation to produce observations like relative position, squeezed area, and object contour, which are transferable to real robots. For a more realistic contact simulation, a new simulation environment including elastic, plastic, and elastoplastic deformations is created. We utilize RL strategies to train agents in the simulation, and expert demonstrations are applied for challenging tasks. Finally, we build a real experimental platform to complete the sim-to-real transfer and achieve a 90% success rate on difficult tasks such as cylinder and sphere. To test the robustness of our method, we use plasticine of different hardness and sizes to repeat the tasks including cylinder and sphere. The experimental results show superior performances of deformable object manipulation with the proposed method.

5/14/2024