Tactile perception is key for robotics applications such as manipulation.
However, tactile data collection is time-consuming, especially when compared to
vision. This limits the use of the tactile modality in machine learning solutions
in robotics. In this paper, we propose a generative model to simulate realistic
tactile sensory data for use in downstream tasks. Starting with easily-obtained
camera images, we train Neural Radiance Fields (NeRF) for objects of interest.
We then use NeRF-rendered RGB-D images as inputs to a conditional Generative
Adversarial Network model (cGAN) to generate tactile images from desired orientations. We evaluate the generated data quantitatively using the Structural Similarity Index and Mean Squared Error metrics, and also using a tactile classification
task both in simulation and in the real world. Results show that by augmenting
a manually collected dataset, the generated data is able to increase classification
accuracy by around 10%. In addition, we demonstrate that our model is able to
transfer from one tactile sensor to another with a small fine-tuning dataset.
cross-modal tactile data generation
,camera-based tactile sensing
