Tac2Pose: Tactile object pose estimation from the first touch

In this paper, we present Tac2Pose, an object-specific approach to tactile pose estimation from the first touch for known objects. Given the object geometry, we learn a tailored perception model in simulation that estimates a probability distribution over possible object poses given a tactile observation. To do so, we simulate the contact shapes that a dense set of object poses would produce on the sensor. Then, given a new contact shape obtained from the sensor, we match it against the pre-computed set using an object-specific embedding learned using contrastive learning. We obtain contact shapes from the sensor with an object-agnostic calibration step that maps RGB (red, green, blue) tactile observations to binary contact shapes. This mapping, which can be reused across object and sensor instances, is the only step trained with real sensor data. This results in a perception model that localizes objects from the first real tactile observation. Importantly, it produces pose distributions and can incorporate additional pose constraints coming from other perception systems, multiple contacts, or priors. We provide quantitative results for 20 objects. Tac2Pose provides high accuracy pose estimations from distinctive tactile observations while regressing meaningful pose distributions to account for those contact shapes that could result from different object poses. We extend and test Tac2Pose in multi-contact scenarios where two tactile sensors are simultaneously in contact with the object, as during a grasp with a parallel jaw gripper. We further show that when the output pose distribution is filtered with a prior on the object pose, Tac2Pose is often able to improve significantly on the prior. This suggests synergistic use of Tac2Pose with additional sensing modalities (e.g., vision) even in cases where the tactile observation from a grasp is not sufficiently discriminative. Given a coarse estimate of an object's pose, even ambiguous contacts can be used to determine an object's pose precisely. We also test Tac2Pose on object models reconstructed from a 3D scanner, to evaluate the robustness to uncertainty in the object model. We show that even in the presence of model uncertainty, Tac2Pose is able to achieve fine accuracy comparable to when the object model is the manufacturer's CAD (computer-aided design) model. Finally, we demonstrate the advantages of Tac2Pose compared with three baseline methods for tactile pose estimation: directly regressing the object pose with a neural network, matching an observed contact to a set of possible contacts using a standard classification neural network, and direct pixel comparison of an observed contact with a set of possible contacts. Website: mcube.mit.edu/research/tac2pose.html