Mingxu Zhang1∗ , Xiaoqi Li2∗ , Jiahui Xu1∗ , Kaichen Zhou2 , Hojin Bae2 , Yan Shen2 , Chuyan Xiong3 , Jiaming Liu2 and Hao Dong2
1 Beijing University of Posts and Telecommunications
2 Center on Frontiers of Computing Studies, Peking University
3 Institute of Computing Technology, Chinese Academy of Sciences
we propose a training-free framework SR3D that enables robotic grasping of transparent and specular objects from a single-view observation. Specifically, given single-view RGB and depth images, SR3D first uses the external visual models to generate 3D reconstructed object mesh based on RGB image. Then, the key idea is to determine the 3D object’s pose and scale to accurately localize the reconstructed object back into its original depth corrupted 3D scene. Therefore, we propose view matching and keypoint matching mechanisms, which leverage both the 2D and 3D’s inherent semantic and geometric information in the observation to determine the object’s 3D state within the scene, thereby reconstructing an accurate 3D depth map for effective grasp detection.
demo.mp4Fig. 1. The top part illustrates the pipeline of SR3D, which takes single-view RGB-D observation to realize depth map reconstruction for transparent
or specular objects. The bottom part demonstrates its effectiveness for robotic grasping on transparent objects.
Fig. 2. The Overall Framework. The top part illustrates the overall framework flow of reconstructing depth map for transparent or specular objects,
while the bottom part details the view matching and keypoint matching modules. These modules work together to determine the pose and scale required
to align the reconstructed 3D object mesh with the captured 3D depth scene.