Object recognition and semantic scene labeling for rgb-d data

The availability of RGB-D (Kinect-like) cameras has led to an explosive growth of research on robot perception. RGB-D cameras provide high resolution (640 x 480) synchronized videos of both color (RGB) and depth (D) at 30 frames per second. This dissertation demonstrates the thesis that combining of RGB and depth at high frame rates is helpful for various recognition tasks including object recognition, object detection, and semantic scene labeling. We present the RGB-D Object Dataset, a large dataset of 250,000 RGB-D images of 300 objects in 51 categories, and 22 RGB-D videos of objects in indoor home and office environments. We introduce algorithms for object recognition in RGB-D images that perform category, instance, and pose recognition in a scalable manner. We also present HMP3D, an unsupervised feature learning approach for 3D point cloud data, and demonstrate that HMP3D can be used to learn hierarchies of features from different attributes including color, gradient, shape, and surface normal orientation. Finally, we present a scene labeling approach for scenes constructed from RGB-D videos. The approach uses features learned from both individual RGB-D images and 3D point clouds constructed from entire video sequences. Through these applications, this thesis demonstrates the importance of designing new features and algorithms that specifically utilize the advantages of RGB-D cameras over traditional cameras and range sensors.