We address the glass object localization problem with a RGB-D camera. Our approach uses a nonparametric, data-driven label transfer scheme for local glass boundary estimation. A weighted voting scheme based on a joint feature manifold is adopted to integrate depth and appearance cues, and we learn a distance metric on the depth-encoded feature manifold. Local boundary evidence is then integrated into a MRF framework for spatially coherent glass object detection and segmentation. The efficacy of our approach is verified on a challenging RGB-D glass dataset where we obtained a clear improvement over the state-of-the-art both in terms of accuracy and speed.