We present an efficient scene layout aware object detection method for traffic surveillance. Given an input image, our approach first estimates its scene layout by transferring object annotations in a large dataset to the target image based on nonparametric label transfer. The transferred annotations are then integrated with object hypotheses generated by the state-of-the-art object detectors. We propose an approximate nearest neighbor search scheme for efficient inference in the scene layout estimation. Experiments verified that this simple and efficient approach provides consistent performance improvements to the stateof-the-art object detection baselines on all object categories in the TSWC-2017 localization challenge.