A Multi-object Detection and Tracking Method Based on the Fusion of Lidar and Camera

Aiming at the problem that some detection networks have high accuracy but poor real-time performance and are not easy to deploy on embedded devices when used to intelligent vehicles, the method in this paper is proposed. And it solves the problem of global target tracking respectively and trajectory management in multi-objective environment. It’s an improved method for object detection and tracking based on the data fusion of lidar and camera. In this method, first, Yolov3 algorithm, which is accelerated by KCF algorithm, is used to recognize the object in the image information. And cluster the point cloud. Second, align sensor data in time and space. Then, the object information of the two sensors is supplemented by the Hungarian algorithm matching. Further then, the multi-frame matching is carried out through the key point matching count, and the multi-frame information is used for the object tracking. Finally, the continuous motion trajectory of each object in the surrounding environment can be obtained. Experimental verification shows that the position detection error of this method is less than 5%, and the total processing time of a single frame is 154ms. Compared with other methods, this method can significantly improve the processing speed, which is convenient to enhance the real-time data processing as a step of the perception module.