IOFusion: instance segmentation and optical-flow guided 3D reconstruction in dynamic scenes

To improve the accuracy of camera pose estimation for RGBD-based 3D reconstruction in dynamic scenes, a method based on instance segmentation and optical flow is proposed. Firstly, instance segmentation is used to detect objects, and a semantic map is constructed by removing non-rigid objects. Secondly, motion residual is calculated by optical flow and camera flow to detect dynamic rigid objects, and nonlinear optimization is used to track the dynamic rigid objects extracted from the semantic map. Thirdly, after removing features of non-rigid objects and dynamic rigid ones in each frame, the remaining features are used to optimize the camera pose. Finally, a TSDF model is used to reconstruct the static background, and point clouds are used to reconstruct dynamic rigid objects. Experiments on TUM and Bonn datasets show that the method produces better camera poses than the current state-of-the-art methods in most dynamic scenes. Ablation experiments on Bonn dataset show that retaining features of static rigid objects significantly improves camera pose estimation precision. The annotated datasets and the source code are available at https://github.com/CodingMaplee/IOFusion/tree/main .