RGB-D Images Based 3D Plant Growth Prediction by Sequential Images-to-Images Translation with Plant Priors

This paper presents a neural network based method for 3D plant growth prediction based on sequential images-to-images translation. Especially, we extend an existing image-to-image translation technique based on U-Net to images-to-images translation by incorporating convLSTM into skip connections in U-Net. With this architecture, we can achieve sequential image prediction tasks such that future images are predicted from several past ones. Since depth images are incorporated as additional channel into our network, the prediction can be represented in 3D space. As an application of our method, we develop a 3D plant growth prediction system. In the evaluation, the performance of our network was investigated in terms of the importance of each module in the network. We verified how the prediction accuracy was affected by the internal structure of the network. In addition, the extension of our network with plant priors was further investigated to evaluate the impact for plant growth prediction tasks.