Indoor and Outdoor 3D Scene Graph Generation Via Language-Enabled Spatial Ontologies

This paper proposes an approach to build 3D scene graphs in arbitrary indoorand outdoor environments. Such extension is challenging; the hierarchy ofconcepts that describe an outdoor environment is more complex than for indoors,and manually defining such hierarchy is time-consuming and does not scale.Furthermore, the lack of training data prevents the straightforward applicationof learning-based tools used in indoor settings. To address these challenges,we propose two novel extensions. First, we develop methods to build a spatialontology defining concepts and relations relevant for indoor and outdoor robotoperation. In particular, we use a Large Language Model (LLM) to build such anontology, thus largely reducing the amount of manual effort required. Second,we leverage the spatial ontology for 3D scene graph construction using LogicTensor Networks (LTN) to add logical rules, or axioms (e.g., "a beach containssand"), which provide additional supervisory signals at training time thusreducing the need for labelled data, providing better predictions, and evenallowing predicting concepts unseen at training time. We test our approach in avariety of datasets, including indoor, rural, and coastal environments, andshow that it leads to a significant increase in the quality of the 3D scenegraph generation with sparsely annotated data.