Image Understanding using vision and reasoning through Scene Description Graph.

•We propose an intermediate structure that captures the semantics of an image.•We propose an Image Understanding architecture that combines vision and reasoning modules to generate such structures, and an implementation of the architecture by combining a deep learning based Visual module with probabilistic reasoning on a Commonsense Knowledge Base.•We enhance the Flickr8k dataset with the observable scene constituents (actions and properties involving objects).•We publish the comparative human evaluations dataset for our approach, two popular neural approaches (Karpathy, Li, Vinyals, Toshev, Bengio, Erhan, 2017) and goldtruth captions for three existing Captioning Datasets (Flickr8k, Flickr30k and MS-COCO), which can be used to propose better automatic caption evaluation metrics (this dataset is used in Anderson et al., 2016 to propose SPICE).