A Bayesian model of visual search in natural images and the role of memory

Abstract Modelling human eye movements during visual search has been a research topic of interest for many years. Here, we investigate if a Bayesian ideal observer model can be configured to effectively explain human eye movements when searching for a known object target in natural images. We collected eye movements from participants who searched for a known target in 18 different natural textured images and compared their performance and strategy to the ideal observer model. Our model chooses search fixations that maximize information gain, given a probabilistic model of target detectability, at various eccentricities which is in turn approximated from features of a convolutional neural network. We collect and use ground truth data from a second experiment with human observers (observers detecting a target at various eccentricities on the 18 natural textured backgrounds) to fine-tune the parameters of our proposed pipeline for detectability estimation and verify our results. We further study the role of short-term memory in the ideal observer model and find that a model with limited memory best fits the human search patterns. Our results suggest that the visual system's strategy for choosing next fixation location in a visual search task is consistent with a Bayesian ideal observer model.