Interactive Deep Learning for Exploratory Sorting of PlantImages by Visual Phenotypes

This paper proposes an interactive system called Andromeda1 that enables users to interact with machine learning models to allow for exploratory sorting of images through a cognitive approach that uses a reduced dimension plot. In our system, a dimension reduction algorithm projects the images into a 2D space representing similarities between the images based on visual features extracted by a deep neural network. With Andromeda, users can alter the projection by dragging a subset of the images into groups according to their domain expertise. The underlying machine learning model learns the new projection by optimizing a weighted distance function in the feature space, and the model re-projects the images accordingly. The users can explore multiple custom projections to learn about the visual support for di↵erent groupings based on explainable-AI feedback. Our approach incorporates user preferences into machine learning model construction and allows transfer learning from pre-trained image processing models to accomplish new tasks based on user inputs. Using edamame pod images as an example, we interactively re-project the images into di↵erent groupings based on maturity and disease, and identify important visual features from the pixels highlighted by the model.