Peaks2Image: Reconstructing fMRI Statistical Maps from Peaks

Neuroscience is striving to overcome the lack of power due to the small sample size of standard studies. An important step forward has been the creation of large-scale public image repositories, such as NeuroVault. Such repositories enable comparing images across studies and automatically associating them with cognitive terms. Yet, this type of meta-analysis faces a major roadblock: the scarcity and inconsistency of image annotations and metadata. Another resource containing rich annotations is the neuroscientific literature. However it only yields a handful of brain-space coordinates per publication, those of the main activity peaks reported in each study. In this work, we propose Peaks2Image, a neuralnetwork approach to reconstructing continuous spatial representations of brain activity from peak activation tables. Using reconstructions of studies published in the neuroscientific literature, we train a decoder using tf-idf features as labels, leading to a much broader set of decoded terms than current image-based studies. We validate the decoder on 43,000 NeuroVault images, successfully decoding 58 out of 81 concepts in a zero-shot setting.