Neural Text Line Segmentation of Multilingual Print and Handwriting with Recognition-Based Evaluation

We present a novel method for detecting text lines in historical handwritten and printed document images. Our hybrid technique begins by leveraging deep neural networks to perform multi-class pixel-wise prediction. The predictor not only discovers text and graphics pixels in the document, but it is also designed to automatically adhere contiguous regions from the same text line while also predicting buffers that prevent disassociated text lines from merging. The system breaks neural "ties" through dynamic programming. To the best of our knowledge, our system is the first neural system to predict the entire perimeters of full text lines. Also, to aid in scaling and full-scope awareness, the network during training is initially given small regions of the image to study and then expands its scope to full images as training continues. Our goal for line segmentation is to enable automatic transcription on huge heterogenous collections of historical images, so we use transcription accuracy as our metric. We document and leverage our state-of-the-art transcription system as an evaluation harness for scoring our segmenter along with various other competitor segmenters. We then show the effectiveness of our system as it relates to other systems by comparing it to both known data sets (IAM) and to three 50K-word "in the wild" test sets consisting of (a) US handwritten wills and deeds, (b) US historical newsprint images, and (c) Spanish Church and Government records.