Automatic Human Utility Evaluation Of Asr Systems: Does Wer Really Predict Performance?

We propose an alternative evaluation metric to Word Error Rate (WER) for the decision audit task of meeting recordings, which exemplifies how to evaluate speech recognition within a legitimate application context. Using machine learning on an initial seed of human-subject experimental data, our alternative metric handily outperforms WER, which correlates very poorly with human subjects' success in finding decisions given ASR transcripts with a range of WERs.