Artie Bias Corpus: An Open Dataset for Detecting Demographic Bias in Speech Applications

P. 6462–6468.

Meyer J., Rauchenstein L., Eisenberg J.

We describe the creation of the Artie Bias Corpus, an English dataset of expert-validated <audio, transcript> pairs with demographic tags for age, gender, accent. We also release open software which may be used with the Artie Bias Corpus to detect demographic bias in Automatic Speech Recognition systems, and can be extended to other speech technologies. The Artie Bias Corpus is a curated subset of the Mozilla Common Voice corpus, which we release under a Creative Commons CC0 license – the most open and permissive license for data. This article contains information on the criteria used to select and annotate the Artie Bias Corpus in addition to experiments in which we detect and attempt to mitigate bias in end-to-end speech recognition models. We we observe a significant accent bias in our baseline DeepSpeech model, with more accurate transcriptions of US English compared to Indian English. We do not, however, find evidence for a significant gender bias. We then show significant improvements on individual demographic groups from fine-tuning.

Keywords: speech corpus, automatic speech recognition, demographic bias, bias detection

In book

Proceedings of The 12th Language Resources and Evaluation Conference

Vol. 12. , European Language Resources Association (ELRA), 2020.