Speech perception studies typically rely on trained research assistants to score orthographic listener transcripts for words correctly identified. While the accuracy of the human scoring protocol has been validated with strong intra- and inter-rater reliability, the process of hand-scoring the transcripts is time-consuming and resource intensive. Here, an open-source computer-based tool for automated scoring of listener transcripts is built (Autoscore) and validated on three different human-scored data sets. Results show that not only is Autoscore highly accurate, achieving approximately 99% accuracy, but extremely efficient. Thus, Autoscore affords a practical research tool, with clinical application, for scoring listener intelligibility of speech.

1.
Allison
,
K. M.
, and
Hustad
,
K. C.
(
2014
). “
Impact of sentence length and phonetic complexity on intelligibility of 5-year-old children with cerebral palsy
,”
Int. J. Speech Lang. Pathol.
16
(
4
),
396
407
.
2.
Bache
,
S. M.
, and
Wickham
,
H.
(
2014
). “
magrittr: A forward-pipe operator for R
,” R package version 1.5. https://CRAN.R-project.org/package=magrittr (Last viewed December 1, 2018).
3.
Barrett
,
T. S.
, and
Brignone
,
E.
(
2017
). “
Furniture for quantitative scientists
,”
R Journal
9
,
142
148
.
4.
Bilger
,
R. C.
,
Nuetzel
,
J. M.
,
Rabinowitz
,
W. M.
, and
Rzeczkowski
,
C.
(
1984
). “
Standardization of a test of speech perception in noise
,”
J. Speech Lang. Hear. Res.
27
,
32
48
.
5.
Borrie
,
S. A.
,
Lansford
,
K. L.
, and
Barrett
,
T. S.
(
2017a
). “
Generalized adaptation to dysarthric speech
,”
J. Speech Lang. Hear. Res.
60
,
3110
3117
.
6.
Borrie
,
S. A.
,
Lansford
,
K. L.
, and
Barrett
,
T. S.
(
2017b
). “
Rhythm perception and its role in recognition and learning of dysrhythmic speech
,”
J. Speech Lang. Hear. Res.
60
,
561
570
.
7.
Borrie
,
S. A.
,
McAuliffe
,
M. J.
,
Liss
,
J. M.
,
Kirk
,
C.
,
O'Beirne
,
G. A.
, and
Anderson
,
T.
(
2012
). “
Familiarisation conditions and the mechanisms that underlie improved recognition of dysarthric speech
,”
Lang. Cogn. Process.
27
,
1039
1055
.
8.
Bradlow
,
A. R.
, and
Bent
,
T.
(
2008
). “
Perceptual adaptation to non-native speech
,”
Cognition
106
,
707
729
.
9.
Cooke
,
M.
,
Mayo
,
C.
,
Valentini-Botinhao
,
C.
,
Stylianou
,
Y.
,
Sauert
,
B.
, and
Tang
,
Y.
(
2013
). “
Evaluating the intelligibility benefit of speech modifications in known noise conditions
,”
Speech Commun.
55
(
4
),
572
585
.
10.
Csárdi
,
G.
(
2017
). “
crayon: Colored Terminal Output
,” R package version 1.3.4. https://CRAN.R-project.org/package=crayon (Last viewed December 1, 2018).
11.
Davis
,
M. H.
,
Johnsrude
,
I. S.
,
Hervais-Adelman
,
A.
,
Taylor
,
K.
, and
McGettigan
,
C.
(
2005
). “
Lexical information drives perceptual learning of distorted speech: Evidence from the comprehension of noise-vocoded sentences
,”
J. Exp. Psychol. Gen.
134
,
222
241
.
12.
Feinerer
,
I.
,
Hornik
,
K.
, and
Meyer
,
D.
(
2008
). “
Text mining infrastructure in R
,”
J. Stat. Software
25
,
1
54
.
13.
Festen
,
J. M.
, and
Plomp
,
R.
(
1990
). “
Effects of fluctuating noise and interfering speech on the speech-reception threshold for impaired and normal hearing
,”
J. Acoust. Soc. Am.
88
(
4
),
1725
1736
.
14.
Guediche
,
S.
,
Fiez
,
J. A.
, and
Holt
,
L. L.
(
2016
). “
Adaptive plasticity in speech perception: Effects of external information and internal predictions
,”
J. Exp. Psychol. Hum. Percept. Perform.
42
(
7
),
1048
1059
.
15.
Healy
,
E. W.
,
Yoho
,
S. E.
,
Wang
,
Y.
, and
Wang
,
D.
(
2013
). “
An algorithm to improve speech recognition in noise for hearing-impaired listeners
,”
J. Acoust. Soc. Am.
134
(
4
),
3029
3038
.
16.
Henry
,
L.
, and
Wickham
,
H.
(
2018
). “
purrr: Functional programming tools
,” R package version 0.2.5. https://CRAN.R-project.org/package=purrr (Last viewed December 1, 2018).
17.
Hogan
,
C. A.
, and
Turner
,
C. W.
(
1998
). “
High-frequency audibility: Benefits for hearing-impaired listeners
,”
J. Acoust. Soc. Am.
104
,
432
441
.
18.
Hustad
,
K. C.
(
2006
). “
A closer look at transcription intelligibility for speakers with dysarthria: Evaluation of scoring paradigms and linguistic errors made by listeners
,”
Am. J. Speech Lang. Pathol.
15
,
268
277
.
19.
Hustad
,
K. C.
,
Jones
,
T.
, and
Dailey
,
S.
(
2003
). “
Implementing speech supplementation strategies: Effects on intelligibility and speech rate of individuals with chronic severe dysarthria
,”
J. Speech Lang. Hear. Res.
46
,
462
474
.
20.
Huyck
,
J.
(
2018
). “
Comprehension of degraded speech matures during adolescence
,”
J. Speech Lang. Hear. Res.
61
,
1012
1022
.
21.
IEEE
(
1969
). “
IEEE recommended practice for speech quality measurements
,”
IEEE Trans. Audio Electroacoust.
17
,
225
246
.
22.
Killion
,
M. C.
,
Niquette
,
P. A.
,
Revit
,
L. J.
, and
Skinner
,
M. W.
(
2001
). “
Quick SIN and BKB-SIN, two new speech-in-noise tests permitting SNR-50 estimates in 1 to 2 min
,”
J. Acoust. Soc. Am.
109
(
5
),
2502
.
23.
Liss
,
J. M.
,
Spitzer
,
S. M.
,
Caviness
,
J. N.
, and
Adler
,
C.
(
2002
). “
The effects of familiarization on intelligibility and lexical segmentation in hypokinetic and ataxic dysarthria
,”
J. Acoust. Soc. Am.
112
,
3022
3030
.
24.
Liss
,
J. M.
,
Spitzer
,
S.
,
Caviness
,
J. N.
,
Adler
,
C.
, and
Edwards
,
B.
(
1998
). “
Syllabic strength and lexical boundary decisions in the perception of hypokinetic dysarthric speech
,”
J. Acoust. Soc. Am.
104
,
2457
2466
.
25.
Liss
,
J. M.
,
Spitzer
,
S.
,
Caviness
,
J. N.
,
Adler
,
C.
, and
Edwards
,
B.
(
2000
). “
Lexical boundary error analysis in hypokinetic and ataxic dysarthria
,”
J. Acoust. Soc. Am.
107
,
3415
3424
.
26.
Luce
,
P. A.
, and
Pisoni
,
D. B.
(
1998
). “
Recognizing spoken words: The neighborhood activation Model
,”
Ear Hear.
19
(
1
),
1
36
.
27.
McAuliffe
,
M. J.
,
Gibson
,
E. M. R.
,
Kerr
,
S. E.
,
Anderson
,
T.
, and
LaShell
,
P. J.
(
2013
). “
Vocabulary influences older and younger listeners' processing of dysarthric speech
,”
J. Acoust. Soc. Am.
134
,
1358
1368
.
28.
Müller
,
K.
, and
Wickham
,
H.
(
2018
). “
tibble: Simple data frames
,” R package version 1.4.2. https://CRAN.R-project.org/package=tibble (Last viewed December 1, 2018).
29.
Munro
,
M. J.
(
1998
). “
The effects of noise on the intelligibility of foreign-accented speech
,”
Stud. Second Lang. Acquist.
20
,
139
154
.
30.
Nilsson
,
M.
,
Soli
,
S.
, and
Sullivan
,
J.
(
1994
). “
Development of the hearing in noise test for the measurement of speech reception thresholds in quiet and in noise
,”
J. Acoust. Soc. Am.
95
,
1085
1099
.
31.
R Core Team
(
2018
). “
R: A language and environment for statistical computing
,” R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/ (Last viewed December 1, 2018).
32.
Stilp
,
C. E.
,
Kiefte
,
M.
,
Alexander
,
J. M.
, and
Kluender
,
K. R.
(
2010
). “
Cochlea-scaled spectral entropy predicts rate-invariant intelligibility of temporally distorted sentences
,”
J. Acoust. Soc. Am.
128
,
2112
2126
.
33.
Strand
,
E. A.
, and
Yorkston
,
K. M.
(
1994
). “
Description and classification of individuals with dysarthria: A 10-year review
,” in
Motor Speech Disorders: Advances in Assessment and Treatment
, edited by
J. A.
Till
,
K. M.
Yorkston
, and
D. R.
Beukelman
(
Paul H. Brooks
,
Baltimore, MD
), pp.
37
56
.
34.
Turner
,
C. W.
, and
Cummings
,
K. J.
(
1999
). “
Speech audibility for listeners with high-frequency hearing loss
,”
Am. J. Audiol.
8
,
47
56
.
35.
Tye-Murray
,
N.
,
Sommers
,
M. S.
, and
Spehar
,
B.
(
2007
). “
Audiovisual integration and lip reading abilities of older adults with normal and impaired hearing
,”
Ear Hear.
28
(
5
),
656
668
.
36.
Van Engen
,
K.
,
Phelps
,
J. E. B.
,
Smiljanic
,
R.
, and
Chandrasekaran
,
B.
(
2014
). “
Enhancing speech intelligibility: Interactions among context, modality, speech style, and masker
,”
J. Speech Lang. Hear. Res.
57
,
1908
1918
.
37.
Wang
,
D.
,
Kjems
,
U.
,
Pedersen
,
M. S.
,
Boldt
,
J. B.
, and
Lunner
,
T.
(
2009
). “
Speech intelligibility in background noise with ideal binary time-frequency masking
,”
J. Acoust. Soc. Am.
125
(
4
),
2336
2347
.
38.
Wickham
,
H.
(
2018
). “
stringr: Simple, consistent wrappers for common string operations
,” R package version 1.3.1. https://CRAN.R-project.org/package=string (Last viewed December 1, 2018).
39.
Wickham
,
H.
,
François
,
R.
,
Henry
,
L.
, and
Müller
,
K.
(
2018
). “
dplyr: A grammar of data manipulation
,” R package version 0.7.6. https://CRAN.R-project.org/package=dplyr (Last viewed December 1, 2018).
40.
Wickham
,
H.
, and
Henry
,
L.
(
2018
). “
tidyr: Easily tidy data with ‘spread()’ and ‘gather()’ functions
,” R package version 0.8.1. https://CRAN.R-project.org/package=tidyr (Last viewed December 1, 2018).
41.
Wild
,
A.
,
Vorperian
,
H. K.
,
Kent
,
R. D.
,
Bolt
,
D. M.
, and
Austin
,
D.
(
2018
). “
Single-word speech intelligibility in children and adults with down syndrome
,”
Am. J. Speech Lang. Pathol.
27
,
222
236
.
42.
Yoho
,
S. E.
,
Borrie
,
S. A.
,
Barrett
,
T. S.
, and
Whittaker
,
D.
(
2018
). “
Are there sex effects for speech intelligibility in American English? Examining the influence of talker, listener, and methodology
,”
Atten. Percept. Psychophys
. (in press).
43.
Yorkston
,
K.
, and
Beukelman
,
D.
(
1980
). “
A clinician-judged technique for quantifying dysarthric speech based on single-word intelligibility
,”
J. Commun. Disord.
13
,
15
31
.
44.
Yorkston
,
K. M.
,
Hammen
,
V. L.
,
Beukelman
,
D. R.
, and
Traynor
,
C. D.
(
1990
). “
The effect of rate control on the intelligibility and naturalness of dysarthric speech
,”
J. Speech Hear. Disord.
55
(
3
),
550
561
.
You do not currently have access to this content.