Costs of dynamic cocktail party listening: Investigating the effects of cognitive abilities and hearing impairment^a)

In everyday life, communication often takes place in situations in which multiple talkers speak simultaneously. These so-called cocktail party situations pose high demands not only perceptually but also cognitively, since they require the listener to separate different auditory streams and to direct their attention to the target talker while ignoring competing sound sources.

Cocktail party listening can be static or dynamic. In static situations, the talker of interest is fixed, whereas in dynamic situations, the target talker may change occasionally and in a possibly unpredictable manner. The latter thus requires simultaneously monitoring multiple sound sources as well as switching attention from one target talker to the other and is therefore associated with an additional cognitive load (e.g., Lin and Carlile, 2015). As a consequence, performance in dynamic listening relative to a static condition may decrease, which is referred to as “costs” (e.g., Lin and Carlile, 2015, 2019; Meister et al., 2020).

Different factors contribute to the costs of dynamic listening. Brungart and Simpson (2007) investigated speech recognition in static and dynamic multi-talker situations with normal-hearing (NH) listeners (no age reported). In dynamic conditions, they observed a decrease in the percentage of correctly recognized words relative to the static conditions. Conditions with frequent talker switches were associated with lower speech recognition than conditions with rare switches. In addition, subjects showed better speech recognition if the voice of the target talker was fixed across trials relative to conditions with a changing target voice.

To examine possible age-related effects, dynamic cocktail party listening in younger and older participants has been compared in several studies, some of which revealed different attentional mechanisms in these groups. For example, using a similar setup as Brungart and Simpson (2007), Meister et al. (2020) showed that costs were similar between groups of younger and older NH listeners, but a more fine-grained analysis found that older listeners tend to have more difficulty in monitoring multiple information sources at once. In a study by Oberem et al. (2017), younger and older NH participants were required to recognize single digits presented from occasionally switching locations in the presence of one other competing talker. While the costs due to switching attention from one location to the other were not affected by aging, the ability to suppress the competing talker was worse in older than in younger listeners. Getzmann et al. (2015) examined younger and older NH listeners in dynamic cocktail party situations and found indications of an increased allocation of attentional resources in the older group based on event-related potentials. However, performance did not differ significantly, although it was somewhat higher in the younger listeners. In addition, they showed that performance in both groups dropped significantly following target talker switches and required multiple trials to get back to its initial state, pointing to a prolonged period of reorientation.

As dynamic cocktail party situations are cognitively demanding, costs may depend on individual cognitive abilities. However, only a few studies have tapped into this issue. In a study conducted by Lin and Carlile (2015) involving young NH adults, correlations between switching costs and results of a working memory test were shown. By contrast, in a follow-up study (Lin and Carlile, 2019) considering non-spatial attention switching as well as in the study of Meister et al. (2020), no significant association between costs and cognitive abilities was found.

The studies presented above considered young and older listeners with good hearing. However, cocktail party listening is more difficult for hearing-impaired (HI) than for NH listeners [see Bronkhorst (2000) for a review]. Besides the decreased audibility caused by the elevated hearing thresholds, this finding might be partially attributed to the reduced recognition benefit HI listeners gain from temporal (Festen and Plomp, 1990) and spectral (Peters et al., 1998) dips in the masker compared to NH listeners. In addition, benefits arising from the spatial separation of sound sources may also be reduced in HI listeners (e.g., Glyde et al., 2013). Understanding speech that is degraded by impaired peripheral processing causes an additional cognitive load, as proposed by the ELU-model (Rönnberg et al., 2013) and the FUEL model (Pichora-Fuller et al., 2016). As a consequence, given that cognitive capacity is limited, fewer cognitive resources can be allocated to other tasks (Pichora-Fuller et al., 2016). In the case of dynamic cocktail party listening, these tasks may include monitoring multiple sources and switching attention from one talker to the other, which are vital for following a conversation. Thus, dynamic situations might be especially problematic for older listeners with hearing loss. However, as pointed out by Roverud et al. (2018), very little is known about how hearing impairment affects speech recognition in multi-talker situations involving target talker switches.

In the present study, we hypothesize that, due to the increased cognitive load associated with impaired peripheral processing, detrimental effects of hearing loss on speech recognition are more pronounced in cognitively demanding dynamic situations than in static situations. As a consequence, we anticipate higher costs for HI than for NH listeners. Furthermore, we expect the costs to be smaller for listeners with good cognitive abilities, as they have more capacity to handle the increased cognitive demand caused by the dynamic situations.

Data from 30 listeners aged between 62 and 83 years were included in the study. All participants passed DemTect dementia screening (Kalbe et al., 2004). Fifteen listeners [aged 71.1 ± 5.0 years, mean ± standard deviation (SD)] revealed symmetrical hearing impairment with a better ear hearing loss (BEHL; averaged across 0.5, 1, 2, and 4 kHz) larger than 25 dB HL and thus exhibited mild-to-moderate hearing impairment according to the classification of the World Health Organization (WHO) (Mathers et al., 2000). Mean BEHL was 32.9 ± 5.9 dB HL. They were therefore denoted as older hearing-impaired listeners (old HI). These listeners were age-matched with a group of 15 listeners (70.3 ± 4.7 years), including 13 participants of Meister et al. (2020) and two additional participants, all showing a BEHL substantially below 25 dB HL (i.e., 11.6 ± 3.9 dB HL), thus denoted as older normal-hearing listeners (old NH). This choice allowed for age matching within a margin of 1 year or less for 11 and 2 years or less for 13 of the listener pairs. The mean BEHL value across all participants (22.3 dB HL) is in good agreement with data of the corresponding age group from a large-scale study by von Gablenz and Holube (2015) investigating the hearing status of the German population.

The setup and stimuli used for this study were the same as in Meister et al. (2020). Participants were seated in front of three loudspeakers located at −60° (left), 0° (center), and +60° (right) azimuth angle; the distance between the participants' heads and each of the loudspeakers was 1.2 m. A sentence was presented through each of the speakers at 65 dB sound pressure level (SPL). All three sentences were played back simultaneously. The sentences were uttered by three talkers with different voices: a male voice (fundamental frequency 122 Hz) as well as a deep and a high female voice with fundamental frequencies of 143 and 202 Hz, respectively. The sentences were taken from the German version of the Oldenburg sentence test (Wagener et al., 1999), which is a matrix test with five-word sentences, such as “Stefan kauft acht nasse Autos” (Stefan buys eight wet cars). All sentences had a length of 2.5 s. For each trial, sentences were combined in such a way that they did not have any words in common. Participants were instructed to repeat back the target sentence, which was indicated by the keyword “Stefan,” and to ignore the competing talkers.

This setup was used to investigate multi-talker listening in static as well as in dynamic situations. In the static condition, the position and the voice of the target talker remained constant across a test list of 10 trials. In addition, before each test list, listeners were given a priori information about the position and voice of the target talker. One test list was presented for each of the 3 × 3 combinations of position and voice.

In the dynamic conditions, the listeners were not given any a priori information regarding the position or the voice of the target talker. The rationale here is to create stimulus uncertainty, meaning listeners were required to monitor all three speakers to identify the target talker on their own using the keyword.

Two different switching types (STs), both adapted from Brungart and Simpson (2007), were used in this study: In switching type A (STA), the three talkers remained at fixed positions, and the talker uttering the target sentences changed. By contrast, in switching type B (STB), the talker uttering the target sentence remained the same, but the talkers switched spatial positions. This condition was included to examine the benefit of a fixed target voice across the trials. For a more detailed description of the STs, including examples, see Meister et al. (2020).

Each ST was combined with two switching probabilities (SPs) of 20% and 100%, resulting in a total of four different dynamic conditions (STA_SP20, STA_SP100, STB_SP20, STB_SP100). In test lists with 20% SP (comprising 30 trials in total), six switches occurred in unpredictable intervals of 2–5 trials, while the remaining 24 were non-switch trials. In test lists with 100% SP, every trial (30 in total) involved a switch.

Participants underwent three neuropsychological tests tapping into cognitive domains that are assumed to be important for speech recognition in cocktail party situations. These include working memory capacity (WMC), which is generally considered to play an important role in speech comprehension in difficult situations (cf. ELU-model; Rönnberg et al., 2013). More specifically with regard to listening in multi-talker situations, processing speed and cognitive flexibility may be important (Woods et al., 2013). These domains are also thought to play a role in dynamic cocktail party situations: Lin and Carlile (2015) showed a correlation of WMC and the costs associated with target talker switching. It can also be assumed that cognitive flexibility is of great importance in dynamic situations due to the need to (re-)adjust the focus of attention. Finally, we assume that the basal ability of focusing and sustaining attention is important in order to be able to focus on the target talker and to ignore irrelevant information from competing sound sources.

WMC was tested using the German version of the Reading Span Test (RST) (Carroll et al., 2015). During this test, sentences are successively presented on a computer screen in blocks of 3–6 sentences. Immediately after each sentence, subjects are asked if the sentence was meaningful. After the presentation of a sentence block, subjects are instructed to repeat back either the first or the last word in each sentence. The percentage of correctly recalled words is used as the outcome measure.

The Trail Making Test (TMT) (Reitan, 1958) is a paper and pencil test and was used to assess cognitive flexibility. Participants connect a series of dots in sequential order that are pseudo-randomly distributed across the test sheet. In version A, the order of the dots is indicated using numbers. Version B requires the participants to alternate between numbers and letters. The time to complete the task is used as an outcome measure. To obtain a measure of cognitive flexibility, the ratio of the outcomes of versions B and A (“TMT B:A”) was calculated as described by Arbuthnott and Frank (2000).

Focused/sustained attention was assessed using the test d2-R (Brickenkamp et al., 2010). In this paper and pencil test, subjects are instructed to search for and cross out target characters (letter d with two dashes) while ignoring distractor characters (letter d with more or less than two dashes and the letter p with one or two dashes) within a given time limit. The total number of identified target characters minus the number of false alarms is used as the outcome measure.

After obtaining informed consent from each participant, pure-tone thresholds were determined; next, participants completed the dementia screening and the three neuropsychological tests. Afterward, the multi-talker speech recognition test was conducted. The latter began with a familiarization phase during which participants listened to various sentences from the Oldenburg sentence test uttered by the three different talkers. Then a practice condition with 30 trials was presented so that participants could familiarize themselves with the task at hand.

The actual multi-talker test comprised nine static and four dynamic conditions (as described above). The order of the conditions was quasi-randomized and reversed for half of the subjects to mitigate learning and fatigue effects. The listening tests were conducted in a sound-attenuating and sound-treated booth. The test session lasted for about 2.5–3 h, including several individual breaks. The study was approved by the local ethics committee.

Costs were calculated in two different ways: first, as the relative change in word recognition performance between the static and the dynamic conditions {i.e., [performance(static baseline) – performance(dynamic)]/performance(static baseline)} and second as the absolute change between static and dynamic conditions [performance(static baseline) – performance(dynamic)]. The second is in accordance with the study by Meister et al. (2020). Here, we additionally used relative changes, as we expected a large performance variation in the static condition across participants due to the different degrees of hearing loss. Importantly, our cost calculation considered that the nine different combinations of target voices and positions did not occur equally often in the dynamic test lists to prevent listeners from anticipating switches. Thus, for each dynamic condition, an individual static baseline was established reflecting exactly the presentation schemes in the dynamic conditions. This was done by averaging the proportion of correctly recognized words weighted with the frequency of occurrences of the different target talker combinations in the dynamic test list. For example, one of the dynamic conditions included 7 target sentences presented from the right position and uttered by the high female voice, 9 from the center by the male voice, and 14 from the left by the deep female voice. The baseline score for this condition was thus (7*P_{highfemale,right}+9*P_male,center+14*P_{deepfemale,left})/30, with P_x being the static performance for the respective voice-position combination. The rationale behind the use of costs—that is, performance changes rather than absolute values—is the fact that this study focusses on the additional detrimental effects that dynamic multi-talker situations have on the performance compared to static situations.

While speech recognition in cocktail party situations is assumed to be associated with the cognitive domains tested here, we had no detailed hypothesis regarding the specific contribution of the single tests. Thus, a global cognitive score was calculated for each individual by standardizing the outcome of each of the neuropsychological tests separately using z-transformation and then averaging across the three tests. This yielded a global composite score for each participant, reflecting cognitive abilities potentially important for the costs of dynamic cocktail party listening.

Two separate generalized linear mixed models (GLMMs) with participants as random-effect factor were used for assessing the effects of ST, SP, and listener group on relative and absolute costs as the dependent variable. t-tests were used for pairwise comparisons. Pearson correlation coefficients r_p were calculated to analyze the relationship between costs and cognitive measures as well as the BEHL. A significance level of 0.05 was used.

In the RST, 44.3% ± 12.1% (mean ± SD) and 40.5% ± 10.3% of the target words were recalled by the NH and the HI group, respectively. Outcomes of the Trailmaking Test (i.e., TMT B:A) amounted to 2.23 ± 0.61 for the NH and to 2.35 ± 0.68 for the HI listeners. The test d2-R yielded 133.2 ± 32.8 items for the NH and 110.9 ± 44.8 items for the HI. Except for the TMT, higher values indicate a better result, meaning the test scores for the HI participants tend to be worse than those of the NH participants. However, none of the neuropsychological tests nor the composite score revealed a significant group difference (t-test, all uncorrected p-values > 0.05).

Table 1 shows the percentages of correctly recognized words for both the static and the dynamic conditions; results for the static conditions are expressed as baseline scores.

Based on these values, relative and absolute costs were calculated as described above. Figure 1 shows the costs of dynamic multi-talker listening for the two STs and SPs. The left panel depicts the relative costs, whereas the right one shows the absolute costs. Both cost types seem to yield roughly similar patterns. Differences between the two listener groups appear to be small, except for the STA_SP20 condition, in which the HI listeners tend to show somewhat higher costs than the NH group. STB reveals lower costs than STA, and costs in SP20 appear to be lower than in SP100.

Table 2 shows the outcome of the GLMMs. F-statistics and p-values are reported separately for the relative and absolute costs. For both cost types, the GLMMs reveal significant effects of ST and SP as well as group*ST and ST*SP interactions. The three-way interaction group*ST*SP is only significant for the relative but not the absolute costs. There was no significant effect of listener group for either cost type.

To uncover exclusively those effects that are relevant regardless of the type of costs, a follow-up analysis was performed only on the interactions that were significant in both models. This applied to group*ST and ST*SP. Regarding the first interaction, pairwise contrasts with Bonferroni correction revealed a significant difference between STA and STB in the HI listeners (relative costs: t_1,112 = 4.90, p < 0.001; absolute costs: t_1,112 = 4.82, p < 0.001) but not in the NH listeners (relative costs: t_1,112 = 1.33, p = 0.187; absolute costs: t_1,112 = 1.74, p = 0.085). In terms of the ST*SP interaction, pairwise contrasts with Bonferroni correction revealed a significant difference between SP20 and SP100 in STB (relative costs: t_1,112 = 3.85, p < 0.001; absolute costs: t_1,112 = 3.69, p < 0.001) but not in STA (relative costs: t_1,112 = 0.06, p = 0.954; absolute costs: t_1,112 = 0.46, p = 0.644). Thus, both models confirm that when a fixed target voice is used (i.e., STB), the reduction in costs is especially pronounced in the HI listeners and that a significant effect of SP (i.e., ST20 vs ST100) can only be seen in STB.

To illustrate the general relationship between costs and cognitive abilities/hearing loss, the costs were averaged across the four dynamic conditions and correlated with the cognitive test scores as well as the BEHL. Averaged costs were used because none of the dynamic conditions in particular were expected to show a stronger correlation with cognitive measures than the other conditions. In Table 3, the corresponding correlation coefficients as well as corresponding p-values are shown.

When correcting p-values for multiple comparisons, significant associations with the relative costs were found for the RST and the composite score. However, none of the cognitive test results showed a significant correlation with the absolute costs. BEHL values were associated with neither the relative nor the absolute costs.

This study examined the costs of dynamic cocktail party listening in relation to static baseline conditions. The main focus was on the effect of hearing loss and cognitive abilities. Two age-matched listener groups with differing hearing were considered. We expected higher costs for the HI listeners than for the NH listeners. In addition, we anticipated that better cognitive abilities are associated with lower costs.

Two cost types were considered. In both cases, a GLMM revealed significant main effects of ST and SP. STB (fixed target voice) caused lower costs than STA (changing target voices). The constant voice might have offered the listeners an additional cue that helped them to identify the target talker. A similar benefit was observed in Brungart and Simpson (2007) and Meister et al. (2020). The finding that conditions with rare switches (SP20) are associated with lower costs than those with switches in every trial (SP100) also agrees with the studies of Brungart and Simpson (2007) and Meister et al. (2020). However, in the present study, significant interactions of group*ST and ST*SP were found additionally. Follow-up revealed that the former interaction was due to the fact that the beneficial effect of the constant target voice did not reach significance in the NH but only in the HI listeners. This was observed even though there is some evidence that hearing loss can negatively impact the ability to identify talkers (Best et al., 2018). The latter interaction was due to the fact that the expected higher costs of frequent talker switches (i.e., SP100) were only found in the condition with fixed target voice (STB). Both interactions stand in contrast to Meister et al. (2020), who examined young and older normal-hearing listeners. This appears to be due to the fact that the data of the HI do not entirely follow the expected pattern (cf. Brungart and Simpson, 2007; Meister et al., 2020) due to their relatively high costs in STA_SP20, a finding that should be addressed in future studies. However, overall, these analyses did not confirm our expectation that costs of dynamic listening are generally higher for the HI listeners. In this context, it should also be considered that the near-ceiling baseline performance in the normal-hearing subjects could have limited their costs—although we do not see any good reason to assume that they are actually larger in NH than in HI listeners.

Further evidence that costs do not differ between the groups is also provided with the outcome of the correlation analysis shown in Table 3, since BEHL was not associated with any of the two cost types. The outcomes of the neuropsychological tests showed somewhat different results. We had hypothesized that better cognitive abilities are associated with lower costs. Indeed, the composite score revealed a significant negative correlation with relative costs but not with absolute costs (p = 0.064). Since the composite score might be related to the subjects' general abilities to cope with the different tests, inspection of the single cognitive domains may be more informative—although we did not have specific hypotheses in this respect. Here, the outcome of the RST, assumed to reflect WMC, was the only domain that revealed a significant association with relative costs. Lin and Carlile (2015) also reported a significant correlation of WMC and costs in their young normal-hearing listeners. In their study, absolute costs as the difference between non-switch and switch stimuli were considered. Similar to the present study, they used a setup involving three competing talkers as well as occasional and unpredictable switches of the target talker location. However, their participants had the task of listening to two successive sets of concurrent sentences and to repeat back six words plus a keyword, which may have imposed a higher memory load than in the present study. In a follow-up study only including non-spatial switches, no significant correlation of costs and WMC was found (Lin and Carlile, 2019). This is in line with Meister et al. (2020), who did not find a significant contribution of the cognitive measures to absolute costs in young and older listeners with normal hearing using the same experimental setup and cognitive tests as in the present study. Thus, methodological variations might account for these different findings.

The fact that only relative costs were significantly associated with WMC of the listeners requires a closer inspection. The rationale behind calculating relative costs was to consider individual differences in the static baseline condition. A more detailed analysis revealed that the performance averaged across the static conditions was actually also significantly related to WMC (r_p(28) = 0.44, p = 0.015). This is in line with the meta-analysis by Akeroyd (2008), who showed that the outcome of RSTs was most effective in predicting speech recognition in noise as well as the general assumption that better WMC is associated with better speech recognition in adverse listening conditions (cf. ELU-model; Rönnberg et al., 2013). Likewise, performance averaged across the dynamic conditions was significantly related to WMC (r_p(28) = 0.52, p = 0.003). Thus, listeners with poorer WMC tended to show lower performance in both dynamic and static listening, and the latter in turn yielded higher relative costs. Hence, while performance in static and dynamic cocktail party listening was significantly related to WMC, this study does not give clear evidence that the costs of dynamic listening depend on cognitive abilities.

This study extends prior research on dynamic cocktail party listening by taking participants with hearing impairment into account. In contrast to our hypothesis, overall costs were not generally higher for the HI participants than for their age-matched peers with good hearing. However, it shows that HI listeners are at least as able to benefit from a fixed target voice as are normal-hearing listeners. Also, in contrast to our expectation, this study did not show clear evidence that costs of dynamic listening are significantly related to the individuals' cognitive abilities, whereas speech recognition in both static and dynamic cocktail party situations was. Since our participants showed mild-to-moderate hearing impairment, future studies should take more severe hearing losses into account.

This study was supported by Deutsche Forschungsgemeinschaft (DFG) Grant No. ME 2751/3-1. The authors thank Fabian Wenzel for his support with conducting the experiments. We are grateful to two anonymous reviewers for their constructive comments on a former version of the manuscript.