Effect of temporal misalignment on understanding Mandarin sentences in simulated combined electric-and-acoustic stimulation Open Access

While electric (E) stimulation based cochlear implant (CI) devices have remarkably restored partial hearing to listeners with profound-to-severe hearing loss, a number of studies have suggested employing combined electric-and-acoustic stimulation (EAS) to CI users preserving low-frequency (LF) hearing in the implanted ear [e.g., Gantz et al. (2006), Luo and Fu (2006), Chang et al. (2006), Micheyl and Oxenham (2012), and Chen and Chen (2018)]. With the EAS technique, a short electrode array is implanted only into the base region of the cochlea in order to preserve the residual acoustic (A) hearing at low frequencies (typically, a hearing level of 20–60 dB up to 750 Hz and severe-to-profound hearing loss at 1000 Hz and above), which many CI patients still have [e.g., Gantz et al. (2006)]. The low-frequency and high-frequency speech information is provided to these patients via a hearing aid (HA) and a CI, respectively. Recently, attempting hearing preservation, many EAS and CI cases used full-length electrodes with soft surgical techniques [e.g., Mick et al. (2014)]. The combined-stimulation advantage (or combined EAS advantage) refers to an improvement in speech recognition when the E portion in a CI is supplemented with the low-frequency information in the A portion [e.g., Micheyl and Oxenham (2012)]. Many early studies have reported that adding low-frequency acoustic information significantly improves CI speech perception (i.e., in the E-only condition), such as in noisy environments [e.g., Chang et al. (2006)] and in the understanding of a tonal language [e.g., Luo and Fu (2006)]. Meanwhile, studies are still actively ongoing to investigate how various factors affect the perceptual benefits of combined EAS. These investigations are of particular importance, as nowadays many advanced speech processing algorithms have been developed for CI users so that their communication performance is improved, e.g., noise suppression with advanced machine learning-based algorithms [e.g., Lai et al. (2018)]. The outcomes of these studies may provide us with more knowledge regarding which conditions EAS can yield improved speech perception performance in relation to that of traditional CI listeners and how many perceptual benefits EAS listeners can achieve. For instance, Chen and Chen (2018) studied the perceptual importance of accurate tonal contour information for understanding EAS-processed Mandarin sentences, and their simulation studies showed that if there are severe distortions to the fundamental frequency (F0) contours (carrying lexical importance for Mandarin word recognition), the combined EAS-based speech perception does not provide a significant advantage compared to the CI-based speech understanding.

While previous studies have suggested that the combined-stimulation advantage may be affected by many factors, the present work specifically assessed the impact of temporal misalignment between the E and A portions on EAS-based sentence understanding. The temporal misalignment in EAS may be caused by at least two reasons, i.e., processing delay difference and inherent biological delay of CI and HA speech processing. First, separate CIs and HAs may have different processing delays and are not designed specifically to work together [e.g., Francart et al. (2015)], which yields a difference of processing delay between a CI and a HA [e.g., Francart and McDermott (2013)]. Recent EAS speech processing combines a CI processor and a HA in one device [e.g., Lorens (2012)], or hybrid processor. While delay compensation can potentially solve the temporal delay differences between CIs and HAs within and without hybrid processors, so far little is known on how this compensation is implemented in commercial devices and how the parameters of CI and HA processing should be set. Second, temporal misalignment between low and high frequency signals in EAS may be partially due to basilar membrane mechanics and its relation to electrical stimulation. Although the CI processor can take longer to process incoming sound, the electrical stimulation occurs almost instantly. In contrast, the acoustic sound from the HA or hybrid processor is presented through the ear canal and middle ear, and takes time to travel the basilar membrane to the apical portion of the cochlea. This inherent biological delay of CI and HA speech processing has an interaction with the electrical delays of the CI- and HA-processed signals.

The aim of this work was to examine whether this temporal misalignment between the two portions in EAS processing affects sentence perception. This study was also motivated by several previous findings on the perceptual influence of the cross-band correlation of speech waveforms on speech intelligibility. For example, Healy et al. (2005) measured the correlations between the fluctuating envelopes of the acoustic stimuli and found that the rate at which the acoustic correlation fell as a function of between-band asynchrony closely matched the rate at which intelligibility fell for normal-hearing (NH) listeners. As temporal misalignment may potentially lead to asynchrony of speech waveforms across bands, it was hypothesized that the temporal misalignment between the E and A portions in EAS would negatively influence the understanding of EAS-processed sentences.

This experiment involved nine listeners (six males and three females) with NH (pure-tone thresholds better than a hearing level of 20 dB at octave frequencies of 125–8000 Hz in both ears). All participants were native speakers of Mandarin Chinese and were paid for their participation. The experimental procedures were reviewed and approved by the Research Ethics Committee of Southern University of Science and Technology.

The speech materials consisted of sentences from the Mandarin Hearing in Noise Test (MHINT) (Wong et al., 2007). Each MHINT list consists of ten sentences, and each sentence contains ten words. Two types of maskers were used to corrupt the sentences, which included a steady-state speech-spectrum-shaped noise (SSN) and a two-talker (i.e., 2 T) babble. To generate the SSN masker, a finite impulse response filter was designed based on the average spectrum of the MHINT sentences, and white noise was filtered and scaled to the same long-term average spectrum and level as the sentences. The two-talker babble contained two (one male and one female) equal-level interfering talkers. A noise segment of the same length as the clean intact (i.e., full-length) speech signal was randomly cut out of the noise recordings, appropriately scaled to reach the desired input signal-to-noise ratio (SNR), and finally added to the speech signals at (0, –2, and –5) dB and (2, 0, and –3) dB input SNRs for the SSN and two-talker maskers, respectively. The input SNR level was chosen to avoid ceiling/floor effects.

This experiment included three signal processing conditions. The first condition simulated the acoustic-only (A-only) stimulation. The speech signal was processed by low-pass filtering to generate the low-pass-processed stimuli. Low-pass filtering was implemented by using a linear-phase finite impulse response filter with a filter order of 10 × fs/fcut, where fs is the sampling rate (16 kHz) and fcut is the low-pass cutoff frequency (fcut = 600 Hz in this study).

The second signal processing condition simulated an eight-band electric-only (E-only) stimulation by using a noise vocoder. To implement the noise vocoder, the speech signals were first processed through a pre-emphasis filter (first-order high-pass filter with a cutoff frequency of 1200 Hz). Then, the signals were bandpass-filtered into eight frequency bands between 80 and 6000 Hz with sixth-order Butterworth filters. The cutoff frequencies for the channel allocation of bandpass filters were 80, 221, 426, 724, 1158, 1790, 2710, 4050, and 6000 Hz, computed according to the cochlear frequency-position mapping function (Greenwood, 1990). From each band, the envelope was extracted by half-wave rectification and low-pass filtering with a 200-Hz cutoff frequency by way of a fourth-order Butterworth filter. White noise was used as the carrier signal, and it was amplitude-modulated by the extracted envelope. The output from each band was further band-limited with the same bandpass filter at that band. All amplitude-modulated noises (with band-limiting processing) were summed to generate the noise-vocoded stimulus, with its amplitude adjusted to have the same root-mean-square (RMS) power as the input speech signal.

The third signal processing condition simulated the combined electric-and-acoustic stimulation. To simulate the effect of EAS with residual hearing <600 Hz, we combined the low-pass stimulus described earlier with the upper five bands (ranging from 724 to 6000 Hz) of the eight-band noise vocoder. Note that the level of the low-pass stimulus was equalized to that of the E-only simulation. Again, RMS power scaling was performed with respect to the input speech signal. More details on generating the EAS-processed stimuli can be found in Chen and Loizou (2010). Note that for the first and second conditions (i.e., A-only and E-only), only one input SNR level was used, i.e., an input SNR of 0 dB for the SSN masker and an input SNR of 2 dB for the two-talker masker, and these conditions served as the baseline performance to demonstrate the combined EAS advantage.

To simulate the effect of temporal misalignment between the E and A portions, a relative time shift was added between the two portions. Three time shift (or temporal misalignment) values were used in this work, including 15, 0, and –15 ms, yielding three E + A conditions of E + A(15), E + A(0), and E + A(–15). The E + A(0) condition denotes that there was no temporal misalignment between the E and A portions; while the E + A(15) and E + A(–15) conditions mean that the E portion was temporally moved to 15 ms earlier and later relative to the A portion, respectively. Note that early study [e.g., Francart and McDermott (2013)] suggested that for combined HA and CI stimulation, the total temporal delay of the E and A portions differs in a range up to tens of milliseconds. Hence, this work used relative temporal delays of 15 and –15 ms to study the potential influence of temporal misalignment on combined stimulation advantage.

The experiment was performed in a sound booth, and stimuli were played to listeners monaurally through an HD 650 circumaural headphone (Sennheiser, Germany) set at a comfortable listening level (i.e., the volume level was adjusted by each participant and maintained during the whole experiment). Before the actual testing session, each participant attended a 10-min training session and was given four lists of ten MHINT sentences [different from those used in the testing session, and in conditions of E-only and E + A(0) with an SSN masker at an input SNR of 0 dB and a two-talker masker at an input SNR of 2 dB]. The training session familiarized the participants with the testing procedure and the EAS-processed test conditions. During the training session, the participants were allowed to read transcriptions of the training sentences while they were listening to the sentences. In the testing session, the order of the conditions was randomized across participants, and the participants were asked to orally repeat all of the words they heard. In addition, the lists were randomized across listeners. Each participant completed a total of 22 conditions [= 2 maskers × 3 input SNR levels × 3 E + A conditions (i.e., E + A(15), E + A(0), and E + A(–15)) + 2 maskers × 1 input SNR level × 2 conditions (i.e., A-only and E-only)]. One list of ten Mandarin sentences was used per test condition, and none of the sentences were repeated across conditions. The participants were allowed to listen to each stimulus for a maximum of three times, and they were asked to repeat as many words as they could recognize. During the testing session, a tester accompanied the participant and scored his/her responses. A 5-min break was given every 30 min to avoid listening fatigue. The intelligibility score for each condition was computed as the ratio between the number of correctly recognized words and the total number of words contained in each MHINT list. The total testing time was one hour.

For all statistical analyses in this work, the recognition scores as percentages were first converted to rational arcsine units by using the rationalized arcsine transform (Studebaker, 1985). Figure 1 shows the mean sentence recognition results (in percentages and under the two masker conditions) to illustrate the combined EAS advantage under the temporally aligned conditions. Under the SSN masker condition with an input SNR of 0 dB, the sentence recognition scores for the E-only and E + A(0) conditions were 59.7% and 93.6%, respectively; while under the two-talker condition with an input SNR of 2 dB, the scores for the E-only and E + A(0) conditions were 33.8% and 86.6%, respectively. Under both conditions, the scores were significantly (p < 0.05) improved from the E-only condition to the E + A(0) condition, suggesting a combined EAS advantage.

Figure 2(a) shows the mean sentence recognition results for all E + A conditions with an SSN masker. Statistical significance was determined by using the recognition score as the dependent variable, and the SNR level and the temporal misalignment condition (i.e., 15, 0, and –15 ms) as the two within-subject factors. Two-way analysis of variance with repeated measures indicated significant effects of the SNR level (F_2,16 = 34.876, p < 0.001) and the temporal misalignment condition (F_2,16 = 5.420, p < 0.05), and a non-significant interaction between the SNR level and the temporal misalignment condition (F_4,32 = 1.421, p = 0.249). Post hoc pairwise comparisons were carried out for each group of conditions with the same SNR level to analyze the effect of the temporal misalignment condition (i.e., among the three E + A conditions). Alpha level for statistical significance was Bonferroni corrected, and only those tests with a p-value lower than 0.017 (=0.05/3) were considered as significant. The results showed that with an input SNR of 0 dB, the mean recognition score under the E and A aligned condition [i.e., E + A(0)] was significantly (p < 0.017) greater than those under the E and A misaligned conditions [i.e., E + A(15) and E + A(–15)], and the mean recognition scores under the two misaligned conditions were not significantly (p > 0.017) different. With an input SNR of –2 dB, the mean recognition scores under the three conditions were not significantly (p > 0.017) different. With an input SNR of –5 dB, the mean recognition scores under the E + A(15) and E + A(0) conditions were significantly (p < 0.017) greater than that under the E + A(–15) condition, and there was no significant difference (p > 0.017) between the mean recognition scores under the E + A(15) and E + A(0) conditions.

Figure 2(b) shows the mean sentence recognition results for all E + A conditions with a two-talker masker. The statistical significance was determined by using the recognition score as the dependent variable and the SNR level and temporal misalignment condition (i.e., 15, 0, and –15 ms) as the two within-subject factors. Two-way analysis of variance with repeated measures indicated significant effects of the SNR level (F_2,16 = 49.039, p < 0.001) and the temporal misalignment condition (F_2,16 = 7.111, p < 0.05), and a non-significant interaction between the SNR level and the temporal misalignment condition (F_4,32 = 2.259, p = 0.085). Post hoc pairwise comparisons were carried out for each group of conditions with the same SNR level to analyze the effect of the temporal misalignment condition (i.e., among the three E + A conditions). Alpha level for statistical significance was Bonferroni corrected, and only those tests with a p-value lower than 0.017 (=0.05/3) were considered as significant. The results showed that with an input SNR of 2 dB, the mean recognition score under the E and A aligned condition [i.e., E + A(0)] was significantly (p < 0.017) greater than those under the E and A misaligned conditions [i.e., E + A(15) and E + A(–15)], and the mean recognition score under the E + A(15) condition was significantly (p < 0.017) greater than that under the E + A(–15) condition. With an input SNR of 0 dB, the mean recognition scores under the E + A(15) and E + A(0) conditions were not significantly (p > 0.017) different, but both were significantly (p < 0.017) greater than that under the E + A(–15) condition; while with an input SNR of –3 dB, the mean recognition scores under the three E + A conditions were not significantly (p > 0.017) different.

The present work assessed the effect of temporal misalignment between the electric and acoustic portions on simulated EAS-based sentence understanding. There were two main findings in this work. First, this work showed that the temporal misalignment examined in EAS processing may potentially cause a negative influence on the sentence perception task. As shown in Fig. 1, when there was no extra delay between the two portions, i.e., in the temporally aligned condition E + A(0), the greatest combined-stimulation advantage was obtained (i.e., 33.9% and 52.8% under the SSN and two-talker maskers, respectively). When temporal misalignment was introduced between the two portions in EAS, yielding the temporally misaligned conditions E + A(15) and E + A(–15), the combined-stimulation advantage was significantly reduced (see Fig. 2). This finding indicated the importance of preserving temporal alignment between the E and A portions in EAS processing. To some extent, this result was consistent with previous findings regarding the relative importance of between-band synchrony on speech perception [e.g., Healy et al. (2005)]. This work simulated the effect of asynchrony between the E and A portions in EAS processing by adding a relative delay between the two portions. In real scenarios, this asynchrony between the E and A portions may be caused or exaggerated by the difference of the processing time of the E and A portions' signal processing blocks. Note that the present work used Mandarin Chinese as the test language. Mandarin Chinese differs from English in many aspects [e.g., Howie (1976) and Chen et al. (2013)], including the perceptual importance of the low-frequency region (Fogerty and Chen, 2014), the increased perceptual importance of vowel segments (e.g., Chen et al., 2013), its monosyllabic word structure (e.g., Chen et al., 2013), etc. It is unclear whether and to which extent the misalignment influence will affect the perception of English speech.

Second, the results depicted in Fig. 2 show that while the two temporally misaligned E + A conditions [i.e., E + A(15) and E + A(–15)] affected the combined-stimulation advantage, their influences were different. Figure 2 shows that at several conditions (e.g., an SSN masker at an input SNR of –5 dB, and a two-talker masker at an input SNR of 0 dB), the E + A condition with a preceding A portion, i.e., E + A(–15), caused a much greater decrease of the intelligibility score than the E + A condition with the preceding E portion, i.e., E + A(15). This finding suggests that the difference between the group delays of the E and A portions may have a different influence on the combined EAS advantage. The exact reason for the relatively better understanding under the E + A condition with the preceding E portion or the worse performance under the E + A condition with the preceding A portion is unclear. We hypothesize that under the E + A condition with the preceding E portion, listeners can still reliably identify the landmarks (i.e., onset and offset) for vowel segments with the presence of important acoustic cues (e.g., F0 contour) in the A portion. This is because Mandarin Chinese (the test language used in this work) has a monosyllabic word structure, i.e., consonants (mainly in the high-frequency region or E portion) followed by vowels (mainly in the low-frequency region or A portion), and the preceding E portion does not cause a severe interference between the E and A portions. This onset (or landmark) information (contained in the A portion) carries importance for speech perception under challenging conditions, which has been shown in many previous studies [e.g., Chen and Loizou (2010)]. On the other hand, under the E + A condition with the preceding A portion, the early appearance of the A portion will interfere with the ending region of the E portion, potentially deteriorating the perception of the onset of vowel segments and consonant-vowel transitions in Mandarin sentences and subsequently decreasing sentence understanding. Further investigations are needed to test this hypothesis. Because of its interaction with the electrical delays of the CI- and HA-processed signals, the inherent biological delay the CI- and HA-processed signals may also potentially affect the observed large changes in benefit with the E portion being moved after than before the acoustic signal.

Note that the present work was studied with a vocoder based acoustic simulation, which has been long used to specifically investigate the impact of individual factor (i.e., temporal misalignment in this work) on CI and EAS speech perception [e.g., Chang et al. (2006), Luo and Fu (2006), and Chen and Loizou (2010)] and was shown to predict the trend in performance observed in CI and EAS users well. Hence, the findings of this work have potential clinical relevance for EAS listeners and insightful implications for the design of speech processing and coding strategies for the combined EAS technique, e.g., the potential delay difference between CIs and HAs within and without hybrid processors. Early work [e.g., Francart and McDermott (2013)] showed that for combined HA and CI stimulation, the total temporal delay of the E and A portions differs in a range up to tens of milliseconds. An algorithm optimized for a hearing aid may cause a group delay of roughly 20–30 ms [e.g., Stone and Moore (1999)]. Currently, powerful and complicated speech processing methods are being used to improve the speech perception of CI and EAS users in challenging conditions, e.g., in noisy surroundings (Lai et al., 2018). While the applications of these signal processing tools bring significant speech perception benefits, the associated negative influence should also be taken into account, e.g., the temporal misalignment between the E and A portions in EAS processing caused by the difference of group delays of the processed signals in the E and A portions. Particularly, this work suggested that, for Mandarin Chinese-based EAS speech perception, the E + A condition with a preceding A portion (e.g., due to a relatively smaller group delay in the A portion than in the E portion) may cause a much more significant performance reduction than that with a preceding E portion. Hence, signal processing in EAS should be carefully designed to avoid a large difference of group delays between the E and A portions. On the other hand, note that vocoder simulations are carried out with presenting synthesized stimuli to normal-hearing listeners, and do not work in the same temporal manner as CIs do, i.e., using multi-channel modulated electrical pulse trains to elicit speech sensation. Particularly, using noise as carrier signal in vocoder-based stimulus synthesis may yield extra waveform fluctuation, which is absent in CI processing. In this regard, further investigation with EAS individuals is warranted.

In conclusion, the present work studied the effect of temporal misalignment between the electric and acoustic portions on Mandarin sentence perception in simulated EAS processing. The results showed that the temporal misalignment between the two portions in EAS may potentially decrease the combined-stimulation advantage in relation to EAS with temporally aligned E and A portions. The preceding A portion caused a much more negative influence on the combined-stimulation advantage in understanding Mandarin sentences than the preceding E portion.

This work was supported by the National Natural Science Foundation of China (Grant Nos. 61971212, 61771023), the Basic Research Foundation of Shenzhen (Grant No. KQJSCX20180319114453986), and High-level University Fund G02236002 of Southern University of Science and Technology.