Some, but not all, cochlear implant users prefer music stimuli with congruent haptic stimulation Open Access

Tactile vibrations can enhance music listening experiences (Merchel and Altinsoy, 2014), body movement, and ratings of enjoyment (Hove , 2020). Since cochlear implant (CI) users are often unsatisfied with how music sounds after implantation, a haptic device might seem like a viable, hearing-agnostic way to enhance music listening. How can we optimize a music-haptic device for a cochlear implant user? A previous study from Aker (2022) showed that vibrotactile music enjoyment enhancement in normal-hearing listeners can occur by ensuring congruence between the acoustic intensity and timing and the tactile intensity and timing in audio-tactile music stimuli. While perceptual abilities related to intensity and timing are considered similar between CI users and normal-hearing listeners, it is not known whether the importance of ensuring audio-tactile congruence for intensity and timing that was found in normal-hearing listeners is also important in cochlear implant listeners. In this study, we therefore investigated whether the tactile parameters of intensity, fundamental frequency, and timing must be congruent with their auditory counterparts for vibrotactile music enjoyment enhancement to occur in CI users.

A CI is a surgically implanted medical device for severely hearing-impaired listeners. Cochlear implants can significantly increase speech understanding for individuals with hearing impairment; 3 months after implantation, people with severe hearing loss achieve a mean sentence recognition score of more than 80% (Wilson and Dorman, 2008). Cochlear implants operate by stimulating the auditory nerve with an intensity related to the temporal envelope of an acoustic signal picked up by a microphone. In the case of speech understanding, the envelope of the acoustic signal is enough to perceive much of the speech information (Shannon , 1995). However, music perception is a more complicated challenge (see Limb and Roy 2014; McDermott 2004; Marozeau , 2014, for reviews).

Music perception in cochlear implant users can be judged on at least two criteria: accuracy of perception (performance metrics) and appraisal (subjective metrics) (Gfeller , 2008). Perceptual accuracy in CI users can be quite good for certain musical features, particularly those captured by the temporal envelope of the signal. These include rhythm (Brockmeier , 2011; Cooper , 2008; Kong , 2004), meter (Cooper , 2008), tempo (Kong , 2004), and broad temporal features of timbre (Kong , 2011). However, perceptual accuracy suffers in CI users for features that depend on the fine temporal or spectral features of a music signal, such as pitch (Brockmeier , 2011; Cooper , 2008; Drennan , 2015; Kong , 2004), spectral or fine temporal features of timbre (Brockmeier , 2011; Kong , 2011), or auditory streaming (Brockmeier , 2011; Marozeau , 2013; Paredes-Gallardo , 2018a; Paredes-Gallardo , 2018b). CI users have also been shown to have reduced accuracy in identifying musical emotion (Hopyan , 2011; Paquette , 2018).

However, determining the accuracy with which music is perceived is only one way to assess the effect of cochlear implantation on music listening. As music listening is often done for pleasure, the appraisal of musical preferences is also important when measuring music perception in CI users. Measuring music appraisal scores independently of music accuracy performance is important as appraisal scores are shown to not correlate well with performance (Drennan , 2015; Gfeller , 2008; Wright and Uchanski, 2012). CI users often report listening to music less after implantation than prior to deafness (Drennan , 2015; Gfeller , 2000; Migirov , 2009; Mirza , 2003). Reduced listening habits have been correlated with increasing age and reduced speech perception abilities (Drennan , 2015; Gfeller , 2000; Mirza , 2003); however, Gfeller (2008) demonstrated that speech perception abilities are only a predictor of music perception tests that involve lyrics. What is common to all studies is that the music listening habits of CI users are highly variable and strong predictors of music enjoyment in CI users have not yet been found. Additionally, even though music is considered a challenge for CI users, most CI users still actively listen to music for a myriad of reasons, including the social aspects, or simply because music processed via a CI is still better than no music at all (Gfeller , 2000; Gfeller , 2019a; Gfeller , 2019b).

Tactile vibrations could be a way to enhance music enjoyment for CI users. Tactile vibrations are known to enhance the music listening experiences for normal-hearing listeners (Merchel and Altinsoy, 2014) as well as body movement and enjoyment ratings listening to music (Hove , 2020). There are several commercial and research devices that utilize vibrotactile music enjoyment enhancement. The Woojer (San Jose, CA) (Woojer, 2020), SUBPAC (Toronto, Canada) (SUBPAC, 2022), Skinetic (Paris, France) (Actronika, 2023) and TactSuit (Daejeon, South Korea) (bHaptics, 2023) are commercially available haptic gilets marketed for music enthusiasts and gamers. Prototypes have also been developed in research settings (Haynes , 2021; Karam , 2010; Nanayakkara , 2009; Remache-Vinueza , 2022). Two panels of CI users discussing their experiences with music in 2018 and 2021 provided the basis for Gfeller (2019b) and Gfeller (2022). The CI users were chosen as they were considered to be highly engaged in music, so they might not be representative of the general CI user population. However, the utilization of all senses in music listening (including tactile, visual, and proprioception) was the most common music listening strategy reported in Gfeller (2019b). One CI user reports, “I believe that engaging all of the senses is very helpful to regain music perception” (Gfeller , 2019b, p. 5). In Gfeller (2022), playing a musical instrument was reported to be an important part of music listening rehabilitation due to its multisensory nature.

Would the positive benefits of tactile vibration that normal-hearing listeners experience for music also apply for CI users? In order to start working towards an optimized audio-to-tactile coding in music-haptic devices, Aker (2022) investigated how audio-tactile congruence affected vibrotactile music enjoyment enhancement. The procedure used ensured that the audio-tactile stimulus being rated was only directly compared to audio-tactile stimuli with a similar physical distribution of onset/offsets, durations, frequencies, and intensities in the vibrotactile component. This paradigm ensured that participants were rating each vibrotactile stimulus based on its relation to the auditory stimulus, rather than any pleasure derived innately from the physical stimulation itself (like a massage). The results of the study indicated that participants preferred audio-tactile music stimuli in which the intensity and timing were congruent between the auditory and tactile components. Intensity and timing are two features that are well perceived by CI users.

Vibrotactile stimulation has already been used to enhance the perceptual abilities of CI users. It has been shown that presenting the envelope of speech (Fletcher , 2019), or the fundamental frequency (Huang , 2017) through vibrotactile stimulation, can increase speech intelligibility in CI users. Huang et al. has also shown that presenting the fundamental frequency can enhance melody recognition (Huang , 2020). By using the intensity of tactile stimulation as a proxy for interaural level differences, CI users are able to more accurately determine the spatial location of auditory stimuli (Fletcher , 2020a). In normal-hearing listeners with a simulated CI, tactile stimulation has been shown to enhance pitch discrimination by using the location of the vibrotactile stimulation on the arm as a map for pitch (Fletcher , 2020b) and melodic contour identification with the fundamental frequency mapped to vibration frequency (Luo and Hayes, 2019). With all that considered, it seems likely that most CI users' music listening experience will be enhanced when auditory music is accompanied by vibrotactile stimulation in which the intensity and timing are congruent.

In this experiment, we tested the hypothesis that audio-tactile congruence would affect CI users like normal-hearing listeners, and that only intensity and timing congruence between the modalities is required for vibrotactile music enjoyment enhancement to occur in CI users. We tested CI users and normal-hearing listeners in a paradigm similar to Aker (2022). Participants were presented with simple melodies through a CI streaming device or headphones, respectively, alongside sinusoidal vibrotactile stimulation which was either congruent or incongruent with the auditory stimulus in terms of intensity, fundamental frequency, and timing. Participants rated both the congruent and incongruent stimuli based on preference. The effect of each parameter's congruence or incongruence on participant preference could then be measured.

The experiment was preceded by a Goldsmith Music Sophistication Index (Gold-MSI) (Müllensiefen , 2014) questionnaire to get data on each participant's musical background, a vibrotactile intensity matching task to equalize tactile intensity for each frequency, and finally, a vibrotactile frequency discrimination task to get data on each participant's sensitivity to changes in tactile vibration frequency.

The methodology was adapted from Aker (2022). The procedure ensures that participants only directly compare audio-tactile stimuli with similar physical characteristics to control for the presence of vibrotactile stimulation, which may be enjoyable and increase preference ratings irrespective of its information content. The primary changes from Aker (2022) include adding four new auditory melodies, and changing the acoustic waveform from a sine wave to a sawtooth wave to ensure that pitch was perceivable in the frequency range of the CI. Additionally, five combinations of congruent–incongruent parameters were tested on every screen.

Participants consisted of one cohort of 18 CI users between the ages of 41 and 82 years (mean age 64 years; 12 female) and a second control cohort of 24 self-reported normal-hearing listeners between the ages of 19 and 27 years (mean age 24 years; 11 female). CI users were recruited from an internal database with an effort to have all four brands of CI represented. For a table of statistics related to the CI users, see Table I.

The sample size was determined a priori with a power analysis performed with G*Power version 3.1.9.7 (Düsseldorf, Germany) (Faul , 2007) on data from Aker (2022). The minimum sample size for a Friedman test with two groups with a significance criterion of $α = 0.05$ and a power of 0.8 was found to be N = 9. Therefore, a sample size of 18 CI users and 24 normal-hearing listeners was determined to be sufficient.

A large difference in ages between the CI user cohort [standard deviation (SD) = 12.68] and the normal-hearing listener cohort (SD = 2.23) was made abundantly clear by a two-sample t-test t(df) = 15.1110; p = 6.8237 × 10⁻¹⁸, where df represents degrees for freedom. Younger, normal-hearing listeners were chosen as the control group to evaluate whether the present study's results were consistent with previous findings (Aker , 2022). However, results from the present study must also, therefore, be evaluated through the lens of the large difference in age groups between the two cohorts.

All participants completed the Gold-MSI (Müllensiefen , 2014) questionnaire to get a background on their musical training and previous experience. The Gold-MSI questionnaire is meant to give a total score which is made up of five sub-scores in five possible categories: active engagement, perceptual abilities, musical training, emotions, and singing abilities. While all components of the Gold-MSI were given to participants, the questionnaire was administered primarily for the musical training sub-score, which consists of questions like “I engaged in regular, daily practice of a musical instrument [including voice] for ___ years.” The test was administered since musicianship has been previously shown to affect audio-tactile congruence (Aker , 2022). CI participants were also given a second questionnaire at this time, to determine when they were implanted, the CI brand, and their device setup [bilateral, bimodal, single-sided deafness (SSD), etc.].

A preliminary analysis of the participants' Gold-MSI musical training sub-score was performed using a two-sided Wilcoxon rank sum test to compare the CI user and normal-hearing listener cohorts. There was a significant difference in the musical training sub-score between the CI user cohort (M = 15.4; SD = 7.26) and normal-hearing listener cohort (M = 28.3; SD = 11.4); z = –3.3802; p = 7.2434 × 10⁻⁴. While this difference may be representative of real life differences in musical training between CI users and normal-hearing listeners, it is still an additional, significant difference between the two cohorts. Any results, therefore, must be framed with this confound variable in mind. A histogram of both cohorts sub-score, and the sub-scores combined can be seen in Fig. 1. Since Gold-MSI scores are based on a Likert scale, scores could not be analyzed as continuous variables. Therefore, participants were classified as “musicians” or “non-musicians” for any future analysis using a k-medoids cluster analysis.

Auditory stimuli could be presented either acoustically for normal-hearing participants, or electrically for CI users. Acoustic stimuli were presented over a pair of Sennheiser HD200 headphones (Sennheiser, Wedemark, Germany) and a Focusrite Scarlett 4i4 audio interface (Focusrite, High Wycombe, United Kingdom). To mask any perceivable sound generated by the haptic stimulator, acoustic stimulation was presented with pink masking noise at a constant level with overall signal-to-noise (SNR) of 30 dB to hide any possible noise from the haptic stimulation.

Electric stimuli were presented through participants' wireless listening device, according to their CI manufacturer: the Oticon Medical Streamer (Demant A/S, Smørum, Denmark), Phonak ComPilot (Sonova Group, Stäfa, Switzerland), Cochlear Wireless Mini Microphone 2+ (Cochlear Ltd, Sydney, Australia), or the MED-EL AudioLink (MED-EL, Innsbruck, Austria). Audio was wired to the streaming device through a Focusrite Scarlett 4i4 audio interface with a 3.5 mm audio jack, which then connected to the appropriate sound processor wirelessly. Bilateral CI users were instructed to connect to the wireless listening device as they normally listened to music, which included both cochlear implants in all cases. Bimodal participants were asked to remove their hearing aid, and the two participants with SSD used an ear plug in their normal-hearing ear. Based on the intensity of the haptic stimulation and the residual hearing of the participants, the masking methods (removal of hearing aids and ear plugs) were deemed sufficient to prevent any auditory cues from the haptic device. Participants gave an additional, informal confirmation that they could not hear the haptic stimulation.

Tactile stimuli were presented through a Lofelt L5, a linear resonant actuator developed by (Lofelt, Berlin, Germany). The Lofelt is a flat, 2 ×2 × 0.5 cm cuboid shape that covers the fingertip when held between the thumb and finger. When stimulated with an alternating current (AC) waveform, the cuboid vibrates laterally relative to the finger. Participants were instructed to hold the Lofelt comfortably between their thumb and forefinger, inside a fabric-lined box, which was intended to both dampen the sound of the actuator as well as hide it visually. The actuator was connected to the computer via a Focusrite Scarlett 4i4 audio interface.

Music stimuli consisted of five melody excerpts manually recreated from their original source. Music notation for each melody can be seen in Fig. 2. Excerpts were manually transcribed to a musical instrument digital interface (MIDI) format using Ableton Live 10 (Berlin, Germany). The MIDI files were then transposed so that the lowest note corresponded to a frequency of 73.42 Hz (D2). The velocities were set to have an root mean square (RMS) of either 58 or 70 dB sound pressure level (SPL) based on the original excerpts. The MIDI files were then rendered into digital waveforms to be presented either acoustically or electrically. Each note was transformed into a sawtooth wave with only the first ten harmonics, a fundamental frequency equal to the pitch of the respective MIDI note, a 31 ms linear onset, and an exponential decay with an exponential decay constant of 2. Electric stimuli had a 12 digital dB RMS difference between high and low intensity stimuli. Participants were instructed to adjust the overall level with a virtual, on-screen slider to a comfortable level prior to the experiment.

It is impossible to fully represent the world's catalogue of music with a limited selection of single track melodies. Even within the constraints of using single track melodies, it is impossible to account for every possible permutation of intensity, timing, and fundamental frequencies. Therefore, the five melody excerpts were selected specifically to challenge the central hypothesis that CI users and normal-hearing listeners would have their vibrotactile music enjoyment affected by intensity and timing congruent haptic stimulation. The first melody, “Bass Track,” was selected as a baseline stimulus since it was used in a previous study (Aker , 2022). The four remaining songs were then chosen to test secondary hypotheses by being different from the “Bass Track” song on some aspect, while still being adapted directly from an existing song. The excerpts, or songs, were then analyzed categorically (as opposed to along a continuous variable, such as tempo) to evaluate holistically how these varying parameters could affect the vibrotactile music enjoyment enhancement process. Regardless, the songs chosen do not, and cannot, represent all music and there are still numerous confounds to each stimulus. An overview of the parameters in the excerpt choices can be seen in Table II.

The first melody chosen, “Bass Track,” was the one used in Aker (2022), adapted from the bass track of a pop song from the Hungarian band “Chalga” titled “szárad a száj.” The melody was chosen to enable direct comparisons to be made between the present study and the previous work described in Aker (2022). The song was initially selected because it is a high-valence, positive song, and therefore assumed to be easier to evaluate by participants vs a “sad” song in which participants could interpret “enhancement” as “more sad” or “less sad.” It was also chosen as it would not be recognizable to most participants, to avoid any pre-existing emotional associations. The bass track of the song was adapted to resemble natural situations, in which it is lower acoustic frequencies that will be transduced mechanically and perceived through the tactile system. As another benefit, the songs bass line follows the overall rhythm of the song, so the adapted melody is salient in the actual music recording. The tempo was set at 100 bpm.

The “Bass Track” song was used as a baseline to then select the other four stimuli.

The “Beethoven” song tested the secondary hypothesis that the effect of intensity and timing congruence on participant enjoyment was due to the original “Bass Track” stimulus being a rhythm track. Being part of the rhythm section of the band, it was thought that the “Bass Track” stimulus might overly emphasize timing and intensity changes in its composition. Therefore, another stimulus dubbed “Beethoven” was adapted from the Beethoven song violin sonata No. 9 “Kreutzer,” which was thought to be less rhythmic (being classical vs pop) and not adapted from the bass track. If the “Beethoven” song showed a smaller effect size of intensity and timing congruence, then this secondary hypothesis would have evidence supporting it. The tempo was set at 270 bpm, as it was in the piece.

The “Amen break” is an oft-sampled drum track in hip hop and electronic music. In Aker (2022), which exclusively used the “Bass Track” song, frequency congruence between the audio and tactile stimuli components was not shown to have an effect on participant preference ratings. It was hypothesized one reason could be due to the frequency differences between notes being too small to be consistently detected by the tactile system. Therefore, the third stimulus, “Amen Break,” was adapted from a drum track, in which the snare drum was mapped to the highest pitch (146.83 Hz), and the bass drum to the lowest pitch (73.42 Hz). The hi hat was omitted. With only two possible frequencies of vibration, and both being as different as possible with the apparatus used, frequency differences would be easier to detect by participants. Therefore, a lack of congruence between audio and tactile frequencies may have a larger effect on participant preference ratings. The tempo was set at 100 bpm.

A melody called “Isolation Waltz” in 3/3 written by Bryan Teoh in public domain was chosen due to the duration of its notes. Another potential reason frequency congruence between the audio and tactile stimuli components was not shown to have an effect on participant preference ratings in Aker (2022) was the rate at which the notes changed. If the duration of each note was too short, low-frequency tactile vibrations might not run through enough cycles to be consistently detected by participants. Therefore, if an effect of fundamental frequency congruence on participant ratings for the “Isolation Waltz” stimulus is shown, fundamental frequency congruence may only be beneficial for relatively slow songs. The tempo was set at 125 bpm.

A major scale rising to the highest pitch (146.83 Hz) and then falling. The first, rising eight notes were made high intensity, and the seven falling notes were made low intensity. The scale was chosen as a melody with a clear pitch directionality. Similar to the third stimulus, “Amen Break,” it was thought that a stimulus with clear frequency changes might enhance the salience of tactile frequency perception, and therefore show a greater effect on participant preference ratings when congruence with the audio fundamental frequency. The tempo was set at 80 bpm.

Tactile tone stimuli consisted of 1 s–duration sine waves presented through the Lofelt with a frequency of either 50, 75, 100, 125, 150, or 180 Hz. The sine waves had a 50 ms Hanning window applied.

Tactile music stimuli were derived from the same MIDI files used for the auditory stimuli. The MIDI files first had randomization applied to them to achieve the required level of incongruence from the auditory stimulus.

Three possible randomizations could be applied to the MIDI files prior to transformation to a digital waveform for use as tactile stimuli, in any combination. Each randomization made the tactile stimulus incongruent with the corresponding auditory stimulus along one of three parameters: intensity, fundamental frequency, or timing. However, the randomizations were designed to ensure the physical properties, or overall “feeling” of the tactile stimuli remained the same. An overview of these randomizations can be seen in Fig. 3.

Intensity was randomized by shuffling the velocity assignments of each MIDI note prior to waveform transformation. Fundamental frequency was randomized similarly by shuffling the pitch assignment of each MIDI note. Timing was randomized by manually dividing each melody into musical phrases based on the original music excerpts (for example, eight bars of four beats each). New rhythm permutations were then generated for each phrase, each of which had the same number of notes and the same total duration of notes as the original phrase. A $1 16$ note was the smallest possible note denomination. One of these new permutations was then randomly selected for each phrase. The result was a new rhythm with the same time signature and tempo as the original rhythm, and with approximately the same distribution of notes over the entire rhythm. For example, four $1 4$ notes could never be replaced by one whole note, since the two rhythms have two different numbers of notes. However, four $1 4$ notes could be replaced by two $1 8$ notes, a $1 4$ note, and a $1 2$ note.

In Eq. (1), $f stretch$ is the new frequency of the note, and $f orig$ is the original frequency of the note. $f min$ is the lowest frequency in the range (73.42 Hz, or D2). $a$ is the stretching factor.

The tactile intensity of each note was adjusted based on each participant's individual equalization. MIDI notes with a “high intensity” were adjusted to have the same perceived intensity as a 100 Hz sine wave with an RMS voltage of 600 mV at the actuator, corresponding to a maximum displacement of approximately 3 mm. MIDI notes with a “low intensity” were adjusted to have the same perceived intensity as a 100 Hz sine wave with an RMS voltage of 300 mV at the actuator, a peak displacement of approximately 1.5 mm.

The testing procedure had the following steps: first, participants were instructed to fill out the Gold-MSI (Müllensiefen , 2014) to get a background of each participant's musical training. Results for this questionnaire are discussed under Sec. II A. Second, an intensity matching task was completed so that the intensity of the tactile stimulation for future tasks could be calibrated on an individual basis, and tactile frequency would correspond to one of two intensity levels (“low” intensity or “high” intensity). Third, a tactile frequency discrimination task was completed to get a background of each participant's frequency discrimination sensitivity. Finally, the rating task was completed. In the rating task, participants would be presented with an auditory music stimulus and five possible corresponding tactile music stimuli. Each audio-tactile stimulus was congruent along different parameters. Participants were instructed to rate each combination between zero and 100 based on personal preference.

An intensity matching task was included to equalize the perceived intensity for participants across relevant frequencies. Participants were presented with 12 trials, each trial consisting of a virtual knob on a screen and alternating tactile tones being presented to participants. The alternating vibrations consisted first of a reference vibration at 100 Hz and then a comparison vibration that had a frequency of either 50, 75, 100, 125, 150, or 180 Hz, depending on the trial. The comparison tone had a level that could be adjusted with a virtual knob. The reference tone had an RMS voltage level of 600 mV for six trials (the “high” intensity reference level) and an RMS voltage level of 300 mV for six trials (the “low” intensity reference level). The vibrations repeated until the participant was satisfied with their match. Participants were instructed to adjust the level of the comparison vibration until it matched the reference vibration. Equal-intensity frequency curves for both the “high” and “low” reference levels were interpolated between tested frequencies and used for all subsequent tasks.

Tactile frequency just-noticeable differences (JNDs) were determined using a 3-alternative forced choice frequency discrimination task. Participants were presented with three buttons, denoting three different tactile stimuli. Each stimulus was presented to the participant in turn. Participants were instructed to press the button corresponding to the tactile stimulus that was different from the other two. Feedback was given. Two of the stimuli were 75 Hz sine waves, while the other “target” stimulus was a sine wave that started with a frequency 40% higher. After two consecutive correct responses, the difference in frequency between the standard and target stimuli would decrease by 5%, while after a single incorrect response, the difference would increase by 5%. Eight turnarounds were tracked, defined by two correct responses followed by an incorrect response. The first two of the eight turnarounds were omitted. Following the first two turnarounds, the step size was decreased by half. The frequencies of the last six were averaged to determine the participant's tactile frequency JND.

Finally, participants completed the rating task in which audio-tactile music stimuli with differing levels of congruence were rated. Prior to the rating task, CI participants had their wireless listening devices connected and a comfortable listening level established. CI participants verified that they could hear six notes in a six-note test melody which spanned the entire frequency range of the stimuli and both “high” and “low” intensity notes.

The rating task consisted of 12 trials. Ten of the trials corresponded to one of five songs and one of two tactile frequency maps: standard or stretched. The first trial was a practice trial used to explain the procedure, and the 12th trial was an exact duplicate of the second trial to test reliability.

In each trial, participants were presented with five buttons and five corresponding sliders. The graphical user interface (GUI) was based on the MUltiple Stimuli with Hidden Reference and Anchor (MUSHRA) (Series, 2014) protocol. The sliders ranged from zero to 100. When pressed, each button would present a single auditory music stimulus and a corresponding tactile music stimulus. In each trial, the auditory music stimulus was always the same; however, the tactile music stimuli had different randomizations applied in each trial to create a different level of congruence with the auditory music stimulus. Participants were then instructed to rate each audio-tactile stimulus using the sliders based on their personal preference. Specifically, participants were told to rate the auditory component, but to consider how it was enhanced by the tactile component. Participants were not told that any randomization had been applied, only that each version had a different audio-to-tactile map that was being tested. The assignments of the stimuli to the buttons were randomized.

To test the hypothesis that vibrotactile music enhancement is affected by audio-tactile congruence in CI users, a Friedman test was performed to compare the effect of audio-tactile congruence on the mean preference ratings for all songs of the CI user cohort. A separate test was performed on the standard and stretched map ratings. A Friedman test was performed on the normal-hearing listener cohort for both standard and stretched maps as a control, leading to a total of four tests. A significant result of the Friedman test (p ≤ 0.05) would imply that preference ratings were affected by audio-tactile congruence; however, not specifically on what parameter (i.e., intensity, fundamental frequency, and/or timing).

Subsequent analysis made use of linear models which were fitted to each song and map condition to test for an interaction between participant cohort (CI user vs normal-hearing listener) and audio-tactile parameter congruence (fundamental frequency congruence, intensity congruence, and time alignment). Any interaction between the participant cohort and audio-tactile parameter congruence could provide evidence that CI user preference ratings are not affected by audio-tactile congruence along the same parameters that normal-hearing listeners are (intensity and timing) as shown in Aker (2022).

Ratings were first transformed with a rationalized arcsine transform (Studebaker, 1985) to better conform to a normal distribution. A different linear model was constructed for each song and map combination, since each song and map combination corresponded to a different screen, and ratings were instructed to be made relative to the other stimuli on a screen. All linear models had intensity congruence, fundamental frequency congruence, timing congruence, participant cohort (normal-hearing listener vs CI user), and musicianship (musician vs non-musician) as fixed effects. Fixed effects that were non-significant (p > 0.05) were removed.

Song and map conditions can be compared across models to investigate how each song condition interacts with the effect of audio-tactile congruence on preference ratings. In previous work (Aker , 2022), only one music stimulus was used, and therefore its results had several hypothesized confound variables, such as the rhythmicity of the song, the magnitude of frequency difference between notes, the average duration of the notes in the song, and the salience of the directionality of the pitches. The additional stimuli in the present study were selected to address these variables, and therefore, differences in results between the five songs could further elucidate which compositional factors interact with audio-tactile congruence to affect participant preference ratings.

The CI-MUSHRA score was calculated for each CI participant, for each congruence condition. A CI-MUSHRA score was also calculated for each normal-hearing participant for comparison purposes, by following Eq. (2) but omitting the given normal-hearing participant's rating from the mean $μ$ and standard deviation $σ$ calculation.

Overall, preference ratings were higher when the tactile parts of the stimuli were congruent with the audio than when they were not (compare the first pair of bars in Fig. 4 with the fifth pair of bars). In general, although the ratings for normal-hearing listeners are high when all components of the stimuli are congruent, the ratings appear to drop slightly when the intensity becomes incongruent, and dramatically when the timing becomes incongruent. The ratings for CI users were also affected by intensity and timing congruence, but to a lesser degree. A summary of the ratings for each congruence condition can be seen in Fig. 4 for the standard map and Fig. 5 for the stretched map.

A Friedman test was used to test the main hypothesis by comparing the effect of audio-tactile congruence on the mean preference ratings for all songs of the CI user cohort. There was a statistically significant (p < 0.05) difference in mean preference ratings between at least two audio-tactile stimuli in the standard map (chi-squared = 12.4; p = 1.49 × 10⁻²) and the stretched map (chi-squared = 22.4; p = 1.70 × 10⁻⁴). The same test was also performed on the mean preference ratings of the normal-hearing cohort in the standard map and stretched map as a control population. There was a statistically significant (p <0.05) difference in mean preference ratings between at least two audio-tactile stimuli in the standard map (chi-squared = 52.6; p = 1.05 × 10⁻¹⁰) and the stretched map (chi-squared = 58.3; p = 6.50 × 10⁻¹²).

An overview of these results are shown in Table III.

The hypothesis that audio-tactile congruence would affect vibrotactile music enjoyment enhancement in CI users the same as normal-hearing listeners was further investigated using linear models. A linear model was fitted to each song and map condition to test for an interaction between participant cohort (CI user vs normal-hearing listener) and audio-tactile congruence (fundamental frequency congruence, intensity congruence, and time alignment).

The resulting model coefficients are shown in Fig. 6, organized by participant cohort, song, and map. On each graph, the y axis shows the estimated coefficient and the three columns on the x axis show each congruent audio-tactile parameter. A higher estimated coefficient indicates a larger, positive effect size on a participant's preference ratings. In general, timing congruence always had a positive and significant (p <0.05) effect on participant ratings for all cohorts, songs, and maps. However, the effect size for CI users was smaller than for normal-hearing listeners.

Intensity congruence had a significant (p <0.05) and positive effect on ratings for all conditions with the standard map, except for the “Isolation Waltz” song. There was a significant (p <0.05) and positive effect on ratings in the stretched map condition only with the “Bass Track” and “Major Scale” songs. There was no difference between the two cohorts; however, musicians had a greater effect than non-musicians in two conditions: the “Isolation Waltz” with the standard map and the “Amen Break” with the stretched map.

Fundamental frequency congruence showed no significant (p <0.05) effect on participant ratings for all conditions except the “Major Scale” with the standard map, and the “Amen Break” with the stretched map. This was consistent between CI user and normal-hearing listener cohorts. Musicians had a significant (p <0.05), negative effect on participant ratings for the “Isolation Waltz” with the standard map.

Results can be seen in Fig. 7. The largest CI-MUSHRA scores of the CI users, meaning the greatest difference from the normal-hearing average, occurs in the timing incongruent condition. However, the CI-MUSHRA scores are not consistent among all CI users, as only a portion (around nine) are outside the range of the normal-hearing listener scores.

As large variability was found within the CI cohort for the CI-MUSHRA scores in the timing incongruent condition, CI users were subsequently analyzed with a k-medoid clustering analysis on the CI-MUSHRA score for the timing incongruent condition. The k-medoid clustering analysis provided an objective method to determine the two subsets of the CI cohort based on the effect of timing congruence to their preference ratings. The results of the clustering analysis were used to calculate a silhouette coefficient, which indicates the distinctiveness of the two subsets of CI cohort. All participants' silhouette coefficient was over 0.85 with one exception (coefficient of 0.77 in the “different from NH” group), where a silhouette coefficient over 0.6 is considered an indicator of a good k-cluster. Additionally, t-tests were performed between the two clusters on several participant traits including age, time since implantation, the active engagement Gold-MSI sub-score, the musical training Gold-MSI sub-score, and tactile frequency discrimination score to test confounding factors, such as age, musical training, and tactile frequency discrimination ability, or correlates, such as timing since implantation and active engagement. A summary of the results of the t-tests are shown on Table IV; however, no significant differences between the two clusters was found (p >0.05).

It was originally hypothesized that since CI users' timing and intensity discrimination are comparable to normal-hearing listeners (Brockmeier , 2011; Cooper , 2008; Kong , 2004), the effect of preference ratings for timing alignment and intensity congruence in auditory-tactile music stimuli would be similarly comparable. As seen in the results, it was true that both intensity congruence and timing alignment were shown to positively affect preference ratings for CI users, presumably due to a similar perceptual mechanism. However, not to the same degree. Timing alignment was the only statistically significant (p <0.05) interaction the CI user cohort had with audio-tactile congruence. According to the model, participants in the CI cohort were affected less by audio-tactile timing alignment than participants in the normal-hearing cohort. Therefore, the hypothesis that CI users would have similar results to normal-hearing listeners is only partially supported.

A further analysis with the CI-MUSHRA score, however, paints a different picture of the results: an individual CI participant's preference rating was not less affected by timing alignment than normal-hearing participants, but instead timing alignment affected some CI users just as much as normal-hearing participants and some CI users much less. Therefore, the perceptual mechanism that explains the difference between the two groups could also explain the within group differences in the CI cohort.

Certain CI users could have been less affected by timing alignment than other CI users due to confounding factors, such as age. Age is a clear confounding factor between the two cohorts—the normal-hearing listeners were between the ages of 19 and 27 years, while the CI users were between the ages of 41 and 82 years. Tactile sensitivity is known to decrease with age, which could make tactile stimulation less salient than in younger listeners (Merchel and Altinsoy, 2020). Information was gathered on frequency discrimination scores for each participant, and while there was no significant difference (p >0.05) in frequency discrimination between the two intra-cohort CI groups, frequency discrimination abilities might not capture a complete picture of overall tactile sensitivity. For example, absolute thresholds of tactile detection might have affected the salience of rhythm differences in the tactile stimuli.

There could also be differences in terms of the type of music enjoyed and format it is enjoyed in, which might be related to age or CI use. First, frequent purveyors of loud concerts might have a stronger association between tactile stimulation and music going into the experiment. That association could also be related to age, which would affect the genres and types of music participants enjoy. With that said, questions on active engagement were included in the Gold-MSI questionnaire, and no significant difference was found between the two intra-cohort CI groups (p >0.05). Second, the discrepancy related to the importance of the time alignment could be due to the CI usage, but not necessarily its interaction with the tactile stimuli. There is a high variability in the listening habits and music enjoyment of CI users, which is not necessarily correlated with music perception accuracy. Certain studies have shown that CI users who report better enjoyment of music are younger, have better speech perception outcomes, listen to music more before and after implantation, and have a shorter duration of deafness (Gfeller , 2000; Gfeller , 2008; Migirov , 2009; Mirza , 2003). However, these correlations are not found consistently across all studies, and in most cases, the root cause is not known. Certain correlations have particularly unclear cause-and-effect directions. For instance, it was reported in Gfeller (2008) that CI users who listen to music more also report a higher enjoyment from music listening.

Different CI users may have different experiences with haptics as well. A panel of high-performing CI users cited the utilization of all senses in music listening as the most common music listening strategy Gfeller (2019b), and that playing a musical instrument was important for music listening rehabilitation due to its multisensory nature (Gfeller , 2022). Engaging tactile perception to improve music listening might simply be a result of a conscious effort to improve music listening experiences, and therefore not necessarily predictable by demographic data. Different CI users may have a different propensity to tactile information based on their experience using tactile information itself.

One more possibility is that the discrepancy in time alignment importance in CI users is related to a CI user's ability to integrate the audio and the tactile stimulation. For one, there is the possibility that using the wireless streaming device created a delay between the auditory and tactile stimulation, and that this delay may cause a failure of multisensory integration. However, it has been shown that the time window over which audio-tactile stimuli are perceived as synchronous is between −50 and 50 ms (Occelli , 2011), and this window is longer than the type of processing delays found in the wireless streaming devices (9–12 ms for a CI compared to acoustic hearing, and 20 ms for the streamer) (Zirn , 2019).

There is evidence the discrepancy in time alignment importance could be due to CI use disrupting audio-tactile integration. Multisensory integration between auditory and tactile modalities has been studied in CI users. It has been shown that auditory-tactile integration is possible (Landry , 2013; Landry , 2014; Nava , 2014) but can be disrupted (Landry , 2013) in CI users, and that it possibly changes over time (Landry , 2014). The changeover time can be aligned with other studies on CI users with SSD, which show that pitch perception in the implanted ear adjusts to match the normal hearing ear over time (Reiss , 2007). It could be that the differences in the importance of timing alignment in the CI cohort are just snapshots of different CI participants at different stages of this adjustment period. While that might imply that the time since implantation would be a good predictor of these differences, there is a high degree of variation among CI outcomes and so it is difficult to quantify which stage a given participant is at with the available data.

The fundamental mechanism behind this difference, however, still requires explanation. One reason could be related to the principle of inverse effectiveness: multisensory integration is stronger when one of the two modalities is weak in isolation (Holmes and Spence, 2005; Meredith and Stein, 1983). If a CI user's auditory perception of a melody is weak, they may be more likely to perceptually emphasize the tactile stimulation. One could argue, therefore, that a CI user should be more affected by auditory-tactile incongruence than a normal-hearing listener. However, part of the design of the experiment depends on the assumption that the congruence or incongruence of tactile stimulation with sound will correspond to either enhancing or disrupting enjoyment of the auditory stimulation. One could also consider the opposite direction: if the salience of the tactile stimulation is strong, it could be the auditory stimulation either enhancing or disrupting the tactile stimulation. The tactile stimulation was consistent with the auditory stimulation in tempo and distribution of notes. The CI users could have perceived two, equally salient and complementary rhythms, rather than one less salient rhythm enhancing or disrupting another.

Audio-tactile timing misalignment could have affected CI users less than normal-hearing listeners due to the perceived frequency differences between the two stimuli. For audio-tactile integration to occur, the audio and tactile stimuli have certain requirements. Specifically, certain types of auditory-tactile integration require matched frequency content between the acoustic and tactile stimulus. It is known that CI users can often have frequency-to-place mismatches in their device (Canfarotta , 2020; Landsberger , 2015), and that pitch perception can changeover time (Reiss , 2007). It is possible that auditory-tactile integration has certain perceptual requirements of the stimuli that are only met in an experienced CI user, who perceives sound frequency more similar to a normal-hearing listener.

A few participants in the CI user cohort provide some insight due to their specific hearing loss. Within the CI user cohort, one participant was pre-lingually deafened and two participants had SSD. While the participant with pre-lingual deafness was categorized in the “different from normal-hearing” group, the SSD participants were one in each category. One could justify these categorizations a posteriori with a myriad of explanations. For example, perhaps a lack of experience listening to music without hearing loss (such as the pre-lingually deafened individual) would reduce the meaning or musical interpretation that could be gleaned from the tactile stimulation. Similarly, the participants with SSD could listen to music regularly with normal hearing, and so might not be used to listening to music with the CI processing, hampering their overall listening experience. Alternatively, their experience listening to music could enhance their ability to interpret the tactile stimulation through a musical lens (such as interpreting a change of frequency as a “pitch” change). Regardless, these explanations are difficult to legitimize with so few participants. Future studies could focus on these alternate CI participant groups as a way to isolate the effect of variables like “music listening experience” that might be reduced in CI participants vs the innate bottleneck of auditory information through the CI.

The effect of different songs is an interesting finding and partially amends the findings of Aker (2022), which only tested with one melody. In the present study, four additional music stimuli were tested and were selected to address potential confounds of the melody used previously. However, it is important to note that even with the present study's extended stimuli selection, there is an enormous library of music to be potentially sampled and it is difficult to represent all music comprehensively. Not only is there large variability in the genre, instrumentation, rhythm, and tempo of music, but each individual will also have a unique reaction to a given song based on their own experiences. Additionally, there are still confounds in the present study's stimuli selection due to technical and experimental limitations. For example, all auditory stimuli have an identical timbre (sawtooth wave), are only single track, and occur in the frequency range of 73.42 Hz to 146.83 Hz. All conclusions must therefore be viewed through this lens.

The “Bass Track” song was the same stimuli used in Aker (2022) and had similar results, with intensity and timing congruence having a significant effect on participant preference ratings. It was thought that because the “Bass Track” song was adapted from the bass guitar of a pop song, it might emphasize intensity and timing changes over the pitch changes of a melody or harmony. Therefore, the second song “Beethoven,” was chosen as an excerpt of a piece that was not particularly rhythmic or “groovy.” While the effect size of intensity congruence was smaller for the “Beethoven” song when compared to the “Bass Track” song, the results followed a similar pattern with intensity and timing having a significant effect on ratings while fundamental frequency congruence did not.

The same reasoning could explain why the fourth song, “Isolation Waltz,” did not show a significant effect of intensity congruence in either map for non-musicians. “Isolation Waltz” was chosen due to the long duration of the notes in its melody, with the speculation that a long duration of note might make fundamental frequency changes more salient. However, this was not the case, implying a lack of temporal acuity in the tactile frequency domain is likely not a large issue. By virtue of being a slower waltz, however, it is also arguably less rhythmic or “groovy” than the bass guitar of a pop song and the melody of a classical piece. Therefore, it is possible that as a song becomes less rhythmic, intensity changes become less emphasized, and intensity congruence has a smaller effect size on participant ratings. However, as iterated above, three stimuli in the present study are not enough to conclusively decide this, and therefore, it should be tested more thoroughly in a future study.

Notably, fundamental frequency congruence between the auditory and tactile music stimuli was shown to have a significant (p <0.05) effect on participant ratings for “Amen Break” and “Major Scale” in the stretched and standard map, respectively. Both songs were selected for their pitch contents. In previous work (Aker , 2022), the “Bass Track” song did not show an effect of fundamental frequency congruence on participant preference ratings. One theory was it was due to the frequency of the notes lacking large changes in magnitude or a clear directionality to make the tactile frequency changes as salient as possible. Therefore, the “Amen Break” was chosen as another stimuli adapted from a rhythm instrument, drums, in which the bass drum was always mapped to the lowest tactile frequency and the snare drum was mapped to the highest tactile frequency. The large separation between the two possible frequencies could have influenced participant perception of audio-tactile fundamental frequency congruence and made incongruence more obvious. It is also possible that the fundamental frequency of the notes in the “Amen Break” had a more distinct role in the melody. While the auditory stimulus used square waves and did not have the timbre of a drum, it is possible the stimulus was still distinctively more rhythmic to the participants, and the two pitches became perceptually intertwined with the note intensity and timbre. A possibility for future audio-tactile maps could be mapping the auditory timbre to the tactile frequency, or simply utilizing the tactile frequency to map a tactile timbre to the auditory timbre.

Likewise, the “Major Scale” was chosen for the clear directionality of its pitch contour. It is possible a salient pitch contour both increased the perceptual weighting of fundamental frequency in the melody as well as made the fundamental frequency changes in the tactile stimuli more noticeable. While both the “Major Scale” and “Amen Break” songs only showed an effect of fundamental frequency congruence in one of the two maps, the results appear to support the hypothesis that one of the main bottlenecks to an effect of frequency congruence is the poor frequency discrimination of the tactile system relative to the auditory system (Aker , 2022; Merchel and Altinsoy, 2020).

In a broad view, the present study provides evidence that song choice interacts with participant preference rating and audio-tactile congruence. Since the specific musical parameters have been shown to be important, by extension, it also emphasizes that the similarities (due to technical or experimental limitations) of the selected stimuli could be significant confound variables, such as all auditory stimuli having identical timbre (sawtooth wave), only having a single track, and occurring in the same frequency range of 73.42 to 146.83 Hz. The interpretation of the results could be reframed entirely through this lens: that it is the music stimuli and an individual's personal response to the stimuli that determines how audio-tactile congruence affects participant preference ratings. In this reframing, an individual's hearing loss or CI use will only alter their experience with the music stimuli itself. However, this reframing perhaps better serves as a building block for another explanation: that it is a complex, multi-factored interaction that the present study provides a small window into that includes hearing loss, hearing device usage, musical preferences and experience, parameters of the music stimuli itself, and neurological, auditory processing capabilities of the listener.

The study shows that more research is needed into vibrotactile music enjoyment enhancement for CI users to see how music-haptic devices can be optimized for music listening with a CI. On one hand, a subgroup of the CI participants had results similar to the normal-hearing cohort, implying they could benefit from vibrotactile music enjoyment enhancement in a similar way. For these participants, vibrotactile stimulation could provide a simple, hearing-agnostic way to enhance music that does not depend on specific aspects of hearing perception that might be unavailable to a CI user, such as localization.

However, CI participants who had a less strong preference difference between timing aligned and timing unaligned auditory-tactile stimulation might not be benefiting from the vibrotactile stimulation in the same way, if at all. While vibrotactile stimulation remains a promising direction for CI users to enhance music, investigation into the perceptual mechanisms behind these benefits are critical to determine both who can benefit and how music-haptic devices can best be optimized. For example, it is possible that time alignment was not important if the tactile stimulation and auditory stimulation were weighted equally perceptually. In that case, some CI users might be better served by removing a single instrument's track from a piece of music and presenting it exclusively through a haptic device, making the vibration asynchronous, but in-rhythm with the auditory components. Alternatively, the reason could be due to an aspect of a specific participant, and that only certain participants are suited for vibrotactile music enjoyment enhancement, whether related to the CI or not. Anecdotally, the author has seen many normal-hearing colleagues and visitors who describe the haptic accompaniment of music presented through commercial devices as “annoying.” Determining good predictors for these cases would make it easier to determine who can benefit from vibrotactile music enjoyment enhancement, and if it is a large enough population that vibrotactile music enjoyment enhancement is a feasible, large-scale solution. Finally, it is possible that for a subset of CI users, auditory-tactile music simply does not integrate or benefit the listener at all. If only specific, “super-user” CI users benefit from vibrotactile music enjoyment enhancement, then the long-term aspirations for using tactile vibrations to enhance music for CI users, such as wide-spread usage of haptic CI accessories, might be reduced.

One recommendation for music-haptic devices for CI users that is robust to all interpretations of the present study is to ensure the devices are customizable. That is, certain elements of the processing or audio-to-tactile transformation can be adjusted, whether that adjustment is done by the user, manufacturer, or a clinician. There is already a large variability in outcomes for CI users—adding that variability with the innate variability and complexity of music, and any given individual's reactions and experiences with music, means that any solution attempting to account for an individual's music listening experience will not be a one-size-fits-all device. The most basic level of customization would be the ability to adjust the intensity level of the tactile stimulation according to an individual's sensitivity or proclivity to haptic sensations. Ensuring the device could be worn on a variety of locations on the body could also be beneficial to the user. A more complex customization system might have users mapping the auditory parameters to tactile parameters themselves, or in the case of CI users, associating specific electrodes with specific haptic actuators on the body. Users could then adjust the maps on a song-by-song or situation-by-situation basis, mapping tactile parameters like intensity (that are relatively precise in their discrimination ability compared to frequency) to whichever auditory parameter is more important for a given musical experience.

This work was supported by the William Demant Foundation and the Innovation Fund Denmark. Special thanks to Kristen Engel and Viktorija Ratkute for helping to pick the music stimuli, Rikke Skovhøj Sørensen and Maria Grube Sorgenfrei for their help with the participants, as well as the participants for their involvement.

The authors have no conflicts to disclose.

Participants provided informed consent and all experiments were approved by the Science-Ethics Committee for the Capital Region of Denmark (reference H-16036391) and were compensated financially for their time unless they voluntarily declined. All relevant regulations were followed.

The data that support the findings of this study are available from the corresponding author upon reasonable request.