Electromagnetic articulography appears feasible for assessment of speech motor skills in cochlear-implant users

Auditory input is essential to the acquisition and maintenance of speech production skills. Infants born with severe-to-profound sensorineural hearing loss (henceforth HL) who do not receive appropriate prostheses rarely develop intelligible speech. When the onset of HL occurs in adulthood, the quality of speech becomes degraded. But since cochlear implants (CIs) have become the standard of care for patients with severe-to-profound sensorineural HL—regardless of whether that HL is congenital or acquired—speech production skills in this population appear to have been greatly enhanced (Boothroyd et al., 1991; Spencer et al., 2004). However, detailed descriptions are currently lacking when it comes to the speech of children with severe-to-profound HL who get CIs. There are myriad reasons to investigate speech production in children and adults with CIs, but to do so in a comprehensive manner requires a methodology for directly recording speech motor actions. One method that fits the bill is electromagnetic articulography (EMA) (Perkell et al., 1992; Rebernik et al., 2021), but to date no one has shown that current EMA technology can be validly and reliably used with CI wearers. Given that both CIs and EMA make use of magnetic fields, concerns of possible cross-interference exist [see, e.g., Katz et al. (2003) and Rebernik et al. (2021)]. Those concerns must be addressed if we are to refine our investigations, and thus further our understanding of the development and maintenance of speech production skills in deaf individuals.

With that motivation in mind, we undertook an exploration of the feasibility of using EMA with CI wearers that had two distinct goals: first, to test whether the presence of a CI affects the stability of EMA sensor position tracking and, second, to test whether EMA interferes with the signal processing of CIs. In one prior study, Katz and colleagues (Katz, et al., 2003) reported that the magnetic fields generated by an EMA system did not induce voltage artifacts in the output of a CI during the processing of non-speech sounds (white noise and sine waves), and that speech perception was not adversely affected by EMA fields in three postlingually deaf adults with CIs. However, the participants in that study were tested with a now-defunct CI (Clarion 1.2 S-series) and an earlier EMA model (Carstens AG100) was used. Furthermore, no static or dynamic sensor positions were recorded and tracked with versus without a CI present in the EMA field. Another study recorded articulographic data from one adult with a CI, but that CI was an older model without magnets (Matthies et al., 1996). Most recent speech production studies involving adults or children with CIs have used other methods, such as acoustic analyses or ultrasound (Ghayedlou et al., 2020; Grandon and Vilain, 2020; Turgeon et al., 2017). Although these studies have yielded valuable insights, such analyses still do not provide direct examination of inter-articulator speech production.

Developing a novel methodology to record the timing, sequencing, and coordination of articulatory gestures for talkers with CIs is an essential goal if we want to understand speech motor control in deaf individuals, because of the “many-to-one” problem, which refers to the fact that different vocal tract constrictions can produce the same acoustic structure [see, e.g., Nieto-Castañón et al. (2005) and Guenther (2016)], so acoustic structure may not necessarily reveal articulatory actions. Therefore, direct observation is preferable and requires a method such as EMA, which allows researchers to record and track the precise choreography of the jaw, tongue, and lips on a millisecond-by-millisecond basis and localize where in the vocal tract the movements occur (Perkell et al., 1992; Rebernik et al., 2021). Simultaneous acoustic recordings also allow researchers to examine how those movements shape the acoustic speech signal.

Speech movement tracking is performed with EMA using weak magnetic fields, generated by a series of transmitter coils, to localize the positions of small sensor coils temporarily fixated on the surfaces of the articulators (i.e., the lips, tongue, and jaw) during a speaking task. The transmitter coils, located at the vertices of an equilateral triangle above the head of the speaker, generate and radiate out a series of magnetic fields, which induce a current in the sensor coils. The intensity of the current depends on the distance and orientation of the sensor coils from the transmitter coils. This voltage-distance relation is then used to compute the locations of the sensors, and therefore the articulatory surfaces to which they are attached, in near real time [see Hoole (1996), for additional technical details concerning the acquisition and processing of articulographic data]. However, given that CIs use external and internal magnets, their simultaneous use with EMA may induce signal artifacts that affect the precision and accuracy of EMA sensor position measurements. Conversely, the magnetic fields generated by EMA may interfere with the transmission of CI signals across the external and internal magnets, which in turn, may distort auditory processing. If signal artifacts in either direction were found, then researchers would need to develop appropriate methods for eliminating that interference (e.g., filtering algorithms) prior to conducting articulographic experiments with this population.

Here, we report the results of two experiments that provide evidence that there is minimal uncorrected interference between the magnetic fields generated by state-of-the-art EMA and CIs, in either direction. Ensuring that EMA magnetic fields do not affect transmission of CI signals and, vice versa, that CI magnets do not interfere with EMA recordings, is a necessary prerequisite for conducting articulographic testing with deaf CI wearers. In experiment 1, we calibrated a set of EMA sensors with and without a CI present in the EMA field, and measured impedances of all CI electrodes when in and out of the EMA field. Then, in experiment 2, we recorded and analyzed the position stability of three static, head reference sensors during a speaking task with a normal-hearing talker with and without a CI present in the EMA field.

The AG501 EMA system (Carstens Medizinelektronik GmbH, Bovenden, Germany) and the research CI system [Cochlear Ltd, Sydney, Australia; shown in Fig. 1(A)], including the Freedom sound processor (CP910), Freedom programming pod interface, Freedom implant emulator (CI24RE), and implant load board, were used in both the current experiment and in the subsequent experiment. In the current experiment, we calibrated a set of EMA sensors in three experimental conditions: (1) with the CI absent from the EMA field (CI ABS); (2) with the CI (and electrodes) in the EMA field, but with the emulator off (CI OFF); and (3) with the CI (and electrodes) in the EMA field and the emulator turned on (CI ON). Across all three conditions, eight sensors were firmly mounted on the AG501 circal disk (a calibration device) using two magazines [shown in Figs. 1(B) and 1(C)], and then calibrated using Carstens' cs5cal calibration program. Each magazine contained four slots (or notches; spaced by 4 cm) to receive sensors positioned at approximately 7 cm from the rotation center of the disk. As shown in Fig. 1(C), each sensor is mounted in a magazine slot at a 45° angle between the sensor's axis and an imaginary line to the circal disk's center. The disk itself is positioned at the geometric center of the transmitter coils generating the EMA magnetic fields. During the calibration step, the disk gradually rotates the sensors across an area typical of human articulatory space and measures deviations in the Euclidean distance and angles between all pairs of sensors. In the conditions where the CI was present, the research CI system was positioned on a wooden table immediately underneath the circal disk.

We also measured the impedances of CI electrodes 5–22¹ when not in the EMA field (EMA ABS), and in the EMA field, both with EMA off (EMA OFF) and EMA on (EMA ON), using Cochlear Custom Sound software (version 5.2) with a fixed common ground (CG) stimulation mode. For the CI setup configuration, the CI sound processor (CP910) was connected to the programming pod interface, providing the input to the CI emulator (CI24RE). The implant load board was coupled to the electrode terminals of the implant emulator and served as a cochlea simulator to measure the impedances. All CI components except for the programming pod interface were positioned within the EMA field.

Table 1 displays the calibration results provided by the Carstens' cs5cal program as a function of condition (CI ABS vs CI OFF vs CI ON). In Table 1(A), the mean (M) and standard deviation (SD) of the z-coordinates for each sensor channel (1–8) during the revolutions of the circal disk in each condition are given. According to the calibration quality guidelines published by Carstens, the SD should be less than 0.25 mm. This was found to be the case for all sensor channels and all experimental conditions.

The cs5cal program also provided four measures concerning sensor position deviation, given in Table 1(B): (1) The maximum angle deviation (between the sensor's axis and the line to the circal disk's center) that occurred between any two sensor axes (Angle Max); (2) the mean of all measured changes of angle differences (Angle Mean); (3) the maximum deviation that occurred in the Euclidean distance between any two sensors (Position Max); and (4) the mean of all measured changes of the Euclidean differences (Position Mean). As the table shows, there were no substantial differences in any of these measurements obtained across conditions, indicating that the presence of a CI in the measurement volume does not influence sensor position tracking and the sensor calibration procedures of the AG501 EMA system.

Figure 1(D) shows the CI electrode impedances (kOhm) as a function of electrode number (5–22) and condition (EMA ABS vs EMA OFF vs EMA ON). The results indicate that the impedances were similar across most electrodes for each of the three experimental conditions, except electrodes 10, 20, and 21 [highlighted in yellow in Fig. 1(C)], which all showed slightly higher impedance values in the EMA OFF condition than in the EMA ABS and EMA ON conditions.

In the second experiment, EMA sensor positions were recorded for sensors fitted to a normal-hearing female talker in two experimental conditions: (1) with the CI absent (CI ABS) and (2) with the CI (and electrodes) in the EMA field and turned on (CI ON). We used the sensor calibration file that was generated in the CI ABS condition of experiment 1. In the CI ON condition, the model speaker wore the CI microphone and speech processor behind her ear and held the other components of the research system in the approximate position it would be in a CI-implanted individual (shown in Fig. 2). Articulatory sensors were placed on the tongue tip (TT) and the jaw (J) with static head reference sensors placed on the gingiva above the upper incisor (UI) and behind each ear on the left mastoid (LM) and right mastoid (RM) processes. Prior to the speaking task, the participant's mid-occlusal plane was obtained by attaching three sensors to the Carstens biteplane that the participant held between her upper and lower teeth while data are recorded for these three sensors, as well as the reference sensors. The talker was recorded while producing ten repetitions of a list containing eight non-words adapted from a previous articulatory investigation (Nittrouer, 1991); five repetitions were recorded in each condition (CI ABS vs CI ON). Each word ([tVCɑt] where (1) V is short [ɛ] or long [ɑ]; (2) C is [t] or [d]; (3) the first syllable is unstressed or stressed; (4) speaking rate is fast or normal) was spoken in the carrier phrase “It is a ____ again” in order to control pre- and post-vocal tract configurations. The segmental and intonational structure were manipulated in order to engender variation in inter-gestural timing and coordination in the collected data. The trials were blocked by speaking rate: normal first and then fast. The stimuli were controlled using psychopy (version 3.0). Simultaneous audio (sampled at 48 kHz) was recorded using a shotgun microphone (t.bone EM9600) and the EMA sensor signals (sampled at 250 Hz) were recorded using Carstens' cs5recorder and cs5view programs.

The raw acoustic and kinematic data were processed, visualized, and analyzed using the matlab-based MVIEW algorithms (Tiede, 2005, 2010). The raw kinematic data first underwent a series of standardized pre-processing steps to rotate and translate each position signal to a consistent maxillary frame of reference (based on the location of sensor placed on the gingiva of the UI), and to correct for head motion artifacts. The acoustic and kinematic movement signals were then computed, co-registered, and visualized together.

If the presence of the CI does not adversely affect EMA sensor tracking, then the distances between the static, reference sensors (UI, LM, RM) should remain relatively constant across the two speaking conditions. To test this hypothesis, we used a custom matlab script to compute the intersensor Euclidean distances for the three reference sensors at each sampled time point. We then computed the mean and standard deviation of these measurements (pooled across trials) as a function of condition to assess the EMA data. Since the distances between the moving articulatory sensors (TT, J) are not expected to remain constant across recordings, regardless of the presence of the CI, we do not report those data here, but they will be forthcoming in another articulographic study. For the purposes of the current study, we only report the distances between reference sensor pairs with the CI absent and with it present and on.

Table 2 gives the mean intersensor Euclidean distances and standard deviations (SD) for all pairings of the static, head-reference sensors (UI, LM, RM) as a function of condition (CI ABS vs CI ON). As the table shows, these measurements remained fairly constant with and without a CI present during the acquisition of speech movement data.

Since CIs became the standard of care, scarce data have been collected on the speech production patterns of individuals with severe-to-profound HL—especially those with congenital HL. It is our long-term goal to examine those patterns and based on the data we report here EMA appears to be a promising and feasible method of assessment. Across two experiments, we found little to no evidence of cross-interference between EMA and CIs, and therefore, there appears to be no need to develop methods to eliminate interference signals. Experiment 1 showed that, during sensor calibration, the z-coordinates, Euclidean distances, and rotation angles between EMA sensors did not differ regardless of whether there was a CI absent or present and on. Experiment 2 further demonstrated that, during a speaking task (albeit with a normal-hearing speaker), the fluctuation of Euclidean distances between the static, head reference sensors did not vary when there was a CI absent versus present and on in the EMA measurement volume.

Collectively, the present findings suggest that there is minimal cross-interference between current EMA and CIs, and that EMA data can be reliably obtained from CI wearers. The next step in this line of investigation is to determine whether the EMA environment disproportionately affects speech perception and/or patterns of articulatory behavior in deaf speakers (both adult and child) with CIs. Although the current results show that EMA magnetic fields did not alter the impedances of the electrodes in a CI, future studies still need to confirm that the perception of speech by CI wearers is not affected while in the EMA measurement field. Given that Katz et al. (2003) previously reported, using old EMA and CI technology, that measures of sentence reception in background noise did not differ for CI wearers when they were in versus out of the EMA field, it seems unlikely that such cross-interference will occur. Nevertheless, this needs to be empirically demonstrated with current devices.

Future studies also need to ensure that CI wearers are not more adversely affected by having sensors attached to their articulators. To address this issue, we plan to obtain EMA recordings of CI wearers while producing utterances with sensors attached and not attached to their articulators and EMA on and off. Even if kinematic and acoustic analyses were to reveal that there are minor effects of the sensors, the major concern would be whether those effects are different in magnitude for deaf speakers with CIs and those with normal-hearing. Collecting speech samples both prior to and after testing would also allow researchers to determine whether there are practice-related changes that might be greater in magnitude for speakers with CIs. If a greater practice effect were found for CI wearers, then that would suggest that they might benefit more from greater practice prior to EMA data collection. In spite of these limitations, however, the present study demonstrates that EMA is a viable tool for research on speech motor control in deaf individuals who received CIs. As such, EMA may prove to be a valuable asset for experimental studies designed to explicate the role of degraded auditory input on inter-articulator speech production.

This work was supported by a Research Opportunity Seed Fund award from the University of Florida to M. Masapollo and S. Nittrouer, and an Emerging Research Grant from the Hearing Health Foundation to M. Masapollo. This work benefited from helpful discussions with, or comments from, Mark Tiede at Haskins Laboratories (New Haven, CT), and engineers at both Carstens Medizinelektronik GmbH (Bovenden, Germany) and Cochlear Corporation (Cochlear Ltd, Sydney, Australia).