Study of optical-based speech acquisition system using vibration signals from speakers' medical masks Open Access

The classical method for obtaining speech information from users involves air-conducted microphones, which can obtain high-quality speech information from speakers for quiet environments and short distances. However, there is still room for improvement in their performance under challenging conditions, such as for distant speech and under ambient noise.^1–4 Specifically, when sound waves propagate in air, the energy of the speech signals decreases as the distance between the sound source and the target increases.¹ These signals can also be easily contaminated by environmental noise in the vicinity of the sensor.^2,3 Medical masks have become necessary of late because of the COVID-19 outbreak. However, the presence of medical masks may have a deleterious effect on the transmission of sound waves. Their use attenuates the energy and distorts the high-frequency components of speech signals, thereby affecting speech quality.^5,6 In addition, speech intelligibility is negatively affected when speakers wear medical masks because of the reduction in visual information available to listeners.^5,7

Currently, several approaches have been proposed to alleviate the effect of distant speech and ambient noise, such as microphone array-based (directional microphone) speech enhancement approaches that use the minimum variance distortionless response (MVDR) beamforming algorithm,⁸ but the issues of speech signal distortion and energy decay caused by the medical mask continue to be problematic with the microphone array-based technology. However, it can be observed that when people speak, the energy from the sound waves can cause medical masks to vibrate. Therefore, we assumed that the vibration of the mask is rich in speech information, which can be used to acquire the user's speech. Moreover, because the medical mask is the vibrational object surface closest to the sound source, namely, medical personnel or patients, the vibration caused by speech can be directly transmitted to the medical mask. Therefore, the signal-to-noise ratio (SNR) remains high in noisy conditions, even if the sound waves of background noise contact the medical mask. To verify our assumption that the vibration signal of medical masks provides useful speech information, an optical-based sensor, a laser Doppler vibrometer (LDV),^9–13 is used to acquire the medical mask vibration signals in this study.

The LDV sensor provides an alternative method for distant speech acquisition based on the Doppler effect. More specifically, LDV sensors can obtain distant target speech by irradiating the object surface near the target sound source, such as a glass window, metal plate, or plastic box,^9–11 with a laser beam at the micrometer or even nanometer scale. Based on this LDV concept, previous studies (e.g., Sun et al.¹² and Xie et al.¹³) used a combination of LDV and normal acoustic speech features to train a deep neural network (DNN) acoustic model, and thus achieved performance improvements for automatic speech recognition (ASR) under both quiet and noisy conditions. Based on earlier studies on LDV sensors in speech processing tasks, the primary purpose of this study is to further investigate the feasibility of obtaining speech signals from medical masks using LDV sensors. As the surface of the medical mask is a non-smooth reflective surface composed of fibers, it may affect the possibility of optical reflection. Furthermore, we intend to investigate whether LDV sensors can provide advantages for ASR systems under challenging conditions (e.g., distant speech and ambient noise) in comparison to a classical microphone array device.

Figure 1 illustrates the system proposed in this study, where the LDV sensor is used as an optical-based microphone to record speech from a target sound source. The LDV sensor is a non-contact measurement device based on the principle of interferometry. Measurements are made at the point where the laser beam strikes the object surface that is vibrated by the target sound source. The LDV equipment we used (LDV Vector-Series, Optomet, Darmstadt, Germany) is classified as class 2, which is considered safe for normal operation. As the class 2 laser beam is typically less than 1 mW, it will not damage human skin or mask materials. In our case, the laser beam is directed at the KEMAR mask (45BB-4 KEMAR head, GRAS, Holte, Denmark), which is used in this study to replace the real speaker to provide fair testing conditions. When KEMAR's artificial mouth is playing audio, the vibration is directly transmitted to the medical mask and was easily captured by the LDV sensor.

The purpose of this study is to investigate the benefits of an optical-based microphone under distant speech and ambient noise conditions when a speaker is wearing a medical mask. Initially, we asked a speaker to record his voice according to the Taiwan Mandarin hearing in noise test (TMHINT) corpus¹⁴ in advance and normalize each recorded sentence. Subsequently, we used KEMAR (GRAS 45BB-4 KEMAR head) with a medical mask to simulate the application scenario, and the speech was played from KEMAR's artificial mouth. Note that the masks we used are medical masks generally made of polypropylene, which, in recent years, have been recommended for the public to wear to help prevent the spread of COVID-19. Therefore, we can avoid multiple influencing factors, such as the involuntary movement of heads, articulators, and airstreams in real speakers to provide fair testing conditions for different approaches. Moreover, since the energy of each sentence has been adjusted to be the same, we can also ensure the consistency of the acoustic characteristics of each test. As for the microphone systems, we used three types of microphone systems—an omnidirectional microphone (CDCU-04IP VX-1, IPEVO, Sunnydale, CA), a directional microphone (IPEVO CDCU-04IP VX-1), and the proposed system (Optomet LDV Vector-Series) to record these sentences with a 65-dB sound pressure level (SPL). In the experimental stage, we used the aforementioned three approaches to record the speech at three distances (1.5, 2, and 2.5 m) under quiet and noisy conditions. To simulate noise from conversation, the background noise of Institute of Electrical and Electronics Engineers (IEEE) speech-shaped noise (SSN) was used at 65 dB SPL. The entire experiment was conducted inside an audiometric room to ensure that the environmental variables were consistent. After the recording, we conducted two types of objective evaluations to analyze the performance of different microphone approaches, namely spectrogram analysis and the objective evaluation metric of Google ASR, to verify the benefits of the optical-based microphone. In the spectrogram analysis, we can observe which microphone performs better by comparing the similarities between the corpus recorded by different microphones and the original corpus. In the Google ASR test, the accuracy is used to determine the speech quality performance of each microphone approach at different distances.

Figure 2 depicts the spectrograms of the original speech signal and the recorded corpus via three types of microphone configurations at 2 m distances in noisy conditions. We observed that there was a broad noise band in the omnidirectional and directional microphones, whereas the speech derived from the optical-based microphones was clear and similar to the original speech. Moreover, in the high-frequency region around 2 kHz indicated by the red arrows, the high-frequency components of speech recorded by LDV are clearer and more intact compared with those recorded by omnidirectional and directional microphones. In other words, these spectrograms indicated that the optical-based microphones provided clearer low- to mid-frequency information (e.g., below 2 kHz) of speech, which held the richest speech information. However, notably, the signal-derived information of the low- to mid-frequency range was blurry from the air-conducted microphones compared to the optical-based microphones. Based on these results, we concluded that the speech acquired from an optical-based microphone exhibited higher quality compared to air-conducted microphones. It was noted that the harmonics running from the bottom to the top at 3 s in Fig. 2(d) represent the phenomenon called speckle noise. The speckle noise comes from randomly distributed light interference caused by the irregularly arranged polypropylene fibers in the mask.¹⁵ In addition, it is a crackling sound in the recorded audio and would produce a harmonic band across a wide range of frequencies in the spectrogram, which cannot be seen in the target speech in Fig. 1(a). Moreover, certain high-frequency signals are lost in the optical-based corpus, whereas the low-frequency signals remain intact. To solve this issue, the frequency bandwidth extension^16,17 technology, voice conversion^18,19 and speech enhancement²⁰ technologies can synthesize frequency components in the high band (i.e., 4–8 kHz) given the low-band frequencies (i.e., 0–4 kHz) that can be applied in future investigations.

Figure 3 shows the results of the Google ASR accuracy using the aforementioned microphone approaches under quiet conditions. Note that the speech level is at 65 dB SPL in this study. The x axis represents the distance between the microphones and sound source, and the y axis represents the judgment accuracy of Google ASR. Evidently, a performance gap exists between the optical-based microphone and other baseline microphone systems. More specifically, the accuracy of the baseline microphone systems decreased by approximately 16.5% when the distance increased from 1.5 to 2.5 m. In contrast, the accuracy of the optical-based microphone system decreased by only 7% with the same increase in distance. We assumed that this gap may be due to the following two reasons. First, the medical masks attenuated the high-frequency components and absorbed energy, thus blurring the speech and lowering the volume. Second, the energy from the KEMAR was attenuated when the distance between the microphones and the sound source increased. However, this did not occur in the optical-based microphone because the LDV sensor depended on optical-based vibration to acquire sound signals. Therefore, the accuracy of these two baseline systems (omnidirectional and directional microphones) dropped below that of the optical-based microphone approach.

Figure 4 illustrates the results of the Google ASR accuracy for the three microphone types under noisy conditions. Compared to quiet conditions, notably, as shown in Fig. 3, the baseline system accuracies were significantly affected by the noise, particularly at 2.5 m. However, the LDV sensor continued to provide stable accuracy, compared with the 1.5 and 2.0 m testing conditions. Because the medical mask associated with the LDV sensor was the closest object surface near the sound source, the vibration was directly transmitted to the medical mask. As a result, the SNR was very high even under 65 dB SPL background noise. In conclusion, the results indicated that the optical-based approach surpassed the baseline approaches under adverse conditions, such as distant speech and background noise.

Distant speech and ambient noise are bottlenecks for the performance improvement of conventional air-conducted microphones. A commonly used medical mask causes voice distortion and affects speech quality, thereby reducing the accuracy of speech recognition systems. To alleviate these issues, we proposed an optical-based microphone approach to acquire speech signals from a speaker with a medical mask for robust speech recognition performance. The classical approaches involving omnidirectional and directional microphones were used as baseline systems for comparison. The experimental results indicated that the proposed optical-based microphone system achieved better performance than the classical microphone array-based system. These results imply that the optical-based microphone can be a promising approach to enhance the performance of a voice-control-based system under challenging conditions when the speaker has a medical mask. In addition, this study proves that the vibration signals on the medical masks of speakers contain useful information for speech recognition applications.

This study was supported by the Ministry of Science and Technology of Taiwan under the 110–2218-E-A49A-501 project. The authors would like to thank IEA Electro-Acoustic Technology Co., Ltd. for providing the experimental devices.