A programmable magnetoelastic sensor array for self-powered human–machine interface

As the ever-growing presence of 5G infrastructure and the proliferation of the Internet of Things (IoT) become more robust, intelligent devices, such as computers, machines, sensors, and many more, progressively provide humans with more convenience.^1–7 In addition, those technological examples are gradually becoming more adaptive and intuitive, revolutionizing the bridge of communication between humans and machines.^8,9 The global human–machine interface (HMI) market is expected to reach a value of 5.73 × 10⁹ dollars by 2023 at a compound annual growth rate of 9.37%. The growing desire for improved machines to monitor production and respond to fast-changing demands, as well as the necessity for even higher efficiency and lower downtime, has fueled the expansion of the HMI market. This rapid growth suggests that now is an opportunistic time for the development of more innovative and creative approaches to connect human and machine even further, diversely ranging from hardware sensors to software algorithms.

On the one hand, traditional HMIs require complex data collecting units and an enormous amount of power consumption, which call for external power sources that are bulky, rigid, environmentally unfriendly, and limited in lifetime. These disadvantages hinder the HMI equipment capabilities in transferring into practical and sustainable applications since it is nearly impossible to seamlessly incorporate wearable devices with conventional batteries while maintaining breathability and skin conformability, owing to the current designs in materials and volume. On this account, wearable HMI devices^10–12 with minimalistic features related to resistive effect,^13–16 capacitive effect,^17–19 and self-powered mechanisms based on triboelectric effect^20–24 and piezoelectric effect,^25–27 or hybridized systems,^28–30 have emerged to provide the current state-of-the-art technologies, especially those with energy-harvesting strategies for sustainable and environmentally friendly power generation by means of biomechanical motions. Despite the tremendous list of advantages, these working principles can still be vulnerable to humidity and deteriorate in liquid conditions,³¹ which limit their electrical outputs and applications in certain environments, such as exercise causing heavy perspiration or usages in extreme weather.

On the other hand, the magnetoelastic effect is usually observed in rigid bulky alloys in nature.^32–35 Very recently, the giant magnetoelastic effect was discovered in a soft material system with up to four times enhancement more than the traditional rigid counterpart.³⁶ In this work, the discovered giant magnetoelastic effect is employed to develop a programmable and waterproof sensor array for self-powered HMI. Each magnetoelastic sensing unit is revolutionarily conditioned with a characteristic output signal in order to correlate with programmable functionalities in controlling a machine. This unique feature comes from the programmed orientation of the magnetoelastic film during the initial magnetization process. The device demonstrates a strain of up to 150%, a wide pressure sensitivity ranging from 10 kPa to 80 kPa, an optimal signal-to-noise ratio (SNR) of 34 dB, and a rapid response time of 0.2 s at the frequency of 1 Hz. The programmable magnetoelastic sensor array can produce continuously responsive electric signals and productively command electronic devices in real time via touch sensing of finger tapping. Importantly, it is intrinsically waterproof since the magnetic field can penetrate water without much loss in intensity. To pursue a practical application, this device is integrated with a customized circuit system to act as the on and off buttons for a desk lamp and function as four command features: play, pause, next, and previous, to control a music speaker. At the front end, the programmable magnetoelastic sensor array is capable of becoming a key player in the HMI communities whose future may require a self-powered, skin-comfortable, flexible, stretchable, and waterproof innovation.

A 40 × 40 mm² programmable magnetoelastic sensor array, consisting of four sensors, is illustrated in Fig. 1(a) with a waterproof all-in-one body design. Each sensor mainly holds two functional components. One is the giant magnetomechanical coupling (MC) layer that comprises the solid neodymium–iron–boron (NdFeB) nanomagnets and microbubbles-introduced porous silicone rubber matrix, which is able to convert a gentle biomechanical pressure into magnetic flux variation. The scanning electron microscope (SEM) image of the MC layer is displayed in Fig. S1 in the supplementary material, showing the scattered nanomagnets and porous structure. The other functional component is the magnetic induction (MI) layer, which are the patterned liquid metal coils. A photograph of the liquid metal before patterning is shown in Fig. S2 in the supplementary material. The MI layer is responsible to pick up the magnetic field variation and generate electricity on the basis of electromagnetic induction.

The magnetoelastic sensor itself could convert biomechanical activities into electrical signals by using a two-step conversion process: the MC layer is responsible for the mechanical-to-magnetic conversion and the MI layer the magnetic-to-electrical conversion. As illustrated in Fig. 1(b), after magnetization and in the initial state, the nanomagnets are single magnetic dipoles and aligned in a wavy chain structure. When each magnetoelastic sensor receives an applied uniaxial pressure, as shown in Fig. 1(c), the micromagnet chain structure diverges and internally alters the dipole–dipole interaction of the chain (shown in Fig. S3 in the supplementary material).³⁶ The demagnetizing fields are proportional with the decrease in the surface magnetic flux density. Once the uniaxial stress is released, the recovery of the micromagnet wavy chain structure reverses the magnetic flux density back to its original state. The magnetoelastic effect in the magnetoelastic sensor is observed without the necessity of an external magnetic field. The micro-computed tomography (Micro-CT) images in Fig. 1(d) and Movie S1 in the supplementary material reveal that these nanomagnets are evenly distributed and scattered throughout the porous matrix. Figure 1(e) shows the shift in the magnetic flux density mappings of one magnetoelastic sensor. As illustrated in Fig. 1(f), under an applied pressure of 300 kPa, the magnetic flux density declines to about 50%. Owing to the materials' flexibility and durability, the magnetoelastic sensor array can also generate stable power under deformations, rolling, folding, and stretching as in Fig. 1(g). Due to these compelling features, the device can be adopted for human-body powered HMI by transforming human biomechanical activities into electrical signals.

To optimize the biomechanical-to-electrical energy conversion of each individual magnetoelastic sensor, we comprehensively investigate the assembly and properties of the soft magnetoelastic composite. First, by controlling the thicknesses and the magnetic particle concentration, the soft magnetoelastic composite shows different mechanical properties. The thickness to produce an optimal electrical output is plotted according to Fig. 2(a). Accordingly, thicker magnetoelastic composite provides higher magnetic flux. The thickness of 1.5 mm is chosen because it provides high magnetic field variation while still seemingly appears thin enough to exhibit flexibility, stretchability, and deformation. Second, as illustrated in Fig. 2(b), the 83 wt. % soft magnetoelastic film is stretchable up to 150% strain. Since its decrease in magnetic flux density can compete to that of the traditional magnetoelastic system,³⁷ which needs an enormous amount of uniaxial stress of more than 10 MPa, we proceed to sample different micromagnet concentrations to examine a variety of magnetic flux density alterations as appeared in Fig. 2(c). Under a continuous uniaxial applied stress, 83 wt. % micromagnet concentration demonstrates the highest values of magnetic field variation of 10.3 mT more than those with 75 wt. % (8.9 mT) and 67 wt. % (5.2 mT) of nanomagnets. The higher increase in micromagnet concentrations would be more difficult to combine with the polymer matrix and consequently, to form a flexible, stretchable, and deformable sensor. Therefore, 83 wt. % is adequate enough to provide extensive output signals while still keeping the desired properties of our magnetoelastic sensor array. In addition, the Young's modulus and the initial magnetic field strength of the magnetoelastic composite (10 × 10 × 1.5 mm³), with different micromagnet concentrations, are measured, as shown in Fig. 2(d). Increasing in the concentration of the nanomagnets not only could intensify the initial magnetic field but also raise the Young's modulus of the magnetoelastic system. The rearrangement of the nanomagnets in the composite could possibly decrease the remanent magnetization and the coercive field in the compressed state, resulting in a negative fluctuation in the magnetic flux density. Furthermore, we verify the magnetic field variation of the device in different magnetization angles under the original state and the applied pressure of 300 kPa. As evidenced in Fig. 2(e), the orientation that directly applies magnetization on the south and north directions provide the highest values of magnetic field variation, where north is the positive direction. Figure 2(f) displays the systematic configuration of how the magnetoelastic composite was oriented to conditioned different magnetization, so each sensor can perform different commands.

Subsequently, each soft magnetoelastic film is patterned with 20 turns of liquid metal coils to establish a fully completed magnetoelastic sensor array, which includes four different magnetoelastic sensors as shown in Fig. 3(a). In our case, the proposed dimension allows the device to fit onto a human hand or wrist for HMI. However, the sensor array can be miniaturized for various application scenarios. The excellent composition of the materials gives great freedom for versatile sensor designs, including size, thickness, softness, and so on. To incorporate the magnetoelastic effect with the coil's electromagnetic induction, different numbers of coil turns are examined as a way to verify both the performance of the current and voltage outputs against the number of coils. Consequently, Fig. 3(b) shows a well-behaved linear relationship. This is consistent with Faraday's law of induction, which declares that both the number of liquid metal turns and the magnetic field variation of the MC layer are positively proportional to the electrical outputs.³² Consequently, 20 turns of coil are chosen due to their indication of high-output signals while still equipping the magnetoelastic sensor array with the abilities to be skin-conformal, flexible, stretchable, and deformable. On the one hand, to characterize the electrical performance of the magnetoelastic sensor array, we investigate the pulse waveforms under different applied frequencies at a fixed pressure, as illustrated in Figs. 3(c) and S4 in the supplementary material and observe that increasing the applied frequencies yields higher electrical outputs. This result correlates with the working principle that the faster the magnetic field changes (the higher frequency), the greater the output electrical signal will be. On top of that, according to Fig. 3(d), with an increased frequency, the output signals exhibit a shorter response time and higher SNR. The background noise is within a controllable range in the lab environment. Additionally, the magnetoelastic sensor array is sensitive enough to detect pressure in the range of human finger tapping. As shown in Fig. 3(e), both the current and voltage response linearly with an increase in pressure ranging from 10 to 80 kPa, confirming a superb sensitivity in the range of human finger tapping. Meanwhile, 10 000 cycles of constant applied pressure were exerted onto the device to validate its durability. Figure 3(f) substantiates the device's excellent stability and repeatability. The fast response time of 0.2 s at a frequency of 1 Hz, a favorable SNR of 34 dB, and a stable output performance reveal that the magnetoelastic sensor array performs better under high-frequency excitation and expresses significant stability and durability, which can be incorporated toward many long-term HMI applications.

With the fundamentally new working principle, the array system is a promising design in applications of HMI for its wearability, flexibility, skin conformity, and stable electrical performance under the exposure to humid environment and submergence in water as illustrated in Figs. 4(a) and S5 in the supplementary material. The whole system is tested underwater, as shown in Figs. 4(b) and S6 in the supplementary material, to examine its performance in producing adequate energy in a liquid environment, such as accessing to control the equipment in a shower, in extreme weather, or under a heavily perspiring body during exercise. Notably, to validate the performance of the magnetoelastic sensor array as a Bluetooth wireless controller, four similar-in-appearance magnetoelastic sensors are embedded into a structural Ecoflex elastomer. Each is individually magnetized in different orientation to exhibit characteristic output signals in order to separately command the features: play, pause, next, and previous, of a music speaker, as shown in Fig. 4(c). To further explore the possibility of differentiating the control sensors, four participants are requested to touch each key as shown in Figs. S7–S10 in the supplementary material. These results deliver output differences between each sensor but similarities between each subject. Thus, when a user is tapping on the magnetoelastic sensor array, the signal detected is conditioned, transmitted, and then converted into on and off signals for the electrical appliances (Fig. S11 and Movie S2 both in the supplementary material). To integrate this technology with a commercialized music speaker as a part of the HMI application and confirm its competence in enacting wireless communication, a process flow system of the circuit system is developed, consisting of three components: a sensor array (four magnetoelastic sensors connected in parallel), a transmitter unit, and a receiver unit, as shown in Fig. 4(d). For the transmitter unit, the magnetoelastic sensor array directly collects the user's biomechanical finger tapping data. The hand gesture signals will then be collected and transferred to an analog circuit for careful amplification and filtration. This step ensures that the output signal can precisely express adequate details that are suitable for processing by an analog-to-digital converter (ADC) and further incorporating in commanding a third-party equipment. Additionally, the data would further be processed by a microcontroller before wirelessly being delivered from a Bluetooth module to another one at the receiver end. In this way, the second microcontroller can receive the command signals generated from the magnetoelastic sensor array, which can also precisely control the audio and display module inside the music speaker. Simultaneously, a latching relay is connected in series with the electrical appliances (music player, lamp, or fan). Figure 4(e) shows the command signals collected after converting by the relay, and in this way, the magnetoelastic sensor array can successfully control a commercial music speaker, as shown in Movie S3 in the supplementary material.

Enabled by the new discovery of the magnetoelastic effect in the soft polymer system, a self-powered sensor array is developed for human–machine interaction with decent wearability and water resistance. It can effectively convert biomechanical signals from finger tapping into bespoke output signals to connect with desired machines. By integrating with a signal-processing circuit that includes an amplifier, low-pass filters, micro-controllers, Bluetooth modules, a relay, and an audio and display module, the magnetoelastic sensor array not only wirelessly acts as the on and off buttons of a lamp but also as a music player's command features, representing the actions of play, pause, next, and previous. These applications are accomplished by the unique magnetization design of each magnetoelastic sensor in order for it to produce identifiable electrical signals. Importantly, between different users, these four output signals remain similar. The device exhibits a well-behaved linear variation in the forms of output voltage and current that shows a superior sensitivity range of 80 kPa, which is suitable for touch sensing, an optimal SNR of 34 dB, and a rapid response time of 0.2 s at 1 Hz. This work demonstrates a unique and compelling approach for self-powered bioelectronics and promises a great adaptable and versatile solution for users in water-resistant HMI applications to control their third-party machine anytime anywhere, ultimately improving our way of living in the smart generation of the IoT and 5 G technologies.

All the soft MC layers are fabricated using Ecoflex 00-30 part A and Ecoflex 00-30 part B with a weight ratio of 1:1. Then, neodymium–iron–boron nanomagnets (MQFP-B-20076-088) with weight concentrations of 65%, 75%, and 83% are combined with the polymer mixture using a stirring rod. Stirring thoroughly for 10 min introduced air microbubbles to produce desirable porous structure. Then, the mixture is poured into a 3D printed template and cured at 70 °C in the oven (Thermo Fisher Scientific) for 4 h. By using different templates, composited films with given thickness could be fabricated. Finally, the cured composited film, positioned at 0°, 45°, 180°, and 225° angle, is individually magnetized by an impulse field (approximately 2.6 T) using an impulse magnetizer (IM-10-30, ASC Scientific) to introduce different remnant magnetization patterns.

Ga (99.99%) and In (99.99%) ingots were purchased from RotoMetals to assemble the liquid metal. Eutectic gallium indium (EGaIn; 74.5 wt. % Ga and 25.5 wt. % In) is heated in a muffle furnace (Thermo Fisher) at 200 °C for 2 h. Then, the liquid metal is mixed with 10 wt. % Ni particles (99.5%, 5 μm, US Research Nanomaterials) thoroughly using a VWR mini Vortexer to acquire the desired rheological property as a way to improve processability before any usage. A laser cutting machine (ULTRA R5000, Universal Laser System) is used to cut a polyethylene terephthalate (PET) film in the shape of a square helix (length, 12 mm; width, 12 mm). The liquid metal is then patterned onto the soft magnetoelastic film using the PET film mask.

Structural characterization of the soft magnetoelastic film was conducted by SEM (Zeiss supra 40VP) and micro-CT (CrumpCAT). Magnetic flux density measurement is succeeded using a digital Gauss meter (TD8620, Tunkia). Uniaxial stress is applied on the soft magnetoelastic film, and the Gauss meter with an axial probe measures the vertical component of the magnetic field. The stress–strain curves are determined by using a dynamic mechanical analyzer (DMA, RSA III). The Young's modulus is calculated by fitting the experimental curves with a neo-Hookean model.

The voltage signals of the magnetoelastic sensors are measured by a Stanford low-noise voltage pre-amplifier (Model SR560) and current signals, a Stanford low-noise current pre-amplifier (Model SR570). Real-time data acquisition and display are realized using the LabVIEW software. The stability of the magnetoelastic sensor is validated by a calibration electrodynamic transducer (Labworks, ET-126HF) at 20 Hz. The electrical output performance of the magnetoelastic sensor is measured at the different frequencies and applied forces. Finally, the pressure meter (HYPX-017) is used to apply an adjustable pressure to the magnetoelastic sensor.

The magnetoelastic sensor array and interaction system are composed of three parts, including a magnetoelastic sensor array, an integrated signal conditioning circuit (transmitter unit), and an integrated command control circuit (receiver unit). First, the electrical signals from the finger tapping are acquired from the magnetoelastic sensor array. The signals are then amplified and filtered by an analog circuit to remove environmental noise. Then, the analog signals are converted to digital signals by a microcontroller (Arduino UNO) and then transmitted wirelessly to the receiver unit through a Bluetooth module (HC-05). Another Bluetooth module (HC-05) in the receiver unit receives these signals and passes them to a second microcontroller (Arduino UNO). Finally, a latching relay is connected and transforms the signals to different commands, which can precisely control the audio and display module inside a music player as well as the on and off function of a lamp.

See the supplementary material for SEM images of the soft magnetoelastic film, photographs of the magnetoelastic sensor's components, its wearability and waterproofness, and with circuit configuration, additional waveforms of the device under applied frequencies, current outputs on each of the four keys.

The authors acknowledge the Henry Samueli School of Engineering and Applied Science and the Department of Bioengineering at the University of California, Los Angeles, for startup support. J.C. also acknowledges the Hellman Fellows Research Grant, the UCLA Pandemic Resources Program Research Award, and the Research Recovery Grant by the UCLA Academic Senate. T.T. acknowledges the Department of Defense (DoD) through the National Defense Science and Engineering Graduate (NDSEG) Fellowship Program.

A patent based on this work has been filed by UCLA.

Jing Xu: Data curation (lead); Investigation (lead); Validation (lead); Writing – original draft (lead). Trinny Tat: Data curation (equal); Investigation (equal); Validation (equal); Writing – original draft (equal). Xun Zhao: Investigation (supporting). Yihao Zhou: Investigation (supporting). Diantha Ngo: Investigation (supporting). Xiao Xiao: Investigation (supporting). Jun Chen: Conceptualization (lead); Funding acquisition (lead); Project administration (lead); Supervision (lead); Validation (lead); Visualization (lead); Writing – original draft (supporting); Writing – review and editing (lead).

The data that support the findings of this study are available from the corresponding author upon reasonable request.