Detection of keyboard vibrations and effects on perceived piano quality

The somatosensory aspects of piano performance have become the object of research interest for a long time due to their role in the complex perception-action mechanisms occurring when this instrument is played. When the aesthetic viewpoint was largely prevailing, Bryan (1902) started in Nature a lively debate in which Allen (1913), Heaviside (1913), Morton (1913), Pickering (1913), and Wheatley (1913) pioneered a discussion about piano sound quality based on physical arguments. Later, Ortmann (1925) explicitly addressed “The Physical Basis of Piano Touch and Tone,” then starting to issue the links between physics and physiology (Ortmann, 1929). In the same period Cochran (1931) discussed the possibility and limits for pianists to convey expression attributes to a piano tone by varying the pressing and releasing action of the finger.

In recent years this discussion has been fueled by the advent of a new generation of force sensors and tactile actuators whose encumbrance, accuracy, and cost permit their inclusion in experimental settings that were exclusively auditory until a few years ago. On the one hand, such technologies have formed a platform for the multimodal design of novel musical interfaces and instrument prototypes, whose haptic feedback has been shown to increase the engagement and, more in general, the intimate relationship existing between musicians and their instrument during a performance, yet apparently with no improvement in the precision of the execution (Marshall and Wanderley, 2011). On the other hand, they offer new possibilities to study the perceptual role of haptic feedback in traditional music instruments, which typically deliver kinesthetic as well as tactile cues to the performer along with sound. In fact, the development of unintrusive and accurate measurements that can be used outside the experimental laboratory for diverse instrumental settings is far from straightforward (MacRitchie, 2015).

This paper focuses on the tactile feedback that follows the initial keypress and related early feedback during piano playing, and investigates through two experiments the hypotheses that strings vibration at the keyboard is perceived through the fingers, and that the same vibration affects the perceived quality of the piano. The study ascribes to a research line that was started by Askenfelt and Jansson (1992), who made systematic measurements of vibrations produced by some popular Western stringed instruments, then discussing about the possibility for a performer to perceive these cues.

Later, Galembo and Askenfelt (2003) showed that the synergistic effect of tactile and kinesthetic feedback coming from some keyboards during playing was more important to recognize a piano than the sound itself. Specifically, this feedback is commonly referred to by pianists as touch, and gathers together haptic sensations depending on the surface material a key is made of, its perceived mass, and the characteristic force exerted in response to the action of the finger, as a consequence of the key mechanics. Although not delving into details on the importance of key vibrations, this result demonstrated the role of somatosensory feedback in the characterization of a piano during performance, at least when kinesthetic cues were included.

Evidence of the importance of tactile cues alone was presented by Keane and Dodd (2011), who discovered a significant preference of pianists for an upright instrument whose keybed had been modified so as to provide vibrations at the keyboard of intensity comparable to that of a grand piano. In the same year, two authors of the present paper explored the effects of adding vibrations to a digital piano (Fontana et al., 2011): Despite the exploratory nature of that study, it was shown that the addition of vibrotactile feedback to a digital piano significantly modified the performer's preferences during playing. In the same period an industrial trend was marked by a worldwide known manufacturer, equipping its flagship digital pianos with force transducers that made the instrument body vibrate during playing (Guizzo, 2010).

The piano keyboard offers a controlled experimental setting, as the performer can only hit and then release one or more keys with one or more fingers. The rest of his or her body is separate from the instrument if the task does not require it to operate the pedals. This limitation permits to design vibrotactile measurements in which the perception of auditory by-products is masked by insulating the ears and providing sound feedback through headphones (Wilson et al., 2009; Bensmaïa et al., 2005). Furthermore, the velocity with which a hammer hits the string is in good relationship with the sound and string vibrations produced by the corresponding keypress (Kinoshita et al., 2007). If a keyboard is equipped with sensors complying with the MIDI protocol—as that of Disklavier pianos—then this relationship is encoded for each key by simple protocol messages. Even under these assumptions, controlling the tactile stimuli and the simultaneous auditory feedback from the piano is not obvious, but feasible, as exemplified by our experimental apparatus.

Classically, cutaneous sensitivity has been studied by considering sinusoidal vibratory stimuli of varying frequency. That led to identifying four types of mechanoreceptors in the skin which mediate different aspects of touch (Lamoré and Keemink, 1988; Bolanowski et al., 1988). Vibrotactile perception was found to be mainly conveyed by the Pacinian system (also known as P-channel), which shows a U-shaped contour of sensitivity in the 40–800 Hz band, with lowest detection thresholds between 200 and $300 Hz$⁠. The non-Pacinian channels I and III respond to low-frequency stimuli up to $100 Hz$⁠, mostly related to the sensation of flutter and pressure. The non-Pacinian channel II is sensitive to vibrations in the 15–400 Hz range, and while that overlaps with the band targeted by the P-channel, its sensitivity is much lower. Therefore, the P-channel is the main responsible for the detection of piano tones vibrations for fundamental frequencies above $40 Hz$⁠, that is, approximately from E1 on.

The literature reports vibration detection thresholds typically in the acceleration range 105–115 dB for sinusoidal stimuli at 100–250 Hz (Maeda and Griffin, 1994). The lowest reported values lie between 97 and $98.5 dB$ (Aatola et al., 1990), for contact areas comparable to that of a fingertip. Verrillo (1992), often cited, reported lowest displacement thresholds of $−20 dB$ (re $10−6 m$⁠) at $250 Hz$⁠, equivalent to about $105 dB$ root-mean-square (RMS) acceleration.

Note, however, that the above results refer to sinusoidal stimuli. Similarly to what happens within an auditory critical band, the P-channel is capable of energy summation of the frequency components falling within its sensitivity range (Verrillo and Gescheider, 1975; Makous et al., 1995). In particular, it shows ability to convey distinct spectral regions simultaneously with good independence and parallelism (Bensmaïa et al., 2005). A piano tone contains several sinusoidal components above the fundamental, or partials, whose amplitudes fade irregularly following a decay curve that is different for each partial (Fletcher et al., 1962). The vibration at the key contains all these partials, although filtered through the instrument's body. This rich spectral content is delivered to the finger in contact with the key immediately after the initial percussive event. For this reason, comparing amplitudes of single partials against sensitivity thresholds obtained using pure sinusoids at the same frequencies would lead to overlooking spectral summation effects that can contribute to the tactile perception of piano tones.

A few studies are found in the literature dealing with the perceptual thresholds of non-sinusoidal stimuli. Gescheider et al. (1990) studied difference limens for the detection of changes in vibration amplitude, with either sinusoidal stimuli at 25 or $250 Hz$⁠, or narrow-band noise with spectrum centered at $175 Hz$ and $24 dB$/octave falloff at 150 and $200 Hz$⁠. The contact area was $2.9 cm2$⁠. They found that the nature of the stimuli had no effect on difference limens. While stimulating the whole hand through a vibrating wooden tablet (contact area of about 50–80 cm²) with various types of signals, Wyse et al. (2012) found RMS acceleration thresholds of about $80 dB$ (re $10−6 m$⁠) at $250 Hz$⁠.

More recently, Papetti et al. (2017) investigated the relationships between vibrotactile sensitivity thresholds and contact force during an active finger pressing task, using both sinusoidal and broadband vibration signals. They found thresholds considerably lower than those previously reported in the literature, and observed significant effects on thresholds for vibration type, contact force, and contact area. These results support the role of spatial and temporal summation in the P-channel, and moreover suggest that active finger pressing also affects sensitivity thresholds.

The P-channel is, in fact, capable of spatial summation (Gescheider et al., 2004). It is known (Verrillo, 1985) that, for contact areas between 0.02 and $5.1 cm2$ and frequencies in the 40–800 Hz range—covered by the P-channel—displacement thresholds improve by approximately $3 dB$ for every doubling of the contact area. Intuitively, then, the probability that receptors will be activated becomes greater for larger stimulation areas. Typically, pianists play long notes using the whole finger pad, while during short tones, the contact area reduces to the fingertip. Moreover, tactile interaction on the piano is multi-finger: Any pianist, in fact, practices for years to achieve a fluent and well-balanced execution of piano chords. For this reason, sensitivity thresholds on the piano should be investigated also in the case of multi-finger contact with the keyboard, as when playing chords. There is little background on multi-finger tactile perception, with the exception of a recent systematic investigation in which non-musicians, among other tasks, were asked to tap on a steel plate using several fingers simultaneously (Shao et al., 2016). The propagation and consequent interaction of the mechanical waves across the skin of the hand revealed spatial, temporal, and frequency patterns of vibrations suggesting that multi-finger perception cannot be seen as a superposition of the somatosensory processes occurring when the same fingers are stimulated in isolation.

The P-channel is also sensitive to temporal summation, which lowers sensitivity thresholds and enhances sensation magnitude as well (Gescheider et al., 2004). Verrillo (1965) found that thresholds decrease for stimuli at $250 Hz$ with a duration of up to about $1 s$⁠, when delivered through a $2.9 cm2$ contactor to the thenar eminence. Gescheider and Joelson (1983) examined temporal summation at stimulus intensities ranging from the threshold to 40 dB above it. For 80 and 200 Hz stimuli, peak displacement thresholds were lowered by up to about 8 dB for durations increasing from 30 to 1000 ms. The effects, however, decreased as a function of intensity.

The evidence for spectral, spatial, and temporal summation effects provides motivations to revisit the conclusions drawn by Askenfelt and Jansson (1992) about the perception of keyboard vibrations. In particular, by comparing magnitude spectra of displacement of a vibrating piano keyboard against sensitivity thresholds obtained with sinusoidal stimuli (Verrillo, 1971, 1985), they argued that weak or no cues of vibration could be detected by a pianist during playing. Furthermore, almost all the studies discussed above measured sensitivity thresholds in passive conditions, whereas pianists experience touch sensations under active finger pressing conditions. As already mentioned, the work by Papetti et al. (2017) shows that active touch lowers the thresholds. Maeda and Griffin (1994) also found dependencies of the vibrotactile thresholds on the forces pressing down the key and keeping it depressed once it has reached the keybed, where the latter force depends on the pianist's skill and personal style (Parlitz et al., 1998).

Finally, cross-modal effects can arise due to the interaction between the sound of one or more notes and the vibrations arriving at the finger(s) while playing the same note(s). To this regard, instructive conclusions were drawn in a few previous studies on bimodal auditory and vibrotactile stimuli. Ro et al. (2009) showed that sounds with the same frequency of the vibrotactile stimuli enhanced tactile detection. Wilson et al. (2009) found evidence of perceptual integration of auditory and tactile stimuli at near-threshold levels, especially when the components in the two modalities were equal or closely spaced in frequency and were both within the Pacinian range. These additive effects were less pronounced when stimuli fell into narrower frequency ranges, speaking in favor of a cross-modal extension of the critical band model (Wilson et al., 2010). Now, pianists have a long practice with vibrating keys which, once pressed, generate sounds whose spectra share components with the vibratory tones. It is therefore possible that an audible piano tone helps detect a tactile signal near threshold, whose vibratory components are a subset of the auditory components. The abundance of harmonic components forming the auditory and tactile stimuli when a piano key is pressed, however, makes the cross-modal experience of playing the piano especially rich and, hence, difficult to measure.

A quality experiment and a detection experiment were designed, respectively, to test the following hypotheses:

• The perceived quality of a piano is affected by vibration at the keyboard.

• Vibration at the keyboard is perceived through the fingers during playing.

• The quality experiment measured the effect of vibrations on the perceived quality of the piano in an active playing task.

• The detection experiment measured the detection of vibrations at single and multiple tones of various pitch, duration, and dynamics.

Both experiments were performed at two separate laboratories, located in Padova, Italy, and Zurich, Switzerland, using similar setups. Different Yamaha Disklavier pianos (Yamaha Corp., Hamamatsu, Japan) were used at the two laboratories: a grand model DC3 M4 in Padova and an upright model DU1A with control unit DKC-850 in Zurich. The Disklaviers are Musical Instrument Digital Interface (MIDI)-compliant acoustic pianos equipped with sensors for reading the velocity of each key, and electromechanical motors for enabling the keyboard to play back MIDI files. Furthermore, a Disklavier can be switched to “silent mode,” in which the soundboard is decoupled from the keys. This mode, hence, prevents the strings from being struck by the hammers meanwhile preserving the mechanical operation of the keyboard unaltered.

During the experiments, the acoustic and silent modes were switched back and forth across trials, letting subjects receive either natural or no vibration from the keys after the initial percussive event. Since the grand and upright Disklavier pianos adopt different mode switching electronics, we had to implement two solutions: On the upright Disklavier we programmed an Arduino microcontroller to send an electrical signal to the piano switching board upon request; on the grand Disklavier it was sufficient to send via Wi-Fi a specific message to the piano control unit. In both cases, the switch was requested by the software for Pure Data that we used to automatize the experiment. Furthermore, no notable sounds and vibrations were produced by the switching electro-mechanics in both Disklavier models, once the participants wore earphones and did not touch the piano during a switch.

In both configurations participants received the same auditory feedback, consisting of synthetic piano sounds produced by the Modartt Pianoteq 4.5 software piano synthesizer,¹ which was set to simulate a grand or an upright piano, and was driven in real time by the Disklavier's MIDI data. The synthesized sound was reproduced by using Sennheiser HDA-200 isolating reference headphones (Sennheiser electronic GmbH & Co. KG, Wedemark-Wennebostel, Germany; grand piano) or Shure SE425 earphones (Shure Inc., Niles, IL; upright piano). In the latter case, 3M Peltor X5A earmuffs (3M Co., Maplewood, MN) were worn on top of the earphones. Preliminary testing confirmed that the Disklaviers' operating modes (acoustic or silent) were indistinguishable when listening to the piano synthesizer through these setups from the performer's seat position, meaning that any simultaneous auditory feedback coming from the acoustic pianos was masked.

The loudness and dynamic response of the piano synthesizer were calibrated to match those of the corresponding Disklavier model in use. Intensity levels were measured by positioning a KEMAR mannequin (GRAS Sound & Vibration A/S, Holte, Denmark; grand) or a sound level meter (upright) in correspondence of the performer's head position. Both were used to record all “A” keys at various MIDI velocities (Fig. 2 shows the two piano setups). Three intensity measurements for each key velocity were taken in correspondence of every A note, and the average peak RMS intensity value in dB(A) was finally recorded. The dynamic response of the piano synthesizer was then matched to that of the corresponding Disklavier by repeating sound intensity measurements on the KEMAR mannequin while playing back synthetic A notes—through headphones (grand) or earphones (upright)—at the same MIDI velocities used before with the Disklavier pianos.

Participants were only exposed to vibrations generated by the pianos when placing their fingers on the keyboard: The pedals were made inaccessible; the grand piano was decoupled from the floor by inserting layers of insulating rubber under the wheels, while the stool and the player's feet were isolated from the floor by means of thick rubber panels; in the case of the upright piano, the stool and the player were placed on a suspended and isolated tile. Vibration measurements confirmed that, as a result of the mechanical insulation, playing the piano did not cause vibrations exceeding the noise floor in the room.

The software piano synthesizer ran on a laptop computer, and a RME Fireface 800 audio interface (RME AG, Haimhausen, Germany) was used to receive MIDI data from the Disklavier and send the consequent synthetic sounds to the headphone sets with sampling rate 48 kHz and latency set to 64 samples.

The experiments were conducted under human control with the help of programs developed in the Pure Data environment.² This software was used to (i) read computer-generated playlists describing the experimental trials, (ii) set the Disklavier playing mode accordingly, (iii) check if the participants executed the requested tasks correctly, and finally (iv) record the participants' answers.

In both experiments, participants performed first a playing task and then a rating task. The Disklavier operated in acoustic mode when string vibrations were present and in silent mode when string vibrations were absent. In both modes the initial percussive event was present. In the quality experiment, three ranges were considered separately across the keyboard, labeled low (keys below D3), mid (keys between D3 and A5), and high (keys above A5). The ranges were marked with adhesive tape. Participants could play freely, within one range at a time, to compare the quality of the instrument in presence and absence of string vibrations following the initial percussive events. In the detection experiment, participants played either single keys or three-key clusters with specific pitch(es), duration, and key velocity. Correctness of the trial was monitored by the measurement software and the test supervisor.

The detection experiment consisted of two parts, labeled as part A and part B from here on. In (detection) part A, participants played long, single tones lasting eight metronome beats at 120 BPM, with MIDI key velocities in the range 72–108 (mezzoforte to fortissimo). Seven keys were considered: A0, A1, A3, D4, A4, D5, and A5. This choice was based on the findings of a previous pilot study by Fontana et al. (2014), who reported that perceptibility of key vibrations vanishes above A5. For this reason, a denser set of pitches was used in the mid range in order to better estimate the cutoff point, while the range above A5 was not considered.

The task in (detection) part B changed both in terms of duration and dynamics, and three-note clusters (chords) were used in addition to single tones. The goal was to investigate whether altering the duration and dynamics, or the number of depressed keys, affected the detection of vibrations. Specifically, as compared to part A, in part B participants had to play A0 and A1 with either shorter duration (two metronome beats at 120 BPM, and MIDI velocity in the range 72–108 as in part A) or softer dynamics (MIDI velocity in the range 36–54, equivalent to p to mp, and lasting eight metronome beats at 120 BPM as in part A). These modifications were expected to make the detection task harder in the lower range, where vibrations are most easily perceived (Fontana et al., 2014). Additionally, participants had to play clusters (C-D-E) around D4 or D5, with long duration and loud dynamics (eight metronome beats at 120 BPM, MIDI velocity in the range 72–108). By extending the contact area, the clusters were expected to make the detection task easier in the upper range, where vibrations are hardly perceived (Fontana et al., 2014).

The within-subject factors and their respective conditions are summarized in Table I for the quality and detection experiments. Piano type (upright or grand) was a between-subjects factor in both experiments.

N = 25 subjects participated in the quality experiment (n = 15 in the upright and n = 10 in the grand piano condition) and N = 28 in the detection experiment (n = 14 in the upright and n = 14 in the grand piano condition). Their average age was 27 yr and average piano experience was 15 yr. About half of the participants were females. All of the subjects in the upright condition of the quality experiment also participated in the detection experiment. In the grand piano condition, five subjects took part in both experiments. There are missing data from two subjects in the grand piano condition in (detection) part B. The groups in the upright and grand conditions were roughly balanced, consisting of pianists in professional training, mostly classical and a few jazz pianists. Nearly all of the participants played mainly an acoustic piano, either upright or grand.

Both experiments were conducted in a single session, first the quality experiment and then the detection experiment. This order was not balanced, since the quality experiment required participants to be unaware that the experiment concerned key vibrations. Each session took about one hour.

Participants could play freely, without using the pedals. Using a manual control, they could switch at their convenience between two setups, “X” and “Y,” associated with the silent and acoustic modes of the Disklavier. The actual difference between the setups was not explained to them.

The task was to compare the setups with respect to the following attributes: dynamic range, loudness, richness, naturalness, and preference. The first four were rated separately in the low, mid, and high ranges, while the preference rating was given considering the entire keyboard, resulting in 13 ratings per participant. Participants were given definitions of the attributes and informed that dynamic range, loudness, and richness were mainly related to sound, whereas naturalness and preference could also be related to touch. The considered attribute scales were inherited and adapted for the present study from previous experiments on the Disklavier (Fontana et al., 2014), the violin (Saitis et al., 2012), and the upright piano (Keane and Dodd, 2011). A laptop placed next to the piano displayed a graphical user interface with sliders for rating the different attributes. Attributes were rated on a continuous Comparison Category Rating scale (CCR; ITU-T, 1996), ranging from −3 to +3 as follows:

• +3: “X much better than Y”

• +2: “X better than Y”

• +1: “X slightly better than Y”

• 0: “X equal to Y”

• −1: “Y slightly better than X”

• −2: “Y better than X”

• −3: “Y much better than X”

A single slider was available for rating preference and, for each of the three keyboard ranges, a set of four sliders were used to rate the remaining attributes. Ratings could be given in free order.

The experiment followed a yes-no procedure. Each pitch condition was repeated eight times in acoustic mode and eight times in silent mode. The total number of trials, each consisting of a single keystroke, was therefore 16 × 7 = 112 in part A and 16 × 6 = 96 in part B. Presentation order was randomized. The participants' task was to report whether they had detected vibrations during a trial or not. A new trial was prompted through headphones by a voice message, specifying the key(s) to be played, along with dynamics and duration. Right after the voice prompt, a metronome started at 120 BPM and participants could perform the trial. After releasing their finger, participants gave a “yes” or “no” response to the experimenter, who recorded it into the software.

Participants were informed that vibrations were sometimes present and sometimes absent, and that they should report their own perception instead of saying “yes” all the time. They were instructed to use their index fingers for single keys or fingers 2-3-4 for chords, to press with the finger pad, and to wait for the key(s) to reach the keybed before focusing their attention on the vibration. They had to play keys left of the middle C using their left hand (pitches A0, A1, and A3) and the remaining keys using their right hand (pitches CDE4, D4, A4, CDE5, D5, and A5).

Part A was always performed before part B of the detection experiment in order to let participants become familiar with the voice prompt and simpler task of part A before tackling the more complex task in part B. We address the possible effect of the unbalanced presentation order in Sec. IV A.

The results of the detection experiment are presented and discussed first, as they help motivate the outcome of the quality experiment.

Sensitivity index $d′$⁠, as defined in signal detection theory (Green and Swets, 1966), was computed for each subject and pitch as follows:

where Z(p) is the inverse of the Gaussian cumulative distribution function, hits = proportion of “yes” responses with vibrations present, and false alarms = proportion of “yes” responses with vibrations absent. Thus, a proportion of correct responses $p(c)=0.69$ corresponds to $d′$ = 1 and chance performance $p(c)=0.50$ to $d′=0$⁠. Perfect proportions 1 and 0 would result in infinite $d′$ and were therefore corrected by (⁠$1−1/16$⁠) and (1/16), respectively (Green and Swets, 1966).

Results of part A of the detection experiment are presented in Fig. 3. Sensitivity is highest in the lowest range and decreases toward the high range. Performance drops to chance level, where subjects must have been mainly guessing, at D5 and A5. However, at A4 vibrations are still detected with mean $d′=0.84$ for the upright and $d′=0.87$ for the grand piano (95% confidence interval (CI) = [0.35,1.32] and [0.30,1.34], respectively) corresponding to roughly 64% of correct responses. The cutoff pitch, above which vibrations cannot be detected anymore, is thus somewhere between A4 (⁠$440 Hz$⁠) and D5 (⁠$587 Hz$⁠).

Statistical analysis was performed in R (R Core Team, 2015) using the afex package and the lsmeans package for post hoc comparisons. A mixed analysis of variance (ANOVA) was conducted to investigate the effects of pitch (within-subjects factor) and piano type (between-subjects factor) on sensitivity to vibrations. As hypothesized, the main effect of pitch is significant [$F(6,156)=26.98$⁠, p < 0.001, $ηG2=0.401$]. The results for the upright and the grand piano do not differ significantly [$F(1,26)=0.007$⁠, p > 0.05], nor is there a significant interaction of pitch and piano type [$F(6,156)=0.59$⁠, p > 0.05]. The Mauchly test showed that sphericity had not been violated (p = 0.24).

The results show a monotonically decreasing trend with the exception of D4 versus A4. However, this contrast is not significant [$t(156)=0.77$⁠, p = 0.44]. The results were collapsed over upright and grand piano and a trend analysis was conducted. A significant linear trend can be seen [$t(156)=−12.3$⁠, p < 0.0001], indicating that as pitch increases, sensitivity to vibrations decreases. Quadratic, cubic, and quartic trends are not significant.

Results from parts A and B of the detection experiment are presented together in Fig. 4, showing small differences in mean sensitivity between the normal, soft, and short conditions. However, none of the contrasts between normal and short duration or normal and soft dynamics at A0 or A1 is statistically significant.

A larger difference is observed for clusters versus single notes. For the cluster CDE4 sensitivity is considerably higher than for the isolated note D4. The contrast is significant [$t(294)=5.96$⁠, p < 0.0001], whereas the much smaller difference between D5 and the cluster CDE5 is not significant [$t(294)=1.20$⁠, p > 0.05]. Part A was performed before part B, hence, presentation order was balanced within parts but not between them. To assess a possible learning effect, sensitivities were averaged over pianos and over pitches A0 and A1 in part A and part B, respectively: The mean $d′$ in part B is 0.23 higher than in part A. The effect is, however, small compared to the increase in sensitivity for the CDE4 cluster, thus we conclude that at D4—where sensitivity is decreased but still above chance level—playing a cluster of notes facilitates vibration detection.

The results of the quality experiment are presented in Fig. 5, where a positive value signifies preference for the vibrating mode.

Due to limited rating scale, normality of the distribution of ratings could not be assumed and was tested for all scales using the Shapiro-Wilk test. A likely normal distribution as verified for most of the rating scales (p > 0.05). Only dynamic range in the high range of the grand piano (p = 0.02) and richness in the high range the upright piano (p = 0.04) failed the normality test. A further ANOVA assumption, homogeneity of group variances was shown by Bartlett's test for preference [$χ2(1)=0.21$⁠, p > 0.05]. Therefore, a parametric analysis of the preference scores was undertaken. The mean preference scores are M = 1.05 [n = 15, standard deviation (sd) = 1.48] for upright piano and M = 0.77 (n = 10, $sd=1.71$⁠) for grand piano. A one-way univariate ANOVA on the preference scores showed no significant difference between pianos [$F(1,23)=0.20$⁠, p = 0.66]. The joint distribution of preference scores was likely not normally distributed (Shapiro-Wilk, p > 0.05). Thus, a non-parametric Wilcoxon test was carried out and indicated that the joint median = 1.09 is significantly greater than zero (V = 240.5, p < 0.01). A joint histogram of the preference scores for both pianos is shown in Fig. 6.

The attribute rating scales are moderately to highly correlated with each other. The mean Pearson correlation between all pairs of attribute scales is $ρ=0.35$ (⁠$sd=0.31$⁠). The mean correlations between attribute scales within pitch ranges are generally higher: $ρ=0.50$ (⁠$sd=0.28$⁠) in the low range, $ρ=0.36$ (⁠$sd=0.29$⁠) in the mid range, and $ρ=0.45$ (⁠$sd=0.18$⁠) in the high range. The low and mid ranges are generally more correlated with each other (⁠$ρ=0.58, sd=0.17$⁠) than the mid and high (⁠$ρ=0.43, sd=0.21$⁠) or the low and high ranges (⁠$ρ=0.26, sd=0.29$⁠).

The attribute scales are also positively correlated with the preference scores (mean $ρ=0.19, sd=0.41$ and $ρ=0.60, sd=0.28$ for upright and grand pianos, respectively). For the upright piano, naturalness in the mid range had the highest correlation with preference (⁠$ρ=0.86$⁠). For the grand piano, several scales had moderate to high correlation with preference, up to $ρ=0.91$ for naturalness in the low range. Note that the high range (starting at A5) was entirely beyond the pitch range (up to A4) where key vibrations were perceived in the detection experiment. Therefore it is doubtful whether the ratings in the high range result from actual perceived differences between the setups, or subjects rather gave ratings in favor of their preferred setup in all ranges. A new set of dependent variables was therefore computed for each participant and quality attribute as the average of the ratings at the low and mid ranges. A multivariate analysis of variance (MANOVA) on the new dependent variables showed that the differences in attribute ratings between the upright and grand pianos are not statistically significant [$F(13,11)=0.63$⁠, p = 0.79]. Partial correlation coefficients were computed between preference and the new variables for each scale in order to estimate the correlation of the two variables while controlling for the other three: Naturalness has a significant association with preference (⁠$γ=0.55$⁠, p < 0.01), while for the remaining attributes partial correlations are not significant.

To further explore the relationships between preference and the remaining attributes, principal component analysis was performed on the new dependent variables, standardized prior to the analysis. The first two principal components (PCs) accounted for 80% of the variance. PC1 has the highest correlations with richness, naturalness, and preference, which all increase if one increases. PC2 is associated with dynamic range and loudness, which decrease as naturalness and preference increase. The coefficients for the principal components are given in Table II, and individual rating profiles are projected on the first two principal components in Fig. 7. Figure 7 also presents the directions of the attributes relative to the first two PCs: Naturalness and richness are close to preference, whereas dynamic range and loudness together point to another direction. The negative loadings of dynamic range and loudness in PC2 versus PC1 suggest a possible disagreement between participants on these attributes. We target this finding in the following agreement analysis.

Inter-individual consistency was assessed by computing the overall Lin concordance correlation coefficients (Lin, 1989, 2000; Barnhart et al., 2002) based on the new dependent variables. Overall correlations are $ρoc=0.018$ for upright and $ρoc=0.013$ for grand piano. A joint histogram of the pairwise correlations in both conditions is presented in Fig. 8. Participant-specific inter-individual consistency was estimated as the average ρ_c between each single participant and all other participants: Mean participant-specific consistencies are $ρc=0.080$ and $ρc=0.016$ for the upright and grand piano, respectively.

The generally low inter-individual concordance correlations, albeit in line with previous studies on instrument evaluation (Saitis et al., 2012), suggest a high degree of disagreement between participants. Specifically, three participants in the upright and four in the grand piano condition have a negative inter-individual consistency. Most of them belong to the group of five participants who gave a negative preference rating. In Fig. 7, this group segregates as having altogether the most negative values on both principal components. We therefore used a negative preference rating as a criterion for a posteriori segmentation (Næs et al., 2010) to study the differences between the respective groups as shown in Fig. 9. While the negative and positive groups give rather similar ratings for dynamic range and loudness, their mean ratings for richness, naturalness, and preference are nearly opposite to each other. Thus, the second principal component reflects the behavior of the negative group. The mean preference ratings are M = 1.58 (⁠$sd=0.79$⁠, n = 20) and $M=−1.61$ (⁠$sd=1.10$⁠, n = 5) for the positive and negative groups, respectively. Based on the analysis of partial correlations and principal components, presented above, and the behavior of the negative and positive groups, we conclude that, for 80% of the participants, key vibrations contribute to increased perceived quality of the piano, which is associated to perceived naturalness and richness in the low and middle ranges. We address the question, why a minority of the subjects did not prefer the vibrating setup, in Sec. VI, Discussion.

In the upright condition, nine participants gave a spontaneous written statement about the differences between the setups. Seven statements preferred the vibrating setup and two were neutral. The vibrating setup was described as “great, a mixture of piano and bass,” “more responsive to dynamic changes,” “richer in lower keys,” “more natural in the mid range,” “having more of the soft growl of a beast,” “more powerful,” and “more natural and rich in the lower register.” The two neutral statements were “preference depends on previous experience and habits of each player” and “both X and Y were unnatural.” After completing the quality experiment, the experimenter asked the participants what may have caused the difference between the setups. Interestingly, only 1 of the 15 participants could pinpoint vibrations. Thus, while the participants generally preferred the vibrating setup, they were not actively aware of vibrations.

In order to gain further insight into the results of the detection experiment, vibration signals at the keyboard were acquired on both the grand and the upright Disklaviers.

Measurements were made with a Wilcoxon Research 736 piezoelectric accelerometer (Wilcoxon Sensing Technologies Inc., Amphenol, MD) having high sensitivity (⁠$10.2 mV/m/s2$ at $25 °C$⁠) and flat frequency response in the 10–15 000 Hz range (⁠$±5%$⁠). A Wilcoxon Research iT100M Intelligent Transmitter was used to drive the accelerometer and form the vibration signal. The alternate current-coupled output from the transmitter was recorded as an audio signal at $96 kHz$ and 24 bit resolution via a RME Fireface 800 interface connected to a laptop.

Vibration signals were recorded from all keys of the Disklavier pianos by attaching the accelerometer with two-sided adhesive tape close to the outer end of each key, i.e., at the position where pianists normally place their fingers. A digital audio workstation software was used to record vibration signals, while reproducing MIDI tracks that played back each single key of the Disklaviers with constantly increasing MIDI velocity: 12, 23, 34, 45, 56, 67, 78, 89, 100, 111. Additional MIDI tracks were used to play CDE4 and CDE5 clusters, while vibration was recorded with the accelerometer attached to the respective C, D, and E keys in sequence. MIDI “note ON” messages lasted 16 s, in this way allowing a note to fade out completely before releasing the corresponding key. The MIDI velocities were chosen to cover the entire dynamics reproducible by the Disklaviers' motors: Previous measurements made with a sound level meter in fact showed that our Disklaviers, similarly to other actuated pianos (Goebl and Bresin, 2003), can reproduce lower and higher velocity inputs only to a limited extent and tend to saturate extreme dynamics toward the mid range.

Acceleration signals had a large onset in the attack, reporting about the initial fly of the key from the idle to the pressed position. Figure 10 shows a typical attack, recorded from the grand Disklavier actuating the note A2 at MIDI velocity equal to 12. These onsets, appearing in the first 200–250 ms, result from a kinematic event and were therefore manually removed from each sample in order to isolate the string vibrations in the signals.

Accelerations in m/s² were computed from the acquired signals by making use of the nominal sensitivity parameters of the audio interface and the accelerometer: the digital signals, ranging between −1 and 1, were first converted to voltage values through the full scale reference of the audio interface (in our case $0 dBFS$ at $+19 dBu$⁠, reference $0.775 V$⁠), and then transformed into proportional acceleration values through the sensitivity constant of the accelerometer.

Similarly to Askenfelt and Jansson (1992), the spectra of the resulting acceleration signals were compared to the Verrillo (1992) reference vibrotactile sensitivity curve. Note that this curve reports sensitivity as the smallest, frequency-dependent displacement A(f) (in $m$⁠) of a sinusoidal stimulus $s(t)=A(f) sin(2πft)$ that is detected at the fingertip. Therefore, a corresponding acceleration curve was computed from the original displacement curve in order to compare with our acceleration signals. Thanks to the sinusoidal nature of the stimuli employed by Verrillo, the threshold acceleration signal could be found analytically as $s¨(t)=−A(f)(2πf)2sin(2πft)$⁠. Consequently, the acceleration curve $A(f)(2πf)2$ was used for comparison to our signals. Confirming the results of Askenfelt and Jansson (1992), no spectral peaks were found to exceed the threshold acceleration curve, even for notes played with high dynamics. To exemplify this, Fig. 11 shows the spectrum of the highest dynamics of the note our participants detected with the highest sensitivity, i.e., A0 played at MIDI velocity 111, along with the threshold acceleration curve.

Since Verrillo's thresholds do not explain the results of our detection experiment, a different approach was adopted in analogy with the work by Papetti et al. (2017). RMS acceleration values were computed in place of spectral peak amplitudes. Such values account for the averaged power of vibration acceleration in a given time window, and are obtained by integrating the squared values of the acceleration signal in the same window and then extracting the squared root of the result. Such signals were generated by the Disklaviers, driving the respective key actuator with a MIDI velocity averaging those measured during the detection experiment for the same key (see Table III).

Vibration signals were first processed to remove the components falling outside the tactile band from the computation of the RMS value. In order to do that, a filter was designed whose response approximated the inverse of Verrillo's curve in Fig. 11. This design resulted in a low-pass filter with cutoff frequency at 200 Hz and attenuation of $−50 dB$ at $1000 Hz$⁠. RMS values in $dB$ (re $10−6 m/s2$⁠) were then extracted from the filtered signals over time windows equal to the lengths of the stimuli, that is 1 s for short, and 4 s for long trials.

Figures 12 and 13 show the resulting RMS values together with the RMS thresholds of vibration reported by Papetti et al. (2017), respectively, for part A and part B of the detection experiment. Those thresholds were obtained with a psychophysical method targeting 70.7% correct responses, corresponding to a sensitivity index value $d′≈1$⁠. A comparison of the RMS values and the perception thresholds for noise in Figs. 12 and 13 against the sensitivity curves in Figs. 3 and 4 suggests that RMS values of broadband stimuli have more potential to explain the results of our experiment.

The results presented in Sec. IV show that sensitivity to key vibrations is highest in the lowest range, and decreases toward higher pitches. Vibrations are clearly detected in many cases where the vibration acceleration signals hardly reached typical thresholds found in literature for sinusoidal stimuli.

The literature on the detection of complex stimuli provides support to our results, although it does not explain them completely. The RMS acceleration threshold values at $250 Hz$ reported by Wyse et al. (2012) correspond to $80 dB$ (re $10−6 m$⁠), that is a value compatible with our results. However, the characteristics of those stimuli may have occasionally produced significant energy at lower frequencies, causing the thresholds to lower once they were presented to the whole hand.

Once the pianist has received the initial percussive event (Goebl et al., 2005), the peak of energy promptly decreases and the partials start to fade each with its own decay curve. The initial peak may produce an enhancement effect similar to those measured by Verrillo and Gescheider (1975) limited to sinusoids, and hence contribute to increasing sensitivity.

As discussed earlier, the P-channel is sensitive to the signal energy and has no possibility to recognize complex waveforms. Loudness summation instead occurs when a vibration stimulates both the Pacinian and non-Pacinian (NP) channels, lowering the thresholds accordingly (Verrillo and Gescheider, 1975; Makous et al., 1995; Bensmaïa and Hollins, 2000). In our experiment, summation effects were likely to occur when the A0 key was pressed and, probably, also when A1 was pressed. From the A3 key on, only the P-channel became responsible for vibration perception. Figures 3 and 4 seem to confirm these conclusions because they show a pronounced drop in sensitivity between A1 and A3 in both parts of the detection experiment. As Figs. 12 and 13 demonstrate, this drop is only partially motivated by a proportional attenuation of the vibration energy in the grand piano; furthermore, it is not motivated at all in the upright piano. Hence, it is reasonable to conclude that the NP-channel played a perceptual role until A3. Beyond that pitch, the summation effect ceased.

In accordance with the literature on spatial summation of tactile cues (Craig and Sherrick, 1969), our results also suggest a constructive effect when a cluster in the tactile band is played instead of single notes. As Fig. 4 shows, playing the cluster in the fourth octave boosted the detection in that octave, whereas the same effect did not occur in the fifth octave. This evidence opens an interesting question about the interaction of complex vibrations reaching the fingers simultaneously. Yet, this question becomes too complicated if the piano is used as a source of multi-finger vibrations. In fact, as we observed in our measurements, keys played to form a chord vibrate differently from when they are played individually, and the change in energy depends on the keys combination. This happens because every key receives all the partials forming the chord, each one in a portion varying with the key position in the keyboard. The resulting stimulus at the key is therefore impossible to predict, unless all the key combinations of interest are measured one by one. Leaving a systematic investigation of multi-finger vibration perception for future research, our results show the existence of relevant tactile cues of multiple vibration when a cluster in the fourth octave is played, making the detection easier in that octave. Measurements of cutaneous vibration propagation patterns in the hand resulting from finger tapping show, however, an increase in both intensity and propagation distance with the number of fingers involved (Shao et al., 2016), which may contribute to explain the increased sensitivity we observed.

The existence of cross-modal amplification effects needs to be shortly discussed, even though our experiment did not investigate this aspect. As introduced earlier in Sec. I, previous studies on cross-modal integration effects (Ro et al., 2009; Wilson et al., 2009) support the concrete possibility that an audible piano tone helps detect a tactile signal near threshold, whose vibratory components are a subset of the auditory components. Although in our case the sound came from a synthesizer, both the auditory and tactile signal shared the same fundamental frequency of the piano tone and, furthermore, the first partials were close to each other, respecting the hypothesis of proximity in frequency investigated by Wilson et al. (2009). We did not test a condition in which subjects played the piano in acoustic mode in absence of auditory feedback from the synthesizer. Although testing such a condition may provide significant data about the presence or absence of a cross-modal effect, a different experimental setup should be devised: The insulation of our participants' ears, in fact, was not strong enough to mask the piano sounds completely in absence of comparable auditory feedback from the synthesizer. This fact would make such condition difficult to control. Other cross-modal effects that may have instead contributed to impair the detection (Yau et al., 2009) are to be considered as minor with respect to the spectral compatibility and temporal synchronization of the audio-tactile stimulus occurring when a piano key was pressed.

The maximum and minimum sensitivity thresholds (Papetti et al., 2017) appearing in Figs. 12 and 13 correspond to respective pressing forces 1.9 and $8 N$ for noisy vibrations, and 1.9 and $15 N$ for $250 Hz$ sinusoidal vibrations. These force ranges occur when piano keys are pressed at dynamics between pp and f, with negligible difference between struck and pressed touch style (Kinoshita et al., 2007; Galembo and Askenfelt, 2003). Conversely, ff dynamics require stronger forces up to $50 N$⁠. Even if the pianists' kinematic profiles during keypress depend on a range of factors, including the specific key mass and musicians' expertise, and for this reason are to a good extent individual (Minetti et al., 2007; Parlitz et al., 1998; MacRitchie, 2015), it seems reasonable to think that participants to the detection experiment initially pressed the key according to the dynamics required by the trial and then, once the key had reached the keybed, accommodated the finger force on a comfortable value while attending the detection process. If our participants adapted the finger force toward the ranges mentioned above, then their performance during the detection experiment would fall in between the results found by Papetti et al. (2017) for sinusoidal and noisy stimuli. In the same work it was additionally found that musicians on average exhibit slightly better tactile acuity than non-musicians when using low finger force: Even if this difference was not significant, our participants could have reduced the finger force only after starting a trial that required loud dynamics, meanwhile leaving the force substantially unvaried during the entire task in the other cases. This behavior seems indeed quite natural.

Participants' sensitivity to a RMS acceleration in between the thresholds for a sinusoid at $250 Hz$ and a filtered noise admits an explanation. During its steady state, a piano tone closely resembles a noisy sinusoid. For instance, it can be simulated by employing several hundreds of damped oscillators whose outputs are subsequently filtered using a high-order transfer characteristic (Bank et al., 2010). The question is whether the RMS acceleration values of filtered noise plotted in Figs. 12 and 13 explain our thresholds sufficiently, or if there is a need to discuss them further. Other elements in favor of further discussion are the mentioned potential existence of a cross-modal amplification, and evidences of superior tactile acuity in musicians (Zamorano et al., 2015).

Audio-tactile interactions generally exist in loudness perception (Wilson et al., 2009; Yau et al., 2010). Specifically concerning musicians, greater touch sensitivity along with lower tactile spatial acuity was reported in string, brass, or wind instrument players (Zamorano et al., 2015), whereas greater spatial acuity was observed in pianists who went through periods of heavy training during their life (Ragert et al., 2004). Probably all present in our experiment, these elements form an articulate context whose individual factors cannot be disentangled, and hence neither confirmed nor contradicted by our results.

In the quality experiment, the effects of key vibrations were most notably perceived as increased preference, which was associated with richness and naturalness. These results find support from the literature in that richness had a significant association with preference in a study on the violin (Saitis et al., 2012). In a study focusing on vibrotactile feedback, Wollman et al. (2014) reported a clear positive effect of vibrations of natural amplitude versus half amplitude in comparative ratings of violins in an augmented listening situation. The effect was strongest for the attribute rich sound and still significant for the attributes loud and powerful and pleasure, while no effect was observed for the attribute alive and responsive. In an absolute judgment task, however, the effects were much weaker. Our results, although concerning a different instrument and natural vibrations versus no vibrations at all, are in agreement with what was reported by Wollman et al. (2014) when it comes to the effect of vibrations on general preference and increased richness in a comparative task. Our finding that participants could not identify vibrations as the difference between setups might be related to the weakness of the effect in an absolute judgment task observed by Wollman et al. (2014). The relationship between preference and loudness or dynamic range was not as strong in our study, even though the latter two were in favor of the vibrating mode in the low and mid ranges. They were, however, rated similarly both by the majority who preferred the vibrating setup and the minority preferring the non-vibrating setup.

This study focuses on preference. Confirmation of the preliminary conclusions about the underlying factors calls for a larger scale experiment with a factorial design—ruling out cognitive fatigue—where each participant rates only a certain combination of factor levels, one rating scale at a time in randomized order, and setup names X and Y likewise balanced across trials. Furthermore, intra-individual consistency should be addressed through replications, and test validity by including trials whose effect can be expected, such as trials with limited dynamic range or varying loudness in one of the setups. A further point of future interest is to find out why 20% of the participants preferred the non-vibrating setup. Two subjects in the negative group also participated in the detection experiment 2 (both in the upright condition). On average, they scored sensitivities $d′$ slightly but not significantly lower at A0 and significantly lower at A1, A3, and D4 than the positive group (n = 13), suggesting that they perhaps did not perceive the effects of key vibrations as strongly as the positive group. Overall for the upright piano, a weak positive correlation (⁠$ρ=0.23$⁠) exists between the participants' sensitivity scores from the detection experiment, averaged over A1 to A4, and their preference ratings from the quality experiment. However, there may be other reasons, which the present design cannot reveal.

A future goal is to study the importance of vibrotactile feedback in expressive piano performance. Touch quality and its relation to tone quality have recently received great attention due to the long-standing discrepancy between the physical basis of piano timbre and the experience of advanced pianists (Báron, 1958; Askenfelt and Jansson, 1990). According to the former, the player can only control key velocity, which affects the intensity of the tone, whose consequences on tone quality are instrument-dependent. Pianists, on the other hand, claim to control timbre and loudness independently through touch and gestural means. It has been shown that pianists use a common vocabulary to describe the timbre space of the piano (Bellemare and Traube, 2005). Bernays and Traube (2011) clustered these descriptors into five main categories: dry, bright, velvety, round, and dark, and could likewise show that pianists can identify the respective intended timbres from recorded piano performances (Bernays, 2013). There is active research toward filling the gap between playing technique and the acoustic result (Chabassier and Durufle, 2015; Vyasarayani et al., 2009). The importance of finger-key and key-keybed noises to quality of single tones has been discovered (Goebl et al., 2014), and control of balance, synchrony, and articulation have been related to touch and tone quality in chords (Dahl et al., 2010). Parncutt (2013) proposed to renew the notion of timbre to include cross-modal aspects.

The effect of vibrotactile feedback on performance accuracy is a further topic for future research. Higher stability of finger force in presence of proportional vibrotactile feedback has been reported (Ahmaniemi, 2013; Järveläinen et al., 2013). In our detection experiment, at each trial participants played single keystrokes, and therefore they could not benefit from the vibrotactile feedback of the depressed key(s) for planning further keystrokes. Moreover, they did not know whether the upcoming trial would contain vibrations or not. In the quality experiment, participants could play freely and use all the available vibrotactile feedback. The mean ratings for loudness and dynamic range were in favor of the vibrating setup, but a strong relation to preference could not be observed. Such inconclusive result suggests further studies on accuracy of dynamic and timing control under varying feedback conditions. It may further unfold how pianists use vibrations to control tone quality through various aspects of touch.

Two experiments were performed on a grand and an upright piano. A detection experiment showed that pianists can perceive key vibrations in the tactile band up to a pitch estimated between A4 and D5. In the cutoff range, the RMS acceleration of the vibrations was comparable to sensitivity thresholds previously measured for noisy signals in an active task involving finger pressing forces compatible with our experiment. Sensitivity decreased from low to high pitches, suggesting loudness summation across channels in the lowest range.

In a quality experiment, pianists preferred a setup providing normal vibratory feedback to one without vibration. Their preference judgments were associated with the attributes richness and naturalness in the lower and middle octaves. While vibrations had a positive effect on perceived instrument quality, participants could generally not recognize that the only change in the two setups consisted in the presence or absence of vibrations.

Our findings suggest to further investigate the role of vibrotactile feedback in skilled piano performance. Its indisputable effect on perceived touch quality should also manifest in the way that pianists control nuances through touch and, consequently, in the quality of the performance.

This research was pursued as part of project AHMI³ (Audio-Haptic modalities in Musical Interfaces) and F.F.'s International Short Visit entitled Enduring international leadership of ZHdK in Musical Haptics (Project No. IZK0Z2_171102) to the Zurich University of the Arts, both funded by the Swiss National Science Foundation. We gratefully acknowledge Balázs Bank for his insightful comments and help in the measurement and preliminary analysis of piano vibrations, and Francesco and Valerio Zanini for recording and assembling the vibration samples library.