Classification and automatic transcription of primate calls

A central topic in bioacoustics is the description of animal call repertoires, including what the calls are and how they are combined and used. However, traditional acoustic analysis of calls requires a significant amount of manual work, which means that only a fraction of the data collected in the field is actually used, and the majority of the otherwise useful data does not serve its role in answering scientific questions (Kobayasi and Riquimaroux, 2012). Recently, techniques from speech processing have been applied to animal vocalizations. The key advance they offer is to bypass a step where researchers extract preselected acoustic features, such as durations or peak frequencies. Standard speech processing tools represent signals using rich, general purpose spectral representations, with no hand selection of acoustic features. Previous analyses automatically classified isolated calls by call type, species, and caller using such representations (Mielke and Zuberbühler, 2013). Our system, in addition to labeling isolated calls, detects and labels calls in raw field recordings. We apply it to three primate species with acoustically diverse calls (see Fig. 1): Blue monkeys (Cercopithecus mitis), Titi monkeys (Callicebus nigrifrons), and Colobus monkeys (Colobus guereza).

Recordings of three species were taken from several field researchers for a total of 5.58 h of audio. A trained primatologist marked the start and end times (calls typically do not overlap), and labeled each Blue monkey call Hack (also called Ka) or Pyow (Papworth et al., 2008), Colobus calls as Roar or Snort (Marler, 1972), and Titi calls as A or B (Cäsar et al., 2012). Table 1 documents the length of the audio recordings for each data set, the percentage of that time taken up by calls, and the token count for each type of call. Estimated signal-to-noise ratios for these data sets (Vondrášek and Pollák, 2005) were low (between 0.5 and 5.3), typical of field recordings in primatology.

Acoustic features were automatically extracted from the audio recordings using a standard speech feature extraction pipeline, adapted minimally. Since the recordings had non-zero mean and varying average amplitudes within each recording due to recording conditions and manual adjustment of the gain levels by field scientists, we removed the DC component with a notched high pass filter. We increased the ratio between calls and noise with a five-point temporal median filter (i.e., averaging windows of five consecutive samples in the time domain) followed by a two-dimensional three-point median filter pass in the spectral domain. The first enhances the ratio of the amplitude of the calls to noise, and the second flattens the spectrum for low transient noise passages and enhances the contrast with calls. We then estimated a noise signature based on the spectral components of the first half second of each audio file (which never included a call) and subtracted this noise signature from the rest of the audio stream. We calculated spectral representations of the signal using short-term Fourier transforms on overlapping windows of 25 ms shifted by 10 ms, and transformed the frequency components through a set of 40 filters evenly spaced on the Mel scale. This filter distribution is common in speech processing and is copied here for generality. Finally, each filter was mean-variance normalized independently.

In this section, we describe three experiments classifying isolated calls using the generic acoustic features just described. Each call was represented by concatenating the first 50 frames from the call onset (a 40 × 50 = 2000-dimensional vector, corresponding to 515 ms), capturing the full length of 84% of calls. In experiment 1, we assessed the ability to classify call types within each species based on these representations. In experiment 2, we assessed classification of species. In experiment 3, we assessed the six-way labeling of species and call type required when all three species are pooled.

To predict the calls, we used a sparse radial basis function support vector machine (SVM) trained with block coordinate descent with squared hinge loss and L1 regularization. This is a standard statistical approach to classification problems that may not be amenable to classification using a linear model. Instead of computing the full Gram matrix of the kernel, we employ the Nyström approximation to significantly speed up the training time of our classifiers (Williams and Seeger, 2001). The approximation computes the eigendecomposition on a random small subset of the Gram matrix and scales the results up to the original number of dimensions (the number of samples). We achieved good results with a 500-component approximation. Experiments 2 and 3 involve more than two classes, so we employed a one-versus-rest strategy (training N individual binary classifiers, where N is the number of classes). Training was on 80% of the data, with evaluation on the remaining, unseen, 20%. Three hyperparameters (weight of the loss term, C, weight of the penalty term, λ, and kernel coefficient, γ) were optimized using the sequential model-based algorithm configuration (SMAC) technique (Hutter et al., 2011) by fivefold cross-validation within the training set.

Table 2 shows the results of experiments 1–3. We give precision (positive predictive value: among the calls the classifier gives label x, the fraction that are actually x and not false positives) and recall (sensitivity: among the calls that should be labeled x, the fraction that are labeled x and not false negatives), and F₁ [$2 precision recall/(precision+recall)$]. Classification was good, with average F₁ of between 0.91 and 0.99. Experiment 1 extends previous findings using different methodology and new species (Mielke and Zuberbühler, 2013). Experiments 1 and 3 were repeated with subsets of increasing sizes of the full (i.e., 80%) training set. Figure 2 shows the F₁ score on the test set as a function of the number of annotated calls given for training.

In experiment 4, we trained a call transcription system whose input is raw, unsegmented field recordings. It predicts call labels using a support vector machine and uses a conditional random field (CRF) to correct unlikely sequences.

The SVM was trained on annotated data to predict call labels from individual frames. Input features consisted of a concatenation of MFCC features (13 cepstral coefficients with first and second derivatives) with activations from a voice activity detection (VAD) system (Lee and Hasegawa-Johnson, 2007). The classifier was trained within species to predict one of the two call types or a third class indicating the absence of a call. The sequence of Platt-calibrated predictions of the SVM were used as input to a linear chain CRF. The CRF's predictions are also sequences of frame labels, but the CRF takes into account statistical dependencies between adjacent frames and smoothes the predictions in the time domain. The hyperparameters of the SVM and the CRF were optimized using SMAC. We evaluated on a 10% held out test set. The third label (absence of any call) is removed from the output for evaluation.

The system outputs call sequences, time-aligned with an audio file. We evaluate these transcriptions for the held-out test data. Considering the sequences of calls (not the alignment with the audio), we evaluate using word error rate (WER) and match error rate (MER), used in speech recognition (Morris et al., 2004). Results are in Table 3. The majority of calls are correctly identified. Most errors are deletions (missing calls) for Blue and Colobus monkeys and insertions (noise identified as calls) for Titis, perhaps because Titi calls are high frequency, similar to the noise.

To evaluate how well the predicted calls are time-aligned, we match each call in the gold transcription to the nearest predicted call whose onset and offset are within a 200 ms tolerance of the real onset and offset, and count a gold call as having a true positive only if it has such a match, and that match is correctly labeled; otherwise, it counts as a false negative. Similarly, for each predicted call, we look for the nearest such match among the calls in the gold transcription, and count a false positive if there is no match or if the match is mislabeled. Since it is likely easier to accurately mark the onsets of calls than their offsets, both for our human annotator and for the transcription system, we also compute an alternative scoring in which only call onsets need to be matched within the 200 ms tolerance. For both scorings, we compute precision, recall, and F₁, as shown in Table 4. The results show that call onsets are indeed much easier to match to the annotation than offsets, particularly for Colobus monkeys, where performance is relatively poor when offsets are required to be correctly marked.

General purpose acoustic features and voice activity detection techniques, as used in speech recognition, can automate the labeling of primate calls, both in isolation and in unannotated recordings, using data representative of field recordings. The system needs to be bootstrapped by a set of annotated examples. We showed that good isolated call labeling requires less than 200 labeled examples. It accurately transcribes around 90% of the frames in an audio file, vastly reducing the amount of manual work.

The results also imply that generic acoustic features, rather than specialized acoustic measurements taken manually by the researcher, can be used for detailed analysis. For example, there are competing descriptions of the call repertoires of certain species. Previous analyses have appealed to clustering analyses on hand-selected acoustic features as evidence (Fuller, 2014; Keenan et al., 2013). The results here validate an automated process of feature extraction that may be used as the input to these analyses. Both results allow much larger data sets from the field to be used than are currently being used for research and make it easier to create shared databases between researchers. Our tools can be downloaded at http://github.com/mwv/mcr.

This research received funding from the European Research Council under the EU's Seventh Framework Programme (FP/2007–2013), Grant Nos. 324115-FRONTSEM (Schlenker) and ERC-2011-AdG-295810 BOOTPHON (Dupoux), and from the Agence Nationale pour la Recherche ANR-10-IDEX-0001-02 PSL*, ANR-10-LABX-0087 IEC.