Separation of overlapping sources in bioacoustic mixtures

The separation of sources in acoustic mixtures is an inevitable step in the study of many environments when recordings of isolated signals of interest are not available.^1–3 This problem has been studied in the context of many different applications, and proposed solutions typically depend on the relative information contained in sources^4,5 and mixtures.^6,7 Recovering individual sources in a complex and noisy environment (sometimes referred to as the “cocktail party problem”) has been an active area of research for decades and is still in progress.^8–12 Popular approaches in speech separation, singing-voice separation, and signal denoising mainly suffer from underlying assumptions that make them inapplicable to mixtures of other sounds, such as bioacoustic recordings. For example, it is often assumed that the number of sources is known and fixed and/or sources are independent.^2,13,14 When the number of sources is more than the number of recording channels (underdetermined system), sources are similar, the number of sources is unknown, or sources are highly overlapped in time and/or frequency, such as in the case of many bioacoustic recordings, common methods such as independent component analysis (ICA),¹⁵ nonnegative matrix factorization (NMF),¹⁶ or deep clustering (DC)¹⁷ are not applicable. Solutions based on NMF aim to represent the additive linear interpolation of a known number of sources through a basis (or template) matrix. ICA methods are developed for the determined system and rely on the assumption of the independence of sources. The DC methods train a neural network to generate discriminative embeddings for time-frequency bins where target sources are complementary masked spectrograms of the mixture. Mapping-based methods, unlike masking-based ones, look for complimentary masks in a continuous range, but these approaches are designed for dereverberation and denoising, not separation of multiple sources. The specific problem of detecting sound events may vary based on the amount of information in the input and the level of details in the output, but typical approaches are not designed to estimate the number of sources or to discriminate overlapped similar sources. Furthermore, masking-based methods with complementary outputs cannot fully represent overlapping sources because each time-frequency bin is assigned to only one source. Estimating the number of sources is a front-end processing step while estimating the separation function is usually defined as a back-end learning process, with the design of the separation function relying on a fixed number of sources.

This paper describes an approach to separate the bioacoustic signals of an unknown number of overlapping sources. Specifically, mixtures of interest are recordings of many echolocating signals collected within swarms of bats and target representation of sources are single component connected binary regions in the time-frequency domain. This technical problem is motivated by our study of the behavior of individual bats in a swarm. Some bat species form large groups in a cave and emerge from the cave in a dense swarm each evening.¹⁸ The primary sensory modality of these bats is echolocation, in which individuals make frequency-modulated ultrasonic signals.^19–21 When flying together in small groups, bats may adjust their echolocation to avoid interference by switching between active and passive mode, adjusting the time-frequency characteristics of signals, or spending substantial time in silence.^22–24 It is currently unknown, however, how bats in extreme dense swarms, such as during cave emergence, avoid the problem of mutual interference or sonar jamming.^24–29 Pilot data indicate individual bats may produce signals with discrete time-frequency structure,³⁰ but this has not been verified from recordings of dense swarms due to the challenge of overlapping echolocation signals. Two spectrograms of these overlapping signals are depicted in Fig. 1, where Fig. 1(a) is a simulated mixture by adding ten single call recordings and Fig. 1(b) is a real recording within a swarm of approximately 30 echolocating bats. Understanding how individual bats can adapt and use their sonar signaling techniques to navigate in complex environments is the primary incentive of this work. To gain this understanding, complex acoustic mixtures need to be represented in terms of individual sources.^31–34 In this work, individual sources are defined to be single-harmonic bat calls. One can extract bat-wise signals by sequentially associating predicted calls to a specific bat. However, performing the post-processing step of the so-called data association is out of the scope of this work.

The approach presented in this paper is built on ideas that have been used to address the segmentation problem found in computer vision systems. Instance segmentation is the task of assigning every pixel into a category and discriminating between individual object instances that are spatially large and sparse. A common approach to instance segmentation is to split the task into two separate problems: First, detecting all areas that potentially contain an object, known as region proposals; then, second, running a mask generator on each proposal for final segmentation. In segmentation by detection methods, once an object is detected, segmentation can be relatively easy using characteristics of the object. Most of the instance segmentation methods such as mask regional-convolutional neural networks (Mask R-CNN)³⁵ either show poor performance on overlapping objects due to the local non-maximum suppression or stack multiple computationally expensive neural networks.

In this paper, a deep neural network (DNN) with DenseUNet architecture is proposed to model the separation function where the neural network operates to perform in two steps. First, a detection step detects signal measurements that are large enough to allow them to be segmented from the remaining signal content. Then, by reducing the observation to just a few target candidates, the problem reduces to mapping the most probable candidates to the representation of individual sources. The model performs in the time-frequency representation of mixtures. Outputs of the detection and segmentation steps are bounding boxes and binary mappings of individual sources, respectively. Time-domain predicted sources can be achieved from an inverse time-frequency transformation. The performance of the proposed method is evaluated in the time-frequency domain through common metrics including F₁-score and false-negative rate (FNR).

This paper is organized as follows. In Sec. II, the proposed model is described. In Sec. III, the detailed description of experiments is provided and results are evaluated. And finally, a summary of this work and conclusions are presented in Sec. IV.

Let $x∈ℝT$ denote the mixture of S sources $y=[ys]s=1S∈ℝS×T$ where $ys∈ℝT$ and T is the length of the recording. The source separation problem is to estimate the separation function $ĝ$ that minimizes a distance $D(y,ŷ)$ where $ŷ=ĝ(x)$ and $Ŝ$ are predicted sources and their quantity, respectively. $ŷ∈ℝŜ×T$⁠.

This section first reviews different assumptions on the mixing process. The second part then explains the importance of signal representation and provides a transformation option. The model section proposes a DNN which models the separation function g and presents the process of estimating parameters—training—and an algorithm for preparing ground-truth sources used in the training.

Source separation is considered as an inverse problem, and the first assumption underlying most of the proposed solutions is the mixing function $ℝS×T→ℝT$ that represents the environment and process of mixing sources. The separation function g is one of the (possibly many) inverse functions of the mixing function and should reflect the properties of the mixing environment.

A linear and time-invariant system which adds up sources in an instantaneous system does not fully express properties of non-stationary environments in which mixing variables are time-dependent. Furthermore, in reverberant environments, reflections of sources persist after they are produced and the mixing process should be in a convolutive manner to highlight the occurrence of reflections. However, an adequately large enough environment is usually approximated with a linear system. In this paper, the mixing function is assumed to be $x=Ay+b$⁠, where $A∈ℝS$ and $b∈ℝT$ capture the relative power of sources and the environment noise (including non-source signals), respectively.

Source separation is often performed in the time-frequency domain. Signal transformations such as Fourier analysis are not only used as feature extractors that improve the separation performance but also serve as a visualization scheme that provides insight into the analysis of the signal. An appropriate signal transformation is reversible and projects time-domain samples to an interpretable space that highlights the contrast between different sources and similarities between identical sources. Depending on the application, some transformations have shown to be more appropriate. Time-frequency representations like the short-time Fourier transform (STFT) are among the most common transformations used in source separation.

Let $T:ℝT→ℝF×N$ be a two-dimensional (2 D) transformation that projects T time-domain samples into N features of dimension F. Parameters of the transformation including sampling frequency, window size, and hop size play an important role in the resolution of time-frequency space (⁠$F×N$⁠) such that increasing F reduces N and vice versa. Due to this inherent trade-off of short-time analysis, various joint distributions have been proposed to control the interplay between time and frequency by adding more parameters to the transformation $T$⁠.³⁶ Furthermore, useful information in separating sources is primarily embodied in the signal energy and not in its phase.³⁷ Therefore, most separation methods rely on energy-based representations known as spectrograms. In some cases, applying other transformations such as logarithmic and Mel scales and recently proposed per-channel energy normalization (PCEN)³⁸ on spectrograms can help with the suppression of non-source components. The log and PCEN compression reduce the dynamic range of Mel-bank energy so that scales down a large range of low-level signals, such as silence and small noise (non-source). Since a log function is loudness dependent and devotes a large range to weak signals, PCEN has shown to be more effective in suppressing non-stationary noise.

Using the spectrogram of a mixture $X=T(x)$ and sources $Ys=T(ys)$ for $s∈{1,…,S}$⁠, the separation function $ĝ$ extracts $Ŝ$ sources $[Ŷs]s=1Ŝ=ĝ(X)$ and the source separation problem is to estimate a function $ĝ$ such that the distance

is minimized where S ground-truth and $Ŝ$ predicted sources are noted by Y_s and $Ŷŝ$⁠, respectively. For an F × N-dimensional A, the norm $‖A‖22$ is the average over norms of A element, i.e., $‖A‖=∑iF∑jN‖aij‖/FN$⁠. In a $S×Ŝ$ matrix of distances $‖Ys−Ŷŝ‖22$ for $s=1,…,S$ and $ŝ=1,…,Ŝ$⁠, the first term of averages over minimums of rows, while the second term overages over the minimums of columns. Considering unique predictions, Eq. (1) is minimized if predictions are a permutation of S ground-truth sources. A missing prediction increases the first term of Eq. (1), while the second term quantifies extra predictions.

Instead of directly looking for the source representations within an F × N-dimensional real-valued space, it can be useful to reduce the range of the outputs to the binary-valued $F×N$-dimensional space.³⁷ The time-frequency masking approach was proposed to identify sources with the largest amplitude in each individual time-frequency bin. The ideal binary mask (IBM),³⁹ inspired by the auditory masking phenomenon, is defined as $M=1ω(Y)$⁠, where $1ω(·)$ is the element-wise indicator function with an amplitude threshold of ω. Let M_s be the mask of the sth source for $s∈{1,…,S}$⁠. Having a mask M_s and the phase of the mixture, one can reconstruct the time-domain source s as $ys=T−1(Ms⊙X)$ where $⊙$ denotes the element-wise multiplication.

Conventional separating functions are dependent on the number of sources (or an estimation of the number of sources when it is unknown) and are typically composed of source-independent and source-dependent parts. The source-independent part projects the mixture to a latent space in which sources are easily separable. This part aims to maximize the distance between samples from different sources and the similarity between identical sources. Finally, for each source, a source-dependent function projects the latent vector onto the corresponding hyperplane of the source. In a separation function modeled with a DNN, the last layer with S-channels operates as the source-dependent part. However, when the number of sources is unknown and variable, the last layer needs to operate adaptively.

To model a separation function that handles a variable number of overlapping sources, three intermediate variables are defined to capture the location of sources and their bounding boxes in the spectrogram. The first variable, C, determines the center of sources, which consequently indicates the number of sources. The other two variables, H and W, are responsible for the extent of sources, i.e., the heights and widths of sources in the spectrogram. Therefore, the separation function operates in two steps of the detection and segmentation. During the detection step, the center of all sources and their corresponding bounding boxes are predicted. The segmentation step aims to extract the mask of each detected source in the first step from its corresponding resized version. For a mixture of S sources, the separation function ideally operates in S + 1 steps, where S center and bounding boxes of sources are detected in the first step and S masks M_s for $s∈{1,…,S}$ is predicted in the next S steps of segmentation. An example of inputs and outputs of the separation function during the training and the test are depicted in Fig. 2. In the detection step, Fig. 2(a), the resized version is not available and the first three output variables are of interest. Only the resized version is involved in the segmentation step, Fig. 2(b), that extracts the corresponding mask. The first and last outputs are binary variables, while the other two have positive values indicating heights and widths of all sources. During the training, Fig. 2(c), the function is forced to map the mixture X and a resized version Z_s to four aforementioned variables of C, H, W, and M_s. Figure 3 shows the results of the proposed method for two steps of detection and segmentation. The input which is a mixture of an unknown number of bat echolocation calls is depicted in Fig. 3(a).

The detection step (equivalent to the source-independent part) manages the variable number of sources. Any estimator of the number of sources captures an underlying structure in which sources are different. When sources are characterized based on their position (temporal and spectral) and extent, one can estimate the number of sources by detecting the position of sources in the spectrogram. Therefore, a mapping from the mixture X to $C∈[0,1]F×N$ space could be interpreted as the probability of the presence of a source in each time-frequency bin, i.e., $C(f,n)=1$ indicates the presence of a source centering at (f, n) in the spectrogram. Accordingly, this variable, C, not only provides the information about the number of sources since $Σf,nC(f,n)=S$⁠, but also specifies the position of sources in the spectrogram. When the function is modeled with a neural network, to ensure the binary range of the C, the last layer is followed by a mapping function to $[0,1]$⁠, like the Sigmoid function, and an indicator function.

Since knowing the position of centers of sources is not enough to uniquely identify sources, the extent of sources along the time and frequency axes needs to be predicted. To record the extent of sources, two variables, H and W $∈ℕF×N$⁠, are assigned to describe the height and width of sources. For the source $s∈{1,…,S}$ centering at the time-frequency bin (f, n), i.e., $C(f,n)=1$⁠, having frequency range h_s and duration w_s, two time-frequency bins H(f, n) and W(f, n) are assigned accordingly, i.e., $H(f,n)=hs$ and $W(f,n)=ws$⁠. Two mappings from the mixture X to H and W $∈ℝ+F×N$ model the extent of sources where $ℝ+$ denotes the set of non-negative real numbers. Three variables of C, H, and W are equivalent to S bounding boxes on the spectrogram.

Having the position and extent of sources only specifies rectangular areas of the spectrogram, not their detailed bin-level masks, see Fig. 3(b). Thus, another step is needed to fully segment sources in the time-frequency domain. The segmentation step that extracts masks of individual sources from a mixture is conditioned on results of the detection step, i.e., $X;C,H,W,s→Ms$ for $s∈{1,…,Ŝ}$⁠, and should be independent of the position and extent of sources. It is trivial to show the mapping from X to M_s given $C,H,W,s$ is equivalent to the mapping from Z_s to M_s where Z_s is the sth resized version of the mixture. One can get the sth resized version of the mixture X by first cropping the area of the X that contains the source Y_s and then resizing the rectangular spectrogram of size $hs×ws$ to F × N. The sth rectangular region of X is the smallest rectangular window that $Ms(f,n)$ is not zero, i.e., $∀f,n:arg minf,nMs≤f,n≤arg maxf,nMs$⁠. Without this resizing operation, the segmentation part needs to learn the already available position and extent of the source unnecessarily. This operation takes advantage of having similar sources that only differ in the position and size in the spectrogram. The result of the segmentation step is shown in Fig. 3(c). In the case of using a neural network, a post-processing step is needed to reverse the resizing process in the following way: First, the output of the last layer is resized from F × N to $hs×ws$⁠, and then, the spectrogram of size $hs×ws$ is padded with zeros so that it is centered at the location of the sth source. Also, since the desired output M_s is binary, the last layer of the network is followed by a mapping function to $[0,1]$⁠.

There are two approaches to implement the detection and segmentation steps in a supervised fashion. One can search for two separate functions using ground-truth outputs of $[C,H,W]$ and $[Ms]s=1S$ and then use the detection function to predict the position and extent of $Ŝ$ sources and run the segmentation function $Ŝ$ times to reconstruct individual sources. The approach in this paper is to find a single function that projects a mixture X to $Ŝ$ masks M_s for $s∈{1,…,S}$ operating in $Ŝ+1$ steps: one step of detection, $X→[C,H,W]$⁠, and $Ŝ$ steps of segmentation, $Zs→Ms$⁠. This function is presented as $g:[X,Zs]→[C,H,W,Ms]$ where Z_s is the resized version of sth source in X and M_s is its corresponding mask. g operates in $1+Ŝ$ steps: $g:[X,·]→[C,H,W,·]$ in the first step, and $g:[·,Zs]→[·,·,·,Ms]$ in the next $Ŝ$ steps. In the following, first, a recipe for training this function is provided and then its usage in the evaluation and test time is explained. Finally, a DNN is suggested for implementing this function.

Training or estimating the parameters of the function in a supervised manner is an optimization process in which a loss function between predictions and ground-truth values is minimized. Given the input of $[X,Zs]$⁠, ground-truth labels are $[C,H,W,Ms]$ or equivalently, a mixture X has four labels of C, H, W, and $M=[Ms]s=1S$ indicating bounding boxes (the first three) and masks of all sources, respectively. It is worth noting that given the first three labels, it is trivial to find S resized versions of X as Z_s for $s∈{1,…,S}$⁠. Also, the input and output of the separation function have 2 and 4 channels, respectively. During the training, as shown in Fig. 2, the function inputs $[X,Zs]$ for a random s from ${1,…,S}$ and outputs a prediction of $[C,H,W,Ms]$⁠. Parameters of g are estimated by minimizing the loss function of

which measures the distance between the prediction $Ŷ$ and the ground-truth Y where parameters $λ={λ1,λ2,λ3}$ are regularization coefficients. The first and the last terms of Eq. (2) are binary cross-entropy between ground-truth binary variables and their predictions, whereas the other term contains mean squared errors between ground-truth heights and widths of bounding boxes and their predictions. Since $Ĥ$ and $Ŵ$ are positive and real-valued, mean squared error is used to measure the model performance. However, cross-entropy has shown to be a more effective distance on categorical predictions like binary values.

During the evaluation or test time, the separation function operates $Ŝ+1$ times and in two steps of detection and segmentation. Since there is no resized version of the mixture during the detection step of test and evaluation, let the mixture accompanied by itself, i.e., $[X,·]$ be the initial input to the network where the second variable is not available. The output of the network during the detection step is noted by $[Ĉ,Ĥ,Ŵ,·]$⁠, where the last dimension of the output is not useful. To prepare resized versions for the segmentation step, a couple of post-processings are employed on the $Ŷ$⁠. Since $Ĉ∈[0,1]$⁠, a thresholding operation is applied to the first dimension of the output, i.e., $Ŝ=Σf,n1ϕ(Ĉ(f,n))$⁠, where $ϕ$ is the detection threshold. This operation chooses $Ŝ$ bins with high chances of being centers of sources and excludes the rest of $FN−Ŝ$ bins at $Ĉ, Ĥ$⁠, and $Ŵ$⁠. Therefore, the output of the detection step could be represented as $[1ϕ(Ĉ),1ϕ(Ĉ)⊙Ĥ,1ϕ(Ĉ)⊙Ŵ,·]$⁠. By having the first three dimensions of the output $Ŷ$⁠, one can form $Ŝ$ inputs of the form $[·,Ẑs]$ and get $Ŝ$ outputs of $[·,·,·,M̂s]$ for $s∈{1,…Ŝ}$⁠. In a similar manner as the prediction of centers of sources, the final output is achieved by $Ŷ=[·,·,·,1ψ(M̂s)]$⁠, where ψ is the segmentation threshold. Since threshold-based predictions are cluttered with non-source segments, as a post-processing step, after filling holes of the predicted binary mask, the largest segment is extracted as the final mask.

An auto-encoder convolutional neural network (CNN) from the UNet family of architectures is used to model the separation function. The UNet architecture, initially proposed for segmentation in medical images, has shown to be successful in the encoding and decoding of spacial features. The encoding part represents images in a low-dimensional space, known as latent space, and the decoding part reconstructs outputs of interest using features in the latent space.⁴⁰ In this paper, a neural network with the DenseUNet architecture⁴¹ (a slightly modified UNet) is used to model the separation function that detects and segments individual sources in the mixture. The DenseUNet is an auto-encoder neural network with the path alternating downsample/upsample units in encoding/decoding parts respectively and blocks of bottleneck layers. Downsample layers consist of a convolution followed by an average pooling and upsample layers are transposed convolutions. Bottleneck layers in both encoding and decoding paths include two convolution operations and the output of each layer is dropped out with a small probability. A batch normalization followed by a rectified linear unit (ReLU) is applied to the inputs of both convolution and transposed convolution. Parameters of this function (weights of convolution, transposed convolution, and batch normalization) are estimated through the training process.

In order to minimize the loss function of Eq. (2), input mixtures X and their corresponding ground-truth labels $[Ms]s=1S$ are required. Regarding the input mixture x, a common approach in the development of source separation algorithms is to mix S individual sources $[xs1]s=1S$ and estimate a function that predicts binary masks of individual sources based on their dominance in the spectrogram of X as the transformation of x. To make a mixture of S sources, x, one can mix S individual sources $[xs1]s=1S$ using

for signal-to-noise ratios (SNRs) of $(αs)s=2S$ where $x0=0$⁠. Therefore, a mixture X is labeled with $[Ms]s=1S$⁠, where the mask of $Xs1$ is noted by M_s.

For ground-truth labels M, since manually labeling bioacoustic signals in the time-frequency domain is expensive, the development of a simple and fast labeling algorithm is needed. Given a recording x¹ with an assumption of containing non-overlapping calls and its transformation X¹, the single-channel binary mask M is extracted using the iterative thresholding Algorithm 1 for the maximum of U iterations. ω is the amplitude threshold, and $⟨M⟩$ is the number of connected regions in the mask M.

Experiments of the proposed method are conducted on real and artificial datasets, and the performance is evaluated in terms of common metrics used in binary decision problems.

The artificial dataset is made by mixing single source recordings that are pulled out of recordings of bat echolocation calls sampled at 256 kHz using Algorithm 1. The time-frequency representation is drawn using PCEN³⁸ spectral magnitude of a Mel scaled spectrogram with F = 128 bands between 20 and 60 kHz. Spectrograms are obtained using a 1024-point short-time Fourier transform (STFT) with window and hop size of 256 and 32 samples, respectively. By framing time-domain recordings in segments of length T = 32 ms, inputs to Algorithm 1 are N = 256 features of dimension F = 128 and U = 5. Having three field recordings with a total duration of one hour, around 2700 bat echolocation calls are pulled out and divided into five folds of sources used in making development and test sets of mixtures. For each set of individual sources, a maximum of $S∈{5,10}$ sources is chosen and mixed at randomly selected SNRs from $[−5,5]$ (see Table I). The number of mixtures made out of each set of sources is proportional to the number of sources and a total number of 100–5000 mixtures are generated for each fold (see Table II). Figure 4 depicts a sample of artificial dataset, i.e., a mixture of ten echolocation calls and its corresponding label, ten masks.

The real dataset contains 13 recordings (with the total duration of around 4 s) within swarms of up to 300 echolocating bats. Recordings are framed into overlapping segments so that some sources appear in more than one segment. Using a DenseUNet of $2/4/8/4$ bottleneck layers and trained with mixtures of maximum ten sources and regularization coefficients of $0.1/0.1/0.5$⁠, more than 900 sources are extracted. Figure 3(a) shows a spectrogram of a 32 ms frame from this dataset, where Fig. 3(b) depicts outputs of the detection step and extracted masks are presented in Fig. 3(c).

The DenseUNet has four dense blocks of bottleneck layers in both encoding and decoding paths (see Table III). Each downsample layer is composed of a 3 × 3 convolution with the stride of 1 and a 2 × 2 average pooling with the stride of 2, while upsample layers have 2 × 2 transposed convolutions with a stride of 2. In bottleneck layers, 1 × 1 and 3 × 3 convolutions with strides of 1 are followed by a dropout with p = 0.1. The number of channels in the DenseUNet repeatedly increases and decreases with growth and reduction rates of 16 and 0.5, respectively. To estimate parameters of this network, the development set is split into two validation and training sets with the proportion of 1/4, and the Adam algorithm⁴² is used to minimize the loss function of Eq. (2) over the training set where its learning rate decays linearly starting from 10⁴ ending at 10⁵ (see Table IV). The total number of epochs is set to 100 with an early stop rule based on no change in the value of the loss function for the validation set for ten epochs.

The performance of the proposed method is evaluated in the time-frequency domain. Without any extra post-processing decision steps, all predicted binary masks and their corresponding bounding boxes are examined in contrast with the ground-truth ones. Since the proposed model operates in two separate steps and the output of the first step affects the final prediction, the results are evaluated for both detection and segmentation steps. Also, for each step, the performance is measured in two levels of sources and time-frequency bins. A source-level metric compares the number of detected or segmented sources with ground-truth ones and considers a prediction as a correct output if its intersection over union (IOU) between the ground-truth is larger than 0.7. Within correctly predicted sources, the time-frequency bins level metric similarly compares time-frequency bins of the spectrogram in the prediction and ground-truth. Each metric is presented for both detection and segmentation steps and source-level and bin-level metrics are provided separately. The results of different experiments on the artificial dataset are reported in terms of average false negative rate (FNR), event error rate (EER) (when appropriate), and F₁ score (F) over five test folds for detection and segmentation thresholds of $ϕ=0.1$ and $ψ=0.5$⁠, respectively. All the threshold values are chosen based on the FNR. The thresholding Algorithm 1 is not accurate enough to capture all sources in a recording, so F₁ score does not reflect the real performance of the proposed method compared to FNR. Also, F₁ score takes into account false positive predictions, which is not the primary concern compared to FNR that represents the ratio of missed sources and time-frequency bins.

The results of the experiment exploring the complexity of the training dataset are shown in Table I. Two datasets composed of mixtures of maximum five and ten sources are used for training and each is evaluated against test sets of mixtures of maximum five and ten sources. As Table I shows, testing mixtures of maximum ten sources on a function trained on mixtures of maximum five sources results in the worst FNR. Therefore, training on mixtures of more sources is necessary for detecting highly overlapped sources because the neural network learns to recognize the source of interest in the presence of other sources if the cropped version is more complicated. However, since frame size and duration of sources and overlap of sources are inversely related, training on mixtures of more than 20 sources does not provide a better model. Table II shows the performance for various sizes of the training dataset. The larger the training dataset, the faster the convergence rate. However, since all mixtures are generated similarly, the model reaches its maximum capacity after 2000 mixtures. To examine the size of the DNN, which is mainly defined by the number of encoding layers (set to 4) and bottlenecks in each layer, training is performed on a different combination of bottleneck layers. Increasing layers constantly improves the FNR up to a total of around 20 layers and $2/4/6/8$ shows the best performance (Table III). Other orders like $8/6/4/2$ have decreased the FNR substantially. Furthermore, minor changes in the architecture such as $2/5/5/8$ do not yield considerable enhancement. Since during the training, the first three variables of C, H, and W are repeated proportional to the number of sources, weighting λ₃ should result in better performance. Table IV shows the effects of the various regularization coefficients. FNR decreases constantly when λ₁ and λ₂ values are larger than λ₃. However, training with regularization coefficients of $1/1/10$ results in better FNRs in the source level, but the last variable of the output overfits to the average mask so the shape of sources is not preserved well.

This work proposes a novel source separation algorithm for extracting an unknown number of overlapped bat echolocation calls. The suggested method operates in two steps of detection and segmentation, where the first one finds bounding boxes potentially containing a source and the second one extracts the final masks of sources. Experiments on real mixtures of bat echolocation calls demonstrate that extracted segments maintain essential features of the original shape. Different components of the proposed methods such as DNN architectures, loss function, and size of the training dataset are founded thorough a numerical search over a wide range of parameters. This method performs the best on signals with high SNR and low-to-moderate levels of overlap. The primary motivation for this algorithm was the separation of bat echolocation calls, but this method can easily be applied and/or extended to other bioacoustic recordings with overlapping calls. We developed this method for the separation of sources without harmonics, but an association function could be applied to merge harmonics of identical sources at the end. Future directions can include modifications to the approach to separate more complex overlapping bioacoustic signals, such as those from dolphins, birds, or frogs vocalizing in large groups.

This project was funded by an Office of Naval Research Young Investigator Award (N00014-16-1-2478) and ONR N00014-18-1-2522, both awarded to L.N.K. The authors would like to thank Nicole Blaha, Morgan Kinniry, Kathryn McGowan, Allison Pudlo, and Lilias Zusi for assistance with data collection and extraction.