Abstract

Speaker Sequence Segmentation is the first step in many audio-processing applications and aims to solve the problem ”who spoke when”. It therefore relies on efficient use of temporal information from extracted audio features. In this project, we have utilised the Linear Predictive Coefficients of the speech signal and it’s derived features to segment out the speech of individual speakers. We have employed both supervised and unsupervised learning methods to approach the problem.

1. Introduction

1.3 Figure

2. Proposed Approach

The process of speech sequence segmenting comprises of the following steps:

1. Splitting audio into frames - The complete audio is divided into smaller time frames of 20ms each with a hop of 10ms. This is done because in smaller time scales audio signals are statistically unchanged.

2. Voice Activity Detection - Voice activity detection (VAD), also known as speech activity detection or speech detection, is a technique used in speech processing in which the presence or absence of human speech is detected. This was performed on each of the 20ms time frames and the time frames in which speech is not detected were discarded.

3. Feature Extraction - Linear Predictive Coefficients (LPCs) are widely used features in automatic speech sequence segmenting. The redundancy in the speech signal is exploited in the LP analysis. The prediction of current sample as a linear combination of past p samples form the basis of linear prediction analysis where p is the order of prediction. The predicted sample s^(n) can be represented as follows,
   Various LPC-derived features such as pitch, formants, Linear Predictive Cepstral Coefficients (LPCC), Line Spectral Frequencies (LSF), Pseudo-Cepstral Coefficients (PCC), Pseudo-Cepstrum (PCEP) were included in the feature vector extracted. LP analysis separates the given short term sequence of speech into its slowly varying vocal tract component represented by LP filter (H(z)) and fast varying excitation component given by the LP residual (e(n)). The LP filter (H(z)) induces the desired spectral shape for the shape on the flat spectrum (E(z)) of the noise like excitation sequence as given in equation (20). As the LP spectrum provides the vocal-tract characteristics, the vocal-tract resonances (formants) can be estimated from the LP spectrum. Various formant locations can be obtained by picking the peaks from the magnitude LP spectrum (|H(z)|). The extraction process of the LPCC features from the LPC coefficients is formulated in the z -transform domain, using the complex logarithm of the LPC system transfer function, which is analogous to the cepstrum computation from the discrete Fourier transform of the speech signal. The i-th LPCC parameter is given by the following recursive equation where ai is the i-th LPC parameter, p is the LPC system order and G is the gain factor of the system. The PCC is computed directly from the LSFs. However, its derivation is based on the LPCC. Mathematical manipulations and approximations allow it to be expressed in terms of the LSFs. The n-th PCC is given by the equation where wi is the i -th LSF parameter. The n-th PCEP expression is given by It is fair to expect a good spectral performance of the PCEP because they provide a spectral envelope very similar to the one provided by the Cepstrum, which is generated from the original speech signal.

   Feature Used	Number of components of each feature used to compose the feature vector
   Pitch	1
   Formants	2
   Linear Predictive Coefficients (LPCs)	20
   Linear Predictive Cepstral Coefficients (LPCC)	10
   Line Spectral Frequencies (LSF)	10
   Pseudo-Cepstral Coefficients (PCC)	10
   Pseudo-Cepstrum (PCEP)	10

   These features were normalised and cumulated to form a feature vector of length 63.

4. Speech Segmentation based on speakers - The speech segmentation was performed separately using the following two methods -

   1. Supervised Approach - The k-nearest neighbours (kNN) model was trained on a large numbers of voice samples to incorporate total variability in feature space. Hence, parameters of the model were set accordingly to classify individual speakers. This model was applied on a test data to cluster the 20ms time frames pertaining to individual speakers. For every 1 sec (50 time frames of 20ms each), the mode of the data was found and assigned to the whole 1 sec time frame. In order to eliminate the sporadic predictions, we have used a window function which passes through each time frame and computes the mode of elements left of the considered time-frame as well as the right side. If it does not match with the class predicted by the considered time-frame, it is assigned the mode computed. Hence, we were able to predict the transitions in the complete audio samples and segment out the speech of each individual speaker.

   2. Unsupervised Approach - The feature vectors pertaining to each time-frame were passed through stack of 4 sparse auto-encoders to carry out feature transfer learning. Gaussian Mixture Model was applied to cluster the output into 20 components. A graph was plotted depicting the predicted class at each time-frame. We could observe a change in distribution in the clusters at the speaker transition points. The time instances where such a change occurs across a significant number of clusters were recorded and marked as the transition points. This effectively predicts the transitions in the complete audio samples and segments out the speech of each individual speaker.

   

3. Experiments & Results

3.1 Dataset Description

Firstly, we acquired youtube videos of various lecturers from nptel channel. Later we extracted the audio signal from the videos at a sampling rate of 8000 Hz. In order to generate an audio signal comprising of variable number of speakers of varied durations we concatenated the audio signals in such a way as to create various experimental test cases.
Link: Audio Dataset

Supervised:

The probability that the model assigns the rightful class to a 20ms time-frame is shown in the plot below:

To view the transition point of speakers click here

To view the transition point of speakers click here

Unsupervised:

The features vectors of five speakers was passed through four stacked sparse autoencoders with ReLU as activation function. The output of the autoencoder was then modelled using Gaussian Mixture Model with no. of clusters set to 20. Decoded Value are the output of the sparse autoencoders.

Training of Sparse Autoencoder

Visualisation of the Autoencoder layer

Consecutive segments were taken of the order 1000 and total number of each type of clusters was calculated in them. The above graph plots the number of occurence of cluster X per 1000 segments. Each feature was then given a binary code depending on the relative occurrence of different gaussian model. The difference in bits of these binary code does not vary significantly for a particular speaker. On the contrary, the no of bit change is high when there is speaker transition. These binary codes can thus be used for detecting speaker transition as well as speaker identification.
1st Dataset: 5 unique speakers with each speaker having equal speaking time.

Visualization of Clusters generated by the GMM Model

Density of each Cluster

Speaker Identification

---

To view the transition point of speakers click here

2nd Dataset: 5 unique speakers with duration of each speaker increasing progressively.

Visualization of Clusters generated by the GMM Model

Density of each Cluster

Speaker Identification

3rd Dataset: 5 unique speakers with each speaker having equal speaking time and after half of the signal the sequence of speakers repeat.

Density of each Cluster

Speaker Identification

---

4th Dataset: 5 unique speakers with each speaker having variable duration.

Density of each Cluster

Speaker Identification

---

Code

4. Conclusion