-
Sparks of Large Audio Models: A Survey and Outlook
Authors:
Siddique Latif,
Moazzam Shoukat,
Fahad Shamshad,
Muhammad Usama,
Yi Ren,
Heriberto Cuayáhuitl,
Wenwu Wang,
Xulong Zhang,
Roberto Togneri,
Erik Cambria,
Björn W. Schuller
Abstract:
This survey paper provides a comprehensive overview of the recent advancements and challenges in applying large language models to the field of audio signal processing. Audio processing, with its diverse signal representations and a wide range of sources--from human voices to musical instruments and environmental sounds--poses challenges distinct from those found in traditional Natural Language Pr…
▽ More
This survey paper provides a comprehensive overview of the recent advancements and challenges in applying large language models to the field of audio signal processing. Audio processing, with its diverse signal representations and a wide range of sources--from human voices to musical instruments and environmental sounds--poses challenges distinct from those found in traditional Natural Language Processing scenarios. Nevertheless, \textit{Large Audio Models}, epitomized by transformer-based architectures, have shown marked efficacy in this sphere. By leveraging massive amount of data, these models have demonstrated prowess in a variety of audio tasks, spanning from Automatic Speech Recognition and Text-To-Speech to Music Generation, among others. Notably, recently these Foundational Audio Models, like SeamlessM4T, have started showing abilities to act as universal translators, supporting multiple speech tasks for up to 100 languages without any reliance on separate task-specific systems. This paper presents an in-depth analysis of state-of-the-art methodologies regarding \textit{Foundational Large Audio Models}, their performance benchmarks, and their applicability to real-world scenarios. We also highlight current limitations and provide insights into potential future research directions in the realm of \textit{Large Audio Models} with the intent to spark further discussion, thereby fostering innovation in the next generation of audio-processing systems. Furthermore, to cope with the rapid development in this area, we will consistently update the relevant repository with relevant recent articles and their open-source implementations at https://github.com/EmulationAI/awesome-large-audio-models.
△ Less
Submitted 21 September, 2023; v1 submitted 24 August, 2023;
originally announced August 2023.
-
A New Approach to Extract Fetal Electrocardiogram Using Affine Combination of Adaptive Filters
Authors:
Yu Xuan,
Xiangyu Zhang,
Shuyue Stella Li,
Zihan Shen,
Xin Xie,
Leibny Paola Garcia,
Roberto Togneri
Abstract:
The detection of abnormal fetal heartbeats during pregnancy is important for monitoring the health conditions of the fetus. While adult ECG has made several advances in modern medicine, noninvasive fetal electrocardiography (FECG) remains a great challenge. In this paper, we introduce a new method based on affine combinations of adaptive filters to extract FECG signals. The affine combination of m…
▽ More
The detection of abnormal fetal heartbeats during pregnancy is important for monitoring the health conditions of the fetus. While adult ECG has made several advances in modern medicine, noninvasive fetal electrocardiography (FECG) remains a great challenge. In this paper, we introduce a new method based on affine combinations of adaptive filters to extract FECG signals. The affine combination of multiple filters is able to precisely fit the reference signal, and thus obtain more accurate FECGs. We proposed a method to combine the Least Mean Square (LMS) and Recursive Least Squares (RLS) filters. Our approach found that the Combined Recursive Least Squares (CRLS) filter achieves the best performance among all proposed combinations. In addition, we found that CRLS is more advantageous in extracting FECG from abdominal electrocardiograms (AECG) with a small signal-to-noise ratio (SNR). Compared with the state-of-the-art MSF-ANC method, CRLS shows improved performance. The sensitivity, accuracy, and F1 scores improved by 3.58%, 2.39%, and 1.36%, respectively.
△ Less
Submitted 26 February, 2023; v1 submitted 20 October, 2022;
originally announced October 2022.
-
Automated Sex Classification of Children's Voices and Changes in Differentiating Factors with Age
Authors:
Fuling Chen,
Roberto Togneri,
Murray Maybery,
Diana Weiting Tan
Abstract:
Sex classification of children's voices allows for an investigation of the development of secondary sex characteristics which has been a key interest in the field of speech analysis. This research investigated a broad range of acoustic features from scripted and spontaneous speech and applied a hierarchical clustering-based machine learning model to distinguish the sex of children aged between 5 a…
▽ More
Sex classification of children's voices allows for an investigation of the development of secondary sex characteristics which has been a key interest in the field of speech analysis. This research investigated a broad range of acoustic features from scripted and spontaneous speech and applied a hierarchical clustering-based machine learning model to distinguish the sex of children aged between 5 and 15 years. We proposed an optimal feature set and our modelling achieved an average F1 score (the harmonic mean of the precision and recall) of 0.84 across all ages. Our results suggest that the sex classification is generally more accurate when a model is developed for each year group rather than for children in 4-year age bands, with classification accuracy being better for older age groups. We found that spontaneous speech could provide more helpful cues in sex classification than scripted speech, especially for children younger than 7 years. For younger age groups, a broad range of acoustic factors contributed evenly to sex classification, while for older age groups, F0-related acoustic factors were found to be the most critical predictors generally. Other important acoustic factors for older age groups include vocal tract length estimators, spectral flux, loudness and unvoiced features.
△ Less
Submitted 26 September, 2022;
originally announced September 2022.
-
End-to-End Lyrics Recognition with Self-supervised Learning
Authors:
Xiangyu Zhang,
Shuyue Stella Li,
Zhanhong He,
Roberto Togneri,
Leibny Paola Garcia
Abstract:
Lyrics recognition is an important task in music processing. Despite traditional algorithms such as the hybrid HMM- TDNN model achieving good performance, studies on applying end-to-end models and self-supervised learning (SSL) are limited. In this paper, we first establish an end-to-end baseline for lyrics recognition and then explore the performance of SSL models on lyrics recognition task. We e…
▽ More
Lyrics recognition is an important task in music processing. Despite traditional algorithms such as the hybrid HMM- TDNN model achieving good performance, studies on applying end-to-end models and self-supervised learning (SSL) are limited. In this paper, we first establish an end-to-end baseline for lyrics recognition and then explore the performance of SSL models on lyrics recognition task. We evaluate a variety of upstream SSL models with different training methods (masked reconstruction, masked prediction, autoregressive reconstruction, and contrastive learning). Our end-to-end self-supervised models, evaluated on the DAMP music dataset, outperform the previous state-of-the-art (SOTA) system by 5.23% for the dev set and 2.4% for the test set even without a language model trained by a large corpus. Moreover, we investigate the effect of background music on the performance of self-supervised learning models and conclude that the SSL models cannot extract features efficiently in the presence of background music. Finally, we study the out-of-domain generalization ability of the SSL features considering that those models were not trained on music datasets.
△ Less
Submitted 26 October, 2022; v1 submitted 26 September, 2022;
originally announced September 2022.
-
Voice Gender Scoring and Independent Acoustic Characterization of Perceived Masculinity and Femininity
Authors:
Fuling Chen,
Roberto Togneri,
Murray Maybery,
Diana Tan
Abstract:
Previous research has found that voices can provide reliable information to be used for gender classification with a high level of accuracy. In social psychology, perceived masculinity and femininity (masculinity and femininity rated by humans) has often been considered an important feature when investigating the influence of vocal features on social behaviours. While previous studies have charact…
▽ More
Previous research has found that voices can provide reliable information to be used for gender classification with a high level of accuracy. In social psychology, perceived masculinity and femininity (masculinity and femininity rated by humans) has often been considered an important feature when investigating the influence of vocal features on social behaviours. While previous studies have characterised the acoustic features that contributed to perceivers' judgements of speakers' masculinity or femininity, there is limited research on developing a machine masculinity/femininity scoring model and characterizing the independent acoustic factors that contribute to perceivers' masculinity and femininity judgements. In this work, we first propose a machine scoring model of perceived masculinity/femininity based on the Extreme Random Forest and then characterize the independent and meaningful acoustic factors that contribute to perceivers' judgements by using a correlation matrix based hierarchical clustering method. Our results show that the machine ratings of masculinity and femininity strongly correlated with the human ratings of masculinity and femininity when we used an optimal speech duration of 7 seconds, with a correlation coefficient of up to .63 for females and .77 for males. Nine independent clusters of acoustic measures were generated from our modelling of femininity judgements for female voices and eight clusters were found for masculinity judgements for male voices. The results revealed that, for both genders, the F0 mean is the most important acoustic measure affecting the judgement of acoustic-related masculinity and femininity. The F3 mean, F4 mean and VTL estimators were found to be highly inter-correlated and appeared in the same cluster, forming the second most significant factor in influencing the assessment of acoustic-related masculinity and femininity.
△ Less
Submitted 4 August, 2022; v1 submitted 16 February, 2021;
originally announced February 2021.
-
Multi-task Learning Based Spoofing-Robust Automatic Speaker Verification System
Authors:
Yuanjun Zhao,
Roberto Togneri,
Victor Sreeram
Abstract:
Spoofing attacks posed by generating artificial speech can severely degrade the performance of a speaker verification system. Recently, many anti-spoofing countermeasures have been proposed for detecting varying types of attacks from synthetic speech to replay presentations. While there are numerous effective defenses reported on standalone anti-spoofing solutions, the integration for speaker veri…
▽ More
Spoofing attacks posed by generating artificial speech can severely degrade the performance of a speaker verification system. Recently, many anti-spoofing countermeasures have been proposed for detecting varying types of attacks from synthetic speech to replay presentations. While there are numerous effective defenses reported on standalone anti-spoofing solutions, the integration for speaker verification and spoofing detection systems has obvious benefits. In this paper, we propose a spoofing-robust automatic speaker verification (SR-ASV) system for diverse attacks based on a multi-task learning architecture. This deep learning based model is jointly trained with time-frequency representations from utterances to provide recognition decisions for both tasks simultaneously. Compared with other state-of-the-art systems on the ASVspoof 2017 and 2019 corpora, a substantial improvement of the combined system under different spoofing conditions can be obtained.
△ Less
Submitted 5 December, 2020;
originally announced December 2020.
-
Enhanced ${q}$-Least Mean Square
Authors:
Shujaat Khan,
Alishba Sadiq,
Imran Naseem,
Roberto Togneri,
Mohammed Bennamoun
Abstract:
In this work, a new class of stochastic gradient algorithm is developed based on $q$-calculus. Unlike the existing $q$-LMS algorithm, the proposed approach fully utilizes the concept of $q$-calculus by incorporating time-varying $q$ parameter. The proposed enhanced $q$-LMS ($Eq$-LMS) algorithm utilizes a novel, parameterless concept of error-correlation energy and normalization of signal to ensure…
▽ More
In this work, a new class of stochastic gradient algorithm is developed based on $q$-calculus. Unlike the existing $q$-LMS algorithm, the proposed approach fully utilizes the concept of $q$-calculus by incorporating time-varying $q$ parameter. The proposed enhanced $q$-LMS ($Eq$-LMS) algorithm utilizes a novel, parameterless concept of error-correlation energy and normalization of signal to ensure high convergence, stability and low steady-state error. The proposed algorithm automatically adapts the learning rate with respect to the error. For the evaluation purpose the system identification problem is considered. Extensive experiments show better performance of the proposed $Eq$-LMS algorithm compared to the standard $q$-LMS approach.
△ Less
Submitted 1 January, 2018;
originally announced January 2018.
