Extremely Simple Out-of-distribution Detection for Audio-visual Generalized Zero-shot Learning

Abstract

Report issue for preceding element

Zero-shot Learning (ZSL) attains knowledge transfer from seen classes to unseen classes by exploring auxiliary category information, which is a promising yet difficult research topic. In this field, Audio-Visual Generalized Zero-Shot Learning (AV-GZSL) has aroused researchers’ great interest in which intricate relations within triple modalities (audio, video, and natural language) render this task quite challenging but highly research-worthy. However, both existing embedding-based and generative-based AV-GZSL methods tend to suffer from domain shift problem a lot and we propose an extremely simple Out-of-distribution (OOD) detection based AV-GZSL method (EZ-AVOOD) to further mitigate bias problem by differentiating seen and unseen samples at the initial beginning. EZ-AVOOD accomplishes effective seen-unseen separation by exploiting the intrinsic discriminative information held in class-specific logits and class-agnostic feature subspace without training an extra OOD detector network. Followed by seen-unseen binary classification, we employ two expert models to classify seen samples and unseen samples separately. Compared to existing state-of-the-art methods, our model achieves superior ZSL and GZSL performances on three audio-visual datasets and becomes the new SOTA, which comprehensively demonstrates the effectiveness of the proposed EZ-AVOOD.

Report issue for preceding element

1 Introduction

Report issue for preceding element

Multimodal learning has become one of the most trending research topics nowadays involving vision-language tasks including visual question answering [7], image captioning [34], cross-modal retrieval [39], visual entailment [37], and so on, and audio-language tasks like audio captioning [22], emotion recognition [9], speech translation [31], etc., since vision, audio, and language are the most common signal sources in real world. However, it is undoubtedly an overwhelming burden to collect and label quantities of videos/pictures, audio signals, and natural language corpus to fulfill aforementioned tasks, where Zero-shot Learning (ZSL) emerges as an attainable approach to get rid of redundant data collection by mining auxiliary information like semantic attributes and word embeddings to achieve knowledge transfer without contact with unseen samples. Concerning video-audio multimodal tasks, AV-GZSL dependent on the fusion of audio and visual input, and natural language description is the chosen methodology to cope with classification, retrieval, and other problems.

Report issue for preceding element

Figure 1: Harmonic mean (%) evaluating GZSL performance of our EZ-AVOOD model and other comparison methods on three datasets. EZ-AVOOD (the red bar) consistently outperforms the rest opponents with a lead margin up to 5% on the UCF-GZSL benchmark.

Report issue for preceding element

[28] sets out to apply GZSL to audio-visual features. AVGZSLNet [21] leverages late fusion for integrating information from bi-modalities. Most of the subsequent works have focused on efficient fusion of audio-visual feature representations: AVCA [25] initially utilizes cross-attention [33] block which is inherited by [24, 12, 27]. Additionally, generative approach AVFS [40] facilitates contrastive learning with the synthesized negative unseen samples. However, domain shift problem is always unavoidable for ZSL with the bias towards seen classes, AV-GZSL simply behaves the same way, especially for embedding-based approaches. Generative solutions aim to reduce the bias with synthesized unseen samples and Calibrated Stacking [4] searches extra hyper-parameters to suppress the model’s tendency towards seen samples. Considering that generative model is unstable to train while Calibrated Stacking’s effect is limited, in this paper we propose an extremely simple OOD detection based AV-GZSL method named EZ-AVOOD to effectively alleviate the bias problem in an explicit way. Figure 1 fully exhibits the competence of our model where our model outperforms all the contrasting methods on three audio-visual benchmarks.

Report issue for preceding element

Typically, the fundamental framework of OOD approaches in ZSL shares the universal three parts: an OOD detector, one seen classifier and an unseen classifier, and each of them plays a vital role in the final GZSL performance. Additionally, these three components indeed work quite independently and consequently they are relatively substitutable when stronger components are developed. Unlike previous work which utilizes generative method like WGAN-GP [1, 10] to first synthesize unseen samples features, and then jointly train a completely new OOD detector with seen samples and synthesized unseen samples [35], the proposed EZ-AVOOD model accomplishes OOD detection without training the OOD detector with the aid of class-specific logits produced by supervised seen classifier and class-agnostic information hidden in feature subspace. More specifically, our method only needs to train the seen classifier composed of one MLP (Multilayer Perceptrons), and fine-tune an existing embedding-based method [14] as the unseen expert classifier, which considerably reduces the complexity of the entire model. Moreover, the unseen classifier of our EZ-AVOOD can be substituted by arbitrary optimal AV-GZSL method for future researchers to achieve higher overall GZSL performance. The main contributions of this paper can be summarized into the following three aspects:

Report issue for preceding element

• We proposed an extremely simple OOD detection based model EZ-AVOOD to address AV-GZSL problem with OOD score derived from class-specific logits and class-agnostic feature subspace instead of training a completely new OOD detector.

  Report issue for preceding element

• We comprehensively demonstrate the effectiveness of EZ-AVOOD model through practical ZSL and GZSL experiments on three different audio-visual benchmarks and observe a substantial enhancement in comparison with the current state-of-the-art methodologies.

  Report issue for preceding element

• The proposed effective OOD detection method EZ-OOD possesses strong compatibility with existing AV-GZSL approaches, which indicates that future researchers could effortlessly improve the GZSL performance by simply replacing the unseen expert with more powerful substitutes.

  Report issue for preceding element

2 Related Work

Report issue for preceding element

2.1 Audio-visual Generalized Zero-shot Learning

Report issue for preceding element

Embedding approaches mapping the videos, audio signals, and text information into a shared common space to get joint feature representations aimed for subsequent classification or retrieval tasks enjoy wide popularity in AV-GZSL. Among them, prior CJME [28] employs triplet loss to restrict the distance between audio-visual features and class embedding on the proposed AudioSetZSL [28] dataset. Recently, TCaF [24] proposes a temporal cross-attention framework enhanced from AVCA [25] by factoring in the temporal information. Hyperbolic [12] method employs hyperbolic alignment loss and cross-attention module to further improve the separability of joint feature representations. EZ-AVGZL [27] utilizes class embedding optimization to achieve better discriminability of class embeddings while maintaining their original semantics. ClipClap [14] enhances the feature quality by pre-trained CLIP [29] and CLAP [23] models. As for the generative method, AVFS [40](Audio-Visual Feature Synthesis) generates unseen samples to facilitate contrastive training.

Report issue for preceding element

2.2 Out-of-distribution Detection and Post-hoc Methods

Report issue for preceding element

Out-of-distribution (OOD) detection is proposed to secure the smooth deployment of machine learning models in real-world scenarios since these models are typically trained and validated in close-world settings. OOD detection is implemented with different OOD scores (scalars) produced by ID (in-distribution) and OOD samples on which certain thresholds are applied to finish ID-OOD separation.

Report issue for preceding element

Post-hoc methods for OOD detection typically involve training a model or a classifier with ID data first and subsequently, the model with frozen parameters is converted into an OOD detector during the test stage, which is cost-effective compared with training an extra detector. Post-hoc methods compute an OOD score based on the pre-trained model’s output. One simple solution MSP [11] utilizes the maximum softmax probability as the OOD score to distinguish ID and OOD samples. To improve the discriminability of the softmax score, ODIN [18] employs temperature scaling to make the softmax score distribution more uniform and also applies input perturbation. Logits-based Energy score [20, 19] makes use of output logits in conjunction with the LogSumExp function. Features-based methods [8, 32, 13] like WDiscOOD [6] leverages LDA (linear discriminant analysis) [38] to enlarge the inter-class discrepancy and reduce the intra-class gap for better ID-OOD separation.

Report issue for preceding element

3 Proposed Approach

Report issue for preceding element

Figure 2: The general framework of EZ-AVOOD. Four key modules “Feature Extractor”, “OOD Detector”, “Seen Classifier” and “Unseen Classifier” make up the complete model. Parameter-fixed feature extractor simply produces audio-visual features $\boldsymbol{a}\oplus\boldsymbol{v}$ ($\oplus$ represents concatenation operation) and text embeddings $\boldsymbol{t}$ without further optimization. Seen classifier and OOD detector are implemented with two identical MLPs, which means they share the same copy of parameters and need to train only one of them to make two modules work. The process of OOD score formulation is illustrated in Figure 3. At evaluation stage, OOD detector distinguishes seen and unseen samples and input them to the trained seen expert and unseen expert classifiers respectively (red arrows).

Report issue for preceding element

3.1 Problem Statement of Audio-visual GZSL

Report issue for preceding element

AV-GZSL aims to efficiently recognize previously seen and even unseen video-audio combinations through the set of seen (training) audio-visual events together with human-readable text descriptions. Thus, we denote samples from seen classes with $\boldsymbol{S}=(\boldsymbol{a}_{i}^{s},\boldsymbol{v}_{i}^{s},\boldsymbol{t}_{i}^{s},y_{i}^{s})_{i\in\{1,\cdots,M\}}$ , where $M$ seen samples in total consist of audio features $\boldsymbol{a}_{i}^{s}$, visual features $\boldsymbol{v}_{i}^{s}$ and textual description $\boldsymbol{t}_{i}^{s}$ as well as the corresponding ground truth class label $y_{i}^{s}$. Likewise, unseen dataset is denoted as $\boldsymbol{U}=(\boldsymbol{a}_{j}^{u},\boldsymbol{v}_{j}^{u},\boldsymbol{t}_{j}^{u},y_{j}^{u})_{j\in\{1,\cdots,K\}}$ with $K$ samples and notably ${\boldsymbol{S}}\cap{\boldsymbol{U}}=\varnothing$. The number of seen classes labels is $C_{s}$: ${\boldsymbol{Y}^{s}}={({y_{1}^{s}},\cdots,{y}_{M}^{s})}\in{\{{1,\cdots,C_{s}}\}}$ , and the number of unseen classes is $C_{u}$: ${\boldsymbol{Y}^{u}}={({y_{1}^{u}},\cdots,{y}_{K}^{u})}\in{\{{1,\cdots,C_{u}}\}}$. Hence,for the ZSL task, AV-GZSL learns the model $f_{ZSL}:\boldsymbol{X}\rightarrow{{{\boldsymbol{Y}}^{u}}}$ to classify unseen samples only, where $\boldsymbol{X}=(\boldsymbol{a}_{z},\boldsymbol{v}_{z})$ denotes the test dataset and in GZSL, the classifier $f_{GZSL}:\boldsymbol{X}\rightarrow{\boldsymbol{Y}^{u}\cup\boldsymbol{Y}^{s}}$ aims to classify seen and unseen examples.

Report issue for preceding element

3.2 Model Architecture

Report issue for preceding element

The proposed model is depicted in Figure 2 and there are three essential components in our model: the OOD detector, the supervised seen classifier, and the unseen classifier adapted from an existing embedding method. The greatest strength of our method is that there is no need to train a new OOD detector, since our OOD detector shares the same parameters with the seen classifier, which means the supervised classifier trained with seen samples also serves as the OOD detector. Therefore, EZ-AVOOD significantly reduces the complexity of the model and brings a considerable decrease in the computational overhead and training time.

Report issue for preceding element

3.2.1 Seen Classifier

Report issue for preceding element

To cope with seen samples, a vanilla and efficient 3-layer MLP optimized with Cross Entropy Loss $\boldsymbol{\mathcal{L}}_{xent}=\textbf{CrossEntropy}(\boldsymbol{x},y(\boldsymbol{x}))$ is adopted as the seen expert classifier, where $\boldsymbol{x}$ denotes the joint audio-visual features from seen classes constructed through simple concatenation: $\boldsymbol{x}=\boldsymbol{a}\oplus\boldsymbol{v}$. Moreover, once the seen classifier is trained, it can be leveraged as the OOD detector, and more details are thoroughly elaborated in the next part.

Report issue for preceding element

Figure 3: The process of the EZ-OOD score formulation. During training phase, “Residual Subspace” is derived from the eigen-decomposition on all seen samples features matrix. At test time (pink arrows), concatenated audio-visual feature $\boldsymbol{a}\oplus\boldsymbol{v}$ projects onto the residual subspace to get “Residual Score” and “Energy Score” is calculated with the logits of the test sample produced by the MLP (the trained seen classifier actually). The final OOD score is defined by the weighted sum of energy score and residual score.

Report issue for preceding element

3.2.2 Out-of-distribution Detector

Report issue for preceding element

We adopt post-hoc idea to design OOD algorithm tackling seen-unseen separation in AV-GZSL problem. Since the output of pre-trained model usually includes high-dimensional features, logits, or Softmax probability and we choose to exploit the intrinsic information held in class-specific logits and feature representation to construct OOD score. Consequently, the trained seen classifier now becomes the “OOD detector” in our method to output class-dependent logits. The proposed extremely simple OOD detection method is named as “EZ-OOD” and the formulation pipeline is illustrated in Figure 3. The EZ-OOD score consists of Energy Score calculated by logits and Residual Score derived from residual subspace.

Report issue for preceding element

Class-specific Logits and Energy Score  Seen classifier takes the high-dimensional fused audio-visual features as input and produces the logits corresponding to specific seen classes labels. We adopt the widely used Energy Score function LogSumExp $\boldsymbol{E(x;\,l)}$ as one vital part of our OOD score, mapping the logits $\boldsymbol{l}$ of sample $\boldsymbol{x}$ to a scalar:

Report issue for preceding element

$$\boldsymbol{E}(\boldsymbol{x};\,\boldsymbol{l})=-\log\sum_{i=1}^{C}e^{l_{i}(\boldsymbol{x})}~{},$$ (1)

where $C$ is the number of seen classes and $l_{i}(\boldsymbol{x})$ is the logit of class-$i$ in correspondence with sample $\boldsymbol{x}$. Negative inversion $\boldsymbol{E}(\boldsymbol{x})=-\boldsymbol{E}(\boldsymbol{x};\,\boldsymbol{l})$ is the score practically used to ensure that seen samples produce higher scores, which is consistent with the tradition in OOD detection.

Report issue for preceding element

Feature Representation and Class-agnostic Residual Score  Here $\boldsymbol{x}\in{\mathbb{R}}^{D}$ is the D-dimensional fused bi-modal sample feature and $\boldsymbol{X}$ denotes the audio-visual feature matrix of all seen samples. Therefore, the principal subspace ${P}$ is defined by the $N$-dimensional space spanned by the eigenvetors corresponding to the top-$N$ eigenvalues of matrix $\boldsymbol{X}^{T}\boldsymbol{X}$ and the the residual subspace ${P}^{\perp}$ is the orthogonal complements of the space ${P}$. Thus we have $\boldsymbol{x}=\boldsymbol{x}^{P}+\boldsymbol{x}^{{P}^{\perp}}$ where $\boldsymbol{x}^{P}$ is the projection of sample feature $\boldsymbol{x}$ onto subspace $P$ and $\boldsymbol{x}^{{P}^{\perp}}$ is the mapping on ${P}^{\perp}$. Suppose the eigen-decomposition of matrix $\boldsymbol{X}^{T}\boldsymbol{X}$ is

Report issue for preceding element

$$\boldsymbol{X}^{T}\boldsymbol{X}=\boldsymbol{W{\Lambda}W}^{T}~{},$$ (2)

where $\boldsymbol{W}$ refers to a set of standard orthogonal bases that are arranged according to the decreasing order of the eigenvalues within the diagonal matrix $\boldsymbol{\Lambda}$. $N$-dimensional principal subspace $P$ is defined by the first $N$ column vectors of $\boldsymbol{W}$, and span of the $(N+1)$-th column to the $D$-th column vectors in $\boldsymbol{W}$ is the residual subspace ${P}^{\perp}$. Then the matrix $\boldsymbol{W}$ can be separated into matrix $\boldsymbol{Q}\in{\mathbb{R}}^{D\times{N}}$ and matrix $\boldsymbol{O}\in{\mathbb{R}}^{D\times(D-N)}$ formed by the last $(D-N)$ eigenvetors, and we can get

Report issue for preceding element

$$\boldsymbol{x}^{P}={\boldsymbol{Q}}^{T}\boldsymbol{x};\ \boldsymbol{x}^{{P}^{\perp}}={\boldsymbol{O}}^{T}\boldsymbol{x}~{}.$$ (3)

Given that the principal subspace and the residual subspace are constructed based on the feature representations of all training samples, they just ignore the information specific to individual seen category, namely characteristics hidden in these two subspaces are class-agnostic. Moreover, we argue that seen samples are relatively closer to principal subspace while deviate a lot from residual subspace. Therefore we define the Residual Score $bm{R}(\boldsymbol{x})$ as the norm of $\boldsymbol{x}^{{P}^{\perp}}$:

Report issue for preceding element

$$\boldsymbol{R}(\boldsymbol{x})=-\|\boldsymbol{x}^{{P}^{\perp}}\|=-\left(\boldsymbol{x}^{T}{\boldsymbol{O}}{\boldsymbol{O}}^{T}\boldsymbol{x}\right)^{1/2}~{}.$$ (4)

Just like the Energy Score, we take a minus norm to make ID samples produce higher OOD scores.

Report issue for preceding element

EZ-OOD Score Formulation  The final OOD score $\boldsymbol{S}(\boldsymbol{x})$ is formulated by the weighted sum of energy score and residual score to unify the class-specific and class-agnostic information for better ID-OOD separation:

Report issue for preceding element

$$\boldsymbol{S}(\boldsymbol{x})=\boldsymbol{E}(\boldsymbol{x})+\gamma\boldsymbol{R}(\boldsymbol{x})~{},$$ (5)

where $\gamma$ is the weight hyper-parameter to balance the scale of these two different scores and enhance the overall OOD detection performance.

Report issue for preceding element

OOD detection process is defined as below, $A$ is the binary classification outcome,

Report issue for preceding element

$$A_{\lambda}(\boldsymbol{x})=\begin{cases}\text{ Seen }&\boldsymbol{S}(\boldsymbol{x})\geq\lambda\\ \text{ Unseen }&\boldsymbol{S}(\boldsymbol{x})<\lambda~{},\end{cases}$$ (6)

where $\lambda$ is the threshold and samples possess higher $\boldsymbol{S}(\boldsymbol{x})$ are tend to be treated as seen classes. The threshold is uniquely determined by the training samples and has nothing to do with the test data.

Report issue for preceding element

Methods	VGGSound-GZSLcls				UCF-GZSLcls				ActivityNet-GZSLcls
	$acc_{\boldsymbol{S}}$	$acc_{\boldsymbol{U}}$	$\boldsymbol{H}$	$acc_{ZSL}$	$acc_{\boldsymbol{S}}$	$acc_{\boldsymbol{U}}$	$\boldsymbol{H}$	$acc_{ZSL}$	$acc_{\boldsymbol{S}}$	$acc_{\boldsymbol{U}}$	$\boldsymbol{H}$	$acc_{ZSL}$
CJME [28]	11.96	5.41	7.45	6.84	48.18	17.68	25.87	20.46	16.06	9.13	11.64	9.92
AVGZSLNet [21]	13.02	2.88	4.71	5.44	56.26	34.37	42.67	35.66	14.81	11.11	12.70	12.39
AVCA [25]	32.47	6.81	11.26	8.16	34.90	38.67	36.69	38.67	24.04	19.88	21.76	20.88
Hyper-multiple [12]	21.99	8.12	11.87	8.47	43.52	39.77	41.56	40.28	20.52	21.30	20.90	22.18
ClipClap [14]	29.68	11.12	16.18	11.53	77.14	43.91	55.97	46.96	45.98	20.06	27.93	22.76
EZ-AVOOD (Ours)	39.33	11.84	18.21	13.28	83.53	48.01	60.97	50.92	41.56	21.06	27.95	25.20

Table 1: Comparison with existing state-of-the-art methods on VGGSound-GZSL^cls, UCF-GZSL^cls and ActivityNet-GZSL^cls datasets. Performances in percentage of GZSL ($acc_{\boldsymbol{S}}$/$acc_{\boldsymbol{U}}$/H) and ZSL ($acc_{ZSL}$) are reported. For fair comparison, results of all five baseline methods are obtained using audio-visual features and text embeddings extracted by CLIP and CLAP models. Bold values represent the best results and the second-ranked numbers are underlined.

Report issue for preceding element

3.2.3 Unseen Classifier

Report issue for preceding element

Here we fine-tune the ClipClap [14] model to enhance the unseen classes average accuracy for the purpose of improving final GZSL performance. As illustrated in Figure 2, the general framework consists of two branches of Encoder-Encoder-Decoder pipeline to deal with concatenated audio-visual features and fused text embeddings, respectively. With respect to the feature extraction, to be specific, visual features $\boldsymbol{v}$ and part of concatenated text embeddings $\boldsymbol{t^{v}}$ are extracted by vision-language pre-trained model CLIP [29] and CLAP [23] model dedicated to audio-language tasks produces audio features $\boldsymbol{a}$ and another part of text embeddings $\boldsymbol{t^{a}}$.

Report issue for preceding element

The first encoder block $\boldsymbol{O}_{enc}$ from audio-visual branch takes concatenated features $\boldsymbol{x}=\boldsymbol{a}\oplus\boldsymbol{v}$ as input and outputs the multimodal sample features $\boldsymbol{o}$ :

Report issue for preceding element

$$\boldsymbol{o}=\boldsymbol{O}_{enc}(\boldsymbol{x})~{}.$$ (7)

In the same way, we get unified text embeddings $\boldsymbol{w}$ with encoder $\boldsymbol{W}_{enc}$:

Report issue for preceding element

$$\boldsymbol{w}=\boldsymbol{W}_{enc}(\boldsymbol{t^{a}}\oplus\boldsymbol{t^{v}})~{}.$$ (8)

With multimodal sample features $\boldsymbol{o}$ and fused text embeddings $\boldsymbol{w}$ as the input of the rest two simple and effective Encoder-Decoder compound modules, we have the following formulations:

Report issue for preceding element

$$\boldsymbol{\theta}_{o}=\boldsymbol{O}_{proj}(\boldsymbol{o});\ \boldsymbol{\rho}_{o}=\boldsymbol{D}_{o}(\boldsymbol{\theta}_{o})~{},$$ (9)

$$\boldsymbol{\theta}_{w}=\boldsymbol{W}_{proj}(\boldsymbol{w});\ \boldsymbol{\rho}_{w}=\boldsymbol{D}_{w}(\boldsymbol{\theta}_{w})~{},$$ (10)

where $\boldsymbol{\theta}_{o}$ and $\boldsymbol{\theta}_{w}$ represent the projection outcomes, while the reconstruction process produces $\boldsymbol{\rho}_{o}$ and $\boldsymbol{\rho}_{w}$. Then the training objectives of unseen classifier include Cross Entropy Loss $\boldsymbol{\mathcal{L}}_{xe}$:

Report issue for preceding element

$$\boldsymbol{\mathcal{L}}_{xe}=-\frac{1}{n}\sum_{i}^{n}\log\left(\frac{\exp\left(\boldsymbol{\theta}_{w_{y_{i}^{s}}}\boldsymbol{\theta}_{o_{i}}\right)}{\sum_{j}^{C_{s}}\exp\left(\boldsymbol{\theta}_{w_{y_{j}^{s}}}\boldsymbol{\theta}_{o_{i}}\right)}\right)~{},$$ (11)

where $y_{i}^{s}$ is the label of seen sample $i$, and $\boldsymbol{\theta}_{w_{y_{i}^{s}}}$ denotes the $\boldsymbol{\theta}_{w}$-projection of text embedding belonging to seen class $y_{i}^{s}$. $n$ and $C_{s}$ represent the number of training samples and seen categories, respectively. Another loss function is Reconstruction Loss $\boldsymbol{\mathcal{L}}_{rec}$ to minimize the discrepancy between $\boldsymbol{\rho}$ and text embeddings $\boldsymbol{w}$ with MSE (mean squared error):

Report issue for preceding element

$$\boldsymbol{\mathcal{L}}_{rec}=\frac{1}{n}\sum_{i}^{n}\left[{\left(\boldsymbol{\rho}_{o_{i}}-\boldsymbol{w}_{i}\right)}^{2}+{\left(\boldsymbol{\rho}_{w_{i}}-\boldsymbol{w}_{i}\right)}^{2}\right]~{}.$$ (12)

Moreover, a Regression Loss $\boldsymbol{\mathcal{L}}_{reg}$ calculated by MSE function between $\boldsymbol{\theta}_{o}$ and $\boldsymbol{\theta}_{w}$ is defined as:

Report issue for preceding element

$$\boldsymbol{\mathcal{L}}_{reg}=\frac{1}{n}\sum_{i}^{n}{\left(\boldsymbol{\theta}_{o_{i}}-\boldsymbol{\theta}_{w_{i}}\right)}^{2}~{}.$$ (13)

And the overall loss for unseen classifier is defined as:

Report issue for preceding element

$$\boldsymbol{\mathcal{L}}_{total}=\boldsymbol{\mathcal{L}}_{xe}+\boldsymbol{\mathcal{L}}_{rec}+\boldsymbol{\mathcal{L}}_{reg}~{}.$$ (14)

Following the original loss function design, there is no weight parameter applied on final loss $\boldsymbol{\mathcal{L}}_{total}$.

Report issue for preceding element

During test phase, the classification result is determined by the nearest neighbor principle which means class text embedding closest to sample feature projection $\boldsymbol{\theta}_{o}^{i}$ is selected as the predicted label $c_{i}$:

Report issue for preceding element

$$c_{i}=\underset{j}{\operatorname{argmin}}\left(\left\|\boldsymbol{\theta}_{w}^{j}-\boldsymbol{\theta}_{o}^{i}\right\|_{2}\right)~{},$$ (15)

where $\boldsymbol{\theta}_{w}^{j}$ is the encoded text embeddings corresponding to class $i$.

Report issue for preceding element

Methods	VGGSound-GZSLmain				UCF-GZSLmain				ActivityNet-GZSLmain
	$acc_{\boldsymbol{S}}$	$acc_{\boldsymbol{U}}$	$\boldsymbol{H}$	$acc_{ZSL}$	$acc_{\boldsymbol{S}}$	$acc_{\boldsymbol{U}}$	$\boldsymbol{H}$	$acc_{ZSL}$	$acc_{\boldsymbol{S}}$	$acc_{\boldsymbol{U}}$	$\boldsymbol{H}$	$acc_{ZSL}$
CJME [28]	8.69	4.78	6.17	5.16	26.04	8.21	12.48	8.29	5.55	4.75	5.12	5.84
AVGZSLNet [21]	18.15	3.48	5.83	5.28	52.52	10.90	18.05	13.65	8.93	5.04	6.44	5.40
TCaF [24]	9.64	5.91	7.33	6.06	58.60	21.74	31.72	24.81	18.70	7.50	10.71	7.91
VIB-GZSL [17]	18.42	6.00	9.05	6.41	90.35	21.41	34.62	22.49	22.12	8.94	12.73	9.29
AVMST [15]	14.14	5.28	7.68	6.61	44.08	22.63	29.91	28.19	17.75	9.90	12.71	10.37
MDFT [16]	16.14	5.97	8.72	7.13	48.79	23.11	31.36	31.53	18.32	10.55	13.39	12.55
Hyper-multiple [12]	15.02	6.75	9.32	7.97	63.08	19.10	29.32	22.24	23.38	8.67	12.65	9.50
AVCA [25]	14.90	4.00	6.31	6.00	51.53	18.43	27.15	20.01	24.86	8.02	12.13	9.13
OOD-entropy+AVCA [35]	13.31	7.01	9.19	7.48	63.94	26.99	37.96	30.56	29.84	9.54	14.46	11.41
EZ-OOD+AVCA (Ours)	24.94	6.38	10.16	7.48	79.71	27.94	41.38	30.56	30.65	9.29	14.26	11.41

Table 2: Compatibility of EZ-OOD with existing method. We make comparisons between existing state-of-the-art AV-GZSL methods and our new model on VGGSound-GZSL^main, UCF-GZSL^main, and ActivityNet-GZSL^main datasets. GZSL ($acc_{\boldsymbol{S}}$/$acc_{\boldsymbol{U}}$/$\boldsymbol{H}$) and ZSL ($acc_{ZSL}$) performances are reported in percentage. Bold numbers denote the best results and the second highest values are underlined.

Report issue for preceding element

4 Experiments and Results Analysis

Report issue for preceding element

4.1 Setup for Audio-visual GZSL

Report issue for preceding element

4.1.1 Datasets and Evaluation Metrics

Report issue for preceding element

Following experimental setting in AVCA [25], we adopt the curated version of three audio-visual datasets: VGGSound [5], UCF101 [30], and ActivityNet [3] to evaluate our EZ-AVOOD method and they are VGGSound-GZSL^cls, UCF-GZSL^cls and ActivityNet-GZSL^cls. The upper $cls$ represents $cls$-$split$ introduced in [24] instead of $main$-$split$ utilized in [25].

Report issue for preceding element

Consistent with ZSL conventions [36, 25], we adopt the average per-class classification accuracy as the evaluation metric, where $acc_{\boldsymbol{S}}$ and $acc_{\boldsymbol{U}}$ denotes the mean class accuracy of seen classes and unseen classes separately. To comprehensively evaluate GZSL performance, the harmonic mean $\boldsymbol{H}$ of seen and unseen accuracy is calculated as:

Report issue for preceding element

$$\boldsymbol{H}=\frac{2*acc_{\boldsymbol{S}}*acc_{\boldsymbol{U}}}{acc_{\boldsymbol{S}}+acc_{\boldsymbol{U}}}~{}.$$ (16)

For ZSL tasks aimed to classify unseen samples only, mean class accuracy $acc_{ZSL}$ is also obtained.

Report issue for preceding element

4.1.2 Implementation Details

Report issue for preceding element

OOD Detector  The trained seen classifier is transformed into our OOD detector to produce logits and energy score. As for residual score, the dimension N of principal subspace and the scaling factor $\gamma$ are valued at 64/90 for VGGSound-GZSL^cls, 256/205 for UCF-GZSL^cls and 256/285 for ActivityNet-GZSL^cls.

Report issue for preceding element

More details about Feature Extractor, Seen Classifier, and Unseen Classifier are provided in supplementary material.

Report issue for preceding element

4.2 Experimental Results

Report issue for preceding element

4.2.1 Quantitative Results

Report issue for preceding element

As shown in Table 1, our EZ-AVOOD model consistently takes the lead on all three benchmarks in terms of both harmonic mean $\boldsymbol{H}$ for GZSL task and $acc_{ZSL}$ under ZSL setup. For VGGSound-GZSL^cls dataset, we achieve the best performances on all metrics and specially EZ-AVOOD considerably outperforms current state-of-the-art ClipClap with the lead of 9.65%@$acc_{\boldsymbol{S}}$, 0.72%@$acc_{\boldsymbol{U}}$, 2.03%@$\boldsymbol{H}$, and 1.75%@$acc_{ZSL}$. In addition, our method substantially overtakes the ClipClap on UCF-GZSL^cls benchmark with even bigger lead margins of 6.39%@$acc_{\boldsymbol{S}}$, 4.10%@$acc_{\boldsymbol{U}}$, 5.00%@$\boldsymbol{H}$, and 3.96%@$acc_{ZSL}$ respectively. Though the proposed EZ-AVOOD “merely” takes the second place on $acc_{\boldsymbol{S}}$ and $acc_{\boldsymbol{U}}$ of ActivityNet-GZSL^cls, in which ClipClap and Hyper-multiple ranks the top separately, our method does holds the first place on the more comprehensive metric $\boldsymbol{H}$ and attains significant performance on $acc_{ZSL}$ better than all other baseline methods with a 2.44% lead margin at least.

Report issue for preceding element

4.3 Compatibility of EZ-OOD with Existing Method

Report issue for preceding element

4.3.1 Experimental Setup

Report issue for preceding element

Here, we replace the unseen classifier of EZ-AVOOD with AVCA [25] and explore the new model’s experimental performance. We provide some key details of this experiment: $main$-$split$ of three datasets is adopted; audio features and visual features are extracted by self-supervised SeLaVi [2] pre-trained on VGGSound dataset; and text embeddings are obtained using word2vec model [26] pre-trained with Wikipedia.

Report issue for preceding element

4.3.2 Baseline Method

Report issue for preceding element

AV-OOD[35] is another OOD-based method that takes the AVCA model as unseen expert and proposes the OOD-entropy method for OOD detection, consequently quite suitable for making contrasts with our method right here. To ensure a just comparison, we only train the seen classifier (also working as the OOD detector) and utilize the same unseen classifier as AV-OOD method. Notably, we re-run the AV-OOD with the provided checkpoint files and get ZSL and GZSL performances to facilitate fair comparison. As can be seen in Table 2, the $acc_{ZSL}$ results of our method and OOD-entropy are totally the same on three benchmarks.

Report issue for preceding element

Methods	VGGSound-GZSLcls				UCF-GZSLcls				ActivityNet-GZSLcls
	AUROC	FPR95	AUPR	$\boldsymbol{H}$	AUROC	FPR95	AUPR	$\boldsymbol{H}$	AUROC	FPR95	AUPR	$\boldsymbol{H}$
	$\uparrow$	$\downarrow$	$\uparrow$	$\uparrow$	$\uparrow$	$\downarrow$	$\uparrow$	$\uparrow$	$\uparrow$	$\downarrow$	$\uparrow$	$\uparrow$
Residual Score	68.73	85.74	85.51	14.91	91.30	42.09	90.96	58.65	67.30	85.28	44.73	25.34
Energy Score	81.97	67.69	92.65	17.84	80.41	74.09	86.55	47.57	70.99	86.95	54.57	26.13
EZ-OOD (full)	84.33	66.24	93.82	18.21	95.35	33.87	96.01	60.97	77.57	80.61	63.62	27.95

Table 3: Ablation studies on EZ-OOD method. We make a comparison with Energy Score, Resdual Score, and their $\gamma$-weighted sum the full EZ-OOD on VGGSound-GZSL^cls, UCF-GZSL^cls and ActivityNet-GZSL^cls datasets. Out-of-distribution detection metrics (AUROC/FPR95/AUPR) and GZSL (harmonic mean $\boldsymbol{H}$) performance are reported in percentage. $\downarrow$ indicates that lower results are better while $\uparrow$ means the opposite. Bold values denote the best results and the second-best outcomes are underlined.

Report issue for preceding element

Figure 4: ROC curves of EZ-OOD, Energy Score, and Residual Score on three datasets. Evidently, the full EZ-OOD consistently outperforms Energy Score and Residual Score with larger AUROC metric.

Report issue for preceding element

4.3.3 Quantitative Results

Report issue for preceding element

In the first place, compared with baseline method AVCA, the new model has secured a full-fledged lead on all metrics of three different datasets. More accurately, the greatest lead margins can amount to 28.18%@$acc_{\boldsymbol{S}}$, 9.51%@$acc_{\boldsymbol{U}}$, 14.23%@$\boldsymbol{H}$, and 10.55%@$acc_{ZSL}$ on UCF-GZSL^main dataset. In addition, our new model significantly increases the ZSL and GZSL performances on the other 2 benchmarks from the baseline which comprehensively verifies the compatibility of the proposed EZ-OOD method.

Report issue for preceding element

Secondly, compared with OOD-entropy+AVCA [35], our EZ-OOD leveraging the same unseen classifier attains a remarkable lead on the harmonic mean metric of both VGGSound-GZSL^main and UCF-GZSL^main datasets (0.97% and 3.42% separately), and lags behind by merely 0.2%@$\boldsymbol{H}$ on ActivityNet-GZSL^main benchmark. Notably, the OOD detection performance of our method is ahead of OOD-entropy actually on ActivityNet benchmark which is illustrated in the supplementary material. Since AV-GZSL evaluates the average per-class classification accuracy, while OOD detection simply considers ID-OOD separation of all test samples without caring about class labels, as a result, better but close OOD performance not always brings stronger GZSL performance.

Report issue for preceding element

4.4 Ablation Studies

Report issue for preceding element

To gain an insight into the concrete effect of Energy Score and Residual Score, we conduct additional ablation studies to compare the OOD detection performance and AV-GZSL results within EZ-OOD and its two key components. Experimental setup is consistent with the proposed EZ-AVOOD model in Experiment 4.2. Here in Table 3 we report the AUROC (Area Under the Receiver Operating Characteristic curve), FPR95 (FPR@TPR95), and AUPR (Area Under the Precision versus Recall curve) to evaluate OOD detection capability as well as the harmonic mean $\boldsymbol{H}$ of GZSL task on three datasets. Also, we draw the ROC curves belonging to the three methods to explicitly display their OOD detection performance on each benchmark in Figure 4.

Report issue for preceding element

Figure 5: Effect of scaling factor $\gamma$ on AUROC for three datasets. OOD detection performance of EZ-OOD reaches the top when energy score and residual score are properly matched with the linear combination scaled by a suitable $\gamma$.

Report issue for preceding element

4.4.1 Quantitative Results and Qualitative Results

Report issue for preceding element

According to the results in Table 3, the full EZ-OOD undoubtedly takes the first place on all three OOD detection metrics with the highest AUROC and AUPR and the lowest FPR95 and naturally achieves the best $\boldsymbol{H}$ for GZSL. In terms of the two components, Energy Score ranks second on VGGSound-GZSL^cls and ActivityNet-GZSL^cls datasets and Residual Score attains a remarkable lead over Energy Score on UCF-GZSL^cls benchmark. Moreover, we observe that the harmonic mean $\boldsymbol{H}$ produce by Energy Score@VGGSound-GZSL^cls (17.84%) and Residual Score@UCF-GZSL^cls (58.65%) effortlessly defeat all the contrasting methods in Table 1. Therefore, we conclude that both Energy Score and Residual Score play a vital role in separating seen and unseen samples to facilitate subsequent ZSL classification objectives.

Report issue for preceding element

Figure 4 depicts the AUROC discrepancy between EZ-OOD score and its two crucial components on three benchmarks. In addition, two individual OOD scores manifest competitive detection performance whose ROC curves are close to the upper-left in the graph. To sum up, the proposed OOD score effectively combines the strengths of the two powerful scores with the weighted sum to achieve stronger OOD detection performance and higher GZSL classification accuracy.

Report issue for preceding element

4.5 Parameter Sensitivity Studies

Report issue for preceding element

4.5.1 Effect of Scaling Factor $\gamma$

Report issue for preceding element

Here we test the scaling factor $\gamma$ from the set: $\gamma\in\{0.1,1,10,100,250,500,1000\}$ with a fixed N specific to each dataset. Typically, better OOD detection performance will bring higher $\boldsymbol{H}$ for GZSL task, hence, AUROC is evaluated to avoid extra computational burden instead of $\boldsymbol{H}$. We follow the same experimental setup in Section 4.2. Figure 5 illustrates the AUROC on three datasets with different scaling factors $\gamma$. When the scaling factor is valued at 0.1 or 1000, the linear combination EZ-OOD score will reduce to ordinary energy score or individual residual score, resulting in lower AUROC at both ends of the curve, which is consistent with the outcomes in ablation studies. Moreover, our method is capable of effectively integrating the discriminative information held by class-specific energy score and class-agnostic residual score under a wide range of scaling factors to achieve enhanced OOD detection performance and better audio-visual GZSL results than individual scores.

Report issue for preceding element

Figure 6: Effect of principal subspace dimension N on AUROC for three datasets. OOD detection performance of EZ-OOD method is quite robust with principal subspace dimension since the AUROC changes little towards a wide range of N values.

Report issue for preceding element

4.5.2 Effect of Principle Subspace Dimension N

Report issue for preceding element

Different N parameters will have a direct influence on the OOD detection performance of residual score, followed by producing different EZ-OOD scores and finally change the overall audio-visual GZSL results. Additionally, since the concatenated audio-visual feature is 1536-d and here we adopt different N as 32, 64, 128, 256, 384, 512 and 768 together with a fixed $\gamma$ for each benchmark, and the AUROC value is reported as the evaluation metric. As depicted in Figure 6, the fitted curves reach the peak at $N=64$, $N=256$ and $N=256$ for VGGSound-GZSL^cls, UCF-GZSL^cls and ActivityNet-GZSL^cls benchmarks respectively and are generally “flat” which indicates the proposed EZ-OOD method is less sensitive with the dimension N. As a result, we can select this hyperparameter from a wide range of numbers with little influence on the final GZSL performance, which convincingly validates the excellent robustness of our method.

Report issue for preceding element

5 Conclusion

Report issue for preceding element

In this paper, we propose an extremely simple OOD detection based model EZ-AVOOD for Audio-Visual Generalized Zero-Shot Learning (AV-GZSL) by ingeniously integrating the discriminative information held by class-specific logits and class-agnostic feature subspace. Superior experimental results on 3 audio-visual datasets fully demonstrate the effectiveness of our model. Moreover, the excellent compatibility of the proposed OOD detection method EZ-OOD is verified through deploying a different unseen classifier to construct a new model that outperforms the contrasting methods on both OOD detection performance and GZSL classification accuracy. Therefore, we conclude that EZ-AVOOD is new state-of-the-art of AV-GZSL.

Report issue for preceding element