Perception- and Fidelity-aware Reduced-Reference Super-Resolution Image Quality Assessment

Abstract

With the advent of image super-resolution (SR) algorithms, how to evaluate the quality of generated SR images has become an urgent task. Although full-reference methods perform well in SR image quality assessment (SR-IQA), their reliance on high-resolution (HR) images limits their practical applicability. Leveraging available reconstruction information as much as possible for SR-IQA, such as low-resolution (LR) images and the scale factors, is a promising way to enhance assessment performance for SR-IQA without HR for reference. In this letter, we attempt to evaluate the perceptual quality and reconstruction fidelity of SR images considering LR images and scale factors. Specifically, we propose a novel dual-branch reduced-reference SR-IQA network, i.e., Perception- and Fidelity-aware SR-IQA (PFIQA). The perception-aware branch evaluates the perceptual quality of SR images by leveraging the merits of global modeling of Vision Transformer (ViT) and local relation of ResNet, and incorporating the scale factor to enable comprehensive visual perception. Meanwhile, the fidelity-aware branch assesses the reconstruction fidelity between LR and SR images through their visual perception. The combination of the two branches substantially aligns with the human visual system, enabling a comprehensive SR image evaluation. Experimental results indicate that our PFIQA outperforms current state-of-the-art models across three widely-used SR-IQA benchmarks. Notably, PFIQA excels in assessing the quality of real-world SR images.

I Introduction

Image super-resolution (SR) technology aims to produce more detailed high-resolution (HR) images from the given low-resolution (LR) images. It has been widely used in various fields, such as security and surveillance[1], medical imaging[2], and remote sensing imaging[3, 4]. Given the remarkable advancements in recent research on SR algorithms, including blind SR[5, 6], lightweight SR models[7, 8], arbitrary scale SR[9], and multimodal SR[10, 11], it has become imperative to evaluate the quality of the generated SR images. This evaluation is crucial for facilitating comparative analysis of reconstruction performance across various SR models and guiding the development of SR algorithms.

Numerous methods have been proposed for image quality assessment (IQA) [12, 13, 14], which can be categorized as subjective or objective. While subjective ones are more reliable, they are impractical due to high costs and external factors. Hence, objective IQA methods that align with subjective evaluations are currently a focus of research. IQT[15], MANIQA[14], AHIQ[16], and MAMIQA[17] are recently proposed IQA methods that achieve excellent subjective consistency. While generic IQA methods have shown satisfactory results, they often neglect the specific characteristics of SR, making them unsuitable to directly apply to SR images. SR algorithms aim to recover detailed information from LR images, so SR-IQA needs to not only focus on the visual quality of the image, but also consider the consistency with the LR image, a factor that generic IQA methods overlook. At the same time, SR images often exhibit mixed degradations, such as blurring, ringing, and aliasing artifacts, which are not effectively addressed by current generic IQA methods.

Figure 1: Overview of proposed PFIQA method. This framework consists of Perception-aware and Fidelity-aware Assessment Branches for SR-IQA.

Recently, there has been an increase in the development of SR-IQA methods. Depending on the availability of lossless reference images, SR-IQA methods can be categorized into full-reference[18], reduced-reference[19, 11], and no-reference methods[20, 21]. For all three SR-IQA paradigms, evaluating the perceptual quality is of paramount importance, as it directly relates to human perceptual judgments and assessment of the given SR images[22, 18]. Moreover, reconstruction fidelity[23, 24] is also highly important since it reflects how faithfully the SR image represents the reference image details and content. Only reference-based SR-IQA methods can effectively consider reconstruction fidelity by comparing the SR image against the reference image. However, in practical scenarios where HR images are unavailable, the application of full-reference SR-IQA methods becomes challenging. Fortunately, the SR task inherently possesses available reference or auxiliary information, such as the paired LR image and the scale factor. These cues have significant reference value for evaluating SR images[18], but are not directly utilized by most SR-IQA methods[21, 20]. Recently, a few reduced-reference SR-IQA methods [19, 18, 25] have adopted CNN-based networks to get feature maps of LR and SR images, and simply calculated the similarity between the feature maps and utilized multi-layer perceptron to regress the quality scores. The entire process can be seen as a pixel-by-pixel comparison between the LR and SR images, which yet neglects the spatial coherence between the LR and SR images, resulting in an insufficient emphasis on reconstruction fidelity.

To address these issues, we present a novel reduced-reference Perception- and Fidelity-aware SR-IQA (PFIQA), which integrates LR images and scale factors as prior knowledge to assist in SR-IQA. Specifically, PFIQA consists of two assessment branches: the Perception-aware Assessment Branch (PA Branch) for evaluating perceptual quality and the Fidelity-aware Assessment Branch (FA Branch) for assessing reconstruction fidelity. Given that each patch holds unique visual details, we design a patch scoring module for each branch and a patch weighting module to assign varying degrees of attention to each patch, thus achieving fine-grained patch-wise prediction for SR-IQA. The outputs of the two branches are combined through a sum to provide a comprehensive evaluation of the SR images. Based on this delicate design, PFIQA can enhance the consistency between the network’s assessment results and human visual system (HVS).

Our main contributions can be summarized as follows:

• We introduce PFIQA, a novel dual-branch reduced-reference SR-IQA network that comprehensively assesses the perceptual quality and reconstruction fidelity of SR images without requiring any reference HR images.

• Our proposed PFIQA takes pairs of SR and LR images as input, leveraging the merits of global modeling of ViT and local relation of ResNet to enable comprehensive visual perception, and incorporates the scale factor to effectively align with HVS.

• Extensive experiments on three widely-used SR-IQA benchmarks demonstrate that PFIQA shows superior performance compared with other SR-IQA methods.

II Methodology

As shown in Fig. 1, our proposed PFIQA is composed of two assessment branches, underging three key phases: (1) feature extraction to acquire global and local visual features; (2) adaptive fusion of visual features and the scale factor to obtain fine-grained representations conducive for SR-IQA; (3) patch-weighted quality regression to produce a perception-aware score and a fidelity-aware score and perform a sum of them to obtain the final quality score.

II-A Feature Extraction

The first phase of PFIQA is feature extraction, which consists of pre-trained Vision Transformer (ViT)[26] and ResNet [27]. Understanding the broader context of an image is crucial for the IQA task as it provides essential information about the overall structure and content. ViT excels in this aspect by primarily focusing on extracting global visual features. Its self-attention mechanism effectively captures long-range dependencies and encodes images into comprehensive global feature representations, which are valuable for SR-IQA. In addition to global features, paying attention to fine-grained details is equally important, especially in SR-IQA where humans tend to focus on intricate elements. Incorporating local visual information alongside global features can significantly enhance the accuracy and robustness of SR-IQA methods. Inspired by previous work [28, 29], we augment ViT with ResNet to better capture local visual features, thereby enriching the model’s ability to comprehensively represent images.

Specifically, a pair of LR and SR images are fed into ViT and ResNet, and then we take out the ViT feature maps of the early stage and ResNet feature maps of different scales. Due to the similarity in the feature extraction process between SR and LR images, we illustrate the process taking the SR image as an example. For ViT, we utilize the outputs from five intermediate blocks $\{0,1,2,3,4\}$ within ViT. The output feature from each block $f_{sr}\in\mathbb{R}^{p\times p\times c}$, where $c=768$, $p=28$, is concatenated into $f_{sr}^{G}\in\mathbb{R}^{p\times p\times 5c}$, and then reshaped into $f_{sr}^{G^{\prime}}\in\mathbb{R}^{5c\times p\times p}$. For ResNet, we extract multi-scale features from four different stages of ResNet, and then interpolate these features to resize them to the same dimensions. After that, we concatenate them along the channel dimension and reshape them to obtain $f_{sr}^{L^{\prime}}\in\mathbb{R}^{3840\times p\times p}$. We apply a convolutional dimensionality reduction to get the final output global features $f_{sr}^{G}\in\mathbb{R}^{256\times p\times p}$ and local features $f_{sr}^{L}\in\mathbb{R}^{256\times p\times p}$. Similarly, the paired LR image undergoes the same process, resulting in the output of global features $f_{lr}^{G}$ and local features $f_{lr}^{L}$.

For PA branch, the target is to evaluate the perceptual quality of the SR image itself. Therefore, we utilize the extracted feature of the SR images $f_{sr}^{G}$ and $f_{sr}^{L}$ as input to PA branch. In contrast, for FA branch, in order to measure the fidelity between the SR image and its paired LR image, we take into consideration that LR and SR images exhibit differences in the feature space. After obtaining the global and local visual features extracted respectively by ViT and ResNet, we represent the features $f_{diff}^{L}=f_{sr}^{L}-f_{lr}^{L}$ and $f_{diff}^{G}=f_{sr}^{G}-f_{lr}^{G}$ as input to FA Branch. These features are represented as the difference between the global and local features to capture the distinction between the SR image and its paired LR image.

II-B Adaptive Fusion

To effectively combine the extracted visual features for the two branches, as well as auxiliary information from the scale factor for SR-IQA, we employ Adaptive Fusion Modules (AFMs). For each branch, the global and local features along with the scale factor, are utilized as inputs to the AFM. Since the processing by AFM in two branches is similar, with the only difference lying in their respective inputs, we provide a detailed description of the AFM in PA branch as an example. As shown in Fig. 1 (right), we first adaptively fuse global and local visual features, and then incorporate the scale factor information.

Global and Local Visual Features. The AFM is capable of learning the importance of features from global and local features and adaptively assigning appropriate weights to them, thereby obtaining comprehensive visual features for SR-IQA.

For PA branch, global features ($f_{sr}^{G}$) and local features ($f_{sr}^{L}$) are first fed into the AFM and then concatenated along a new dimension. Then, to effectively learn the weights of global and local features to fuse the two features adaptively, we employ a fusion operation implemented by a fully-connected layer followed by a ReLU activation function, which yields the features $f_{sr}$. Similarly, for FA branch, we fuse $f_{diff}^{G}$ and $f_{diff}^{L}$ to obtain $f_{diff}$.

Scale Factor. The scale factor has a statistically significant influence on the subjective quality scores of SR images [20], suggesting its potential as a valuable indicator for assessing the quality of SR images and guiding SR-IQA. Therefore, we input the vector representing the scale factor $S$ into the AFM to generate SR-IQA-related features. First, the scale factor is passed through two fully-connected layers and then reshaped to obtain the scale factor feature $f_{S}$.

Subsequently, for PA branch, the scale factor feature $f_{S}$ is concatenated along the channel dimension with the features $f_{sr}$. The concatenated features then go through a $3\times 3$ convolution, a ReLU activation, and another $3\times 3$ convolution to obtain the perceptual features $f_{sr}^{S}$. Similarly, for FA branch, the differential features $f_{diff}^{S}$ are obtained.

II-C Patch-Weighted Quality Regression

The final phase of PFIQA is patch-weighted quality regression, which comprises two patch scoring modules and a patch weighting module to generate the score $S_{PF}$ of the input SR image. Since each pixel in the deep feature map corresponds to a distinct patch of the input image and encapsulates abundant information, the spatial dimension’s information is crucial. To capture the relationships between image patches, we predict scores for each pixel in the feature maps of the two branches and calculate the attention maps for each corresponding score. We obtain the scores for the two branches separately by performing a weighted sum of the individual scores. The weighted summation is employed to capture the significance of each region, simulating the behavior of the HVS. Finally, we add the scores from these two complementary branches to obtain the final predicted score.

Specifically, patch scoring module employs perceptual features $f_{sr}^{S}$ and differential features $f_{diff}^{S}$ to evaluate the perception and fidelity of the SR image in a patch-wise manner. Taking the PA branch as an example, the visual features $f_{sr}^{S}$ pass through a $3\times 3$ convolution, followed by a ReLU activation, and another $3\times 3$ convolution to produce the perception-aware score map $s_{p}$. Similarly, for the FA branch, the process yields the fidelity-aware score map $s_{f}$.

In parallel, the perceptual features $f_{sr}^{S}$ and differential features $f_{diff}^{S}$ are concatenated and fed into a patch weighting module, which computes the perception-aware score weight map $w_{p}$ and the fidelity-aware score weight map $w_{f}$ for each image patch. This process can be represented as:

$$\leavevmode\resizebox{422.77661pt}{}{$w=Sigmoid(Conv1(ReLU(Conv3(Concat(f_{sr}^{S},f_{diff}^{S})))))$},$$ (1)

where $Conv1$ and $Conv3$ represent $1\times 1$ and $3\times 3$ convolution, the score weight map $w$ is a two-channel tensor, with the first channel representing the perception-aware score weight map $w_{p}$ and the second channel representing the fidelity-aware score weight map $w_{f}$, i.e., $w=[w_{p},w_{f}]$.

Finally, we utilize the two score weight maps to perform weighted summation on the perception-aware score map and fidelity-aware score map and then add them to obtain the final predicted quality score $S_{PF}$. This can be represented as:

$$S_{PF}=\frac{s_{p}*w_{p}}{\sum w_{p}}+\frac{s_{f}*w_{f}}{\sum w_{f}},$$ (2)

where $*$ means Hadamard product.

TABLE I: Performance comparison on three benchmark datasets. We highlight the best and the second results. Algorithms marked with asterisks (*) have been reproduced, while unmarked ones use results from the original paper.

Type	Reference	Methods	QADS[30]		WIND[31]		RealSRQ[32]		Average
			PLCC	SRCC	PLCC	SRCC	PLCC	SRCC	PLCC	SRCC
Generic IQA	$\times$	NIQE*[13]	0.4055	0.3916	0.6286	0.5520	0.1452	0.0983	0.3931	0.3473
	$\times$	BRISQUE*[12]	0.5568	0.5851	0.5397	0.5085	0.0515	0.0075	0.3827	0.3670
	$\times$	CNNIQA*[33]	0.8791	0.8742	0.9328	0.8902	0.6711	0.6671	0.8277	0.8105
	$\times$	MANIQA*[14]	0.9804	0.9771	0.9771	0.9321	0.8631	0.7812	0.9402	0.8634
	HR	SIS [30]	0.9137	0.9132	0.8913	0.8777	-	-	-	-
SR-IQA	$\times$	NRQM*[34]	0.6867	0.6898	0.6869	0.6201	0.1451	0.0042	0.5062	0.4380
	$\times$	C2MT [21]	0.9719	0.9690	-	-	0.7184	0.7043	-	-
	$\times$	SFSN [35]	0.9174	0.9163	0.9525	0.9157	-	-	-	-
	$\times$	SGH*[20]	0.9362	0.9475	0.9733	0.9563	0.6847	0.7069	0.8647	0.8702
	LR	DISQ*[19]	0.9127	0.9110	0.9785	0.9333	0.7885	0.7175	0.8932	0.8539
	LR	PSCT[18]	0.9620	0.9600	0.9596	0.9290	-	-	-	-
	LR	PFIQA (Ours)	0.9830	0.9815	0.9827	0.9637	0.9269	0.8597	0.9642	0.9350

III Experiments

III-A Experimental Settings

Datasets and Evaluation Metrics. The evaluations are implemented on three datasets which are commonly used in the research of SR-IQA, including QADS[30], WIND[31] and RealSRQ[32]. We split the datasets randomly into training and test sets at an 8:2 ratio, repeat the partitioning and evaluation process 5 times for fair comparison, and report the average results as the final performance. We utilize two of the most commonly used metrics, including Pearson Linear Correlation Coefficient (PLCC) and Spearman Rankorder Correlation Coefficient (SRCC).

Implementation Details. We use ViT-B [26] and ResNet50 [27] models pre-trained on ImageNet [36], which are not updated during training. We bilinearly interpolate the LR image to achieve the same resolution as the SR image. For model training, we normalize all input images and randomly crop them into a size of $224\times 224$. Additionally, a random horizontal flip rotation is applied to augment the training data. To optimize the model, we utilize the MSE loss between the predicted score and the ground truth score. During the test phase, we crop the four corners and the center of the original image, and the final score is the average of the scores from these cropped sections. We train PFIQA on a single NVIDIA GeForce RTX3090 GPU. The training process utilizes the AdamW optimizer. The minibatch size is set to 4.

III-B Experimental Results and Analysis

TABLE II: Comparison of leveraging available information of SR algorithms on RealSRQ[32].

#	Assessment Branch		Scale Factor	PLCC	SRCC
	Perception	Fidelity
(a)	✓			0.9007	0.8070
(b)		✓		0.9233	0.8564
(c)	✓	✓		0.9242	0.8573
(d)	✓	✓	✓	0.9269	0.8597

Quantitative Comparison. The primary experimental results are reported in Table I, from which we can observe that: (1) On the whole, SR-IQA methods outperform most generic IQA methods because there are specifically designed for SR, including the model architecture and training data. (2) The proposed PFIQA model surpasses the compared SR-IQA methods on three benchmarks. This can be attributed to the fact that other SR-IQA methods do not consider the integration between perception and fidelity awareness when evaluating the quality of SR images. (3) When evaluated on the real-world SR-IQA dataset RealSRQ, PFIQA showcases a substantial improvement compared to other methods. The PLCC and SRCC metrics show an improvement of approximately 7.39$\%$ and 10.05$\%$ compared to the second-best MANIQA[14], which illustrates the superiority of PFIQA in real-world scenarios.

Effects of Reference Information. To further investigate the impact of reference information for SR-IQA, we compare the performance of PFIQA utilizing different reference information or not. In Table II, we can see that: (1) Using only the PA Branch (Table II (a)) can be viewed as a no-reference SR-IQA method, focusing solely on the perceptual quality of the SR image itself, hence leading to sub-optimal performance. (2) Considering the LR image as reference and using only the FA Branch (Table II (b)) can be viewed as a plain reduced-reference SR-IQA method, which accounts for the consistency between the SR and LR images, thereby ensuring reconstruction fidelity to a certain extent. Compared to Table II (a), this brings a significant performance improvement. (3) Combining the PA Branch and FA Branch (Table II (c)), integrates both perceptual quality and reconstruction fidelity, thus obtaining better performance compared to Table II (a,b). (4) Building upon considering the LR image as the reference, and additionally taking into account the scale factor (Table II (d)), PFIQA maximally leverages the available information from SR algorithms, thereby achieving the best SR-IQA performance.

TABLE III: Comparison of different visual feature extraction and fusion strategies on RealSRQ[32].

#	Feature		Fusion Method	PLCC	SRCC
	ResNet	ViT
(a)	✓		-	0.9070	0.8302
(b)		✓	-	0.9010	0.8105
(c)	✓	✓	Concatenation	0.9198	0.8460
(d)	✓	✓	Adaptive Fusion	0.9269	0.8597

Effects of Different Visual Feature Extraction Methods and Fusion Strategies. Since we use ViT and ResNet to extract global and local visual features from images, it is necessary to investigate how they impact the performance of SR-IQA. In Table III, we can find that: (1) Using only each of the visual features extracted by ViT and ResNet (Table III (a,b)) can achieve satisfactory SR-IQA results compared to other SOTA methods in Table I. (2) Utilizing both global and local visual features (Table III (c)), leads to better performance compared to Table III (a,b). This demonstrates the complementary nature of global and local features for comprehensive SR-IQA. (3) Further analysis of the adaptive fusion of global and local features, as indicated in Table III (d), reveals that PFIQA attains superior performance. This demonstrates the efficacy of the AFMs in learning the importance of features from global and local features.

IV Conclusion

In this letter, we propose a novel reduced-reference Perception- and Fidelity-aware SR-IQA (PFIQA) network, which integrates LR images and the scale factors as prior knowledge to comprehensively assess the perceptual quality and reconstruction fidelity of SR images. We leverage the merits of global modeling of ViT and local relation of CNN to enable comprehensive visual feature extraction, and incorporate the scale factors to obtain features that are more relevant to SR-IQA. The extensive results from three benchmark datasets showcase the efficacy of PFIQA.