Showing 1–50 of 194 results for author: Zhang, F

Serial coherent diffraction imaging of dynamic samples based on inter-frame constraint

Authors: Pengju Sheng, Fucai Zhang

Abstract: We proposed a novel approach to coherent imaging of dynamic samples. The inter-frame similarity of the sample's local structures is found to be a powerful constraint in phasing a sequence of diffraction patterns. We devised a new image reconstruction algorithm that exploits this inter-frame constraint enabled by an adaptive similar region determination approach. We demonstrated the feasibility of… ▽ More We proposed a novel approach to coherent imaging of dynamic samples. The inter-frame similarity of the sample's local structures is found to be a powerful constraint in phasing a sequence of diffraction patterns. We devised a new image reconstruction algorithm that exploits this inter-frame constraint enabled by an adaptive similar region determination approach. We demonstrated the feasibility of this technique in visible light experiments with various real samples, achieving reconstructions of good quality within a few hundred iterations. With a setup as simple as conventional coherent diffraction imaging but with much-improved convergence and robustness to missing data and noise, our method is expected to enrich X-ray imaging techniques and electron microscopy, offering a new tool for dynamics studies. △ Less

Submitted 9 July, 2024; originally announced July 2024.

LDGCN: An Edge-End Lightweight Dual GCN Based on Single-Channel EEG for Driver Drowsiness Monitoring

Authors: Jingwei Huang, Chuansheng Wang, Jiayan Huang, Haoyi Fan, Antoni Grau, Fuquan Zhang

Abstract: Driver drowsiness electroencephalography (EEG) signal monitoring can timely alert drivers of their drowsiness status, thereby reducing the probability of traffic accidents. Graph convolutional networks (GCNs) have shown significant advancements in processing the non-stationary, time-varying, and non-Euclidean nature of EEG signals. However, the existing single-channel EEG adjacency graph construct… ▽ More Driver drowsiness electroencephalography (EEG) signal monitoring can timely alert drivers of their drowsiness status, thereby reducing the probability of traffic accidents. Graph convolutional networks (GCNs) have shown significant advancements in processing the non-stationary, time-varying, and non-Euclidean nature of EEG signals. However, the existing single-channel EEG adjacency graph construction process lacks interpretability, which hinders the ability of GCNs to effectively extract adjacency graph features, thus affecting the performance of drowsiness monitoring. To address this issue, we propose an edge-end lightweight dual graph convolutional network (LDGCN). Specifically, we are the first to incorporate neurophysiological knowledge to design a Baseline Drowsiness Status Adjacency Graph (BDSAG), which characterizes driver drowsiness status. Additionally, to express more features within limited EEG data, we introduce the Augmented Graph-level Module (AGM). This module captures global and local information at the graph level, ensuring that BDSAG features remain intact while enhancing effective feature expression capability. Furthermore, to deploy our method on the fourth-generation Raspberry Pi, we utilize Adaptive Pruning Optimization (APO) on both channels and neurons, reducing inference latency by almost half. Experiments on benchmark datasets demonstrate that LDGCN offers the best trade-off between monitoring performance and hardware resource utilization compared to existing state-of-the-art algorithms. All our source code can be found at https://github.com/BryantDom/Driver-Drowsiness-Monitoring. △ Less

Submitted 8 July, 2024; originally announced July 2024.

Depth-Aware Endoscopic Video Inpainting

Authors: Francis Xiatian Zhang, Shuang Chen, Xianghua Xie, Hubert P. H. Shum

Abstract: Video inpainting fills in corrupted video content with plausible replacements. While recent advances in endoscopic video inpainting have shown potential for enhancing the quality of endoscopic videos, they mainly repair 2D visual information without effectively preserving crucial 3D spatial details for clinical reference. Depth-aware inpainting methods attempt to preserve these details by incorpor… ▽ More Video inpainting fills in corrupted video content with plausible replacements. While recent advances in endoscopic video inpainting have shown potential for enhancing the quality of endoscopic videos, they mainly repair 2D visual information without effectively preserving crucial 3D spatial details for clinical reference. Depth-aware inpainting methods attempt to preserve these details by incorporating depth information. Still, in endoscopic contexts, they face challenges including reliance on pre-acquired depth maps, less effective fusion designs, and ignorance of the fidelity of 3D spatial details. To address them, we introduce a novel Depth-aware Endoscopic Video Inpainting (DAEVI) framework. It features a Spatial-Temporal Guided Depth Estimation module for direct depth estimation from visual features, a Bi-Modal Paired Channel Fusion module for effective channel-by-channel fusion of visual and depth information, and a Depth Enhanced Discriminator to assess the fidelity of the RGB-D sequence comprised of the inpainted frames and estimated depth images. Experimental evaluations on established benchmarks demonstrate our framework's superiority, achieving a 2% improvement in PSNR and a 6% reduction in MSE compared to state-of-the-art methods. Qualitative analyses further validate its enhanced ability to inpaint fine details, highlighting the benefits of integrating depth information into endoscopic inpainting. △ Less

Submitted 2 July, 2024; originally announced July 2024.

Comments: Accepted by MICCAI 2024

CEC: A Noisy Label Detection Method for Speaker Recognition

Authors: Yao Shen, Yingying Gao, Yaqian Hao, Chenguang Hu, Fulin Zhang, Junlan Feng, Shilei Zhang

Abstract: Noisy labels are inevitable, even in well-annotated datasets. The detection of noisy labels is of significant importance to enhance the robustness of speaker recognition models. In this paper, we propose a novel noisy label detection approach based on two new statistical metrics: Continuous Inconsistent Counting (CIC) and Total Inconsistent Counting (TIC). These metrics are calculated through Cros… ▽ More Noisy labels are inevitable, even in well-annotated datasets. The detection of noisy labels is of significant importance to enhance the robustness of speaker recognition models. In this paper, we propose a novel noisy label detection approach based on two new statistical metrics: Continuous Inconsistent Counting (CIC) and Total Inconsistent Counting (TIC). These metrics are calculated through Cross-Epoch Counting (CEC) and correspond to the early and late stages of training, respectively. Additionally, we categorize samples based on their prediction results into three categories: inconsistent samples, hard samples, and easy samples. During training, we gradually increase the difficulty of hard samples to update model parameters, preventing noisy labels from being overfitted. Compared to contrastive schemes, our approach not only achieves the best performance in speaker verification but also excels in noisy label detection. △ Less

Submitted 19 June, 2024; originally announced June 2024.

Comments: interspeech 2024

Communication-Efficient MARL for Platoon Stability and Energy-efficiency Co-optimization in Cooperative Adaptive Cruise Control of CAVs

Authors: Min Hua, Dong Chen, Kun Jiang, Fanggang Zhang, Jinhai Wang, Bo Wang, Quan Zhou, Hongming Xu

Abstract: Cooperative adaptive cruise control (CACC) has been recognized as a fundamental function of autonomous driving, in which platoon stability and energy efficiency are outstanding challenges that are difficult to accommodate in real-world operations. This paper studied the CACC of connected and autonomous vehicles (CAVs) based on the multi-agent reinforcement learning algorithm (MARL) to optimize pla… ▽ More Cooperative adaptive cruise control (CACC) has been recognized as a fundamental function of autonomous driving, in which platoon stability and energy efficiency are outstanding challenges that are difficult to accommodate in real-world operations. This paper studied the CACC of connected and autonomous vehicles (CAVs) based on the multi-agent reinforcement learning algorithm (MARL) to optimize platoon stability and energy efficiency simultaneously. The optimal use of communication bandwidth is the key to guaranteeing learning performance in real-world driving, and thus this paper proposes a communication-efficient MARL by incorporating the quantified stochastic gradient descent (QSGD) and a binary differential consensus (BDC) method into a fully-decentralized MARL framework. We benchmarked the performance of our proposed BDC-MARL algorithm against several several non-communicative andcommunicative MARL algorithms, e.g., IA2C, FPrint, and DIAL, through the evaluation of platoon stability, fuel economy, and driving comfort. Our results show that BDC-MARL achieved the highest energy savings, improving by up to 5.8%, with an average velocity of 15.26 m/s and an inter-vehicle spacing of 20.76 m. In addition, we conducted different information-sharing analyses to assess communication efficacy, along with sensitivity analyses and scalability tests with varying platoon sizes. The practical effectiveness of our approach is further demonstrated using real-world scenarios sourced from open-sourced OpenACC. △ Less

Submitted 17 June, 2024; originally announced June 2024.

A high-performance reconstruction method for partially coherent ptychography

Authors: Wenhui Xu, Shoucong Ning, Pengju Sheng, Huixiang Lin, Angus I Kirkland, Yong Peng, Fucai Zhang

Abstract: Ptychography is now integrated as a tool in mainstream microscopy allowing quantitative and high-resolution imaging capabilities over a wide field of view. However, its ultimate performance is inevitably limited by the available coherent flux when implemented using electrons or laboratory X-ray sources. We present a universal reconstruction algorithm with high tolerance to low coherence for both f… ▽ More Ptychography is now integrated as a tool in mainstream microscopy allowing quantitative and high-resolution imaging capabilities over a wide field of view. However, its ultimate performance is inevitably limited by the available coherent flux when implemented using electrons or laboratory X-ray sources. We present a universal reconstruction algorithm with high tolerance to low coherence for both far-field and near-field ptychography. The approach is practical for partial temporal and spatial coherence and requires no prior knowledge of the source properties. Our initial visible-light and electron data show that the method can dramatically improve the reconstruction quality and accelerate the convergence rate of the reconstruction. The approach also integrates well into existing ptychographic engines. It can also improve mixed-state and numerical monochromatisation methods, requiring a smaller number of coherent modes or lower dimensionality of Krylov subspace while providing more stable and faster convergence. We propose that this approach could have significant impact on ptychography of weakly scattering samples. △ Less

Submitted 9 June, 2024; originally announced June 2024.

MVAD: A Multiple Visual Artifact Detector for Video Streaming

Authors: Chen Feng, Duolikun Danier, Fan Zhang, David Bull

Abstract: Visual artifacts are often introduced into streamed video content, due to prevailing conditions during content production and/or delivery. Since these can degrade the quality of the user's experience, it is important to automatically and accurately detect them in order to enable effective quality measurement and enhancement. Existing detection methods often focus on a single type of artifact and/o… ▽ More Visual artifacts are often introduced into streamed video content, due to prevailing conditions during content production and/or delivery. Since these can degrade the quality of the user's experience, it is important to automatically and accurately detect them in order to enable effective quality measurement and enhancement. Existing detection methods often focus on a single type of artifact and/or determine the presence of an artifact through thresholding objective quality indices. Such approaches have been reported to offer inconsistent prediction performance and are also impractical for real-world applications where multiple artifacts co-exist and interact. In this paper, we propose a Multiple Visual Artifact Detector, MVAD, for video streaming which, for the first time, is able to detect multiple artifacts using a single framework that is not reliant on video quality assessment models. Our approach employs a new Artifact-aware Dynamic Feature Extractor (ADFE) to obtain artifact-relevant spatial features within each frame for multiple artifact types. The extracted features are further processed by a Recurrent Memory Vision Transformer (RMViT) module, which captures both short-term and long-term temporal information within the input video. The proposed network architecture is optimized in an end-to-end manner based on a new, large and diverse training database that is generated by simulating the video streaming pipeline and based on Adversarial Data Augmentation. This model has been evaluated on two video artifact databases, Maxwell and BVI-Artifact, and achieves consistent and improved prediction results for ten target visual artifacts when compared to seven existing single and multiple artifact detectors. The source code and training database will be available at https://chenfeng-bristol.github.io/MVAD/. △ Less

Submitted 31 May, 2024; originally announced June 2024.

Comments: 9 pages

Image-based retrieval of all-day cloud physical parameters for FY4A/AGRI and its application over the Tibetan Plateau

Authors: Zhijun Zhao, Feng Zhang, Wenwen Li, Jingwei Li

Abstract: Satellite remote sensing serves as a crucial means to acquire cloud physical parameters. However, existing official cloud products derived from the advanced geostationary radiation imager (AGRI) onboard the Fengyun-4A geostationary satellite suffer from limitations in computational precision and efficiency. In this study, an image-based transfer learning model (ITLM) was developed to realize all-d… ▽ More Satellite remote sensing serves as a crucial means to acquire cloud physical parameters. However, existing official cloud products derived from the advanced geostationary radiation imager (AGRI) onboard the Fengyun-4A geostationary satellite suffer from limitations in computational precision and efficiency. In this study, an image-based transfer learning model (ITLM) was developed to realize all-day and high-precision retrieval of cloud physical parameters using AGRI thermal infrared measurements and auxiliary data. Combining the observation advantages of geostationary and polar-orbiting satellites, ITLM was pre-trained and transfer-trained with official cloud products from advanced Himawari imager (AHI) and Moderate Resolution Imaging Spectroradiometer (MODIS), respectively. Taking official MODIS products as the benchmarks, ITLM achieved an overall accuracy of 79.93% for identifying cloud phase and root mean squared errors of 1.85 km, 6.72 um, and 12.79 for estimating cloud top height, cloud effective radius, and cloud optical thickness, outperforming the precision of official AGRI and AHI products. Compared to the pixel-based random forest model, ITLM utilized the spatial information of clouds to significantly improve the retrieval performance and achieve more than a 6-fold increase in speed for a single full-disk retrieval. Moreover, the AGRI ITLM products with spatiotemporal continuity and high precision were used to accurately describe the spatial distribution characteristics of cloud fractions and cloud properties over the Tibetan Plateau (TP) during both daytime and nighttime, and for the first time provide insights into the diurnal variation of cloud cover and cloud properties for total clouds and deep convective clouds across different seasons. △ Less

Submitted 28 March, 2024; originally announced May 2024.

Survey on Visual Signal Coding and Processing with Generative Models: Technologies, Standards and Optimization

Authors: Zhibo Chen, Heming Sun, Li Zhang, Fan Zhang

Abstract: This paper provides a survey of the latest developments in visual signal coding and processing with generative models. Specifically, our focus is on presenting the advancement of generative models and their influence on research in the domain of visual signal coding and processing. This survey study begins with a brief introduction of well-established generative models, including the Variational A… ▽ More This paper provides a survey of the latest developments in visual signal coding and processing with generative models. Specifically, our focus is on presenting the advancement of generative models and their influence on research in the domain of visual signal coding and processing. This survey study begins with a brief introduction of well-established generative models, including the Variational Autoencoder (VAE) models, Generative Adversarial Network (GAN) models, Autoregressive (AR) models, Normalizing Flows and Diffusion models. The subsequent section of the paper explores the advancements in visual signal coding based on generative models, as well as the ongoing international standardization activities. In the realm of visual signal processing, our focus lies on the application and development of various generative models in the research of visual signal restoration. We also present the latest developments in generative visual signal synthesis and editing, along with visual signal quality assessment using generative models and quality assessment for generative models. The practical implementation of these studies is closely linked to the investigation of fast optimization. This paper additionally presents the latest advancements in fast optimization on visual signal coding and processing with generative models. We hope to advance this field by providing researchers and practitioners a comprehensive literature review on the topic of visual signal coding and processing with generative models. △ Less

Submitted 23 May, 2024; originally announced May 2024.

RMT-BVQA: Recurrent Memory Transformer-based Blind Video Quality Assessment for Enhanced Video Content

Authors: Tianhao Peng, Chen Feng, Duolikun Danier, Fan Zhang, David Bull

Abstract: With recent advances in deep learning, numerous algorithms have been developed to enhance video quality, reduce visual artefacts and improve perceptual quality. However, little research has been reported on the quality assessment of enhanced content - the evaluation of enhancement methods is often based on quality metrics that were designed for compression applications. In this paper, we propose a… ▽ More With recent advances in deep learning, numerous algorithms have been developed to enhance video quality, reduce visual artefacts and improve perceptual quality. However, little research has been reported on the quality assessment of enhanced content - the evaluation of enhancement methods is often based on quality metrics that were designed for compression applications. In this paper, we propose a novel blind deep video quality assessment (VQA) method specifically for enhanced video content. It employs a new Recurrent Memory Transformer (RMT) based network architecture to obtain video quality representations, which is optimised through a novel content-quality-aware contrastive learning strategy based on a new database containing 13K training patches with enhanced content. The extracted quality representations are then combined through linear regression to generate video-level quality indices. The proposed method, RMT-BVQA, has been evaluated on the VDPVE (VQA Dataset for Perceptual Video Enhancement) database through a five-fold cross validation. The results show its superior correlation performance when compared to ten existing no-reference quality metrics. △ Less

Submitted 15 May, 2024; v1 submitted 14 May, 2024; originally announced May 2024.

Comments: 8pages, 2figures

Near-Field Channel Characterization for Mid-band ELAA Systems: Sounding, Parameter Estimation, and Modeling

Authors: Wei Fan, Zhiqiang Yuan, Yejian Lyu, Jianhua Zhang, Gert Pedersen, Jonathan Borrill, Fengchun Zhang

Abstract: 6G communication will greatly benefit from using extremely large-scale antenna arrays (ELAAs) and new mid-band spectrums (7-24 GHz). These techniques require a thorough exploration of the challenges and potentials of the associated near-field (NF) phenomena. It is crucial to develop accurate NF channel models that include spherical wave propagation and spatial non-stationarity (SnS). However, chan… ▽ More 6G communication will greatly benefit from using extremely large-scale antenna arrays (ELAAs) and new mid-band spectrums (7-24 GHz). These techniques require a thorough exploration of the challenges and potentials of the associated near-field (NF) phenomena. It is crucial to develop accurate NF channel models that include spherical wave propagation and spatial non-stationarity (SnS). However, channel measurement campaigns for mid-band ELAA systems have rarely been reported in the state-of-the-art. To this end, this work develops a channel sounder dedicated to mid-band ELAA systems based on a distributed modular vector network analyzer incorporating radio-over-fiber (RoF), phase compensation, and virtual antenna array schemes. This novel channel-sounding testbed based on off-the-shelf VNA has the potential to enable large-scale experimentation due to its generic and easy-accessible nature. The main challenges and solutions for developing NF channel models for mid-band ELAA systems are discussed, including channel sounders, multipath parameter estimation algorithms, and channel modeling frameworks. Besides, the study reports a measurement campaign in an indoor scenario using a 720-element virtual uniform circular array ELAA operating at {16-20} GHz, highlighting the presence of spherical wavefronts and spatial non-stationary effects. The effectiveness of the proposed near-field channel parameter estimator and channel modeling framework is also demonstrated using the measurement data. △ Less

Submitted 9 May, 2024; originally announced May 2024.

Comments: Submitted to IEEE Communication Magazine

Data-Driven Dynamics Modeling of Miniature Robotic Blimps Using Neural ODEs With Parameter Auto-Tuning

Authors: Yongjian Zhu, Hao Cheng, Feitian Zhang

Abstract: Miniature robotic blimps, as one type of lighter-than-air aerial vehicles, have attracted increasing attention in the science and engineering community for their enhanced safety, extended endurance, and quieter operation compared to quadrotors. Accurately modeling the dynamics of these robotic blimps poses a significant challenge due to the complex aerodynamics stemming from their large lifting bo… ▽ More Miniature robotic blimps, as one type of lighter-than-air aerial vehicles, have attracted increasing attention in the science and engineering community for their enhanced safety, extended endurance, and quieter operation compared to quadrotors. Accurately modeling the dynamics of these robotic blimps poses a significant challenge due to the complex aerodynamics stemming from their large lifting bodies. Traditional first-principle models have difficulty obtaining accurate aerodynamic parameters and often overlook high-order nonlinearities, thus coming to its limit in modeling the motion dynamics of miniature robotic blimps. To tackle this challenge, this letter proposes the Auto-tuning Blimp-oriented Neural Ordinary Differential Equation method (ABNODE), a data-driven approach that integrates first-principle and neural network modeling. Spiraling motion experiments of robotic blimps are conducted, comparing the ABNODE with first-principle and other data-driven benchmark models, the results of which demonstrate the effectiveness of the proposed method. △ Less

Submitted 29 April, 2024; originally announced April 2024.

Comments: 8 pages, 8 figures

Disturbance Rejection-Guarded Learning for Vibration Suppression of Two-Inertia Systems

Authors: Fan Zhang, Jinfeng Chen, Yu Hu, Zhiqiang Gao, Ge Lv, Qin Lin

Abstract: Model uncertainty presents significant challenges in vibration suppression of multi-inertia systems, as these systems often rely on inaccurate nominal mathematical models due to system identification errors or unmodeled dynamics. An observer, such as an extended state observer (ESO), can estimate the discrepancy between the inaccurate nominal model and the true model, thus improving control perfor… ▽ More Model uncertainty presents significant challenges in vibration suppression of multi-inertia systems, as these systems often rely on inaccurate nominal mathematical models due to system identification errors or unmodeled dynamics. An observer, such as an extended state observer (ESO), can estimate the discrepancy between the inaccurate nominal model and the true model, thus improving control performance via disturbance rejection. The conventional observer design is memoryless in the sense that once its estimated disturbance is obtained and sent to the controller, the datum is discarded. In this research, we propose a seamless integration of ESO and machine learning. On one hand, the machine learning model attempts to model the disturbance. With the assistance of prior information about the disturbance, the observer is expected to achieve faster convergence in disturbance estimation. On the other hand, machine learning benefits from an additional assurance layer provided by the ESO, as any imperfections in the machine learning model can be compensated for by the ESO. We validated the effectiveness of this novel learning-for-control paradigm through simulation and physical tests on two-inertial motion control systems used for vibration studies. △ Less

Submitted 15 April, 2024; originally announced April 2024.

MTKD: Multi-Teacher Knowledge Distillation for Image Super-Resolution

Authors: Yuxuan Jiang, Chen Feng, Fan Zhang, David Bull

Abstract: Knowledge distillation (KD) has emerged as a promising technique in deep learning, typically employed to enhance a compact student network through learning from their high-performance but more complex teacher variant. When applied in the context of image super-resolution, most KD approaches are modified versions of methods developed for other computer vision tasks, which are based on training stra… ▽ More Knowledge distillation (KD) has emerged as a promising technique in deep learning, typically employed to enhance a compact student network through learning from their high-performance but more complex teacher variant. When applied in the context of image super-resolution, most KD approaches are modified versions of methods developed for other computer vision tasks, which are based on training strategies with a single teacher and simple loss functions. In this paper, we propose a novel Multi-Teacher Knowledge Distillation (MTKD) framework specifically for image super-resolution. It exploits the advantages of multiple teachers by combining and enhancing the outputs of these teacher models, which then guides the learning process of the compact student network. To achieve more effective learning performance, we have also developed a new wavelet-based loss function for MTKD, which can better optimize the training process by observing differences in both the spatial and frequency domains. We fully evaluate the effectiveness of the proposed method by comparing it to five commonly used KD methods for image super-resolution based on three popular network architectures. The results show that the proposed MTKD method achieves evident improvements in super-resolution performance, up to 0.46dBでしべる (based on PSNR), over state-of-the-art KD approaches across different network structures. The source code of MTKD will be made available here for public evaluation. △ Less

Submitted 15 April, 2024; originally announced April 2024.

Cross-domain Fiber Cluster Shape Analysis for Language Performance Cognitive Score Prediction

Authors: Yui Lo, Yuqian Chen, Dongnan Liu, Wan Liu, Leo Zekelman, Fan Zhang, Yogesh Rathi, Nikos Makris, Alexandra J. Golby, Weidong Cai, Lauren J. O'Donnell

Abstract: Shape plays an important role in computer graphics, offering informative features to convey an object's morphology and functionality. Shape analysis in brain imaging can help interpret structural and functionality correlations of the human brain. In this work, we investigate the shape of the brain's 3D white matter connections and its potential predictive relationship to human cognitive function.… ▽ More Shape plays an important role in computer graphics, offering informative features to convey an object's morphology and functionality. Shape analysis in brain imaging can help interpret structural and functionality correlations of the human brain. In this work, we investigate the shape of the brain's 3D white matter connections and its potential predictive relationship to human cognitive function. We reconstruct brain connections as sequences of 3D points using diffusion magnetic resonance imaging (dMRI) tractography. To describe each connection, we extract 12 shape descriptors in addition to traditional dMRI connectivity and tissue microstructure features. We introduce a novel framework, Shape--fused Fiber Cluster Transformer (SFFormer), that leverages a multi-head cross-attention feature fusion module to predict subject-specific language performance based on dMRI tractography. We assess the performance of the method on a large dataset including 1065 healthy young adults. The results demonstrate that both the transformer-based SFFormer model and its inter/intra feature fusion with shape, microstructure, and connectivity are informative, and together, they improve the prediction of subject-specific language performance scores. Overall, our results indicate that the shape of the brain's connections is predictive of human language function. △ Less

Submitted 29 March, 2024; v1 submitted 27 March, 2024; originally announced March 2024.

Comments: 2 figures, 11 pages

Audio-Visual Segmentation via Unlabeled Frame Exploitation

Authors: Jinxiang Liu, Yikun Liu, Fei Zhang, Chen Ju, Ya Zhang, Yanfeng Wang

Abstract: Audio-visual segmentation (AVS) aims to segment the sounding objects in video frames. Although great progress has been witnessed, we experimentally reveal that current methods reach marginal performance gain within the use of the unlabeled frames, leading to the underutilization issue. To fully explore the potential of the unlabeled frames for AVS, we explicitly divide them into two categories bas… ▽ More Audio-visual segmentation (AVS) aims to segment the sounding objects in video frames. Although great progress has been witnessed, we experimentally reveal that current methods reach marginal performance gain within the use of the unlabeled frames, leading to the underutilization issue. To fully explore the potential of the unlabeled frames for AVS, we explicitly divide them into two categories based on their temporal characteristics, i.e., neighboring frame (NF) and distant frame (DF). NFs, temporally adjacent to the labeled frame, often contain rich motion information that assists in the accurate localization of sounding objects. Contrary to NFs, DFs have long temporal distances from the labeled frame, which share semantic-similar objects with appearance variations. Considering their unique characteristics, we propose a versatile framework that effectively leverages them to tackle AVS. Specifically, for NFs, we exploit the motion cues as the dynamic guidance to improve the objectness localization. Besides, we exploit the semantic cues in DFs by treating them as valid augmentations to the labeled frames, which are then used to enrich data diversity in a self-training manner. Extensive experimental results demonstrate the versatility and superiority of our method, unleashing the power of the abundant unlabeled frames. △ Less

Submitted 16 March, 2024; originally announced March 2024.

Comments: Accepted by CVPR 2024

Speech-driven Personalized Gesture Synthetics: Harnessing Automatic Fuzzy Feature Inference

Authors: Fan Zhang, Zhaohan Wang, Xin Lyu, Siyuan Zhao, Mengjian Li, Weidong Geng, Naye Ji, Hui Du, Fuxing Gao, Hao Wu, Shunman Li

Abstract: Speech-driven gesture generation is an emerging field within virtual human creation. However, a significant challenge lies in accurately determining and processing the multitude of input features (such as acoustic, semantic, emotional, personality, and even subtle unknown features). Traditional approaches, reliant on various explicit feature inputs and complex multimodal processing, constrain the… ▽ More Speech-driven gesture generation is an emerging field within virtual human creation. However, a significant challenge lies in accurately determining and processing the multitude of input features (such as acoustic, semantic, emotional, personality, and even subtle unknown features). Traditional approaches, reliant on various explicit feature inputs and complex multimodal processing, constrain the expressiveness of resulting gestures and limit their applicability. To address these challenges, we present Persona-Gestor, a novel end-to-end generative model designed to generate highly personalized 3D full-body gestures solely relying on raw speech audio. The model combines a fuzzy feature extractor and a non-autoregressive Adaptive Layer Normalization (AdaLN) transformer diffusion architecture. The fuzzy feature extractor harnesses a fuzzy inference strategy that automatically infers implicit, continuous fuzzy features. These fuzzy features, represented as a unified latent feature, are fed into the AdaLN transformer. The AdaLN transformer introduces a conditional mechanism that applies a uniform function across all tokens, thereby effectively modeling the correlation between the fuzzy features and the gesture sequence. This module ensures a high level of gesture-speech synchronization while preserving naturalness. Finally, we employ the diffusion model to train and infer various gestures. Extensive subjective and objective evaluations on the Trinity, ZEGGS, and BEAT datasets confirm our model's superior performance to the current state-of-the-art approaches. Persona-Gestor improves the system's usability and generalization capabilities, setting a new benchmark in speech-driven gesture synthesis and broadening the horizon for virtual human technology. Supplementary videos and code can be accessed at https://zf223669.github.io/Diffmotion-v2-website/ △ Less

Submitted 16 March, 2024; originally announced March 2024.

Comments: 12 pages,

Experimental Study of Spatial Statistics for Ultra-Reliable Communications

Authors: Tobias Kallehauge, Anders E. Kalør, Fengchun Zhang, Petar Popovski

Abstract: This paper presents an experimental validation for prediction of rare fading events using channel distribution information (CDI) maps that predict channel statistics from measurements acquired at surrounding locations using spatial interpolation. Using experimental channel measurements from 127 locations, we demonstrate the use case of providing statistical guarantees for rate selection in ultra-r… ▽ More This paper presents an experimental validation for prediction of rare fading events using channel distribution information (CDI) maps that predict channel statistics from measurements acquired at surrounding locations using spatial interpolation. Using experimental channel measurements from 127 locations, we demonstrate the use case of providing statistical guarantees for rate selection in ultra-reliable low-latency communication (URLLC) using CDI maps. By using only the user location and the estimated map, we are able to meet the desired outage probability with a probability between 93.6-95.6% targeting 95%. On the other hand, a model-based baseline scheme that assumes Rayleigh fading meets the target outage requirement with a probability of 77.2%. The results demonstrate the practical relevance of CDI maps for resource allocation in URLLC. △ Less

Submitted 17 February, 2024; originally announced February 2024.

Comments: Accepted for IEEE International Conference on Communications (ICC) in june 2024

Immersive Video Compression using Implicit Neural Representations

Authors: Ho Man Kwan, Fan Zhang, Andrew Gower, David Bull

Abstract: Recent work on implicit neural representations (INRs) has evidenced their potential for efficiently representing and encoding conventional video content. In this paper we, for the first time, extend their application to immersive (multi-view) videos, by proposing MV-HiNeRV, a new INR-based immersive video codec. MV-HiNeRV is an enhanced version of a state-of-the-art INR-based video codec, HiNeRV,… ▽ More Recent work on implicit neural representations (INRs) has evidenced their potential for efficiently representing and encoding conventional video content. In this paper we, for the first time, extend their application to immersive (multi-view) videos, by proposing MV-HiNeRV, a new INR-based immersive video codec. MV-HiNeRV is an enhanced version of a state-of-the-art INR-based video codec, HiNeRV, which was developed for single-view video compression. We have modified the model to learn a different group of feature grids for each view, and share the learnt network parameters among all views. This enables the model to effectively exploit the spatio-temporal and the inter-view redundancy that exists within multi-view videos. The proposed codec was used to compress multi-view texture and depth video sequences in the MPEG Immersive Video (MIV) Common Test Conditions, and tested against the MIV Test model (TMIV) that uses the VVenC video codec. The results demonstrate the superior performance of MV-HiNeRV, with significant coding gains (up to 72.33\%) over TMIV. The implementation of MV-HiNeRV is published for further development and evaluation. △ Less

Submitted 23 February, 2024; v1 submitted 2 February, 2024; originally announced February 2024.

Neural Implicit Surface Reconstruction of Freehand 3D Ultrasound Volume with Geometric Constraints

Authors: Hongbo Chen, Logiraj Kumaralingam, Shuhang Zhang, Sheng Song, Fayi Zhang, Haibin Zhang, Thanh-Tu Pham, Edmond H. M. Lou, Kumaradevan Punithakumar, Yuyao Zhang, Lawrence H. Le, Rui Zheng

Abstract: Three-dimensional (3D) freehand ultrasound (US) is a widely used imaging modality that allows non-invasive imaging of medical anatomy without radiation exposure. Surface reconstruction of US volume is vital to acquire the accurate anatomical structures needed for modeling, registration, and visualization. However, traditional methods cannot produce a high-quality surface due to image noise. Despit… ▽ More Three-dimensional (3D) freehand ultrasound (US) is a widely used imaging modality that allows non-invasive imaging of medical anatomy without radiation exposure. Surface reconstruction of US volume is vital to acquire the accurate anatomical structures needed for modeling, registration, and visualization. However, traditional methods cannot produce a high-quality surface due to image noise. Despite improvements in smoothness, continuity, and resolution from deep learning approaches, research on surface reconstruction in freehand 3D US is still limited. This study introduces FUNSR, a self-supervised neural implicit surface reconstruction method to learn signed distance functions (SDFs) from US volumes. In particular, FUNSR iteratively learns the SDFs by moving the 3D queries sampled around volumetric point clouds to approximate the surface, guided by two novel geometric constraints: sign consistency constraint and onsurface constraint with adversarial learning. Our approach has been thoroughly evaluated across four datasets to demonstrate its adaptability to various anatomical structures, including a hip phantom dataset, two vascular datasets and one publicly available prostate dataset. We also show that smooth and continuous representations greatly enhance the visual appearance of US data. Furthermore, we highlight the potential of our method to improve segmentation performance, and its robustness to noise distribution and motion perturbation. △ Less

Submitted 11 July, 2024; v1 submitted 11 January, 2024; originally announced January 2024.

Comments: Preprint

A Deep Network for Explainable Prediction of Non-Imaging Phenotypes using Anatomical Multi-View Data

Authors: Yuxiang Wei, Yuqian Chen, Tengfei Xue, Leo Zekelman, Nikos Makris, Yogesh Rathi, Weidong Cai, Fan Zhang, Lauren J. O' Donnell

Abstract: Large datasets often contain multiple distinct feature sets, or views, that offer complementary information that can be exploited by multi-view learning methods to improve results. We investigate anatomical multi-view data, where each brain anatomical structure is described with multiple feature sets. In particular, we focus on sets of white matter microstructure and connectivity features from dif… ▽ More Large datasets often contain multiple distinct feature sets, or views, that offer complementary information that can be exploited by multi-view learning methods to improve results. We investigate anatomical multi-view data, where each brain anatomical structure is described with multiple feature sets. In particular, we focus on sets of white matter microstructure and connectivity features from diffusion MRI, as well as sets of gray matter area and thickness features from structural MRI. We investigate machine learning methodology that applies multi-view approaches to improve the prediction of non-imaging phenotypes, including demographics (age), motor (strength), and cognition (picture vocabulary). We present an explainable multi-view network (EMV-Net) that can use different anatomical views to improve prediction performance. In this network, each individual anatomical view is processed by a view-specific feature extractor and the extracted information from each view is fused using a learnable weight. This is followed by a wavelet transform-based module to obtain complementary information across views which is then applied to calibrate the view-specific information. Additionally, the calibrator produces an attention-based calibration score to indicate anatomical structures' importance for interpretation. △ Less

Submitted 13 January, 2024; v1 submitted 9 January, 2024; originally announced January 2024.

Comments: 2023 The Medical Image Computing and Computer Assisted Intervention Society workshop

A Closed-loop Brain-Machine Interface SoC Featuring a 0.2$μみゅー$J/class Multiplexer Based Neural Network

Authors: Chao Zhang, Yongxiang Guo, Dawid Sheng, Zhixiong Ma, Chao Sun, Yuwei Zhang, Wenxin Zhao, Fenyan Zhang, Tongfei Wang, Xing Sheng, Milin Zhang

Abstract: This work presents the first fabricated electrophysiology-optogenetic closed-loop bidirectional brain-machine interface (CL-BBMI) system-on-chip (SoC) with electrical neural signal recording, on-chip sleep staging and optogenetic stimulation. The first multiplexer with static assignment based table lookup solution (MUXnet) for multiplier-free NN processor was proposed. A state-of-the-art average a… ▽ More This work presents the first fabricated electrophysiology-optogenetic closed-loop bidirectional brain-machine interface (CL-BBMI) system-on-chip (SoC) with electrical neural signal recording, on-chip sleep staging and optogenetic stimulation. The first multiplexer with static assignment based table lookup solution (MUXnet) for multiplier-free NN processor was proposed. A state-of-the-art average accuracy of 82.4% was achieved with an energy consumption of only 0.2$μみゅー$J/class in sleep staging task. △ Less

Submitted 7 January, 2024; originally announced January 2024.

Comments: 2 pages, 6 figures. Accepted by IEEE Custom Integrated Circuits Conference (CICC) 2024. The codes for the MUXnet (constructing neural networks using multiplexers instead of multipliers) will be open-sourced after the Journal version of this work is accepted

Compressing Deep Image Super-resolution Models

Authors: Yuxuan Jiang, Jakub Nawala, Fan Zhang, David Bull

Abstract: Deep learning techniques have been applied in the context of image super-resolution (SR), achieving remarkable advances in terms of reconstruction performance. Existing techniques typically employ highly complex model structures which result in large model sizes and slow inference speeds. This often leads to high energy consumption and restricts their adoption for practical applications. To addres… ▽ More Deep learning techniques have been applied in the context of image super-resolution (SR), achieving remarkable advances in terms of reconstruction performance. Existing techniques typically employ highly complex model structures which result in large model sizes and slow inference speeds. This often leads to high energy consumption and restricts their adoption for practical applications. To address this issue, this work employs a three-stage workflow for compressing deep SR models which significantly reduces their memory requirement. Restoration performance has been maintained through teacher-student knowledge distillation using a newly designed distillation loss. We have applied this approach to two popular image super-resolution networks, SwinIR and EDSR, to demonstrate its effectiveness. The resulting compact models, SwinIRmini and EDSRmini, attain an 89% and 96% reduction in both model size and floating-point operations (FLOPs) respectively, compared to their original versions. They also retain competitive super-resolution performance compared to their original models and other commonly used SR approaches. The source code and pre-trained models for these two lightweight SR approaches are released at https://pikapi22.github.io/CDISM/. △ Less

Submitted 21 February, 2024; v1 submitted 31 December, 2023; originally announced January 2024.

GreenScan: Towards large-scale terrestrial monitoring the health of urban trees using mobile sensing

Authors: Akshit Gupta, Simone Mora, Fan Zhang, Martine Rutten, R. Venkatesha Prasad, Carlo Ratti

Abstract: Healthy urban greenery is a fundamental asset to mitigate climate change phenomena such as extreme heat and air pollution. However, urban trees are often affected by abiotic and biotic stressors that hamper their functionality, and whenever not timely managed, even their survival. While the current greenery inspection techniques can help in taking effective measures, they often require a high amou… ▽ More Healthy urban greenery is a fundamental asset to mitigate climate change phenomena such as extreme heat and air pollution. However, urban trees are often affected by abiotic and biotic stressors that hamper their functionality, and whenever not timely managed, even their survival. While the current greenery inspection techniques can help in taking effective measures, they often require a high amount of human labor, making frequent assessments infeasible at city-wide scales. In this paper, we present GreenScan, a ground-based sensing system designed to provide health assessments of urban trees at high spatio-temporal resolutions, with low costs. The system utilises thermal and multi-spectral imaging sensors fused using a custom computer vision model in order to estimate two tree health indexes. The evaluation of the system was performed through data collection experiments in Cambridge, USA. Overall, this work illustrates a novel approach for autonomous mobile ground-based tree health monitoring on city-wide scales at high temporal resolutions with low-costs. △ Less

Submitted 6 April, 2024; v1 submitted 21 December, 2023; originally announced December 2023.

Comments: 13 pages, submitted to IEEE Sensors

Full-reference Video Quality Assessment for User Generated Content Transcoding

Authors: Zihao Qi, Chen Feng, Duolikun Danier, Fan Zhang, Xiaozhong Xu, Shan Liu, David Bull

Abstract: Unlike video coding for professional content, the delivery pipeline of User Generated Content (UGC) involves transcoding where unpristine reference content needs to be compressed repeatedly. In this work, we observe that existing full-/no-reference quality metrics fail to accurately predict the perceptual quality difference between transcoded UGC content and the corresponding unpristine references… ▽ More Unlike video coding for professional content, the delivery pipeline of User Generated Content (UGC) involves transcoding where unpristine reference content needs to be compressed repeatedly. In this work, we observe that existing full-/no-reference quality metrics fail to accurately predict the perceptual quality difference between transcoded UGC content and the corresponding unpristine references. Therefore, they are unsuited for guiding the rate-distortion optimisation process in the transcoding process. In this context, we propose a bespoke full-reference deep video quality metric for UGC transcoding. The proposed method features a transcoding-specific weakly supervised training strategy employing a quality ranking-based Siamese structure. The proposed method is evaluated on the YouTube-UGC VP9 subset and the LIVE-Wild database, demonstrating state-of-the-art performance compared to existing VQA methods. △ Less

Submitted 19 December, 2023; originally announced December 2023.

Comments: 5 pages, 4 figures

RankDVQA-mini: Knowledge Distillation-Driven Deep Video Quality Assessment

Authors: Chen Feng, Duolikun Danier, Haoran Wang, Fan Zhang, Benoit Vallade, Alex Mackin, David Bull

Abstract: Deep learning-based video quality assessment (deep VQA) has demonstrated significant potential in surpassing conventional metrics, with promising improvements in terms of correlation with human perception. However, the practical deployment of such deep VQA models is often limited due to their high computational complexity and large memory requirements. To address this issue, we aim to significantl… ▽ More Deep learning-based video quality assessment (deep VQA) has demonstrated significant potential in surpassing conventional metrics, with promising improvements in terms of correlation with human perception. However, the practical deployment of such deep VQA models is often limited due to their high computational complexity and large memory requirements. To address this issue, we aim to significantly reduce the model size and runtime of one of the state-of-the-art deep VQA methods, RankDVQA, by employing a two-phase workflow that integrates pruning-driven model compression with multi-level knowledge distillation. The resulting lightweight full reference quality metric, RankDVQA-mini, requires less than 10% of the model parameters compared to its full version (14% in terms of FLOPs), while still retaining a quality prediction performance that is superior to most existing deep VQA methods. The source code of the RankDVQA-mini has been released at https://chenfeng-bristol.github.io/RankDVQA-mini/ for public evaluation. △ Less

Submitted 7 March, 2024; v1 submitted 14 December, 2023; originally announced December 2023.

Comments: The paper has been accepted by Picture Coding Symposium (PCS) 2024

Accelerating Learnt Video Codecs with Gradient Decay and Layer-wise Distillation

Authors: Tianhao Peng, Ge Gao, Heming Sun, Fan Zhang, David Bull

Abstract: In recent years, end-to-end learnt video codecs have demonstrated their potential to compete with conventional coding algorithms in term of compression efficiency. However, most learning-based video compression models are associated with high computational complexity and latency, in particular at the decoder side, which limits their deployment in practical applications. In this paper, we present a… ▽ More In recent years, end-to-end learnt video codecs have demonstrated their potential to compete with conventional coding algorithms in term of compression efficiency. However, most learning-based video compression models are associated with high computational complexity and latency, in particular at the decoder side, which limits their deployment in practical applications. In this paper, we present a novel model-agnostic pruning scheme based on gradient decay and adaptive layer-wise distillation. Gradient decay enhances parameter exploration during sparsification whilst preventing runaway sparsity and is superior to the standard Straight-Through Estimation. The adaptive layer-wise distillation regulates the sparse training in various stages based on the distortion of intermediate features. This stage-wise design efficiently updates parameters with minimal computational overhead. The proposed approach has been applied to three popular end-to-end learnt video codecs, FVC, DCVC, and DCVC-HEM. Results confirm that our method yields up to 65% reduction in MACs and 2x speed-up with less than 0.3dBでしべる drop in BD-PSNR. Supporting code and supplementary material can be downloaded from: https://jasminepp.github.io/lightweightdvc/ △ Less

Submitted 5 December, 2023; originally announced December 2023.

Report number: 2312.02605

A Novel Deep Clustering Framework for Fine-Scale Parcellation of Amygdala Using dMRI Tractography

Authors: Haolin He, Ce Zhu, Le Zhang, Yipeng Liu, Xiao Xu, Yuqian Chen, Leo Zekelman, Jarrett Rushmore, Yogesh Rathi, Nikos Makris, Lauren J. O'Donnell, Fan Zhang

Abstract: The amygdala plays a vital role in emotional processing and exhibits structural diversity that necessitates fine-scale parcellation for a comprehensive understanding of its anatomico-functional correlations. Diffusion MRI tractography is an advanced imaging technique that can estimate the brain's white matter structural connectivity to potentially reveal the topography of the amygdala for studying… ▽ More The amygdala plays a vital role in emotional processing and exhibits structural diversity that necessitates fine-scale parcellation for a comprehensive understanding of its anatomico-functional correlations. Diffusion MRI tractography is an advanced imaging technique that can estimate the brain's white matter structural connectivity to potentially reveal the topography of the amygdala for studying its subdivisions. In this work, we present a deep clustering pipeline to perform automated, fine-scale parcellation of the amygdala using diffusion MRI tractography. First, we incorporate a newly proposed deep learning approach to enable accurate segmentation of the amygdala directly on the dMRI data. Next, we design a novel streamline clustering-based structural connectivity feature for a robust representation of voxels within the amygdala. Finally, we improve the popular joint dimensionality reduction and k-means clustering approach to enable amygdala parcellation at a finer scale. With the proposed method, we obtain nine unique amygdala parcels. Experiments show that these parcels can be consistently identified across subjects and have good correspondence to the widely used coarse-scale amygdala parcellation. △ Less

Submitted 25 November, 2023; originally announced November 2023.

Scanning phase imaging without accurate positioning system

Authors: Tao Liu, Bingyang Wang, JiangTao Zhao, Fu rong Chen, Fucai Zhang

Abstract: Ptychography, a high-resolution phase imaging technique using precise in-plane translation information, has been widely applied in modern synchrotron radiation sources across the globe. A key requirement for successful ptychographic reconstruction is the precise knowledge of the scanning positions, which are typically obtained by a physical interferometric positioning system. Whereas high-throughp… ▽ More Ptychography, a high-resolution phase imaging technique using precise in-plane translation information, has been widely applied in modern synchrotron radiation sources across the globe. A key requirement for successful ptychographic reconstruction is the precise knowledge of the scanning positions, which are typically obtained by a physical interferometric positioning system. Whereas high-throughput positioning poses a challenge in engineering, especially in nano or even smaller scale. In this work, we propose a novel scanning imaging framework that does not require any prior position information from the positioning system. Specifically, our scheme utilizes the wavefront modulation mechanism to reconstruct the object functions at each scan position and the shared illumination function, simultaneously. The scanning trajectory information is extracted by our subpixel image registration algorithm from the overlap region of reconstructed object functions. Then, a completed object function can be obtained by assembling each part of the reconstructed sample functions. High-quality imaging of biological sample and position recovery with sub-pixel accuracy are demonstrated in proof-of-concept experiment. Based on current results, we find it may have great potential applications in high-resolution and high throughput phase imaging. △ Less

Submitted 31 October, 2023; originally announced November 2023.

Comments: 9 pages,4 figures

Enhancing the machine vision performance with multi-spectral light sources

Authors: Feng Zhang, Rui Bao, Congqi Dai, Wanlu Zhang, Shu Liu, Ruiqian Guo

Abstract: This study mainly focuses on the performance of different multi-spectral light sources on different object colors in machine vision and tries to enhance machine vision with multi-spectral light sources. Using different color pencils as samples, by recognizing the collected images with two classical neural networks, AlexNet and VGG19, the performance was investigated under 35 different multi-spectr… ▽ More This study mainly focuses on the performance of different multi-spectral light sources on different object colors in machine vision and tries to enhance machine vision with multi-spectral light sources. Using different color pencils as samples, by recognizing the collected images with two classical neural networks, AlexNet and VGG19, the performance was investigated under 35 different multi-spectral light sources. The results show that for both models there are always some non-pure white light sources, whose accuracy is better than pure white light, which suggests the potential of multi-spectral light sources to further enhance the effectiveness of machine vision. The comparison of both models is also performed, and surprised to find that the overall performance of VGG19 is lower than that of AlexNet, which shows that the importance of the choice of multi-spectral light sources and models. △ Less

Submitted 20 October, 2023; originally announced November 2023.

Comments: 12 pages, 7 figures

Cross-Domain Dual-Functional OFDM Waveform Design for Accurate Sensing/Positioning

Authors: Fan Zhang, Tianqi Mao, Ruiqi Liu, Zhu Han, Sheng Chen, Zhaocheng Wang

Abstract: Orthogonal frequency division multiplexing (OFDM) has been widely recognized as the representative waveform for 5G wireless networks, which can directly support sensing/positioning with existing infrastructure. To guarantee superior sensing/positioning accuracy while supporting high-speed communications simultaneously, the dual functions tend to be assigned with different resource elements (REs) d… ▽ More Orthogonal frequency division multiplexing (OFDM) has been widely recognized as the representative waveform for 5G wireless networks, which can directly support sensing/positioning with existing infrastructure. To guarantee superior sensing/positioning accuracy while supporting high-speed communications simultaneously, the dual functions tend to be assigned with different resource elements (REs) due to their diverse design requirements. This motivates optimization of resource allocation/waveform design across time, frequency, power and delay-Doppler domains. Therefore, this article proposes two cross-domain waveform optimization strategies for effective convergence of OFDM-based communications and sensing/positioning, following communication- and sensing-centric criteria, respectively. For the communication-centric design, to maximize the achievable data rate, a fraction of REs are optimally allocated for communications according to prior knowledge of the communication channel. The remaining REs are then employed for sensing/positioning, where the sidelobe level and peak-to-average power ratio are suppressed by optimizing its power-frequency and phase-frequency characteristics for sensing performance improvement. For the sensing-centric design, a `locally' perfect auto-correlation property is ensured for accurate sensing and positioning by adjusting the unit cells of the ambiguity function within its region of interest (RoI). Afterwards, the irrelevant cells beyond RoI, which can readily determine the sensing power allocation, are optimized with the communication power allocation to enhance the achievable data rate. Numerical results demonstrate the superiority of the proposed waveform designs. △ Less

Submitted 19 March, 2024; v1 submitted 8 November, 2023; originally announced November 2023.

Design and Performance Analysis of a Class of Generalized Predictive Controllers

Authors: Feilong Zhang

Abstract: The design and structure of generalized predictive control (GPC) are not simple and intuitive. The performance analysis does not deeply analyze how the controller parameters affect the system characteristics and the relationship between the tracking error caused by the noise and the selected controller parameters. This paper proposes a generalized predictive control, and its design is simple and i… ▽ More The design and structure of generalized predictive control (GPC) are not simple and intuitive. The performance analysis does not deeply analyze how the controller parameters affect the system characteristics and the relationship between the tracking error caused by the noise and the selected controller parameters. This paper proposes a generalized predictive control, and its design is simple and intuitive for unnecessary solving the Diophantine equation. Then the relationship between desired output, disturbance, and system output is analyzed by the characteristic equation and steady-state analysis. Based on this, the study presents research findings on the steady state of the system and verifies them through simulations. Furthermore, this paper introduces GPC with disturbance compensation and incremental generalized minimum variance control (IGMVC) with disturbance compensation. The conditions for the elimination of disturbance are presented in theory and simulation for the first time. △ Less

Submitted 6 November, 2023; originally announced November 2023.

Broadband ptychographic imaging of biological samples using a deconvolution algorithm

Authors: Huixiang Lin, Fucai Zhang

Abstract: Ptychography is an attractive advance of coherent diffraction imaging (CDI), which can provide high lateral resolution and wide field of view. The theoretical resolution of ptychography is dose-limited, therefore making ptychography workable with a broadband source will be highly beneficial. However, broad spectra of light source conflict with the high coherence assumption in CDI that the current… ▽ More Ptychography is an attractive advance of coherent diffraction imaging (CDI), which can provide high lateral resolution and wide field of view. The theoretical resolution of ptychography is dose-limited, therefore making ptychography workable with a broadband source will be highly beneficial. However, broad spectra of light source conflict with the high coherence assumption in CDI that the current reconstruction algorithm were built upon. In this paper, we demonstrated that incorporation of a blind deconvolution in the reconstruction algorithm can improve the image quality of ptychography with broadband source. This broadband reconstruction algorithm can obtain high-quality amplitude and phase images of complex-valued samples requiring no knowledge of the illumination spectrum. Optical experiments using biological samples demonstrate the effectiveness of our method. The significant improvement in low coherence tolerance by our approach can pave the way for implementing ultrafast imaging with femtosecond or attosecond lasers or high-flux ptychographic imaging with laboratory EUV or X-ray sources. △ Less

Submitted 3 November, 2023; originally announced November 2023.

Comments: 6 pages, 2 figures

Lookup Table meets Local Laplacian Filter: Pyramid Reconstruction Network for Tone Mapping

Authors: Feng Zhang, Ming Tian, Zhiqiang Li, Bin Xu, Qingbo Lu, Changxin Gao, Nong Sang

Abstract: Tone mapping aims to convert high dynamic range (HDR) images to low dynamic range (LDR) representations, a critical task in the camera imaging pipeline. In recent years, 3-Dimensional LookUp Table (3D LUT) based methods have gained attention due to their ability to strike a favorable balance between enhancement performance and computational efficiency. However, these methods often fail to deliver… ▽ More Tone mapping aims to convert high dynamic range (HDR) images to low dynamic range (LDR) representations, a critical task in the camera imaging pipeline. In recent years, 3-Dimensional LookUp Table (3D LUT) based methods have gained attention due to their ability to strike a favorable balance between enhancement performance and computational efficiency. However, these methods often fail to deliver satisfactory results in local areas since the look-up table is a global operator for tone mapping, which works based on pixel values and fails to incorporate crucial local information. To this end, this paper aims to address this issue by exploring a novel strategy that integrates global and local operators by utilizing closed-form Laplacian pyramid decomposition and reconstruction. Specifically, we employ image-adaptive 3D LUTs to manipulate the tone in the low-frequency image by leveraging the specific characteristics of the frequency information. Furthermore, we utilize local Laplacian filters to refine the edge details in the high-frequency components in an adaptive manner. Local Laplacian filters are widely used to preserve edge details in photographs, but their conventional usage involves manual tuning and fixed implementation within camera imaging pipelines or photo editing tools. We propose to learn parameter value maps progressively for local Laplacian filters from annotated data using a lightweight network. Our model achieves simultaneous global tone manipulation and local edge detail preservation in an end-to-end manner. Extensive experimental results on two benchmark datasets demonstrate that the proposed method performs favorably against state-of-the-art methods. △ Less

Submitted 3 January, 2024; v1 submitted 26 October, 2023; originally announced October 2023.

Comments: 12 pages, 6 figures, accepted by NeurlPS 2023

First realization of macroscopic Fourier ptychography for hundred-meter distance sub-diffraction imaging

Authors: Qi Zhang, Yuran Lu, Yinghui Guo, Yingjie Shang, Mingbo Pu, Yulong Fan, Rui Zhou, Xiaoyin Li, Fei Zhang, Mingfeng Xu, Xiangang Luo

Abstract: Fourier ptychography (FP) imaging, drawing on the idea of synthetic aperture, has been demonstrated as a potential approach for remote sub-diffraction-limited imaging. Nevertheless, the farthest imaging distance is still limited around 10 m even though there has been a significant improvement in macroscopic FP. The most severely issue in increasing the imaging distance is FoV limitation caused by… ▽ More Fourier ptychography (FP) imaging, drawing on the idea of synthetic aperture, has been demonstrated as a potential approach for remote sub-diffraction-limited imaging. Nevertheless, the farthest imaging distance is still limited around 10 m even though there has been a significant improvement in macroscopic FP. The most severely issue in increasing the imaging distance is FoV limitation caused by far-field condition for diffraction. Here, we propose to modify the Fourier far-field condition for rough reflective objects, aiming to overcome the small FoV limitation by using a divergent beam to illuminate objects. A joint optimization of pupil function and target image is utilized to attain the aberration-free image while estimating the pupil function simultaneously. Benefiting from the optimized reconstruction algorithm which effectively expands the camera's effective aperture, we experimentally implement several FP systems suited for imaging distance of 12 m, 90 m, and 170 m with the maximum synthetic aperture of 200 mm. The maximum imaging distance and synthetic aperture are thus improved by more than one order of magnitude of the state-of-the-art works with a fourfold improvement in the resolution. Our findings demonstrate significant potential for advancing the field of macroscopic FP, propelling it into a new stage of development. △ Less

Submitted 22 October, 2023; originally announced October 2023.

Learning Regularized Monotone Graphon Mean-Field Games

Authors: Fengzhuo Zhang, Vincent Y. F. Tan, Zhaoran Wang, Zhuoran Yang

Abstract: This paper studies two fundamental problems in regularized Graphon Mean-Field Games (GMFGs). First, we establish the existence of a Nash Equilibrium (NE) of any $λらむだ$-regularized GMFG (for $λらむだ\geq 0$). This result relies on weaker conditions than those in previous works for analyzing both unregularized GMFGs ($λらむだ=0$) and $λらむだ$-regularized MFGs, which are special cases of GMFGs. Second, we propose provab… ▽ More This paper studies two fundamental problems in regularized Graphon Mean-Field Games (GMFGs). First, we establish the existence of a Nash Equilibrium (NE) of any $λらむだ$-regularized GMFG (for $λらむだ\geq 0$). This result relies on weaker conditions than those in previous works for analyzing both unregularized GMFGs ($λらむだ=0$) and $λらむだ$-regularized MFGs, which are special cases of GMFGs. Second, we propose provably efficient algorithms to learn the NE in weakly monotone GMFGs, motivated by Lasry and Lions [2007]. Previous literature either only analyzed continuous-time algorithms or required extra conditions to analyze discrete-time algorithms. In contrast, we design a discrete-time algorithm and derive its convergence rate solely under weakly monotone conditions. Furthermore, we develop and analyze the action-value function estimation procedure during the online learning process, which is absent from algorithms for monotone GMFGs. This serves as a sub-module in our optimization algorithm. The efficiency of the designed algorithm is corroborated by empirical evaluations. △ Less

Submitted 12 October, 2023; originally announced October 2023.

Deshadow-Anything: When Segment Anything Model Meets Zero-shot shadow removal

Authors: Xiao Feng Zhang, Tian Yi Song, Jia Wei Yao

Abstract: Segment Anything (SAM), an advanced universal image segmentation model trained on an expansive visual dataset, has set a new benchmark in image segmentation and computer vision. However, it faced challenges when it came to distinguishing between shadows and their backgrounds. To address this, we developed Deshadow-Anything, considering the generalization of large-scale datasets, and we performed F… ▽ More Segment Anything (SAM), an advanced universal image segmentation model trained on an expansive visual dataset, has set a new benchmark in image segmentation and computer vision. However, it faced challenges when it came to distinguishing between shadows and their backgrounds. To address this, we developed Deshadow-Anything, considering the generalization of large-scale datasets, and we performed Fine-tuning on large-scale datasets to achieve image shadow removal. The diffusion model can diffuse along the edges and textures of an image, helping to remove shadows while preserving the details of the image. Furthermore, we design Multi-Self-Attention Guidance (MSAG) and adaptive input perturbation (DDPM-AIP) to accelerate the iterative training speed of diffusion. Experiments on shadow removal tasks demonstrate that these methods can effectively improve image restoration performance. △ Less

Submitted 2 January, 2024; v1 submitted 20 September, 2023; originally announced September 2023.

Comments: it needs revised

Self-supervised Domain-agnostic Domain Adaptation for Satellite Images

Authors: Fahong Zhang, Yilei Shi, Xiao Xiang Zhu

Abstract: Domain shift caused by, e.g., different geographical regions or acquisition conditions is a common issue in machine learning for global scale satellite image processing. A promising method to address this problem is domain adaptation, where the training and the testing datasets are split into two or multiple domains according to their distributions, and an adaptation method is applied to improve t… ▽ More Domain shift caused by, e.g., different geographical regions or acquisition conditions is a common issue in machine learning for global scale satellite image processing. A promising method to address this problem is domain adaptation, where the training and the testing datasets are split into two or multiple domains according to their distributions, and an adaptation method is applied to improve the generalizability of the model on the testing dataset. However, defining the domain to which each satellite image belongs is not trivial, especially under large-scale multi-temporal and multi-sensory scenarios, where a single image mosaic could be generated from multiple data sources. In this paper, we propose an self-supervised domain-agnostic domain adaptation (SS(DA)2) method to perform domain adaptation without such a domain definition. To achieve this, we first design a contrastive generative adversarial loss to train a generative network to perform image-to-image translation between any two satellite image patches. Then, we improve the generalizability of the downstream models by augmenting the training data with different testing spectral characteristics. The experimental results on public benchmarks verify the effectiveness of SS(DA)2. △ Less

Submitted 25 September, 2023; v1 submitted 20 September, 2023; originally announced September 2023.

Snapp: An Agile Robotic Fish with 3-D Maneuverability for Open Water Swim

Authors: Timothy J. K. Ng, Nan Chen, Fu Zhang

Abstract: Fish exhibit impressive locomotive performance and agility in complex underwater environments, using their undulating tails and pectoral fins for propulsion and maneuverability. Replicating these abilities in robotic fish is challenging; existing designs focus on either fast swimming or directional control at limited speeds, mainly within a confined environment. To address these limitations, we de… ▽ More Fish exhibit impressive locomotive performance and agility in complex underwater environments, using their undulating tails and pectoral fins for propulsion and maneuverability. Replicating these abilities in robotic fish is challenging; existing designs focus on either fast swimming or directional control at limited speeds, mainly within a confined environment. To address these limitations, we designed Snapp, an integrated robotic fish capable of swimming in open water with high speeds and full 3-dimensional maneuverability. A novel cyclic-differential method is layered on the mechanism. It integrates propulsion and yaw-steering for fast course corrections. Two independent pectoral fins provide pitch and roll control. We evaluated Snapp in open water environments. We demonstrated significant improvements in speed and maneuverability, achieving swimming speeds of 1.5 m/s (1.7 Body Lengths per second) and performing complex maneuvers, such as a figure-8 and S-shape trajectory. Instantaneous yaw changes of 15$^{\circ}$ in 0.4 s, a minimum turn radius of 0.85 m, and maximum pitch and roll rates of 3.5 rad/s and 1 rad/s, respectively, were recorded. Our results suggest that Snapp's swimming capabilities have excellent practical prospects for open seas and contribute significantly to developing agile robotic fishes. △ Less

Submitted 24 August, 2023; originally announced August 2023.

Comments: 8 pages, 17 figures, to be publish in IEEE Robotics and Automation Letters The accompanying video can be found at this link: https://youtu.be/1bGmlN0Jriw

Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained model

Authors: Fan Zhang, Naye Ji, Fuxing Gao, Siyuan Zhao, Zhaohan Wang, Shunman Li

Abstract: The generation of co-speech gestures for digital humans is an emerging area in the field of virtual human creation. Prior research has made progress by using acoustic and semantic information as input and adopting classify method to identify the person's ID and emotion for driving co-speech gesture generation. However, this endeavour still faces significant challenges. These challenges go beyond t… ▽ More The generation of co-speech gestures for digital humans is an emerging area in the field of virtual human creation. Prior research has made progress by using acoustic and semantic information as input and adopting classify method to identify the person's ID and emotion for driving co-speech gesture generation. However, this endeavour still faces significant challenges. These challenges go beyond the intricate interplay between co-speech gestures, speech acoustic, and semantics; they also encompass the complexities associated with personality, emotion, and other obscure but important factors. This paper introduces "diffmotion-v2," a speech-conditional diffusion-based and non-autoregressive transformer-based generative model with WavLM pre-trained model. It can produce individual and stylized full-body co-speech gestures only using raw speech audio, eliminating the need for complex multimodal processing and manually annotated. Firstly, considering that speech audio not only contains acoustic and semantic features but also conveys personality traits, emotions, and more subtle information related to accompanying gestures, we pioneer the adaptation of WavLM, a large-scale pre-trained model, to extract low-level and high-level audio information. Secondly, we introduce an adaptive layer norm architecture in the transformer-based layer to learn the relationship between speech information and accompanying gestures. Extensive subjective evaluation experiments are conducted on the Trinity, ZEGGS, and BEAT datasets to confirm the WavLM and the model's ability to synthesize natural co-speech gestures with various styles. △ Less

Submitted 13 April, 2024; v1 submitted 11 August, 2023; originally announced August 2023.

Comments: This article needs major revision

Unleashing the Strengths of Unlabeled Data in Pan-cancer Abdominal Organ Quantification: the FLARE22 Challenge

Authors: Jun Ma, Yao Zhang, Song Gu, Cheng Ge, Shihao Ma, Adamo Young, Cheng Zhu, Kangkang Meng, Xin Yang, Ziyan Huang, Fan Zhang, Wentao Liu, YuanKe Pan, Shoujin Huang, Jiacheng Wang, Mingze Sun, Weixin Xu, Dengqiang Jia, Jae Won Choi, Natália Alves, Bram de Wilde, Gregor Koehler, Yajun Wu, Manuel Wiesenfarth, Qiongjie Zhu , et al. (4 additional authors not shown)

Abstract: Quantitative organ assessment is an essential step in automated abdominal disease diagnosis and treatment planning. Artificial intelligence (AI) has shown great potential to automatize this process. However, most existing AI algorithms rely on many expert annotations and lack a comprehensive evaluation of accuracy and efficiency in real-world multinational settings. To overcome these limitations,… ▽ More Quantitative organ assessment is an essential step in automated abdominal disease diagnosis and treatment planning. Artificial intelligence (AI) has shown great potential to automatize this process. However, most existing AI algorithms rely on many expert annotations and lack a comprehensive evaluation of accuracy and efficiency in real-world multinational settings. To overcome these limitations, we organized the FLARE 2022 Challenge, the largest abdominal organ analysis challenge to date, to benchmark fast, low-resource, accurate, annotation-efficient, and generalized AI algorithms. We constructed an intercontinental and multinational dataset from more than 50 medical groups, including Computed Tomography (CT) scans with different races, diseases, phases, and manufacturers. We independently validated that a set of AI algorithms achieved a median Dice Similarity Coefficient (DSC) of 90.0\% by using 50 labeled scans and 2000 unlabeled scans, which can significantly reduce annotation requirements. The best-performing algorithms successfully generalized to holdout external validation sets, achieving a median DSC of 89.5\%, 90.9\%, and 88.3\% on North American, European, and Asian cohorts, respectively. They also enabled automatic extraction of key organ biology features, which was labor-intensive with traditional manual measurements. This opens the potential to use unlabeled data to boost performance and alleviate annotation shortages for modern AI models. △ Less

Submitted 10 August, 2023; originally announced August 2023.

Comments: MICCAI FLARE22: https://flare22.grand-challenge.org/

Towards General Low-Light Raw Noise Synthesis and Modeling

Authors: Feng Zhang, Bin Xu, Zhiqiang Li, Xinran Liu, Qingbo Lu, Changxin Gao, Nong Sang

Abstract: Modeling and synthesizing low-light raw noise is a fundamental problem for computational photography and image processing applications. Although most recent works have adopted physics-based models to synthesize noise, the signal-independent noise in low-light conditions is far more complicated and varies dramatically across camera sensors, which is beyond the description of these models. To addres… ▽ More Modeling and synthesizing low-light raw noise is a fundamental problem for computational photography and image processing applications. Although most recent works have adopted physics-based models to synthesize noise, the signal-independent noise in low-light conditions is far more complicated and varies dramatically across camera sensors, which is beyond the description of these models. To address this issue, we introduce a new perspective to synthesize the signal-independent noise by a generative model. Specifically, we synthesize the signal-dependent and signal-independent noise in a physics- and learning-based manner, respectively. In this way, our method can be considered as a general model, that is, it can simultaneously learn different noise characteristics for different ISO levels and generalize to various sensors. Subsequently, we present an effective multi-scale discriminator termed Fourier transformer discriminator (FTD) to distinguish the noise distribution accurately. Additionally, we collect a new low-light raw denoising (LRD) dataset for training and benchmarking. Qualitative validation shows that the noise generated by our proposed noise model can be highly similar to the real noise in terms of distribution. Furthermore, extensive denoising experiments demonstrate that our method performs favorably against state-of-the-art methods on different sensors. △ Less

Submitted 17 August, 2023; v1 submitted 31 July, 2023; originally announced July 2023.

Comments: 11 pages, 7 figures. Accepted by ICCV 2023

Journal ref: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 10820-10830

Data-Driven Optimal Control of Tethered Space Robot Deployment with Learning Based Koopman Operator

Authors: Ao Jin, Fan Zhang, Panfeng Huang

Abstract: To avoid complex constraints of the traditional nonlinear method for tethered space robot (TSR) deployment, this paper proposes a data-driven optimal control framework with an improved deep learning based Koopman operator that could be applied to complex environments. In consideration of TSR's nonlinearity, its finite dimensional lifted representation is derived with the state-dependent only embed… ▽ More To avoid complex constraints of the traditional nonlinear method for tethered space robot (TSR) deployment, this paper proposes a data-driven optimal control framework with an improved deep learning based Koopman operator that could be applied to complex environments. In consideration of TSR's nonlinearity, its finite dimensional lifted representation is derived with the state-dependent only embedding functions in the Koopman framework. A deep learning approach is adopted to approximate the global linear representation of TSR. Deep neural networks (DNN) are developed to parameterize Koopman operator and its embedding functions. An auxiliary neural network is developed to encode the nonlinear control term of finite dimensional lifted system. In addition, the state matrix A and control matrix B of lifted linear system in the embedding space are also estimated during training DNN. Then three loss functions that related to reconstruction and prediction ability of network and controllability of lifted linear system are designed for training the entire network. With the global linear system produced from DNN, Linear Quadratic Regulator (LQR) is applied to derive the optimal control policy for the TSR deployment. Finally, simulation results verify the effectiveness of proposed framework and show that it could deploy tethered space robot more quickly with less swing of in-plane angle. △ Less

Submitted 15 July, 2023; originally announced July 2023.

Comments: 10pages, 10figures

Reconstructing Three-decade Global Fine-Grained Nighttime Light Observations by a New Super-Resolution Framework

Authors: Jinyu Guo, Feng Zhang, Hang Zhao, Baoxiang Pan, Linlu Mei

Abstract: Satellite-collected nighttime light provides a unique perspective on human activities, including urbanization, population growth, and epidemics. Yet, long-term and fine-grained nighttime light observations are lacking, leaving the analysis and applications of decades of light changes in urban facilities undeveloped. To fill this gap, we developed an innovative framework and used it to design a new… ▽ More Satellite-collected nighttime light provides a unique perspective on human activities, including urbanization, population growth, and epidemics. Yet, long-term and fine-grained nighttime light observations are lacking, leaving the analysis and applications of decades of light changes in urban facilities undeveloped. To fill this gap, we developed an innovative framework and used it to design a new super-resolution model that reconstructs low-resolution nighttime light data into high resolution. The validation of one billion data points shows that the correlation coefficient of our model at the global scale reaches 0.873, which is significantly higher than that of other existing models (maximum = 0.713). Our model also outperforms existing models at the national and urban scales. Furthermore, through an inspection of airports and roads, only our model's image details can reveal the historical development of these facilities. We provide the long-term and fine-grained nighttime light observations to promote research on human activities. The dataset is available at \url{https://doi.org/10.5281/zenodo.7859205}. △ Less

Submitted 14 July, 2023; originally announced July 2023.

Perch a quadrotor on planes by the ceiling effect

Authors: Yuying Zou, Haotian Li, Yunfan Ren, Wei Xu, Yihang Li, Yixi Cai, Shenji Zhou, Fu Zhang

Abstract: Perching is a promising solution for a small unmanned aerial vehicle (UAV) to save energy and extend operation time. This paper proposes a quadrotor that can perch on planar structures using the ceiling effect. Compared with the existing work, this perching method does not require any claws, hooks, or adhesive pads, leading to a simpler system design. This method does not limit the perching by sur… ▽ More Perching is a promising solution for a small unmanned aerial vehicle (UAV) to save energy and extend operation time. This paper proposes a quadrotor that can perch on planar structures using the ceiling effect. Compared with the existing work, this perching method does not require any claws, hooks, or adhesive pads, leading to a simpler system design. This method does not limit the perching by surface angle or material either. The design of the quadrotor that only uses its propeller guards for surface contact is presented in this paper. We also discussed the automatic perching strategy including trajectory generation and power management. Experiments are conducted to verify that the approach is practical and the UAV can perch on planes with different angles. Energy consumption in the perching state is assessed, showing that more than 30% of power can be saved. Meanwhile, the quadrotor exhibits improved stability while perching compared to when it is hovering. △ Less

Submitted 3 July, 2023; originally announced July 2023.

HiNeRV: Video Compression with Hierarchical Encoding-based Neural Representation

Authors: Ho Man Kwan, Ge Gao, Fan Zhang, Andrew Gower, David Bull

Abstract: Learning-based video compression is currently a popular research topic, offering the potential to compete with conventional standard video codecs. In this context, Implicit Neural Representations (INRs) have previously been used to represent and compress image and video content, demonstrating relatively high decoding speed compared to other methods. However, existing INR-based methods have failed… ▽ More Learning-based video compression is currently a popular research topic, offering the potential to compete with conventional standard video codecs. In this context, Implicit Neural Representations (INRs) have previously been used to represent and compress image and video content, demonstrating relatively high decoding speed compared to other methods. However, existing INR-based methods have failed to deliver rate quality performance comparable with the state of the art in video compression. This is mainly due to the simplicity of the employed network architectures, which limit their representation capability. In this paper, we propose HiNeRV, an INR that combines light weight layers with novel hierarchical positional encodings. We employs depth-wise convolutional, MLP and interpolation layers to build the deep and wide network architecture with high capacity. HiNeRV is also a unified representation encoding videos in both frames and patches at the same time, which offers higher performance and flexibility than existing methods. We further build a video codec based on HiNeRV and a refined pipeline for training, pruning and quantization that can better preserve HiNeRV's performance during lossy model compression. The proposed method has been evaluated on both UVG and MCL-JCV datasets for video compression, demonstrating significant improvement over all existing INRs baselines and competitive performance when compared to learning-based codecs (72.3% overall bit rate saving over HNeRV and 43.4% over DCVC on the UVG dataset, measured in PSNR). △ Less

Submitted 26 January, 2024; v1 submitted 16 June, 2023; originally announced June 2023.

MFSN: Multi-perspective Fusion Search Network For Pre-training Knowledge in Speech Emotion Recognition

Authors: Haiyang Sun, Fulin Zhang, Yingying Gao, Zheng Lian, Shilei Zhang, Junlan Feng

Abstract: Speech Emotion Recognition (SER) is an important research topic in human-computer interaction. Many recent works focus on directly extracting emotional cues through pre-trained knowledge, frequently overlooking considerations of appropriateness and comprehensiveness. Therefore, we propose a novel framework for pre-training knowledge in SER, called Multi-perspective Fusion Search Network (MFSN). Co… ▽ More Speech Emotion Recognition (SER) is an important research topic in human-computer interaction. Many recent works focus on directly extracting emotional cues through pre-trained knowledge, frequently overlooking considerations of appropriateness and comprehensiveness. Therefore, we propose a novel framework for pre-training knowledge in SER, called Multi-perspective Fusion Search Network (MFSN). Considering comprehensiveness, we partition speech knowledge into Textual-related Emotional Content (TEC) and Speech-related Emotional Content (SEC), capturing cues from both semantic and acoustic perspectives, and we design a new architecture search space to fully leverage them. Considering appropriateness, we verify the efficacy of different modeling approaches in capturing SEC and fills the gap in current research. Experimental results on multiple datasets demonstrate the superiority of MFSN. △ Less

Submitted 26 June, 2024; v1 submitted 12 June, 2023; originally announced June 2023.

RGBlimp: Robotic Gliding Blimp -- Design, Modeling, Development, and Aerodynamics Analysis

Authors: Hao Cheng, Zeyu Sha, Yongjian Zhu, Feitian Zhang

Abstract: A miniature robotic blimp, as one type of lighter-than-air aerial vehicle, has attracted increasing attention in the science and engineering field for its long flight duration and safe aerial locomotion. While a variety of miniature robotic blimps have been developed over the past decade, most of them utilize the buoyant lift and neglect the aerodynamic lift in their design, thus leading to a medi… ▽ More A miniature robotic blimp, as one type of lighter-than-air aerial vehicle, has attracted increasing attention in the science and engineering field for its long flight duration and safe aerial locomotion. While a variety of miniature robotic blimps have been developed over the past decade, most of them utilize the buoyant lift and neglect the aerodynamic lift in their design, thus leading to a mediocre aerodynamic performance. This letter proposes a new design of miniature robotic blimp that combines desirable features of both a robotic blimp and a fixed-wing glider, named the Robotic Gliding Blimp, or RGBlimp. This robot, equipped with an envelope filled with helium and a pair of wings, uses an internal moving mass and a pair of propellers for its locomotion control. This letter presents the design, dynamic modeling, prototyping, and system identification of the RGBlimp. To the best of the authors' knowledge, this is the first effort to systematically design and develop such a miniature robotic blimp with hybrid lifts and moving mass control. Experimental results are presented to validate the design and the dynamic model of the RGBlimp. Analysis of the RGBlimp aerodynamics is conducted which confirms the performance improvement of the proposed RGBlimp in aerodynamic efficiency and flight stability. △ Less

Submitted 20 October, 2023; v1 submitted 6 June, 2023; originally announced June 2023.

Journal ref: IEEE Robotics and Automation Letters, vol. 8, no. 11, pp. 7273-7280, Nov. 2023

Doppler-Resilient Design of CAZAC Sequences for mmWave/THz Sensing Applications

Authors: Fan Zhang, Tianqi Mao, Zhaocheng Wang

Abstract: Ultra-high-resolution target sensing has emerged as a key enabler for various cutting-edge applications, which can be realized by utilizing the millimeter wave/terahertz frequencies. However, the extremely high operating frequency inevitably leads to significant Doppler shift effects, especially for high-mobility applications, causing the degradation of sensing performance with high false alarm ra… ▽ More Ultra-high-resolution target sensing has emerged as a key enabler for various cutting-edge applications, which can be realized by utilizing the millimeter wave/terahertz frequencies. However, the extremely high operating frequency inevitably leads to significant Doppler shift effects, especially for high-mobility applications, causing the degradation of sensing performance with high false alarm rate. To this end, this paper proposes a parameter design methodology of the well-known constant amplitude zero auto correlation (CAZAC) sequences, which aims at enhancing their resilience to Doppler shifts. Specifically, we suppress the sidelobes incurred by Doppler shifts for the peak-to-sidelobe ratio (PSLR) improvement within the range of interest (RoI) of the radar range profile. The Zadoff-Chu (ZC) sequence, as a representative member in the CAZAC family, is firstly considered. The impacts of its root index on range sidelobes are investigated based on number theory. For an arbitrary-length ZC sequence, a feasible range of the root index is derived to satisfy the requirement of PSLR within the scope of RoI. Furthermore, these design guidelines are extended to a general form of CAZAC sequences, where a low-complexity heuristic algorithm is developed for PSLR improvement. Simulation results demonstrate that under severe Doppler shifts, our proposed methodology could enhance the sensing performance by lowering the false alarm rate while maintaining the same detection rate, compared with its classical counterpart. △ Less

Submitted 12 May, 2023; originally announced May 2023.

Real-time Autonomous Glider Navigation Software

Authors: Ruochu Yang, Mengxue Hou, Chad Lembke, Catherine Edwards, Fumin Zhang

Abstract: Underwater gliders are widely utilized for ocean sampling, surveillance, and other various oceanic applications. In the context of complex ocean environments, gliders may yield poor navigation performance due to strong ocean currents, thus requiring substantial human effort during the manual piloting process. To enhance navigation accuracy, we developed a real-time autonomous glider navigation sof… ▽ More Underwater gliders are widely utilized for ocean sampling, surveillance, and other various oceanic applications. In the context of complex ocean environments, gliders may yield poor navigation performance due to strong ocean currents, thus requiring substantial human effort during the manual piloting process. To enhance navigation accuracy, we developed a real-time autonomous glider navigation software, named GENIoS Python, which generates waypoints based on flow predictions to assist human piloting. The software is designed to closely check glider status, provide customizable experiment settings, utilize lightweight computing resources, offer stably communicate with dockservers, robustly run for extended operation time, and quantitatively compare flow estimates, which add to its value as an autonomous tool for underwater glider navigation. △ Less

Submitted 20 December, 2023; v1 submitted 25 April, 2023; originally announced April 2023.

Comments: OCEANS 2023 Limerick