Hierarchical Convolutional Neural Network with Knowledge Complementation for Long-Tailed Classification
Article No.: 154, Pages 1 - 22
Abstract
Existing methods based on transfer learning leverage auxiliary information to help tail generalization and improve the performance of the tail classes. However, they cannot fully exploit the relationships between auxiliary information and tail classes and bring irrelevant knowledge to the tail classes. To solve this problem, we propose a hierarchical CNN with knowledge complementation, which regards hierarchical relationships as auxiliary information and transfers relevant knowledge to tail classes. First, we integrate semantics and clustering relationships as hierarchical knowledge into the CNN to guide feature learning. Then, we design a complementary strategy to jointly exploit the two types of knowledge, where semantic knowledge acts as a prior dependence and clustering knowledge reduces the negative information caused by excessive semantic dependence (i.e., semantic gaps). In this way, the CNN facilitates the utilization of the two complementary hierarchical relationships and transfers useful knowledge to tail data to improve long-tailed classification accuracy. Experimental results on public benchmarks show that the proposed model outperforms existing methods. In particular, our model improves accuracy by 3.46% compared with the second-best method on the long-tailed tieredImageNet dataset.
References
[1]
Lida Abdi and Sattar Hashemi. 2015. To combat multi-class imbalanced problems by means of over-sampling techniques. Transactions on Knowledge and Data Engineering 28, 1 (2015), 238–251.
[2]
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. 2019. Learning imbalanced datasets with label-distribution-aware margin loss. In International Conference on Neural Information Processing Systems. 1567–1578.
[3]
Jianlong Chang, Gaofeng Meng, Lingfeng Wang, Shiming Xiang, and Chunhong Pan. 2018. Deep self-evolution clustering. Transactions on Pattern Analysis and Machine Intelligence 42, 4 (2018), 809–823.
[4]
Nitesh V. Chawla, Kevin W. Bowyer, Lawrence O. Hall, and W. Philip Kegelmeyer. 2002. SMOTE: Synthetic minority over-sampling technique. Artificial Intelligence Research 16, 1 (2002), 321–357.
[5]
Haibin Chen, Qianli Ma, Zhenxi Lin, and Jiangyue Yan. 2021. Hierarchy-aware label semantics matching network for hierarchical text classification. In Annual Meeting of the Association for Computational Linguistics. 4370–4379.
[6]
Yin Cui, Menglin Jia, Tsung Yi Lin, Yang Song, and Serge Belongie. 2019. Class-balanced loss based on effective number of samples. In Conference on Computer Vision and Pattern Recognition. 9268–9277.
[7]
Jia Deng, Wei Dong, Richard Socher, Li Jia Li, Kai Li, and Li Fei Fei. 2009. ImageNet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition. 248–255.
[8]
Jia Deng, Jonathan Krause, Alexander C. Berg, and Li Fei Fei. 2012. Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition. In Conference on Computer Vision and Pattern Recognition. 3450–3457.
[9]
Saite Fan, Xinmin Zhang, and Zhihuan Song. 2022. Imbalanced sample selection with deep reinforcement learning for fault diagnosis. Transactions on Industrial Informatics 18, 4 (2022), 2518–2527.
[10]
Yubin Ge, Site Li, Xuyang Li, Fangfang Fan, Wanqing Xie, Jane You, and Xiaofeng Liu. 2021. Embedding semantic hierarchy in discrete optimal transport for risk minimization. In International Conference on Acoustics, Speech and Signal Processing. 2835–2839.
[11]
Hao Guo and Song Wang. 2021. Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings. In Conference on Computer Vision and Pattern Recognition. 15089–15098.
[12]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition. 770–778.
[13]
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. 2016. Learning deep representation for imbalanced classification. In Conference on Computer Vision and Pattern Recognition. 5375–5384.
[14]
Ling Chien Hung, Ya Han Hu, Chih Fong Tsai, and Min Wei Huang. 2022. A dynamic time warping approach for handling class imbalanced medical datasets with missing values: A case study of protein localization site prediction. Expert Systems with Applications 192 (2022), 116437.
[15]
Matheus Inoue, Carlos Henrique Forster, and Antonio Carlos dos Santos. 2020. Semantic hierarchy-based convolutional neural networks for image classification. In International Joint Conference on Neural Networks. 1–8.
[16]
Nathalie Japkowicz. 2000. The class imbalance problem: Significance and strategies. In International Conference on Artificial Intelligence. 111–117.
[17]
Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin. 2020. M2m: Imbalanced classification via major-to-minor translation. In Conference on Computer Vision and Pattern Recognition. 13896–13905.
[18]
Aris Kosmopoulos, Ioannis Partalas, Eric Gaussier, Georgios Paliouras, and Ion Androutsopoulos. 2015. Evaluation measures for hierarchical classification: A unified view and novel approaches. Data Mining and Knowledge Discovery 29, 3 (2015), 820–865.
[19]
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2009. Learning Multiple Layers of Features from Tiny Images. Master’s thesis, University of Toronto. 1–58.
[20]
Tsung Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollr. 2017. Focal loss for dense object detection. In International Conference on Computer Vision. 2980–2988.
[21]
Wei Chao Lin, Chih Fong Tsai, Ya Han Hu, and Jing Shang Jhang. 2017. Clustering-based undersampling in class-imbalanced data. Information Sciences 409 (2017), 17–26.
[22]
Huafeng Liu, Jiaqi Wang, and Liping Jing. 2021. Cluster-wise hierarchical generative model for deep amortized clustering. In Conference on Computer Vision and Pattern Recognition. 15109–15118.
[23]
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X. Yu. 2019. Large-scale long-tailed recognition in an open world. In Conference on Computer Vision and Pattern Recognition. 2537–2546.
[24]
Jianghong Ma, Tommy W. S. Chow, and Haijun Zhang. 2022. Semantic-gap-oriented feature selection and classifier construction in multilabel learning. Transactions on Cybernetics 52, 1 (2022), 101–115.
[25]
Sebastin Maldonado, Carla Vairetti, Alberto Fernandez, and Francisco Herrera. 2022. FW-SMOTE: A feature-weighted oversampling approach for imbalanced classification. Pattern Recognition 124 (2022), 108511.
[26]
Stanislav Naumov, Grigory Yaroslavtsev, and Dmitrii Avdiukhin. 2021. Objective-based hierarchical clustering of deep embedding vectors. In Conference on Artificial Intelligence. 9055–9063.
[27]
Abraham Montoya Obeso, Jenny Benois-Pineau, Mireya Sara Garca Vzquez, and Alejandro lvaro Ramrez Acosta. 2022. Visual vs internal attention mechanisms in deep neural networks for image classification and object detection. Pattern Recognition 123 (2022), 108411.
[28]
Peter Oram. 2001. WordNet: An electronic lexical database. Applied Psycholinguistics 22, 1 (2001), 131–134.
[29]
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B. Tenenbaum, Hugo Larochelle, and Richard S. Zemel. 2018. Meta-learning for semi-supervised few-shot classification. In International Conference on Learning Representations.
[30]
Yong Rui, Thomas S. Huang, and Shih Fu Chang. 1999. Image retrieval: Current techniques, promising directions, and open issues. Journal of Visual Communication and Image Representation 10, 1 (1999), 39–62.
[31]
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115, 3 (2015), 211–252.
[32]
Sungho Suh, Paul Lukowicz, and Yong Oh Lee. 2022. Discriminative feature generation for classification of imbalanced data. Pattern Recognition 122 (2022), 108302.
[33]
Muhammad Atif Tahir, Josef Kittler, and Fei Yan. 2012. Inverse random under sampling for class imbalance problem and its application to multi-label classification. Pattern Recognition 45, 10 (2012), 3738–3750.
[34]
Jingru Tan, Xin Lu, Gang Zhang, Changqing Yin, and Quanquan Li. 2021. Equalization loss v2: A new gradient balance approach for long-tailed object detection. In Conference on Computer Vision and Pattern Recognition. 1685–1694.
[35]
Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. 2020. Equalization loss for long-tailed object recognition. In Conference on Computer Vision and Pattern Recognition. 11662–11671.
[36]
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. 2018. The iNaturalist species classification and detection dataset. In Conference on Computer Vision and Pattern Recognition. 8769–8778.
[37]
Guoyin Wang, Jie Yang, and Ji Xu. 2017. Granular computing: From granularity optimization to multi-granularity joint problem solving. Granular Computing 2, 3 (2017), 105–120.
[38]
Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. 2021. Seesaw loss for long-tailed instance segmentation. In Conference on Computer Vision and Pattern Recognition. 9695–9704.
[39]
Yu Wang, Ruonan Liu, Di Lin, Dongyue Chen, Ping Li, Qinghua Hu, and C. L. Philip Chen. 2023. Coarse-to-fine: Progressive knowledge transfer based multi-task convolutional neural network for intelligent large-scale fault diagnosis. Transactions on Neural Networks and Learning Systems 34, 2 (2023), 761–774.
[40]
Yu Xiong Wang, Deva Ramanan, and Martial Hebert. 2017. Learning to model the tail. In Conference on Neural Information Processing Systems. 7032–7042.
[41]
Jianxiong Xiao, Krista A. Ehinger, James Hays, Antonio Torralba, and Aude Oliva. 2016. SUN database: Exploring a large collection of scene categories. International Journal of Computer Vision 119, 1 (2016), 3–22.
[42]
Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. 2010. SUN database: Large-scale scene recognition from abbey to zoo. In Conference on Computer Vision and Pattern Recognition. 3485–3492.
[43]
Chaoyang Xu, Renjie Lin, Jinyu Cai, and Shiping Wang. 2022. Deep image clustering by fusing contrastive learning and neighbor relation mining. Knowledge-Based Systems 238 (2022), 107967.
[44]
Huaikuan Yi, Qingchao Jiang, Xuefeng Yan, and Bei Wang. 2021. Imbalanced classification based on minority clustering synthetic minority oversampling technique with wind turbine fault detection application. Transactions on Industrial Informatics 17, 9 (2021), 5867–5875.
[45]
Renhui Zhang, Tiancheng Lin, Rui Zhang, and Yi Xu. 2022. Solving the long-tailed problem via intra-and inter-category balance. In International Conference on Acoustics, Speech and Signal Processing. 2355–2359.
[46]
Hong Zhao, Qinghua Hu, Pengfei Zhu, Yu Wang, and Ping Wang. 2021. A recursive regularization based feature selection framework for hierarchical classification. Transactions on Knowledge and Data Engineering 33, 7 (2021), 2833–2846.
[47]
Wei Zhong and Feng Gu. 2022. Predicting local protein 3D structures using clustering deep recurrent neural network. Transactions on Computational Biology and Bioinformatics 19, 1 (2022), 593–604.
[48]
Boyan Zhou, Quan Cui, Xiu Shen Wei, and Zhao Min Chen. 2020. BBN: Bilateral-branch network with cumulative learning for long-tailed visual recognition. In Conference on Computer Vision and Pattern Recognition. 9719–9728.
[49]
Ning Zhou and Jianping Fan. 2013. Jointly learning visually correlated dictionaries for large-scale visual recognition applications. Transactions on Pattern Analysis and Machine Intelligence 36, 4 (2013), 715–730.
[50]
Yu Zhou, Xiaoni Li, Yucan Zhou, Yu Wang, Qinghua Hu, and Weiping Wang. 2022. Deep collaborative multi-task network: A human decision process inspired model for hierarchical image classification. Pattern Recognition 124 (2022), 108449.
[51]
Linchao Zhu and Yi Yang. 2022. Label independent memory for semi-supervised few-shot video classification. Transactions on Pattern Analysis and Machine Intelligence 44, 1 (2022), 273–285.
Index Terms
- Hierarchical Convolutional Neural Network with Knowledge Complementation for Long-Tailed Classification
Recommendations
Multi-task convolutional neural network with coarse-to-fine knowledge transfer for long-tailed classification
AbstractLong-tailed classifications make it very challenging to deal with class-imbalanced problems using deep convolutional neural networks (CNNs). Existing solutions based on re-balancing methods perform well and use single-task CNNs to train each fine-...
Hierarchical long-tailed classification based on multi-granularity knowledge transfer driven by multi-scale feature fusion
AbstractLong-tailed learning is attracting increasing attention due to the unbalanced distributions of real-world data. The aim is to train well-performing depth models. Traditional knowledge transfer methods for long-tailed learning are classified into ...
Highlights- We propose a multi-scale feature fusion network about channel and spatial features.
- We investigate a multi-granularity relationship of class space.
- We explore a vertical transfer of coarse- to fine-grained knowledge.
Geometric Prior Guided Feature Representation Learning for Long-Tailed Classification
AbstractReal-world data are long-tailed, the lack of tail samples leads to a significant limitation in the generalization ability of the model. Although numerous approaches of class re-balancing perform well for moderate class imbalance problems, ...
Comments
Information & Contributors
Information
Published In
July 2024
760 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 April 2024
Online AM: 22 March 2024
Accepted: 16 March 2024
Revised: 12 December 2023
Received: 09 October 2022
Published in TKDD Volume 18, Issue 6
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- National Natural Science Foundation of China
- Natural Science Foundation of Fujian Province
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 157Total Downloads
- Downloads (Last 12 months)157
- Downloads (Last 6 weeks)24
Reflects downloads up to 09 Aug 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in