Semi-supervised Multi-view Clustering based on NMF with Fusion Regularization

Most of unsupervised MVNMF methods are based on the idea of aligning the multiple views in a shared common subspace to fuse the information. In [32], Liu et al. proposed a pioneer MVNMF method which tries to learn a consensus representation by minimizing a centroid co-regularization. With this regularization, the feature matrices of the different views are pushed toward a consensus feature matrix, and then multiview alignment is achieved. Furthermore, a feature scale normalization matrix is introduced to constrain the feature scales of different views to be similar hoping to facilitate the feature alignment. In [39], Rai et al. have adopted the same strategy to make the scales of different views comparable. In [52], Yang et al. have explored another way to reduce the distribution divergences of the feature matrix. They presented a MVNMF based on nonnegative matrix tri-factorization [53]. In order to restrict the feature scale of different views to be similar, the column-summation of the product matrix of basis matrix and shared embedding matrix are constrained to be 1. Essentially, Liu et al. [32], Rai et al. [39], and Yang et al. [52] have adopted the same feature normalizing strategy, in which the column-summation of the basis matrix is constrained to be 1, i.e., \(\sum \nolimits _i {{{\bf W}}_{i\cdot }^v = 1}\) , and the norms are compensated to the coefficient matrix in the co-regularization term. Actually, to improve the performance of unsupervised MVNMFs, various normalizing operations are explored. For example, to guarantee the numerical stability of NMF, Shao et al. [41] have constrained the column-summation of the basis matrix to be 1 and do not compensate the norms to the coefficient matrix in the co-regularization term. To ensure the solution uniqueness of NMF, Hu et al. [17] have constrained the row-summation of the coefficient matrix to be 1, i.e., \(\sum \nolimits _j {{{\bf H}}_{\cdot j}^v = 1}\) . Most of existing methods are based on the centroid co-regularization. While, in [46], a pair-wise co-regularization was proposed to align the multiple views pair-wisely. By minimizing this co-regularization, the alignment is acquired through pushing the representations of different views together. Until now, we can find that, most of the MVNMF methods were developed from aligning the multiple views, and ignored the diversity of the different views. Addressing this issue, in [42], an MVNMF called diverse nonnegative matrix factorization has been proposed. In this method, a diverse term is designed to encourage the heterogeneity of different views and try to learn some more comprehensive information. Recently, with the flourish of deep learning in computer vision and pattern recognition, some “deep” models are proposed for multi-view clustering task [13]. For example, a deep multi-view semi-nonnegative matrix factorization is proposed in [60]. In this method, an adaptive weighting strategy is adopted to balance the effect of different views. Similarly, a deep MVNMF is proposed in [26]. The graph regularization is applied in all layers not only in the final layer like [60] and all factor matrices are restricted to be nonnegative. All of the previously reviewed methods focus on dealing with “well-established” data. It means that all views have the corrected correspondence on sample-level. In [20], Huang et al. have developed a partially view-aligned clustering method which can tackle the multi-view data not fully view-aligned. “Partially view-aligned” means that only a part of data have the corrected correspondence on sample-level.

In recent years, several semi-supervised MVNMFs are proposed and promising results are obtained with label information. Wang et al. [43] have proposed a semi-supervised MVNMF based on CNMF for the first time. In this work, an adaptive weighting strategy is applied to balance the effect of different views. To learn a consensus representation, a similar centroid co-regularization like [32] is utilized in their method. In [5] and [6], Cai et al. have developed two semi-supervised MVNMF methods; these two methods are also based on CNMF. Both methods have adopted pair-wise co-regularization with Euclidean distance to align multiple views like [46]. The difference between [5] and [6] is the second regularizing term. In [5], \(\ell _{2,1}\) -norm regularization of the auxiliary matrix is used, and an orthonormality constraint on the auxiliary matrix is used in [6]. The former one tries to get sparse representations for each view, and the latter one tries to constrain the feature scale of different views to be similar. Yang et al. [54] have improved the work in [6] by relaxing the label constraint matrix. The common drawback of above methods is that they all have ignored the discriminative information of data. To make use of this information, Jiang et al. [22] have proposed a unified latent factor learning model trying to reconstruct the label matrix with the product of partial shared coefficient matrix and an auxiliary matrix. Further, Liu et al. [31] have improved the above work by splitting the coefficient matrix into the private part and the common part. Based on [31], Liang et al. [28] have tried to improve the performance of the method by imposing a graph regularization on the coefficient matrix. The problem with [28] is that it uses too many hyper-parameters, which makes this method very complex and hard to fine-tune. Wang et al. [44] have presented a semi-supervised multi-view clustering model with weighted anchor graph embedding, in which the anchors are constructed based on label information. Nie et al. [36] have developed a semi-supervised multi-view learning method which can model the structure of data adaptively. Based on this work, Liang et al. [29] have presented a semi-supervised MVNMF method based on label propagation. Zhang et al. [58] have proposed a semi-supervised MVNMF method, called dual hypergraph regularized partially shared NMF, in which hypergraph regularization was imposed on both the coefficient matrix and the basis matrix. Similar to [28], these two methods also suffer from involving too many hyper-parameters. Recently, some deep multi-view clustering methods were also developed. Zhao et al. [62] have proposed a deep semi-supervised MVNMF method which encodes the discriminative information of data by constructing an affinity graph for intra-class compactness and a penalty graph for inter-class distinctness. Chen et al. [11] have presented an autoencoder-based semi-supervised multi-view clustering method which encodes the discriminative information of data by introducing a pairwise constraint.

From above review, we can find that, although various matrix normalizing strategies are widely explored in many unsupervised MVNMF methods, these stategies are rarely discussed under the semi-supervised MVNMF framework. In these previously introduced normalizing strategies, the column-summation of the basis matrix is usually normalized to be 1, and sometimes the norms are compensated to the coefficient matrix in the co-regularization term. However, it is more reasonable to constrain the columns of the basis to be unit vectors, i.e., \(||{{\bf W}}_{\cdot j}^v|{|_2} = 1\) . Additionally, recently proposed semi-supervised MVNMF methods have involved too many hyper-parameters, which has made these methods very complex and hard to fine-tune. Addressing these issues, in the following sections, we will introduce a novel semi-supervised MVNMF framework with fusion regularization. This framework can be implemented with two different feature normalizing strategies. And the fusion regularization can make use of the discriminative information of data and reduce the number of hyper-parameters.

The optimization problem of minimizing Equation (8) is not direct. In this section, an alternating optimization strategy is developed, which breaks the original problem into several subproblems such that each subproblem is tractable.

For a specific view \(v\) , its optimization with \({\bf W}^{v}\) and \({\bf H}^{v}\) does not depend on other views. The problem of minimizing Equation (8) can be written as follows:

Comparing with FRSMNMF_N1, the main difference of optimizing procedure for FRSMNMF_N2 lies in the updating rule of \({\bf W}^v\) . So, in this section, only the updating rule of \({\bf W}^v\) is deduced. Fixing \({\bf H}_c\) and \({\bf H}^v\) , the objective function of FRSMNMF_N2 with respect to \({\bf W}^v\) is written as follows:where \({\bf Q}^v\) is defined as in Equation (10).

As \({{\bf W}^v} \ge 0\) , introducing the Lagrangian multipliers \(\psi _{ih}\) for the constraints \({\bf W}^v_{ih} \ge 0\) . Let \(\Psi =[\psi _{ih}]\) . Then, the Lagrangian function \({\mathcal {L}}\) is as follows:

The partial derivative of (30) with respect to \({\bf W}^v\) can be written as follows:where \({\bf Y}^v_1\) , \({\bf Y}^v_2\) and \({{\bf Y}^v_3}\) are defined in Equation (13) and Equation (17).

Setting the above equation to zero and using the KKT condition of \(\psi _{ih} {\bf W}^v_{ih}=0\) , then we havewhich leads to the following update rule:

The updating rules of \({{\bf H}^v}\) and \({{\bf H}_c}\) in FRSMNMF_N2 are the same as those in Equation (26) and Equation (28). The optimizing scheme of FRSMNMF_N2 is summarized in Algorithm 2. The convergence proofs for FRSMNMF_N1 and FRSMNMF_N2 are presented in the Appendix.

In this section, the computational complexity of FRSMNMF_N1 and FRSMNMF_N2 is analyzed. The computational complexity is expressed in a big \(O\) notation [12]. For a specific view \(v\) of FRSMNMF_N1 in one iteration, updating \({\bf W}^v\) needs to calculate \({{\bf W}^v}{{\bf H}^v}{{\bf H}^v}^T\) , \({{\bf X}^v}{{\bf H}^v}^T\) , \({{\bf W}^v}{\bf Y}_1^v\) , \({{\bf W}^v}{\bf Y}_2^v,\) and \({{\bf W}^v}({{\bf Q}^v})^{ - 1}{\bf Y}_3^v\) . \({{\bf W}^v}{{\bf H}^v}{{\bf H}^v}^T\) and \({{\bf X}^v}{{\bf H}^v}^T\) cost \(O((n + m^v)d^{v2})\) and \(O(m^v n d^v),\) respectively. Total cost of \({{\bf W}^v}{\bf Y}_1^v\) , \({{\bf W}^v}{\bf Y}_2^v\) and \({{\bf W}^v}({{\bf Q}^v})^{ - 1}{\bf Y}_3^v\) is \(O(m^vd^v + Kd^v n + d^{v2} n)\) . \(K\) is the number of \(k\) nearest neighbors in graph. So, the cost for updating \({\bf W}^v\) is \(O(Kd ^v n + m^v n d^v)\) . Normalize \({\bf W}^v\) and \({\bf H}^v\) in Equation (21) needs \(O(nd^v + m^v d^v)\) computation. The main cost of updating \({\bf H}^v\) is on the calculation of \({{\bf W}^v}^T{{\bf X}^v}\) , \({{\bf W}^v}^T{{\bf W}^v}{{\bf H}^v}\) , \({\bf H}^v{\bf S}^v\) and \({\bf H}^v{\bf D}^v\) . The cost on \({{\bf W}^v}^T{{\bf X}^v}\) and \({{\bf W}^v}^T{{\bf W}^v}{{\bf H}^v}\) are \(O(m^v n d^v)\) and \(O((n + m^v)d^{v2})\) . While \({{\bf H}^v}{{\bf S}^v}\) and \({{\bf H}^v}{{\bf D}^v}\) need \(O(Kd^v n)\) and \(O(d^v n)\) computation respectively. The cost for updating \({\bf H}^v\) is \(O(Kd^v n + m^v n d^v)\) . Comparing with the cost of updating \({\bf W}^v\) and \({\bf H}^v\) , the computation cost on updating \({\bf H}_c\) is ignorable. So, the final cost for FRSMNMF_N1 is \(O(t V m^v n d^v)\) . \(t\) is the number of iterations and \(V\) denotes the number of views.

For FRSMNMF_N2, the main difference of computation cost resides on the updating of \({\bf W}^v\) . Except the computation cost on calculating \({{\bf W}^v}{{\bf H}^v}{{\bf H}^v}^T\) and \({{\bf X}^v}{{\bf H}^v}^T\) , it also need to compute \({{\bf Q}^v}{\bf Y}_1^v\) , \({{\bf Q}^v}{\bf Y}_2^v\) and \({\bf Y}_3^v\) . The total computation cost on these additional terms is \(O(Kd^v n + d^{v2}n)\) . So, the cost for updating \({\bf W}^v\) in FRSMNMF_N2 is \(O(Kd^v n + m^v n d^v)\) . The final cost for FRSMNMF_N2 is \(O(t V m^v n d^v)\) .

Two widely used clustering metrics are adopted to evaluate the performance of the methods: clustering accuracy (AC) [49] and normalized mutual information (NMI) [40].

AC is defined aswhere \(n\) is the number of samples in each multiview dataset. \(gnd_i\) is the ground truth label of \(x_i\) . \(z_i\) corresponds to the label that is obtained by clustering methods due to the sample \(x_i\) . \(map(\cdot)\) is an optimal mapping function [33] which can map the obtained label \(z_i\) to match the ground truth label provided by the datasets. \(\delta (a,b)\) is an indicator function. \(\delta (a,b) = 1\) , when \(a=b\) , otherwise, \(\delta (a,b) = 0\) .

NMI is defined as follows:where \(C\) and \(\hat{C}\) are two cluster sets, one for ground truth and another for the label obtained by clustering methods. \(\rm {H}(C)\) and \(\rm {H}(\hat{C})\) denote the entropy of cluster sets \(C\) and \(\hat{C}\) . \(\rm {MI}(C,\hat{C})\) is the mutual information between \(C\) and \(\hat{C}\) , which is defined aswhere \(p(c_i,c_j)\) denotes the joint probability that a randomly selected item belongs to \(c_i\) and \(c_j\) simultaneously, while \(p(c_i)\) and \(p(c_j)\) denote the probability that a randomly selected item belongs to \(c_i\) and \(c_j\) , respectively.

In the experiments, six multiview datasets are used to evaluate the effectiveness of the proposed method on the clustering task. The statistics are summarized in Table 1.

For Yale, ORL, and FEI datasets, the multiple views are the image intensity, Gabor [23], and LBP [37] with the dimensions of 1,024; 2,048; and 256, respectively.

The clustering results are obtained by conducting k-means clustering method on the learned representations of above algorithms. To avoid errors from randomness, all unsupervised methods were tested 10 times, and the average value is reported. To evaluate the performance of semi-supervised methods, for parameter analysis experiments, 10 \(\%\) , 20 \(\%,\) and 30 \(\%\) data points are labeled randomly for 5 times, and the average clustering results are reported. For comparison experiments, 10 \(\%\) , 20 \(\%,\) and 30 \(\%\) data points are labeled randomly for 20 times, and the average clustering results and statistically significant differences (p-value) are reported.

To evaluate the effectiveness of the proposed methods, they are compared with several representative NMF-based multi-view clustering approaches, including seven unsupervised ones and four semi-supervised ones. These methods are described as follows:

VAGNMF. Conduct GNMF [4] on each view individually and average feature of multiple views are treated as final representation.

VCGNMF. Conduct GNMF on each view individually and concatenate the features of multiple views as final representation.

LP-DiNMF [42]. The objective function of this method involves a diverse term which is defined to enforce the heterogeneity of the different views. Local geometric structure information is also utilized to improve the performance.

rNNMF [10]. This method attempts to deal with the noise and outliers among the views through defining a reconstruction term and a neighbor-structure-preserving term with \(\ell _{2,1}\) -norm.

MPMNMF_1 [46]. This method has adopted the Euclidean distance based pair-wise co-regularization to align multiple features. Graph regularization is constructed to encode the geometry information of each view.

MPMNMF_2 [46]. This method has adopted the kernel based pair-wise co-regularization to align multiple features. Graph regularization is constructed to encoding the geometry information of each view.

UDNMF [52]. This method factorizes the each view into three matrices and constrain the columns of product matrix of basis matrix and shared embedding matrix to be unit vector. Graph regularization is imposed on the coefficient matrix of each view. Centroid co-regularization is used to learn a common consensus matrix as the final representation.

MVCDMF [60]. This is a deep multi-view semi-NMF algorithm. “semi-nonnegative” means that it only constrains the shared coefficient matrix to be nonnegative. Graph regularization is also used to utilize the geometrical structure information of data. The shared coefficient matrix is used as the final representation.

MvDGNMF [26]. This is a deep multi-view NMF algorithm. It iteratively factorizes the coefficient matrix in each layer to get hierarchical representations of data. Similar to UDNMF, it adopts centroid co-regularization to fuse the multiple views to obtain the final representation.

AMVNMF [43]. This method factorizes each view in CNMF [30] framework and try to learn a consensus auxiliary matrix. The entries summation of each columns of auxiliary matrix is constrained to be 1. An auto-weighting strategy is imposed on the consensus learning terms.

MVCNMF [5]. This method factorizes each view in CNMF framework and align multiple views with Euclidean distance based pair-wise co-regularization. \(\ell _{2,1}\) -norm regularization is imposed on the auxiliary matrix in each view.

MVOCNMF [6]. This method factorizes each view in CNMF framework and align multiple views with Euclidean distance based pair-wise co-regularization. An orthonormality constraint is imposed on the auxiliary matrix in each view to normalize the scale of the feature.

GPSNMF [28]. This method is a semi-supervised partially shared MVNMF, which tries to make use of both distinct and shared information of multiple views. A graph regularization is also constructed for each view.

The results of above methods with optimal parameter settings (obtained by grid search) are reported. Note that, before conducting all methods, the data points of the original data are normalized as unit vectors. The source code of this paper is available at: https://github.com/GuoshengCui/FRSMNMF.

To verify the convergence of FRSMNMF_N1 and FRSMNMF_N2, the convergence curves on six datasets with labeling ratios of 10 \(\%\) , 20 \(\%,\) and 30 \(\%\) are visualized in Figure 1. We also visualize the variation of clustering performance with different iterations. In each figure, the left y-axis denotes the objective function value, the right y-axis denotes the clustering accuracy and the x-axis is the iteration number. We can see that the proposed methods converge on all datasets and the clustering performances on all datasets become stable in 300 iterations.