Eigendecomposition of a matrix
![]() | This article may be too technical for most readers to understand.(August 2024) |
In linear algebra, eigendecomposition is the factorization of a matrix into a canonical form, whereby the matrix is represented in terms of its eigenvalues and eigenvectors. Only diagonalizable matrices can be factorized in this way. When the matrix being factorized is a normal or real symmetric matrix, the decomposition is called "spectral decomposition", derived from the spectral theorem.
Fundamental theory of matrix eigenvectors and eigenvalues
[edit]A (nonzero) vector v of dimension N is an eigenvector of a square N × N matrix A if it satisfies a linear equation of the form
for some scalar
This yields an equation for the eigenvalues
We call p(
If the field of scalars is algebraically closed, then we can factor p as
The integer ni is termed the algebraic multiplicity of eigenvalue
For each eigenvalue
The eigenvectors can be indexed by eigenvalues, using a double index, with vij being the jth eigenvector for the ith eigenvalue. The eigenvectors can also be indexed using the simpler notation of a single index vk, with k = 1, 2, ..., Nv.
Eigendecomposition of a matrix
[edit]Let A be a square n × n matrix with n linearly independent eigenvectors qi (where i = 1, ..., n). Then A can be factorized as
where Q is the square n × n matrix whose ith column is the eigenvector qi of A, and
The n eigenvectors qi are usually normalized, but they don't have to be. A non-normalized set of n eigenvectors, vi can also be used as the columns of Q. That can be understood by noting that the magnitude of the eigenvectors in Q gets canceled in the decomposition by the presence of Q−1. If one of the eigenvalues
The decomposition can be derived from the fundamental property of eigenvectors: The linearly independent eigenvectors qi with nonzero eigenvalues form a basis (not necessarily orthonormal) for all possible products Ax, for x ∈ Cn, which is the same as the image (or range) of the corresponding matrix transformation, and also the column space of the matrix A. The number of linearly independent eigenvectors qi with nonzero eigenvalues is equal to the rank of the matrix A, and also the dimension of the image (or range) of the corresponding matrix transformation, as well as its column space.
The linearly independent eigenvectors qi with an eigenvalue of zero form a basis (which can be chosen to be orthonormal) for the null space (also known as the kernel) of the matrix transformation A.
Example
[edit]The 2 × 2 real matrix A may be decomposed into a diagonal matrix through multiplication of a non-singular matrix Q
Then for some real diagonal matrix .
Multiplying both sides of the equation on the left by Q:
The above equation can be decomposed into two simultaneous equations:
Factoring out the eigenvalues x and y:
Letting
this gives us two vector equations:
And can be represented by a single vector equation involving two solutions as eigenvalues:
where
Shifting
Putting the solutions back into the above simultaneous equations
Solving the equations, we have Thus the matrix Q required for the eigendecomposition of A is that is:
Matrix inverse via eigendecomposition
[edit]If a matrix A can be eigendecomposed and if none of its eigenvalues are zero, then A is invertible and its inverse is given by
If is a symmetric matrix, since is formed from the eigenvectors of , is guaranteed to be an orthogonal matrix, therefore . Furthermore, because
Practical implications
[edit]When eigendecomposition is used on a matrix of measured, real data, the inverse may be less valid when all eigenvalues are used unmodified in the form above. This is because as eigenvalues become relatively small, their contribution to the inversion is large. Those near zero or at the "noise" of the measurement system will have undue influence and could hamper solutions (detection) using the inverse.[4]
Two mitigations have been proposed: truncating small or zero eigenvalues, and extending the lowest reliable eigenvalue to those below it. See also Tikhonov regularization as a statistically motivated but biased method for rolling off eigenvalues as they become dominated by noise.
The first mitigation method is similar to a sparse sample of the original matrix, removing components that are not considered valuable. However, if the solution or detection process is near the noise level, truncating may remove components that influence the desired solution.
The second mitigation extends the eigenvalue so that lower values have much less influence over inversion, but do still contribute, such that solutions near the noise will still be found.
The reliable eigenvalue can be found by assuming that eigenvalues of extremely similar and low value are a good representation of measurement noise (which is assumed low for most systems).
If the eigenvalues are rank-sorted by value, then the reliable eigenvalue can be found by minimization of the Laplacian of the sorted eigenvalues:[5] where the eigenvalues are subscripted with an s to denote being sorted. The position of the minimization is the lowest reliable eigenvalue. In measurement systems, the square root of this reliable eigenvalue is the average noise over the components of the system.
Functional calculus
[edit]The eigendecomposition allows for much easier computation of power series of matrices. If f (x) is given by
then we know that
Because
The off-diagonal elements of f (
A similar technique works more generally with the holomorphic functional calculus, using from above. Once again, we find that
Examples
[edit]which are examples for the functions . Furthermore, is the matrix exponential.
Decomposition for spectral matrices
[edit]![]() | This section needs expansion. You can help by adding to it. (June 2008) |
Spectral matrices are matrices that possess distinct eigenvalues and a complete set of eigenvectors. This characteristic allows spectral matrices to be fully diagonalizable, meaning they can be decomposed into simpler forms using eigendecomposition. This decomposition process reveals fundamental insights into the matrix's structure and behavior, particularly in fields such as quantum mechanics, signal processing, and numerical analysis.[6]
Normal matrices
[edit]A complex-valued square matrix is normal (meaning , , where is the conjugate transpose) if and only if it can be decomposed as , where is a unitary matrix (meaning ) and diag() is a diagonal matrix.[7] The columns of form an orthonormal basis and are eigenvectors of with corresponding eigenvalues .[8]
For example, consider the 2 x 2 normal matrix .
The eigenvalues are and .
The (normalized) eigenvectors corresponding to these eigenvalues are and .
The diagonalization is , where , and .
The verification is .
This example illustrates the process of diagonalizing a normal matrix by finding its eigenvalues and eigenvectors, forming the unitary matrix , the diagonal matrix , and verifying the decomposition.
![](https://upload.wikimedia.org/wikipedia/commons/thumb/d/d1/Taxonomy_of_Complex_Matrices.svg/428px-Taxonomy_of_Complex_Matrices.svg.png)
Real symmetric matrices
[edit]As a special case, for every n × n real symmetric matrix, the eigenvalues are real and the eigenvectors can be chosen real and orthonormal. Thus a real symmetric matrix A can be decomposed as , where Q is an orthogonal matrix whose columns are the real, orthonormal eigenvectors of A, and
Diagonalizable Matrices
[edit]Diagonalizable matrices can be decomposed using eigendecomposition, provided they have a full set of linearly independent eigenvectors. They can be expressed as, where is a matrix whose columns are eigenvectors of and is a diagonal matrix consisting of the corresponding eigenvalues of .[8]
Positive Definite Matrices
[edit]Positive definite matrices are matrices for which all eigenvalues are positive. They can be decomposed as using the Cholesky decomposition, where is a lower triangular matrix.[10]
Unitary and Hermitian Matrices
[edit]Unitary Matrices satisfy (real case) or (complex case), where denotes the conjugate transpose and denotes the conjugate transpose. They diagonalize using unitary transformations.[8]
Hermitian matrices satisfy , where denotes the conjugate transpose. They can be diagonalized using unitary or orthogonal matrices.[8]
Useful facts
[edit]Useful facts regarding eigenvalues
[edit]- The product of the eigenvalues is equal to the determinant of A Note that each eigenvalue is raised to the power ni, the algebraic multiplicity.
- The sum of the eigenvalues is equal to the trace of A Note that each eigenvalue is multiplied by ni, the algebraic multiplicity.
- If the eigenvalues of A are
λ i, and A is invertible, then the eigenvalues of A−1 are simplyλ −1
i. - If the eigenvalues of A are
λ i, then the eigenvalues of f (A) are simply f (λ i), for any holomorphic function f.
Useful facts regarding eigenvectors
[edit]- If A is Hermitian and full-rank, the basis of eigenvectors may be chosen to be mutually orthogonal. The eigenvalues are real.
- The eigenvectors of A−1 are the same as the eigenvectors of A.
- Eigenvectors are only defined up to a multiplicative constant. That is, if Av =
λ v then cv is also an eigenvector for any scalar c ≠ 0. In particular, −v and eiθ v (for anyθ ) are also eigenvectors. - In the case of degenerate eigenvalues (an eigenvalue having more than one eigenvector), the eigenvectors have an additional freedom of linear transformation, that is to say, any linear (orthonormal) combination of eigenvectors sharing an eigenvalue (in the degenerate subspace) is itself an eigenvector (in the subspace).
Useful facts regarding eigendecomposition
[edit]- A can be eigendecomposed if and only if the number of linearly independent eigenvectors, Nv, equals the dimension of an eigenvector: Nv = N
- If the field of scalars is algebraically closed and if p(
λ ) has no repeated roots, that is, if then A can be eigendecomposed. - The statement "A can be eigendecomposed" does not imply that A has an inverse as some eigenvalues may be zero, which is not invertible.
- The statement "A has an inverse" does not imply that A can be eigendecomposed. A counterexample is , which is an invertible defective matrix.
Useful facts regarding matrix inverse
[edit]- A can be inverted if and only if all eigenvalues are nonzero:
- If
λ i ≠ 0 and Nv = N, the inverse is given by
Numerical computations
[edit]Numerical computation of eigenvalues
[edit]Suppose that we want to compute the eigenvalues of a given matrix. If the matrix is small, we can compute them symbolically using the characteristic polynomial. However, this is often impossible for larger matrices, in which case we must use a numerical method.
In practice, eigenvalues of large matrices are not computed using the characteristic polynomial. Computing the polynomial becomes expensive in itself, and exact (symbolic) roots of a high-degree polynomial can be difficult to compute and express: the Abel–Ruffini theorem implies that the roots of high-degree (5 or above) polynomials cannot in general be expressed simply using nth roots. Therefore, general algorithms to find eigenvectors and eigenvalues are iterative.
Iterative numerical algorithms for approximating roots of polynomials exist, such as Newton's method, but in general it is impractical to compute the characteristic polynomial and then apply these methods. One reason is that small round-off errors in the coefficients of the characteristic polynomial can lead to large errors in the eigenvalues and eigenvectors: the roots are an extremely ill-conditioned function of the coefficients.[11]
A simple and accurate iterative method is the power method: a random vector v is chosen and a sequence of unit vectors is computed as
This sequence will almost always converge to an eigenvector corresponding to the eigenvalue of greatest magnitude, provided that v has a nonzero component of this eigenvector in the eigenvector basis (and also provided that there is only one eigenvalue of greatest magnitude). This simple algorithm is useful in some practical applications; for example, Google uses it to calculate the page rank of documents in their search engine.[12] Also, the power method is the starting point for many more sophisticated algorithms. For instance, by keeping not just the last vector in the sequence, but instead looking at the span of all the vectors in the sequence, one can get a better (faster converging) approximation for the eigenvector, and this idea is the basis of Arnoldi iteration.[11] Alternatively, the important QR algorithm is also based on a subtle transformation of a power method.[11]
Numerical computation of eigenvectors
[edit]Once the eigenvalues are computed, the eigenvectors could be calculated by solving the equation using Gaussian elimination or any other method for solving matrix equations.
However, in practical large-scale eigenvalue methods, the eigenvectors are usually computed in other ways, as a byproduct of the eigenvalue computation. In power iteration, for example, the eigenvector is actually computed before the eigenvalue (which is typically computed by the Rayleigh quotient of the eigenvector).[11] In the QR algorithm for a Hermitian matrix (or any normal matrix), the orthonormal eigenvectors are obtained as a product of the Q matrices from the steps in the algorithm.[11] (For more general matrices, the QR algorithm yields the Schur decomposition first, from which the eigenvectors can be obtained by a backsubstitution procedure.[13]) For Hermitian matrices, the Divide-and-conquer eigenvalue algorithm is more efficient than the QR algorithm if both eigenvectors and eigenvalues are desired.[11]
Additional topics
[edit]Generalized eigenspaces
[edit]Recall that the geometric multiplicity of an eigenvalue can be described as the dimension of the associated eigenspace, the nullspace of
This usage should not be confused with the generalized eigenvalue problem described below.
Conjugate eigenvector
[edit]A conjugate eigenvector or coneigenvector is a vector sent after transformation to a scalar multiple of its conjugate, where the scalar is called the conjugate eigenvalue or coneigenvalue of the linear transformation. The coneigenvectors and coneigenvalues represent essentially the same information and meaning as the regular eigenvectors and eigenvalues, but arise when an alternative coordinate system is used. The corresponding equation is For example, in coherent electromagnetic scattering theory, the linear transformation A represents the action performed by the scattering object, and the eigenvectors represent polarization states of the electromagnetic wave. In optics, the coordinate system is defined from the wave's viewpoint, known as the Forward Scattering Alignment (FSA), and gives rise to a regular eigenvalue equation, whereas in radar, the coordinate system is defined from the radar's viewpoint, known as the Back Scattering Alignment (BSA), and gives rise to a coneigenvalue equation.
Generalized eigenvalue problem
[edit]A generalized eigenvalue problem (second sense) is the problem of finding a (nonzero) vector v that obeys
where A and B are matrices. If v obeys this equation, with some
If n linearly independent vectors {v1, …, vn} can be found, such that for every i ∈ {1, …, n}, Avi =
And since P is invertible, we multiply the equation from the right by its inverse, finishing the proof.
The set of matrices of the form A −
If B is invertible, then the original problem can be written in the form which is a standard eigenvalue problem. However, in most situations it is preferable not to perform the inversion, but rather to solve the generalized eigenvalue problem as stated originally. This is especially important if A and B are Hermitian matrices, since in this case B−1A is not generally Hermitian and important properties of the solution are no longer apparent.
If A and B are both symmetric or Hermitian, and B is also a positive-definite matrix, the eigenvalues
See also
[edit]- Eigenvalue perturbation
- Frobenius covariant
- Householder transformation
- Jordan normal form
- List of matrices
- Matrix decomposition
- Singular value decomposition
- Sylvester's formula
Notes
[edit]- ^ Golub, Gene H.; Van Loan, Charles F. (1996), Matrix Computations (3rd ed.), Baltimore: Johns Hopkins University Press, p. 310, ISBN 978-0-8018-5414-9
- ^ Kreyszig, Erwin (1972), Advanced Engineering Mathematics (3rd ed.), New York: Wiley, p. 273, ISBN 978-0-471-50728-4
- ^ Nering, Evar D. (1970). Linear Algebra and Matrix Theory (2nd ed.). New York: Wiley. p. 270. LCCN 76091646.
- ^ Hayde, A. F.; Twede, D. R. (2002). Shen, Sylvia S. (ed.). "Observations on relationship between eigenvalues, instrument noise and detection performance". Imaging Spectrometry VIII. Proceedings of SPIE. 4816: 355. Bibcode:2002SPIE.4816..355H. doi:10.1117/12.453777. S2CID 120953647.
- ^ Twede, D. R.; Hayden, A. F. (2004). Shen, Sylvia S; Lewis, Paul E (eds.). "Refinement and generalization of the extension method of covariance matrix inversion by regularization". Imaging Spectrometry IX. Proceedings of SPIE. 5159: 299. Bibcode:2004SPIE.5159..299T. doi:10.1117/12.506993. S2CID 123123072.
- ^ Allaire, Grégoire (2008). Numerical linear algebra. ISBN 978-0-387-34159-0.
- ^ Horn & Johnson 1985, p. 133, Theorem 2.5.3
- ^ a b c d Shores, Thomas S (2006). "Applied linear algebra and matrix analysis".
- ^ Horn & Johnson 1985, p. 136, Corollary 2.5.11
- ^ Carl D. Meyer (2023). Matrix analysis and applied linear algebra (2nd ed.). Society for Industrial and Applied Mathematics. ISBN 9781611977431.
- ^ a b c d e f Trefethen, Lloyd N.; Bau, David (1997). Numerical Linear Algebra. SIAM. ISBN 978-0-89871-361-9.
- ^ Ipsen, Ilse, and Rebecca M. Wills, Analysis and Computation of Google's PageRank Archived 2018-09-21 at the Wayback Machine, 7th IMACS International Symposium on Iterative Methods in Scientific Computing, Fields Institute, Toronto, Canada, 5–8 May 2005.
- ^ Quarteroni, Alfio; Sacco, Riccardo; Saleri, Fausto (2000). "section 5.8.2". Numerical Mathematics. Springer. p. 15. ISBN 978-0-387-98959-4.
- ^ a b c Bai, Z.; Demmel, J.; Dongarra, J.; Ruhe, A.; Van Der Vorst, H., eds. (2000). "Generalized Hermitian Eigenvalue Problems". Templates for the Solution of Algebraic Eigenvalue Problems: A Practical Guide. Philadelphia: SIAM. ISBN 978-0-89871-471-5. Archived from the original on 2010-08-21. Retrieved 2022-09-09.
- ^ Parlett, Beresford N. (1998). The symmetric eigenvalue problem (Reprint. ed.). Philadelphia: Society for Industrial and Applied Mathematics. p. 345. doi:10.1137/1.9781611971163. ISBN 978-0-89871-402-9.
References
[edit]- Franklin, Joel N. (1968). Matrix Theory. Dover Publications. ISBN 978-0-486-41179-8.
- Horn, Roger A.; Johnson, Charles R. (1985). Matrix Analysis. Cambridge University Press. ISBN 978-0-521-38632-6.
- Horn, Roger A.; Johnson, Charles R. (1991). Topics in Matrix Analysis. Cambridge University Press. ISBN 978-0-521-46713-1.
- Strang, G. (1998). Introduction to Linear Algebra (3rd ed.). Wellesley-Cambridge Press. ISBN 978-0-9614088-5-5.