Eigenvalue decomposition - Fundamentals of Numerical Computation

To this point we have dealt frequently with the solution of the linear system $\mathbf{A}\mathbf{x}=\mathbf{b}$ . Alongside this problem in its importance to linear algebra is the eigenvalue problem.

7.2.1Complex matrices¶

A matrix with real entries can have complex eigenvalues. Therefore we assume all matrices, vectors, and scalars may be complex in what follows. Recall that a complex number can be represented as $a+i b$ for real $a$ and $b$ and where $i^2=-1$ . The complex conjugate of $x=a+i b$ is denoted $\bar{x}$ and is given by $\bar{x}=a-i b$ . The magnitude or modulus of a complex number $z$ is

|z| = \sqrt{z\cdot \bar{z}}.

(7.2.2)

Definition 7.2.2 (Terms for complex matrices)

The adjoint or hermitian of a matrix $\mathbf{A}$ is denoted $\mathbf{A}^*$ and is given by $\mathbf{A}^*=(\overline{\mathbf{A}})^T=\overline{\mathbf{A}^T}$ . The matrix is self-adjoint or hermitian if $\mathbf{A}^*=\mathbf{A}$ .

The 2-norm of a complex vector $\mathbf{u}$ is $\sqrt{\mathbf{u}^*\mathbf{u}}$ . Other vector norms, and all matrix norms, are as defined in Vector and matrix norms.

Complex vectors $\mathbf{u}$ and $\mathbf{v}$ of the same dimension are orthogonal if $\mathbf{u}^*\mathbf{v}=0$ . Orthonormal vectors are mutually orthogonal and have unit 2-norm. A unitary matrix is a square matrix with orthonormal columns, or, equivalently, a matrix satisfying $\mathbf{A}^* = \mathbf{A}^{-1}$ .

For the most part, “adjoint” replaces “transpose,” “hermitian” replaces “symmetric,” and “unitary matrix” replaces “orthogonal matrix” when applying our previous results to complex matrices.

7.2.2Eigenvalue decomposition¶

An easy rewrite of the eigenvalue definition (7.2.1) is that $(\mathbf{A} - \lambda\mathbf{I}) \mathbf{x} = \boldsymbol{0}$ . Hence $(\mathbf{A} - \lambda\mathbf{I})$ is singular, and it therefore must have a zero determinant. This is the property most often used to compute eigenvalues by hand.

The determinant $\det(\mathbf{A} - \lambda \mathbf{I})$ is called the characteristic polynomial. Its roots are the eigenvalues, so we know that an $n\times n$ matrix has $n$ eigenvalues, counting algebraic multiplicity.

Suppose that $\mathbf{A}\mathbf{v}_k=\lambda_k\mathbf{v}_k$ for $k=1,\ldots,n$ . We can summarize these as

\begin{split} \begin{bmatrix} \mathbf{A}\mathbf{v}_1 & \mathbf{A}\mathbf{v}_2 & \cdots & \mathbf{A}\mathbf{v}_n \end{bmatrix} &= \begin{bmatrix} \lambda_1 \mathbf{v}_1 & \lambda_2\mathbf{v}_2 & \cdots & \lambda_n \mathbf{v}_n \end{bmatrix}, \\[1mm] \mathbf{A} \begin{bmatrix} \mathbf{v}_1 & \mathbf{v}_2 & \cdots & \mathbf{v}_n \end{bmatrix} &= \begin{bmatrix} \mathbf{v}_1 & \mathbf{v}_2 & \cdots & \mathbf{v}_n \end{bmatrix} \begin{bmatrix} \lambda_1 & & & \\ & \lambda_2 & & \\ & & \ddots & \\ & & & \lambda_n \end{bmatrix}, \end{split}

(7.2.5)

which we write as

\mathbf{A} \mathbf{V} = \mathbf{V} \mathbf{D}.

(7.2.6)

If we find that $\mathbf{V}$ is a nonsingular matrix, then we arrive at a key factorization.^[1]

Observe that if $\mathbf{A}\mathbf{v} = \lambda \mathbf{v}$ for nonzero $\mathbf{v}$ , then the equation remains true for any nonzero multiple of $\mathbf{v}$ . Therefore, eigenvectors are not unique, and thus neither is an EVD.

We stress that while (7.2.6) is possible for all square matrices, (7.2.7) is not. One simple example of a nondiagonalizable matrix is

\mathbf{B} = \begin{bmatrix} 1 & 1\\0 & 1 \end{bmatrix}.

(7.2.8)

There is a common circumstance in which we can guarantee an EVD exists. The proof of the following theorem can be found in many elementary texts on linear algebra.

Example 7.2.2 (Eigenvalues and eigenvectors Eigenvalues and eigenvectors)

Julia

MATLAB

Python

Example 7.2.2

The eigvals function returns a vector of the eigenvalues of a matrix.

A = π * ones(2, 2)

2×2 Matrix{Float64}:
 3.14159  3.14159
 3.14159  3.14159

λ = eigvals(A)

2-element Vector{Float64}:
 0.0
 6.283185307179586

If you want the eigenvectors as well, use eigen.

λ, V = eigen(A)

Eigen{Float64, Float64, Matrix{Float64}, Vector{Float64}}
values:
2-element Vector{Float64}:
 0.0
 6.283185307179586
vectors:
2×2 Matrix{Float64}:
 -0.707107  0.707107
  0.707107  0.707107

norm(A * V[:, 2] - λ[2] * V[:, 2])

0.0

Both functions allow you to sort the eigenvalues by specified criteria.

A = diagm(-2.3:1.7)
@show eigvals(A, sortby=real);
@show eigvals(A, sortby=abs);

eigvals(A, sortby = real) = [-2.3, -1.3, -0.3, 0.7, 1.7]


eigvals(A, sortby = abs) =

[-0.3, 0.7, -1.3, 1.7, -2.3]

If the matrix is not diagonalizable, no message is given, but V will be singular. The robust way to detect that circumstance is via $\kappa(\mathbf{V})$ .

A = [-1 1; 0 -1]
λ, V = eigen(A)

Eigen{Float64, Float64, Matrix{Float64}, Vector{Float64}}
values:
2-element Vector{Float64}:
 -1.0
 -1.0
vectors:
2×2 Matrix{Float64}:
 1.0  -1.0
 0.0   2.22045e-16

cond(V)

9.007199254740991e15

Even in the nondiagonalizable case, $\mathbf{A}\mathbf{V} = \mathbf{V}\mathbf{D}$ holds.

opnorm(A * V - V * diagm(λ))

2.220446049250313e-16

Example 7.2.2

The eig function with one output argument returns a vector of the eigenvalues of a matrix.

A = pi * ones(2, 2);
lambda = eig(A)

With two output arguments given, eig returns a matrix eigenvectors and a diagonal matrix with the eigenvalues.

[V, D] = eig(A)

We can check the fact that this is an EVD.

norm( A - V*D/V )   % / V is like * inv(V)

If the matrix is not diagonalizable, no message is given, but V will be singular. The robust way to detect that circumstance is via $\kappa(\mathbf{V})$ .

A = [-1 1; 0 -1];
[V, D] = eig(A)

cond(V)

Even in the nondiagonalizable case, $\mathbf{A}\mathbf{V} = \mathbf{V}\mathbf{D}$ holds.

norm(A * V - V * D)

Example 7.2.2

The eig function from scipy.linalg will return a vector of eigenvalues and a matrix of associated eigenvectors.

from numpy.linalg import eig
A = pi * ones([2, 2])
d, V = eig(A)
print("eigenvalues:", d)

eigenvalues: [ 6.28318531e+00 -6.97573700e-16]

We can check the fact that this is an EVD (although in practice we never invert a matrix).

from numpy.linalg import inv
D = diag(d)
print(f"should be near zero: {norm(A - V @ D @ inv(V), 2):.2e}")

should be near zero: 1.26e-15

If the matrix is not diagonalizable, no message is given, but V will be singular. The robust way to detect that circumstance is via $\kappa(\mathbf{V})$ .

from numpy.linalg import cond
A = array([[1, 1], [0, 1]])
d, V = eig(A)
print(f"cond(V) is {cond(V):.2e}")

cond(V) is 9.01e+15

But even in the nondiagonalizable case, $\mathbf{A}\mathbf{V} = \mathbf{V}\mathbf{D}$ holds up to roundoff error.

print(f"should be near zero: {norm(A @ V - V @ diag(d), 2):.2e}")

should be near zero: 2.22e-16

7.2.3Similarity and matrix powers¶

The particular relationship between matrices $\mathbf{A}$ and $\mathbf{D}$ in (7.2.7) is important.

A similarity transformation does not change eigenvalues, a fact that is typically proved in elementary linear algebra texts.

The EVD is especially useful for matrix powers. To begin,

\mathbf{A}^2=(\mathbf{V}\mathbf{D}\mathbf{V}^{-1})(\mathbf{V}\mathbf{D}\mathbf{V}^{-1})=\mathbf{V}\mathbf{D}(\mathbf{V}^{-1}\mathbf{V})\mathbf{D}\mathbf{V}^{-1}=\mathbf{V}\mathbf{D}^2\mathbf{V}^{-1}.

(7.2.9)

Multiplying this result by $\mathbf{A}$ repeatedly, we find that

\mathbf{A}^k = \mathbf{V}\mathbf{D}^k\mathbf{V}^{-1}.

(7.2.10)

Because $\mathbf{D}$ is diagonal, its power $\mathbf{D}^k$ is just the diagonal matrix of the $k$ th powers of the eigenvalues.

Furthermore, given a polynomial $p(z)=c_0+c_1 z + \cdots + c_m z^m$ , we can apply the polynomial to the matrix in a straightforward way,

p(\mathbf{A}) = c_0\mathbf{I} +c_1 \mathbf{A} + \cdots + c_m \mathbf{A}^m.

(7.2.11)

Applying (7.2.10) leads to

\begin{split} p(\mathbf{A}) & = c_0\mathbf{V}\mathbf{V}^{-1} +c_1 \mathbf{V}\mathbf{D}\mathbf{V}^{-1} + \cdots + c_m \mathbf{V}\mathbf{D}^m\mathbf{V}^{-1} \\ &= \mathbf{V} \cdot [ c_0\mathbf{I} +c_1 \mathbf{D} + \cdots + c_m \mathbf{D}^m] \cdot \mathbf{V}^{-1} \\[1mm] &= \mathbf{V} \cdot \begin{bmatrix} p(\lambda_1) & & & \\ & p(\lambda_2) & & \\ & & \ddots & \\ & & & p(\lambda_n) \end{bmatrix} \cdot \mathbf{V}^{-1}. \end{split}

(7.2.12)

Finally, given the convergence of Taylor polynomials to common functions, we are able to apply a function $f$ to a square matrix by replacing $p$ with $f$ in (7.2.11).

7.2.4Conditioning of eigenvalues¶

Just as linear systems have condition numbers that quantify the effect of finite precision, eigenvalue problems may be poorly conditioned too. While many possible results can be derived, we will use just one, the Bauer–Fike theorem.

The Bauer–Fike theorem tells us that eigenvalues can be perturbed by an amount that is $\kappa(\mathbf{V})$ times larger than perturbations to the matrix. This result is a bit less straightforward than it might seem—eigenvectors are not unique, so there are multiple possible values for $\kappa(\mathbf{V})$ . Even so, the theorem indicates caution when a matrix has eigenvectors that form an ill-conditioned matrix. The limiting case of $\kappa(\mathbf{V})=\infty$ might be interpreted as indicating a nondiagonalizable matrix $\mathbf{A}$ . The other extreme is also of interest: $\kappa(\mathbf{V})=1$ , which implies that $\mathbf{V}$ is unitary.

As we will see in Symmetry and definiteness, hermitian and real symmetric matrices are normal. Since the condition number of a unitary matrix is equal to 1, (7.2.13) guarantees that a perturbation of a normal matrix changes the eigenvalues by the same amount or less.

Example 7.2.3 (Eigenvalue conditioning)

Julia

MATLAB

Python

Example 7.2.3

We first define a hermitian matrix. Note that the ' operation is the adjoint and includes complex conjugation.

n = 7
A = randn(n, n) + 1im * randn(n, n)
A = (A + A') / 2

7×7 Matrix{ComplexF64}:
 0.325505+0.0im        0.490234-0.743158im   …   0.396406+0.590502im
 0.490234+0.743158im    1.85613+0.0im           -0.547247+1.12398im
 0.515176-1.17467im    0.404081+0.0817125im     -0.457196-0.442748im
   0.1643-1.43759im   -0.438945+0.207581im      -0.988933-0.24625im
 0.396863+0.555028im   0.363285-0.815487im      -0.572039+0.129132im
 -1.23095+1.61663im    0.620866-0.0761188im  …  -0.469493-0.741019im
 0.396406-0.590502im  -0.547247-1.12398im        -1.19368+0.0im

We confirm that the matrix $\mathbf{A}$ is normal by checking that $\kappa(\mathbf{V}) = 1$ (to within roundoff).

λ, V = eigen(A)
@show cond(V);

cond(V) = 1.0000000000000007

Now we perturb $\mathbf{A}$ and measure the effect on the eigenvalues. The Bauer–Fike theorem uses absolute differences, not relative ones.

ΔA = 1e-8 * normalize(randn(n, n) + 1im * randn(n, n))
λ̃ = eigvals(A + ΔA)
dist = minimum([abs(x - y) for x in λ̃, y in λ], dims=2)

7×1 Matrix{Float64}:
 1.1607368132587166e-9
 8.506184094346277e-10
 1.901690883719605e-9
 1.2162453210897967e-9
 1.3795003762031114e-9
 1.1221499944880294e-9
 1.2396386976731458e-9

As promised, the perturbations in the eigenvalues do not exceed the normwise perturbation to the original matrix.

Now we see what happens for a triangular matrix.

n = 20
x = 1:n
A = triu(x * ones(n)')
A[1:5, 1:5]

5×5 Matrix{Float64}:
 1.0  1.0  1.0  1.0  1.0
 0.0  2.0  2.0  2.0  2.0
 0.0  0.0  3.0  3.0  3.0
 0.0  0.0  0.0  4.0  4.0
 0.0  0.0  0.0  0.0  5.0

This matrix is not especially close to normal.

λ, V = eigen(A)
@show cond(V);

cond(V) = 6.149906664929389e9

As a result, the eigenvalues can change by a good deal more.

ΔA = 1e-8 * normalize(randn(n, n) + 1im * randn(n, n))
λ̃ = eigvals(A + ΔA)
dist = minimum([abs(x - y) for x in λ̃, y in λ], dims=2)
BF_bound = cond(V) * norm(ΔA)
@show maximum(dist), BF_bound;

(maximum(dist), BF_bound) = (0.494851347444697, 61.499066649293894)

If we plot the eigenvalues of many perturbations, we get a cloud of points that roughly represents all the possible eigenvalues when representing this matrix with single-precision accuracy.

using Plots
plt = scatter(λ, zeros(n), aspect_ratio=1)
for _ in 1:200
    ΔA = eps(Float32) * normalize(randn(n, n) + 1im * randn(n, n))
    λ̃ = eigvals(A + ΔA)
    scatter!(real(λ̃), imag(λ̃), m=1, color=:black)
end
plt

The plot shows that some eigenvalues are much more affected than others. This situation is not unusual, but it is not explained by the Bauer–Fike theorem.

Example 7.2.3

We first define a hermitian matrix. Note that the ' operation is the adjoint and includes complex conjugation.

n = 7;
A = randn(n, n) + 1i * randn(n, n);
A = (A + A') / 2;

We confirm that the matrix $\mathbf{A}$ is normal by checking that $\kappa(\mathbf{V}) = 1$ (to within roundoff).

[V, D] = eig(A);
lambda = diag(D);
cond(V)

Now we perturb $\mathbf{A}$ and measure the effect on the eigenvalues. The Bauer–Fike theorem uses absolute differences, not relative ones. Note: since the ordering of eigenvalues can change, we look at all pairwise differences and take the minima.

E = randn(n, n) + 1i * randn(n, n);
E = 1e-8 * E / norm(E);
dd = eig(A + E);
dist = [];
for j = 1:n
    dist = [dist; min(abs(dd - lambda(j)))];
end
dist

As promised, the perturbations in the eigenvalues do not exceed the normwise perturbation to the original matrix.

Now we see what happens for a triangular matrix.

n = 20;
x = (1:n)';
A = triu(x * ones(1, n));
A(1:5, 1:5)

This matrix is not at all close to normal.

[V, D] = eig(A);
lambda = diag(D);
cond(V)

As a result, the eigenvalues can change by a good deal more.

E = randn(n, n) + 1i * randn(n, n);
E = 1e-8 * E / norm(E);
dd = eig(A + E);
dist = -Inf;
for j = 1:n
    dist = max(dist, min(abs(dd - lambda(j))));
end
fprintf("max change in eigenvalues: %.2e", dist)
fprintf("Bauer-Fike upper bound: %.2e", cond(V) * norm(E))

max change in eigenvalues:

6.84e-01

Bauer-Fike upper bound:

6.15e+01

If we plot the eigenvalues of many perturbations, we get a cloud of points that roughly represents all the possible eigenvalues when representing this matrix with single-precision accuracy.

clf
plot(lambda, 0*lambda, 'o')
axis equal; hold on
for k = 1:60
    E = randn(n, n) + 1i * randn(n, n);
    E = eps(single(1)) * E / norm(E);
    dd = eig(A + E);
    plot(real(dd), imag(dd), 'k.', markersize=2)
end

The plot shows that some eigenvalues are much more affected than others. This situation is not unusual, but it is not explained by the Bauer–Fike theorem.

Example 7.2.3

We first define a hermitian matrix. Note that we add the conjugate transpose of a matrix to itself.

n = 7
A = random.randn(n, n) + 1j * random.randn(n, n)
A = (A + conj(A.T)) / 2

We confirm that the matrix $\mathbf{A}$ is normal by checking that $\kappa(\mathbf{V}) = 1$ (to within roundoff).

from numpy.linalg import eig
d, V = eig(A)
print(f"eigenvector matrix has condition number {cond(V):.5f}")

eigenvector matrix has condition number 1.00000

Now we perturb $\mathbf{A}$ and measure the effect on the eigenvalues. Note that the Bauer–Fike theorem uses absolute differences, not relative ones. Since the ordering of eigenvalues can change, we look at all pairwise differences and take the minima.

E = random.randn(n, n) + 1j * random.randn(n, n)
E = 1e-8 * E / norm(E, 2)
dd, _ = eig(A + E)
dist = array([min([abs(x - y) for x in dd]) for y in d])
print(dist)

[9.13062970e-10 3.34658175e-09 7.06122371e-10 4.31937276e-10
 1.05432865e-09 3.97864390e-09 2.06839348e-09]

As promised, the perturbations in the eigenvalues do not exceed the normwise perturbation to the original matrix.

Now we see what happens for a triangular matrix.

n = 20
x = arange(n) + 1
A = triu(outer(x, ones(n)))
print(A[:5, :5])

[[1. 1. 1. 1. 1.]
 [0. 2. 2. 2. 2.]
 [0. 0. 3. 3. 3.]
 [0. 0. 0. 4. 4.]
 [0. 0. 0. 0. 5.]]

This matrix is not at all close to normal.

d, V = eig(A)
print(f"eigenvector matrix has condition number {cond(V):.2e}")

eigenvector matrix has condition number 6.15e+09

As a result, the eigenvalues can change by a good deal more.

E = random.randn(n, n) + 1j * random.randn(n, n)
E = 1e-8 * E / norm(E, 2)
dd, _ = eig(A + E)
dist = array([min([abs(x - y) for x in dd]) for y in d])
print(f"Maximum eigenvalue change is {max(dist):.2e}")
print(f"The Bauer-Fike upper bound is {cond(V) * norm(E, 2):.2e}")

Maximum eigenvalue change is 8.18e-01
The Bauer-Fike upper bound is 6.15e+01

If we plot the eigenvalues of many perturbations, we get a cloud of points that roughly represents all the possible eigenvalues when representing this matrix with single-precision accuracy.

clf
scatter(d, zeros(n), 18)
axis("equal") 
for _ in range(100):
    E = random.randn(n, n) + 1j * random.randn(n, n)
    E = finfo(np.float32).eps * E / norm(E, 2)
    dd, _ = eig(A + E)
    scatter(real(dd), imag(dd), 2, 'k')

The plot shows that some eigenvalues are much more affected than others. This situation is not unusual, but it is not explained by the Bauer–Fike theorem.

7.2.5Computing the EVD¶

Roots of the characteristic polynomial are not used in numerical methods for finding eigenvalues.^[2] Practical algorithms for computing the EVD go beyond the scope of this book. The essence of the matter is the connection to matrix powers indicated in (7.2.10). (We will see much more about the importance of matrix powers in Chapter 8.)

If the eigenvalues have different complex magnitudes, then as $k\to\infty$ the entries on the diagonal of $\mathbf{D}^k$ become increasingly well separated and easy to pick out. It turns out that there is an astonishingly easy and elegant way to accomplish this separation without explicitly computing the matrix powers.

Example 7.2.4 (Francis QR iteration)

Julia

MATLAB

Python

Example 7.2.4

Let’s start with a known set of eigenvalues and an orthogonal eigenvector basis.

D = diagm([-6, -1, 2, 4, 5])
V, R = qr(randn(5, 5))    # V is unitary
A = V * D * V'

5×5 Matrix{Float64}:
  3.23016    0.376963  -1.19976   -0.659159   0.169185
  0.376963   0.688255  -0.843471  -2.79237   -3.01583
 -1.19976   -0.843471  -0.512361   0.120006  -1.97367
 -0.659159  -2.79237    0.120006  -1.38242   -2.99132
  0.169185  -3.01583   -1.97367   -2.99132    1.97636

eigvals(A)

5-element Vector{Float64}:
 -6.000000000000002
 -0.9999999999999978
  1.9999999999999993
  3.9999999999999982
  4.999999999999995

Now we will take the QR factorization and just reverse the factors.

Q, R = qr(A)
A = R * Q;

It turns out that this is a similarity transformation, so the eigenvalues are unchanged.

eigvals(A)

5-element Vector{Float64}:
 -5.999999999999998
 -1.0000000000000002
  2.0000000000000004
  3.999999999999998
  5.000000000000002

What’s remarkable, and not elementary, is that if we repeat this transformation many times, the resulting matrix converges to $\mathbf{D}$ .

for k in 1:40
    Q, R = qr(A)
    A = R * Q
end
A

5×5 Matrix{Float64}:
 -5.99982      -0.0445231     4.5239e-7    2.09391e-15   1.01254e-15
 -0.0445231     4.99982       0.000186824  3.73623e-15   3.97368e-16
  4.5239e-7     0.000186824   4.0          4.98983e-12  -2.41858e-16
  7.69591e-20   7.14865e-16   4.98951e-12  2.0           2.6241e-12
 -4.03397e-31   2.47249e-28  -3.87812e-24  2.62456e-12  -1.0

Example 7.2.4

Let’s start with a known set of eigenvalues and an orthogonal eigenvector basis.

D = diag([-6, -1, 2, 4, 5]);
[V, R]= qr(randn(5, 5));    % V is unitary
A = V * D * V';

sort(eig(A))

Now we will take the QR factorization and just reverse the factors.

[Q, R] = qr(A);
A = R * Q;

It turns out that this is a similarity transformation, so the eigenvalues are unchanged.

sort(eig(A))

What’s remarkable, and not elementary, is that if we repeat this transformation many times, the resulting matrix converges to $\mathbf{D}$ .

for k = 1:40
    [Q, R] = qr(A);
    A = R * Q;
end
format short e
A

Example 7.2.4

Let’s start with a known set of eigenvalues and an orthogonal eigenvector basis.

from numpy.linalg import qr
D = diag([-6, -1, 2, 4, 5])
V, R = qr(random.randn(5, 5))
A = V @ D @ V.T    # note that V.T = inv(V) here

print(sort(eig(A)[0]))

[-6. -1.  2.  4.  5.]

Now we will take the QR factorization and just reverse the factors.

Q, R = qr(A)
A = R @ Q;

It turns out that this is a similarity transformation, so the eigenvalues are unchanged.

print(sort(eig(A)[0]))

[-6. -1.  2.  4.  5.]

What’s remarkable, and not elementary, is that if we repeat this transformation many times, the resulting matrix converges to $\mathbf{D}$ .

for k in range(40):
    Q, R = qr(A)
    A = R @ Q
set_printoptions(precision=4)
print(A)

[[-6.0000e+00 -6.3163e-05 -7.7494e-07  8.0433e-16 -6.6814e-16]
 [-6.3163e-05  5.0000e+00  3.4480e-03  1.3223e-15 -1.2786e-16]
 [-7.7494e-07  3.4480e-03  4.0000e+00 -2.8128e-13  2.4507e-16]
 [ 7.1896e-20  6.4328e-16 -2.8163e-13  2.0000e+00  3.8647e-13]
 [-3.5579e-32 -9.6403e-29  1.2418e-26  3.8653e-13 -1.0000e+00]]

The process demonstrated in Demo 7.2.4 is known as the Francis QR iteration, and it can be formulated as an $O(n^3)$ algorithm for finding the EVD. It forms the basis of most practical eigenvalue computations, at least until the matrix size approaches 10⁴ or so.

7.2.6Exercises¶

(a) ✍ Suppose that matrix $\mathbf{A}$ has an eigenvalue λ. Show that for any induced matrix norm, $\| \mathbf{A} \|\ge |\lambda|$ .
(b) ✍ Find a matrix $\mathbf{A}$ such that $\| \mathbf{A} \|_2$ is strictly larger than $|\lambda|$ for all eigenvalues λ. (Proof-by-computer isn’t allowed here. You don’t need to compute $\| \mathbf{A} \|_2$ exactly, just a lower bound for it.)

✍ Prove that the matrix $\mathbf{B}$ in (7.2.8) does not have two independent eigenvectors.
⌨ Find the eigenvalues of each matrix. Then, for each eigenvalue λ, use rank to verify that $\lambda\mathbf{I}$ minus the given matrix is singular.
$\mathbf{A} = \begin{bmatrix} 2 & -1 & 0 \\ -1 & 2 & -1 \\ 0 & -1 & 2 \end{bmatrix}\qquad$ $\mathbf{B} = \begin{bmatrix} 2 & -1 & -1 \\ -2 & 2 & -1 \\ -1 & -2 & 2 \end{bmatrix} \qquad$ $\mathbf{C} = \begin{bmatrix} 2 & -1 & -1 \\ -1 & 2 & -1 \\ -1 & -1 & 2 \end{bmatrix}$

$\mathbf{D} = \begin{bmatrix} 3 & 1 & 0 & 0 \\ 1 & 3 & 1 & 0 \\ 0 & 1 & 3 & 1 \\ 0 & 0 & 1 & 3 \end{bmatrix}\qquad$ $\mathbf{E} = \begin{bmatrix} 4 & -3 & -2 & -1\\ -2 & 4 & -2 & -1 \\ -1 & -2 & 4 & -1 \\ -1 & -2 & -1 & 4 \\ \end{bmatrix}$

(a) ✍ Show that the eigenvalues of a diagonal $n\times n$ matrix $\mathbf{D}$ are the diagonal entries of $\mathbf{D}$ . (That is, produce the associated eigenvectors.)
(b) ✍ The eigenvalues of a triangular matrix are its diagonal entries. Prove this in the $3\times 3$ case,
$\mathbf{T} = \begin{bmatrix} t_{11} & t_{12}& t_{13}\\ 0 & t_{22} & t_{23} \\ 0 & 0 & t_{33} \end{bmatrix},$
(7.2.14)
by finding the eigenvectors. (Start by showing that $[1,0,0]^T$ is an eigenvector. Then show how to make $[a,1,0]^T$ an eigenvector, except for one case that does not change the outcome. Continue the same logic for $[a,b,1]^T$ .)
✍ Let $\mathbf{A}=\displaystyle\frac{\pi}{6}\begin{bmatrix} 4 & 1 \\ 4 & 4 \end{bmatrix}$ .
(a) Show that
$\lambda_1=\pi,\, \mathbf{v}_1=\begin{bmatrix}1 \\ 2 \end{bmatrix}, \quad \lambda_2=\frac{\pi}{3},\, \mathbf{v}_2=\begin{bmatrix}1 \\ -2 \end{bmatrix}$
(7.2.15)
yield an EVD of $\mathbf{A}$ .
(b) Use (7.2.12) to evaluate $p(\mathbf{A})$ , where $p(x) = (x-\pi)^4$ .
(c) Use the function analog of (7.2.12) to evaluate $\cos(\mathbf{A})$ .

⌨ In Exercise 2.3.5, you showed that the displacements of point masses placed along a string satisfy a linear system $\mathbf{A}\mathbf{q}=\mathbf{f}$ for an $(n-1)\times(n-1)$ matrix $\mathbf{A}$ . The eigenvalues and eigenvectors of $\mathbf{A}$ correspond to resonant frequencies and modes of vibration of the string. For $n=40$ and the physical parameters given in part (b) of that exercise, find the eigenvalue decomposition of $\mathbf{A}$ . Report the three eigenvalues with smallest absolute value, and plot all three associated eigenvectors on a single graph (as functions of the vector row index).
⌨ Demo 7.2.4 suggests that the result of the Francis QR iteration as $k\to\infty$ sorts the eigenvalues on the diagonal according to a particular ordering. Following the code there as a model, create a random matrix with eigenvalues equal to $-9.6,-8.6,\ldots,10.4$ , perform the iteration 200 times, and check whether the sorting criterion holds in your experiment as well.
⌨ Eigenvalues of random matrices and their perturbations can be very interesting.
(a) Let A=randn(60,60).^[3] Scatter plot its eigenvalues in the complex plane, using a plot aspect ratio of 1 and red diamonds as markers.
(b) Let $\mathbf{E}$ be another random $60\times 60$ matrix, and on top of the previous graph, plot the eigenvalues of $\mathbf{A}+0.05\mathbf{E}$ as blue dots. Repeat this for 100 different values of $\mathbf{E}$ .
(c) Let T=triu(A). On a new graph, scatter plot the eigenvalues of $\mathbf{T}$ in the complex plane. (They all lie on the real axis.)
(d) Repeat part (b) with $\mathbf{T}$ in place of $\mathbf{A}$ .
(e) Compute some condition numbers and apply Theorem 7.2.3 to explain the dramatic difference between your plots with respect to the dot distributions.

Footnotes¶

The terms “factorization” and “decomposition” are equivalent; they coexist mainly for historical reasons.
↩
In fact, the situation is reversed: eigenvalue methods are among the best ways to compute the roots of a given polynomial.
↩
The randn function generates random numbers from a standard normal distribution. In Python, it is found in the numpy.random module.
↩

Preface

From matrix to insight

Preface

Singular value decomposition