Efficiency of matrix computations

Predicting how long an algorithm will take to solve a particular problem, on a particular computer, as written in a particular way in a particular programming language, is an enormously difficult undertaking. It’s more practical to predict how the required time will scale as a function of the size of the problem. In the case of a linear system of equations, the problem size is $n$, the number of equations and variables in the system. Because expressions of computational time are necessarily approximate, it’s customary to suppress all but the term that is dominant as $n\to\infty$. We first need to build some terminology for these expressions.

2.5.1Asymptotic analysis¶

One immediate consequence is that $f\sim g$ implies $f=O(g)$.^[1]

It’s conventional to use asymptotic notation that is as specific as possible. For instance, while it is true that $n^2+n=O(n^{10})$, it’s more informative, and usually expected, to say $n^2+n=O(n^2)$. There are additional notations that enforce this requirement strictly, but we will just stick to the informal understanding.

There is a memorable way to use asymptotic notation to simplify sums:

These formulas greatly resemble the definite integral of $x^p$.

2.5.2Flop counting¶

Traditionally, in numerical linear algebra we count floating-point operations, or flops for short. In our interpretation each scalar addition, subtraction, multiplication, division, and square root counts as one flop. Given any algorithm, we simply add up the number of scalar flops and ignore everything else.

Suppose that the running time $t$ of an algorithm obeys a function that is $O(n^p)$. For sufficiently large $n$, $t\approx Cn^p$ for a constant $C$ should be a good approximation. Hence

So we expect that a graph of $\log t$ as a function of $\log n$ will be a straight line of slope $p$.

2.5.3Solution of linear systems¶

Recall the steps of Algorithm 2.4.2 for the system $\mathbf{A}\mathbf{x}=\mathbf{b}$:

1. Factor $\mathbf{L}\mathbf{U}=\mathbf{A}$ using Gaussian elimination.
2. Solve $\mathbf{L}\mathbf{z}=\mathbf{b}$ for $\mathbf{z}$ using forward substitution.
3. Solve $\mathbf{U}\mathbf{x}=\mathbf{z}$ for $\mathbf{x}$ using backward substitution.

The second and third steps are solved by Function 2.3.1 and Function 2.3.2. Only one line in each of these functions dominates the arithmetic. Take forwardsub, for instance. It has a single flop in line 11. Line 13 computes

sum( L[i,j]*x[j] for j in 1:i-1 )

This line requires $i-1$ multiplications and $(i-2)$ additions, for a total of $2i-3$ flops. Line 14 adds two more flops. These lines are performed within a loop as $i$ ranges from 1 to $n$, so the total count is

It is not hard to find an exact formula for the sum, but we use (2.5.5) to simplify it to $\sim n^2$. After all, since flop counting is only an approximation of true running time, why bother with the more complicated exact expression? An analysis of backward substitution yields the same result.

Before counting flops for the LU factorization, we have to admit that Function 2.4.1 is not written as economically as it could be. Recall from our motivating example in Demo 2.4.3 that we zero out the first row and column of $\mathbf{A}$ with the first outer product, the second row and column with the second outer product, and so on. There is no good reason to do multiplications and additions with values known to be zero, so we could replace lines 15–19 of lufact with

15
16
17
18
19
for k in 1:n-1
    U[k,k:n] = Aₖ[k,k:n]
    L[k:n,k] = Aₖ[k:n,k]/U[k,k]
    Aₖ[k:n,k:n] -= L[k:n,k]*U[k,k:n]'
end

We will use the following handy fact.

Line 17 above divides each element of the vector Aₖ[k:n,k] by a scalar. Hence the number of flops equals the length of the vector, which is $n-k+1$.

Line 18 has an outer product followed by a matrix subtraction. The definition (5) of the outer product makes it clear that that computation takes one flop (multiplication) per element of the result, which here results in $(n-k+1)^2$ flops. The number of subtractions is identical.

Altogether the factorization takes

There are different ways to simplify this expression. We will make a change of summation index using $j=n-k$. The endpoints of the sum are $j=n-1$ when $k=1$ and $j=1$ when $k=n-1$. Since the order of terms in a sum doesn’t matter, we get

We have proved the following.

In practice, flops are not the only aspect of an implementation that occupies significant time. Our position is that counting flops as a measure of performance is a useful oversimplification. We will assume that LU factorization (and as a result, the solution of a linear system of $n$ equations) requires a real-world time that is roughly $O(n^3)$. This growth rate is a great deal more tolerable than, say, $O(2^n)$, but it does mean that for (at this writing) $n$ greater than 10,000 or so, something other than general LU factorization will have to be used.

2.5.4Exercises¶

1. ✍ The following are asymptotic assertions about the limit $n\rightarrow\infty$. In each case, prove the statement true or false.

   (a) $n^2 = O(\log n),\quad$ (b) $n^{a} = O(n^b)$ if $a\le b,\quad$ (c) $e^n \sim e^{2n},\quad$ (d) $n+\sqrt{n}\sim n+2\sqrt{n}$.

2. ✍ The following are asymptotic assertions about the limit $h\to 0$. In each case, prove the statement true or false.

   (a) $h^2\log(h) = O(h^3),\quad$ (b) $h^{a} = O(h^b)$ if $a < b,\quad$ (c) $\sin(h) \sim h,\quad$ (d) $(e^{2h}-1)\sim h$.

3. ✍ Show that the inner product of two $n$-vectors takes exactly $2n-1$ flops.

4. ✍ Show that the multiplication of two $n\times n$ matrices takes $\sim 2n^3$ flops.

5. ✍ This problem is about evaluation of a polynomial $c_1 + c_2 x + \cdots + c_{n}x^{n-1}$.

   (a) Here is a little code to do the evaluation.

   y = c[1]
   xpow = 1
   for i in 2:n
       xpow *= x
       y += c[i]*xpow
   end

   Assuming that x is a scalar, how many flops does this function take, as a function of $n$?

   (b) Compare the count from (a) to the flop count for Horner’s algorithm, Function 1.3.1.

6. The exact sums for $p=1,2$ in (2.5.5) are as follows:

   $$\sum_{k=1}^{n} k = \frac{n(n+1)}{2}, \qquad \sum_{k=1}^{n} k^2 = \frac{n(n+1)(2n+1)}{6}.$$

   (2.5.11)

   (a) ✍ Use these to find the exact result for (2.5.9).

   (b) ⌨ Plot the ratio of your result from (a) and the asymptotic result $2n^3/3$ for all $n=10^{1+0.03i}$, $i=0,\dots,100$, using a log scale for $n$ and a linear scale for the ratio. (The curve should approach 1 asymptotically.)

7. ✍ Show that for any nonnegative constant integer $m$,

   $$\sum_{k=0}^{n-m} k^p \sim \frac{n^{p+1}}{p+1}.$$

   (2.5.12)

8. ⌨ The UpperTriangular and LowerTriangular matrix types cause specialized algorithms to be invoked by the backslash. Define

   A = rand(1000,1000)
   B = tril(A)
   C = LowerTriangular(B)
   b = rand(1000)

   Using @elapsed with the backslash solver, time how long it takes to solve the linear system $\mathbf{A}\mathbf{x}=\mathbf{b}$ 100 times; then, do the same for matrices $\mathbf{B}$ and $\mathbf{C}$. Is the timing for $\mathbf{B}$ closer to $\mathbf{A}$ or to $\mathbf{C}$? (Hint: Remember to make one timing run without recording results, so that compilation time is not counted.)