Class 34: Algorithm Analysis (2)

Held: Monday, 31 October 2005

Summary: Today we formalize techniques for comparatively evaluating the running time of algorithms.

Related Pages:

• EBoard.
• Examples.

Assignments

• Do the lab on algorithm analysis in The TAO of Java.

Notes:

• Happy halloween!
• Grading not completed. Sorry. Probably not until Friday.

Overview:

• Formalizing notions.
• Eliminating constants.
• Using the notation: Relating functions.
• Analyzing recursive functions.

Asymptotic Analysis, Formalized

• Noting problems in providing a precise analysis of the running time of programs, computer scientists developed a technique which is often called asymptotic analysis. In asymptotic analysis of algorithms, one describes the general behavior of algorithms in terms of the size of input, but without delving into precise details.
  • The analysis is asymptotic in that we look at the behavior as the input gets arbitrary large
• There are many issues to consider in analyzing the asymptotic behavior of a program. One particularly useful metric is an upper bound on the running time of an algorithm. We call this the Big-O of an algorithm.
• Informally, Big-O gives the general shape of the curve of the graph of running time vs. input size.
  • That is, is it linear, quadratic, logarithmic, ...
• Big-O is defined somewhat mathematically, as a relationship between functions.
• f(n) is in O(g(n)) iff
  • there exists a number n₀
  • there exists a number d > 0
  • for all but finitely many n > n₀, |f(n)| <= |d*g(n)|
• What does this say? It says that after a certain value, n₀, f(n) is bounded above by a constant (that is, d) times g(n).
  • The constant, d, helps accommodate the variation in the algorithm.
  • We don't usually identify the d precisely.
  • The n₀ says for big enough n.
• We can apply big-O to algorithms.
  • n is the size of the input (e.g., the number of items in a list or vector to be manipulated).
  • f(n) is the running time of the algorithm.
• Some common Big-O bounds
  • An algorithm that is in O(1) takes constant time. That is, the running time is independent of the input. Getting the size of a vector should be an O(1) algorithm.
  • An algorithm that is in O(n) takes time linear in the size of the input. That is, we basically do constant work for each element of the input. Finding the smallest element in a list is often an O(n) algorithm.
  • An algorithm that is in O(log₂(n)) takes logarithmic time. While the running time is dependent on the size of the input, it is clear that not every element of the input is processed. Many such algorithms involve the strategy of divide and conquer.
• We will typically use Big O notation after informal analysis.

Eliminating Constants

• One of the nice things about asymptotic analysis is that it makes constants unimportant because they can be hidden in the d.
• If f(n) is 100n seconds and g(n) is 0.5n seconds, then
  • f(n) is in O(g(n)) [let d be 200]
  • g(n) is in f(n) [let d be 1]
• If f(n) is 100n seconds and g(n) is n² seconds, then f(n) is in O(g(n)) [let n₀ be 100 and d be 1]
• However, g(n) is not in O(f(n)). Why not?
  • Suppose there were an n₀ and a d.
  • Consider what happens for n = 101d.
  • d*f(n) = d*100*101*d = d*d*100*101.
  • However, g(n) = d*d*101*101, which is even larger.
  • If n₀ is greater than 101d, we'll still have this problem [proof left to reader].
• Since constants can be eliminated, we normally don't write them.
  • That is, we say that the running time of an algorithm is O(n) or O(n²) or ....

Relating Functions

• Note that some of the counting can be hard, so we may also need to estimate throughout the process.
• We can often express the running time of an algorithm as a composition of other functions.
  • For example, we might have a primary control structure which repeats its contents O(h(n)) times and within that control structure, we have a subalgorithm that takes O(g(n)) time.
  • Similarly, we might break our algorithm up into two independent algorithms, one of which takes O(h(n)) time and one of which takes O(g(n)) time.
  • We might also look at other relationships.
• Here are some questions we might ask ourselves about composition of functions.
  • If f(n) is in O(h(n)) and g(n) is O(k(n)), then what can we say about f(n) + g(n)?
  • If f(n) is in O(h(n)) and g(n) is O(k(n)), then what can we say about f(n) * g(n)?
• We might also ask more general questions about Big-O notation.
  • If f(n) is in O(g(n)) and g(n) is in O(h(n)) then what can we say about f(n) and h(n)?
  • If f(n) is in O(kg(n)) and k is a constant, then what else can we say about f(n) and g(n)?
  • If f(n) is in O(h(n)) and g(n) is in O(h(n)), what can we say about f(n) + g(n)?
  • If f(n) is in O(g(n) + h(n)) and g(n) is in O(h(n)), then what simpler thing can we say about f(n)?

Analyzing Recursive Functions

• Recursive functions are more difficult to analyze.
• After you've taken combinatorics, you can use recurrence relations for recursive functions.
  • We may do some informal recurrence relations when we get to appropriate recursive functions
• We may also try to rephrase recursive methods iteratively.
• We can also use some informal analysis of what happens in the procedure.

An Example, Recursive Sum

• Consider, for example, the recursive sum method from Scheme.

  (define sum
    (lambda (lst)
      (if (null? lst)
          0
          (+ (car lst) (sum (cdr lst))))))
  

• Informal: Since we do one addition for each element in the list (as well as one test), we can say that this is likely to be in O(n).
• Rephrase iteratively (in Java):

  public static double sum(SchemeListOfDoubles sl)
  {
    SchemeListOfDoubles tmp = sl;
    int sum = 0;
    while (!tmp.isEmpty()) {
      sum = sum + tmp.car();
      tmp = tmp.cdr();
    } // while
  } // sum(SchemeListOfDoubles)
  

• Recurrence relations: Let's define a function, t(n), that gives the running time of sum for a list of length n. We know that
  • t(0) = c, for some constant c, because we just have do do the test and return 0.
  • t(n) = d + t(n-1), because we need to do a test, an addition, a car, and a cdr plus a recursive call with a list of size n-1.
• How do we define t(n) non-recursively. For this class, we can repeatedly expand and look for a pattern.
  • t(n) = d + t(n-1)
  • t(n) = d + d + t(n-2) = 2d + t(n-2)
  • t(n) = 2d + d + t(n-3) = 3d + t(n-3)
  • t(n) = 3d + d + t(n-4) = 3d + t(n-4)
  • I see the pattern t(n) = kd + t(n-k)
  • Let k = n. t(n) = dn + t(0) = dn + c is in O(n)
• Some more examples
  • t(n) = d + n + t(n-1)
  • t(n) = d + t(n/2) // Binary search
  • t(n) = d + 2*t(n/2) // Generic divide-and-conquer
  • t(n) = 2*t(n/2)
  • t(n) = d + n + t(n/2) // ???
  • t(n) = d + 2*t(n-1) // Upper bound on Fibonacci
  • t(n) = d + 2*t(n-2) // Lower bound on Fibonacci