Class 27: More Efficient Sorting Algorithms

Held Wednesday, March 8, 2000

Overview

Today we consider three more-efficient sorting algorithms.

Notes

• Are there questions on Assignment 3?
  • Some of you asked about the questions from chapter 5. If you see questions from chapter 5, you're using last semester's version of the assignment.
• Are there questions on Project, Phase 3?
• Reminder: I'll be gone from noon Wednesday until Sunday. I'll be at the SIGCSE Symposium on Computer Science Education.
  • Friday's class is optional. Vivek and Rachel will go over Assignment 3
• Assignments:
  • Exam 2 (due Friday, March 17, 2000)

Contents

• Sorting with Divide and Conquer
  • Merge Sort
    • Running Time
    • Running Time, Revisited
    • A Problem
    • Other Versions
  • Quicksort
    • Variations
• Better Sorting Techniques

Summary

• Divide and conquer sorts
  • Merge sort
  • Quick sort
• Other sorting techniques
• Sorting, concluded
• Handouts:
  • Exam 2 (due Friday, March 17, 2000)

Sorting with Divide and Conquer

• In the previous class, we identified a number of interesting sorting algorithms which took O(n²) time.
• Can we do better? Well, sometimes using divide-and-conquer helps speed up algorithms (in our experience from O(n) to O(log₂n)).
• We'll look at two different ways of ``splitting'' the array.

Merge Sort

• We can develop a number of sorting techniques based on the divide and conquer technique. One of the most straightforward is merge sort.
• In merge sort, you split the list, array, collection or ... into two parts, sort each part, and then merge them together.
• Unlike the previous algorithms, merge sort requires extra space for the sorted arrays or subarrays.
• We'll write this as a non-inplace routine which returns a new, sorted array, rather than sorting the existing array.
• In approximate Java code,

  
  import SimpleOutput;
  
  /**
   * A collection of techniques for sorting an input array.
   *
   * @author Samuel A. Rebelsky
   * @version 1.1 of March 2000
   */
  public class MergeSorter
  {
    // +--------+--------------------------------------------------
    // | Fields |
    // +--------+
  
    /** The current indent level.  Used when logging steps. */
    String indent = "";
  
    // +----------------+------------------------------------------
    // | Public Methods |
    // +----------------+
  
    /**
     * Sort an array, creating a new sorted version of the array.
     * If the SimpleOutput object is non-null, prints a simple log
     * of what's happening.
     * Pre: The elements in the array can be compared to each other.
     * Pre: There is sufficient memory to complete the creation of the
     *   new array (and the other steps of the algorithm).
     * Post: Returns a sorted version of the array (where sorted is
     *   defined carefully elsewhere).
     * Post: Does not affect the original array.
     */
    public Object[] sort(Object[] stuff, 
                         Comparator compare, 
                         SimpleOutput observer) 
      throws IncomparableException
    {
      return mergeSort(stuff, 0, stuff.length-1, compare, observer);
    } // sort(Object[])
    
    // +----------------+------------------------------------------
    // | Helper Methods |
    // +----------------+
  
    /**
     * Sort part of an array, creating a new sorted version of the
     * part of the array.
     * Pre: The elements in the array can be compared to each other.
     * Pre: There is sufficient memory to complete the creation of the
     *   new array (and the other steps of the algorithm).
     * Post: Returns a sorted version of the array (where sorted is
     *   defined carefully elsewhere).
     * Post: Does not affect the original array.
     */
    protected Object[] mergeSort(Object[] stuff, 
                                 int lb, int ub, 
                                 Comparator compare,
                                 SimpleOutput observer) 
      throws IncomparableException
    {
      Object[] sorted;		// The sorted version
      int middle;			// Index of middle element
      // Print some basic information.
      if (observer != null) {
        observer.print(indent + "Sorting: ");
        printSubArray(stuff, lb, ub, observer);
        indent = indent + "  ";
      }
      // Base case: vector of size 0 or 1.  Make a fresh copy so that
      // it's safe to modify (and is the appropriate size.
      if (ub <= lb) {
        sorted = copySubArray(stuff, lb, ub);
      } // base case
      // Recursive case: split and merge
      else {
        // Find the middle of the subarray.
        middle = (lb + ub) / 2;
        // Sort the two halves.
        Object[] left = mergeSort(stuff, lb, middle, compare, observer);
        Object[] right = mergeSort(stuff, middle+1, ub, compare, observer);
        sorted = merge(left, right, compare);
      } // recursive case
      // Print information, if appropriate
      if (observer != null) {
        indent = indent.substring(2);
        observer.print(indent + "Sorted: ");
        printSubArray(sorted, 0, sorted.length-1, observer);
      } 
      // That's it.
      return sorted;
    } // mergeSort(Object[], int, int, Comparator)
    
    /**
     * Merge two sorted arrays into a new single sorted array.
     * Pre: Both vectors are sorted.
     * Pre: Elements in both vectors may be compared to each other.
     * Pre: There is sufficient memory to allocate the new array.
     * Post: The returned array is sorted, and contains all the
     *   elements of the two arrays (no more, no less).
     * Post: The two arguments are not changed
     */
    public Object[] merge(Object[] left, Object[] right, Comparator compare) 
      throws IncomparableException
    {
      // Create a new array of the appropriate size.
      Object[] result = new Object[left.length + right.length];
      // Create indices into the three arrays.
      int leftIndex=0;	// Index into left array.
      int rightIndex=0;	// Index into right array.
      int index=0;		// Index into result array.
      // As long both vectors have elements, copy the smaller one.
      while ((leftIndex < left.length) && (rightIndex < right.length)) {
        if(compare.precedes(left[leftIndex],right[rightIndex])) {
          result[index++] = left[leftIndex++];
        } // first element in left subvector is smaller
        else {
          result[index++] = right[rightIndex++];
        } // first element in right subvector is smaller
      } // while both vectors have elements
      // Copy any remaining parts of each vector.  
      while(leftIndex < left.length) {
        result[index++] = left[leftIndex++];
      } // while the left vector has elements
      while(rightIndex < right.length) {
        result[index++] = right[rightIndex++];
      } // while the right vector has elements
      // That's it
      return result;
    } // merge
  
    /**
     * Copy a subarray (so that we can return it without affecting it).
     * Pre: 0 <= lb <= ub < stuff.length
     * Post: Does not affect stuff.
     * Post: Returns a new array containing only stuff[lb] .. stuff[ub].
     */
    protected Object[] copySubArray(Object[] stuff, int lb, int ub) {
      // Create the new array.  
      Object[] result = new Object[ub-lb+1];
      for (int i = lb; i <= ub; i++) {
        result[i-lb] = stuff[i];
      }
      return result;
    } // copySubArray
  
    /**
     * Print a subarray.
     * Pre: 0 <= lb <= ub < stuff.length
     * Post: Does not affect stuff.
     */
    protected void printSubArray(Object[] stuff, 
                                 int lb, int ub,
                                 SimpleOutput out) { 
      // Print all but the last element followed by a comma
      for (int i = lb; i < ub; ++i) {
        out.print(stuff[i].toString() + ",");
      }
      // Print the last element
      out.println(stuff[ub]);
    } // printSubArray
  } // MergeSorter
  
  
  

• Here's the corresponding test class.

  
  import MergeSorter;
  import SimpleOutput;
  import StringComparator;
  
  /**
   * A simple test of selection sort.
   *
   * @author Samuel A. Rebelsky
   * @version 1.0 of September 1999
   */
  public class TestMergeSorter {
    public static void main(String[] args) 
      throws Exception
    {
      SimpleOutput out = new SimpleOutput();
      MergeSorter sorter = new MergeSorter();
      Object[] sorted = sorter.sort(args, new StringComparator(), out);
      for (int i = 0; i < sorted.length; ++i) {
        out.println(i + ": " + sorted[i]);
      } // for
    } // main(String[])
  } // class TestMergeSorter
  
  
  

Running Time

• What is the running time?
• We can use recurrence relations:
  • Let f(n) be the running time of merge sort on input of size n.
  • f(1) = 1
  • f(n) = n + 2*f(n/2)
• Let's run this for a few steps
  • f(n)
  • = n + 2*f(n/2)
  • = n + 2*f(n/2)
  • = n + 2*(n/2 + 2*f(n/2/2))
  • = n + n + 4*f(n/4)
  • = n + n + 4*(n/4 + 2*f(n/4/2))
  • = n + n + n + 8*f(n/8)
• Generalizing
  • f(n) = kn + 2^k*f(n/2^k)
• When k = log₂n
  • f(n) = nlog₂n + n*1
• Therefore, f(n) is in O(nlog₂n).

Running Time, Revisited

• We can also use a somewhat nontraditional analysis technique.
• Assume that we're dealing with n = 2^x for some x.
• Consider all the sorts of Arrays of size k as the same "level".
• There are n/k Arrays of size k.
• Because we divide in half each time, there are log_2(n) levels.
• Going from level k to level k+1, we do O(n) work to merge.
• So, the running time is O(n*log_2(n)).

A Problem

• Unfortunately, merge sort requires significantly more memory than do the other sorting routines (you can spend some time trying to come up with an ``in place'' merge sort, but you are quite likely to fail).

Other Versions

• We've written merge sort so that it does not affect the original array.
• How might we write it so that it creates a sorted array?
  • Will that save space?
• How might we write it like the previous sorting methods (which were for subclasses of our Array class)?

Quicksort

• Is it possible to write an O(n*log₂n) sorting algorithm that is based on comparing and swapping, but doesn't require significantly extra space?
• Yes, if you're willing to rely on probabilities.
• In the Quicksort algorithm, you split (partition) the array to be sorted into two pieces, those smaller than or equal to the pivot and those greater than the pivot. You don't include the pivot in either piece (so that the recursive case is ``smaller'').
  • You can also split into three parts: those smaller than some middle element, those equal to some middle element, and those larger than some middle element.
• With a little work, you can do this partitioning in place, so that there is no overhead (and so that ``glueing'' is basically a free operation).
• (It is not necesarrily okay to partition the array into two parts: those less than or equal to the element, and those greater than or equal to the element.)

  /**
   * Sort an array using Quicksort.
   * Pre: All elements in the array can be compared to each other.
   * Post: The vector is sorted (using the standard meaning).
   */
  public void quickSort(Object[] stuff) {
    quickSort(stuff, 0, stuff.length-1);
  } // quickSort(Object[])
  
  /**
   * Sort part of an array using Quicksort.
   * Pre: All elements in the subarray can be compared to each other.
   * Pre: 0 <= lb <= ub < stuff.length
   * Post: The vector is sorted (using the standard meaning).
   */
  public void quickSort(Object[] stuff, int lb, int ub, Comparator compare) {
    // Variables
    Object pivot;	// The pivot used to split the vector
    int mid;		// The position of the pivot
    // Base case: size one arrays are sorted.
    if (lb == ub) return;
    // Pick a pivot and put it at the front of the array.  
    putPivotAtFront(stuff,lb,ub);
    // Determine the position of the pivot, while rearranging the array.
    mid = partition(stuff, lb, ub);
    // Recurse on nonempty subarrays.
    if (lb<=mid-1) quickSort(stuff, lb,mid-1);
    if (mid+1<=ub) quickSort(stuff, mid+1,ub);
  } // quickSrt
  
  /**
   * Split the array given by [lb .. ub] into ``smaller'' and
   * ``larger'' elements, where smaller and larger are defined by
   * their relationship to a pivot.  Return the index of the pivot 
   * between those elements.  Uses the first element of the array 
   * as the pivot.
   */
  public int partition(Object[] stuff, int lb, int ub) {
    // STUB.  Can you figure it out?
    return 0;
  } // partition(Object[])
  

• What is the running time of Quicksort? It depends on how well we partition. If we partition into two equal halves, then we can say
  • Partitioning a vector of length n takes O(n) steps.
  • The time to partition those two partitions into four parts is also O(n).
  • If each partition is perfect (splits it exactly in half), we can stop the process after O(log_2(n)) levels.
  • This gives a running time of O(n*log_2(n)).
• On average, we don't quite do half, but it's close enough that it doesn't make a significant difference.
• However, bad choice of pivots can give significantly worse running time. If we always chose the largest element as the pivot, this algorithm would be equivalent to selection sort, and would take time O(n*n).
• How do we partition? Typically, using something like the following strategy:

  Set pivot to the first element of the subarray
  Set left to the start of the subarray
  Set right to the end of the subarray
  Move left and right toward each other, 
    Swap their contents when you observe that one side is
      "wrong" (something on the left is larger than the pivot,
      something on the right is larger than the pivot)
  

Variations

• How might you change Mergesort so that the pivot need not be an element of the array?
• How might you rewrite Mergesort iteratively?

Better Sorting Techniques

• Are there better sorting techniques (ones that take less than O(nlog₂n))? Not if you limit yourself to basic operations of comparing and swapping.
• But if you're willing to use extra space and know something about the original data, then you can do better.
• In bucket sort (the one Rob suggested), you create separate ``buckets'' for kinds of elements, put each element into the appropriate bucket, sort each bucket, and then take them out again.
  • If you can arrange things so that each bucket contains only a few elements (say no more than four), then the main cost is putting in to buckets and taking out of buckets.
  • Usually we choose simple criteria for the buckets, such as the first (or nth) letter in a string.
  • Running time can be a constant (as long as you can guarantee the number of items in any bucket)!
• In radix sort, we sort using a binary representation of the things we're sorting. We'll do an example later.