Topics covered: Divide-and-Conquer: Strassen, Fibonacci, Polynomial Multiplication
Instructors: Prof. Erik Demaine, Prof. Charles Leiserson
Lecture 3: Divide-and-Conqu...
Related Resources
Lecture Notes (PDF)
Assignments
Exams
Good. Log base 2 of 7 is going be a Good morning everyone. Today we are going to do some algorithms, back to algorithms, and we are going to use a lot of the, well, some of the simpler mathematics that we developed last class like the master theorem for solving recurrences. We are going to use this a lot. Because we are going to talk about recursive algorithms today. And so we will find their running time using the master theorem. This is just the same as it was last time, I hope, unless I made a mistake. A couple of reminders. You should all go to recitation on Friday. That is required. If you want to, you can go to homework lab on Sunday. That may be a good excuse for you to actually work on your problem set a few hours early. Well, actually, it's due on Wednesday so you have lots of time. And there is no class on Monday. It is the holiday known as Student Holiday, so don't come. Today we are going to cover something called Divide and Conquer. Also known as divide and rule or divide et impera for those of you who know Latin, which is a tried and tested way of conquering a land by dividing it into sections of some kind. It could be different political factions, different whatever. And then somehow making them no longer like each other. Like starting a family feud is always a good method. You should remember this on your quiz. I'm kidding. And if you can separate this big power structure into little power structures such that you dominate each little power structure then you can conquer all of them individually, as long as you make sure they don't get back together. That is divide and conquer as practiced, say, by the British. But today we are going to do divide and conquer as practiced in Cormen, Leiserson, Rivest and Stein or every other algorithm textbook. This is a very basic and very powerful algorithm design technique. So, this is our first real algorithm design experience. We are still sort of mostly in the analysis mode, but we are going to do some actual design. We're going to cover maybe only three or four major design techniques. This is one of them, so it is really important. And it will lead to all sorts of recurrences, so we will get to use everything from last class and see why it is useful. As you might expect, the first step in divide-and-conquer is divide and the second step is conquer. But you may not have guessed that there are three steps. And I am leaving some blank space here, so you should, too. Divide-and-conquer is an algorithmic technique. You are given some big problem you want to solve, you don't really know how to solve it in an efficient way, so you are going to split it up into subproblems. That is the divide. You are going to divide that problem, or more precisely the instance of that problem, the particular instance of that problem you have into subproblems. And those subproblems should be smaller in some sense. And smaller means normally that the value of N is smaller than it was in the original problem. So, you sort of made some progress. Now you have one, or more likely you have several subproblems you need to solve. Each of them is smaller. So, you recursively solve each subproblem. That is the conquer step. You conquer each subproblem recursively. And then somehow you combine those solutions into a solution for the whole problem. So, this is the general divide-and-conquer paradigm. And lots of algorithms fit it. You have already seen one algorithm that fits this paradigm, if you can remember. Merge sort, good. Wow, you are all awake. I'm impressed. So, we saw merge sort. And, if I am clever, I could fit it in this space. Sure. Let's be clever. A quick review on merge sort. Phrased in this 1, 2, 3 kind of method. The first step was to divide your array into two halves. This really doesn't mean anything because you just sort of think, oh, I will pretend my array is divided into two halves. There is no work here. This is zero time. You just look at your array. Here is your array. I guess maybe you compute n/2 and take the floor. That takes constant time. And you say OK, I am pretending my array is now divided into the left part and the right part. And then the interesting part is that you recursively solve each one. That's the conquer. We recursively sort each subarray. And then the third step is to combine those solutions. And so here we really see what this means. We now have a sorted version of this array by induction. We have a sorted version of this array by induction. We now want the sorted version of the whole array. And we saw that was the merge problem, merging two sorted arrays. And that we saw how to do in linear time, order n time. I am not going to repeat that, but the point is it falls into that framework. I want to write the running time and merge sort as a recurrence. You have already seen the recurrence, you have already been told the solution, but now we actually know how to solve it. And, furthermore, every algorithm that follows the divide-and-conquer paradigm will have a recurrence of pretty much the same form, very much like our good friend the master method. Let's do it for merge sort where we sort of already know the answer and get a bit of practice. This is the merge sort recurrence. You should know and love this recurrence because it comes up all over the place. It comes from this general approach by just seeing what are the sizes of the subproblems you are solving and how many there are and how much extra work you are doing. You have here the size of the subproblems. It happens here that both subproblems have the same size roughly. There is this sloppiness that we have, which really should be T of floor of n over 2 plus T of ceiling of n over 2. And when you go to recitation on Friday you will see why that is OK, the floors and ceilings don't matter. There is a theorem you can prove that that's happy. You can assume that N is a power of 2, but we are just going to assume that for now. We just have two problems with size n over 2. This 2 is the number of subproblems. And this order n is all the extra work we are doing. Now, what is the extra work potentially? Well, the conquering is always just recursion. There is sort of no work there except this lead part. The dividing in this case is trivial, but in general it might involve some work. And the combining here involves linear work. So, this is the divide-and-conquer running times. So, this is the nonrecursive work. And that is generally how you convert a divide-and-conquer algorithm into a recurrence. It's really easy and you usually get to apply the master method. Here we are in Case 2. Very good. This is Case 2. And k is zero here. And so in the recursion tree, all of the costs are roughly the same. They are all n to the log base b of a. Here n to the log base 2 of 2 is just n. So these are equal. We get an extra log factor because of the number of levels in the recursion tree. Remember the intuition behind the master method. This is n log n, and that is good. Merge sort is a fast sorting algorithm n log n. Insertion sort was n squared. In some sense, n log n is the best you can do. We will cover that in two lectures from now, but just foreshadowing. Today we are going to do different divide-and-conquer algorithms. Sorting is one problem. There are all sorts of problems we might want to solve. It turns out a lot of them you can apply divide-and-conquer to. Not every problem. Like how to wake up in the morning, it's not so easy to solve a divide-and-conquer, although maybe that's a good problem set problem. The next divide-and-conquer algorithm we are going to look at is even simpler than sorting, even simpler than merge sort, but it drives home the point of when you have only one subproblem. How many people have seen binary search before? Anyone hasn't? One, OK. I will go very quickly then. You have some element X. You want to find X in a sorted array. How many people had not seen it before they saw it in recitation? No one, OK. Good. You have seen it in another class, probably 6.001 or something. Very good. You took the prerequisites. OK. I just want to phrase it as a divide-and-conquer because you don't normally see it that way. The divide step is you compare X with the middle element in your array. Then the conquer step. Here is your array. Here is the middle element. You compare X with this thing if let's say X is smaller than the middle element in your array. You know that X is in the left half because it is sorted, a nice loop invariant there, whatever, but we are just going to think of that as recursively I am going to solve the problem of finding X in this subarray. We recurse in one subarray, unlike merge sort where we had two recursions. And then the combined step we don't do anything. I mean if you find X in here then you've found X in the whole array. There is nothing to bring it back up really. So, this is just phrasing binary search in the divide-and-conquer paradigm. It is kind of a trivial example, but there are lots of circumstances where you only need to recurse in one side. And it is important to see how much of a difference making one recursion versus making two recursions can be. This is the recurrence for binary search. We start with a problem size n. We reduce it to 1. There is an implicit 1 factor here. One subproblem of size n over 2 roughly. Again, floors and ceilings don't matter. Plus a constant which is to compare X with the middle element, so it is actually like one comparison. This has a solution, log n. And you all know the running time of binary search, but here it is at solving the recurrence. I mean, there are a couple of differences here. We don't have the additive order n term. If we did, it would be linear, the running the time. Still better than n log n. So, we are getting rid of the 2, bringing it down to a 1, taking the n and bringing it down to a 1. That is making the running time a lot faster, the whole factor of n faster. No big surprise there. Let's do some more interesting algorithms. The powering a number problem is I give you a number X. I give that as like a real number or floating point number or whatever. And I give you an integer n, at least zero. I want to compute X to the power n. So, it's a very simple problem. It is, in some sense, even easier than all of these. But here it is. And divide-and-conquer is sort of the right thing to do. So, the naÔve algorithm is very simple. How do you compute X to the power n? Well, the definition of X to the power n is I take X and I multiply by X n times. So, I take X times X times X times X where there are n copies of X totally. And that's X to the n. No big surprise. That is n multiplications, or n minus 1 multiplications, theta n time. But that's not the best you can do for this problem. Any suggestions on what we might do using divide-and-conquer? Has anyone seen this algorithm before? A few, OK. For the rest? Testing on the spot creativity, which is very difficult, but I always like a challenge. I mean random ideas. What could we possibly do to solve this problem in less than linear time? How is this sort of a divide-and-conquer problem? We have to inputs, X and n. Yeah? We could try to divide on X. It seems a bit hard. It is just some number. Or, we could try to divide on n. Any guesses? Look at X to the n over 2, very good. That is exactly the idea of the divide-and-conquer algorithm. We would like to look at X to the n over 2. This is going to be a little bit tricky. Now we are going to have to pay attention to floors and ceilings. What I would like to say is while X to the n is X to the n over 2 times X to the n over 2. And this is true if n is even. If it is odd then I need to be a little bit more careful. But let's just think about the intuition why this is a good divide-and-conquer algorithm. We have a problem of size n, let's say. We convert it into, it looks like two subproblems of size n over 2, but in fact they are the same subproblems. So, I only have to solve one of them. If I compute X to the n over 2. Yeah, I know X to the n over 2. So, there is one conversive call, problem of size n over 2, then I square that number. And that is one computation. So, exactly the same recurrence as binary search, log n time much better than n. Cool. I also have to solve the odd case. So, n is odd. I will look at n minus 1 over N minus 1 better be even. And then I am missing another factor of X. If n is odd, I am going to have to do one recursive call and two multiplications. The same recurrence. One recursive problem of size n over 2, plus constant time. The dividing work here is dividing by 2 and the combination work is doing one or possibly two multiplications. And this is lg n. And if all you are allowed to do is multiply numbers, lg n is the best you can do. Good. Simple but powerful. Whenever you want to compute a power of a number, now you know what to do. Does anyone not know the definition of Fibonacci numbers and is willing to admit it? OK, so this is a good old friend. I will write down the definition as just a reminder. And, in particular, the base cases. Fibonacci numbers, I will claim, are very important because it appears throughout nature. You look at certain fruits, you see the Fibonacci sequence. If you count the number of little bumps around each ring. If you look at the sand on the beach and how the waves hit it, it's the Fibonacci sequence I am told. If you look all over the place, Fibonacci sequence is there. How does nature compute the Fibonacci sequence? Well, that is a different class. But how are we going to compute the Fibonacci sequence as fast as possible? You have probably seen two algorithms. The most naÔve algorithm is the recursive algorithm. Where you say OK, f of n. I say well, if n is zero, return zero, if n is 1, return one. Otherwise, recursively compute f of n minus 1 and f of n minus 2, add them together. How much time does this algorithm take, for those who have seen it before? This is not obvious to guess. It doesn't have to be exact. OK. And how many people have seen this algorithm before and analyzed it? Half, OK. So what is the running time? Really, really long, very good. Anymore precise answers? What's that? Exponential, yes. That is also correct and more precise. I will be even more precise. Maybe you haven't seen this analysis before. It's phi to the n where phi is the Golden Ratio. Again, Golden Ratio appears throughout the world in mathematics. This is probably the only time in this class, I'm afraid, but there we go. It made its cameo so we are happy. This is called exponential time. This is bigger than one, that's all you need to know. This is exponential time. Exponential time means basically some constant to the power n. Exponential time is a very long time. It's bad. Polynomial time is good. [LAUGHTER] This is what we want, polynomial time algorithms. This class is basically entirely about polynomial time algorithms. Question? Oh, say what the algorithm does again. Define function Fibonacci of n? I check for the base cases. Otherwise, I recursively call Fibonacci of n minus 1. I recursively call Fibonacci of n minus 2 and add those two numbers together. So, you get this branching tree. You are solving two subproblems of almost the same size, just additively smaller by one or two. I mean you are almost not reducing the problem size at all, so that's intuitively why it is exponential. You can draw a recursion tree and you will see how big it gets and how quickly. I mean by n over two levels, you've only reduced on one branch the problem from n to n over 2. The other one, maybe you've gotten from n down to one, but none of the branches have stopped after n of two levels. You have at least 2 to the power n over 2 which is like square root of 2 to the power n, which is getting close to phi to the n. So, this is definitely exponential. And exponential is bad. We want polynomial. N squared, n cubed, log n would be nice. Anything that is bounded above by a polynomial is good. This is an old idea. It goes back to one of the main people who said polynomial is good, Jack Edmonds who is famous in the combinatorial optimization world. He is my academic grandfather on one side. He is a very interesting guy. OK, so that's a really bad algorithm. You have probably seen a somewhat better algorithm, which you might think of as the bottom-up implantation of that recursive algorithm. Or, another way to think of it is if you build out the recursion tree for Fibonacci of n, you will see that there are lots of common subtrees that you are just wasting time on. When you solve Fibonacci of n minus 1, it again solves Fibonacci of n minus 2. Why solve it twice? You only need to solve it once. So, it is really easy to do that if you do it bottom-up. But you could also do it recursively with some cache of things you have already computed. So, no big surprise. You compute the Fibonacci numbers in order. And each time, when you compute Fibonacci of n, let's say, you have already computed the previous two, you add them together, it takes constant time. So, the running time here is linear, linear in n, and as our input. Great. Is that the best we can do? No. Any ideas on how we could compute Fibonacci of n faster than linear time? Now we should diverge from what you have already seen, most of you. Any ideas using techniques you have already seen? Yes? Yes. We can use the mathematical trick of phi and psi to the nth powers. In fact, you can just use phi, phi, pi, pho, phum, whatever you want to call this Greek letter. Good. Here is the mathematical trick. And, indeed, this is cheating, as you have said. This is no good, but so it is. We will call it naÔve recursive squaring and say well, we know recursive squaring. Recursive squaring takes log n time. Let's use recursive squaring. And if you happen to know lots of properties of the Fibonacci numbers, you don't have to, but here is one of them. If you take phi to the n divided by root 5 and you round it to the nearest integer that is the nth Fibonacci number. This is pretty cool. Fibonacci of n is basically phi to the n. We could apply recursive squaring to compute phi to the n in log n time, divide by root 5, assume that our computer has an operation that rounds a number to its nearest integer and poof, we are done. That doesn't work for many different reasons. On a real machine, probably you would represent phi and root 5 as floating point numbers which have some fixed amount of precise bits. And if you do this computation, you will lose some of the important bits. And when you round to the nearest integer you won't get the right answer. So, floating point round off will kill you on a floating point machine. On a theoretical machine where we magically have numbers that can do crazy things like this, I mean it really takes more than constant time per multiplication. So, we are sort of in a different model. You cannot multiply phi times phi in constant time. I mean that's sort of outside the boundaries of this course, but that's the way it is. In fact, in a normal machine, some problems you can only solve in exponential time. In a machine where you can multiply real numbers and round them to the nearest integers, you can solve them in polynomial time. So, it really breaks the model. You can do crazy things if you were allowed to do this. This is not allowed. And I am foreshadowing like three classes ahead, or three courses ahead, so I shouldn't talk more about it. But it turns out we can use recursive squaring in a different way if we use a different property of Fibonacci numbers. And then we will just stick with integers and everything will be happy. Don't forget to go to recitation and if you want to homework lab. Don't come here on Monday. That is required. This is sort of the right recursive squaring algorithm. And this is a bit hard to guess if you haven't already seen it, so I will just give it to you. I will call this a theorem. It's the first time I get to use the word theorem in this class. It turns out the nth Fibonacci number is the nth power of this matrix. Cool. If you look at it a little bit you say oh, yeah, of course. And we will prove this theorem in a second. But once we have this theorem, we can compute f of n by computing the nth power of this matrix. It's a two-by-two matrix. You multiply two two-by-two matrixes together, you get a two-by-two matrix. So that is constant size, four numbers. I can handle four numbers. We don't have crazy precision problems on the floating point side. There are only four numbers to deal with. Matrixes aren't getting bigger. So, the running time of this divide-and-conquer algorithm will still be log n because it takes a constant time per two-by-two matrix multiplication. Yes? Oh, yes. Thank you. I have a type error. Sorry about that. F of n is indeed the upper left corner, I hope. I better check I don't have it off by one. I do. It's F_n upper right corner, indeed. That's what you said. F of n. I need more space. Sorry. I really ought to have a two-by-two matrix on the left-hand side there. Thank you. So, I compute this nth power of a matrix in log n time, I take the upper right corner or the lower left corner, your choice, that is the nth Fibonacci number. This implies an order log n time algorithm with the same recurrence as the last two, binary search and really the recursive squaring algorithm. It is log n plus a constant, so log n. Let's prove that theorem. Any suggestions on what techniques we might use for proving this theorem, or what technique, singular? Induction, very good. I think any time I ask that question the answer is induction. Hint for the future in this class. A friend of mine, when he took an analysis class, whenever the professor asked, and what is the answer to this question, the answer was always zero. If you have taken analysis class that is funny. [LAUGHTER] Maybe I will try to ask some questions whose answers are zero just for our own amusement. We are going to induct on n. It's pretty much the obvious thing to do. But we have to check some cases. So, the base case is we have this to the first power. And that is itself [(1, 1), (1, 0)]. And I should have said n is at least 1. And you can check. This is supposed to be F_2, F_1, F_1, F_0. And you can check it is, F_0 is 0, F_1 is 1 and F_2 is Base case is correct, step case is about as exciting, but you've got to prove that your algorithm works. Suppose this is what we want to compute. I am just going to sort of, well, there are many ways I can do this. I will just do it the fast way because it's really not that exciting. Which direction? Let's do this direction. I want to use induction on n. If I want to use induction on n, presumably I should use what I already know is true. If I decrease n by 1, I have this property that this thing is going to be [(1, 1), (1, 0)] to the power n minus 1. This I already know, by the induction hypothesis, [(1, 1), (1, 0)] to the n minus 1. So, presumably I should use it in some way. This equality is not yet true, you may have noticed. So, I need to add something on. What could I possibly add on to be correct? Well, another factor of [(1, 1), (1, 0)]. The way I am developing this proof is the only way it could possibly be, in some sense. If you know its induction, this is all that you could do. And then you check. Indeed, this equality holds conveniently. For example, F_(n plus 1) is the product of these two things. It is this row times this column. So, it is F_n times 1 plus F_(n minus 1) times 1, which is indeed the definition of F_(n plus 1). And you could check four of the entries. This is true. Great. If that is true then I would just put these together. That is [(1, 1), (1, 0)] to the n minus 1 times [(1, 1), (1, 0)], which is [(1, 1), (1, 0)] to the n, end of proof. A very simple proof, but you have to do that in order to know if this algorithm really works. Good. Question? Oh, yes. Thank you. This, in the lower right, we should have F_(n minus 1). This is why you should really check your proofs. We would have discovered that when I checked that this was that row times that column, but that is why you are here, to fix my bugs. That's the great thing about being up here instead of in a quiz. But that is a minor mistake. You wouldn't lose much for that. All right. More divide-and-conquer algorithms. Still, we have done relatively simple ones so far. In fact, the fanciest has been merge sort, which we already saw. So, that is not too exciting. The rest have all be log n time. Let's break out of the log n world. Well, you all have the master method memorized, right, so I can erase that. Good. This will be a good test. Next problem is matrix multiplication, following right up on this two-by-two matrix multiplication. Let's see how we can compute n-by-n matrix multiplications. Just for a recap, you should know how to multiply matrixes, but here is the definition so we can turn it into an algorithm. You have two matrixes, A and B, which are capital levels. The ijth entry. Ith row, jth column is called little a_ij or little b_ij. And your goal is to compute the products of those matrixes. I should probably say that i and j range from 1 to n. So, they are square matrixes. The output is to compute C which has entry c_ij which is the product of A and B. And, for a recap, the ijth entry of the product is the inner product of the ith row of A with the jth column of B. But you can write that out as a sum like so. We want to compute this thing for every i and j. What is the obvious algorithm for doing this? Well, for every i and j you compute the sum. You compute all the products. You compute the sum. So, it's like n operations here roughly. I mean like 2n minus 1, whatever. It is order n operations. There are n^2 entries of C that I need to compute, so that's n^3 time. I will write this out just for the programmers at heart. Here is the pseudocode. It's rare that I will write pseudocode. And this is a simple enough algorithm that I can write it in gory detail. But it gives you some basis for this analysis if you like to program. It is a triply nested for loop. And I made a coding error. Hopefully you haven't written that far yet. I need c_ij to be initialized to zero. And then I add to c_ij the appropriate product, a_ik, b_kj. That is the algorithm. And the point is you have a nesting of n for loops from 1 to n. That takes n^3 time because this is constant and that is constant. So, very simple, n^3. Let's do better. And, of course, we are going to use divide-and-conquer. Now, how are we going to divide a matrix? There are a lot of numbers in a matrix, n^2 of them in each one. There are all sorts of ways you could divide. So far all of the divide-and-conquers we have done have been problems of size n into some number of problems of size n over 2. I am going to say I start with some matrixes of size n-by-n. I want to convert it down to something like n/2-by-n/2. Any suggestions how I might do that? Yeah? Block form the matrix, indeed. That is the right answer. So, this is the first divide-and-conquer algorithm. This will not work, but it has the first idea that we need. We have a n-by-n matrix. We can view it, this equality is more, you can think of it as, it's really the thing, a two-by-two block matrix where each entry in this two-by-two block matrix is a block of n/2-by-n/2 submatrixes. I will think of C as being divided into three parts, r, s, t and u. Even though I write them as lower case letters they are really matrixes. Each is n/2-by-n/2. And A, I will split into a, b, c, d, times B, I will split into e, f, g, h. Why not? This is certainly true. And if you've seen some linear algebra, this is basically what you can do with matrixes. Now I can pretend these are two-by-two and sort of forget the fact that these little letters are matrixes and say well, r is the inner product of this row with this column. It is ae times bg. Let me not cheat or else it will be too easy. r=ae+bg, s=af+bh, t=ce+dh and u=cf+dg. It's nothing like making it too hard on yourself. OK, got them right. Good. I mean this is just a fact about how you would expand out this product. And so now we have a recursive algorithm. In fact, we have a divide-and-conquer algorithm. We start with our n-by-n matrix. Well, we have two of them actually. We divide it into eight little pieces, a, b, c, d, e, f, g, h. Then we compute these things and that gives us C, just by sticking them together. Now, how do we compute these things? Well, these innocent-looking little products between these two little numbers are actually recursive matrix multiplications. Because each of these little letters is an n/2-by-n/2 matrix so I have to recursively compute the product. There are like eight recursive multiplications of n/2-by-n/2 matrixes. That is what bites us. And then there are like four additions, plus minor work of gluing things together. How long does it take to add two matrixes together? n^2. This is cheap. It just takes n^2. Remember, we are trying to beat n^3 for our matrix multiplication. Addition is a really easy problem. You just have to add every number. There is no way you can do better than n^2. So, that is not recursive. That is the nice thing. But the bad thing is we have eight of these recursions. We have T(n)=8T(n/2)+Theta(n^2). And I have erased the master method, but you should all have it memorized. What is the solution to this recurrence? Theta(n^3). That is annoying. All right. A is 8, b is 2, log base 2 of 8 is 3. Every computer scientist should know that. n^log_b(a)=n^3. That is polynomially larger than n^2, so we are in Case 1. Thank you. Let's get them upside down. This is n^3, no better than our previous algorithm. That kind of sucks. And now comes the divine inspiration. Let's go over here. There are some algorithms like this Fibonacci algorithm where if you sat down for a little while, it's no big deal, you would figure it out. I mean it is kind of clever to look at that matrix and then everything works happily. It is not obvious but it is not that amazingly clever. This is an algorithm that is amazingly clever. You may have seen it before which steals the thunder a little bit, but it is still really, really cool so you should be happy to see it again. And how Strassen came up with this algorithm, he must have been very clever. The idea is we've got to get rid of these multiplications. I could do a hundred additions. That only costs Theta(n^2). I have to reduce this 8 to something smaller. It turns out, if you try to split the matrices into three-by-three or something, that doesn't help you. You get the same problem because we're using fundamentally the same algorithm, just in a different order. We have got to somehow reduce the number of multiplications. We are going to reduce it to 7. The claim is that if we have two two-by-two matrices we can take their product using seven multiplications. If that were true, we would reduce the 8 to a 7 and presumably make things run faster. We will see how fast in a moment. You can compute it in your head. If you are bored and like computing logs that are non-integral logs then go ahead. All right. Here we are. This algorithm is unfortunately rather long, but it is only seven multiplications. Each of these P's is a product of two things which only involves addition or subtraction, the same thing. Those are seven multiplications. And I can compute those in 7T(n/2). Oh, indeed it is. Six was wrong. Six and seven are the same, very good. You know, you think that copying something would not be such a challenging task. But when you become an absent-minded professor like me then you will know how easy it is. OK. We have them all correct, hopefully. We continue. That wasn't enough. Of course we had seven things. Clearly we have to reduce this down to four things, the elements of C. Here they are, the elements of C, r, s, t, u. It turns out r=P_5+P_4-P_2+P_6. Of course. Didn't you all see that? [LAUGHTER] I mean this one is really easy, s=P_1+P2. t=P_3+P_4. I mean that is clearly how they were chosen. And then u is another tricky one, u=P_5+P_1-P_3-P_7. OK. Now, which one of these would you like me to check? Don't be so nice. How about s? I can show you s is right. Any preferences? u. Oh, no, sign errors. OK. Here we go. The claim that this really works is you have to check all four of them. And I did in my notes. u=P_5. P_5=(ae + ah + de + dh). That is P_5. Check me. If I screw up, I am really hosed. (af - ah) = P_1. P_3 has a minus sign in front, so that is (ce + de). And then we have minus P_7, which is a big one, (ae + af - ce - cf). OK. Now I need like the assistant that crosses off things in parallel like the movie, right? ah, de, af, ce, ae, thank you, and hopefully these survive, dh minus minus cf. And, if we are lucky, that is exactly what is written here, except in the opposite order. Magic, right? Where the hell did Strassen get this? You have to be careful. It is OK that the plus is in the wrong order because plus is commutative, but the multiplications better not be in the wrong order because multiplication over matrixes is not commutative. I check cf, OK, dh, they are in the right order. I won't check the other three. That is matrix multiplication in hopefully subcubic time. Let's write down the recurrence. T(n) is now 7. Maybe I should write down the algorithm for kicks. Why not? Assuming I have time. Lots of time. Last lecture I was ten minutes early. I ended ten minutes early. I apologize for that. I know it really upsets you. And I didn't realize exactly when the class was supposed to end. So, today, I get to go ten minutes late. OK. Good. I'm glad you all agree. [LAUGHTER] I am kidding. Don't worry. OK. Algorithm. This is Strassen. First we divide, then we conquer and then we combine. As usual, I don't have it written anywhere here. Fine. Divide A and B. This is sort of trivial. Then we compute the terms -- -- for the products. This means we get ready to compute all the P's. We compute a+b, c+d, g-e, a+d, e+h and so on. All of the terms that appear in here, we compute those. That takes n2 time because it is just a bunch of additions and subtractions. No big deal. A constant number of them. Then we conquer by recursively computing all the P_i's. That is each our product of seven of them. We have P_1, P_2 up to P_7. And, finally, we combine, which is to compute r, s, t and u. And those are just additions and subtractions again, so they take n^2 times. So, here we finally get an algorithm that is nontrivial both in dividing and in combining. Recursion is always recursion, but now we have interesting steps one and three. The recurrence is T(n) is seven recursive subproblems, each are size n/2 plus order n^2, to do all this addition work. Now we need to solve this recurrence. We compute n^log_b(a), which here is nlog_2(7). And we know log base 2 of 8 is little bit less than 3 but still bigger than 2 because log base 2 of 4 is 2. So, it is going to be polynomially larger than n^2 but polynomially smaller than n^3. We are again in Case 1. And this is the cheating way to write n log base 2 of 7, nlg7. lg means log base 2. You should know that. It is all over the textbook and in our problem sets and what not, nlg7. And, in particular, if I have my calculator here. This is a good old-fashion calculator. No, that is wrong. Sorry. It is strictly less than 2.81. It is strictly less than 2.81. That is cool. I mean it is polynomially better than n^3. Still not as good as addition, which is n^2. It is generally believed, although we don't know whether you can multiply as fast as you can divide for matrices. We think you cannot get n^2, but who knows? It could still happen. There are no lower bounds. This is not the best algorithm for matrix multiplication. It is sort of the simplest that beats n^3. The best so far is like n^2.376. Getting closer to 2. You might think these numbers are a bit weird. Maybe the constants out here dominate the improvement you are getting in the exponent. It turns out improving the exponent is a big deal. I mean, as n gets larger exponents really come out to bite you. So, n^3 is pretty impractical for any very large values of n. And we known that Strassen will beat normal matrix multiplication if n is sufficiently large. The claim is that roughly at about 32 or so already you get an improvement, for other reasons, not just because the exponent gets better, but there you go. So, this is pretty good. This is completely impractical, so don't use whatever this algorithm is. I don't have the reference handy, but it is just trying to get a theoretical improvement. There may be others that are in between and more reasonable but that is not it. Wow, lots of time. Any questions? We're not done yet, but any questions before we move on for matrix multiplication? OK. I have one more problem. Divide-and-conquer is a pretty general idea. I mean, you can use it to dominate countries. You can use it to multiply matrices. I mean, who would have thought? Here is a very different kind of problem you can solve with divide-and-conquer. It is not exactly an algorithmic problem, although it is computer science. That is clear. This is very large-scale integration. The chips, they are very large scale integrated. Probably even more these days, but that is the catch phrase. Here is a problem, and it arises in VLSI layout. We won't get into too many details why, but you have some circuit. And here I am going to assume that the circuit is a binary tree. This is just part of a circuit. Assume for now here that it is just a complete binary tree. A complete binary tree looks like this. In all of my teachings, I have drawn this figure for sure the most. It is my favorite figure, the height four complete binary tree. OK, there it is. I have some tree like that as some height. I want to imbed it into some chip layout on a grid. Let's say it has n leaves. I want to imbed it into a grid with minimum area. This is a very cute problem and it really shows you another way in which divide-and-conquer is a useful and powerful tool. So, I have this tree. I like to draw it in this way. I want to somehow draw it on the grid. What that means is the vertices have to be imbedded onto dots on the grid, and I am talking about the square grid. It has to go to vertices of the grid. And these edges have to be routed as sort of orthogonal paths between one dot and another, so that should be an edge and they shouldn't cross and all these good things because wires do not like to cross. There is the obvious way to solve this problem and there is the right way. And let's talk about both of them. Neither of them is particularly obvious, but divide-and-conquer sort of gives you a hint in the right direction. So, the naÔve imbedding. I seem to like the word naÔve here. I am going to draw this bottom up because it is easier, so leave three grid lines and then start drawing. I don't know how big that is going to be. Here is the bottom of our tree. This is like the little three nodes there. And then I leave a blank column and then a blank column. I don't actually need to leave those blank columns, but it makes a prettier drawing. And then we work our way up. There is the tree, which should be aligned, on a grid. No crossings. Everything is happy. How much area does it take? By area, I mean sort of the area of the bounding box. So, I count this blank space even though I am not using it and I count all this blank space even though I am not using it. I want to look at the height. Let's call this H(n). And to look at the width, which I will call W(n). Now, it is probably pretty obvious that H(n) is like log n, W(n) is like n or whatever. But I want to write it as a recurrence because that will inspire us to do the right thing. H(n). Well, if you think of this as a recursion-tree, in some sense. We start with the big tree. We split it into two halves, two subtrees of size n/2 indeed because we are counting leaves. It is exactly n/2 on each side. Then for height they are in parallel so it is no big deal. The height is just the height of this thing, one of these subproblems plus one. The width, you have to add together the two widths and also add on 1. You don't have to add on 1 here, but it doesn't matter. It is certainly at most 1. H(n) = H(n/2) + Theta(1), there you do have to add 1, and W(n) = 2W(n/2) + O(1). The usual base cases. I mean, these are recurrences we should know and love. This is log n, I sort of have already given away the answers, and this better be linear. This is again Case 1. And to the log base 2 of 2 is n, which is the answer, much bigger than 1. And here n to the log base 2 of 1 is n to the zero, which is 1, which is the same so we get log n. The area is n log n, but if you are making chips you want the area as small as possible so you can fit more good stuff in there. So, we would like to aim for, well, we certainly cannot do a better area than n. You've got to put the leaves down somewhere, but this is already pretty good. It is only a log factor off, but we want to aim for n. How could we get n? Any guesses on what needs to change in this layout? Not how to do it because that is not obvious, but in terms of height and width what should we do? It is pretty hard to get the height smaller than log n, I will tell you, because this is a tree. It cannot really get its width down to less than log n. What could we do to make the product linear? Just random ideas. What are two functions whose product is n? Square root of n and square root of n. That is a good choice. Were there other suggestions? n times constant. Yeah, n times constant would be nice. But I claim you cannot get either of these down to less than a constant. You could aim for n over log n by log n, that is more likely, but I think that is almost impossible. Root n by root n is the right answer, so let's go with that. So, root n by root n. We haven't seen any recurrences whose solution is root n, but surely they are out there. Let's say the goal is to get W(n) = Theta(root n) and to get H(n) = Theta(root n). If we did that we would be happy, because then the area is the product is linear. How? What is a recurrence that is in the usual master method form whose solution is root n? I mean, you could think of it that way. Recurrence is a bit tricky, but let's just think of n^log_b(a). When is log base b of a Ω? Because then n^log_b(a) is root n. And there is some hope that I could get a root n solution to recurrence. This is designed by knowing that it is divide-and-conquer, and therefore it must be something like this. It is easy once you know the approach you are supposed to take and you can try this approach. When is log base b of a Ω? Lots of solutions, shout them out. 4 and 2, that is a good one. I better get this right. Log base 4 of 2 is Ω because the square root of 4 is 2. So, let's aim for this. Why not? When would we get log base 4 of 2? This is b, this is a, so it should be 2T(n/4) plus something. And if I want the n^log_b(a) to dominate, it has got to be polynomially smaller than root n. So, this should be n^1/2-epsilon. But it could be smaller. It could be 1. Zero would be nice, but that is probably too much to hope for. So, something smaller, strictly polynomially smaller than root n. That is our goal. And now comes the magic. If you played with this for a while you would find it, I think, at this point. When you know that you are somehow solve this problem of size n with two subproblems of size n/4 what could you do? Well, if you start thinking of things as squares, this is the natural thing that happens. This is called the H layout. You can imagine why. It would be much easier to draw if I had a grid board, a graph board, whatever, if that exists. This is a recursive layout. I am only going to draw a couple iterations, but hopefully you can imagine the generalization. I take four Hs, a good plan because I want problems of size n/4. This has n/4 leaves. This has n/4 leaves. This is the root, by the way, in the middle. This has n/4 leaves. This has n/4 leaves. So, I have four problems of size n/4. Somehow I have got to get that down to two. Thankfully, if I look at width or if I look at height, there are only two that matter. And these two matter and these two get along for free. They are going in parallel, just like we had with height over here. But I get that both in height and in width. If I measure, well, now they are equal, so I will just call them the length. We have L(n)-by-L(n). And if I compute well, what is L(n)? I have here L(n/4) because there are only a quarter of the leaves in this one or in that one, then I have a constant, Theta(1), no big deal, and then I have L(n/4) again. So, I get the recurrence that I wanted. L(n) = 2L(n/4) + Theta(1). And that has solution square root of n, as we claimed before. Again, we are in Case 1 of the master method. Cool, ha? This is a much more compact layout. Charles, did you invent this layout? No. But I know it appears on your PhD thesis and you extended it in various directions. So, this is sort of a classic cool layout of trees into grids and another application of divide-and-conquer. I mean this is not particularly useful for algorithms directly. It is useful for VLSI layout directly. But it gives you more flavor of how you should think. If you know what running time you are aiming for, like in problem sets in quizzes often we say here is the running time you have got to get, think about the recurrence that will get you there. And that could inspire you. And that is it. Recitation Friday. Homework lab Sunday. No class Monday. See you Wednesday.
Free Downloads
Free Streaming
Video
- iTunes U (MP4 - 143MB)
- Internet Archive (MP4 - 282MB)
Caption
- English-US (SRT)