1 00:00:07,270 --> 00:00:12,540 OK, here's the last lecture in the chapter on orthogonality. 2 00:00:12,540 --> 00:00:16,900 So we met orthogonal vectors, two vectors, 3 00:00:16,900 --> 00:00:22,530 we met orthogonal subspaces, like the row space and null 4 00:00:22,530 --> 00:00:23,590 space. 5 00:00:23,590 --> 00:00:28,720 Now today we meet an orthogonal basis, 6 00:00:28,720 --> 00:00:31,370 and an orthogonal matrix. 7 00:00:31,370 --> 00:00:32,920 So we really -- 8 00:00:32,920 --> 00:00:36,170 this chapter cleans up orthogonality. 9 00:00:36,170 --> 00:00:38,930 And really I want -- 10 00:00:38,930 --> 00:00:43,880 I should use the word orthonormal. 11 00:00:43,880 --> 00:00:51,690 Orthogonal is -- so my vectors are q1,q2 up to qn -- 12 00:00:51,690 --> 00:00:56,180 I use the letter "q", here, to remind me, 13 00:00:56,180 --> 00:01:01,270 I'm talking about orthogonal things, not just any vectors, 14 00:01:01,270 --> 00:01:02,690 but orthogonal ones. 15 00:01:02,690 --> 00:01:04,230 So what does that mean? 16 00:01:04,230 --> 00:01:08,600 That means that every q is orthogonal to every other q. 17 00:01:08,600 --> 00:01:13,880 It's a natural idea, to have a basis that's 18 00:01:13,880 --> 00:01:17,060 headed off at ninety-degree angles, 19 00:01:17,060 --> 00:01:19,120 the inner products are all zero. 20 00:01:19,120 --> 00:01:26,260 Of course if q is -- certainly qi is not orthogonal to itself. 21 00:01:26,260 --> 00:01:29,200 But there we'll make the best choice again, 22 00:01:29,200 --> 00:01:31,050 make it a unit vector. 23 00:01:31,050 --> 00:01:35,610 Then qi transpose qi is one, for a unit vector. 24 00:01:35,610 --> 00:01:37,590 The length squared is one. 25 00:01:37,590 --> 00:01:41,430 And that's what I would use the word normal. 26 00:01:41,430 --> 00:01:48,479 So for this part, normalized, unit length for this part. 27 00:01:48,479 --> 00:01:48,979 OK. 28 00:01:48,979 --> 00:01:55,120 So first part of the lecture is how 29 00:01:55,120 --> 00:01:59,320 does having an orthonormal basis make things nice? 30 00:01:59,320 --> 00:02:00,330 It certainly does. 31 00:02:00,330 --> 00:02:02,490 It makes all the calculations better, 32 00:02:02,490 --> 00:02:05,150 a whole lot of numerical linear algebra 33 00:02:05,150 --> 00:02:10,520 is built around working with orthonormal vectors, 34 00:02:10,520 --> 00:02:12,230 because they never get out of hand, 35 00:02:12,230 --> 00:02:16,220 they never overflow or underflow. 36 00:02:16,220 --> 00:02:20,390 And I'll put them into a matrix Q, 37 00:02:20,390 --> 00:02:22,430 and then the second part of the lecture 38 00:02:22,430 --> 00:02:26,630 will be suppose my basis, my columns of A 39 00:02:26,630 --> 00:02:28,710 are not orthonormal. 40 00:02:28,710 --> 00:02:30,660 How do I make them so? 41 00:02:30,660 --> 00:02:35,110 And the two names associated with that simple idea 42 00:02:35,110 --> 00:02:36,840 are Graham and Schmidt. 43 00:02:36,840 --> 00:02:43,410 So the first part is we've got a basis like this. 44 00:02:43,410 --> 00:02:46,040 Let's put those into the columns of a matrix. 45 00:02:48,620 --> 00:02:53,990 So a matrix Q that has -- 46 00:02:53,990 --> 00:02:56,680 I'll put these orthonormal vectors, 47 00:02:56,680 --> 00:03:01,650 q1 will be the first column, qn will be the n-th column. 48 00:03:04,350 --> 00:03:09,610 And I want to say, I want to write this property, 49 00:03:09,610 --> 00:03:13,890 qi transpose qj being zero, I want 50 00:03:13,890 --> 00:03:17,640 to put that in a matrix form. 51 00:03:17,640 --> 00:03:23,390 And just the right thing is to look at Q transpose Q. 52 00:03:23,390 --> 00:03:27,520 So this chapter has been looking at A transpose A. 53 00:03:27,520 --> 00:03:30,470 So it's natural to look at Q transpose Q. 54 00:03:30,470 --> 00:03:33,490 And the beauty is it comes out perfectly. 55 00:03:33,490 --> 00:03:38,160 Because Q transpose has these vectors in its rows, 56 00:03:38,160 --> 00:03:45,510 the first row is q1 transpose, the nth row is qn transpose. 57 00:03:45,510 --> 00:03:49,070 So that's Q transpose. 58 00:03:49,070 --> 00:03:51,610 And now I want to multiply by Q. 59 00:03:51,610 --> 00:03:56,810 That has q1 along to qn in the columns. 60 00:03:56,810 --> 00:03:58,430 That's Q. 61 00:03:58,430 --> 00:03:59,410 And what do I get? 62 00:04:01,940 --> 00:04:06,240 You really -- this is the first simplest most basic fact, 63 00:04:06,240 --> 00:04:10,930 that how do orthonormal vectors, orthonormal columns 64 00:04:10,930 --> 00:04:17,660 in a matrix, what happens if I compute Q transpose Q? 65 00:04:17,660 --> 00:04:18,850 Do you see it? 66 00:04:18,850 --> 00:04:23,060 If I take the first row times the first column, 67 00:04:23,060 --> 00:04:25,170 what do I get? 68 00:04:25,170 --> 00:04:26,680 A one. 69 00:04:26,680 --> 00:04:29,570 If I take the first row times the second column, 70 00:04:29,570 --> 00:04:31,330 what do I get? 71 00:04:31,330 --> 00:04:32,280 Zero. 72 00:04:32,280 --> 00:04:34,170 That's the orthogonality. 73 00:04:34,170 --> 00:04:37,460 The first row times the last column is zero. 74 00:04:37,460 --> 00:04:40,630 And so I'm getting ones on the diagonal 75 00:04:40,630 --> 00:04:43,620 and I'm getting zeroes everywhere else. 76 00:04:43,620 --> 00:04:45,080 I'm getting the identity matrix. 77 00:04:47,880 --> 00:04:50,920 You see how that's -- it's just like the right calculation 78 00:04:50,920 --> 00:04:51,740 to do. 79 00:04:51,740 --> 00:04:55,540 If you have orthonormal columns, and the matrix 80 00:04:55,540 --> 00:04:59,420 doesn't have to be square here. 81 00:04:59,420 --> 00:05:02,290 We might have just two columns. 82 00:05:02,290 --> 00:05:05,670 And they might have four, lots of components. 83 00:05:05,670 --> 00:05:13,370 So but they're orthonormal, and when we do Q transpose times Q, 84 00:05:13,370 --> 00:05:17,110 that Q transpose times Q or A transpose A 85 00:05:17,110 --> 00:05:22,680 just asks for all those dot products. 86 00:05:22,680 --> 00:05:24,530 Rows times columns. 87 00:05:24,530 --> 00:05:29,390 And in this orthonormal case, we get the best possible answer, 88 00:05:29,390 --> 00:05:30,850 the identity. 89 00:05:30,850 --> 00:05:33,330 OK, so this is -- 90 00:05:33,330 --> 00:05:39,690 so I mean now we have a new bunch of important matrices. 91 00:05:39,690 --> 00:05:41,070 What have we seen previously? 92 00:05:41,070 --> 00:05:43,440 We've seen in the distant past we 93 00:05:43,440 --> 00:05:48,180 had triangular matrices, diagonal matrices, permutation 94 00:05:48,180 --> 00:05:51,760 matrices, that was early chapters, 95 00:05:51,760 --> 00:05:59,190 then we had row echelon forms, then in this chapter 96 00:05:59,190 --> 00:06:02,900 we've already seen projection matrices, 97 00:06:02,900 --> 00:06:08,830 and now we're seeing this new class of matrices 98 00:06:08,830 --> 00:06:11,490 with orthonormal columns. 99 00:06:11,490 --> 00:06:13,510 That's a very long expression. 100 00:06:13,510 --> 00:06:19,020 I sorry that I can't just call them orthogonal matrices. 101 00:06:19,020 --> 00:06:22,090 But that word orthogonal matrices -- 102 00:06:22,090 --> 00:06:25,440 or maybe I should be able to call it orthonormal matrices, 103 00:06:25,440 --> 00:06:28,090 why don't we call it orthonormal -- 104 00:06:28,090 --> 00:06:31,290 I mean that would be an absolutely perfect name. 105 00:06:31,290 --> 00:06:33,860 For Q, call it an orthonormal matrix 106 00:06:33,860 --> 00:06:36,370 because its columns are orthonormal. 107 00:06:36,370 --> 00:06:42,270 OK, but the convention is that we only use that name 108 00:06:42,270 --> 00:06:46,930 orthogonal matrix, we only use this -- 109 00:06:46,930 --> 00:06:49,670 this word orthogonal, we don't even 110 00:06:49,670 --> 00:06:51,950 say orthonormal for some unknown reason, 111 00:06:51,950 --> 00:06:54,530 matrix when it's square. 112 00:06:59,130 --> 00:07:04,740 So in the case when this is a square matrix, that's the case 113 00:07:04,740 --> 00:07:07,830 we call it an orthogonal matrix. 114 00:07:07,830 --> 00:07:12,290 And what's special about the case when it's square? 115 00:07:12,290 --> 00:07:19,390 When it's a square matrix, we've got its inverse, so -- 116 00:07:19,390 --> 00:07:33,850 so in the case if Q is square, then Q transpose Q equals I 117 00:07:33,850 --> 00:07:35,730 tells us -- 118 00:07:35,730 --> 00:07:37,820 let me write that underneath -- 119 00:07:37,820 --> 00:07:47,670 tells us that Q transpose is Q inverse. 120 00:07:47,670 --> 00:07:51,000 There we have the easy to remember 121 00:07:51,000 --> 00:07:57,440 property for a square matrix with orthonormal columns. 122 00:07:57,440 --> 00:08:01,960 That -- I need to write some examples down. 123 00:08:01,960 --> 00:08:04,180 Let's see. 124 00:08:04,180 --> 00:08:07,750 Some examples like if I take any -- so examples, 125 00:08:07,750 --> 00:08:08,720 let's do some examples. 126 00:08:12,100 --> 00:08:14,830 Any permutation matrix, let me take just 127 00:08:14,830 --> 00:08:17,360 some random permutation matrix. 128 00:08:17,360 --> 00:08:23,910 Permutation Q equals let's say oh, make it three by three, 129 00:08:23,910 --> 00:08:30,050 say zero, zero, one, one, zero, zero, zero, one, zero. 130 00:08:30,050 --> 00:08:30,550 OK. 131 00:08:36,470 --> 00:08:41,860 That certainly has unit vectors in its columns. 132 00:08:41,860 --> 00:08:45,940 Those vectors are certainly perpendicular to each other. 133 00:08:45,940 --> 00:08:48,700 And if I -- and so that's it. 134 00:08:48,700 --> 00:08:50,440 That makes it a Q. 135 00:08:50,440 --> 00:08:56,360 And -- if I took its transpose, if I multiplied by Q transpose, 136 00:08:56,360 --> 00:08:59,640 shall I do that -- and let me stick in Q transpose 137 00:08:59,640 --> 00:09:00,400 here. 138 00:09:00,400 --> 00:09:02,990 Just to do that multiplication once more, 139 00:09:02,990 --> 00:09:05,200 transpose it'll put the -- 140 00:09:05,200 --> 00:09:08,720 make that into a column, make that into a column, 141 00:09:08,720 --> 00:09:10,920 make that into a column. 142 00:09:10,920 --> 00:09:14,200 And the transpose is also -- 143 00:09:14,200 --> 00:09:15,690 another Q. 144 00:09:15,690 --> 00:09:17,070 Another orthonormal matrix. 145 00:09:17,070 --> 00:09:22,110 And when I multiply that product I get I. OK, 146 00:09:22,110 --> 00:09:23,890 so there's an example. 147 00:09:23,890 --> 00:09:26,860 And actually there's a second example. 148 00:09:26,860 --> 00:09:29,420 But those are real easy examples, right, 149 00:09:29,420 --> 00:09:34,780 I mean to get orthogonal columns by just 150 00:09:34,780 --> 00:09:40,630 putting ones in different places is like too easy. 151 00:09:40,630 --> 00:09:43,030 So let me keep going with examples. 152 00:09:43,030 --> 00:09:46,170 So here's another simple example. 153 00:09:46,170 --> 00:09:51,170 Cos theta sine theta, there's a unit vector, 154 00:09:51,170 --> 00:09:53,920 oh, let me even take it, well, yeah. 155 00:09:53,920 --> 00:09:58,500 Cos theta sine theta and now the other way 156 00:09:58,500 --> 00:10:01,790 I want sine theta cos theta. 157 00:10:01,790 --> 00:10:05,700 But I want the inner product to be zero. 158 00:10:05,700 --> 00:10:09,150 And if I put a minus there, it'll do it. 159 00:10:09,150 --> 00:10:12,760 So that's -- unit vector, that's a unit vector. 160 00:10:12,760 --> 00:10:16,920 And if I take the dot product, I get minus plus zero. 161 00:10:19,470 --> 00:10:19,970 OK. 162 00:10:19,970 --> 00:10:27,230 For example Q equals say one, one, one, minus one, 163 00:10:27,230 --> 00:10:30,710 is that an orthogonal matrix? 164 00:10:33,280 --> 00:10:35,600 I've got orthogonal columns there, 165 00:10:35,600 --> 00:10:37,920 but it's not quite an orthogonal matrix. 166 00:10:37,920 --> 00:10:43,000 How shall I fix it to be an orthogonal matrix? 167 00:10:43,000 --> 00:10:45,780 Well, what's the length of those column vectors, 168 00:10:45,780 --> 00:10:51,540 the dot product with themselves is -- right now it's two, 169 00:10:51,540 --> 00:10:52,760 right, the -- 170 00:10:52,760 --> 00:10:55,040 the length squared. 171 00:10:55,040 --> 00:10:57,710 The length squared would be one plus one would be two, 172 00:10:57,710 --> 00:10:59,340 the length would be square root of two, 173 00:10:59,340 --> 00:11:02,790 so I better divide by square root of two. 174 00:11:02,790 --> 00:11:03,750 OK. 175 00:11:03,750 --> 00:11:08,080 So there's a -- there now I have got an orthogonal matrix, 176 00:11:08,080 --> 00:11:13,350 in fact, it's this one -- when theta is pi over four. 177 00:11:13,350 --> 00:11:16,210 The cosines and well almost, I guess 178 00:11:16,210 --> 00:11:19,610 the minus sine is down there, so maybe, I 179 00:11:19,610 --> 00:11:23,650 don't know, maybe minus pi over four or something. 180 00:11:23,650 --> 00:11:24,150 OK. 181 00:11:26,760 --> 00:11:28,760 Let me do one final example, just 182 00:11:28,760 --> 00:11:32,000 to show that you can get bigger ones. 183 00:11:32,000 --> 00:11:38,270 Q equals let me take that matrix up in the corner 184 00:11:38,270 --> 00:11:41,830 and I'll sort of repeat that pattern, 185 00:11:41,830 --> 00:11:47,010 repeat it again, and then minus it down here. 186 00:11:51,110 --> 00:11:57,800 That's one of the world's favorite orthogonal matrices. 187 00:11:57,800 --> 00:12:00,090 I hope I got it right, is -- 188 00:12:00,090 --> 00:12:02,500 can you see whether -- 189 00:12:02,500 --> 00:12:05,890 if I take the inner product of one column with another one, 190 00:12:05,890 --> 00:12:07,990 let's see, if I take the inner product 191 00:12:07,990 --> 00:12:11,070 of that column with that I have two minuses and two pluses, 192 00:12:11,070 --> 00:12:11,960 that's good. 193 00:12:11,960 --> 00:12:14,280 When I take the inner product of that with that 194 00:12:14,280 --> 00:12:17,390 I have a plus and a minus, a minus and a plus. 195 00:12:17,390 --> 00:12:17,890 Good. 196 00:12:17,890 --> 00:12:19,530 I think it all works out. 197 00:12:19,530 --> 00:12:22,210 And what do I have to divide by now? 198 00:12:22,210 --> 00:12:24,500 To make those into unit vectors. 199 00:12:27,320 --> 00:12:34,520 Right now the vector one, one, one, one has length two. 200 00:12:34,520 --> 00:12:35,660 Square root of four. 201 00:12:35,660 --> 00:12:39,680 So I have to divide by two to make it unit vector, 202 00:12:39,680 --> 00:12:41,190 so there's another. 203 00:12:41,190 --> 00:12:45,550 That's my entire array of simple examples. 204 00:12:49,190 --> 00:12:56,610 This construction is named after a guy called Adhemar and we 205 00:12:56,610 --> 00:13:02,960 know how to do it for two, four, sixteen, 206 00:13:02,960 --> 00:13:10,670 sixty-four and so on, but we -- nobody knows exactly which size 207 00:13:10,670 --> 00:13:12,970 matrices have -- 208 00:13:12,970 --> 00:13:18,250 which size -- which sizes allow orthogonal matrices of ones 209 00:13:18,250 --> 00:13:19,300 and minus ones. 210 00:13:19,300 --> 00:13:24,450 So Adhemar matrix is an orthogonal matrix that's got 211 00:13:24,450 --> 00:13:30,070 ones and minus ones, and a lot of ones -- some we know, 212 00:13:30,070 --> 00:13:34,130 some other sizes, there couldn't be a five by five I think. 213 00:13:34,130 --> 00:13:35,970 But there are some sizes that nobody 214 00:13:35,970 --> 00:13:42,220 yet knows whether there could be or can't be a matrix like that. 215 00:13:42,220 --> 00:13:43,160 OK. 216 00:13:43,160 --> 00:13:47,860 You see those orthogonal matrices. 217 00:13:47,860 --> 00:13:54,910 Now let me ask what -- why is it good to have orthogonal 218 00:13:54,910 --> 00:13:56,050 matrices? 219 00:13:56,050 --> 00:13:59,830 What calculation is made easy? 220 00:13:59,830 --> 00:14:02,040 If I have an orthogonal matrix. 221 00:14:02,040 --> 00:14:06,660 And -- let me remember that the matrix could be rectangular. 222 00:14:06,660 --> 00:14:07,880 Shall I put down -- 223 00:14:07,880 --> 00:14:10,680 I better put a rectangular example down. 224 00:14:10,680 --> 00:14:13,270 So the -- these were all square examples. 225 00:14:13,270 --> 00:14:14,910 Can I put down just -- 226 00:14:14,910 --> 00:14:18,190 a rectangular one just to be sure 227 00:14:18,190 --> 00:14:22,080 that we realize that this is possible. 228 00:14:22,080 --> 00:14:23,930 let's help me out. 229 00:14:23,930 --> 00:14:33,870 Let's see, if I put like a one, two, two and a minus two, 230 00:14:33,870 --> 00:14:35,630 minus one, two. 231 00:14:40,530 --> 00:14:45,340 That's -- a matrix -- oh its columns aren't normalized yet. 232 00:14:45,340 --> 00:14:47,950 I always have to remember to do that. 233 00:14:47,950 --> 00:14:50,590 I always do that last because it's easy to do. 234 00:14:50,590 --> 00:14:53,460 What's the length of those columns? 235 00:14:53,460 --> 00:14:56,420 So if I wanted them -- if I wanted them to be length one, 236 00:14:56,420 --> 00:14:59,840 I should divide by their length, which is -- 237 00:14:59,840 --> 00:15:03,440 so I'd look at one squared plus two squared plus two squared, 238 00:15:03,440 --> 00:15:06,420 that's one and four and four is nine, 239 00:15:06,420 --> 00:15:11,420 so I take the square root and I need to divide by three. 240 00:15:11,420 --> 00:15:11,920 OK. 241 00:15:11,920 --> 00:15:14,780 So there is -- 242 00:15:14,780 --> 00:15:21,320 well, without that, I've got one orthonormal vector. 243 00:15:21,320 --> 00:15:24,060 I mean just one unit vector. 244 00:15:24,060 --> 00:15:26,130 Now put that guy in. 245 00:15:26,130 --> 00:15:29,230 Now I have a basis for the column 246 00:15:29,230 --> 00:15:34,980 space for a two-dimensional space, an orthonormal basis, 247 00:15:34,980 --> 00:15:35,500 right? 248 00:15:35,500 --> 00:15:38,140 These two columns are orthonormal, 249 00:15:38,140 --> 00:15:40,660 they would be an orthonormal basis 250 00:15:40,660 --> 00:15:45,190 for this two-dimensional space that they span. 251 00:15:45,190 --> 00:15:48,700 Orthonormal vectors by the way have got to be independent. 252 00:15:48,700 --> 00:15:52,910 It's easy to show that orthonormal vectors 253 00:15:52,910 --> 00:15:55,640 since they're headed off all at ninety degrees 254 00:15:55,640 --> 00:15:58,630 there's no combination that gives zero. 255 00:15:58,630 --> 00:16:06,970 Now if I wanted to create now a third one, 256 00:16:06,970 --> 00:16:13,220 I could either just put in some third vector that was 257 00:16:13,220 --> 00:16:18,800 independent and go to this Graham-Schmidt calculation that 258 00:16:18,800 --> 00:16:22,860 I'm going to explain, or I could be inspired and say look, 259 00:16:22,860 --> 00:16:26,990 that -- with that pattern, why not put a one in there, 260 00:16:26,990 --> 00:16:29,650 and a two in there, and a two in there, 261 00:16:29,650 --> 00:16:33,260 and try to fix up the signs so that they worked. 262 00:16:36,530 --> 00:16:37,070 Hmm. 263 00:16:37,070 --> 00:16:41,250 I don't know if I've done this too brilliantly. 264 00:16:41,250 --> 00:16:43,230 Let's see, what signs, that's minus, 265 00:16:43,230 --> 00:16:49,270 maybe I'd make a minus sign there, how would that be? 266 00:16:49,270 --> 00:16:52,990 Yeah, maybe that works. 267 00:16:52,990 --> 00:17:00,170 I think that those three columns are orthonormal and they -- 268 00:17:00,170 --> 00:17:03,720 the beauty of this -- this is the last example I'll probably 269 00:17:03,720 --> 00:17:08,089 find where there's no square root, the -- 270 00:17:08,089 --> 00:17:11,250 the punishing thing in Graham-Schmidt, 271 00:17:11,250 --> 00:17:14,720 maybe we better know that in advance, 272 00:17:14,720 --> 00:17:19,950 is that because I want these vectors to be unit vectors, 273 00:17:19,950 --> 00:17:21,900 I'm always running into square roots. 274 00:17:21,900 --> 00:17:24,270 I'm always dividing by lengths. 275 00:17:24,270 --> 00:17:26,109 And those lengths are square roots. 276 00:17:26,109 --> 00:17:29,900 So you'll see as soon as I do a Graham-Schmidt example, 277 00:17:29,900 --> 00:17:32,190 square roots are going to show up. 278 00:17:32,190 --> 00:17:34,960 But here are some examples where we did it 279 00:17:34,960 --> 00:17:36,850 without any square root. 280 00:17:36,850 --> 00:17:38,900 OK. 281 00:17:38,900 --> 00:17:42,110 So -- so great. 282 00:17:42,110 --> 00:17:50,760 Now next question is what's the good of having a Q? 283 00:17:50,760 --> 00:17:52,860 What formulas become easier? 284 00:17:52,860 --> 00:17:57,540 Suppose I want to project, so suppose Q -- 285 00:17:57,540 --> 00:18:03,010 suppose Q has orthonormal columns. 286 00:18:03,010 --> 00:18:05,150 I'm using the letter Q to mean this, 287 00:18:05,150 --> 00:18:07,130 I'll write it this one more time, 288 00:18:07,130 --> 00:18:12,120 but I always mean when I write a Q, 289 00:18:12,120 --> 00:18:14,910 I always mean that it has orthonormal columns. 290 00:18:14,910 --> 00:18:25,910 So suppose I want to project onto its column space. 291 00:18:31,590 --> 00:18:33,170 So what's the projection matrix? 292 00:18:36,480 --> 00:18:40,760 What's the projection matrix is I project onto a column space? 293 00:18:40,760 --> 00:18:46,310 OK, that gives me a chance to review the projection section, 294 00:18:46,310 --> 00:18:50,770 including that big formula, which used to be -- 295 00:18:50,770 --> 00:18:53,580 those four As in a row, but now it's 296 00:18:53,580 --> 00:18:57,350 got Qs, because I'm projecting onto the column space of Q, 297 00:18:57,350 --> 00:18:58,780 so do you remember what it was? 298 00:18:58,780 --> 00:19:05,700 It's Q Q transpose Q inverse Q transpose. 299 00:19:08,500 --> 00:19:12,390 That's my four Qs in a row. 300 00:19:12,390 --> 00:19:13,690 But what's good here? 301 00:19:16,490 --> 00:19:21,040 What -- what makes this formula nice if I'm projecting onto 302 00:19:21,040 --> 00:19:24,660 a column space when I have orthonormal basis for that 303 00:19:24,660 --> 00:19:25,440 space? 304 00:19:25,440 --> 00:19:29,110 What makes it nice is this is the identity. 305 00:19:29,110 --> 00:19:31,360 I don't have to do any inversion. 306 00:19:31,360 --> 00:19:33,190 I just get Q Q transpose. 307 00:19:40,170 --> 00:19:44,250 So Q Q transpose is a projection matrix. 308 00:19:44,250 --> 00:19:45,590 Oh, I can't help -- 309 00:19:45,590 --> 00:19:47,970 I can't resist just checking the properties, 310 00:19:47,970 --> 00:19:52,640 what are the properties of a projection matrix? 311 00:19:52,640 --> 00:19:57,280 There are two properties to know for any projection matrix. 312 00:19:57,280 --> 00:20:00,560 And I'm saying that this is the right projection 313 00:20:00,560 --> 00:20:04,500 matrix when we've got this orthonormal basis 314 00:20:04,500 --> 00:20:06,980 in the columns. 315 00:20:06,980 --> 00:20:07,540 OK. 316 00:20:07,540 --> 00:20:10,650 So there's the projection matrix. 317 00:20:10,650 --> 00:20:13,670 Suppose the matrix is square. 318 00:20:13,670 --> 00:20:17,090 First just tell me first this extreme case. 319 00:20:17,090 --> 00:20:22,190 If my matrix is square and it's got these orthonormal columns, 320 00:20:22,190 --> 00:20:26,170 then what's the column space? 321 00:20:26,170 --> 00:20:31,270 If I have a square matrix and I have independent columns, 322 00:20:31,270 --> 00:20:34,950 and even orthonormal columns, then the column space 323 00:20:34,950 --> 00:20:37,170 is the whole space, right? 324 00:20:37,170 --> 00:20:41,250 And what's the projection matrix onto the whole space? 325 00:20:41,250 --> 00:20:43,790 The identity matrix. 326 00:20:43,790 --> 00:20:45,360 If I'm projecting in the whole space, 327 00:20:45,360 --> 00:20:49,690 every vector B is right where it's supposed to be 328 00:20:49,690 --> 00:20:52,810 and I don't have to move it by projection. 329 00:20:52,810 --> 00:20:57,800 So this would be -- 330 00:20:57,800 --> 00:21:02,390 I'll put in parentheses this is I if Q is square. 331 00:21:07,890 --> 00:21:10,360 Well that we said that already. 332 00:21:10,360 --> 00:21:14,990 If Q is square, that's the case where Q transpose is Q inverse, 333 00:21:14,990 --> 00:21:18,140 we can put it on the right, we can put it on the left, 334 00:21:18,140 --> 00:21:23,000 we always get the identity matrix, if it's square. 335 00:21:23,000 --> 00:21:29,070 But if it's not a square matrix then it's not -- 336 00:21:29,070 --> 00:21:32,510 we don't get the identity matrix. 337 00:21:32,510 --> 00:21:37,710 We have Q Q transpose, and just again 338 00:21:37,710 --> 00:21:41,050 what are those two properties of a projection matrix? 339 00:21:41,050 --> 00:21:44,400 First of all, it's symmetric. 340 00:21:44,400 --> 00:21:48,870 OK, no problem, that's certainly a symmetric So what's 341 00:21:48,870 --> 00:21:50,860 that second property of a projection? 342 00:21:50,860 --> 00:21:51,360 matrix. 343 00:21:51,360 --> 00:21:55,010 That if you project and project again you don't move the second 344 00:21:55,010 --> 00:21:55,760 time. 345 00:21:55,760 --> 00:21:58,220 So the other property of a projection matrix 346 00:21:58,220 --> 00:22:04,070 should be that Q Q transpose twice 347 00:22:04,070 --> 00:22:09,320 should be the same as Q Q transpose once. 348 00:22:09,320 --> 00:22:11,550 That's projection matrices. 349 00:22:11,550 --> 00:22:14,170 And that property better fall out 350 00:22:14,170 --> 00:22:18,540 right away because from the fact we 351 00:22:18,540 --> 00:22:24,060 know about orthonormal matrices, Q transpose Q is I. OK, 352 00:22:24,060 --> 00:22:25,010 you see it. 353 00:22:25,010 --> 00:22:30,320 In the middle here is sitting Q Q t- Q transpose Q, sorry, 354 00:22:30,320 --> 00:22:34,030 that's what I meant to say, Q transpose Q is I. 355 00:22:34,030 --> 00:22:37,000 So that's sitting right in the middle, that cancels out, 356 00:22:37,000 --> 00:22:40,760 to give the identity, we're left with one Q Q transpose, 357 00:22:40,760 --> 00:22:43,480 and we're all set. 358 00:22:43,480 --> 00:22:44,180 OK. 359 00:22:44,180 --> 00:22:48,840 So this is the projection matrix -- 360 00:22:48,840 --> 00:22:54,670 all the equation -- all the messy equations of this chapter 361 00:22:54,670 --> 00:22:59,370 become trivial when our matrix -- 362 00:22:59,370 --> 00:23:02,540 when we have this orthonormal basis. 363 00:23:02,540 --> 00:23:04,630 I mean what do I mean by all the equations? 364 00:23:04,630 --> 00:23:06,280 Well, the most important equation 365 00:23:06,280 --> 00:23:10,680 was the normal equation, do you remember old A transpose 366 00:23:10,680 --> 00:23:15,620 A x hat equals A transpose b? 367 00:23:15,620 --> 00:23:22,200 But now -- now A is Q. 368 00:23:22,200 --> 00:23:28,140 Now I'm thinking I have Q transpose Q X hat 369 00:23:28,140 --> 00:23:32,000 equals Q transpose b. 370 00:23:32,000 --> 00:23:33,280 And what's good about that? 371 00:23:37,220 --> 00:23:42,950 What's good is that matrix on the left side is the identity. 372 00:23:42,950 --> 00:23:46,110 The matrix on the left is the identity, Q transpose Q, 373 00:23:46,110 --> 00:23:49,510 normally it isn't, normally it's that matrix of inner products 374 00:23:49,510 --> 00:23:53,870 and you've to compute all those dopey inner products and -- 375 00:23:53,870 --> 00:23:56,030 and -- and solve the system. 376 00:23:56,030 --> 00:23:59,950 Here the inner products are all one or zero. 377 00:23:59,950 --> 00:24:01,540 This is the identity matrix. 378 00:24:01,540 --> 00:24:03,280 It's gone. 379 00:24:03,280 --> 00:24:05,910 And there's the answer. 380 00:24:05,910 --> 00:24:09,140 There's no inversion involved. 381 00:24:09,140 --> 00:24:15,730 Each component of x is a Q times b. 382 00:24:15,730 --> 00:24:21,240 What that equation is saying is that the i-th component is 383 00:24:21,240 --> 00:24:26,420 the i-th basis vector times b. 384 00:24:26,420 --> 00:24:34,760 That's -- probably the most important formula in some major 385 00:24:34,760 --> 00:24:40,720 parts of mathematics, that if we have orthonormal basis, 386 00:24:40,720 --> 00:24:47,660 then the component in the -- in the i-th, along the i-th -- 387 00:24:47,660 --> 00:24:54,630 the projection on the i-th basis vector is just qi transpose b. 388 00:24:54,630 --> 00:25:00,580 That number x that we look for is just a dot product. 389 00:25:00,580 --> 00:25:01,080 OK. 390 00:25:04,050 --> 00:25:10,040 OK, so I'm ready now for the sort of like second half 391 00:25:10,040 --> 00:25:11,530 of the lecture. 392 00:25:11,530 --> 00:25:16,320 Where we don't start with an orthogonal matrix, 393 00:25:16,320 --> 00:25:18,910 orthonormal vectors. 394 00:25:18,910 --> 00:25:21,610 We just start with independent vectors 395 00:25:21,610 --> 00:25:25,370 and we want to make them orthonormal. 396 00:25:25,370 --> 00:25:27,860 So I'm going to -- can I do that now? 397 00:25:27,860 --> 00:25:29,940 Now here comes Graham-Schmidt. 398 00:25:29,940 --> 00:25:31,490 So -- Graham-Schmidt. 399 00:25:39,460 --> 00:25:43,990 So this is a calculation, I won't say -- 400 00:25:43,990 --> 00:25:53,190 I can't quite say it's like elimination, because it's 401 00:25:53,190 --> 00:25:56,630 different, our goal isn't triangular anymore. 402 00:25:56,630 --> 00:26:00,780 With elimination our goal was make the matrix triangular. 403 00:26:00,780 --> 00:26:04,200 Now our goal is make the matrix orthogonal. 404 00:26:04,200 --> 00:26:07,580 Make those columns orthonormal. 405 00:26:07,580 --> 00:26:10,240 So let me start with two columns. 406 00:26:10,240 --> 00:26:13,000 So I start with vectors a and b. 407 00:26:16,960 --> 00:26:20,430 And they're just like -- here, let me draw them. 408 00:26:20,430 --> 00:26:22,780 Here's a. 409 00:26:22,780 --> 00:26:23,400 Here's b. 410 00:26:26,370 --> 00:26:27,580 For example. 411 00:26:27,580 --> 00:26:29,700 A isn't specially horizontal, wasn't 412 00:26:29,700 --> 00:26:34,150 meant to be, just a is one vector, b is another. 413 00:26:34,150 --> 00:26:38,040 I want to produce those two vectors, 414 00:26:38,040 --> 00:26:40,760 they might be in twelve-dimensional space, 415 00:26:40,760 --> 00:26:43,940 or they might be in two-dimensional space. 416 00:26:43,940 --> 00:26:46,300 They're independent, anyway. 417 00:26:46,300 --> 00:26:49,500 So I better be sure I say that. 418 00:26:49,500 --> 00:26:51,330 I start with independent vectors. 419 00:26:54,860 --> 00:26:58,500 And I want to produce out of that q 1 and q2, 420 00:26:58,500 --> 00:27:00,970 I want to produce orthonormal vectors. 421 00:27:03,880 --> 00:27:09,730 And Graham and Schmidt tell me how. 422 00:27:09,730 --> 00:27:10,260 OK. 423 00:27:10,260 --> 00:27:14,330 Well, actually you could tell me how, we don't need -- frankly, 424 00:27:14,330 --> 00:27:17,800 I don't know -- there's only one idea here, 425 00:27:17,800 --> 00:27:24,850 if Graham had the idea, I don't know what Schmidt did. 426 00:27:24,850 --> 00:27:28,140 But OK. 427 00:27:28,140 --> 00:27:29,470 So you'll see it. 428 00:27:29,470 --> 00:27:31,900 We don't need either of them, actually. 429 00:27:31,900 --> 00:27:33,630 OK, so what I going to do. 430 00:27:33,630 --> 00:27:36,630 I'll take that -- this first guy. 431 00:27:36,630 --> 00:27:37,970 OK. 432 00:27:37,970 --> 00:27:39,790 Well, he's fine. 433 00:27:43,480 --> 00:27:46,620 That direction is fine except -- 434 00:27:46,620 --> 00:27:50,300 yeah, I'll say OK, I'll settle for that direction. 435 00:27:50,300 --> 00:27:51,410 So I'm going to -- 436 00:27:51,410 --> 00:27:53,630 I'm going to get, so what I going to -- 437 00:27:53,630 --> 00:27:59,370 my goal is I'm going to get orthogonal vectors 438 00:27:59,370 --> 00:28:02,780 and I'll call those capital A and B. 439 00:28:02,780 --> 00:28:07,590 So that's the key step is to get from any two vectors 440 00:28:07,590 --> 00:28:09,430 to two orthogonal vectors. 441 00:28:09,430 --> 00:28:14,530 And then at the end, no problem, I'll get orthonormal vectors, 442 00:28:14,530 --> 00:28:20,630 how will -- what will those will be my qs, q1 and q2, 443 00:28:20,630 --> 00:28:21,450 and what will they 444 00:28:21,450 --> 00:28:21,950 be? 445 00:28:25,980 --> 00:28:30,270 Once I've got A and B orthogonal, well, look, 446 00:28:30,270 --> 00:28:35,390 it's no big deal -- maybe that's what Schmidt did, he, 447 00:28:35,390 --> 00:28:38,840 brilliant Schmidt, thought OK, divide by the length, 448 00:28:38,840 --> 00:28:40,310 all right. 449 00:28:40,310 --> 00:28:42,820 That's Schmidt's contribution. 450 00:28:45,640 --> 00:28:46,140 OK. 451 00:28:51,250 --> 00:28:55,620 But Graham had a little more thinking to do, right? 452 00:28:55,620 --> 00:28:58,780 We haven't done Graham's part. 453 00:28:58,780 --> 00:29:03,050 This part except OK, I'm happy with A, 454 00:29:03,050 --> 00:29:07,170 A can be A. That first direction is fine. 455 00:29:07,170 --> 00:29:09,620 Why should -- no complaint about that. 456 00:29:09,620 --> 00:29:13,620 The trouble is the second direction is not fine. 457 00:29:13,620 --> 00:29:18,170 Because it's not orthogonal to the first. 458 00:29:18,170 --> 00:29:23,710 I'm looking for a vector that's -- starts with B, 459 00:29:23,710 --> 00:29:29,550 but makes it orthogonal to A. 460 00:29:29,550 --> 00:29:30,980 What's the vector? 461 00:29:30,980 --> 00:29:32,870 How do I do that? 462 00:29:32,870 --> 00:29:35,960 How do I produce from this vector 463 00:29:35,960 --> 00:29:41,820 a piece that's orthogonal to this one? 464 00:29:41,820 --> 00:29:44,900 And the -- remember these vectors might be in two 465 00:29:44,900 --> 00:29:48,570 dimensions or they might be in twelve dimensions. 466 00:29:48,570 --> 00:29:51,210 I'm just looking for the idea. 467 00:29:51,210 --> 00:29:53,840 So what's the idea? 468 00:29:53,840 --> 00:29:57,110 Where did we have orthogonal -- 469 00:29:57,110 --> 00:30:01,440 a vector showing up that was orthogonal to this guy? 470 00:30:01,440 --> 00:30:03,990 Well, that was the first basic calculation 471 00:30:03,990 --> 00:30:05,780 of the whole chapter. 472 00:30:05,780 --> 00:30:10,850 We -- we did a projection and the projection gave us this 473 00:30:10,850 --> 00:30:15,890 part, which was the part in the A direction. 474 00:30:15,890 --> 00:30:19,500 Now, the part we want is the other part, the e part. 475 00:30:19,500 --> 00:30:21,050 This part. 476 00:30:21,050 --> 00:30:23,790 This is going to be our -- 477 00:30:23,790 --> 00:30:25,450 that guy is that guy. 478 00:30:25,450 --> 00:30:31,110 This is our vector B. That gives us that ninety-degree angle. 479 00:30:31,110 --> 00:30:33,240 So B is you could say -- 480 00:30:33,240 --> 00:30:35,730 B is really what we previously called 481 00:30:35,730 --> 00:30:37,140 e. 482 00:30:37,140 --> 00:30:41,060 The error vector. 483 00:30:41,060 --> 00:30:42,950 And what is it? 484 00:30:42,950 --> 00:30:45,950 I mean what do I -- what is B here? 485 00:30:45,950 --> 00:30:47,690 A is A, no problem. 486 00:30:47,690 --> 00:30:52,410 B is -- 487 00:30:52,410 --> 00:30:54,120 OK, what's this error piece? 488 00:30:54,120 --> 00:30:56,530 Do you remember? 489 00:30:56,530 --> 00:31:04,150 It's I start with the original B and I take away what? 490 00:31:04,150 --> 00:31:08,670 Its projection, this P. This -- the vector B, 491 00:31:08,670 --> 00:31:12,160 this error vector, is the original vector removing 492 00:31:12,160 --> 00:31:12,890 the projection. 493 00:31:12,890 --> 00:31:15,590 So instead of wanting the projection, 494 00:31:15,590 --> 00:31:20,260 now that's what I want to throw away. 495 00:31:20,260 --> 00:31:22,780 I want to get the part that's perpendicular. 496 00:31:22,780 --> 00:31:24,720 And there will be a perpendicular part, 497 00:31:24,720 --> 00:31:25,450 it won't be zero. 498 00:31:28,890 --> 00:31:32,880 Because these vectors were independent, so B -- 499 00:31:32,880 --> 00:31:34,880 if B was along the direction of A, 500 00:31:34,880 --> 00:31:37,830 then if the original B and A were in the same direction, 501 00:31:37,830 --> 00:31:38,760 then I'm -- 502 00:31:38,760 --> 00:31:40,500 I've only got one direction. 503 00:31:40,500 --> 00:31:43,440 But here they're in two independent directions 504 00:31:43,440 --> 00:31:46,190 and all I'm doing is getting that guy. 505 00:31:46,190 --> 00:31:49,550 So what's its formula? 506 00:31:49,550 --> 00:31:53,510 What's the formula for that if -- 507 00:31:53,510 --> 00:31:55,540 I want to subtract the projection, 508 00:31:55,540 --> 00:31:57,450 so do you remember the projection? 509 00:31:57,450 --> 00:32:03,490 It's some multiple of A and what's that multiple? 510 00:32:03,490 --> 00:32:07,030 It's -- it's that thing we called x in the very very first 511 00:32:07,030 --> 00:32:10,150 lecture on this chapter. 512 00:32:10,150 --> 00:32:16,540 There's an A transpose A in the bottom 513 00:32:16,540 --> 00:32:23,990 and there's an A transpose B, isn't that it? 514 00:32:29,780 --> 00:32:32,240 I think that's Graham's formula. 515 00:32:32,240 --> 00:32:33,300 Or Graham-Schmidt. 516 00:32:33,300 --> 00:32:34,600 No, that's Graham. 517 00:32:34,600 --> 00:32:38,120 Schmidt has got to divide the whole thing by the length, 518 00:32:38,120 --> 00:32:39,660 so he -- 519 00:32:39,660 --> 00:32:43,280 his formula makes a mess which I'm not willing to write down. 520 00:32:43,280 --> 00:32:48,070 So let's just see that what I saying here? 521 00:32:48,070 --> 00:32:51,670 I'm saying that this vector is perpendicular to A. 522 00:32:51,670 --> 00:32:53,260 That these are orthogonal. 523 00:32:53,260 --> 00:32:56,710 A is perpendicular to B. 524 00:32:56,710 --> 00:32:58,490 Can you check that? 525 00:32:58,490 --> 00:33:01,430 How do you see that yes, of course, we -- 526 00:33:01,430 --> 00:33:04,160 our picture is telling us, yes, we did it right. 527 00:33:04,160 --> 00:33:08,720 How would I check that this matrix is perpendicular to A? 528 00:33:11,390 --> 00:33:16,740 I would multiply by A transpose and I better get zero, right? 529 00:33:16,740 --> 00:33:18,410 I should check that. 530 00:33:18,410 --> 00:33:22,930 A transpose B should come out zero. 531 00:33:22,930 --> 00:33:26,920 So this is A transpose times -- now what did we say B was? 532 00:33:26,920 --> 00:33:30,370 We start with the original B, and we take away 533 00:33:30,370 --> 00:33:38,090 this projection, and that should come out zero. 534 00:33:38,090 --> 00:33:42,950 Well, here we get an A transpose B minus -- 535 00:33:42,950 --> 00:33:46,050 and here is another A transpose B, and the -- 536 00:33:46,050 --> 00:33:50,120 and it's an A transpose A over A transpose A, a one, 537 00:33:50,120 --> 00:33:53,840 those cancel, and we do get zero. 538 00:33:53,840 --> 00:33:54,340 Right. 539 00:33:58,360 --> 00:34:05,940 Now I guess I can do numbers in there. 540 00:34:05,940 --> 00:34:09,420 But I have to take a third vector 541 00:34:09,420 --> 00:34:13,330 to be sure we've got this system down. 542 00:34:13,330 --> 00:34:20,250 So now I have to say if I have independent vectors A, B and C, 543 00:34:20,250 --> 00:34:25,820 I'm looking for orthogonal vectors A, B and capital C, 544 00:34:25,820 --> 00:34:29,170 and then of course the third guy will just 545 00:34:29,170 --> 00:34:32,969 be C over its length, the unit vector. 546 00:34:35,860 --> 00:34:39,980 So this is now the problem. 547 00:34:39,980 --> 00:34:42,929 I got B here. 548 00:34:42,929 --> 00:34:45,889 I got A very easily. 549 00:34:45,889 --> 00:34:52,610 And now -- if you see the idea, we could figure out a formula 550 00:34:52,610 --> 00:35:01,390 for C. So now that -- so this is like a typical homework quiz 551 00:35:01,390 --> 00:35:02,150 problem. 552 00:35:02,150 --> 00:35:07,260 I give you two vectors, you do this, I give you three vectors, 553 00:35:07,260 --> 00:35:10,460 and you have to make them orthonormal. 554 00:35:10,460 --> 00:35:13,520 So you do this again, the first vector's fine, 555 00:35:13,520 --> 00:35:17,160 the second vector is perpendicular to the first, 556 00:35:17,160 --> 00:35:19,720 and now I need a third vector that's 557 00:35:19,720 --> 00:35:23,240 perpendicular to the first one and the second one. 558 00:35:23,240 --> 00:35:25,560 Right? 559 00:35:25,560 --> 00:35:29,660 Tthis is the end of a -- the lecture is to find this guy. 560 00:35:29,660 --> 00:35:33,980 Find this vector -- this vector C, that's perpendicular we n- 561 00:35:33,980 --> 00:35:39,410 at this point we know A and B. 562 00:35:39,410 --> 00:35:44,540 But C, the little c that we're given, is off in some -- 563 00:35:44,540 --> 00:35:48,030 it's got to come out of the blackboard to be independent, 564 00:35:48,030 --> 00:35:52,440 so -- so can I sort of draw off -- off comes a c somewhere. 565 00:35:52,440 --> 00:35:54,240 I don't know, where I going to put the darn 566 00:35:54,240 --> 00:35:55,050 thing? 567 00:35:55,050 --> 00:35:59,450 Maybe I'll put it off, oh, I don't know, 568 00:35:59,450 --> 00:36:02,160 like that somehow, C, little c. 569 00:36:05,610 --> 00:36:08,930 And I already know that perpendicular direction, 570 00:36:08,930 --> 00:36:10,750 that one and that one. 571 00:36:10,750 --> 00:36:13,800 So now what's the idea? 572 00:36:13,800 --> 00:36:17,080 Give me the Graham-Schmidt formula for C. 573 00:36:17,080 --> 00:36:20,700 What is this C here? 574 00:36:20,700 --> 00:36:21,520 Equals what? 575 00:36:27,340 --> 00:36:28,160 What I going to do? 576 00:36:28,160 --> 00:36:31,300 I'll start with the given one. 577 00:36:31,300 --> 00:36:32,730 As before. 578 00:36:32,730 --> 00:36:33,280 Right? 579 00:36:33,280 --> 00:36:36,410 I start with the vector I'm given. 580 00:36:36,410 --> 00:36:38,400 And what do I do with it? 581 00:36:38,400 --> 00:36:42,060 I want to remove out of it, I want to subtract off, 582 00:36:42,060 --> 00:36:46,510 so I'll put a minus sign in, I want to subtract off 583 00:36:46,510 --> 00:36:53,010 its components in the A, capital A and capital B directions. 584 00:36:53,010 --> 00:36:55,850 I just want to get those out of there. 585 00:36:55,850 --> 00:36:57,290 Well, I know how to do that. 586 00:36:57,290 --> 00:36:58,710 I did it with B. 587 00:36:58,710 --> 00:37:02,410 So I'll just -- so let me take away -- 588 00:37:02,410 --> 00:37:03,440 what if I do this? 589 00:37:07,960 --> 00:37:08,850 What have I done? 590 00:37:12,060 --> 00:37:16,520 I've got little c and what have I subtracted from it? 591 00:37:16,520 --> 00:37:22,280 Its component, its projection if you like, in the A direction. 592 00:37:25,010 --> 00:37:30,820 And now I've got to subtract off its component B transpose 593 00:37:30,820 --> 00:37:35,200 C over B transpose B, that multiple of B, 594 00:37:35,200 --> 00:37:37,570 is its component in the B direction. 595 00:37:40,100 --> 00:37:47,480 And that gives me the vector capital C that if anything is 596 00:37:47,480 --> 00:37:48,160 -- 597 00:37:48,160 --> 00:37:54,700 if there's any justice, this C should be perpendicular to A 598 00:37:54,700 --> 00:37:58,750 and it should be perpendicular to B. 599 00:37:58,750 --> 00:38:02,660 And the only thing it's -- hasn't got is unit vector, 600 00:38:02,660 --> 00:38:05,830 so we divide by its length to get that too. 601 00:38:08,660 --> 00:38:10,390 OK. 602 00:38:10,390 --> 00:38:14,940 Let me do an example. 603 00:38:14,940 --> 00:38:16,410 Can I -- 604 00:38:16,410 --> 00:38:20,990 I'll make my life easy, I'll just take two vectors. 605 00:38:20,990 --> 00:38:23,880 So let me do a numerical example. 606 00:38:23,880 --> 00:38:26,480 If I'll give you two vectors, you 607 00:38:26,480 --> 00:38:31,220 give me back the Graham-Schmidt orthonormal basis, 608 00:38:31,220 --> 00:38:35,990 and we'll see how to express that in matrix form. 609 00:38:35,990 --> 00:38:36,530 OK. 610 00:38:36,530 --> 00:38:41,080 So let me give you the two vectors. 611 00:38:41,080 --> 00:38:46,420 So I'll take the vector A equals let's say one, one, one, 612 00:38:46,420 --> 00:38:47,910 why not? 613 00:38:47,910 --> 00:38:55,810 And B equals let's say one, zero, two, OK? 614 00:39:02,400 --> 00:39:05,300 I didn't want to cheat and make them orthogonal 615 00:39:05,300 --> 00:39:07,330 in the first place because then Graham-Schmidt 616 00:39:07,330 --> 00:39:08,430 wouldn't be needed. 617 00:39:08,430 --> 00:39:08,930 OK. 618 00:39:08,930 --> 00:39:10,760 So those are not orthogonal. 619 00:39:10,760 --> 00:39:12,330 So what is capital A? 620 00:39:12,330 --> 00:39:14,280 Well that's the same as big A. 621 00:39:14,280 --> 00:39:15,350 That was fine. 622 00:39:15,350 --> 00:39:18,590 What's B? 623 00:39:18,590 --> 00:39:21,600 So B is this b -- is the original B, 624 00:39:21,600 --> 00:39:29,680 and then I subtract off some multiple of the A. 625 00:39:29,680 --> 00:39:30,900 And what's the multiple? 626 00:39:33,810 --> 00:39:36,520 What goes in here? 627 00:39:36,520 --> 00:39:41,240 B -- here's the A -- this is the -- this is the little b, 628 00:39:41,240 --> 00:39:45,430 this is the big A, also the little a, and I want 629 00:39:45,430 --> 00:39:48,680 to multiply it by that right -- that right ratio, 630 00:39:48,680 --> 00:39:53,480 which has A transpose A, here's my ratio. 631 00:39:53,480 --> 00:39:57,100 I'm just doing this. 632 00:39:57,100 --> 00:40:00,610 So it's A transpose B, what is A transpose B, 633 00:40:00,610 --> 00:40:02,790 it looks like three. 634 00:40:02,790 --> 00:40:06,780 And what is A -- oh, my -- 635 00:40:06,780 --> 00:40:08,470 what's A transpose A? 636 00:40:08,470 --> 00:40:08,970 Three. 637 00:40:11,540 --> 00:40:12,690 I'm sorry. 638 00:40:12,690 --> 00:40:14,990 I didn't know that was going to happen. 639 00:40:14,990 --> 00:40:15,490 OK. 640 00:40:15,490 --> 00:40:16,440 But it happened. 641 00:40:16,440 --> 00:40:18,750 Why should we knock it? 642 00:40:18,750 --> 00:40:19,610 OK. 643 00:40:19,610 --> 00:40:21,430 So do you see it all right? 644 00:40:21,430 --> 00:40:25,040 That's A transpose B, there's A transpose A, that's 645 00:40:25,040 --> 00:40:28,640 the fraction, so I take this away, 646 00:40:28,640 --> 00:40:33,640 and I get one take away one is a zero, zero minus this one 647 00:40:33,640 --> 00:40:39,180 is a minus one, and two minus the one is a one. 648 00:40:39,180 --> 00:40:39,930 OK. 649 00:40:39,930 --> 00:40:42,330 And what's this vector that we finally found? 650 00:40:42,330 --> 00:40:47,020 This is B. 651 00:40:47,020 --> 00:40:48,580 And how do I know it's right? 652 00:40:51,930 --> 00:40:54,690 How do I know I've got a vector I want? 653 00:40:54,690 --> 00:40:57,700 I check that B is perpendicular to -- 654 00:40:57,700 --> 00:40:59,980 that A and B are perpendicular. 655 00:40:59,980 --> 00:41:02,280 That A is perpendicular to B. 656 00:41:02,280 --> 00:41:03,190 Just look at that. 657 00:41:03,190 --> 00:41:06,550 That one -- the dot product of that with that is zero. 658 00:41:06,550 --> 00:41:07,180 OK. 659 00:41:07,180 --> 00:41:10,850 So now what is my q1 and q2? 660 00:41:14,720 --> 00:41:17,700 Why don't I put them in a matrix? 661 00:41:17,700 --> 00:41:18,480 Of course. 662 00:41:18,480 --> 00:41:20,970 Since I'm always putting these -- so the Q, 663 00:41:20,970 --> 00:41:24,260 I'll put the q1 and the q2 in a matrix. 664 00:41:24,260 --> 00:41:25,720 And what are they? 665 00:41:29,270 --> 00:41:32,700 Now when I'm writing q-s I'm supposed 666 00:41:32,700 --> 00:41:34,200 to make things normalized. 667 00:41:34,200 --> 00:41:36,450 I'm supposed to make things unit vectors. 668 00:41:36,450 --> 00:41:39,740 So I'm going to take that A but I'm going to divide it 669 00:41:39,740 --> 00:41:41,650 by square root of three. 670 00:41:46,920 --> 00:41:48,780 And I'm going to take this B but I'm 671 00:41:48,780 --> 00:41:53,370 going to divide it by square root of two 672 00:41:53,370 --> 00:41:57,720 to make it a unit vector, and there is my matrix. 673 00:42:01,000 --> 00:42:05,430 That's my matrix with orthonormal columns coming from 674 00:42:05,430 --> 00:42:08,790 Graham-Schmidt and it sort of it -- 675 00:42:08,790 --> 00:42:14,740 it came from the original one, one, one, one, zero, two, 676 00:42:14,740 --> 00:42:15,240 right? 677 00:42:15,240 --> 00:42:16,620 That was my original guys. 678 00:42:20,880 --> 00:42:23,200 These were the two I started with. 679 00:42:23,200 --> 00:42:25,950 These are the two that I'm happy to end with. 680 00:42:25,950 --> 00:42:30,110 Because those are orthonormal. 681 00:42:30,110 --> 00:42:33,490 So that's what Graham-Schmidt did. 682 00:42:33,490 --> 00:42:38,050 It -- well, tell me about the column spaces of these 683 00:42:38,050 --> 00:42:40,260 matrices. 684 00:42:40,260 --> 00:42:44,180 How is the column space of Q related to the column space of 685 00:42:44,180 --> 00:42:44,900 A? 686 00:42:44,900 --> 00:42:47,150 So I'm always asking you things like this, 687 00:42:47,150 --> 00:42:49,970 and that makes you think, OK, the column space 688 00:42:49,970 --> 00:42:54,550 is all combinations of the columns, it's that plane, 689 00:42:54,550 --> 00:42:55,540 right? 690 00:42:55,540 --> 00:42:58,770 I've got two vectors in three-dimensional space, 691 00:42:58,770 --> 00:43:03,630 their column space is a plane, the column space of this matrix 692 00:43:03,630 --> 00:43:08,200 is a plane, what's the relation between the planes? 693 00:43:08,200 --> 00:43:09,740 Between the two column spaces? 694 00:43:12,750 --> 00:43:15,190 They're one and the same, right? 695 00:43:15,190 --> 00:43:17,760 It's the same column space. 696 00:43:17,760 --> 00:43:23,580 All I'm taking is here this B thing that I computed, 697 00:43:23,580 --> 00:43:30,120 this B thing that I computed is a combination of B and A, 698 00:43:30,120 --> 00:43:34,630 and A was little A, so I'm always working here 699 00:43:34,630 --> 00:43:36,920 with this in the same space. 700 00:43:36,920 --> 00:43:42,450 I'm just like getting ninety-degree angles in there. 701 00:43:42,450 --> 00:43:47,450 Where my original column space had a perfectly good basis, 702 00:43:47,450 --> 00:43:50,470 but it wasn't as good as this basis, 703 00:43:50,470 --> 00:43:53,610 because it wasn't orthonormal. 704 00:43:53,610 --> 00:43:59,560 Now this one is orthonormal, and I have a basis then that -- 705 00:43:59,560 --> 00:44:03,050 so now projections, all the calculations I would ever want 706 00:44:03,050 --> 00:44:09,570 to do are -- are a cinch with this orthonormal basis. 707 00:44:09,570 --> 00:44:12,710 One final point. 708 00:44:12,710 --> 00:44:14,480 One final point in this chapter. 709 00:44:17,200 --> 00:44:21,240 And it's -- just like elimination. 710 00:44:21,240 --> 00:44:23,530 We learned how to do elimination, 711 00:44:23,530 --> 00:44:26,160 we know all the steps, we can do it. 712 00:44:26,160 --> 00:44:34,720 But then I came back to it and said look at it as a matrix 713 00:44:34,720 --> 00:44:40,140 in matrix language and elimination gave me -- 714 00:44:40,140 --> 00:44:42,480 what was elimination in matrix language? 715 00:44:42,480 --> 00:44:44,000 I'll just put it up there. 716 00:44:44,000 --> 00:44:46,240 A was LU. 717 00:44:46,240 --> 00:44:49,720 That was matrix, that was elimination. 718 00:44:49,720 --> 00:44:53,180 Now, I want to do the same for Graham-Schmidt. 719 00:44:53,180 --> 00:44:56,200 Everybody who works in linear algebra 720 00:44:56,200 --> 00:44:58,530 isn't going to write out the columns 721 00:44:58,530 --> 00:45:01,220 are orthogonal, or orthonormal. 722 00:45:01,220 --> 00:45:04,530 And isn't going to write out these formulas. 723 00:45:04,530 --> 00:45:08,820 They're going to write out the connection between the matrix A 724 00:45:08,820 --> 00:45:11,320 and the matrix Q. 725 00:45:11,320 --> 00:45:14,230 And the two matrices have the same column space, 726 00:45:14,230 --> 00:45:17,720 but there's some -- some matrix is taking the -- 727 00:45:17,720 --> 00:45:25,110 and I'm going to call it R, so A equals QR is the magic formula 728 00:45:25,110 --> 00:45:25,610 here. 729 00:45:28,200 --> 00:45:30,260 It's the expression of Graham-Schmidt. 730 00:45:32,860 --> 00:45:38,430 And I'll -- let me just capture that. 731 00:45:38,430 --> 00:45:42,240 So that's the -- my final step then is A equal QR. 732 00:45:42,240 --> 00:45:44,310 Maybe I can squeeze it in here. 733 00:45:47,300 --> 00:45:50,960 So A has columns, let's say a1 and a2. 734 00:45:55,420 --> 00:45:59,150 Let me suppose n is two, just two vectors. 735 00:45:59,150 --> 00:46:00,290 OK. 736 00:46:00,290 --> 00:46:06,300 So that's some combination of q1 and q2. 737 00:46:06,300 --> 00:46:13,170 And times some matrix R. 738 00:46:13,170 --> 00:46:16,330 They have the same column space. 739 00:46:16,330 --> 00:46:20,420 This is just -- this matrix just includes in it whatever these 740 00:46:20,420 --> 00:46:23,430 numbers like three over three and one over square root 741 00:46:23,430 --> 00:46:25,360 of three and one over square root of two, 742 00:46:25,360 --> 00:46:28,400 probably that's what it's got. 743 00:46:28,400 --> 00:46:31,140 One over square root of three, one over square root of two, 744 00:46:31,140 --> 00:46:34,390 something there, but actually it's got a zero there. 745 00:46:37,690 --> 00:46:45,260 So the main point about this A equal QR is this R 746 00:46:45,260 --> 00:46:48,270 turns out to be upper triangular. 747 00:46:48,270 --> 00:46:50,750 It turns out that this zero is upper triangular. 748 00:46:53,510 --> 00:46:56,440 We could see why. 749 00:46:56,440 --> 00:47:00,640 Let me see, I can put in general formulas for what these 750 00:47:00,640 --> 00:47:05,190 This I think in here should be the inner product of a1 751 00:47:05,190 --> 00:47:05,740 with q1. are. 752 00:47:08,600 --> 00:47:12,160 And this one should be the -- 753 00:47:12,160 --> 00:47:16,230 the inner product of a1 with q2. 754 00:47:16,230 --> 00:47:18,970 And that's what I believe is zero. 755 00:47:21,710 --> 00:47:25,110 This will be something here, and this will be something here 756 00:47:25,110 --> 00:47:33,960 with inner -- a1 transpose q2, sorry a2 transpose q1 and a2 757 00:47:33,960 --> 00:47:35,060 transpose q2. 758 00:47:35,060 --> 00:47:37,130 But why is that guy zero? 759 00:47:40,100 --> 00:47:43,900 Why is a1 q2 zero? 760 00:47:43,900 --> 00:47:47,580 That's the key to this being -- this R here being upper 761 00:47:47,580 --> 00:47:49,110 triangular. 762 00:47:49,110 --> 00:47:55,100 You know why a1q2 is zero, because a1 -- 763 00:47:55,100 --> 00:47:57,310 that was my -- 764 00:47:57,310 --> 00:48:00,180 this was really a and b here. 765 00:48:00,180 --> 00:48:02,700 This was really a and b. 766 00:48:02,700 --> 00:48:05,780 So this is a transpose q2. 767 00:48:05,780 --> 00:48:09,380 And the whole point of Graham-Schmidt was that we 768 00:48:09,380 --> 00:48:14,820 constructed these later q-s to be perpendicular to the earlier 769 00:48:14,820 --> 00:48:19,030 vectors, to the earlier -- all the earlier vectors. 770 00:48:19,030 --> 00:48:21,090 So that's why we get a triangular matrix. 771 00:48:23,800 --> 00:48:29,750 The -- result is extremely satisfactory. 772 00:48:32,530 --> 00:48:37,450 That if I have a matrix with independent columns, 773 00:48:37,450 --> 00:48:40,540 the Graham-Schmidt produces a matrix 774 00:48:40,540 --> 00:48:44,780 with orthonormal columns, and the connection between those 775 00:48:44,780 --> 00:48:48,230 is a triangular matrix. 776 00:48:48,230 --> 00:48:51,200 That last point, that the connection is a triangular 777 00:48:51,200 --> 00:48:53,510 matrix, please look in the book, you 778 00:48:53,510 --> 00:48:56,390 have to see that one more time. 779 00:48:56,390 --> 00:48:56,930 OK. 780 00:48:56,930 --> 00:48:59,170 Thanks, that's great.