WEBVTT 0:00:01.301 --> 0:00:05.707 Okay So Welcome to Today's Lecture. 0:00:06.066 --> 0:00:12.592 I'm sorry for the inconvenience. 0:00:12.592 --> 0:00:19.910 Sometimes they are project meetings. 0:00:19.910 --> 0:00:25.843 There will be one other time. 0:00:26.806 --> 0:00:40.863 So what we want to talk today about is want to start with neural approaches to machine 0:00:40.863 --> 0:00:42.964 translation. 0:00:43.123 --> 0:00:51.285 I guess you have heard about other types of neural models for other types of neural language 0:00:51.285 --> 0:00:52.339 processing. 0:00:52.339 --> 0:00:59.887 This was some of the first steps in introducing neal networks to machine translation. 0:01:00.600 --> 0:01:06.203 They are similar to what you know they see in as large language models. 0:01:06.666 --> 0:01:11.764 And today look into what are these neuro-language models? 0:01:11.764 --> 0:01:13.874 What is the difference? 0:01:13.874 --> 0:01:15.983 What is the motivation? 0:01:16.316 --> 0:01:21.445 And first will use them in statistics and machine translation. 0:01:21.445 --> 0:01:28.935 So if you remember how fully like two or three weeks ago we had this likely model where you 0:01:28.935 --> 0:01:31.052 can integrate easily any. 0:01:31.351 --> 0:01:40.967 We just have another model which evaluates how good a system is or how good a fluent language 0:01:40.967 --> 0:01:41.376 is. 0:01:41.376 --> 0:01:53.749 The main advantage compared to the statistical models we saw on Tuesday is: Next week we will 0:01:53.749 --> 0:02:06.496 then go for a neural machine translation where we replace the whole model. 0:02:11.211 --> 0:02:21.078 Just as a remember from Tuesday, we've seen the main challenge in language world was that 0:02:21.078 --> 0:02:25.134 most of the engrams we haven't seen. 0:02:26.946 --> 0:02:33.967 So this was therefore difficult to estimate any probability because you've seen that normally 0:02:33.967 --> 0:02:39.494 if you have not seen the endgram you will assign the probability of zero. 0:02:39.980 --> 0:02:49.420 However, this is not really very good because we don't want to give zero probabilities to 0:02:49.420 --> 0:02:54.979 sentences, which still might be a very good English. 0:02:55.415 --> 0:03:02.167 And then we learned a lot of techniques and that is the main challenging statistical machine 0:03:02.167 --> 0:03:04.490 translate statistical language. 0:03:04.490 --> 0:03:10.661 What's how we can give a good estimate of probability to events that we haven't seen 0:03:10.661 --> 0:03:12.258 smoothing techniques? 0:03:12.258 --> 0:03:15.307 We've seen this interpolation and begoff. 0:03:15.435 --> 0:03:21.637 And they invent or develop very specific techniques. 0:03:21.637 --> 0:03:26.903 To deal with that, however, it might not be. 0:03:28.568 --> 0:03:43.190 And therefore maybe we can do things different, so if we have not seen an gram before in statistical 0:03:43.190 --> 0:03:44.348 models. 0:03:45.225 --> 0:03:51.361 Before and we can only get information from exactly the same words. 0:03:51.411 --> 0:04:06.782 We don't have some on like approximate matching like that, maybe in a sentence that cures similarly. 0:04:06.782 --> 0:04:10.282 So if you have seen a. 0:04:11.191 --> 0:04:17.748 And so you would like to have more something like that where endgrams are represented, more 0:04:17.748 --> 0:04:21.953 in a general space, and we can generalize similar numbers. 0:04:22.262 --> 0:04:29.874 So if you learn something about walk then maybe we can use this knowledge and also apply. 0:04:30.290 --> 0:04:42.596 The same as we have done before, but we can really better model how similar they are and 0:04:42.596 --> 0:04:45.223 transfer to other. 0:04:47.047 --> 0:04:54.236 And we maybe want to do that in a more hierarchical approach that we know okay. 0:04:54.236 --> 0:05:02.773 Some words are similar but like go and walk is somehow similar and I and P and G and therefore 0:05:02.773 --> 0:05:06.996 like maybe if we then merge them in an engram. 0:05:07.387 --> 0:05:15.861 If we learn something about our walk, then it should tell us also something about Hugo. 0:05:15.861 --> 0:05:17.113 He walks or. 0:05:17.197 --> 0:05:27.327 You see that there is some relations which we need to integrate for you. 0:05:27.327 --> 0:05:35.514 We need to add the s, but maybe walks should also be here. 0:05:37.137 --> 0:05:45.149 And luckily there is one really convincing method in doing that: And that is by using 0:05:45.149 --> 0:05:47.231 a neural mechanism. 0:05:47.387 --> 0:05:58.497 That's what we will introduce today so we can use this type of neural networks to try 0:05:58.497 --> 0:06:04.053 to learn this similarity and to learn how. 0:06:04.324 --> 0:06:14.355 And that is one of the main advantages that we have by switching from the standard statistical 0:06:14.355 --> 0:06:15.200 models. 0:06:15.115 --> 0:06:22.830 To learn similarities between words and generalized, and learn what is called hidden representations 0:06:22.830 --> 0:06:29.705 or representations of words, where we can measure similarity in some dimensions of words. 0:06:30.290 --> 0:06:42.384 So we can measure in which way words are similar. 0:06:42.822 --> 0:06:48.902 We had it before and we've seen that words were just easier. 0:06:48.902 --> 0:06:51.991 The only thing we did is like. 0:06:52.192 --> 0:07:02.272 But this energies don't have any meaning, so it wasn't that word is more similar to words. 0:07:02.582 --> 0:07:12.112 So we couldn't learn anything about words in the statistical model and that's a big challenge. 0:07:12.192 --> 0:07:23.063 About words even like in morphology, so going goes is somehow more similar because the person 0:07:23.063 --> 0:07:24.219 singular. 0:07:24.264 --> 0:07:34.924 The basic models we have to now have no idea about that and goes as similar to go than it 0:07:34.924 --> 0:07:37.175 might be to sleep. 0:07:39.919 --> 0:07:44.073 So what we want to do today. 0:07:44.073 --> 0:07:53.096 In order to go to this we will have a short introduction into. 0:07:53.954 --> 0:08:05.984 It very short just to see how we use them here, but that's a good thing, so most of you 0:08:05.984 --> 0:08:08.445 think it will be. 0:08:08.928 --> 0:08:14.078 And then we will first look into a feet forward neural network language models. 0:08:14.454 --> 0:08:23.706 And there we will still have this approximation. 0:08:23.706 --> 0:08:33.902 We have before we are looking only at a fixed window. 0:08:34.154 --> 0:08:35.030 The case. 0:08:35.030 --> 0:08:38.270 However, we have the umbellent here. 0:08:38.270 --> 0:08:43.350 That's why they're already better in order to generalize. 0:08:44.024 --> 0:08:53.169 And then at the end we'll look at language models where we then have the additional advantage. 0:08:53.093 --> 0:09:04.317 Case that we need to have a fixed history, but in theory we can model arbitrary long dependencies. 0:09:04.304 --> 0:09:12.687 And we talked about on Tuesday where it is not clear what type of information it is to. 0:09:16.396 --> 0:09:24.981 So in general molecular networks I normally learn to prove that they perform some tasks. 0:09:25.325 --> 0:09:33.472 We have the structure and we are learning them from samples so that is similar to what 0:09:33.472 --> 0:09:34.971 we have before. 0:09:34.971 --> 0:09:42.275 So now we have the same task here, a language model giving input or forwards. 0:09:42.642 --> 0:09:48.959 And is somewhat originally motivated by human brain. 0:09:48.959 --> 0:10:00.639 However, when you now need to know about artificial neural networks, it's hard to get similarity. 0:10:00.540 --> 0:10:02.889 There seemed to be not that point. 0:10:03.123 --> 0:10:11.014 So what they are mainly doing is summoning multiplication and then one non-linear activation. 0:10:12.692 --> 0:10:16.085 So the basic units are these type of. 0:10:17.937 --> 0:10:29.891 Perceptron basic blocks which we have and this does processing so we have a fixed number 0:10:29.891 --> 0:10:36.070 of input features and that will be important. 0:10:36.096 --> 0:10:39.689 So we have here numbers to xn as input. 0:10:40.060 --> 0:10:53.221 And this makes partly of course language processing difficult. 0:10:54.114 --> 0:10:57.609 So we have to model this time on and then go stand home and model. 0:10:58.198 --> 0:11:02.099 Then we are having weights, which are the parameters and the number of weights exactly 0:11:02.099 --> 0:11:03.668 the same as the number of weights. 0:11:04.164 --> 0:11:06.322 Of input features. 0:11:06.322 --> 0:11:15.068 Sometimes he has his fires in there, and then it's not really an input from. 0:11:15.195 --> 0:11:19.205 And what you then do is multiply. 0:11:19.205 --> 0:11:26.164 Each input resists weight and then you sum it up and then. 0:11:26.606 --> 0:11:34.357 What is then additionally later important is that we have an activation function and 0:11:34.357 --> 0:11:42.473 it's important that this activation function is non linear, so we come to just a linear. 0:11:43.243 --> 0:11:54.088 And later it will be important that this is differentiable because otherwise all the training. 0:11:54.714 --> 0:12:01.907 This model by itself is not very powerful. 0:12:01.907 --> 0:12:10.437 It was originally shown that this is not powerful. 0:12:10.710 --> 0:12:19.463 However, there is a very easy extension, the multi layer perceptual, and then things get 0:12:19.463 --> 0:12:20.939 very powerful. 0:12:21.081 --> 0:12:27.719 The thing is you just connect a lot of these in this layer of structures and we have our 0:12:27.719 --> 0:12:35.029 input layer where we have the inputs and our hidden layer at least one where there is everywhere. 0:12:35.395 --> 0:12:39.817 And then we can combine them all to do that. 0:12:40.260 --> 0:12:48.320 The input layer is of course somewhat given by a problem of dimension. 0:12:48.320 --> 0:13:00.013 The outward layer is also given by your dimension, but the hidden layer is of course a hyperparameter. 0:13:01.621 --> 0:13:08.802 So let's start with the first question, now more language related, and that is how we represent. 0:13:09.149 --> 0:13:23.460 So we've seen here we have the but the question is now how can we put in a word into this? 0:13:26.866 --> 0:13:34.117 Noise: The first thing we're able to be better is by the fact that like you are said,. 0:13:34.314 --> 0:13:43.028 That is not that easy because the continuous vector will come to that. 0:13:43.028 --> 0:13:50.392 So from the neo-network we can directly put in the bedding. 0:13:50.630 --> 0:13:57.277 But if we need to input a word into the needle network, it has to be something which is easily 0:13:57.277 --> 0:13:57.907 defined. 0:13:59.079 --> 0:14:12.492 The one hood encoding, and then we have one out of encoding, so one value is one, and all 0:14:12.492 --> 0:14:15.324 the others is the. 0:14:16.316 --> 0:14:25.936 That means we are always dealing with fixed vocabulary because what said is we cannot. 0:14:26.246 --> 0:14:38.017 So you cannot easily extend your vocabulary because if you mean you would extend your vocabulary. 0:14:39.980 --> 0:14:41.502 That's also motivating. 0:14:41.502 --> 0:14:43.722 We're talked about biperriagoding. 0:14:43.722 --> 0:14:45.434 That's a nice thing there. 0:14:45.434 --> 0:14:47.210 We have a fixed vocabulary. 0:14:48.048 --> 0:14:55.804 The big advantage of this one encoding is that we don't implicitly sum our implement 0:14:55.804 --> 0:15:04.291 similarity between words, but really re-learning because if you first think about this, this 0:15:04.291 --> 0:15:06.938 is a very, very inefficient. 0:15:07.227 --> 0:15:15.889 So you need like to represent end words, you need a dimension of an end dimensional vector. 0:15:16.236 --> 0:15:24.846 Imagine you could do binary encoding so you could represent words as binary vectors. 0:15:24.846 --> 0:15:26.467 Then you would. 0:15:26.806 --> 0:15:31.177 Will be significantly more efficient. 0:15:31.177 --> 0:15:36.813 However, then you have some implicit similarity. 0:15:36.813 --> 0:15:39.113 Some numbers share. 0:15:39.559 --> 0:15:46.958 Would somehow be bad because you would force someone to do this by hand or clear how to 0:15:46.958 --> 0:15:47.631 define. 0:15:48.108 --> 0:15:55.135 So therefore currently this is the most successful approach to just do this one watch. 0:15:55.095 --> 0:15:59.563 Representations, so we take a fixed vocabulary. 0:15:59.563 --> 0:16:06.171 We map each word to the inise, and then we represent a word like this. 0:16:06.171 --> 0:16:13.246 So if home will be one, the representation will be one zero zero zero, and. 0:16:14.514 --> 0:16:30.639 But this dimension here is a vocabulary size and that is quite high, so we are always trying 0:16:30.639 --> 0:16:33.586 to be efficient. 0:16:33.853 --> 0:16:43.792 We are doing then some type of efficiency because typically we are having this next layer. 0:16:44.104 --> 0:16:51.967 It can be still maybe two hundred or five hundred or one thousand neurons, but this is 0:16:51.967 --> 0:16:53.323 significantly. 0:16:53.713 --> 0:17:03.792 You can learn that directly and there we then have similarity between words. 0:17:03.792 --> 0:17:07.458 Then it is that some words. 0:17:07.807 --> 0:17:14.772 But the nice thing is that this is then learned that we are not need to hand define that. 0:17:17.117 --> 0:17:32.742 We'll come later to the explicit architecture of the neural language one, and there we can 0:17:32.742 --> 0:17:35.146 see how it's. 0:17:38.418 --> 0:17:44.857 So we're seeing that the other one or our representation always has the same similarity. 0:17:45.105 --> 0:17:59.142 Then we're having this continuous factor which is a lot smaller dimension and that's important 0:17:59.142 --> 0:18:00.768 for later. 0:18:01.121 --> 0:18:06.989 What we are doing then is learning these representations so that they are best for language. 0:18:07.487 --> 0:18:14.968 So the representations are implicitly training the language for the cards. 0:18:14.968 --> 0:18:19.058 This is the best way for doing language. 0:18:19.479 --> 0:18:32.564 And the nice thing that was found out later is these representations are really good. 0:18:33.153 --> 0:18:39.253 And that is why they are now even called word embeddings by themselves and used for other 0:18:39.253 --> 0:18:39.727 tasks. 0:18:40.360 --> 0:18:49.821 And they are somewhat describing very different things so they can describe and semantic similarities. 0:18:49.789 --> 0:18:58.650 Are looking at the very example of today mass vector space by adding words and doing some 0:18:58.650 --> 0:19:00.618 interesting things. 0:19:00.940 --> 0:19:11.178 So they got really like the first big improvement when switching to neurostaff. 0:19:11.491 --> 0:19:20.456 Are like part of the model, but with more complex representation, but they are the basic 0:19:20.456 --> 0:19:21.261 models. 0:19:23.683 --> 0:19:36.979 In the output layer we are also having one output layer structure and a connection function. 0:19:36.997 --> 0:19:46.525 That is, for language learning we want to predict what is the most common word. 0:19:47.247 --> 0:19:56.453 And that can be done very well with this so called soft back layer, where again the dimension. 0:19:56.376 --> 0:20:02.825 Vocabulary size, so this is a vocabulary size, and again the case neural represents the case 0:20:02.825 --> 0:20:03.310 class. 0:20:03.310 --> 0:20:09.759 So in our case we have again one round representation, someone saying this is a core report. 0:20:10.090 --> 0:20:17.255 Our probability distribution is a probability distribution over all works, so the case entry 0:20:17.255 --> 0:20:21.338 tells us how probable is that the next word is this. 0:20:22.682 --> 0:20:33.885 So we need to have some probability distribution at our output in order to achieve that this 0:20:33.885 --> 0:20:37.017 activation function goes. 0:20:37.197 --> 0:20:46.944 And we can achieve that with a soft max activation we take the input to the form of the value, 0:20:46.944 --> 0:20:47.970 and then. 0:20:48.288 --> 0:20:58.021 So by having this type of activation function we are really getting this type of probability. 0:20:59.019 --> 0:21:15.200 At the beginning was also very challenging because again we have this inefficient representation. 0:21:15.235 --> 0:21:29.799 You can imagine that something over is maybe a bit inefficient with cheap users, but definitely. 0:21:36.316 --> 0:21:44.072 And then for training the models that will be fine, so we have to use architecture now. 0:21:44.264 --> 0:21:48.491 We need to minimize the arrow. 0:21:48.491 --> 0:21:53.264 Are we doing it taking the output? 0:21:53.264 --> 0:21:58.174 We are comparing it to our targets. 0:21:58.298 --> 0:22:03.830 So one important thing is by training them. 0:22:03.830 --> 0:22:07.603 How can we measure the error? 0:22:07.603 --> 0:22:12.758 So what is if we are training the ideas? 0:22:13.033 --> 0:22:15.163 And how well we are measuring. 0:22:15.163 --> 0:22:19.768 It is in natural language processing, typically the cross entropy. 0:22:19.960 --> 0:22:35.575 And that means we are comparing the target with the output. 0:22:35.335 --> 0:22:44.430 It gets optimized and you're seeing that this, of course, makes it again very nice and easy 0:22:44.430 --> 0:22:49.868 because our target is again a one-hour representation. 0:22:50.110 --> 0:23:00.116 So all of these are always zero, and what we are then doing is we are taking the one. 0:23:00.100 --> 0:23:04.615 And we only need to multiply the one with the logarithm here, and that is all the feedback 0:23:04.615 --> 0:23:05.955 signal we are taking here. 0:23:06.946 --> 0:23:13.885 Of course, this is not always influenced by all the others. 0:23:13.885 --> 0:23:17.933 Why is this influenced by all the. 0:23:24.304 --> 0:23:34.382 Have the activation function, which is the current activation divided by some of the others. 0:23:34.354 --> 0:23:45.924 Otherwise it could easily just increase this volume and ignore the others, but if you increase 0:23:45.924 --> 0:23:49.090 one value all the others. 0:23:51.351 --> 0:23:59.912 Then we can do with neometrics one very nice and easy type of training that is done in all 0:23:59.912 --> 0:24:07.721 the neometrics where we are now calculating our error and especially the gradient. 0:24:07.707 --> 0:24:11.640 So in which direction does the error show? 0:24:11.640 --> 0:24:18.682 And then if we want to go to a smaller arrow that's what we want to achieve. 0:24:18.682 --> 0:24:26.638 We are taking the inverse direction of the gradient and thereby trying to minimize our 0:24:26.638 --> 0:24:27.278 error. 0:24:27.287 --> 0:24:31.041 And we have to do that, of course, for all the weights. 0:24:31.041 --> 0:24:36.672 And to calculate the error of all the weights, we won't do the defectvagation here. 0:24:36.672 --> 0:24:41.432 But but what you can do is you can propagate the arrow which measured. 0:24:41.432 --> 0:24:46.393 At the end you can propagate it back its basic mass and basic derivation. 0:24:46.706 --> 0:24:58.854 For each way in your model measure how much you contribute to the error and then change 0:24:58.854 --> 0:25:01.339 it in a way that. 0:25:04.524 --> 0:25:11.625 So to summarize what for at least machine translation on your machine translation should 0:25:11.625 --> 0:25:19.044 remember, you know, to understand on this problem is that this is how a multilayer first the 0:25:19.044 --> 0:25:20.640 problem looks like. 0:25:20.580 --> 0:25:28.251 There are fully two layers and no connections. 0:25:28.108 --> 0:25:29.759 Across layers. 0:25:29.829 --> 0:25:35.153 And what they're doing is always just a waited sum here and then in activation production. 0:25:35.415 --> 0:25:38.792 And in order to train you have this forward and backward pass. 0:25:39.039 --> 0:25:41.384 So We Put in Here. 0:25:41.281 --> 0:25:41.895 Inputs. 0:25:41.895 --> 0:25:45.347 We have some random values at the beginning. 0:25:45.347 --> 0:25:47.418 Then calculate the output. 0:25:47.418 --> 0:25:54.246 We are measuring how our error is propagating the arrow back and then changing our model 0:25:54.246 --> 0:25:57.928 in a way that we hopefully get a smaller arrow. 0:25:57.928 --> 0:25:59.616 And then that is how. 0:26:01.962 --> 0:26:12.893 So before we're coming into our neural networks language models, how can we use this type of 0:26:12.893 --> 0:26:17.595 neural network to do language modeling? 0:26:23.103 --> 0:26:33.157 So how can we use them in natural language processing, especially machine translation? 0:26:33.157 --> 0:26:41.799 The first idea of using them was to estimate: So we have seen that the output can be monitored 0:26:41.799 --> 0:26:42.599 here as well. 0:26:43.603 --> 0:26:50.311 A probability distribution and if we have a full vocabulary we could mainly hear estimating 0:26:50.311 --> 0:26:56.727 how probable each next word is and then use that in our language model fashion as we've 0:26:56.727 --> 0:26:58.112 done it last time. 0:26:58.112 --> 0:27:03.215 We got the probability of a full sentence as a product of individual. 0:27:04.544 --> 0:27:12.820 And: That was done in the ninety seven years and it's very easy to integrate it into this 0:27:12.820 --> 0:27:14.545 lot of the year model. 0:27:14.545 --> 0:27:19.570 So we have said that this is how the locker here model looks like. 0:27:19.570 --> 0:27:25.119 So we are searching the best translation which minimizes each waste time. 0:27:25.125 --> 0:27:26.362 The Future About You. 0:27:26.646 --> 0:27:31.647 We have that with minimum error rate training if you can remember where we search for the 0:27:31.647 --> 0:27:32.147 optimal. 0:27:32.512 --> 0:27:40.422 The language model and many others, and we can just add here a neuromodel, have a knock 0:27:40.422 --> 0:27:41.591 of features. 0:27:41.861 --> 0:27:45.761 So that is quite easy as said. 0:27:45.761 --> 0:27:53.183 That was how statistical machine translation was improved. 0:27:53.183 --> 0:27:57.082 You just add one more feature. 0:27:58.798 --> 0:28:07.631 So how can we model the language modeling with a network? 0:28:07.631 --> 0:28:16.008 So what we have to do is model the probability of the. 0:28:16.656 --> 0:28:25.047 The problem in general in the head is that mostly we haven't seen long sequences. 0:28:25.085 --> 0:28:35.650 Mostly we have to beg off to very short sequences and we are working on this discrete space where 0:28:35.650 --> 0:28:36.944 similarity. 0:28:37.337 --> 0:28:50.163 So the idea is if we have now a real network, we can make words into continuous representation. 0:28:51.091 --> 0:29:00.480 And the structure then looks like this, so this is a basic still feed forward neural network. 0:29:01.361 --> 0:29:10.645 We are doing this at perximation again, so we are not putting in all previous words, but 0:29:10.645 --> 0:29:11.375 it is. 0:29:11.691 --> 0:29:25.856 This is done because we said that in the real network we can have only a fixed type of input. 0:29:25.945 --> 0:29:31.886 You can only do a fixed step and then we'll be doing that exactly in minus one. 0:29:33.593 --> 0:29:39.536 So here you are, for example, three words and three different words. 0:29:39.536 --> 0:29:50.704 One and all the others are: And then we're having the first layer of the neural network, 0:29:50.704 --> 0:29:56.230 which like you learns is word embedding. 0:29:57.437 --> 0:30:04.976 There is one thing which is maybe special compared to the standard neural member. 0:30:05.345 --> 0:30:11.918 So the representation of this word we want to learn first of all position independence. 0:30:11.918 --> 0:30:19.013 So we just want to learn what is the general meaning of the word independent of its neighbors. 0:30:19.299 --> 0:30:26.239 And therefore the representation you get here should be the same as if in the second position. 0:30:27.247 --> 0:30:36.865 The nice thing you can achieve is that this weights which you're using here you're reusing 0:30:36.865 --> 0:30:41.727 here and reusing here so we are forcing them. 0:30:42.322 --> 0:30:48.360 You then learn your word embedding, which is contextual, independent, so it's the same 0:30:48.360 --> 0:30:49.678 for each position. 0:30:49.909 --> 0:31:03.482 So that's the idea that you want to learn the representation first of and you don't want 0:31:03.482 --> 0:31:07.599 to really use the context. 0:31:08.348 --> 0:31:13.797 That of course might have a different meaning depending on where it stands, but we'll learn 0:31:13.797 --> 0:31:14.153 that. 0:31:14.514 --> 0:31:20.386 So first we are learning here representational words, which is just the representation. 0:31:20.760 --> 0:31:32.498 Normally we said in neurons all input neurons here are connected to all here, but we're reducing 0:31:32.498 --> 0:31:37.338 the complexity by saying these neurons. 0:31:37.857 --> 0:31:47.912 Then we have a lot denser representation that is our three word embedded in here, and now 0:31:47.912 --> 0:31:57.408 we are learning this interaction between words, a direction between words not based. 0:31:57.677 --> 0:32:08.051 So we have at least one connected layer here, which takes a three embedding input and then 0:32:08.051 --> 0:32:14.208 learns a new embedding which now represents the full. 0:32:15.535 --> 0:32:16.551 Layers. 0:32:16.551 --> 0:32:27.854 It is the output layer which now and then again the probability distribution of all the. 0:32:28.168 --> 0:32:48.612 So here is your target prediction. 0:32:48.688 --> 0:32:56.361 The nice thing is that you learn everything together, so you don't have to teach them what 0:32:56.361 --> 0:32:58.722 a good word representation. 0:32:59.079 --> 0:33:08.306 Training the whole number together, so it learns what a good representation for a word 0:33:08.306 --> 0:33:13.079 you get in order to perform your final task. 0:33:15.956 --> 0:33:19.190 Yeah, that is the main idea. 0:33:20.660 --> 0:33:32.731 This is now a days often referred to as one way of self supervise learning. 0:33:33.053 --> 0:33:37.120 The output is the next word and the input is the previous word. 0:33:37.377 --> 0:33:46.783 But it's not really that we created labels, but we artificially created a task out of unlabeled. 0:33:46.806 --> 0:33:59.434 We just had pure text, and then we created the telescopes by predicting the next word, 0:33:59.434 --> 0:34:18.797 which is: Say we have like two sentences like go home and the second one is go to prepare. 0:34:18.858 --> 0:34:30.135 And then we have to predict the next series and my questions in the labels for the album. 0:34:31.411 --> 0:34:42.752 We model this as one vector with like probability for possible weights starting again. 0:34:44.044 --> 0:34:57.792 Multiple examples, so then you would twice train one to predict KRT, one to predict home, 0:34:57.792 --> 0:35:02.374 and then of course the easel. 0:35:04.564 --> 0:35:13.568 Is a very good point, so you are not aggregating examples beforehand, but you are taking each. 0:35:19.259 --> 0:35:37.204 So when you do it simultaneously learn the projection layer and the endgram for abilities 0:35:37.204 --> 0:35:39.198 and then. 0:35:39.499 --> 0:35:47.684 And later analyze it that these representations are very powerful. 0:35:47.684 --> 0:35:56.358 The task is just a very important task to model what is the next word. 0:35:56.816 --> 0:35:59.842 Is motivated by nowadays. 0:35:59.842 --> 0:36:10.666 In order to get the meaning of the word you have to look at its companies where the context. 0:36:10.790 --> 0:36:16.048 If you read texts in days of word which you have never seen, you often can still estimate 0:36:16.048 --> 0:36:21.130 the meaning of this word because you do not know how it is used, and this is typically 0:36:21.130 --> 0:36:22.240 used as a city or. 0:36:22.602 --> 0:36:25.865 Just imagine you read a text about some city. 0:36:25.865 --> 0:36:32.037 Even if you've never seen the city before, you often know from the context of how it's 0:36:32.037 --> 0:36:32.463 used. 0:36:34.094 --> 0:36:42.483 So what is now the big advantage of using neural neckworks? 0:36:42.483 --> 0:36:51.851 So just imagine we have to estimate that I bought my first iPhone. 0:36:52.052 --> 0:36:56.608 So you have to monitor the probability of ad hitting them. 0:36:56.608 --> 0:37:00.237 Now imagine iPhone, which you have never seen. 0:37:00.600 --> 0:37:11.588 So all the techniques we had last time at the end, if you haven't seen iPhone you will 0:37:11.588 --> 0:37:14.240 always fall back to. 0:37:15.055 --> 0:37:26.230 You have no idea how to deal that you won't have seen the diagram, the trigram, and all 0:37:26.230 --> 0:37:27.754 the others. 0:37:28.588 --> 0:37:43.441 If you're having this type of model, what does it do if you have my first and then something? 0:37:43.483 --> 0:37:50.270 Maybe this representation is really messed up because it's mainly on a cavalry word. 0:37:50.730 --> 0:37:57.793 However, you have still these two information that two words before was first and therefore. 0:37:58.098 --> 0:38:06.954 So you have a lot of information in order to estimate how good it is. 0:38:06.954 --> 0:38:13.279 There could be more information if you know that. 0:38:13.593 --> 0:38:25.168 So all this type of modeling we can do that we couldn't do beforehand because we always 0:38:25.168 --> 0:38:25.957 have. 0:38:27.027 --> 0:38:40.466 Good point, so typically you would have one token for a vocabulary so that you could, for 0:38:40.466 --> 0:38:45.857 example: All you're doing by parent coding when you have a fixed thing. 0:38:46.226 --> 0:38:49.437 Oh yeah, you have to do something like that that that that's true. 0:38:50.050 --> 0:38:55.420 So yeah, auto vocabulary are by thanking where you don't have other words written. 0:38:55.735 --> 0:39:06.295 But then, of course, you might be getting very long previous things, and your sequence 0:39:06.295 --> 0:39:11.272 length gets very long for unknown words. 0:39:17.357 --> 0:39:20.067 Any more questions to the basic stable. 0:39:23.783 --> 0:39:36.719 For this model, what we then want to continue is looking a bit into how complex or how we 0:39:36.719 --> 0:39:39.162 can make things. 0:39:40.580 --> 0:39:49.477 Because at the beginning there was definitely a major challenge, it's still not that easy, 0:39:49.477 --> 0:39:58.275 and I mean our likeers followed the talk about their environmental fingerprint and so on. 0:39:58.478 --> 0:40:05.700 So this calculation is not really heavy, and if you build systems yourselves you have to 0:40:05.700 --> 0:40:06.187 wait. 0:40:06.466 --> 0:40:14.683 So it's good to know a bit about how complex things are in order to do a good or efficient 0:40:14.683 --> 0:40:15.405 affair. 0:40:15.915 --> 0:40:24.211 So one thing where most of the calculation really happens is if you're doing it in a bad 0:40:24.211 --> 0:40:24.677 way. 0:40:25.185 --> 0:40:33.523 So in generally all these layers we are talking about networks and zones fancy. 0:40:33.523 --> 0:40:46.363 In the end it is: So what you have to do in order to calculate here, for example, these 0:40:46.363 --> 0:40:52.333 activations: So make it simple a bit. 0:40:52.333 --> 0:41:06.636 Let's see where outputs and you just do metric multiplication between your weight matrix and 0:41:06.636 --> 0:41:08.482 your input. 0:41:08.969 --> 0:41:20.992 So that is why computers are so powerful for neural networks because they are very good 0:41:20.992 --> 0:41:22.358 in doing. 0:41:22.782 --> 0:41:28.013 However, for some type for the embedding layer this is really very inefficient. 0:41:28.208 --> 0:41:39.652 So because remember we're having this one art encoding in this input, it's always like 0:41:39.652 --> 0:41:42.940 one and everything else. 0:41:42.940 --> 0:41:47.018 It's zero if we're doing this. 0:41:47.387 --> 0:41:55.552 So therefore you can do at least the forward pass a lot more efficient if you don't really 0:41:55.552 --> 0:42:01.833 do this calculation, but you can select the one color where there is. 0:42:01.833 --> 0:42:07.216 Therefore, you also see this is called your word embedding. 0:42:08.348 --> 0:42:19.542 So the weight matrix of the embedding layer is just that in each color you have the embedding 0:42:19.542 --> 0:42:20.018 of. 0:42:20.580 --> 0:42:30.983 So this is like how your initial weights look like and how you can interpret or understand. 0:42:32.692 --> 0:42:39.509 And this is already relatively important because remember this is a huge dimensional thing. 0:42:39.509 --> 0:42:46.104 So typically here we have the number of words is ten thousand or so, so this is the word 0:42:46.104 --> 0:42:51.365 embeddings metrics, typically the most expensive to calculate metrics. 0:42:51.451 --> 0:42:59.741 Because it's the largest one there, we have ten thousand entries, while for the hours we 0:42:59.741 --> 0:43:00.393 maybe. 0:43:00.660 --> 0:43:03.408 So therefore the addition to a little bit more to make this. 0:43:06.206 --> 0:43:10.538 Then you can go where else the calculations are very difficult. 0:43:10.830 --> 0:43:20.389 So here we then have our network, so we have the word embeddings. 0:43:20.389 --> 0:43:29.514 We have one hidden there, and then you can look how difficult. 0:43:30.270 --> 0:43:38.746 Could save a lot of calculation by not really calculating the selection because that is always. 0:43:40.600 --> 0:43:46.096 The number of calculations you have to do here is so. 0:43:46.096 --> 0:43:51.693 The length of this layer is minus one type projection. 0:43:52.993 --> 0:43:56.321 That is a hint size. 0:43:56.321 --> 0:44:10.268 So the first step of calculation for this metrics modification is how much calculation. 0:44:10.730 --> 0:44:18.806 Then you have to do some activation function and then you have to do again the calculation. 0:44:19.339 --> 0:44:27.994 Here we need the vocabulary size because we need to calculate the probability for each 0:44:27.994 --> 0:44:29.088 next word. 0:44:29.889 --> 0:44:43.155 And if you look at these numbers, so if you have a projector size of and a vocabulary size 0:44:43.155 --> 0:44:53.876 of, you see: And that is why there has been especially at the beginning some ideas how 0:44:53.876 --> 0:44:55.589 we can reduce. 0:44:55.956 --> 0:45:01.942 And if we really need to calculate all of our capabilities, or if we can calculate only 0:45:01.942 --> 0:45:02.350 some. 0:45:02.582 --> 0:45:10.871 And there again the one important thing to think about is for what will use my language 0:45:10.871 --> 0:45:11.342 mom. 0:45:11.342 --> 0:45:19.630 I can use it for generations and that's what we will see next week in an achiever which 0:45:19.630 --> 0:45:22.456 really is guiding the search. 0:45:23.123 --> 0:45:30.899 If it just uses a feature, we do not want to use it for generations, but we want to only 0:45:30.899 --> 0:45:32.559 know how probable. 0:45:32.953 --> 0:45:39.325 There we might not be really interested in all the probabilities, but we already know 0:45:39.325 --> 0:45:46.217 we just want to know the probability of this one word, and then it might be very inefficient 0:45:46.217 --> 0:45:49.403 to really calculate all the probabilities. 0:45:51.231 --> 0:45:52.919 And how can you do that so? 0:45:52.919 --> 0:45:56.296 Initially, for example, the people look into shortness. 0:45:56.756 --> 0:46:02.276 So this calculation at the end is really very expensive. 0:46:02.276 --> 0:46:05.762 So can we make that more efficient. 0:46:05.945 --> 0:46:17.375 And most words occur very rarely, and maybe we don't need anger, and so there we may want 0:46:17.375 --> 0:46:18.645 to focus. 0:46:19.019 --> 0:46:29.437 And so they use the smaller vocabulary, which is maybe. 0:46:29.437 --> 0:46:34.646 This layer is used from to. 0:46:34.646 --> 0:46:37.623 Then you merge. 0:46:37.937 --> 0:46:45.162 So you're taking if the word is in the shortest, so in the two thousand most frequent words. 0:46:45.825 --> 0:46:58.299 Of this short word by some normalization here, and otherwise you take a back of probability 0:46:58.299 --> 0:46:59.655 from the. 0:47:00.020 --> 0:47:04.933 It will not be as good, but the idea is okay. 0:47:04.933 --> 0:47:14.013 Then we don't have to calculate all these probabilities here at the end, but we only 0:47:14.013 --> 0:47:16.042 have to calculate. 0:47:19.599 --> 0:47:32.097 With some type of cost because it means we don't model the probability of the infrequent 0:47:32.097 --> 0:47:39.399 words, and maybe it's even very important to model. 0:47:39.299 --> 0:47:46.671 And one idea is to do what is reported as so so structured out there. 0:47:46.606 --> 0:47:49.571 Network language models you see some years ago. 0:47:49.571 --> 0:47:53.154 People were very creative and giving names to new models. 0:47:53.813 --> 0:48:00.341 And there the idea is that we model the output vocabulary as a clustered treat. 0:48:00.680 --> 0:48:06.919 So you don't need to model all of our bodies directly, but you are putting words into a 0:48:06.919 --> 0:48:08.479 sequence of clusters. 0:48:08.969 --> 0:48:15.019 So maybe a very intriguant world is first in cluster three and then in cluster three. 0:48:15.019 --> 0:48:21.211 You have subclusters again and there is subclusters seven and subclusters and there is. 0:48:21.541 --> 0:48:40.134 And this is the path, so that is what was the man in the past. 0:48:40.340 --> 0:48:52.080 And then you can calculate the probability of the word again just by the product of the 0:48:52.080 --> 0:48:55.548 first class of the world. 0:48:57.617 --> 0:49:07.789 That it may be more clear where you have this architecture, so this is all the same. 0:49:07.789 --> 0:49:13.773 But then you first predict here which main class. 0:49:14.154 --> 0:49:24.226 Then you go to the appropriate subclass, then you calculate the probability of the subclass 0:49:24.226 --> 0:49:26.415 and maybe the cell. 0:49:27.687 --> 0:49:35.419 Anybody have an idea why this is more efficient or if you do it first, it looks a lot more. 0:49:42.242 --> 0:49:51.788 You have to do less calculations, so maybe if you do it here you have to calculate the 0:49:51.788 --> 0:49:59.468 element there, but you don't have to do all the one hundred thousand. 0:49:59.980 --> 0:50:06.115 The probabilities in the set classes that you're going through and not for all of them. 0:50:06.386 --> 0:50:18.067 Therefore, it's more efficient if you don't need all output proficient because you have 0:50:18.067 --> 0:50:21.253 to calculate the class. 0:50:21.501 --> 0:50:28.936 So it's only more efficient and scenarios where you really need to use a language model 0:50:28.936 --> 0:50:30.034 to evaluate. 0:50:35.275 --> 0:50:52.456 How this works was that you can train first in your language one on the short list. 0:50:52.872 --> 0:51:03.547 But on the input layer you have your full vocabulary because at the input we saw that 0:51:03.547 --> 0:51:06.650 this is not complicated. 0:51:06.906 --> 0:51:26.638 And then you can cluster down all your words here into classes and use that as your glasses. 0:51:29.249 --> 0:51:34.148 That is one idea of doing it. 0:51:34.148 --> 0:51:44.928 There is also a second idea of doing it, and again we don't need. 0:51:45.025 --> 0:51:53.401 So sometimes it doesn't really need to be a probability to evaluate. 0:51:53.401 --> 0:51:56.557 It's only important that. 0:51:58.298 --> 0:52:04.908 And: Here it's called self normalization what people have done so. 0:52:04.908 --> 0:52:11.562 We have seen that the probability is in this soft mechanism always to the input divided 0:52:11.562 --> 0:52:18.216 by our normalization, and the normalization is a summary of the vocabulary to the power 0:52:18.216 --> 0:52:19.274 of the spell. 0:52:19.759 --> 0:52:25.194 So this is how we calculate the software. 0:52:25.825 --> 0:52:41.179 In self normalization of the idea, if this would be zero then we don't need to calculate 0:52:41.179 --> 0:52:42.214 that. 0:52:42.102 --> 0:52:54.272 Will be zero, and then you don't even have to calculate the normalization because it's. 0:52:54.514 --> 0:53:08.653 So how can we achieve that and then the nice thing in your networks? 0:53:09.009 --> 0:53:23.928 And now we're just adding a second note with some either permitted here. 0:53:24.084 --> 0:53:29.551 And the second lost just tells us he'll be strained away. 0:53:29.551 --> 0:53:31.625 The locks at is zero. 0:53:32.352 --> 0:53:38.614 So then if it's nearly zero at the end we don't need to calculate this and it's also 0:53:38.614 --> 0:53:39.793 very efficient. 0:53:40.540 --> 0:53:49.498 One important thing is this, of course, is only in inference. 0:53:49.498 --> 0:54:04.700 During tests we don't need to calculate that because: You can do a bit of a hyperparameter 0:54:04.700 --> 0:54:14.851 here where you do the waiting, so how good should it be estimating the probabilities and 0:54:14.851 --> 0:54:16.790 how much effort? 0:54:18.318 --> 0:54:28.577 The only disadvantage is no speed up during training. 0:54:28.577 --> 0:54:43.843 There are other ways of doing that, for example: Englishman is in case you get it. 0:54:44.344 --> 0:54:48.540 Then we are coming very, very briefly like just one idea. 0:54:48.828 --> 0:54:53.058 That there is more things on different types of language models. 0:54:53.058 --> 0:54:58.002 We are having a very short view on restricted person-based language models. 0:54:58.298 --> 0:55:08.931 Talk about recurrent neural networks for language mines because they have the advantage that 0:55:08.931 --> 0:55:17.391 we can even further improve by not having a continuous representation on. 0:55:18.238 --> 0:55:23.845 So there's different types of neural networks. 0:55:23.845 --> 0:55:30.169 These are these boxing machines and the interesting. 0:55:30.330 --> 0:55:39.291 They have these: And they define like an energy function on the network, which can be in restricted 0:55:39.291 --> 0:55:44.372 balsam machines efficiently calculated in general and restricted needs. 0:55:44.372 --> 0:55:51.147 You only have connection between the input and the hidden layer, but you don't have connections 0:55:51.147 --> 0:55:53.123 in the input or within the. 0:55:53.393 --> 0:56:00.194 So you see here you don't have an input output, you just have an input, and you calculate. 0:56:00.460 --> 0:56:15.612 Which of course nicely fits with the idea we're having, so you can then use this for 0:56:15.612 --> 0:56:19.177 an N Gram language. 0:56:19.259 --> 0:56:25.189 Retaining the flexibility of the input by this type of neon networks. 0:56:26.406 --> 0:56:30.589 And the advantage of this type of model was there's. 0:56:30.550 --> 0:56:37.520 Very, very fast to integrate it, so that one was the first one which was used during the 0:56:37.520 --> 0:56:38.616 coding model. 0:56:38.938 --> 0:56:45.454 The engram language models were that they were very good and gave performance. 0:56:45.454 --> 0:56:50.072 However, calculation still with all these tricks takes. 0:56:50.230 --> 0:56:58.214 We have talked about embest lists so they generated an embest list of the most probable 0:56:58.214 --> 0:57:05.836 outputs and then they took this and best list scored each entry with a new network. 0:57:06.146 --> 0:57:09.306 A language model, and then only change the order again. 0:57:09.306 --> 0:57:10.887 Select based on that which. 0:57:11.231 --> 0:57:17.187 The neighboring list is maybe only like hundred entries. 0:57:17.187 --> 0:57:21.786 When decoding you look at several thousand. 0:57:26.186 --> 0:57:35.196 Let's look at the context so we have now seen your language models. 0:57:35.196 --> 0:57:43.676 There is the big advantage we can use this word similarity and. 0:57:44.084 --> 0:57:52.266 Remember for engram language ones is not always minus one words because sometimes you have 0:57:52.266 --> 0:57:59.909 to back off or interpolation to lower engrams and you don't know the previous words. 0:58:00.760 --> 0:58:04.742 And however in neural models we always have all of this importance. 0:58:04.742 --> 0:58:05.504 Can some of. 0:58:07.147 --> 0:58:20.288 The disadvantage is that you are still limited in your context, and if you remember the sentence 0:58:20.288 --> 0:58:22.998 from last lecture,. 0:58:22.882 --> 0:58:28.328 Sometimes you need more context and there is unlimited context that you might need and 0:58:28.328 --> 0:58:34.086 you can always create sentences where you may need this five context in order to put a good 0:58:34.086 --> 0:58:34.837 estimation. 0:58:35.315 --> 0:58:44.956 Can also do it different in order to understand that it makes sense to view language. 0:58:45.445 --> 0:58:59.510 So secret labeling tasks are a very common type of task in language processing where you 0:58:59.510 --> 0:59:03.461 have the input sequence. 0:59:03.323 --> 0:59:05.976 So you have one output for each input. 0:59:05.976 --> 0:59:12.371 Machine translation is not a secret labeling cast because the number of inputs and the number 0:59:12.371 --> 0:59:14.072 of outputs is different. 0:59:14.072 --> 0:59:20.598 So you put in a string German which has five words and the output can be: See, for example, 0:59:20.598 --> 0:59:24.078 you always have the same number and the same number of offices. 0:59:24.944 --> 0:59:39.779 And you can more language waddling as that, and you just say the label for each word is 0:59:39.779 --> 0:59:43.151 always a next word. 0:59:45.705 --> 0:59:50.312 This is the more generous you can think of it. 0:59:50.312 --> 0:59:56.194 For example, Paddle Speech Taking named Entity Recognition. 0:59:58.938 --> 1:00:08.476 And if you look at now, this output token and generally sequenced labeling can depend 1:00:08.476 --> 1:00:26.322 on: The input tokens are the same so we can easily model it and they only depend on the 1:00:26.322 --> 1:00:29.064 input tokens. 1:00:31.011 --> 1:00:42.306 But we can always look at one specific type of sequence labeling, unidirectional sequence 1:00:42.306 --> 1:00:44.189 labeling type. 1:00:44.584 --> 1:01:00.855 The probability of the next word only depends on the previous words that we are having here. 1:01:01.321 --> 1:01:05.998 That's also not completely true in language. 1:01:05.998 --> 1:01:14.418 Well, the back context might also be helpful by direction of the model's Google. 1:01:14.654 --> 1:01:23.039 We will always admire the probability of the word given on its history. 1:01:23.623 --> 1:01:30.562 And currently there is approximation and sequence labeling that we have this windowing approach. 1:01:30.951 --> 1:01:43.016 So in order to predict this type of word we always look at the previous three words. 1:01:43.016 --> 1:01:48.410 This is this type of windowing model. 1:01:49.389 --> 1:01:54.780 If you're into neural networks you recognize this type of structure. 1:01:54.780 --> 1:01:57.515 Also, the typical neural networks. 1:01:58.938 --> 1:02:11.050 Yes, yes, so like engram models you can, at least in some way, prepare for that type of 1:02:11.050 --> 1:02:12.289 context. 1:02:14.334 --> 1:02:23.321 Are also other types of neonamic structures which we can use for sequins lately and which 1:02:23.321 --> 1:02:30.710 might help us where we don't have this type of fixed size representation. 1:02:32.812 --> 1:02:34.678 That we can do so. 1:02:34.678 --> 1:02:39.391 The idea is in recurrent new networks traction. 1:02:39.391 --> 1:02:43.221 We are saving complete history in one. 1:02:43.623 --> 1:02:56.946 So again we have to do this fixed size representation because the neural networks always need a habit. 1:02:57.157 --> 1:03:09.028 And then the network should look like that, so we start with an initial value for our storage. 1:03:09.028 --> 1:03:15.900 We are giving our first input and calculating the new. 1:03:16.196 --> 1:03:35.895 So again in your network with two types of inputs: Then you can apply it to the next type 1:03:35.895 --> 1:03:41.581 of input and you're again having this. 1:03:41.581 --> 1:03:46.391 You're taking this hidden state. 1:03:47.367 --> 1:03:53.306 Nice thing is now that you can do now step by step by step, so all the way over. 1:03:55.495 --> 1:04:06.131 The nice thing we are having here now is that now we are having context information from 1:04:06.131 --> 1:04:07.206 all the. 1:04:07.607 --> 1:04:14.181 So if you're looking like based on which words do you, you calculate the probability of varying. 1:04:14.554 --> 1:04:20.090 It depends on this part. 1:04:20.090 --> 1:04:33.154 It depends on and this hidden state was influenced by two. 1:04:33.473 --> 1:04:38.259 So now we're having something new. 1:04:38.259 --> 1:04:46.463 We can model like the word probability not only on a fixed. 1:04:46.906 --> 1:04:53.565 Because the hidden states we are having here in our Oregon are influenced by all the trivia. 1:04:56.296 --> 1:05:02.578 So how is there to be Singapore? 1:05:02.578 --> 1:05:16.286 But then we have the initial idea about this P of given on the history. 1:05:16.736 --> 1:05:25.300 So do not need to do any clustering here, and you also see how things are put together 1:05:25.300 --> 1:05:26.284 in order. 1:05:29.489 --> 1:05:43.449 The green box this night since we are starting from the left to the right. 1:05:44.524 --> 1:05:51.483 Voices: Yes, that's right, so there are clusters, and here is also sometimes clustering happens. 1:05:51.871 --> 1:05:58.687 The small difference does matter again, so if you have now a lot of different histories, 1:05:58.687 --> 1:06:01.674 the similarity which you have in here. 1:06:01.674 --> 1:06:08.260 If two of the histories are very similar, these representations will be the same, and 1:06:08.260 --> 1:06:10.787 then you're treating them again. 1:06:11.071 --> 1:06:15.789 Because in order to do the final restriction you only do a good base on the green box. 1:06:16.156 --> 1:06:28.541 So you are now still learning some type of clustering in there, but you are learning it 1:06:28.541 --> 1:06:30.230 implicitly. 1:06:30.570 --> 1:06:38.200 The only restriction you're giving is you have to stall everything that is important 1:06:38.200 --> 1:06:39.008 in this. 1:06:39.359 --> 1:06:54.961 So it's a different type of limitation, so you calculate the probability based on the 1:06:54.961 --> 1:06:57.138 last words. 1:06:57.437 --> 1:07:04.430 And that is how you still need to somehow cluster things together in order to do efficiently. 1:07:04.430 --> 1:07:09.563 Of course, you need to do some type of clustering because otherwise. 1:07:09.970 --> 1:07:18.865 But this is where things get merged together in this type of hidden representation. 1:07:18.865 --> 1:07:27.973 So here the probability of the word first only depends on this hidden representation. 1:07:28.288 --> 1:07:33.104 On the previous words, but they are some other bottleneck in order to make a good estimation. 1:07:34.474 --> 1:07:41.231 So the idea is that we can store all our history into or into one lecture. 1:07:41.581 --> 1:07:44.812 Which is the one that makes it more strong. 1:07:44.812 --> 1:07:51.275 Next we come to problems that of course at some point it might be difficult if you have 1:07:51.275 --> 1:07:57.811 very long sequences and you always write all the information you have on this one block. 1:07:58.398 --> 1:08:02.233 Then maybe things get overwritten or you cannot store everything in there. 1:08:02.662 --> 1:08:04.514 So,. 1:08:04.184 --> 1:08:09.569 Therefore, yet for short things like single sentences that works well, but especially if 1:08:09.569 --> 1:08:15.197 you think of other tasks and like symbolizations with our document based on T where you need 1:08:15.197 --> 1:08:20.582 to consider the full document, these things got got a bit more more more complicated and 1:08:20.582 --> 1:08:23.063 will learn another type of architecture. 1:08:24.464 --> 1:08:30.462 In order to understand these neighbors, it is good to have all the bus use always. 1:08:30.710 --> 1:08:33.998 So this is the unrolled view. 1:08:33.998 --> 1:08:43.753 Somewhere you're over the type or in language over the words you're unrolling a network. 1:08:44.024 --> 1:08:52.096 Here is the article and here is the network which is connected by itself and that is recurrent. 1:08:56.176 --> 1:09:04.982 There is one challenge in this networks and training. 1:09:04.982 --> 1:09:11.994 We can train them first of all as forward. 1:09:12.272 --> 1:09:19.397 So we don't really know how to train them, but if you unroll them like this is a feet 1:09:19.397 --> 1:09:20.142 forward. 1:09:20.540 --> 1:09:38.063 Is exactly the same, so you can measure your arrows here and be back to your arrows. 1:09:38.378 --> 1:09:45.646 If you unroll something, it's a feature in your laptop and you can train it the same way. 1:09:46.106 --> 1:09:57.606 The only important thing is again, of course, for different inputs. 1:09:57.837 --> 1:10:05.145 But since parameters are shared, it's somehow a similar point you can train it. 1:10:05.145 --> 1:10:08.800 The training algorithm is very similar. 1:10:10.310 --> 1:10:29.568 One thing which makes things difficult is what is referred to as the vanish ingredient. 1:10:29.809 --> 1:10:32.799 That's a very strong thing in the motivation of using hardness. 1:10:33.593 --> 1:10:44.604 The influence here gets smaller and smaller, and the modems are not really able to monitor. 1:10:44.804 --> 1:10:51.939 Because the gradient gets smaller and smaller, and so the arrow here propagated to this one 1:10:51.939 --> 1:10:58.919 that contributes to the arrow is very small, and therefore you don't do any changes there 1:10:58.919 --> 1:10:59.617 anymore. 1:11:00.020 --> 1:11:06.703 And yeah, that's why standard art men are undifficult or have to pick them at custard. 1:11:07.247 --> 1:11:11.462 So everywhere talking to me about fire and ants nowadays,. 1:11:11.791 --> 1:11:23.333 What we are typically meaning are LSDN's or long short memories. 1:11:23.333 --> 1:11:30.968 You see they are by now quite old already. 1:11:31.171 --> 1:11:39.019 So there was a model in the language model task. 1:11:39.019 --> 1:11:44.784 It's some more storing information. 1:11:44.684 --> 1:11:51.556 Because if you only look at the last words, it's often no longer clear this is a question 1:11:51.556 --> 1:11:52.548 or a normal. 1:11:53.013 --> 1:12:05.318 So there you have these mechanisms with ripgate in order to store things for a longer time 1:12:05.318 --> 1:12:08.563 into your hidden state. 1:12:10.730 --> 1:12:20.162 Here they are used in in in selling quite a lot of works. 1:12:21.541 --> 1:12:29.349 For especially machine translation now, the standard is to do transform base models which 1:12:29.349 --> 1:12:30.477 we'll learn. 1:12:30.690 --> 1:12:38.962 But for example, in architecture we have later one lecture about efficiency. 1:12:38.962 --> 1:12:42.830 So how can we build very efficient? 1:12:42.882 --> 1:12:53.074 And there in the decoder in parts of the networks they are still using. 1:12:53.473 --> 1:12:57.518 So it's not that yeah our hands are of no importance in the body. 1:12:59.239 --> 1:13:08.956 In order to make them strong, there are some more things which are helpful and should be: 1:13:09.309 --> 1:13:19.683 So one thing is there is a nice trick to make this new network stronger and better. 1:13:19.739 --> 1:13:21.523 So of course it doesn't work always. 1:13:21.523 --> 1:13:23.451 They have to have enough training data. 1:13:23.763 --> 1:13:28.959 But in general there's the easiest way of making your models bigger and stronger just 1:13:28.959 --> 1:13:30.590 to increase your pyramids. 1:13:30.630 --> 1:13:43.236 And you've seen that with a large language models they are always bragging about. 1:13:43.903 --> 1:13:56.463 This is one way, so the question is how do you get more parameters? 1:13:56.463 --> 1:14:01.265 There's ways of doing it. 1:14:01.521 --> 1:14:10.029 And the other thing is to make your networks deeper so to have more legs in between. 1:14:11.471 --> 1:14:13.827 And then you can also get to get more calm. 1:14:14.614 --> 1:14:23.340 There's more traveling with this and it's very similar to what we just saw with our hand. 1:14:23.603 --> 1:14:34.253 We have this problem of radiant flow that if it flows so fast like a radiant gets very 1:14:34.253 --> 1:14:35.477 swollen,. 1:14:35.795 --> 1:14:42.704 Exactly the same thing happens in deep LSD ends. 1:14:42.704 --> 1:14:52.293 If you take here the gradient, tell you what is the right or wrong. 1:14:52.612 --> 1:14:56.439 With three layers it's no problem, but if you're going to ten, twenty or hundred layers. 1:14:57.797 --> 1:14:59.698 That's Getting Typically Young. 1:15:00.060 --> 1:15:07.000 Are doing is using what is called decisional connections. 1:15:07.000 --> 1:15:15.855 That's a very helpful idea, which is maybe very surprising that it works. 1:15:15.956 --> 1:15:20.309 And so the idea is that these networks. 1:15:20.320 --> 1:15:29.982 In between should no longer calculate what is a new good representation, but they're more 1:15:29.982 --> 1:15:31.378 calculating. 1:15:31.731 --> 1:15:37.588 Therefore, in the end you're always the output of a layer is added with the input. 1:15:38.318 --> 1:15:48.824 The knife is later if you are doing back propagation with this very fast back propagation. 1:15:49.209 --> 1:16:02.540 Nowadays in very deep architectures, not only on other but always has this residual or highway 1:16:02.540 --> 1:16:04.224 connection. 1:16:04.704 --> 1:16:06.616 Has two advantages. 1:16:06.616 --> 1:16:15.409 On the one hand, these layers don't need to learn a representation, they only need to learn 1:16:15.409 --> 1:16:18.754 what to change the representation. 1:16:22.082 --> 1:16:24.172 Good. 1:16:23.843 --> 1:16:31.768 That much for the new map before, so the last thing now means this. 1:16:31.671 --> 1:16:33.750 Language was are yeah. 1:16:33.750 --> 1:16:41.976 I were used in the molds itself and now were seeing them again, but one thing which at the 1:16:41.976 --> 1:16:53.558 beginning they were reading was very essential was: So people really train part of the language 1:16:53.558 --> 1:16:59.999 models only to get this type of embedding. 1:16:59.999 --> 1:17:04.193 Therefore, we want to look. 1:17:09.229 --> 1:17:15.678 So now some last words to the word embeddings. 1:17:15.678 --> 1:17:27.204 The interesting thing is that word embeddings can be used for very different tasks. 1:17:27.347 --> 1:17:31.329 The knife wing is you can train that on just large amounts of data. 1:17:31.931 --> 1:17:41.569 And then if you have these wooden beddings we have seen that they reduce the parameters. 1:17:41.982 --> 1:17:52.217 So then you can train your small mark to do any other task and therefore you are more efficient. 1:17:52.532 --> 1:17:55.218 These initial word embeddings is important. 1:17:55.218 --> 1:18:00.529 They really depend only on the word itself, so if you look at the two meanings of can, 1:18:00.529 --> 1:18:06.328 the can of beans or I can do that, they will have the same embedding, so some of the embedding 1:18:06.328 --> 1:18:08.709 has to save the ambiguity inside that. 1:18:09.189 --> 1:18:12.486 That cannot be resolved. 1:18:12.486 --> 1:18:24.753 Therefore, if you look at the higher levels in the context, but in the word embedding layers 1:18:24.753 --> 1:18:27.919 that really depends on. 1:18:29.489 --> 1:18:33.757 However, even this one has quite very interesting. 1:18:34.034 --> 1:18:39.558 So that people like to visualize them. 1:18:39.558 --> 1:18:47.208 They're always difficult because if you look at this. 1:18:47.767 --> 1:18:52.879 And drawing your five hundred damage, the vector is still a bit challenging. 1:18:53.113 --> 1:19:12.472 So you cannot directly do that, so people have to do it like they look at some type of. 1:19:13.073 --> 1:19:17.209 And of course then yes some information is getting lost by a bunch of control. 1:19:18.238 --> 1:19:24.802 And you see, for example, this is the most famous and common example, so what you can 1:19:24.802 --> 1:19:31.289 look is you can look at the difference between the main and the female word English. 1:19:31.289 --> 1:19:37.854 This is here in your embedding of king, and this is the embedding of queen, and this. 1:19:38.058 --> 1:19:40.394 You can do that for a very different work. 1:19:40.780 --> 1:19:45.407 And that is where the masks come into, that is what people then look into. 1:19:45.725 --> 1:19:50.995 So what you can now, for example, do is you can calculate the difference between man and 1:19:50.995 --> 1:19:51.410 woman? 1:19:52.232 --> 1:19:55.511 Then you can take the embedding of tea. 1:19:55.511 --> 1:20:02.806 You can add on it the difference between man and woman, and then you can notice what are 1:20:02.806 --> 1:20:04.364 the similar words. 1:20:04.364 --> 1:20:08.954 So you won't, of course, directly hit the correct word. 1:20:08.954 --> 1:20:10.512 It's a continuous. 1:20:10.790 --> 1:20:23.127 But you can look what are the nearest neighbors to this same, and often these words are near 1:20:23.127 --> 1:20:24.056 there. 1:20:24.224 --> 1:20:33.913 So it somehow learns that the difference between these words is always the same. 1:20:34.374 --> 1:20:37.746 You can do that for different things. 1:20:37.746 --> 1:20:41.296 He also imagines that it's not perfect. 1:20:41.296 --> 1:20:49.017 He says the world tends to be swimming and swimming, and with walking and walking you. 1:20:49.469 --> 1:20:51.639 So you can try to use them. 1:20:51.639 --> 1:20:59.001 It's no longer like saying yeah, but the interesting thing is this is completely unsupervised. 1:20:59.001 --> 1:21:03.961 So nobody taught him the principle of their gender in language. 1:21:04.284 --> 1:21:09.910 So it's purely trained on the task of doing the next work prediction. 1:21:10.230 --> 1:21:20.658 And even for really cementing information like the capital, this is the difference between 1:21:20.658 --> 1:21:23.638 the city and the capital. 1:21:23.823 --> 1:21:25.518 Visualization. 1:21:25.518 --> 1:21:33.766 Here we have done the same things of the difference between country and. 1:21:33.853 --> 1:21:41.991 You see it's not perfect, but it's building some kinds of a right direction, so you can't 1:21:41.991 --> 1:21:43.347 even use them. 1:21:43.347 --> 1:21:51.304 For example, for question answering, if you have the difference between them, you apply 1:21:51.304 --> 1:21:53.383 that to a new country. 1:21:54.834 --> 1:22:02.741 So it seems these ones are able to really learn a lot of information and collapse all 1:22:02.741 --> 1:22:04.396 this information. 1:22:05.325 --> 1:22:11.769 At just to do the next word prediction: And that also explains a bit maybe or not explains 1:22:11.769 --> 1:22:19.016 wrong life by motivating why what is the main advantage of this type of neural models that 1:22:19.016 --> 1:22:26.025 we can use this type of hidden representation, transfer them and use them in different. 1:22:28.568 --> 1:22:43.707 So summarize what we did today, so what you should hopefully have with you is for machine 1:22:43.707 --> 1:22:45.893 translation. 1:22:45.805 --> 1:22:49.149 Then how we can do language modern Chinese literature? 1:22:49.449 --> 1:22:55.617 We looked at three different architectures: We looked into the feet forward language mode 1:22:55.617 --> 1:22:59.063 and the one based on Bluetooth machines. 1:22:59.039 --> 1:23:05.366 And finally there are different architectures to do in your networks. 1:23:05.366 --> 1:23:14.404 We have seen feet for your networks and we'll see the next lectures, the last type of architecture. 1:23:15.915 --> 1:23:17.412 Have Any Questions. 1:23:20.680 --> 1:23:27.341 Then thanks a lot, and next on Tuesday we will be again in our order to know how to play.