demo_data/lectures/Lecture-07-16.05.2023/English.vtt

WEBVTT

0:00:01.301 --> 0:00:05.707
Okay So Welcome to Today's Lecture.

0:00:06.066 --> 0:00:12.592
I'm sorry for the inconvenience.

0:00:12.592 --> 0:00:19.910
Sometimes they are project meetings.

0:00:19.910 --> 0:00:25.843
There will be one other time.

0:00:26.806 --> 0:00:40.863
So what we want to talk today about is want
to start with neural approaches to machine

0:00:40.863 --> 0:00:42.964
translation.

0:00:43.123 --> 0:00:51.285
I guess you have heard about other types of
neural models for other types of neural language

0:00:51.285 --> 0:00:52.339
processing.

0:00:52.339 --> 0:00:59.887
This was some of the first steps in introducing
neal networks to machine translation.

0:01:00.600 --> 0:01:06.203
They are similar to what you know they see
in as large language models.

0:01:06.666 --> 0:01:11.764
And today look into what are these neuro-language
models?

0:01:11.764 --> 0:01:13.874
What is the difference?

0:01:13.874 --> 0:01:15.983
What is the motivation?

0:01:16.316 --> 0:01:21.445
And first will use them in statistics and
machine translation.

0:01:21.445 --> 0:01:28.935
So if you remember how fully like two or three
weeks ago we had this likely model where you

0:01:28.935 --> 0:01:31.052
can integrate easily any.

0:01:31.351 --> 0:01:40.967
We just have another model which evaluates
how good a system is or how good a fluent language

0:01:40.967 --> 0:01:41.376
is.

0:01:41.376 --> 0:01:53.749
The main advantage compared to the statistical
models we saw on Tuesday is: Next week we will

0:01:53.749 --> 0:02:06.496
then go for a neural machine translation where
we replace the whole model.

0:02:11.211 --> 0:02:21.078
Just as a remember from Tuesday, we've seen
the main challenge in language world was that

0:02:21.078 --> 0:02:25.134
most of the engrams we haven't seen.

0:02:26.946 --> 0:02:33.967
So this was therefore difficult to estimate
any probability because you've seen that normally

0:02:33.967 --> 0:02:39.494
if you have not seen the endgram you will assign
the probability of zero.

0:02:39.980 --> 0:02:49.420
However, this is not really very good because
we don't want to give zero probabilities to

0:02:49.420 --> 0:02:54.979
sentences, which still might be a very good
English.

0:02:55.415 --> 0:03:02.167
And then we learned a lot of techniques and
that is the main challenging statistical machine

0:03:02.167 --> 0:03:04.490
translate statistical language.

0:03:04.490 --> 0:03:10.661
What's how we can give a good estimate of
probability to events that we haven't seen

0:03:10.661 --> 0:03:12.258
smoothing techniques?

0:03:12.258 --> 0:03:15.307
We've seen this interpolation and begoff.

0:03:15.435 --> 0:03:21.637
And they invent or develop very specific techniques.

0:03:21.637 --> 0:03:26.903
To deal with that, however, it might not be.

0:03:28.568 --> 0:03:43.190
And therefore maybe we can do things different,
so if we have not seen an gram before in statistical

0:03:43.190 --> 0:03:44.348
models.

0:03:45.225 --> 0:03:51.361
Before and we can only get information from
exactly the same words.

0:03:51.411 --> 0:04:06.782
We don't have some on like approximate matching
like that, maybe in a sentence that cures similarly.

0:04:06.782 --> 0:04:10.282
So if you have seen a.

0:04:11.191 --> 0:04:17.748
And so you would like to have more something
like that where endgrams are represented, more

0:04:17.748 --> 0:04:21.953
in a general space, and we can generalize similar
numbers.

0:04:22.262 --> 0:04:29.874
So if you learn something about walk then
maybe we can use this knowledge and also apply.

0:04:30.290 --> 0:04:42.596
The same as we have done before, but we can
really better model how similar they are and

0:04:42.596 --> 0:04:45.223
transfer to other.

0:04:47.047 --> 0:04:54.236
And we maybe want to do that in a more hierarchical
approach that we know okay.

0:04:54.236 --> 0:05:02.773
Some words are similar but like go and walk
is somehow similar and I and P and G and therefore

0:05:02.773 --> 0:05:06.996
like maybe if we then merge them in an engram.

0:05:07.387 --> 0:05:15.861
If we learn something about our walk, then
it should tell us also something about Hugo.

0:05:15.861 --> 0:05:17.113
He walks or.

0:05:17.197 --> 0:05:27.327
You see that there is some relations which
we need to integrate for you.

0:05:27.327 --> 0:05:35.514
We need to add the s, but maybe walks should
also be here.

0:05:37.137 --> 0:05:45.149
And luckily there is one really convincing
method in doing that: And that is by using

0:05:45.149 --> 0:05:47.231
a neural mechanism.

0:05:47.387 --> 0:05:58.497
That's what we will introduce today so we
can use this type of neural networks to try

0:05:58.497 --> 0:06:04.053
to learn this similarity and to learn how.

0:06:04.324 --> 0:06:14.355
And that is one of the main advantages that
we have by switching from the standard statistical

0:06:14.355 --> 0:06:15.200
models.

0:06:15.115 --> 0:06:22.830
To learn similarities between words and generalized,
and learn what is called hidden representations

0:06:22.830 --> 0:06:29.705
or representations of words, where we can measure
similarity in some dimensions of words.

0:06:30.290 --> 0:06:42.384
So we can measure in which way words are similar.

0:06:42.822 --> 0:06:48.902
We had it before and we've seen that words
were just easier.

0:06:48.902 --> 0:06:51.991
The only thing we did is like.

0:06:52.192 --> 0:07:02.272
But this energies don't have any meaning,
so it wasn't that word is more similar to words.

0:07:02.582 --> 0:07:12.112
So we couldn't learn anything about words
in the statistical model and that's a big challenge.

0:07:12.192 --> 0:07:23.063
About words even like in morphology, so going
goes is somehow more similar because the person

0:07:23.063 --> 0:07:24.219
singular.

0:07:24.264 --> 0:07:34.924
The basic models we have to now have no idea
about that and goes as similar to go than it

0:07:34.924 --> 0:07:37.175
might be to sleep.

0:07:39.919 --> 0:07:44.073
So what we want to do today.

0:07:44.073 --> 0:07:53.096
In order to go to this we will have a short
introduction into.

0:07:53.954 --> 0:08:05.984
It very short just to see how we use them
here, but that's a good thing, so most of you

0:08:05.984 --> 0:08:08.445
think it will be.

0:08:08.928 --> 0:08:14.078
And then we will first look into a feet forward
neural network language models.

0:08:14.454 --> 0:08:23.706
And there we will still have this approximation.

0:08:23.706 --> 0:08:33.902
We have before we are looking only at a fixed
window.

0:08:34.154 --> 0:08:35.030
The case.

0:08:35.030 --> 0:08:38.270
However, we have the umbellent here.

0:08:38.270 --> 0:08:43.350
That's why they're already better in order
to generalize.

0:08:44.024 --> 0:08:53.169
And then at the end we'll look at language
models where we then have the additional advantage.

0:08:53.093 --> 0:09:04.317
Case that we need to have a fixed history,
but in theory we can model arbitrary long dependencies.

0:09:04.304 --> 0:09:12.687
And we talked about on Tuesday where it is
not clear what type of information it is to.

0:09:16.396 --> 0:09:24.981
So in general molecular networks I normally
learn to prove that they perform some tasks.

0:09:25.325 --> 0:09:33.472
We have the structure and we are learning
them from samples so that is similar to what

0:09:33.472 --> 0:09:34.971
we have before.

0:09:34.971 --> 0:09:42.275
So now we have the same task here, a language
model giving input or forwards.

0:09:42.642 --> 0:09:48.959
And is somewhat originally motivated by human
brain.

0:09:48.959 --> 0:10:00.639
However, when you now need to know about artificial
neural networks, it's hard to get similarity.

0:10:00.540 --> 0:10:02.889
There seemed to be not that point.

0:10:03.123 --> 0:10:11.014
So what they are mainly doing is summoning
multiplication and then one non-linear activation.

0:10:12.692 --> 0:10:16.085
So the basic units are these type of.

0:10:17.937 --> 0:10:29.891
Perceptron basic blocks which we have and
this does processing so we have a fixed number

0:10:29.891 --> 0:10:36.070
of input features and that will be important.

0:10:36.096 --> 0:10:39.689
So we have here numbers to xn as input.

0:10:40.060 --> 0:10:53.221
And this makes partly of course language processing
difficult.

0:10:54.114 --> 0:10:57.609
So we have to model this time on and then
go stand home and model.

0:10:58.198 --> 0:11:02.099
Then we are having weights, which are the
parameters and the number of weights exactly

0:11:02.099 --> 0:11:03.668
the same as the number of weights.

0:11:04.164 --> 0:11:06.322
Of input features.

0:11:06.322 --> 0:11:15.068
Sometimes he has his fires in there, and then
it's not really an input from.

0:11:15.195 --> 0:11:19.205
And what you then do is multiply.

0:11:19.205 --> 0:11:26.164
Each input resists weight and then you sum
it up and then.

0:11:26.606 --> 0:11:34.357
What is then additionally later important
is that we have an activation function and

0:11:34.357 --> 0:11:42.473
it's important that this activation function
is non linear, so we come to just a linear.

0:11:43.243 --> 0:11:54.088
And later it will be important that this is
differentiable because otherwise all the training.

0:11:54.714 --> 0:12:01.907
This model by itself is not very powerful.

0:12:01.907 --> 0:12:10.437
It was originally shown that this is not powerful.

0:12:10.710 --> 0:12:19.463
However, there is a very easy extension, the
multi layer perceptual, and then things get

0:12:19.463 --> 0:12:20.939
very powerful.

0:12:21.081 --> 0:12:27.719
The thing is you just connect a lot of these
in this layer of structures and we have our

0:12:27.719 --> 0:12:35.029
input layer where we have the inputs and our
hidden layer at least one where there is everywhere.

0:12:35.395 --> 0:12:39.817
And then we can combine them all to do that.

0:12:40.260 --> 0:12:48.320
The input layer is of course somewhat given
by a problem of dimension.

0:12:48.320 --> 0:13:00.013
The outward layer is also given by your dimension,
but the hidden layer is of course a hyperparameter.

0:13:01.621 --> 0:13:08.802
So let's start with the first question, now
more language related, and that is how we represent.

0:13:09.149 --> 0:13:23.460
So we've seen here we have the but the question
is now how can we put in a word into this?

0:13:26.866 --> 0:13:34.117
Noise: The first thing we're able to be better
is by the fact that like you are said,.

0:13:34.314 --> 0:13:43.028
That is not that easy because the continuous
vector will come to that.

0:13:43.028 --> 0:13:50.392
So from the neo-network we can directly put
in the bedding.

0:13:50.630 --> 0:13:57.277
But if we need to input a word into the needle
network, it has to be something which is easily

0:13:57.277 --> 0:13:57.907
defined.

0:13:59.079 --> 0:14:12.492
The one hood encoding, and then we have one
out of encoding, so one value is one, and all

0:14:12.492 --> 0:14:15.324
the others is the.

0:14:16.316 --> 0:14:25.936
That means we are always dealing with fixed
vocabulary because what said is we cannot.

0:14:26.246 --> 0:14:38.017
So you cannot easily extend your vocabulary
because if you mean you would extend your vocabulary.

0:14:39.980 --> 0:14:41.502
That's also motivating.

0:14:41.502 --> 0:14:43.722
We're talked about biperriagoding.

0:14:43.722 --> 0:14:45.434
That's a nice thing there.

0:14:45.434 --> 0:14:47.210
We have a fixed vocabulary.

0:14:48.048 --> 0:14:55.804
The big advantage of this one encoding is
that we don't implicitly sum our implement

0:14:55.804 --> 0:15:04.291
similarity between words, but really re-learning
because if you first think about this, this

0:15:04.291 --> 0:15:06.938
is a very, very inefficient.

0:15:07.227 --> 0:15:15.889
So you need like to represent end words, you
need a dimension of an end dimensional vector.

0:15:16.236 --> 0:15:24.846
Imagine you could do binary encoding so you
could represent words as binary vectors.

0:15:24.846 --> 0:15:26.467
Then you would.

0:15:26.806 --> 0:15:31.177
Will be significantly more efficient.

0:15:31.177 --> 0:15:36.813
However, then you have some implicit similarity.

0:15:36.813 --> 0:15:39.113
Some numbers share.

0:15:39.559 --> 0:15:46.958
Would somehow be bad because you would force
someone to do this by hand or clear how to

0:15:46.958 --> 0:15:47.631
define.

0:15:48.108 --> 0:15:55.135
So therefore currently this is the most successful
approach to just do this one watch.

0:15:55.095 --> 0:15:59.563
Representations, so we take a fixed vocabulary.

0:15:59.563 --> 0:16:06.171
We map each word to the inise, and then we
represent a word like this.

0:16:06.171 --> 0:16:13.246
So if home will be one, the representation
will be one zero zero zero, and.

0:16:14.514 --> 0:16:30.639
But this dimension here is a vocabulary size
and that is quite high, so we are always trying

0:16:30.639 --> 0:16:33.586
to be efficient.

0:16:33.853 --> 0:16:43.792
We are doing then some type of efficiency
because typically we are having this next layer.

0:16:44.104 --> 0:16:51.967
It can be still maybe two hundred or five
hundred or one thousand neurons, but this is

0:16:51.967 --> 0:16:53.323
significantly.

0:16:53.713 --> 0:17:03.792
You can learn that directly and there we then
have similarity between words.

0:17:03.792 --> 0:17:07.458
Then it is that some words.

0:17:07.807 --> 0:17:14.772
But the nice thing is that this is then learned
that we are not need to hand define that.

0:17:17.117 --> 0:17:32.742
We'll come later to the explicit architecture
of the neural language one, and there we can

0:17:32.742 --> 0:17:35.146
see how it's.

0:17:38.418 --> 0:17:44.857
So we're seeing that the other one or our
representation always has the same similarity.

0:17:45.105 --> 0:17:59.142
Then we're having this continuous factor which
is a lot smaller dimension and that's important

0:17:59.142 --> 0:18:00.768
for later.

0:18:01.121 --> 0:18:06.989
What we are doing then is learning these representations
so that they are best for language.

0:18:07.487 --> 0:18:14.968
So the representations are implicitly training
the language for the cards.

0:18:14.968 --> 0:18:19.058
This is the best way for doing language.

0:18:19.479 --> 0:18:32.564
And the nice thing that was found out later
is these representations are really good.

0:18:33.153 --> 0:18:39.253
And that is why they are now even called word
embeddings by themselves and used for other

0:18:39.253 --> 0:18:39.727
tasks.

0:18:40.360 --> 0:18:49.821
And they are somewhat describing very different
things so they can describe and semantic similarities.

0:18:49.789 --> 0:18:58.650
Are looking at the very example of today mass
vector space by adding words and doing some

0:18:58.650 --> 0:19:00.618
interesting things.

0:19:00.940 --> 0:19:11.178
So they got really like the first big improvement
when switching to neurostaff.

0:19:11.491 --> 0:19:20.456
Are like part of the model, but with more
complex representation, but they are the basic

0:19:20.456 --> 0:19:21.261
models.

0:19:23.683 --> 0:19:36.979
In the output layer we are also having one
output layer structure and a connection function.

0:19:36.997 --> 0:19:46.525
That is, for language learning we want to
predict what is the most common word.

0:19:47.247 --> 0:19:56.453
And that can be done very well with this so
called soft back layer, where again the dimension.

0:19:56.376 --> 0:20:02.825
Vocabulary size, so this is a vocabulary size,
and again the case neural represents the case

0:20:02.825 --> 0:20:03.310
class.

0:20:03.310 --> 0:20:09.759
So in our case we have again one round representation,
someone saying this is a core report.

0:20:10.090 --> 0:20:17.255
Our probability distribution is a probability
distribution over all works, so the case entry

0:20:17.255 --> 0:20:21.338
tells us how probable is that the next word
is this.

0:20:22.682 --> 0:20:33.885
So we need to have some probability distribution
at our output in order to achieve that this

0:20:33.885 --> 0:20:37.017
activation function goes.

0:20:37.197 --> 0:20:46.944
And we can achieve that with a soft max activation
we take the input to the form of the value,

0:20:46.944 --> 0:20:47.970
and then.

0:20:48.288 --> 0:20:58.021
So by having this type of activation function
we are really getting this type of probability.

0:20:59.019 --> 0:21:15.200
At the beginning was also very challenging
because again we have this inefficient representation.

0:21:15.235 --> 0:21:29.799
You can imagine that something over is maybe
a bit inefficient with cheap users, but definitely.

0:21:36.316 --> 0:21:44.072
And then for training the models that will
be fine, so we have to use architecture now.

0:21:44.264 --> 0:21:48.491
We need to minimize the arrow.

0:21:48.491 --> 0:21:53.264
Are we doing it taking the output?

0:21:53.264 --> 0:21:58.174
We are comparing it to our targets.

0:21:58.298 --> 0:22:03.830
So one important thing is by training them.

0:22:03.830 --> 0:22:07.603
How can we measure the error?

0:22:07.603 --> 0:22:12.758
So what is if we are training the ideas?

0:22:13.033 --> 0:22:15.163
And how well we are measuring.

0:22:15.163 --> 0:22:19.768
It is in natural language processing, typically
the cross entropy.

0:22:19.960 --> 0:22:35.575
And that means we are comparing the target
with the output.

0:22:35.335 --> 0:22:44.430
It gets optimized and you're seeing that this,
of course, makes it again very nice and easy

0:22:44.430 --> 0:22:49.868
because our target is again a one-hour representation.

0:22:50.110 --> 0:23:00.116
So all of these are always zero, and what
we are then doing is we are taking the one.

0:23:00.100 --> 0:23:04.615
And we only need to multiply the one with
the logarithm here, and that is all the feedback

0:23:04.615 --> 0:23:05.955
signal we are taking here.

0:23:06.946 --> 0:23:13.885
Of course, this is not always influenced by
all the others.

0:23:13.885 --> 0:23:17.933
Why is this influenced by all the.

0:23:24.304 --> 0:23:34.382
Have the activation function, which is the
current activation divided by some of the others.

0:23:34.354 --> 0:23:45.924
Otherwise it could easily just increase this
volume and ignore the others, but if you increase

0:23:45.924 --> 0:23:49.090
one value all the others.

0:23:51.351 --> 0:23:59.912
Then we can do with neometrics one very nice
and easy type of training that is done in all

0:23:59.912 --> 0:24:07.721
the neometrics where we are now calculating
our error and especially the gradient.

0:24:07.707 --> 0:24:11.640
So in which direction does the error show?

0:24:11.640 --> 0:24:18.682
And then if we want to go to a smaller arrow
that's what we want to achieve.

0:24:18.682 --> 0:24:26.638
We are taking the inverse direction of the
gradient and thereby trying to minimize our

0:24:26.638 --> 0:24:27.278
error.

0:24:27.287 --> 0:24:31.041
And we have to do that, of course, for all
the weights.

0:24:31.041 --> 0:24:36.672
And to calculate the error of all the weights,
we won't do the defectvagation here.

0:24:36.672 --> 0:24:41.432
But but what you can do is you can propagate
the arrow which measured.

0:24:41.432 --> 0:24:46.393
At the end you can propagate it back its basic
mass and basic derivation.

0:24:46.706 --> 0:24:58.854
For each way in your model measure how much
you contribute to the error and then change

0:24:58.854 --> 0:25:01.339
it in a way that.

0:25:04.524 --> 0:25:11.625
So to summarize what for at least machine
translation on your machine translation should

0:25:11.625 --> 0:25:19.044
remember, you know, to understand on this problem
is that this is how a multilayer first the

0:25:19.044 --> 0:25:20.640
problem looks like.

0:25:20.580 --> 0:25:28.251
There are fully two layers and no connections.

0:25:28.108 --> 0:25:29.759
Across layers.

0:25:29.829 --> 0:25:35.153
And what they're doing is always just a waited
sum here and then in activation production.

0:25:35.415 --> 0:25:38.792
And in order to train you have this forward
and backward pass.

0:25:39.039 --> 0:25:41.384
So We Put in Here.

0:25:41.281 --> 0:25:41.895
Inputs.

0:25:41.895 --> 0:25:45.347
We have some random values at the beginning.

0:25:45.347 --> 0:25:47.418
Then calculate the output.

0:25:47.418 --> 0:25:54.246
We are measuring how our error is propagating
the arrow back and then changing our model

0:25:54.246 --> 0:25:57.928
in a way that we hopefully get a smaller arrow.

0:25:57.928 --> 0:25:59.616
And then that is how.

0:26:01.962 --> 0:26:12.893
So before we're coming into our neural networks
language models, how can we use this type of

0:26:12.893 --> 0:26:17.595
neural network to do language modeling?

0:26:23.103 --> 0:26:33.157
So how can we use them in natural language
processing, especially machine translation?

0:26:33.157 --> 0:26:41.799
The first idea of using them was to estimate:
So we have seen that the output can be monitored

0:26:41.799 --> 0:26:42.599
here as well.

0:26:43.603 --> 0:26:50.311
A probability distribution and if we have
a full vocabulary we could mainly hear estimating

0:26:50.311 --> 0:26:56.727
how probable each next word is and then use
that in our language model fashion as we've

0:26:56.727 --> 0:26:58.112
done it last time.

0:26:58.112 --> 0:27:03.215
We got the probability of a full sentence
as a product of individual.

0:27:04.544 --> 0:27:12.820
And: That was done in the ninety seven years
and it's very easy to integrate it into this

0:27:12.820 --> 0:27:14.545
lot of the year model.

0:27:14.545 --> 0:27:19.570
So we have said that this is how the locker
here model looks like.

0:27:19.570 --> 0:27:25.119
So we are searching the best translation which
minimizes each waste time.

0:27:25.125 --> 0:27:26.362
The Future About You.

0:27:26.646 --> 0:27:31.647
We have that with minimum error rate training
if you can remember where we search for the

0:27:31.647 --> 0:27:32.147
optimal.

0:27:32.512 --> 0:27:40.422
The language model and many others, and we
can just add here a neuromodel, have a knock

0:27:40.422 --> 0:27:41.591
of features.

0:27:41.861 --> 0:27:45.761
So that is quite easy as said.

0:27:45.761 --> 0:27:53.183
That was how statistical machine translation
was improved.

0:27:53.183 --> 0:27:57.082
You just add one more feature.

0:27:58.798 --> 0:28:07.631
So how can we model the language modeling
with a network?

0:28:07.631 --> 0:28:16.008
So what we have to do is model the probability
of the.

0:28:16.656 --> 0:28:25.047
The problem in general in the head is that
mostly we haven't seen long sequences.

0:28:25.085 --> 0:28:35.650
Mostly we have to beg off to very short sequences
and we are working on this discrete space where

0:28:35.650 --> 0:28:36.944
similarity.

0:28:37.337 --> 0:28:50.163
So the idea is if we have now a real network,
we can make words into continuous representation.

0:28:51.091 --> 0:29:00.480
And the structure then looks like this, so
this is a basic still feed forward neural network.

0:29:01.361 --> 0:29:10.645
We are doing this at perximation again, so
we are not putting in all previous words, but

0:29:10.645 --> 0:29:11.375
it is.

0:29:11.691 --> 0:29:25.856
This is done because we said that in the real
network we can have only a fixed type of input.

0:29:25.945 --> 0:29:31.886
You can only do a fixed step and then we'll
be doing that exactly in minus one.

0:29:33.593 --> 0:29:39.536
So here you are, for example, three words
and three different words.

0:29:39.536 --> 0:29:50.704
One and all the others are: And then we're
having the first layer of the neural network,

0:29:50.704 --> 0:29:56.230
which like you learns is word embedding.

0:29:57.437 --> 0:30:04.976
There is one thing which is maybe special
compared to the standard neural member.

0:30:05.345 --> 0:30:11.918
So the representation of this word we want
to learn first of all position independence.

0:30:11.918 --> 0:30:19.013
So we just want to learn what is the general
meaning of the word independent of its neighbors.

0:30:19.299 --> 0:30:26.239
And therefore the representation you get here
should be the same as if in the second position.

0:30:27.247 --> 0:30:36.865
The nice thing you can achieve is that this
weights which you're using here you're reusing

0:30:36.865 --> 0:30:41.727
here and reusing here so we are forcing them.

0:30:42.322 --> 0:30:48.360
You then learn your word embedding, which
is contextual, independent, so it's the same

0:30:48.360 --> 0:30:49.678
for each position.

0:30:49.909 --> 0:31:03.482
So that's the idea that you want to learn
the representation first of and you don't want

0:31:03.482 --> 0:31:07.599
to really use the context.

0:31:08.348 --> 0:31:13.797
That of course might have a different meaning
depending on where it stands, but we'll learn

0:31:13.797 --> 0:31:14.153
that.

0:31:14.514 --> 0:31:20.386
So first we are learning here representational
words, which is just the representation.

0:31:20.760 --> 0:31:32.498
Normally we said in neurons all input neurons
here are connected to all here, but we're reducing

0:31:32.498 --> 0:31:37.338
the complexity by saying these neurons.

0:31:37.857 --> 0:31:47.912
Then we have a lot denser representation that
is our three word embedded in here, and now

0:31:47.912 --> 0:31:57.408
we are learning this interaction between words,
a direction between words not based.

0:31:57.677 --> 0:32:08.051
So we have at least one connected layer here,
which takes a three embedding input and then

0:32:08.051 --> 0:32:14.208
learns a new embedding which now represents
the full.

0:32:15.535 --> 0:32:16.551
Layers.

0:32:16.551 --> 0:32:27.854
It is the output layer which now and then
again the probability distribution of all the.

0:32:28.168 --> 0:32:48.612
So here is your target prediction.

0:32:48.688 --> 0:32:56.361
The nice thing is that you learn everything
together, so you don't have to teach them what

0:32:56.361 --> 0:32:58.722
a good word representation.

0:32:59.079 --> 0:33:08.306
Training the whole number together, so it
learns what a good representation for a word

0:33:08.306 --> 0:33:13.079
you get in order to perform your final task.

0:33:15.956 --> 0:33:19.190
Yeah, that is the main idea.

0:33:20.660 --> 0:33:32.731
This is now a days often referred to as one
way of self supervise learning.

0:33:33.053 --> 0:33:37.120
The output is the next word and the input
is the previous word.

0:33:37.377 --> 0:33:46.783
But it's not really that we created labels,
but we artificially created a task out of unlabeled.

0:33:46.806 --> 0:33:59.434
We just had pure text, and then we created
the telescopes by predicting the next word,

0:33:59.434 --> 0:34:18.797
which is: Say we have like two sentences like
go home and the second one is go to prepare.

0:34:18.858 --> 0:34:30.135
And then we have to predict the next series
and my questions in the labels for the album.

0:34:31.411 --> 0:34:42.752
We model this as one vector with like probability
for possible weights starting again.

0:34:44.044 --> 0:34:57.792
Multiple examples, so then you would twice
train one to predict KRT, one to predict home,

0:34:57.792 --> 0:35:02.374
and then of course the easel.

0:35:04.564 --> 0:35:13.568
Is a very good point, so you are not aggregating
examples beforehand, but you are taking each.

0:35:19.259 --> 0:35:37.204
So when you do it simultaneously learn the
projection layer and the endgram for abilities

0:35:37.204 --> 0:35:39.198
and then.

0:35:39.499 --> 0:35:47.684
And later analyze it that these representations
are very powerful.

0:35:47.684 --> 0:35:56.358
The task is just a very important task to
model what is the next word.

0:35:56.816 --> 0:35:59.842
Is motivated by nowadays.

0:35:59.842 --> 0:36:10.666
In order to get the meaning of the word you
have to look at its companies where the context.

0:36:10.790 --> 0:36:16.048
If you read texts in days of word which you
have never seen, you often can still estimate

0:36:16.048 --> 0:36:21.130
the meaning of this word because you do not
know how it is used, and this is typically

0:36:21.130 --> 0:36:22.240
used as a city or.

0:36:22.602 --> 0:36:25.865
Just imagine you read a text about some city.

0:36:25.865 --> 0:36:32.037
Even if you've never seen the city before,
you often know from the context of how it's

0:36:32.037 --> 0:36:32.463
used.

0:36:34.094 --> 0:36:42.483
So what is now the big advantage of using
neural neckworks?

0:36:42.483 --> 0:36:51.851
So just imagine we have to estimate that I
bought my first iPhone.

0:36:52.052 --> 0:36:56.608
So you have to monitor the probability of
ad hitting them.

0:36:56.608 --> 0:37:00.237
Now imagine iPhone, which you have never seen.

0:37:00.600 --> 0:37:11.588
So all the techniques we had last time at
the end, if you haven't seen iPhone you will

0:37:11.588 --> 0:37:14.240
always fall back to.

0:37:15.055 --> 0:37:26.230
You have no idea how to deal that you won't
have seen the diagram, the trigram, and all

0:37:26.230 --> 0:37:27.754
the others.

0:37:28.588 --> 0:37:43.441
If you're having this type of model, what
does it do if you have my first and then something?

0:37:43.483 --> 0:37:50.270
Maybe this representation is really messed
up because it's mainly on a cavalry word.

0:37:50.730 --> 0:37:57.793
However, you have still these two information
that two words before was first and therefore.

0:37:58.098 --> 0:38:06.954
So you have a lot of information in order
to estimate how good it is.

0:38:06.954 --> 0:38:13.279
There could be more information if you know
that.

0:38:13.593 --> 0:38:25.168
So all this type of modeling we can do that
we couldn't do beforehand because we always

0:38:25.168 --> 0:38:25.957
have.

0:38:27.027 --> 0:38:40.466
Good point, so typically you would have one
token for a vocabulary so that you could, for

0:38:40.466 --> 0:38:45.857
example: All you're doing by parent coding
when you have a fixed thing.

0:38:46.226 --> 0:38:49.437
Oh yeah, you have to do something like that
that that that's true.

0:38:50.050 --> 0:38:55.420
So yeah, auto vocabulary are by thanking where
you don't have other words written.

0:38:55.735 --> 0:39:06.295
But then, of course, you might be getting
very long previous things, and your sequence

0:39:06.295 --> 0:39:11.272
length gets very long for unknown words.

0:39:17.357 --> 0:39:20.067
Any more questions to the basic stable.

0:39:23.783 --> 0:39:36.719
For this model, what we then want to continue
is looking a bit into how complex or how we

0:39:36.719 --> 0:39:39.162
can make things.

0:39:40.580 --> 0:39:49.477
Because at the beginning there was definitely
a major challenge, it's still not that easy,

0:39:49.477 --> 0:39:58.275
and I mean our likeers followed the talk about
their environmental fingerprint and so on.

0:39:58.478 --> 0:40:05.700
So this calculation is not really heavy, and
if you build systems yourselves you have to

0:40:05.700 --> 0:40:06.187
wait.

0:40:06.466 --> 0:40:14.683
So it's good to know a bit about how complex
things are in order to do a good or efficient

0:40:14.683 --> 0:40:15.405
affair.

0:40:15.915 --> 0:40:24.211
So one thing where most of the calculation
really happens is if you're doing it in a bad

0:40:24.211 --> 0:40:24.677
way.

0:40:25.185 --> 0:40:33.523
So in generally all these layers we are talking
about networks and zones fancy.

0:40:33.523 --> 0:40:46.363
In the end it is: So what you have to do in
order to calculate here, for example, these

0:40:46.363 --> 0:40:52.333
activations: So make it simple a bit.

0:40:52.333 --> 0:41:06.636
Let's see where outputs and you just do metric
multiplication between your weight matrix and

0:41:06.636 --> 0:41:08.482
your input.

0:41:08.969 --> 0:41:20.992
So that is why computers are so powerful for
neural networks because they are very good

0:41:20.992 --> 0:41:22.358
in doing.

0:41:22.782 --> 0:41:28.013
However, for some type for the embedding layer
this is really very inefficient.

0:41:28.208 --> 0:41:39.652
So because remember we're having this one
art encoding in this input, it's always like

0:41:39.652 --> 0:41:42.940
one and everything else.

0:41:42.940 --> 0:41:47.018
It's zero if we're doing this.

0:41:47.387 --> 0:41:55.552
So therefore you can do at least the forward
pass a lot more efficient if you don't really

0:41:55.552 --> 0:42:01.833
do this calculation, but you can select the
one color where there is.

0:42:01.833 --> 0:42:07.216
Therefore, you also see this is called your
word embedding.

0:42:08.348 --> 0:42:19.542
So the weight matrix of the embedding layer
is just that in each color you have the embedding

0:42:19.542 --> 0:42:20.018
of.

0:42:20.580 --> 0:42:30.983
So this is like how your initial weights look
like and how you can interpret or understand.

0:42:32.692 --> 0:42:39.509
And this is already relatively important because
remember this is a huge dimensional thing.

0:42:39.509 --> 0:42:46.104
So typically here we have the number of words
is ten thousand or so, so this is the word

0:42:46.104 --> 0:42:51.365
embeddings metrics, typically the most expensive
to calculate metrics.

0:42:51.451 --> 0:42:59.741
Because it's the largest one there, we have
ten thousand entries, while for the hours we

0:42:59.741 --> 0:43:00.393
maybe.

0:43:00.660 --> 0:43:03.408
So therefore the addition to a little bit
more to make this.

0:43:06.206 --> 0:43:10.538
Then you can go where else the calculations
are very difficult.

0:43:10.830 --> 0:43:20.389
So here we then have our network, so we have
the word embeddings.

0:43:20.389 --> 0:43:29.514
We have one hidden there, and then you can
look how difficult.

0:43:30.270 --> 0:43:38.746
Could save a lot of calculation by not really
calculating the selection because that is always.

0:43:40.600 --> 0:43:46.096
The number of calculations you have to do
here is so.

0:43:46.096 --> 0:43:51.693
The length of this layer is minus one type
projection.

0:43:52.993 --> 0:43:56.321
That is a hint size.

0:43:56.321 --> 0:44:10.268
So the first step of calculation for this
metrics modification is how much calculation.

0:44:10.730 --> 0:44:18.806
Then you have to do some activation function
and then you have to do again the calculation.

0:44:19.339 --> 0:44:27.994
Here we need the vocabulary size because we
need to calculate the probability for each

0:44:27.994 --> 0:44:29.088
next word.

0:44:29.889 --> 0:44:43.155
And if you look at these numbers, so if you
have a projector size of and a vocabulary size

0:44:43.155 --> 0:44:53.876
of, you see: And that is why there has been
especially at the beginning some ideas how

0:44:53.876 --> 0:44:55.589
we can reduce.

0:44:55.956 --> 0:45:01.942
And if we really need to calculate all of
our capabilities, or if we can calculate only

0:45:01.942 --> 0:45:02.350
some.

0:45:02.582 --> 0:45:10.871
And there again the one important thing to
think about is for what will use my language

0:45:10.871 --> 0:45:11.342
mom.

0:45:11.342 --> 0:45:19.630
I can use it for generations and that's what
we will see next week in an achiever which

0:45:19.630 --> 0:45:22.456
really is guiding the search.

0:45:23.123 --> 0:45:30.899
If it just uses a feature, we do not want
to use it for generations, but we want to only

0:45:30.899 --> 0:45:32.559
know how probable.

0:45:32.953 --> 0:45:39.325
There we might not be really interested in
all the probabilities, but we already know

0:45:39.325 --> 0:45:46.217
we just want to know the probability of this
one word, and then it might be very inefficient

0:45:46.217 --> 0:45:49.403
to really calculate all the probabilities.

0:45:51.231 --> 0:45:52.919
And how can you do that so?

0:45:52.919 --> 0:45:56.296
Initially, for example, the people look into
shortness.

0:45:56.756 --> 0:46:02.276
So this calculation at the end is really very
expensive.

0:46:02.276 --> 0:46:05.762
So can we make that more efficient.

0:46:05.945 --> 0:46:17.375
And most words occur very rarely, and maybe
we don't need anger, and so there we may want

0:46:17.375 --> 0:46:18.645
to focus.

0:46:19.019 --> 0:46:29.437
And so they use the smaller vocabulary, which
is maybe.

0:46:29.437 --> 0:46:34.646
This layer is used from to.

0:46:34.646 --> 0:46:37.623
Then you merge.

0:46:37.937 --> 0:46:45.162
So you're taking if the word is in the shortest,
so in the two thousand most frequent words.

0:46:45.825 --> 0:46:58.299
Of this short word by some normalization here,
and otherwise you take a back of probability

0:46:58.299 --> 0:46:59.655
from the.

0:47:00.020 --> 0:47:04.933
It will not be as good, but the idea is okay.

0:47:04.933 --> 0:47:14.013
Then we don't have to calculate all these
probabilities here at the end, but we only

0:47:14.013 --> 0:47:16.042
have to calculate.

0:47:19.599 --> 0:47:32.097
With some type of cost because it means we
don't model the probability of the infrequent

0:47:32.097 --> 0:47:39.399
words, and maybe it's even very important to
model.

0:47:39.299 --> 0:47:46.671
And one idea is to do what is reported as
so so structured out there.

0:47:46.606 --> 0:47:49.571
Network language models you see some years
ago.

0:47:49.571 --> 0:47:53.154
People were very creative and giving names
to new models.

0:47:53.813 --> 0:48:00.341
And there the idea is that we model the output
vocabulary as a clustered treat.

0:48:00.680 --> 0:48:06.919
So you don't need to model all of our bodies
directly, but you are putting words into a

0:48:06.919 --> 0:48:08.479
sequence of clusters.

0:48:08.969 --> 0:48:15.019
So maybe a very intriguant world is first
in cluster three and then in cluster three.

0:48:15.019 --> 0:48:21.211
You have subclusters again and there is subclusters
seven and subclusters and there is.

0:48:21.541 --> 0:48:40.134
And this is the path, so that is what was
the man in the past.

0:48:40.340 --> 0:48:52.080
And then you can calculate the probability
of the word again just by the product of the

0:48:52.080 --> 0:48:55.548
first class of the world.

0:48:57.617 --> 0:49:07.789
That it may be more clear where you have this
architecture, so this is all the same.

0:49:07.789 --> 0:49:13.773
But then you first predict here which main
class.

0:49:14.154 --> 0:49:24.226
Then you go to the appropriate subclass, then
you calculate the probability of the subclass

0:49:24.226 --> 0:49:26.415
and maybe the cell.

0:49:27.687 --> 0:49:35.419
Anybody have an idea why this is more efficient
or if you do it first, it looks a lot more.

0:49:42.242 --> 0:49:51.788
You have to do less calculations, so maybe
if you do it here you have to calculate the

0:49:51.788 --> 0:49:59.468
element there, but you don't have to do all
the one hundred thousand.

0:49:59.980 --> 0:50:06.115
The probabilities in the set classes that
you're going through and not for all of them.

0:50:06.386 --> 0:50:18.067
Therefore, it's more efficient if you don't
need all output proficient because you have

0:50:18.067 --> 0:50:21.253
to calculate the class.

0:50:21.501 --> 0:50:28.936
So it's only more efficient and scenarios
where you really need to use a language model

0:50:28.936 --> 0:50:30.034
to evaluate.

0:50:35.275 --> 0:50:52.456
How this works was that you can train first
in your language one on the short list.

0:50:52.872 --> 0:51:03.547
But on the input layer you have your full
vocabulary because at the input we saw that

0:51:03.547 --> 0:51:06.650
this is not complicated.

0:51:06.906 --> 0:51:26.638
And then you can cluster down all your words
here into classes and use that as your glasses.

0:51:29.249 --> 0:51:34.148
That is one idea of doing it.

0:51:34.148 --> 0:51:44.928
There is also a second idea of doing it, and
again we don't need.

0:51:45.025 --> 0:51:53.401
So sometimes it doesn't really need to be
a probability to evaluate.

0:51:53.401 --> 0:51:56.557
It's only important that.

0:51:58.298 --> 0:52:04.908
And: Here it's called self normalization what
people have done so.

0:52:04.908 --> 0:52:11.562
We have seen that the probability is in this
soft mechanism always to the input divided

0:52:11.562 --> 0:52:18.216
by our normalization, and the normalization
is a summary of the vocabulary to the power

0:52:18.216 --> 0:52:19.274
of the spell.

0:52:19.759 --> 0:52:25.194
So this is how we calculate the software.

0:52:25.825 --> 0:52:41.179
In self normalization of the idea, if this
would be zero then we don't need to calculate

0:52:41.179 --> 0:52:42.214
that.

0:52:42.102 --> 0:52:54.272
Will be zero, and then you don't even have
to calculate the normalization because it's.

0:52:54.514 --> 0:53:08.653
So how can we achieve that and then the nice
thing in your networks?

0:53:09.009 --> 0:53:23.928
And now we're just adding a second note with
some either permitted here.

0:53:24.084 --> 0:53:29.551
And the second lost just tells us he'll be
strained away.

0:53:29.551 --> 0:53:31.625
The locks at is zero.

0:53:32.352 --> 0:53:38.614
So then if it's nearly zero at the end we
don't need to calculate this and it's also

0:53:38.614 --> 0:53:39.793
very efficient.

0:53:40.540 --> 0:53:49.498
One important thing is this, of course, is
only in inference.

0:53:49.498 --> 0:54:04.700
During tests we don't need to calculate that
because: You can do a bit of a hyperparameter

0:54:04.700 --> 0:54:14.851
here where you do the waiting, so how good
should it be estimating the probabilities and

0:54:14.851 --> 0:54:16.790
how much effort?

0:54:18.318 --> 0:54:28.577
The only disadvantage is no speed up during
training.

0:54:28.577 --> 0:54:43.843
There are other ways of doing that, for example:
Englishman is in case you get it.

0:54:44.344 --> 0:54:48.540
Then we are coming very, very briefly like
just one idea.

0:54:48.828 --> 0:54:53.058
That there is more things on different types
of language models.

0:54:53.058 --> 0:54:58.002
We are having a very short view on restricted
person-based language models.

0:54:58.298 --> 0:55:08.931
Talk about recurrent neural networks for language
mines because they have the advantage that

0:55:08.931 --> 0:55:17.391
we can even further improve by not having a
continuous representation on.

0:55:18.238 --> 0:55:23.845
So there's different types of neural networks.

0:55:23.845 --> 0:55:30.169
These are these boxing machines and the interesting.

0:55:30.330 --> 0:55:39.291
They have these: And they define like an energy
function on the network, which can be in restricted

0:55:39.291 --> 0:55:44.372
balsam machines efficiently calculated in general
and restricted needs.

0:55:44.372 --> 0:55:51.147
You only have connection between the input
and the hidden layer, but you don't have connections

0:55:51.147 --> 0:55:53.123
in the input or within the.

0:55:53.393 --> 0:56:00.194
So you see here you don't have an input output,
you just have an input, and you calculate.

0:56:00.460 --> 0:56:15.612
Which of course nicely fits with the idea
we're having, so you can then use this for

0:56:15.612 --> 0:56:19.177
an N Gram language.

0:56:19.259 --> 0:56:25.189
Retaining the flexibility of the input by
this type of neon networks.

0:56:26.406 --> 0:56:30.589
And the advantage of this type of model was
there's.

0:56:30.550 --> 0:56:37.520
Very, very fast to integrate it, so that one
was the first one which was used during the

0:56:37.520 --> 0:56:38.616
coding model.

0:56:38.938 --> 0:56:45.454
The engram language models were that they
were very good and gave performance.

0:56:45.454 --> 0:56:50.072
However, calculation still with all these
tricks takes.

0:56:50.230 --> 0:56:58.214
We have talked about embest lists so they
generated an embest list of the most probable

0:56:58.214 --> 0:57:05.836
outputs and then they took this and best list
scored each entry with a new network.

0:57:06.146 --> 0:57:09.306
A language model, and then only change the
order again.

0:57:09.306 --> 0:57:10.887
Select based on that which.

0:57:11.231 --> 0:57:17.187
The neighboring list is maybe only like hundred
entries.

0:57:17.187 --> 0:57:21.786
When decoding you look at several thousand.

0:57:26.186 --> 0:57:35.196
Let's look at the context so we have now seen
your language models.

0:57:35.196 --> 0:57:43.676
There is the big advantage we can use this
word similarity and.

0:57:44.084 --> 0:57:52.266
Remember for engram language ones is not always
minus one words because sometimes you have

0:57:52.266 --> 0:57:59.909
to back off or interpolation to lower engrams
and you don't know the previous words.

0:58:00.760 --> 0:58:04.742
And however in neural models we always have
all of this importance.

0:58:04.742 --> 0:58:05.504
Can some of.

0:58:07.147 --> 0:58:20.288
The disadvantage is that you are still limited
in your context, and if you remember the sentence

0:58:20.288 --> 0:58:22.998
from last lecture,.

0:58:22.882 --> 0:58:28.328
Sometimes you need more context and there
is unlimited context that you might need and

0:58:28.328 --> 0:58:34.086
you can always create sentences where you may
need this five context in order to put a good

0:58:34.086 --> 0:58:34.837
estimation.

0:58:35.315 --> 0:58:44.956
Can also do it different in order to understand
that it makes sense to view language.

0:58:45.445 --> 0:58:59.510
So secret labeling tasks are a very common
type of task in language processing where you

0:58:59.510 --> 0:59:03.461
have the input sequence.

0:59:03.323 --> 0:59:05.976
So you have one output for each input.

0:59:05.976 --> 0:59:12.371
Machine translation is not a secret labeling
cast because the number of inputs and the number

0:59:12.371 --> 0:59:14.072
of outputs is different.

0:59:14.072 --> 0:59:20.598
So you put in a string German which has five
words and the output can be: See, for example,

0:59:20.598 --> 0:59:24.078
you always have the same number and the same
number of offices.

0:59:24.944 --> 0:59:39.779
And you can more language waddling as that,
and you just say the label for each word is

0:59:39.779 --> 0:59:43.151
always a next word.

0:59:45.705 --> 0:59:50.312
This is the more generous you can think of
it.

0:59:50.312 --> 0:59:56.194
For example, Paddle Speech Taking named Entity
Recognition.

0:59:58.938 --> 1:00:08.476
And if you look at now, this output token
and generally sequenced labeling can depend

1:00:08.476 --> 1:00:26.322
on: The input tokens are the same so we can
easily model it and they only depend on the

1:00:26.322 --> 1:00:29.064
input tokens.

1:00:31.011 --> 1:00:42.306
But we can always look at one specific type
of sequence labeling, unidirectional sequence

1:00:42.306 --> 1:00:44.189
labeling type.

1:00:44.584 --> 1:01:00.855
The probability of the next word only depends
on the previous words that we are having here.

1:01:01.321 --> 1:01:05.998
That's also not completely true in language.

1:01:05.998 --> 1:01:14.418
Well, the back context might also be helpful
by direction of the model's Google.

1:01:14.654 --> 1:01:23.039
We will always admire the probability of the
word given on its history.

1:01:23.623 --> 1:01:30.562
And currently there is approximation and sequence
labeling that we have this windowing approach.

1:01:30.951 --> 1:01:43.016
So in order to predict this type of word we
always look at the previous three words.

1:01:43.016 --> 1:01:48.410
This is this type of windowing model.

1:01:49.389 --> 1:01:54.780
If you're into neural networks you recognize
this type of structure.

1:01:54.780 --> 1:01:57.515
Also, the typical neural networks.

1:01:58.938 --> 1:02:11.050
Yes, yes, so like engram models you can, at
least in some way, prepare for that type of

1:02:11.050 --> 1:02:12.289
context.

1:02:14.334 --> 1:02:23.321
Are also other types of neonamic structures
which we can use for sequins lately and which

1:02:23.321 --> 1:02:30.710
might help us where we don't have this type
of fixed size representation.

1:02:32.812 --> 1:02:34.678
That we can do so.

1:02:34.678 --> 1:02:39.391
The idea is in recurrent new networks traction.

1:02:39.391 --> 1:02:43.221
We are saving complete history in one.

1:02:43.623 --> 1:02:56.946
So again we have to do this fixed size representation
because the neural networks always need a habit.

1:02:57.157 --> 1:03:09.028
And then the network should look like that,
so we start with an initial value for our storage.

1:03:09.028 --> 1:03:15.900
We are giving our first input and calculating
the new.

1:03:16.196 --> 1:03:35.895
So again in your network with two types of
inputs: Then you can apply it to the next type

1:03:35.895 --> 1:03:41.581
of input and you're again having this.

1:03:41.581 --> 1:03:46.391
You're taking this hidden state.

1:03:47.367 --> 1:03:53.306
Nice thing is now that you can do now step
by step by step, so all the way over.

1:03:55.495 --> 1:04:06.131
The nice thing we are having here now is that
now we are having context information from

1:04:06.131 --> 1:04:07.206
all the.

1:04:07.607 --> 1:04:14.181
So if you're looking like based on which words
do you, you calculate the probability of varying.

1:04:14.554 --> 1:04:20.090
It depends on this part.

1:04:20.090 --> 1:04:33.154
It depends on and this hidden state was influenced
by two.

1:04:33.473 --> 1:04:38.259
So now we're having something new.

1:04:38.259 --> 1:04:46.463
We can model like the word probability not
only on a fixed.

1:04:46.906 --> 1:04:53.565
Because the hidden states we are having here
in our Oregon are influenced by all the trivia.

1:04:56.296 --> 1:05:02.578
So how is there to be Singapore?

1:05:02.578 --> 1:05:16.286
But then we have the initial idea about this
P of given on the history.

1:05:16.736 --> 1:05:25.300
So do not need to do any clustering here,
and you also see how things are put together

1:05:25.300 --> 1:05:26.284
in order.

1:05:29.489 --> 1:05:43.449
The green box this night since we are starting
from the left to the right.

1:05:44.524 --> 1:05:51.483
Voices: Yes, that's right, so there are clusters,
and here is also sometimes clustering happens.

1:05:51.871 --> 1:05:58.687
The small difference does matter again, so
if you have now a lot of different histories,

1:05:58.687 --> 1:06:01.674
the similarity which you have in here.

1:06:01.674 --> 1:06:08.260
If two of the histories are very similar,
these representations will be the same, and

1:06:08.260 --> 1:06:10.787
then you're treating them again.

1:06:11.071 --> 1:06:15.789
Because in order to do the final restriction
you only do a good base on the green box.

1:06:16.156 --> 1:06:28.541
So you are now still learning some type of
clustering in there, but you are learning it

1:06:28.541 --> 1:06:30.230
implicitly.

1:06:30.570 --> 1:06:38.200
The only restriction you're giving is you
have to stall everything that is important

1:06:38.200 --> 1:06:39.008
in this.

1:06:39.359 --> 1:06:54.961
So it's a different type of limitation, so
you calculate the probability based on the

1:06:54.961 --> 1:06:57.138
last words.

1:06:57.437 --> 1:07:04.430
And that is how you still need to somehow
cluster things together in order to do efficiently.

1:07:04.430 --> 1:07:09.563
Of course, you need to do some type of clustering
because otherwise.

1:07:09.970 --> 1:07:18.865
But this is where things get merged together
in this type of hidden representation.

1:07:18.865 --> 1:07:27.973
So here the probability of the word first
only depends on this hidden representation.

1:07:28.288 --> 1:07:33.104
On the previous words, but they are some other
bottleneck in order to make a good estimation.

1:07:34.474 --> 1:07:41.231
So the idea is that we can store all our history
into or into one lecture.

1:07:41.581 --> 1:07:44.812
Which is the one that makes it more strong.

1:07:44.812 --> 1:07:51.275
Next we come to problems that of course at
some point it might be difficult if you have

1:07:51.275 --> 1:07:57.811
very long sequences and you always write all
the information you have on this one block.

1:07:58.398 --> 1:08:02.233
Then maybe things get overwritten or you cannot
store everything in there.

1:08:02.662 --> 1:08:04.514
So,.

1:08:04.184 --> 1:08:09.569
Therefore, yet for short things like single
sentences that works well, but especially if

1:08:09.569 --> 1:08:15.197
you think of other tasks and like symbolizations
with our document based on T where you need

1:08:15.197 --> 1:08:20.582
to consider the full document, these things
got got a bit more more more complicated and

1:08:20.582 --> 1:08:23.063
will learn another type of architecture.

1:08:24.464 --> 1:08:30.462
In order to understand these neighbors, it
is good to have all the bus use always.

1:08:30.710 --> 1:08:33.998
So this is the unrolled view.

1:08:33.998 --> 1:08:43.753
Somewhere you're over the type or in language
over the words you're unrolling a network.

1:08:44.024 --> 1:08:52.096
Here is the article and here is the network
which is connected by itself and that is recurrent.

1:08:56.176 --> 1:09:04.982
There is one challenge in this networks and
training.

1:09:04.982 --> 1:09:11.994
We can train them first of all as forward.

1:09:12.272 --> 1:09:19.397
So we don't really know how to train them,
but if you unroll them like this is a feet

1:09:19.397 --> 1:09:20.142
forward.

1:09:20.540 --> 1:09:38.063
Is exactly the same, so you can measure your
arrows here and be back to your arrows.

1:09:38.378 --> 1:09:45.646
If you unroll something, it's a feature in
your laptop and you can train it the same way.

1:09:46.106 --> 1:09:57.606
The only important thing is again, of course,
for different inputs.

1:09:57.837 --> 1:10:05.145
But since parameters are shared, it's somehow
a similar point you can train it.

1:10:05.145 --> 1:10:08.800
The training algorithm is very similar.

1:10:10.310 --> 1:10:29.568
One thing which makes things difficult is
what is referred to as the vanish ingredient.

1:10:29.809 --> 1:10:32.799
That's a very strong thing in the motivation
of using hardness.

1:10:33.593 --> 1:10:44.604
The influence here gets smaller and smaller,
and the modems are not really able to monitor.

1:10:44.804 --> 1:10:51.939
Because the gradient gets smaller and smaller,
and so the arrow here propagated to this one

1:10:51.939 --> 1:10:58.919
that contributes to the arrow is very small,
and therefore you don't do any changes there

1:10:58.919 --> 1:10:59.617
anymore.

1:11:00.020 --> 1:11:06.703
And yeah, that's why standard art men are
undifficult or have to pick them at custard.

1:11:07.247 --> 1:11:11.462
So everywhere talking to me about fire and
ants nowadays,.

1:11:11.791 --> 1:11:23.333
What we are typically meaning are LSDN's or
long short memories.

1:11:23.333 --> 1:11:30.968
You see they are by now quite old already.

1:11:31.171 --> 1:11:39.019
So there was a model in the language model
task.

1:11:39.019 --> 1:11:44.784
It's some more storing information.

1:11:44.684 --> 1:11:51.556
Because if you only look at the last words,
it's often no longer clear this is a question

1:11:51.556 --> 1:11:52.548
or a normal.

1:11:53.013 --> 1:12:05.318
So there you have these mechanisms with ripgate
in order to store things for a longer time

1:12:05.318 --> 1:12:08.563
into your hidden state.

1:12:10.730 --> 1:12:20.162
Here they are used in in in selling quite
a lot of works.

1:12:21.541 --> 1:12:29.349
For especially machine translation now, the
standard is to do transform base models which

1:12:29.349 --> 1:12:30.477
we'll learn.

1:12:30.690 --> 1:12:38.962
But for example, in architecture we have later
one lecture about efficiency.

1:12:38.962 --> 1:12:42.830
So how can we build very efficient?

1:12:42.882 --> 1:12:53.074
And there in the decoder in parts of the networks
they are still using.

1:12:53.473 --> 1:12:57.518
So it's not that yeah our hands are of no
importance in the body.

1:12:59.239 --> 1:13:08.956
In order to make them strong, there are some
more things which are helpful and should be:

1:13:09.309 --> 1:13:19.683
So one thing is there is a nice trick to make
this new network stronger and better.

1:13:19.739 --> 1:13:21.523
So of course it doesn't work always.

1:13:21.523 --> 1:13:23.451
They have to have enough training data.

1:13:23.763 --> 1:13:28.959
But in general there's the easiest way of
making your models bigger and stronger just

1:13:28.959 --> 1:13:30.590
to increase your pyramids.

1:13:30.630 --> 1:13:43.236
And you've seen that with a large language
models they are always bragging about.

1:13:43.903 --> 1:13:56.463
This is one way, so the question is how do
you get more parameters?

1:13:56.463 --> 1:14:01.265
There's ways of doing it.

1:14:01.521 --> 1:14:10.029
And the other thing is to make your networks
deeper so to have more legs in between.

1:14:11.471 --> 1:14:13.827
And then you can also get to get more calm.

1:14:14.614 --> 1:14:23.340
There's more traveling with this and it's
very similar to what we just saw with our hand.

1:14:23.603 --> 1:14:34.253
We have this problem of radiant flow that
if it flows so fast like a radiant gets very

1:14:34.253 --> 1:14:35.477
swollen,.

1:14:35.795 --> 1:14:42.704
Exactly the same thing happens in deep LSD
ends.

1:14:42.704 --> 1:14:52.293
If you take here the gradient, tell you what
is the right or wrong.

1:14:52.612 --> 1:14:56.439
With three layers it's no problem, but if
you're going to ten, twenty or hundred layers.

1:14:57.797 --> 1:14:59.698
That's Getting Typically Young.

1:15:00.060 --> 1:15:07.000
Are doing is using what is called decisional
connections.

1:15:07.000 --> 1:15:15.855
That's a very helpful idea, which is maybe
very surprising that it works.

1:15:15.956 --> 1:15:20.309
And so the idea is that these networks.

1:15:20.320 --> 1:15:29.982
In between should no longer calculate what
is a new good representation, but they're more

1:15:29.982 --> 1:15:31.378
calculating.

1:15:31.731 --> 1:15:37.588
Therefore, in the end you're always the output
of a layer is added with the input.

1:15:38.318 --> 1:15:48.824
The knife is later if you are doing back propagation
with this very fast back propagation.

1:15:49.209 --> 1:16:02.540
Nowadays in very deep architectures, not only
on other but always has this residual or highway

1:16:02.540 --> 1:16:04.224
connection.

1:16:04.704 --> 1:16:06.616
Has two advantages.

1:16:06.616 --> 1:16:15.409
On the one hand, these layers don't need to
learn a representation, they only need to learn

1:16:15.409 --> 1:16:18.754
what to change the representation.

1:16:22.082 --> 1:16:24.172
Good.

1:16:23.843 --> 1:16:31.768
That much for the new map before, so the last
thing now means this.

1:16:31.671 --> 1:16:33.750
Language was are yeah.

1:16:33.750 --> 1:16:41.976
I were used in the molds itself and now were
seeing them again, but one thing which at the

1:16:41.976 --> 1:16:53.558
beginning they were reading was very essential
was: So people really train part of the language

1:16:53.558 --> 1:16:59.999
models only to get this type of embedding.

1:16:59.999 --> 1:17:04.193
Therefore, we want to look.

1:17:09.229 --> 1:17:15.678
So now some last words to the word embeddings.

1:17:15.678 --> 1:17:27.204
The interesting thing is that word embeddings
can be used for very different tasks.

1:17:27.347 --> 1:17:31.329
The knife wing is you can train that on just
large amounts of data.

1:17:31.931 --> 1:17:41.569
And then if you have these wooden beddings
we have seen that they reduce the parameters.

1:17:41.982 --> 1:17:52.217
So then you can train your small mark to do
any other task and therefore you are more efficient.

1:17:52.532 --> 1:17:55.218
These initial word embeddings is important.

1:17:55.218 --> 1:18:00.529
They really depend only on the word itself,
so if you look at the two meanings of can,

1:18:00.529 --> 1:18:06.328
the can of beans or I can do that, they will
have the same embedding, so some of the embedding

1:18:06.328 --> 1:18:08.709
has to save the ambiguity inside that.

1:18:09.189 --> 1:18:12.486
That cannot be resolved.

1:18:12.486 --> 1:18:24.753
Therefore, if you look at the higher levels
in the context, but in the word embedding layers

1:18:24.753 --> 1:18:27.919
that really depends on.

1:18:29.489 --> 1:18:33.757
However, even this one has quite very interesting.

1:18:34.034 --> 1:18:39.558
So that people like to visualize them.

1:18:39.558 --> 1:18:47.208
They're always difficult because if you look
at this.

1:18:47.767 --> 1:18:52.879
And drawing your five hundred damage, the
vector is still a bit challenging.

1:18:53.113 --> 1:19:12.472
So you cannot directly do that, so people
have to do it like they look at some type of.

1:19:13.073 --> 1:19:17.209
And of course then yes some information is
getting lost by a bunch of control.

1:19:18.238 --> 1:19:24.802
And you see, for example, this is the most
famous and common example, so what you can

1:19:24.802 --> 1:19:31.289
look is you can look at the difference between
the main and the female word English.

1:19:31.289 --> 1:19:37.854
This is here in your embedding of king, and
this is the embedding of queen, and this.

1:19:38.058 --> 1:19:40.394
You can do that for a very different work.

1:19:40.780 --> 1:19:45.407
And that is where the masks come into, that
is what people then look into.

1:19:45.725 --> 1:19:50.995
So what you can now, for example, do is you
can calculate the difference between man and

1:19:50.995 --> 1:19:51.410
woman?

1:19:52.232 --> 1:19:55.511
Then you can take the embedding of tea.

1:19:55.511 --> 1:20:02.806
You can add on it the difference between man
and woman, and then you can notice what are

1:20:02.806 --> 1:20:04.364
the similar words.

1:20:04.364 --> 1:20:08.954
So you won't, of course, directly hit the
correct word.

1:20:08.954 --> 1:20:10.512
It's a continuous.

1:20:10.790 --> 1:20:23.127
But you can look what are the nearest neighbors
to this same, and often these words are near

1:20:23.127 --> 1:20:24.056
there.

1:20:24.224 --> 1:20:33.913
So it somehow learns that the difference between
these words is always the same.

1:20:34.374 --> 1:20:37.746
You can do that for different things.

1:20:37.746 --> 1:20:41.296
He also imagines that it's not perfect.

1:20:41.296 --> 1:20:49.017
He says the world tends to be swimming and
swimming, and with walking and walking you.

1:20:49.469 --> 1:20:51.639
So you can try to use them.

1:20:51.639 --> 1:20:59.001
It's no longer like saying yeah, but the interesting
thing is this is completely unsupervised.

1:20:59.001 --> 1:21:03.961
So nobody taught him the principle of their
gender in language.

1:21:04.284 --> 1:21:09.910
So it's purely trained on the task of doing
the next work prediction.

1:21:10.230 --> 1:21:20.658
And even for really cementing information
like the capital, this is the difference between

1:21:20.658 --> 1:21:23.638
the city and the capital.

1:21:23.823 --> 1:21:25.518
Visualization.

1:21:25.518 --> 1:21:33.766
Here we have done the same things of the difference
between country and.

1:21:33.853 --> 1:21:41.991
You see it's not perfect, but it's building
some kinds of a right direction, so you can't

1:21:41.991 --> 1:21:43.347
even use them.

1:21:43.347 --> 1:21:51.304
For example, for question answering, if you
have the difference between them, you apply

1:21:51.304 --> 1:21:53.383
that to a new country.

1:21:54.834 --> 1:22:02.741
So it seems these ones are able to really
learn a lot of information and collapse all

1:22:02.741 --> 1:22:04.396
this information.

1:22:05.325 --> 1:22:11.769
At just to do the next word prediction: And
that also explains a bit maybe or not explains

1:22:11.769 --> 1:22:19.016
wrong life by motivating why what is the main
advantage of this type of neural models that

1:22:19.016 --> 1:22:26.025
we can use this type of hidden representation,
transfer them and use them in different.

1:22:28.568 --> 1:22:43.707
So summarize what we did today, so what you
should hopefully have with you is for machine

1:22:43.707 --> 1:22:45.893
translation.

1:22:45.805 --> 1:22:49.149
Then how we can do language modern Chinese
literature?

1:22:49.449 --> 1:22:55.617
We looked at three different architectures:
We looked into the feet forward language mode

1:22:55.617 --> 1:22:59.063
and the one based on Bluetooth machines.

1:22:59.039 --> 1:23:05.366
And finally there are different architectures
to do in your networks.

1:23:05.366 --> 1:23:14.404
We have seen feet for your networks and we'll
see the next lectures, the last type of architecture.

1:23:15.915 --> 1:23:17.412
Have Any Questions.

1:23:20.680 --> 1:23:27.341
Then thanks a lot, and next on Tuesday we
will be again in our order to know how to play.

0:00:01.301 --> 0:00:05.687
Okay, so we're welcome to today's lecture.

0:00:06.066 --> 0:00:18.128
A bit desperate in a small room and I'm sorry
for the inconvenience.

0:00:18.128 --> 0:00:25.820
Sometimes there are project meetings where.

0:00:26.806 --> 0:00:40.863
So what we want to talk today about is want
to start with neural approaches to machine

0:00:40.863 --> 0:00:42.964
translation.

0:00:43.123 --> 0:00:55.779
Guess I've heard about other types of neural
models for natural language processing.

0:00:55.779 --> 0:00:59.948
This was some of the first.

0:01:00.600 --> 0:01:06.203
They are similar to what you know they see
in as large language models.

0:01:06.666 --> 0:01:14.810
And we want today look into what are these
neural language models, how we can build them,

0:01:14.810 --> 0:01:15.986
what is the.

0:01:16.316 --> 0:01:23.002
And first we'll show how to use them in statistical
machine translation.

0:01:23.002 --> 0:01:31.062
If you remember weeks ago, we had this log-linear
model where you can integrate easily.

0:01:31.351 --> 0:01:42.756
And that was how they first were used, so
we just had another model that evaluates how

0:01:42.756 --> 0:01:49.180
good a system is or how good a lot of languages.

0:01:50.690 --> 0:02:04.468
And next week we will go for a neuromachine
translation where we replace the whole model

0:02:04.468 --> 0:02:06.481
by one huge.

0:02:11.211 --> 0:02:20.668
So just as a member from Tuesday we've seen,
the main challenge in language modeling was

0:02:20.668 --> 0:02:25.131
that most of the anthrax we haven't seen.

0:02:26.946 --> 0:02:34.167
So this was therefore difficult to estimate
any probability because we've seen that yet

0:02:34.167 --> 0:02:39.501
normally if you've seen had not seen the N
gram you will assign.

0:02:39.980 --> 0:02:53.385
However, this is not really very good because
we don't want to give zero probabilities to

0:02:53.385 --> 0:02:55.023
sentences.

0:02:55.415 --> 0:03:10.397
And then we learned a lot of techniques and
that is the main challenge in statistical language.

0:03:10.397 --> 0:03:15.391
How we can give somehow a good.

0:03:15.435 --> 0:03:23.835
And they developed very specific, very good
techniques to deal with that.

0:03:23.835 --> 0:03:26.900
However, this is the best.

0:03:28.568 --> 0:03:33.907
And therefore we can do things different.

0:03:33.907 --> 0:03:44.331
If we have not seen an N gram before in statistical
models, we have to have seen.

0:03:45.225 --> 0:03:51.361
Before, and we can only get information from
exactly the same word.

0:03:51.411 --> 0:03:57.567
We don't have an approximate matching like
that.

0:03:57.567 --> 0:04:10.255
Maybe it stood together in some way or similar,
and in a sentence we might generalize the knowledge.

0:04:11.191 --> 0:04:21.227
Would like to have more something like that
where engrams are represented more in a general

0:04:21.227 --> 0:04:21.990
space.

0:04:22.262 --> 0:04:29.877
So if you learn something about eyewalk then
maybe we can use this knowledge and also.

0:04:30.290 --> 0:04:43.034
And thereby no longer treat all or at least
a lot of the ingrams as we've done before.

0:04:43.034 --> 0:04:45.231
We can really.

0:04:47.047 --> 0:04:56.157
And we maybe want to even do that in a more
hierarchical approach, but we know okay some

0:04:56.157 --> 0:05:05.268
words are similar like go and walk is somehow
similar and and therefore like maybe if we

0:05:05.268 --> 0:05:07.009
then merge them.

0:05:07.387 --> 0:05:16.104
If we learn something about work, then it
should tell us also something about Hugo or

0:05:16.104 --> 0:05:17.118
he walks.

0:05:17.197 --> 0:05:18.970
We see already.

0:05:18.970 --> 0:05:22.295
It's, of course, not so easy.

0:05:22.295 --> 0:05:31.828
We see that there is some relations which
we need to integrate, for example, for you.

0:05:31.828 --> 0:05:35.486
We need to add the S, but maybe.

0:05:37.137 --> 0:05:42.984
And luckily there is one really yeah, convincing
methods in doing that.

0:05:42.963 --> 0:05:47.239
And that is by using an evil neck or.

0:05:47.387 --> 0:05:57.618
That's what we will introduce today so we
can use this type of neural networks to try

0:05:57.618 --> 0:06:04.042
to learn this similarity and to learn how some
words.

0:06:04.324 --> 0:06:13.711
And that is one of the main advantages that
we have by switching from the standard statistical

0:06:13.711 --> 0:06:15.193
models to the.

0:06:15.115 --> 0:06:22.840
To learn similarities between words and generalized
and learn what we call hidden representations.

0:06:22.840 --> 0:06:29.707
So somehow representations of words where
we can measure similarity in some dimensions.

0:06:30.290 --> 0:06:42.275
So in representations where as a tubically
continuous vector or a vector of a fixed size.

0:06:42.822 --> 0:06:52.002
We had it before and we've seen that the only
thing we did is we don't want to do.

0:06:52.192 --> 0:06:59.648
But these indices don't have any meaning,
so it wasn't that word five is more similar

0:06:59.648 --> 0:07:02.248
to words twenty than to word.

0:07:02.582 --> 0:07:09.059
So we couldn't learn anything about words
in the statistical model.

0:07:09.059 --> 0:07:12.107
That's a big challenge because.

0:07:12.192 --> 0:07:24.232
If you think about words even in morphology,
so go and go is more similar because the person.

0:07:24.264 --> 0:07:36.265
While the basic models we have up to now,
they have no idea about that and goes as similar

0:07:36.265 --> 0:07:37.188
to go.

0:07:39.919 --> 0:07:53.102
So what we want to do today, in order to go
to this, we will have a short introduction.

0:07:53.954 --> 0:08:06.667
It very short just to see how we use them
here, but that's the good thing that are important

0:08:06.667 --> 0:08:08.445
for dealing.

0:08:08.928 --> 0:08:14.083
And then we'll first look into feet forward,
new network language models.

0:08:14.454 --> 0:08:21.221
And there we will still have this approximation
we had before, then we are looking only at

0:08:21.221 --> 0:08:22.336
fixed windows.

0:08:22.336 --> 0:08:28.805
So if you remember we have this classroom
of language models, and to determine what is

0:08:28.805 --> 0:08:33.788
the probability of a word, we only look at
the past and minus one.

0:08:34.154 --> 0:08:36.878
This is the theory of the case.

0:08:36.878 --> 0:08:43.348
However, we have the ability and that's why
they're really better in order.

0:08:44.024 --> 0:08:51.953
And then at the end we'll look at current
network language models where we then have

0:08:51.953 --> 0:08:53.166
a different.

0:08:53.093 --> 0:09:01.922
And thereby it is no longer the case that
we need to have a fixed history, but in theory

0:09:01.922 --> 0:09:04.303
we can model arbitrary.

0:09:04.304 --> 0:09:06.854
And we can log this phenomenon.

0:09:06.854 --> 0:09:12.672
We talked about a Tuesday where it's not clear
what type of information.

0:09:16.396 --> 0:09:24.982
So yeah, generally new networks are normally
learned to improve and perform some tasks.

0:09:25.325 --> 0:09:38.934
We have this structure and we are learning
them from samples so that is similar to what

0:09:38.934 --> 0:09:42.336
we had before so now.

0:09:42.642 --> 0:09:49.361
And is somehow originally motivated by the
human brain.

0:09:49.361 --> 0:10:00.640
However, when you now need to know artificial
neural networks, it's hard to get a similarity.

0:10:00.540 --> 0:10:02.884
There seems to be not that important.

0:10:03.123 --> 0:10:11.013
So what they are mainly doing is doing summoning
multiplication and then one linear activation.

0:10:12.692 --> 0:10:16.078
So so the basic units are these type of.

0:10:17.937 --> 0:10:29.837
Perceptron is a basic block which we have
and this does exactly the processing.

0:10:29.837 --> 0:10:36.084
We have a fixed number of input features.

0:10:36.096 --> 0:10:39.668
So we have here numbers six zero to x and
as input.

0:10:40.060 --> 0:10:48.096
And this makes language processing difficult
because we know that it's not the case.

0:10:48.096 --> 0:10:53.107
If we're dealing with language, it doesn't
have any.

0:10:54.114 --> 0:10:57.609
So we have to model this somehow and understand
how we model this.

0:10:58.198 --> 0:11:03.681
Then we have the weights, which are the parameters
and the number of weights exactly the same.

0:11:04.164 --> 0:11:15.069
Of input features sometimes you have the spires
in there that always and then it's not really.

0:11:15.195 --> 0:11:19.656
And what you then do is very simple.

0:11:19.656 --> 0:11:26.166
It's just like the weight it sounds, so you
multiply.

0:11:26.606 --> 0:11:38.405
What is then additionally important is we
have an activation function and it's important

0:11:38.405 --> 0:11:42.514
that this activation function.

0:11:43.243 --> 0:11:54.088
And later it will be important that this is
differentiable because otherwise all the training.

0:11:54.714 --> 0:12:01.471
This model by itself is not very powerful.

0:12:01.471 --> 0:12:10.427
We have the X Or problem and with this simple
you can't.

0:12:10.710 --> 0:12:15.489
However, there is a very easy and nice extension.

0:12:15.489 --> 0:12:20.936
The multi layer perception and things get
very powerful.

0:12:21.081 --> 0:12:32.953
The thing is you just connect a lot of these
in these layers of structures where we have

0:12:32.953 --> 0:12:35.088
the inputs and.

0:12:35.395 --> 0:12:47.297
And then we can combine them, or to do them:
The input layer is of course given by your

0:12:47.297 --> 0:12:51.880
problem with the dimension.

0:12:51.880 --> 0:13:00.063
The output layer is also given by your dimension.

0:13:01.621 --> 0:13:08.802
So let's start with the first question, now
more language related, and that is how we represent.

0:13:09.149 --> 0:13:19.282
So we have seen here input to x, but the question
is now okay.

0:13:19.282 --> 0:13:23.464
How can we put into this?

0:13:26.866 --> 0:13:34.123
The first thing that we're able to do is we're
going to set it in the inspector.

0:13:34.314 --> 0:13:45.651
Yeah, and that is not that easy because the
continuous vector will come to that.

0:13:45.651 --> 0:13:47.051
We can't.

0:13:47.051 --> 0:13:50.410
We don't want to do it.

0:13:50.630 --> 0:13:57.237
But if we need to input the word into the
needle network, it has to be something easily

0:13:57.237 --> 0:13:57.912
defined.

0:13:59.079 --> 0:14:11.511
One is the typical thing, the one-hour encoded
vector, so we have a vector where the dimension

0:14:11.511 --> 0:14:15.306
is the vocabulary, and then.

0:14:16.316 --> 0:14:25.938
So the first thing you are ready to see that
means we are always dealing with fixed.

0:14:26.246 --> 0:14:34.961
So you cannot easily extend your vocabulary,
but if you mean your vocabulary would increase

0:14:34.961 --> 0:14:37.992
the size of this input vector,.

0:14:39.980 --> 0:14:42.423
That's maybe also motivating.

0:14:42.423 --> 0:14:45.355
We'll talk about bike parade going.

0:14:45.355 --> 0:14:47.228
That's the nice thing.

0:14:48.048 --> 0:15:01.803
The big advantage of this one putt encoding
is that we don't implement similarity between

0:15:01.803 --> 0:15:06.999
words, but we're really learning.

0:15:07.227 --> 0:15:11.219
So you need like to represent any words.

0:15:11.219 --> 0:15:15.893
You need a dimension of and dimensional vector.

0:15:16.236 --> 0:15:26.480
Imagine you could eat no binary encoding,
so you could represent words as binary vectors.

0:15:26.806 --> 0:15:32.348
So you will be significantly more efficient.

0:15:32.348 --> 0:15:39.122
However, you have some more digits than other
numbers.

0:15:39.559 --> 0:15:46.482
Would somehow be bad because you would force
the one to do this and it's by hand not clear

0:15:46.482 --> 0:15:47.623
how to define.

0:15:48.108 --> 0:15:55.135
So therefore currently this is the most successful
approach to just do this one patch.

0:15:55.095 --> 0:15:59.344
We take a fixed vocabulary.

0:15:59.344 --> 0:16:10.269
We map each word to the initial and then we
represent a word like this.

0:16:10.269 --> 0:16:13.304
The representation.

0:16:14.514 --> 0:16:27.019
But this dimension here is a secondary size,
and if you think ten thousand that's quite

0:16:27.019 --> 0:16:33.555
high, so we're always trying to be efficient.

0:16:33.853 --> 0:16:42.515
And we are doing the same type of efficiency
because then we are having a very small one

0:16:42.515 --> 0:16:43.781
compared to.

0:16:44.104 --> 0:16:53.332
It can be still a maybe or neurons, but this
is significantly smaller, of course, as before.

0:16:53.713 --> 0:17:04.751
So you are learning there this word as you
said, but you can learn it directly, and there

0:17:04.751 --> 0:17:07.449
we have similarities.

0:17:07.807 --> 0:17:14.772
But the nice thing is that this is then learned,
and we do not need to like hand define.

0:17:17.117 --> 0:17:32.377
So yes, so that is how we're typically adding
at least a single word into the language world.

0:17:32.377 --> 0:17:43.337
Then we can see: So we're seeing that you
have the one hard representation always of

0:17:43.337 --> 0:17:44.857
the same similarity.

0:17:45.105 --> 0:18:00.803
Then we're having this continuous vector which
is a lot smaller dimension and that's.

0:18:01.121 --> 0:18:06.984
What we are doing then is learning these representations
so that they are best for language modeling.

0:18:07.487 --> 0:18:19.107
So the representations are implicitly because
we're training on the language.

0:18:19.479 --> 0:18:30.115
And the nice thing was found out later is
these representations are really, really good

0:18:30.115 --> 0:18:32.533
for a lot of other.

0:18:33.153 --> 0:18:39.729
And that is why they are now called word embedded
space themselves, and used for other tasks.

0:18:40.360 --> 0:18:49.827
And they are somehow describing different
things so they can describe and semantic similarities.

0:18:49.789 --> 0:18:58.281
We are looking at the very example of today
that you can do in this vector space by adding

0:18:58.281 --> 0:19:00.613
some interesting things.

0:19:00.940 --> 0:19:11.174
And so they got really was a first big improvement
when switching to neural staff.

0:19:11.491 --> 0:19:20.736
They are like part of the model still with
more complex representation alert, but they

0:19:20.736 --> 0:19:21.267
are.

0:19:23.683 --> 0:19:34.975
Then we are having the output layer, and in
the output layer we also have output structure

0:19:34.975 --> 0:19:36.960
and activation.

0:19:36.997 --> 0:19:44.784
That is the language we want to predict, which
word should be the next.

0:19:44.784 --> 0:19:46.514
We always have.

0:19:47.247 --> 0:19:56.454
And that can be done very well with the softball
softbacked layer, where again the dimension.

0:19:56.376 --> 0:20:03.971
Is the vocabulary, so this is a vocabulary
size, and again the case neuro represents the

0:20:03.971 --> 0:20:09.775
case class, so in our case we have again a
one-hour representation.

0:20:10.090 --> 0:20:18.929
Ours is a probability distribution and the
end is a probability distribution of all works.

0:20:18.929 --> 0:20:28.044
The case entry tells us: So we need to have
some of our probability distribution at our

0:20:28.044 --> 0:20:36.215
output, and in order to achieve that this activation
function goes, it needs to be that all the

0:20:36.215 --> 0:20:36.981
outputs.

0:20:37.197 --> 0:20:47.993
And we can achieve that with a softmax activation
we take each of the value and then.

0:20:48.288 --> 0:20:58.020
So by having this type of activation function
we are really getting that at the end we always.

0:20:59.019 --> 0:21:12.340
The beginning was very challenging because
again we have this inefficient representation

0:21:12.340 --> 0:21:15.184
of our vocabulary.

0:21:15.235 --> 0:21:27.500
And then you can imagine escalating over to
something over a thousand is maybe a bit inefficient

0:21:27.500 --> 0:21:29.776
with cheap users.

0:21:36.316 --> 0:21:43.664
And then yeah, for training the models, that
is how we refine, so we have this architecture

0:21:43.664 --> 0:21:44.063
now.

0:21:44.264 --> 0:21:52.496
We need to minimize the arrow by taking the
output.

0:21:52.496 --> 0:21:58.196
We are comparing it to our targets.

0:21:58.298 --> 0:22:07.670
So one important thing is, of course, how
can we measure the error?

0:22:07.670 --> 0:22:12.770
So what if we're training the ideas?

0:22:13.033 --> 0:22:19.770
And how well when measuring it is in natural
language processing, typically the cross entropy.

0:22:19.960 --> 0:22:32.847
That means we are comparing the target with
the output, so we're taking the value multiplying

0:22:32.847 --> 0:22:35.452
with the horizons.

0:22:35.335 --> 0:22:43.454
Which gets optimized and you're seeing that
this, of course, makes it again very nice and

0:22:43.454 --> 0:22:49.859
easy because our target, we said, is again
a one-hound representation.

0:22:50.110 --> 0:23:00.111
So except for one, all of these are always
zero, and what we are doing is taking the one.

0:23:00.100 --> 0:23:05.970
And we only need to multiply the one with
the logarism here, and that is all the feedback.

0:23:06.946 --> 0:23:14.194
Of course, this is not always influenced by
all the others.

0:23:14.194 --> 0:23:17.938
Why is this influenced by all?

0:23:24.304 --> 0:23:33.554
Think Mac the activation function, which is
the current activation divided by some of the

0:23:33.554 --> 0:23:34.377
others.

0:23:34.354 --> 0:23:44.027
Because otherwise it could of course easily
just increase this value and ignore the others,

0:23:44.027 --> 0:23:49.074
but if you increase one value or the other,
so.

0:23:51.351 --> 0:24:04.433
And then we can do with neon networks one
very nice and easy type of training that is

0:24:04.433 --> 0:24:07.779
done in all the neon.

0:24:07.707 --> 0:24:12.664
So in which direction does the arrow show?

0:24:12.664 --> 0:24:23.152
And then if we want to go to a smaller like
smaller arrow, that's what we want to achieve.

0:24:23.152 --> 0:24:27.302
We're trying to minimize our arrow.

0:24:27.287 --> 0:24:32.875
And we have to do that, of course, for all
the weights, and to calculate the error of

0:24:32.875 --> 0:24:36.709
all the weights we want in the back of the
baggation here.

0:24:36.709 --> 0:24:41.322
But what you can do is you can propagate the
arrow which you measured.

0:24:41.322 --> 0:24:43.792
At the end you can propagate it back.

0:24:43.792 --> 0:24:46.391
That's basic mass and basic derivation.

0:24:46.706 --> 0:24:59.557
Then you can do each weight in your model
and measure how much it contributes to this

0:24:59.557 --> 0:25:01.350
individual.

0:25:04.524 --> 0:25:17.712
To summarize what your machine translation
should be, to understand all this problem is

0:25:17.712 --> 0:25:20.710
that this is how a.

0:25:20.580 --> 0:25:23.056
The notes are perfect thrones.

0:25:23.056 --> 0:25:28.167
They are fully connected between two layers
and no connections.

0:25:28.108 --> 0:25:29.759
Across layers.

0:25:29.829 --> 0:25:35.152
And what they're doing is always just to wait
for some here and then an activation function.

0:25:35.415 --> 0:25:38.794
And in order to train you have this sword
in backwards past.

0:25:39.039 --> 0:25:41.384
So we put in here.

0:25:41.281 --> 0:25:46.540
Our inputs have some random values at the
beginning.

0:25:46.540 --> 0:25:49.219
They calculate the output.

0:25:49.219 --> 0:25:58.646
We are measuring how big our error is, propagating
the arrow back, and then changing our model

0:25:58.646 --> 0:25:59.638
in a way.

0:26:01.962 --> 0:26:14.267
So before we're coming into the neural networks,
how can we use this type of neural network

0:26:14.267 --> 0:26:17.611
to do language modeling?

0:26:23.103 --> 0:26:25.520
So the question is now okay.

0:26:25.520 --> 0:26:33.023
How can we use them in natural language processing
and especially in machine translation?

0:26:33.023 --> 0:26:38.441
The first idea of using them was to estimate
the language model.

0:26:38.999 --> 0:26:42.599
So we have seen that the output can be monitored
here as well.

0:26:43.603 --> 0:26:49.308
Has a probability distribution, and if we
have a full vocabulary, we could mainly hear

0:26:49.308 --> 0:26:55.209
estimate how probable each next word is, and
then use that in our language model fashion,

0:26:55.209 --> 0:27:02.225
as we've done it last time, we've got the probability
of a full sentence as a product of all probabilities

0:27:02.225 --> 0:27:03.208
of individual.

0:27:04.544 --> 0:27:06.695
And UM.

0:27:06.446 --> 0:27:09.776
That was done and in ninety seven years.

0:27:09.776 --> 0:27:17.410
It's very easy to integrate it into this Locklear
model, so we have said that this is how the

0:27:17.410 --> 0:27:24.638
Locklear model looks like, so we're searching
the best translation, which minimizes each

0:27:24.638 --> 0:27:25.126
wage.

0:27:25.125 --> 0:27:26.371
The feature value.

0:27:26.646 --> 0:27:31.642
We have that with the minimum error training,
if you can remember when we search for the

0:27:31.642 --> 0:27:32.148
optimal.

0:27:32.512 --> 0:27:40.927
We have the phrasetable probabilities, the
language model, and we can just add here and

0:27:40.927 --> 0:27:41.597
there.

0:27:41.861 --> 0:27:46.077
So that is quite easy as said.

0:27:46.077 --> 0:27:54.101
That was how statistical machine translation
was improved.

0:27:54.101 --> 0:27:57.092
Add one more feature.

0:27:58.798 --> 0:28:11.220
So how can we model the language mark for
Belty with your network?

0:28:11.220 --> 0:28:22.994
So what we have to do is: And the problem
in generally in the head is that most we haven't

0:28:22.994 --> 0:28:25.042
seen long sequences.

0:28:25.085 --> 0:28:36.956
Mostly we have to beg off to very short sequences
and we are working on this discrete space where.

0:28:37.337 --> 0:28:48.199
So the idea is if we have a meal network we
can map words into continuous representation

0:28:48.199 --> 0:28:50.152
and that helps.

0:28:51.091 --> 0:28:59.598
And the structure then looks like this, so
this is the basic still feed forward neural

0:28:59.598 --> 0:29:00.478
network.

0:29:01.361 --> 0:29:10.744
We are doing this at Proximation again, so
we are not putting in all previous words, but

0:29:10.744 --> 0:29:11.376
it's.

0:29:11.691 --> 0:29:25.089
And this is done because in your network we
can have only a fixed type of input, so we

0:29:25.089 --> 0:29:31.538
can: Can only do a fixed set, and they are
going to be doing exactly the same in minus

0:29:31.538 --> 0:29:31.879
one.

0:29:33.593 --> 0:29:41.026
And then we have, for example, three words
and three different words, which are in these

0:29:41.026 --> 0:29:54.583
positions: And then we're having the first
layer of the neural network, which learns words

0:29:54.583 --> 0:29:56.247
and words.

0:29:57.437 --> 0:30:04.976
There is one thing which is maybe special
compared to the standard neural memory.

0:30:05.345 --> 0:30:13.163
So the representation of this word we want
to learn first of all position independence,

0:30:13.163 --> 0:30:19.027
so we just want to learn what is the general
meaning of the word.

0:30:19.299 --> 0:30:26.244
Therefore, the representation you get here
should be the same as if you put it in there.

0:30:27.247 --> 0:30:35.069
The nice thing is you can achieve that in
networks the same way you achieve it.

0:30:35.069 --> 0:30:41.719
This way you're reusing ears so we are forcing
them to always stay.

0:30:42.322 --> 0:30:49.689
And that's why you then learn your word embedding,
which is contextual and independent, so.

0:30:49.909 --> 0:31:05.561
So the idea is you have the diagram go home
and you don't want to use the context.

0:31:05.561 --> 0:31:07.635
First you.

0:31:08.348 --> 0:31:14.155
That of course it might have a different meaning
depending on where it stands, but learn that.

0:31:14.514 --> 0:31:19.623
First, we're learning key representation of
the words, which is just the representation

0:31:19.623 --> 0:31:20.378
of the word.

0:31:20.760 --> 0:31:37.428
So it's also not like normally all input neurons
are connected to all neurons.

0:31:37.857 --> 0:31:47.209
This is the first layer of representation,
and then we have a lot denser representation,

0:31:47.209 --> 0:31:56.666
that is, our three word embeddings here, and
now we are learning this interaction between

0:31:56.666 --> 0:31:57.402
words.

0:31:57.677 --> 0:32:08.265
So now we have at least one connected, fully
connected layer here, which takes the three

0:32:08.265 --> 0:32:14.213
imbedded input and then learns the new embedding.

0:32:15.535 --> 0:32:27.871
And then if you had one of several layers
of lining which is your output layer, then.

0:32:28.168 --> 0:32:46.222
So here the size is a vocabulary size, and
then you put as target what is the probability

0:32:46.222 --> 0:32:48.228
for each.

0:32:48.688 --> 0:32:56.778
The nice thing is that you learn everything
together, so you're not learning what is a

0:32:56.778 --> 0:32:58.731
good representation.

0:32:59.079 --> 0:33:12.019
When you are training the whole network together,
it learns what representation for a word you

0:33:12.019 --> 0:33:13.109
get in.

0:33:15.956 --> 0:33:19.176
It's Yeah That Is the Main Idea.

0:33:20.660 --> 0:33:32.695
Nowadays often referred to as one way of self-supervised
learning, why self-supervisory learning?

0:33:33.053 --> 0:33:37.120
The output is the next word and the input
is the previous word.

0:33:37.377 --> 0:33:46.778
But somehow it's self-supervised because it's
not really that we created labels, but we artificially.

0:33:46.806 --> 0:34:01.003
We just have pure text, and then we created
the task.

0:34:05.905 --> 0:34:12.413
Say we have two sentences like go home again.

0:34:12.413 --> 0:34:18.780
Second one is go to creative again, so both.

0:34:18.858 --> 0:34:31.765
The starboard bygo and then we have to predict
the next four years and my question is: Be

0:34:31.765 --> 0:34:40.734
modeled this ability as one vector with like
probability or possible works.

0:34:40.734 --> 0:34:42.740
We have musical.

0:34:44.044 --> 0:34:56.438
You have multiple examples, so you would twice
train, once you predict, once you predict,

0:34:56.438 --> 0:35:02.359
and then, of course, the best performance.

0:35:04.564 --> 0:35:11.772
A very good point, so you're not aggregating
examples beforehand, but you're taking each

0:35:11.772 --> 0:35:13.554
example individually.

0:35:19.259 --> 0:35:33.406
So what you do is you simultaneously learn
the projection layer which represents this

0:35:33.406 --> 0:35:39.163
word and the N gram probabilities.

0:35:39.499 --> 0:35:48.390
And what people then later analyzed is that
these representations are very powerful.

0:35:48.390 --> 0:35:56.340
The task is just a very important task to
model like what is the next word.

0:35:56.816 --> 0:36:09.429
It's a bit motivated by people saying in order
to get the meaning of the word you have to

0:36:09.429 --> 0:36:10.690
look at.

0:36:10.790 --> 0:36:18.467
If you read the text in there, which you have
never seen, you can still estimate the meaning

0:36:18.467 --> 0:36:22.264
of this word because you know how it is used.

0:36:22.602 --> 0:36:26.667
Just imagine you read this text about some
city.

0:36:26.667 --> 0:36:32.475
Even if you've never seen the city before
heard, you often know from.

0:36:34.094 --> 0:36:44.809
So what is now the big advantage of using
neural networks?

0:36:44.809 --> 0:36:57.570
Just imagine we have to estimate this: So
you have to monitor the probability of ad hip

0:36:57.570 --> 0:37:00.272
and now imagine iPhone.

0:37:00.600 --> 0:37:06.837
So all the techniques we have at the last
time.

0:37:06.837 --> 0:37:14.243
At the end, if you haven't seen iPhone, you
will always.

0:37:15.055 --> 0:37:19.502
Because you haven't seen the previous words,
so you have no idea how to do that.

0:37:19.502 --> 0:37:24.388
You won't have seen the diagram, the trigram
and all the others, so the probability here

0:37:24.388 --> 0:37:27.682
will just be based on the probability of ad,
so it uses no.

0:37:28.588 --> 0:37:38.328
If you're having this type of model, what
does it do so?

0:37:38.328 --> 0:37:43.454
This is the last three words.

0:37:43.483 --> 0:37:49.837
Maybe this representation is messed up because
it's mainly on a particular word or source

0:37:49.837 --> 0:37:50.260
that.

0:37:50.730 --> 0:37:57.792
Now anyway you have these two information
that were two words before was first and therefore:

0:37:58.098 --> 0:38:07.214
So you have a lot of information here to estimate
how good it is.

0:38:07.214 --> 0:38:13.291
Of course, there could be more information.

0:38:13.593 --> 0:38:25.958
So all this type of modeling we can do and
that we couldn't do beforehand because we always.

0:38:27.027 --> 0:38:31.905
Don't guess how we do it now.

0:38:31.905 --> 0:38:41.824
Typically you would have one talking for awkward
vocabulary.

0:38:42.602 --> 0:38:45.855
All you're doing by carrying coding when it
has a fixed dancing.

0:38:46.226 --> 0:38:49.439
Yeah, you have to do something like that that
the opposite way.

0:38:50.050 --> 0:38:55.413
So yeah, all the vocabulary are by thankcoding
where you don't have have all the vocabulary.

0:38:55.735 --> 0:39:07.665
But then, of course, the back pairing coating
is better with arbitrary context because a

0:39:07.665 --> 0:39:11.285
problem with back pairing.

0:39:17.357 --> 0:39:20.052
Anymore questions to the basic same little
things.

0:39:23.783 --> 0:39:36.162
This model we then want to continue is to
look into how complex that is or can make things

0:39:36.162 --> 0:39:39.155
maybe more efficient.

0:39:40.580 --> 0:39:47.404
At the beginning there was definitely a major
challenge.

0:39:47.404 --> 0:39:50.516
It's still not that easy.

0:39:50.516 --> 0:39:58.297
All guess follow the talk about their environmental
fingerprint.

0:39:58.478 --> 0:40:05.686
So this calculation is normally heavy, and
if you build systems yourself, you have to

0:40:05.686 --> 0:40:06.189
wait.

0:40:06.466 --> 0:40:15.412
So it's good to know a bit about how complex
things are in order to do a good or efficient.

0:40:15.915 --> 0:40:24.706
So one thing where most of the calculation
really happens is if you're.

0:40:25.185 --> 0:40:34.649
So in generally all these layers, of course,
we're talking about networks and the zones

0:40:34.649 --> 0:40:35.402
fancy.

0:40:35.835 --> 0:40:48.305
So what you have to do in order to calculate
here these activations, you have this weight.

0:40:48.488 --> 0:41:05.021
So to make it simple, let's see we have three
outputs, and then you just do a metric identification

0:41:05.021 --> 0:41:08.493
between your weight.

0:41:08.969 --> 0:41:19.641
That is why the use is so powerful for neural
networks because they are very good in doing

0:41:19.641 --> 0:41:22.339
metric multiplication.

0:41:22.782 --> 0:41:28.017
However, for some type of embedding layer
this is really very inefficient.

0:41:28.208 --> 0:41:37.547
So in this input we are doing this calculation.

0:41:37.547 --> 0:41:47.081
What we are mainly doing is selecting one
color.

0:41:47.387 --> 0:42:03.570
So therefore you can do at least the forward
pass a lot more efficient if you don't really

0:42:03.570 --> 0:42:07.304
do this calculation.

0:42:08.348 --> 0:42:20.032
So the weight metrics of the first embedding
layer is just that in each color you have.

0:42:20.580 --> 0:42:30.990
So this is how your initial weights look like
and how you can interpret or understand.

0:42:32.692 --> 0:42:42.042
And this is already relatively important because
remember this is a huge dimensional thing,

0:42:42.042 --> 0:42:51.392
so typically here we have the number of words
ten thousand, so this is the word embeddings.

0:42:51.451 --> 0:43:00.400
Because it's the largest one there, we have
entries, while for the others we maybe have.

0:43:00.660 --> 0:43:03.402
So they are a little bit efficient and are
important to make this in.

0:43:06.206 --> 0:43:10.529
And then you can look at where else the calculations
are very difficult.

0:43:10.830 --> 0:43:20.294
So here we have our individual network, so
here are the word embeddings.

0:43:20.294 --> 0:43:29.498
Then we have one hidden layer, and then you
can look at how difficult.

0:43:30.270 --> 0:43:38.742
We could save a lot of calculations by calculating
that by just doing like do the selection because:

0:43:40.600 --> 0:43:51.748
And then the number of calculations you have
to do here is the length.

0:43:52.993 --> 0:44:06.206
Then we have here the hint size that is the
hint size, so the first step of calculation

0:44:06.206 --> 0:44:10.260
for this metric is an age.

0:44:10.730 --> 0:44:22.030
Then you have to do some activation function
which is this: This is the hidden size hymn

0:44:22.030 --> 0:44:29.081
because we need the vocabulary socks to calculate
the probability for each.

0:44:29.889 --> 0:44:40.474
And if you look at this number, so if you
have a projection sign of one hundred and a

0:44:40.474 --> 0:44:45.027
vocabulary sign of one hundred, you.

0:44:45.425 --> 0:44:53.958
And that's why there has been especially at
the beginning some ideas on how we can reduce

0:44:53.958 --> 0:44:55.570
the calculation.

0:44:55.956 --> 0:45:02.352
And if we really need to calculate all our
capabilities, or if we can calculate only some.

0:45:02.582 --> 0:45:13.061
And there again one important thing to think
about is for what you will use my language.

0:45:13.061 --> 0:45:21.891
One can use it for generations and that's
where we will see the next week.

0:45:21.891 --> 0:45:22.480
And.

0:45:23.123 --> 0:45:32.164
Initially, if it's just used as a feature,
we do not want to use it for generation, but

0:45:32.164 --> 0:45:32.575
we.

0:45:32.953 --> 0:45:41.913
And there we might not be interested in all
the probabilities, but we already know all

0:45:41.913 --> 0:45:49.432
the probability of this one word, and then
it might be very inefficient.

0:45:51.231 --> 0:45:53.638
And how can you do that so initially?

0:45:53.638 --> 0:45:56.299
For example, people look into shortlists.

0:45:56.756 --> 0:46:03.321
So the idea was this calculation at the end
is really very expensive.

0:46:03.321 --> 0:46:05.759
So can we make that more.

0:46:05.945 --> 0:46:17.135
And the idea was okay, and most birds occur
very rarely, and some beef birds occur very,

0:46:17.135 --> 0:46:18.644
very often.

0:46:19.019 --> 0:46:37.644
And so they use the smaller imagery, which
is maybe very small, and then you merge a new.

0:46:37.937 --> 0:46:45.174
So you're taking if the word is in the shortness,
so in the most frequent words.

0:46:45.825 --> 0:46:58.287
You're taking the probability of this short
word by some normalization here, and otherwise

0:46:58.287 --> 0:46:59.656
you take.

0:47:00.020 --> 0:47:00.836
Course.

0:47:00.836 --> 0:47:09.814
It will not be as good, but then we don't
have to calculate all the capabilities at the

0:47:09.814 --> 0:47:16.037
end, but we only have to calculate it for the
most frequent.

0:47:19.599 --> 0:47:39.477
Machines about that, but of course we don't
model the probability of the infrequent words.

0:47:39.299 --> 0:47:46.658
And one idea is to do what is reported as
soles for the structure of the layer.

0:47:46.606 --> 0:47:53.169
You see how some years ago people were very
creative in giving names to newer models.

0:47:53.813 --> 0:48:00.338
And there the idea is that we model the out
group vocabulary as a clustered strip.

0:48:00.680 --> 0:48:08.498
So you don't need to mold all of your bodies
directly, but you are putting words into.

0:48:08.969 --> 0:48:20.623
A very intricate word is first in and then
in and then in and that is in sub-sub-clusters

0:48:20.623 --> 0:48:21.270
and.

0:48:21.541 --> 0:48:29.936
And this is what was mentioned in the past
of the work, so these are the subclasses that

0:48:29.936 --> 0:48:30.973
always go.

0:48:30.973 --> 0:48:39.934
So if it's in cluster one at the first position
then you only look at all the words which are:

0:48:40.340 --> 0:48:50.069
And then you can calculate the probability
of a word again just by the product over these,

0:48:50.069 --> 0:48:55.522
so the probability of the word is the first
class.

0:48:57.617 --> 0:49:12.331
It's maybe more clear where you have the sole
architecture, so what you will do is first

0:49:12.331 --> 0:49:13.818
predict.

0:49:14.154 --> 0:49:26.435
Then you go to the appropriate sub-class,
then you calculate the probability of the sub-class.

0:49:27.687 --> 0:49:34.932
Anybody have an idea why this is more, more
efficient, or if people do it first, it looks

0:49:34.932 --> 0:49:35.415
more.

0:49:42.242 --> 0:49:56.913
Yes, so you have to do less calculations,
or maybe here you have to calculate the element

0:49:56.913 --> 0:49:59.522
there, but you.

0:49:59.980 --> 0:50:06.116
The capabilities in the set classes that you're
going through and not for all of them.

0:50:06.386 --> 0:50:16.688
Therefore, it's only more efficient if you
don't need all awkward preferences because

0:50:16.688 --> 0:50:21.240
you have to even calculate the class.

0:50:21.501 --> 0:50:30.040
So it's only more efficient in scenarios where
you really need to use a language to evaluate.

0:50:35.275 --> 0:50:54.856
How this works is that on the output layer
you only have a vocabulary of: But on the input

0:50:54.856 --> 0:51:05.126
layer you have always your full vocabulary
because at the input we saw that this is not

0:51:05.126 --> 0:51:06.643
complicated.

0:51:06.906 --> 0:51:19.778
And then you can cluster down all your words,
embedding series of classes, and use that as

0:51:19.778 --> 0:51:23.031
your classes for that.

0:51:23.031 --> 0:51:26.567
So yeah, you have words.

0:51:29.249 --> 0:51:32.593
Is one idea of doing it.

0:51:32.593 --> 0:51:44.898
There is also a second idea of doing it again,
the idea that we don't need the probability.

0:51:45.025 --> 0:51:53.401
So sometimes it doesn't really need to be
a probability to evaluate.

0:51:53.401 --> 0:52:05.492
It's only important that: And: Here is called
self-normalization.

0:52:05.492 --> 0:52:19.349
What people have done so is in the softmax
is always to the input divided by normalization.

0:52:19.759 --> 0:52:25.194
So this is how we calculate the soft mix.

0:52:25.825 --> 0:52:42.224
And in self-normalization now, the idea is
that we don't need to calculate the logarithm.

0:52:42.102 --> 0:52:54.284
That would be zero, and then you don't even
have to calculate the normalization.

0:52:54.514 --> 0:53:01.016
So how can we achieve that?

0:53:01.016 --> 0:53:08.680
And then there's the nice thing.

0:53:09.009 --> 0:53:14.743
And our novel Lots and more to maximize probability.

0:53:14.743 --> 0:53:23.831
We have this cross entry lot that probability
is higher, and now we're just adding.

0:53:24.084 --> 0:53:31.617
And the second loss just tells us you're pleased
training the way the lock set is zero.

0:53:32.352 --> 0:53:38.625
So then if it's nearly zero at the end you
don't need to calculate this and it's also

0:53:38.625 --> 0:53:39.792
very efficient.

0:53:40.540 --> 0:53:57.335
One important thing is this is only an inference,
so during tests we don't need to calculate.

0:54:00.480 --> 0:54:15.006
You can do a bit of a hyperparameter here
where you do the waiting and how much effort

0:54:15.006 --> 0:54:16.843
should be.

0:54:18.318 --> 0:54:35.037
The only disadvantage is that it's no speed
up during training and there are other ways

0:54:35.037 --> 0:54:37.887
of doing that.

0:54:41.801 --> 0:54:43.900
I'm with you all.

0:54:44.344 --> 0:54:48.540
Then we are coming very, very briefly like
this one here.

0:54:48.828 --> 0:54:53.692
There are more things on different types of
languages.

0:54:53.692 --> 0:54:58.026
We are having a very short view of a restricted.

0:54:58.298 --> 0:55:09.737
And then we'll talk about recurrent neural
networks for our language minds because they

0:55:09.737 --> 0:55:17.407
have the advantage now that we can't even further
improve.

0:55:18.238 --> 0:55:24.395
There's also different types of neural networks.

0:55:24.395 --> 0:55:30.175
These ballroom machines are not having input.

0:55:30.330 --> 0:55:39.271
They have these binary units: And they define
an energy function on the network, which can

0:55:39.271 --> 0:55:46.832
be in respect of bottom machines efficiently
calculated, and restricted needs.

0:55:46.832 --> 0:55:53.148
You only have connections between the input
and the hidden layer.

0:55:53.393 --> 0:56:00.190
So you see here you don't have input and output,
you just have an input and you calculate what.

0:56:00.460 --> 0:56:16.429
Which of course nicely fits with the idea
we're having, so you can use this for N gram

0:56:16.429 --> 0:56:19.182
language ones.

0:56:19.259 --> 0:56:25.187
Decaying this credibility of the input by
this type of neural networks.

0:56:26.406 --> 0:56:30.582
And the advantage of this type of model of
board that is.

0:56:30.550 --> 0:56:38.629
Very fast to integrate it, so that one was
the first one which was used during decoding.

0:56:38.938 --> 0:56:50.103
The problem of it is that the Enron language
models were very good at performing the calculation.

0:56:50.230 --> 0:57:00.114
So what people typically did is we talked
about a best list, so they generated a most

0:57:00.114 --> 0:57:05.860
probable output, and then they scored each
entry.

0:57:06.146 --> 0:57:10.884
A language model, and then only like change
the order against that based on that which.

0:57:11.231 --> 0:57:20.731
The knifing is maybe only hundred entries,
while during decoding you will look at several

0:57:20.731 --> 0:57:21.787
thousand.

0:57:26.186 --> 0:57:40.437
This but let's look at the context, so we
have now seen your language models.

0:57:40.437 --> 0:57:43.726
There is the big.

0:57:44.084 --> 0:57:57.552
Remember ingram language is not always words
because sometimes you have to back off or interpolation

0:57:57.552 --> 0:57:59.953
to lower ingrams.

0:58:00.760 --> 0:58:05.504
However, in neural models we always have all
of these inputs and some of these.

0:58:07.147 --> 0:58:21.262
The disadvantage is that you are still limited
in your context, and if you remember the sentence

0:58:21.262 --> 0:58:23.008
from last,.

0:58:22.882 --> 0:58:28.445
Sometimes you need more context and there's
unlimited contexts that you might need and

0:58:28.445 --> 0:58:34.838
you can always create sentences where you need
this file context in order to put a good estimation.

0:58:35.315 --> 0:58:44.955
Can we also do it different in order to better
understand that it makes sense to view?

0:58:45.445 --> 0:58:57.621
So sequence labeling tasks are a very common
type of towns in natural language processing

0:58:57.621 --> 0:59:03.438
where you have an input sequence and then.

0:59:03.323 --> 0:59:08.663
I've token so you have one output for each
input so machine translation is not a secret

0:59:08.663 --> 0:59:14.063
labeling cast because the number of inputs
and the number of outputs is different so you

0:59:14.063 --> 0:59:19.099
put in a string German which has five words
and the output can be six or seven or.

0:59:19.619 --> 0:59:20.155
Secrets.

0:59:20.155 --> 0:59:24.083
Lately you always have the same number of
and the same number of.

0:59:24.944 --> 0:59:40.940
And you can model language modeling as that,
and you just say a label for each word is always

0:59:40.940 --> 0:59:43.153
a next word.

0:59:45.705 --> 0:59:54.823
This is the more general you can think of
it, for example how to speech taking entity

0:59:54.823 --> 0:59:56.202
recognition.

0:59:58.938 --> 1:00:08.081
And if you look at now fruit cut token in
generally sequence, they can depend on import

1:00:08.081 --> 1:00:08.912
tokens.

1:00:09.869 --> 1:00:11.260
Nice thing.

1:00:11.260 --> 1:00:21.918
In our case, the output tokens are the same
so we can easily model it that they only depend

1:00:21.918 --> 1:00:24.814
on all the input tokens.

1:00:24.814 --> 1:00:28.984
So we have this whether it's or so.

1:00:31.011 --> 1:00:42.945
But we can always do a look at what specific
type of sequence labeling, unidirectional sequence

1:00:42.945 --> 1:00:44.188
labeling.

1:00:44.584 --> 1:00:58.215
And that's exactly how we want the language
of the next word only depends on all the previous

1:00:58.215 --> 1:01:00.825
words that we're.

1:01:01.321 --> 1:01:12.899
Mean, of course, that's not completely true
in a language that the bad context might also

1:01:12.899 --> 1:01:14.442
be helpful.

1:01:14.654 --> 1:01:22.468
We will model always the probability of a
word given on its history, and therefore we

1:01:22.468 --> 1:01:23.013
need.

1:01:23.623 --> 1:01:29.896
And currently we did there this approximation
in sequence labeling that we have this windowing

1:01:29.896 --> 1:01:30.556
approach.

1:01:30.951 --> 1:01:43.975
So in order to predict this type of word we
always look at the previous three words and

1:01:43.975 --> 1:01:48.416
then to do this one we again.

1:01:49.389 --> 1:01:55.137
If you are into neural networks you recognize
this type of structure.

1:01:55.137 --> 1:01:57.519
Also are the typical neural.

1:01:58.938 --> 1:02:09.688
Yes, so this is like Engram, Louis Couperus,
and at least in some way compared to the original,

1:02:09.688 --> 1:02:12.264
you're always looking.

1:02:14.334 --> 1:02:30.781
However, there are also other types of neural
network structures which we can use for sequence.

1:02:32.812 --> 1:02:34.678
That we can do so.

1:02:34.678 --> 1:02:39.686
The idea is in recurrent neural network structure.

1:02:39.686 --> 1:02:43.221
We are saving the complete history.

1:02:43.623 --> 1:02:55.118
So again we have to do like this fix size
representation because neural networks always

1:02:55.118 --> 1:02:56.947
need to have.

1:02:57.157 --> 1:03:05.258
And then we start with an initial value for
our storage.

1:03:05.258 --> 1:03:15.917
We are giving our first input and then calculating
the new representation.

1:03:16.196 --> 1:03:26.328
If you look at this, it's just again your
network was two types of inputs: in your work,

1:03:26.328 --> 1:03:29.743
in your initial hidden state.

1:03:30.210 --> 1:03:46.468
Then you can apply it to the next type of
input and you're again having.

1:03:47.367 --> 1:03:53.306
Nice thing is now that you can do now step
by step by step, so all the way over.

1:03:55.495 --> 1:04:05.245
The nice thing that we are having here now
is that we are having context information from

1:04:05.245 --> 1:04:07.195
all the previous.

1:04:07.607 --> 1:04:13.582
So if you're looking like based on which words
do you use here, calculate your ability of

1:04:13.582 --> 1:04:14.180
varying.

1:04:14.554 --> 1:04:20.128
It depends on is based on this path.

1:04:20.128 --> 1:04:33.083
It depends on and this hidden state was influenced
by this one and this hidden state.

1:04:33.473 --> 1:04:37.798
So now we're having something new.

1:04:37.798 --> 1:04:46.449
We can really model the word probability not
only on a fixed context.

1:04:46.906 --> 1:04:53.570
Because the in-states we're having here in
our area are influenced by all the trivia.

1:04:56.296 --> 1:05:00.909
So how is that to mean?

1:05:00.909 --> 1:05:16.288
If you're not thinking about the history of
clustering, we said the clustering.

1:05:16.736 --> 1:05:24.261
So do not need to do any clustering here,
and we also see how things are put together

1:05:24.261 --> 1:05:26.273
in order to really do.

1:05:29.489 --> 1:05:43.433
In the green box this way since we are starting
from the left point to the right.

1:05:44.524 --> 1:05:48.398
And that's right, so they're clustered in
some parts.

1:05:48.398 --> 1:05:58.196
Here is some type of clustering happening:
It's continuous representations, but a smaller

1:05:58.196 --> 1:06:02.636
difference doesn't matter again.

1:06:02.636 --> 1:06:10.845
So if you have a lot of different histories,
the similarity.

1:06:11.071 --> 1:06:15.791
Because in order to do the final restriction
you only do it based on the green box.

1:06:16.156 --> 1:06:24.284
So you are now again still learning some type
of clasp.

1:06:24.284 --> 1:06:30.235
You don't have to do this hard decision.

1:06:30.570 --> 1:06:39.013
The only restriction you are giving is you
have to install everything that is important.

1:06:39.359 --> 1:06:54.961
So it's a different type of limitation, so
you calculate the probability based on the

1:06:54.961 --> 1:06:57.138
last words.

1:06:57.437 --> 1:07:09.645
That is how you still need some cluster things
in order to do it efficiently.

1:07:09.970 --> 1:07:25.311
But this is where things get merged together
in this type of hidden representation, which

1:07:25.311 --> 1:07:28.038
is then merged.

1:07:28.288 --> 1:07:33.104
On the previous words, but they are some other
bottleneck in order to make a good estimation.

1:07:34.474 --> 1:07:41.242
So the idea is that we can store all our history
into one lecture.

1:07:41.581 --> 1:07:47.351
Which is very good and makes it more strong.

1:07:47.351 --> 1:07:51.711
Next we come to problems of that.

1:07:51.711 --> 1:07:57.865
Of course, at some point it might be difficult.

1:07:58.398 --> 1:08:02.230
Then maybe things get all overwritten, or
you cannot store everything in there.

1:08:02.662 --> 1:08:04.514
So,.

1:08:04.184 --> 1:08:10.252
Therefore, yet for short things like signal
sentences that works well, but especially if

1:08:10.252 --> 1:08:16.184
you think of other tasks like harmonisation
where a document based on T where you need

1:08:16.184 --> 1:08:22.457
to consider a full document, these things got
a bit more complicated and we learned another

1:08:22.457 --> 1:08:23.071
type of.

1:08:24.464 --> 1:08:30.455
For the further in order to understand these
networks, it's good to have both views always.

1:08:30.710 --> 1:08:39.426
So this is the unroll view, so you have this
type of network.

1:08:39.426 --> 1:08:48.532
Therefore, it can be shown as: We have here
the output and here's your network which is

1:08:48.532 --> 1:08:52.091
connected by itself and that is a recurrent.

1:08:56.176 --> 1:09:11.033
There is one challenge in these networks and
that is the training so the nice thing is train

1:09:11.033 --> 1:09:11.991
them.

1:09:12.272 --> 1:09:20.147
So the idea is we don't really know how to
train them, but if you unroll them like this,.

1:09:20.540 --> 1:09:38.054
It's exactly the same so you can measure your
arrows and then you propagate your arrows.

1:09:38.378 --> 1:09:45.647
Now the nice thing is if you unroll something,
it's a feet forward and you can train it.

1:09:46.106 --> 1:09:56.493
The only important thing is, of course, for
different inputs you have to take that into

1:09:56.493 --> 1:09:57.555
account.

1:09:57.837 --> 1:10:07.621
But since parameters are shared, it's somehow
similar and you can train that the training

1:10:07.621 --> 1:10:08.817
algorithm.

1:10:10.310 --> 1:10:16.113
One thing which makes things difficult is
what is referred to as the vanishing gradient.

1:10:16.113 --> 1:10:21.720
So we are saying there is a big advantage
of these models and that's why we are using

1:10:21.720 --> 1:10:22.111
that.

1:10:22.111 --> 1:10:27.980
The output here does not only depend on the
current input of a last three but on anything

1:10:27.980 --> 1:10:29.414
that was said before.

1:10:29.809 --> 1:10:32.803
That's a very strong thing is the motivation
of using art.

1:10:33.593 --> 1:10:44.599
However, if you're using standard, the influence
here gets smaller and smaller, and the models.

1:10:44.804 --> 1:10:55.945
Because the gradients get smaller and smaller,
and so the arrow here propagated to this one,

1:10:55.945 --> 1:10:59.659
this contributes to the arrow.

1:11:00.020 --> 1:11:06.710
And yeah, that's why standard R&S are
difficult or have to become boosters.

1:11:07.247 --> 1:11:11.481
So if we are talking about our ends nowadays,.

1:11:11.791 --> 1:11:19.532
What we are typically meaning are long short
memories.

1:11:19.532 --> 1:11:30.931
You see there by now quite old already, but
they have special gating mechanisms.

1:11:31.171 --> 1:11:41.911
So in the language model tasks, for example
in some other story information, all this sentence

1:11:41.911 --> 1:11:44.737
started with a question.

1:11:44.684 --> 1:11:51.886
Because if you only look at the five last
five words, it's often no longer clear as a

1:11:51.886 --> 1:11:52.556
normal.

1:11:53.013 --> 1:12:06.287
So there you have these mechanisms with the
right gate in order to store things for a longer

1:12:06.287 --> 1:12:08.571
time into your.

1:12:10.730 --> 1:12:20.147
Here they are used in, in, in, in selling
quite a lot of works.

1:12:21.541 --> 1:12:30.487
For especially text machine translation now,
the standard is to do transformer base models.

1:12:30.690 --> 1:12:42.857
But for example, this type of in architecture
we have later one lecture about efficiency.

1:12:42.882 --> 1:12:53.044
And there in the decoder and partial networks
they are still using our edges because then.

1:12:53.473 --> 1:12:57.542
So it's not that our ends are of no importance.

1:12:59.239 --> 1:13:08.956
In order to make them strong, there are some
more things which are helpful and should be:

1:13:09.309 --> 1:13:19.668
So one thing is it's a very easy and nice trick
to make this neon network stronger and better.

1:13:19.739 --> 1:13:21.619
So, of course, it doesn't work always.

1:13:21.619 --> 1:13:23.451
They have to have enough training to.

1:13:23.763 --> 1:13:29.583
But in general that is the easiest way of
making your mouth bigger and stronger is to

1:13:29.583 --> 1:13:30.598
increase your.

1:13:30.630 --> 1:13:43.244
And you've seen that with a large size model
they are always braggling about.

1:13:43.903 --> 1:13:53.657
This is one way so the question is how do
you get more parameters?

1:13:53.657 --> 1:14:05.951
There's two ways you can make your representations:
And the other thing is its octave deep learning,

1:14:05.951 --> 1:14:10.020
so the other thing is to make your networks.

1:14:11.471 --> 1:14:13.831
And then you can also get more work off.

1:14:14.614 --> 1:14:19.931
There's one problem with this and with more
deeper networks.

1:14:19.931 --> 1:14:23.330
It's very similar to what we saw with.

1:14:23.603 --> 1:14:34.755
With the we have this problem of radiant flow
that if it flows so fast like the radiant gets

1:14:34.755 --> 1:14:35.475
very.

1:14:35.795 --> 1:14:41.114
Exactly the same thing happens in deep.

1:14:41.114 --> 1:14:52.285
If you take the gradient and tell it's the
right or wrong, then you're propagating.

1:14:52.612 --> 1:14:53.228
Three layers.

1:14:53.228 --> 1:14:56.440
It's no problem, but if you're going to ten,
twenty or a hundred layers.

1:14:57.797 --> 1:14:59.690
That is getting typically a problem.

1:15:00.060 --> 1:15:10.659
People are doing and they are using what is
called visual connections.

1:15:10.659 --> 1:15:15.885
That's a very helpful idea, which.

1:15:15.956 --> 1:15:20.309
And so the idea is that these networks.

1:15:20.320 --> 1:15:30.694
In between should calculate really what is
a new representation, but they are calculating

1:15:30.694 --> 1:15:31.386
what.

1:15:31.731 --> 1:15:37.585
And therefore in the end you'll always the
output of a layer is added with the input.

1:15:38.318 --> 1:15:48.824
The nice thing is that later, if you are doing
back propagation with this very fast back,.

1:15:49.209 --> 1:16:01.896
So that is what you're seeing nowadays in
very deep architectures, not only as others,

1:16:01.896 --> 1:16:04.229
but you always.

1:16:04.704 --> 1:16:07.388
Has two advantages.

1:16:07.388 --> 1:16:15.304
On the one hand, it's more easy to learn a
representation.

1:16:15.304 --> 1:16:18.792
On the other hand, these.

1:16:22.082 --> 1:16:24.114
Goods.

1:16:23.843 --> 1:16:31.763
That much for the new record before, so the
last thing now means this.

1:16:31.671 --> 1:16:36.400
Language was used in the molds itself.

1:16:36.400 --> 1:16:46.707
Now we're seeing them again, but one thing
that at the beginning was very essential.

1:16:46.967 --> 1:16:57.655
So people really train part in the language
models only to get this type of embeddings

1:16:57.655 --> 1:17:04.166
and therefore we want to look a bit more into
these.

1:17:09.229 --> 1:17:13.456
Some laugh words to the word embeddings.

1:17:13.456 --> 1:17:22.117
The interesting thing is that word embeddings
can be used for very different tasks.

1:17:22.117 --> 1:17:27.170
The advantage is we can train the word embedded.

1:17:27.347 --> 1:17:31.334
The knife is you can train that on just large
amounts of data.

1:17:31.931 --> 1:17:40.937
And then if you have these wooden beddings
you don't have a layer of ten thousand any

1:17:40.937 --> 1:17:41.566
more.

1:17:41.982 --> 1:17:52.231
So then you can train a small market to do
any other tasks and therefore you're more.

1:17:52.532 --> 1:17:58.761
Initial word embeddings really depend only
on the word itself.

1:17:58.761 --> 1:18:07.363
If you look at the two meanings of can, the
can of beans, or can they do that, some of

1:18:07.363 --> 1:18:08.747
the embedded.

1:18:09.189 --> 1:18:12.395
That cannot be resolved.

1:18:12.395 --> 1:18:23.939
Therefore, you need to know the context, and
if you look at the higher levels that people

1:18:23.939 --> 1:18:27.916
are doing in the context, but.

1:18:29.489 --> 1:18:33.757
However, even this one has quite very interesting.

1:18:34.034 --> 1:18:44.644
So people like to visualize that they're always
a bit difficult because if you look at this

1:18:44.644 --> 1:18:47.182
word, vector or word.

1:18:47.767 --> 1:18:52.879
And drawing your five hundred dimensional
vector is still a bit challenging.

1:18:53.113 --> 1:19:12.464
So you cannot directly do that, so what people
have to do is learn some type of dimension.

1:19:13.073 --> 1:19:17.216
And of course then yes some information gets
lost but you can try it.

1:19:18.238 --> 1:19:28.122
And you see, for example, this is the most
famous and common example, so what you can

1:19:28.122 --> 1:19:37.892
look is you can look at the difference between
the male and the female word English.

1:19:38.058 --> 1:19:40.389
And you can do that for a very different work.

1:19:40.780 --> 1:19:45.403
And that is where, where the masks come into
that, what people then look into.

1:19:45.725 --> 1:19:50.995
So what you can now, for example, do is you
can calculate the difference between man and

1:19:50.995 --> 1:19:51.410
woman.

1:19:52.232 --> 1:19:56.356
And what you can do then you can take the
embedding of peeing.

1:19:56.356 --> 1:20:02.378
You can add on it the difference between men
and women and where people get really excited.

1:20:02.378 --> 1:20:05.586
Then you can look at what are the similar
words.

1:20:05.586 --> 1:20:09.252
So you won't, of course, directly hit the
correct word.

1:20:09.252 --> 1:20:10.495
It's a continuous.

1:20:10.790 --> 1:20:24.062
But you can look at what are the nearest neighbors
to the same, and often these words are near.

1:20:24.224 --> 1:20:33.911
So it's somehow weird that the difference
between these works is always the same.

1:20:34.374 --> 1:20:37.308
Can do different things.

1:20:37.308 --> 1:20:47.520
You can also imagine that the work tends to
be assuming and swim, and with walking and

1:20:47.520 --> 1:20:49.046
walking you.

1:20:49.469 --> 1:20:53.040
So you can try to use him.

1:20:53.040 --> 1:20:56.346
It's no longer like say.

1:20:56.346 --> 1:21:04.016
The interesting thing is nobody taught him
the principle.

1:21:04.284 --> 1:21:09.910
So it's purely trained on the task of doing
the next work prediction.

1:21:10.230 --> 1:21:23.669
And even for some information like the capital,
this is the difference between the capital.

1:21:23.823 --> 1:21:33.760
Is another visualization here where you have
done the same things on the difference between.

1:21:33.853 --> 1:21:41.342
And you see it's not perfect, but it's building
in my directory, so you can even use that for

1:21:41.342 --> 1:21:42.936
pressure answering.

1:21:42.936 --> 1:21:50.345
If you have no three countries, the capital,
you can do what is the difference between them.

1:21:50.345 --> 1:21:53.372
You apply that to a new country, and.

1:21:54.834 --> 1:22:02.280
So these models are able to really learn a
lot of information and collapse this information

1:22:02.280 --> 1:22:04.385
into this representation.

1:22:05.325 --> 1:22:07.679
And just to do the next two are predictions.

1:22:07.707 --> 1:22:22.358
And that also explains a bit maybe or explains
strongly, but motivates what is the main advantage

1:22:22.358 --> 1:22:26.095
of this type of neurons.

1:22:28.568 --> 1:22:46.104
So to summarize what we did today, so what
you should hopefully have with you is: Then

1:22:46.104 --> 1:22:49.148
how we can do language modeling with new networks.

1:22:49.449 --> 1:22:55.445
We looked at three different architectures:
We looked into the feet forward language one,

1:22:55.445 --> 1:22:59.059
the R&N, and the one based the balsamic.

1:22:59.039 --> 1:23:04.559
And finally, there are different architectures
to do in neural networks.

1:23:04.559 --> 1:23:10.986
We have seen feet for neural networks and
base neural networks, and we'll see in the

1:23:10.986 --> 1:23:14.389
next lectures the last type of architecture.

1:23:15.915 --> 1:23:17.438
Any questions.

1:23:20.680 --> 1:23:27.360
Then thanks a lot, and next I'm just there,
we'll be again on order to.