diff --git "a/demo_data/lectures/Lecture-11-15.06.2023/English.vtt" "b/demo_data/lectures/Lecture-11-15.06.2023/English.vtt"
new file mode 100644--- /dev/null
+++ "b/demo_data/lectures/Lecture-11-15.06.2023/English.vtt"
@@ -0,0 +1,12362 @@
+WEBVTT
+
+0:00:00.981 --> 0:00:20.036
+Today about is how to use some type of additional
+resources to improve the translation.
+
+0:00:20.300 --> 0:00:28.188
+We have in the first part of the semester
+two thirds of the semester how to build some
+
+0:00:28.188 --> 0:00:31.361
+of your basic machine translation.
+
+0:00:31.571 --> 0:00:42.317
+Now the basic components are both for statistical
+and for neural, with the encoded decoding.
+
+0:00:43.123 --> 0:00:46.000
+Now, of course, that's not where it stops.
+
+0:00:46.000 --> 0:00:51.286
+It's still what nearly every machine translation
+system is currently in there.
+
+0:00:51.286 --> 0:00:57.308
+However, there's a lot of challenges which
+you need to address in addition and which need
+
+0:00:57.308 --> 0:00:58.245
+to be solved.
+
+0:00:58.918 --> 0:01:09.858
+And there we want to start to tell you what
+else can you do around this, and partly.
+
+0:01:10.030 --> 0:01:14.396
+And one important question there is on what
+do you train your models?
+
+0:01:14.394 --> 0:01:32.003
+Because like this type of parallel data, it's
+easier in machine translation than in other
+
+0:01:32.003 --> 0:01:33.569
+trusts.
+
+0:01:33.853 --> 0:01:41.178
+And therefore an important question is, can
+we also learn from like other sources and through?
+
+0:01:41.701 --> 0:01:47.830
+Because if you remember strongly right at
+the beginning of the election,.
+
+0:01:51.171 --> 0:01:53.801
+This Is How We Train All Our.
+
+0:01:54.194 --> 0:01:59.887
+Machine learning models from statistical to
+neural.
+
+0:01:59.887 --> 0:02:09.412
+This doesn't have changed so we need this
+type of parallel data where we have a source
+
+0:02:09.412 --> 0:02:13.462
+sentence aligned with a target data.
+
+0:02:13.493 --> 0:02:19.135
+We have now a strong model here, a very good
+model to do that.
+
+0:02:19.135 --> 0:02:22.091
+However, we always rely on this.
+
+0:02:22.522 --> 0:02:28.395
+For languages, high risk language pairs say
+from German to English or other European languages,
+
+0:02:28.395 --> 0:02:31.332
+there is decent amount, at least for similarly.
+
+0:02:31.471 --> 0:02:37.630
+But even there if we are going to very specific
+domains it might get difficult and then your
+
+0:02:37.630 --> 0:02:43.525
+system performance might drop because if you
+want to translate now some medical text for
+
+0:02:43.525 --> 0:02:50.015
+example of course you need to also have peril
+data in the medical domain to know how to translate
+
+0:02:50.015 --> 0:02:50.876
+these types.
+
+0:02:51.231 --> 0:02:55.264
+Phrases how to use the vocabulary and so on
+in the style.
+
+0:02:55.915 --> 0:03:04.887
+And if you are going to other languages, there
+is a lot bigger challenge and the question
+
+0:03:04.887 --> 0:03:05.585
+there.
+
+0:03:05.825 --> 0:03:09.649
+So is really this the only resource we can
+use.
+
+0:03:09.889 --> 0:03:19.462
+Can be adapted or training phase in order
+to also make use of other types of models that
+
+0:03:19.462 --> 0:03:27.314
+might enable us to build strong systems with
+other types of information.
+
+0:03:27.707 --> 0:03:35.276
+And that we will look into now in the next
+starting from from just saying the next election.
+
+0:03:35.515 --> 0:03:40.697
+So this idea we already have covered on Tuesday.
+
+0:03:40.697 --> 0:03:45.350
+One very successful idea for this is to do.
+
+0:03:45.645 --> 0:03:51.990
+So that we're no longer doing translation
+between languages, but we can do translation
+
+0:03:51.990 --> 0:03:55.928
+between languages and share common knowledge
+between.
+
+0:03:56.296 --> 0:04:04.703
+And you also learned about things like zero
+shots machine translation so you can translate
+
+0:04:04.703 --> 0:04:06.458
+between languages.
+
+0:04:06.786 --> 0:04:09.790
+Which is the case for many, many language
+pairs.
+
+0:04:10.030 --> 0:04:19.209
+Like even with German, you have not translation
+parallel data to all languages around the world,
+
+0:04:19.209 --> 0:04:26.400
+or most of them you have it to the Europeans
+once, maybe even for Japanese.
+
+0:04:26.746 --> 0:04:35.332
+There is quite a lot of data, for example
+English to Japanese, but German to Japanese
+
+0:04:35.332 --> 0:04:37.827
+or German to Vietnamese.
+
+0:04:37.827 --> 0:04:41.621
+There is some data from Multilingual.
+
+0:04:42.042 --> 0:04:54.584
+So there is a very promising direction if
+you want to build translation systems between
+
+0:04:54.584 --> 0:05:00.142
+language peers, typically not English.
+
+0:05:01.221 --> 0:05:05.887
+And the other ideas, of course, we don't have
+to either just search for it.
+
+0:05:06.206 --> 0:05:12.505
+Some work on a data crawling so if I don't
+have a corpus directly or I don't have an high
+
+0:05:12.505 --> 0:05:19.014
+quality corpus like from the European Parliament
+for a TED corpus so maybe it makes sense to
+
+0:05:19.014 --> 0:05:23.913
+crawl more data and get additional sources
+so you can build stronger.
+
+0:05:24.344 --> 0:05:35.485
+There has been quite a big effort in Europe
+to collect really large data sets for parallel
+
+0:05:35.485 --> 0:05:36.220
+data.
+
+0:05:36.220 --> 0:05:40.382
+How can we do this data crawling?
+
+0:05:40.600 --> 0:05:46.103
+There the interesting thing from the machine
+translation point is not just general data
+
+0:05:46.103 --> 0:05:46.729
+crawling.
+
+0:05:47.067 --> 0:05:50.037
+But how can we explicitly crawl data?
+
+0:05:50.037 --> 0:05:52.070
+Which is some of a peril?
+
+0:05:52.132 --> 0:05:58.461
+So there is in the Internet quite a lot of
+data which has been company websites which
+
+0:05:58.461 --> 0:06:01.626
+have been translated and things like that.
+
+0:06:01.626 --> 0:06:05.158
+So how can you extract them parallel fragments?
+
+0:06:06.566 --> 0:06:13.404
+That is typically more noisy than where you
+do more at hands where mean if you have Parliament.
+
+0:06:13.693 --> 0:06:17.680
+You can do some rules how to extract parallel
+things.
+
+0:06:17.680 --> 0:06:24.176
+Here there is more to it, so the quality is
+later maybe not as good, but normally scale
+
+0:06:24.176 --> 0:06:26.908
+is then a possibility to address it.
+
+0:06:26.908 --> 0:06:30.304
+So you just have so much more data that even.
+
+0:06:33.313 --> 0:06:40.295
+The other thing can be used monolingual data
+and monolingual data has a big advantage that
+
+0:06:40.295 --> 0:06:46.664
+we can have a huge amount of that so that you
+can be autocrawed from the Internet.
+
+0:06:46.664 --> 0:06:51.728
+The nice thing is you can also get it typically
+for many domains.
+
+0:06:52.352 --> 0:06:59.558
+There is just so much more magnitude of monolingual
+data so that it might be very helpful.
+
+0:06:59.559 --> 0:07:03.054
+We can do that in statistical machine translation.
+
+0:07:03.054 --> 0:07:06.755
+It was quite easy to integrate using language
+models.
+
+0:07:08.508 --> 0:07:16.912
+In neural machine translation we have the
+advantage that we have this overall architecture
+
+0:07:16.912 --> 0:07:22.915
+that does everything together, but it has also
+the disadvantage.
+
+0:07:23.283 --> 0:07:25.675
+We'll look today at two things.
+
+0:07:25.675 --> 0:07:32.925
+On the one end you can still try to do a bit
+of language modeling in there and add an additional
+
+0:07:32.925 --> 0:07:35.168
+language model into in there.
+
+0:07:35.168 --> 0:07:38.232
+There is some work, one very successful.
+
+0:07:38.178 --> 0:07:43.764
+A way in which I think is used in most systems
+at the moment is to do some scientific data.
+
+0:07:43.763 --> 0:07:53.087
+Is a very easy thing, but you can just translate
+there and use it as training gator, and normally.
+
+0:07:53.213 --> 0:07:59.185
+And thereby you are able to use like some
+type of monolingual a day.
+
+0:08:00.380 --> 0:08:05.271
+Another way to do it is unsupervised and the
+extreme case.
+
+0:08:05.271 --> 0:08:11.158
+If you have a scenario then you only have
+data, only monolingual data.
+
+0:08:11.158 --> 0:08:13.976
+Can you still build translations?
+
+0:08:14.754 --> 0:08:27.675
+If you have large amounts of data and languages
+are not too dissimilar, you can build translation
+
+0:08:27.675 --> 0:08:31.102
+systems without parallel.
+
+0:08:32.512 --> 0:08:36.267
+That we will see you then next Thursday.
+
+0:08:37.857 --> 0:08:50.512
+And then there is now a third type of pre-trained
+model that recently became very successful
+
+0:08:50.512 --> 0:08:55.411
+and now with large language models.
+
+0:08:55.715 --> 0:09:03.525
+So the idea is we are no longer sharing the
+real data, but it can also help to train a
+
+0:09:03.525 --> 0:09:04.153
+model.
+
+0:09:04.364 --> 0:09:11.594
+And that is now a big advantage of deep learning
+based approaches.
+
+0:09:11.594 --> 0:09:22.169
+There you have this ability that you can train
+a model in some task and then apply it to another.
+
+0:09:22.722 --> 0:09:33.405
+And then, of course, the question is, can
+I have an initial task where there's huge amounts
+
+0:09:33.405 --> 0:09:34.450
+of data?
+
+0:09:34.714 --> 0:09:40.251
+And the test that typically you pre train
+on is more like similar to a language moral
+
+0:09:40.251 --> 0:09:45.852
+task either direct to a language moral task
+or like a masking task which is related so
+
+0:09:45.852 --> 0:09:51.582
+the idea is oh I can train on this data and
+the knowledge about words how they relate to
+
+0:09:51.582 --> 0:09:53.577
+each other I can use in there.
+
+0:09:53.753 --> 0:10:00.276
+So it's a different way of using language
+models.
+
+0:10:00.276 --> 0:10:06.276
+There's more transfer learning at the end
+of.
+
+0:10:09.029 --> 0:10:17.496
+So first we will start with how can we use
+monolingual data to do a Yeah to do a machine
+
+0:10:17.496 --> 0:10:18.733
+translation?
+
+0:10:20.040 --> 0:10:27.499
+That: Big difference is you should remember
+from what I mentioned before is.
+
+0:10:27.499 --> 0:10:32.783
+In statistical machine translation we directly
+have the opportunity.
+
+0:10:32.783 --> 0:10:39.676
+There's peril data for the translation model
+and monolingual data for the language model.
+
+0:10:39.679 --> 0:10:45.343
+And you combine your translation model and
+language model, and then you can make use of
+
+0:10:45.343 --> 0:10:45.730
+both.
+
+0:10:46.726 --> 0:10:53.183
+That you can make use of these large large
+amounts of monolingual data, but of course
+
+0:10:53.183 --> 0:10:55.510
+it has also some disadvantage.
+
+0:10:55.495 --> 0:11:01.156
+Because we say the problem is we are optimizing
+both parts a bit independently to each other
+
+0:11:01.156 --> 0:11:06.757
+and we say oh yeah the big disadvantage of
+newer machine translations now we are optimizing
+
+0:11:06.757 --> 0:11:10.531
+the overall architecture everything together
+to perform best.
+
+0:11:10.890 --> 0:11:16.994
+And then, of course, we can't do there, so
+Leo we can can only do a mural like use power
+
+0:11:16.994 --> 0:11:17.405
+data.
+
+0:11:17.897 --> 0:11:28.714
+So the question is, but this advantage is
+not so important that we can train everything,
+
+0:11:28.714 --> 0:11:35.276
+but we have a moral legal data or even small
+amounts.
+
+0:11:35.675 --> 0:11:43.102
+So in data we know it's not only important
+the amount of data we have but also like how
+
+0:11:43.102 --> 0:11:50.529
+similar it is to your test data so it can be
+that this modeling data is quite small but
+
+0:11:50.529 --> 0:11:55.339
+it's very well fitting and then it's still
+very helpful.
+
+0:11:55.675 --> 0:12:02.691
+At the first year of surprisingness, if we
+are here successful with integrating a language
+
+0:12:02.691 --> 0:12:09.631
+model into a translation system, maybe we can
+also integrate some type of language models
+
+0:12:09.631 --> 0:12:14.411
+into our empty system in order to make it better
+and perform.
+
+0:12:16.536 --> 0:12:23.298
+The first thing we can do is we know there
+is language models, so let's try to integrate.
+
+0:12:23.623 --> 0:12:31.096
+There was our language model because these
+works were mainly done before transformer-based
+
+0:12:31.096 --> 0:12:31.753
+models.
+
+0:12:32.152 --> 0:12:38.764
+In general, of course, you can do the same
+thing with transformer baseball.
+
+0:12:38.764 --> 0:12:50.929
+There is nothing about whether: It's just
+that it has mainly been done before people
+
+0:12:50.929 --> 0:13:01.875
+started using R&S and they tried to do
+this more in cases.
+
+0:13:07.087 --> 0:13:22.938
+So what we're happening here is in some of
+this type of idea, and in key system you remember
+
+0:13:22.938 --> 0:13:25.495
+the attention.
+
+0:13:25.605 --> 0:13:29.465
+Gets it was your last in this day that you
+calculate easy attention.
+
+0:13:29.729 --> 0:13:36.610
+We get the context back, then combine both
+and then base the next in state and then predict.
+
+0:13:37.057 --> 0:13:42.424
+So this is our system, and the question is,
+can we send our integrated language model?
+
+0:13:42.782 --> 0:13:49.890
+And somehow it makes sense to take out a neural
+language model because we are anyway in the
+
+0:13:49.890 --> 0:13:50.971
+neural space.
+
+0:13:50.971 --> 0:13:58.465
+It's not surprising that it contrasts to statistical
+work used and grants it might make sense to
+
+0:13:58.465 --> 0:14:01.478
+take a bit of a normal language model.
+
+0:14:01.621 --> 0:14:06.437
+And there would be something like on Tubbles
+Air, a neural language model, and our man based
+
+0:14:06.437 --> 0:14:11.149
+is you have a target word, you put it in, you
+get a new benchmark, and then you always put
+
+0:14:11.149 --> 0:14:15.757
+in the words and get new hidden states, and
+you can do some predictions at the output to
+
+0:14:15.757 --> 0:14:16.948
+predict the next word.
+
+0:14:17.597 --> 0:14:26.977
+So if we're having this type of in language
+model, there's like two main questions we have
+
+0:14:26.977 --> 0:14:34.769
+to answer: So how do we combine now on the
+one hand our system and on the other hand our
+
+0:14:34.769 --> 0:14:35.358
+model?
+
+0:14:35.358 --> 0:14:42.004
+You see that was mentioned before when we
+started talking about ENCODA models.
+
+0:14:42.004 --> 0:14:45.369
+They can be viewed as a language model.
+
+0:14:45.805 --> 0:14:47.710
+The wine is lengthened, unconditioned.
+
+0:14:47.710 --> 0:14:49.518
+It's just modeling the target sides.
+
+0:14:49.970 --> 0:14:56.963
+And the other one is a conditional language
+one, which is a language one conditioned on
+
+0:14:56.963 --> 0:14:57.837
+the Sewer.
+
+0:14:58.238 --> 0:15:03.694
+So how can you combine to language models?
+
+0:15:03.694 --> 0:15:14.860
+Of course, it's like the translation model
+will be more important because it has access
+
+0:15:14.860 --> 0:15:16.763
+to the source.
+
+0:15:18.778 --> 0:15:22.571
+If we have that, the other question is okay.
+
+0:15:22.571 --> 0:15:24.257
+Now we have models.
+
+0:15:24.257 --> 0:15:25.689
+How do we train?
+
+0:15:26.026 --> 0:15:30.005
+Pickers integrated them.
+
+0:15:30.005 --> 0:15:34.781
+We have now two sets of data.
+
+0:15:34.781 --> 0:15:42.741
+We have parallel data where you can do the
+lower.
+
+0:15:44.644 --> 0:15:53.293
+So the first idea is we can do something more
+like a parallel combination.
+
+0:15:53.293 --> 0:15:55.831
+We just keep running.
+
+0:15:56.036 --> 0:15:59.864
+So here you see your system that is running.
+
+0:16:00.200 --> 0:16:09.649
+It's normally completely independent of your
+language model, which is up there, so down
+
+0:16:09.649 --> 0:16:13.300
+here we have just our NMT system.
+
+0:16:13.313 --> 0:16:26.470
+The only thing which is used is we have the
+words, and of course they are put into both
+
+0:16:26.470 --> 0:16:30.059
+systems, and out there.
+
+0:16:30.050 --> 0:16:42.221
+So we use them somehow for both, and then
+we are doing our decision just by merging these
+
+0:16:42.221 --> 0:16:42.897
+two.
+
+0:16:43.343 --> 0:16:53.956
+So there can be, for example, we are doing
+a probability distribution here, and then we
+
+0:16:53.956 --> 0:17:03.363
+are taking the average of post-perability distribution
+to do our predictions.
+
+0:17:11.871 --> 0:17:18.923
+You could also take the output with Steve's
+to be more in chore about the mixture.
+
+0:17:20.000 --> 0:17:32.896
+Yes, you could also do that, so it's more
+like engaging mechanisms that you're not doing.
+
+0:17:32.993 --> 0:17:41.110
+Another one would be cochtrinate the hidden
+states, and then you would have another layer
+
+0:17:41.110 --> 0:17:41.831
+on top.
+
+0:17:43.303 --> 0:17:56.889
+You think about if you do the conqueredination
+instead of taking the instead and then merging
+
+0:17:56.889 --> 0:18:01.225
+the probability distribution.
+
+0:18:03.143 --> 0:18:16.610
+Introduce many new parameters, and these parameters
+have somehow something special compared to
+
+0:18:16.610 --> 0:18:17.318
+the.
+
+0:18:23.603 --> 0:18:37.651
+So before all the error other parameters can
+be trained independent, the language model
+
+0:18:37.651 --> 0:18:42.121
+can be trained independent.
+
+0:18:43.043 --> 0:18:51.749
+If you have a joint layer, of course you need
+to train them because you have now inputs.
+
+0:18:54.794 --> 0:19:02.594
+Not surprisingly, if you have a parallel combination
+of whether you could, the other way is to do
+
+0:19:02.594 --> 0:19:04.664
+more serial combinations.
+
+0:19:04.924 --> 0:19:10.101
+How can you do a similar combination?
+
+0:19:10.101 --> 0:19:18.274
+Your final decision makes sense to do a face
+on the system.
+
+0:19:18.438 --> 0:19:20.996
+So you have on top of your normal and system.
+
+0:19:21.121 --> 0:19:30.678
+The only thing is now you're inputting into
+your system.
+
+0:19:30.678 --> 0:19:38.726
+You're no longer inputting the word embeddings.
+
+0:19:38.918 --> 0:19:45.588
+So you're training your mainly what you have
+your lower layers here which are trained more
+
+0:19:45.588 --> 0:19:52.183
+on the purely language model style and then
+on top your putting into the NMT system where
+
+0:19:52.183 --> 0:19:55.408
+it now has already here the language model.
+
+0:19:55.815 --> 0:19:58.482
+So here you can also view it.
+
+0:19:58.482 --> 0:20:06.481
+Here you have more contextual embeddings which
+no longer depend only on the word but they
+
+0:20:06.481 --> 0:20:10.659
+also depend on the context of the target site.
+
+0:20:11.051 --> 0:20:19.941
+But you have more understanding of the source
+word, so you have a language in the current
+
+0:20:19.941 --> 0:20:21.620
+target sentence.
+
+0:20:21.881 --> 0:20:27.657
+So if it's like the word can, for example,
+will be put in here always the same independent
+
+0:20:27.657 --> 0:20:31.147
+of its user can of beans, or if it's like I
+can do it.
+
+0:20:31.147 --> 0:20:37.049
+However, because you are having your language
+model style, you have maybe disintegrated this
+
+0:20:37.049 --> 0:20:40.984
+already a bit, and you give this information
+directly to the.
+
+0:20:41.701 --> 0:20:43.095
+An empty cyst.
+
+0:20:44.364 --> 0:20:49.850
+You, if you're remembering more the transformer
+based approach, you have some layers.
+
+0:20:49.850 --> 0:20:55.783
+The lower layers are purely languaged while
+the other ones are with attention to the source.
+
+0:20:55.783 --> 0:21:01.525
+So you can view it also that you just have
+lower layers which don't attend to the source.
+
+0:21:02.202 --> 0:21:07.227
+This is purely a language model, and then
+at some point you're starting to attend to
+
+0:21:07.227 --> 0:21:08.587
+the source and use it.
+
+0:21:13.493 --> 0:21:20.781
+Yes, so this is how you combine them in peril
+or first do the language model and then do.
+
+0:21:23.623 --> 0:21:26.147
+Questions for the integration.
+
+0:21:31.831 --> 0:21:35.034
+Not really sure about the input of the.
+
+0:21:35.475 --> 0:21:38.102
+Model, and in this case in the sequence.
+
+0:21:38.278 --> 0:21:53.199
+Case so the actual word that we transferred
+into a numerical lecture, and this is an input
+
+0:21:53.199 --> 0:21:54.838
+into the.
+
+0:21:56.176 --> 0:22:03.568
+That depends on if you view the word embedding
+as part of the language model.
+
+0:22:03.568 --> 0:22:10.865
+So if you first put the word target word then
+you do the one hot end coding.
+
+0:22:11.691 --> 0:22:13.805
+And then the word embedding there is the r&
+
+0:22:13.805 --> 0:22:13.937
+n.
+
+0:22:14.314 --> 0:22:21.035
+So you can use this together as your language
+model when you first do the word embedding.
+
+0:22:21.401 --> 0:22:24.346
+All you can say is like before.
+
+0:22:24.346 --> 0:22:28.212
+It's more a definition, but you're right.
+
+0:22:28.212 --> 0:22:30.513
+So what's the steps out?
+
+0:22:30.513 --> 0:22:36.128
+You take the word, the one hut encoding, the
+word embedding.
+
+0:22:36.516 --> 0:22:46.214
+What one of these parrots, you know, called
+a language model is definition wise and not
+
+0:22:46.214 --> 0:22:47.978
+that important.
+
+0:22:53.933 --> 0:23:02.264
+So the question is how can you then train
+them and make this this one work?
+
+0:23:02.264 --> 0:23:02.812
+The.
+
+0:23:03.363 --> 0:23:15.201
+So in the case where you combine the language
+one of the abilities you can train them independently
+
+0:23:15.201 --> 0:23:18.516
+and just put them together.
+
+0:23:18.918 --> 0:23:27.368
+Might not be the best because we have no longer
+the stability that we had before that optimally
+
+0:23:27.368 --> 0:23:29.128
+performed together.
+
+0:23:29.128 --> 0:23:33.881
+It's not clear if they really work the best
+together.
+
+0:23:34.514 --> 0:23:41.585
+At least you need to somehow find how much
+do you trust the one model and how much.
+
+0:23:43.323 --> 0:23:45.058
+Still in some cases useful.
+
+0:23:45.058 --> 0:23:48.530
+It might be helpful if you have only data
+and software.
+
+0:23:48.928 --> 0:23:59.064
+However, in MT we have one specific situation
+that at least for the MT part parallel is also
+
+0:23:59.064 --> 0:24:07.456
+always monolingual data, so what we definitely
+can do is train the language.
+
+0:24:08.588 --> 0:24:18.886
+So what we also can do is more like the pre-training
+approach.
+
+0:24:18.886 --> 0:24:24.607
+We first train the language model.
+
+0:24:24.704 --> 0:24:27.334
+The pre-training approach.
+
+0:24:27.334 --> 0:24:33.470
+You first train on the monolingual data and
+then you join the.
+
+0:24:33.933 --> 0:24:41.143
+Of course, the model size is this way, but
+the data size is too bigly the other way around.
+
+0:24:41.143 --> 0:24:47.883
+You often have a lot more monolingual data
+than you have here parallel data, in which
+
+0:24:47.883 --> 0:24:52.350
+scenario can you imagine where this type of
+pretraining?
+
+0:24:56.536 --> 0:24:57.901
+Any Ideas.
+
+0:25:04.064 --> 0:25:12.772
+One example where this might also be helpful
+if you want to adapt to domains.
+
+0:25:12.772 --> 0:25:22.373
+So let's say you do medical sentences and
+if you want to translate medical sentences.
+
+0:25:23.083 --> 0:25:26.706
+In this case it could be or its most probable
+happen.
+
+0:25:26.706 --> 0:25:32.679
+You're learning here up there what medical
+means, but in your fine tuning step the model
+
+0:25:32.679 --> 0:25:38.785
+is forgotten everything about Medicare, so
+you may be losing all the information you gain.
+
+0:25:39.099 --> 0:25:42.366
+So this type of priest training step is good.
+
+0:25:42.366 --> 0:25:47.978
+If your pretraining data is more general,
+very large and then you're adapting.
+
+0:25:48.428 --> 0:25:56.012
+But in the task with moral lingual data, which
+should be used to adapt the system to some
+
+0:25:56.012 --> 0:25:57.781
+general topic style.
+
+0:25:57.817 --> 0:26:06.795
+Then, of course, this is not a good strategy
+because you might forgot about everything up
+
+0:26:06.795 --> 0:26:09.389
+there and you don't have.
+
+0:26:09.649 --> 0:26:14.678
+So then you have to check what you can do
+for them.
+
+0:26:14.678 --> 0:26:23.284
+You can freeze this part and change it any
+more so you don't lose the ability or you can
+
+0:26:23.284 --> 0:26:25.702
+do a direct combination.
+
+0:26:25.945 --> 0:26:31.028
+Where you jointly train both of them, so you
+train the NMT system on the, and then you train
+
+0:26:31.028 --> 0:26:34.909
+the language model always in parallels so that
+you don't forget about.
+
+0:26:35.395 --> 0:26:37.684
+And what you learn of the length.
+
+0:26:37.937 --> 0:26:46.711
+Depends on what you want to combine because
+it's large data and you have a good general
+
+0:26:46.711 --> 0:26:48.107
+knowledge in.
+
+0:26:48.548 --> 0:26:55.733
+Then you normally don't really forget it because
+it's also in the or you use it to adapt to
+
+0:26:55.733 --> 0:26:57.295
+something specific.
+
+0:26:57.295 --> 0:26:58.075
+Then you.
+
+0:27:01.001 --> 0:27:06.676
+Then this is a way of how we can make use
+of monolingual data.
+
+0:27:07.968 --> 0:27:12.116
+It seems to be the easiest one somehow.
+
+0:27:12.116 --> 0:27:20.103
+It's more similar to what we are doing with
+statistical machine translation.
+
+0:27:21.181 --> 0:27:31.158
+Normally always beats this type of model,
+which in some view can be like from the conceptual
+
+0:27:31.158 --> 0:27:31.909
+thing.
+
+0:27:31.909 --> 0:27:36.844
+It's even easier from the computational side.
+
+0:27:40.560 --> 0:27:42.078
+And the idea is OK.
+
+0:27:42.078 --> 0:27:49.136
+We have monolingual data that we just translate
+and then generate some type of parallel data
+
+0:27:49.136 --> 0:27:50.806
+and use that then to.
+
+0:27:51.111 --> 0:28:00.017
+So if you want to build a German-to-English
+system first, take the large amount of data
+
+0:28:00.017 --> 0:28:02.143
+you have translated.
+
+0:28:02.402 --> 0:28:10.446
+Then you have more peril data and the interesting
+thing is if you then train on the joint thing
+
+0:28:10.446 --> 0:28:18.742
+or on the original peril data and on what is
+artificial where you have generated the translations.
+
+0:28:18.918 --> 0:28:26.487
+So you can because you are not doing the same
+era all the times and you have some knowledge.
+
+0:28:28.028 --> 0:28:43.199
+With this first approach, however, there is
+one issue why it might not work the best.
+
+0:28:49.409 --> 0:28:51.177
+Very a bit shown in the image to you.
+
+0:28:53.113 --> 0:28:58.153
+You trade on that quality data.
+
+0:28:58.153 --> 0:29:02.563
+Here is a bit of a problem.
+
+0:29:02.563 --> 0:29:08.706
+Your English style is not really good.
+
+0:29:08.828 --> 0:29:12.213
+And as you're saying, the system always mistranslates.
+
+0:29:13.493 --> 0:29:19.798
+Something then you will learn that this is
+correct because now it's a training game and
+
+0:29:19.798 --> 0:29:23.022
+you will encourage it to make it more often.
+
+0:29:23.022 --> 0:29:29.614
+So the problem with training on your own areas
+yeah you might prevent some areas you rarely
+
+0:29:29.614 --> 0:29:29.901
+do.
+
+0:29:30.150 --> 0:29:31.749
+But errors use systematically.
+
+0:29:31.749 --> 0:29:34.225
+Do you even enforce more and will even do
+more?
+
+0:29:34.654 --> 0:29:40.145
+So that might not be the best solution to
+have any idea how you could do it better.
+
+0:29:44.404 --> 0:29:57.754
+Is one way there is even a bit of more simple
+idea.
+
+0:30:04.624 --> 0:30:10.975
+The problem is yeah, the translations are
+not perfect, so the output and you're learning
+
+0:30:10.975 --> 0:30:12.188
+something wrong.
+
+0:30:12.188 --> 0:30:17.969
+Normally it's less bad if your inputs are
+not bad, but your outputs are perfect.
+
+0:30:18.538 --> 0:30:24.284
+So if your inputs are wrong you may learn
+that if you're doing this wrong input you're
+
+0:30:24.284 --> 0:30:30.162
+generating something correct, but you're not
+learning to generate something which is not
+
+0:30:30.162 --> 0:30:30.756
+correct.
+
+0:30:31.511 --> 0:30:47.124
+So often the case it is that it is more important
+than your target is correct.
+
+0:30:47.347 --> 0:30:52.182
+But you can assume in your application scenario
+you hope that you may only get correct inputs.
+
+0:30:52.572 --> 0:31:02.535
+So that is not harming you, and in machine
+translation we have one very nice advantage:
+
+0:31:02.762 --> 0:31:04.648
+And also the other way around.
+
+0:31:04.648 --> 0:31:10.062
+It's a very similar task, so there's a task
+to translate from German to English, but the
+
+0:31:10.062 --> 0:31:13.894
+task to translate from English to German is
+very similar, and.
+
+0:31:14.094 --> 0:31:19.309
+So what we can do is we can just switch it
+initially and generate the data the other way
+
+0:31:19.309 --> 0:31:19.778
+around.
+
+0:31:20.120 --> 0:31:25.959
+So what we are doing here is we are starting
+with an English to German system.
+
+0:31:25.959 --> 0:31:32.906
+Then we are translating the English data into
+German where the German is maybe not very nice.
+
+0:31:33.293 --> 0:31:51.785
+And then we are training on our original data
+and on the back translated data.
+
+0:31:52.632 --> 0:32:02.332
+So here we have the advantage that our target
+side is human quality and only the input.
+
+0:32:03.583 --> 0:32:08.113
+Then this helps us to get really good.
+
+0:32:08.113 --> 0:32:15.431
+There is one difference if you think about
+the data resources.
+
+0:32:21.341 --> 0:32:27.336
+Too obvious here we need a target site monolingual
+layer.
+
+0:32:27.336 --> 0:32:31.574
+In the first example we had source site.
+
+0:32:31.931 --> 0:32:45.111
+So back translation is normally working if
+you have target size peril later and not search
+
+0:32:45.111 --> 0:32:48.152
+side modeling later.
+
+0:32:48.448 --> 0:32:56.125
+Might be also, like if you think about it,
+understand a little better to understand the
+
+0:32:56.125 --> 0:32:56.823
+target.
+
+0:32:57.117 --> 0:33:01.469
+On the source side you have to understand
+the content.
+
+0:33:01.469 --> 0:33:08.749
+On the target side you have to generate really
+sentences and somehow it's more difficult to
+
+0:33:08.749 --> 0:33:12.231
+generate something than to only understand.
+
+0:33:17.617 --> 0:33:30.734
+This works well if you have to select how
+many back translated data do you use.
+
+0:33:31.051 --> 0:33:32.983
+Because only there's like a lot more.
+
+0:33:33.253 --> 0:33:42.136
+Question: Should take all of my data there
+is two problems with it?
+
+0:33:42.136 --> 0:33:51.281
+Of course it's expensive because you have
+to translate all this data.
+
+0:33:51.651 --> 0:34:00.946
+So if you don't know the normal good starting
+point is to take equal amount of data as many
+
+0:34:00.946 --> 0:34:02.663
+back translated.
+
+0:34:02.963 --> 0:34:04.673
+It depends on the used case.
+
+0:34:04.673 --> 0:34:08.507
+If we have very few data here, it makes more
+sense to have more.
+
+0:34:08.688 --> 0:34:15.224
+Depends on how good your quality is here,
+so the better the more data you might use because
+
+0:34:15.224 --> 0:34:16.574
+quality is better.
+
+0:34:16.574 --> 0:34:22.755
+So it depends on a lot of things, but your
+rule of sum is like which general way often
+
+0:34:22.755 --> 0:34:24.815
+is to have equal amounts of.
+
+0:34:26.646 --> 0:34:29.854
+And you can, of course, do that now.
+
+0:34:29.854 --> 0:34:34.449
+I said already that it's better to have the
+quality.
+
+0:34:34.449 --> 0:34:38.523
+At the end, of course, depends on this system.
+
+0:34:38.523 --> 0:34:46.152
+Also, because the better this system is, the
+better your synthetic data is, the better.
+
+0:34:47.207 --> 0:34:50.949
+That leads to what is referred to as iterated
+back translation.
+
+0:34:51.291 --> 0:34:56.917
+So you play them on English to German, and
+you translate the data on.
+
+0:34:56.957 --> 0:35:03.198
+Then you train a model on German to English
+with the additional data.
+
+0:35:03.198 --> 0:35:09.796
+Then you translate German data and then you
+train to gain your first one.
+
+0:35:09.796 --> 0:35:14.343
+So in the second iteration this quality is
+better.
+
+0:35:14.334 --> 0:35:19.900
+System is better because it's not only trained
+on the small data but additionally on back
+
+0:35:19.900 --> 0:35:22.003
+translated data with this system.
+
+0:35:22.442 --> 0:35:24.458
+And so you can get better.
+
+0:35:24.764 --> 0:35:28.053
+However, typically you can stop quite early.
+
+0:35:28.053 --> 0:35:35.068
+Maybe one iteration is good, but then you
+have diminishing gains after two or three iterations.
+
+0:35:35.935 --> 0:35:46.140
+There is very slight difference because you
+need a quite big difference in the quality
+
+0:35:46.140 --> 0:35:46.843
+here.
+
+0:35:47.207 --> 0:36:02.262
+Language is also good because it means you
+can already train it with relatively bad profiles.
+
+0:36:03.723 --> 0:36:10.339
+It's a design decision would advise so guess
+because it's easy to get it.
+
+0:36:10.550 --> 0:36:20.802
+Replace that because you have a higher quality
+real data, but then I think normally it's okay
+
+0:36:20.802 --> 0:36:22.438
+to replace it.
+
+0:36:22.438 --> 0:36:28.437
+I would assume it's not too much of a difference,
+but.
+
+0:36:34.414 --> 0:36:42.014
+That's about like using monolingual data before
+we go into the pre-train models to have any
+
+0:36:42.014 --> 0:36:43.005
+more crash.
+
+0:36:49.029 --> 0:36:55.740
+Yes, so the other thing which we can do and
+which is recently more and more successful
+
+0:36:55.740 --> 0:37:02.451
+and even more successful since we have this
+really large language models where you can
+
+0:37:02.451 --> 0:37:08.545
+even do the translation task with this is the
+way of using pre-trained models.
+
+0:37:08.688 --> 0:37:16.135
+So you learn a representation of one task,
+and then you use this representation from another.
+
+0:37:16.576 --> 0:37:26.862
+It was made maybe like one of the first words
+where it really used largely is doing something
+
+0:37:26.862 --> 0:37:35.945
+like a bird which you pre trained on purely
+text era and you take it in fine tune.
+
+0:37:36.496 --> 0:37:42.953
+And one big advantage, of course, is that
+people can only share data but also pre-trained.
+
+0:37:43.423 --> 0:37:59.743
+The recent models and the large language ones
+which are available.
+
+0:37:59.919 --> 0:38:09.145
+Where I think it costs several millions to
+train them all, just if you would buy the GPUs
+
+0:38:09.145 --> 0:38:15.397
+from some cloud company and train that the
+cost of training.
+
+0:38:15.475 --> 0:38:21.735
+And guess as a student project you won't have
+the budget to like build these models.
+
+0:38:21.801 --> 0:38:24.598
+So another idea is what you can do is okay.
+
+0:38:24.598 --> 0:38:27.330
+Maybe if these months are once available,.
+
+0:38:27.467 --> 0:38:36.598
+Can take them and use them as an also resource
+similar to pure text, and you can now build
+
+0:38:36.598 --> 0:38:44.524
+models which somehow learn not only from from
+data but also from other models.
+
+0:38:44.844 --> 0:38:49.127
+So it's a quite new way of thinking of how
+to train.
+
+0:38:49.127 --> 0:38:53.894
+We are not only learning from examples, but
+we might also.
+
+0:38:54.534 --> 0:39:05.397
+The nice thing is that this type of training
+where we are not learning directly from data
+
+0:39:05.397 --> 0:39:07.087
+but learning.
+
+0:39:07.427 --> 0:39:17.647
+So the main idea this go is you have a person
+initial task.
+
+0:39:17.817 --> 0:39:26.369
+And if you're working with anLP, that means
+you're training pure taxator because that's
+
+0:39:26.369 --> 0:39:30.547
+where you have the largest amount of data.
+
+0:39:30.951 --> 0:39:35.857
+And then you're defining some type of task
+in order to do your creek training.
+
+0:39:36.176 --> 0:39:43.092
+And: The typical task you can train on on
+that is like the language waddling task.
+
+0:39:43.092 --> 0:39:50.049
+So to predict the next word or we have a related
+task to predict something in between, we'll
+
+0:39:50.049 --> 0:39:52.667
+see depending on the architecture.
+
+0:39:52.932 --> 0:39:58.278
+But somehow to predict something which you
+have not in the input is a task which is easy
+
+0:39:58.278 --> 0:40:00.740
+to generate, so you just need your data.
+
+0:40:00.740 --> 0:40:06.086
+That's why it's called self supervised, so
+you're creating your supervised pending data.
+
+0:40:06.366 --> 0:40:07.646
+By yourself.
+
+0:40:07.646 --> 0:40:15.133
+On the other hand, you need a lot of knowledge
+and that is the other thing.
+
+0:40:15.735 --> 0:40:24.703
+Because there is this idea that the meaning
+of a word heavily depends on the context that.
+
+0:40:25.145 --> 0:40:36.846
+So can give you a sentence with some giverish
+word and there's some name and although you've
+
+0:40:36.846 --> 0:40:41.627
+never heard the name you will assume.
+
+0:40:42.062 --> 0:40:44.149
+And exactly the same thing.
+
+0:40:44.149 --> 0:40:49.143
+The models can also learn something about
+the world by just using.
+
+0:40:49.649 --> 0:40:53.651
+So that is typically the mule.
+
+0:40:53.651 --> 0:40:59.848
+Then we can use this model to train the system.
+
+0:41:00.800 --> 0:41:03.368
+Course we might need to adapt the system.
+
+0:41:03.368 --> 0:41:07.648
+To do that we have to change the architecture
+we might use only some.
+
+0:41:07.627 --> 0:41:09.443
+Part of the pre-trained model.
+
+0:41:09.443 --> 0:41:14.773
+In there we have seen that a bit already in
+the R&N case you can also see that we have
+
+0:41:14.773 --> 0:41:17.175
+also mentioned the pre-training already.
+
+0:41:17.437 --> 0:41:22.783
+So you can use the R&N as one of these
+approaches.
+
+0:41:22.783 --> 0:41:28.712
+You train the R&M language more on large
+pre-train data.
+
+0:41:28.712 --> 0:41:32.309
+Then you put it somewhere into your.
+
+0:41:33.653 --> 0:41:37.415
+So this gives you the ability to really do
+these types of tests.
+
+0:41:37.877 --> 0:41:53.924
+So you can build a system which is knowledge,
+which is just trained on large amounts of data.
+
+0:41:56.376 --> 0:42:01.564
+So the question is maybe what type of information
+so what type of models can you?
+
+0:42:01.821 --> 0:42:05.277
+And we want today to look at briefly at swings.
+
+0:42:05.725 --> 0:42:08.704
+First, that was what was initially done.
+
+0:42:08.704 --> 0:42:15.314
+It wasn't as famous as in machine translation
+as in other things, but it's also used there
+
+0:42:15.314 --> 0:42:21.053
+and that is to use static word embedding, so
+just the first step we know here.
+
+0:42:21.221 --> 0:42:28.981
+So we have this mapping from the one hot to
+a small continuous word representation.
+
+0:42:29.229 --> 0:42:38.276
+Using this one in your NG system, so you can,
+for example, replace the embedding layer by
+
+0:42:38.276 --> 0:42:38.779
+the.
+
+0:42:39.139 --> 0:42:41.832
+That is helpful to be a really small amount
+of data.
+
+0:42:42.922 --> 0:42:48.517
+And we're always in this pre-training phase
+and have the thing the advantage is.
+
+0:42:48.468 --> 0:42:52.411
+More data than the trade off, so you can get
+better.
+
+0:42:52.411 --> 0:42:59.107
+The disadvantage is, does anybody have an
+idea of what might be the disadvantage of using
+
+0:42:59.107 --> 0:43:00.074
+things like.
+
+0:43:04.624 --> 0:43:12.175
+What was one mentioned today giving like big
+advantage of the system compared to previous.
+
+0:43:20.660 --> 0:43:25.134
+Where one advantage was the enter end training,
+so you have the enter end training so that
+
+0:43:25.134 --> 0:43:27.937
+all parameters and all components play optimal
+together.
+
+0:43:28.208 --> 0:43:33.076
+If you know pre-train something on one fast,
+it may be no longer optimal fitting to everything
+
+0:43:33.076 --> 0:43:33.384
+else.
+
+0:43:33.893 --> 0:43:37.862
+So what do pretending or not?
+
+0:43:37.862 --> 0:43:48.180
+It depends on how important everything is
+optimal together and how important.
+
+0:43:48.388 --> 0:43:50.454
+Of large amount.
+
+0:43:50.454 --> 0:44:00.541
+The pre-change one is so much better that
+it's helpful, and the advantage of that.
+
+0:44:00.600 --> 0:44:11.211
+Getting everything optimal together, yes,
+we would use random instructions for raising.
+
+0:44:11.691 --> 0:44:26.437
+The problem is you might be already in some
+area where it's not easy to get.
+
+0:44:26.766 --> 0:44:35.329
+But often in some way right, so often it's
+not about your really worse pre trained monolepsy.
+
+0:44:35.329 --> 0:44:43.254
+If you're going already in some direction,
+and if this is not really optimal for you,.
+
+0:44:43.603 --> 0:44:52.450
+But if you're not really getting better because
+you have a decent amount of data, it's so different
+
+0:44:52.450 --> 0:44:52.981
+that.
+
+0:44:53.153 --> 0:44:59.505
+Initially it wasn't a machine translation
+done so much because there are more data in
+
+0:44:59.505 --> 0:45:06.153
+MPs than in other tasks, but now with really
+large amounts of monolingual data we do some
+
+0:45:06.153 --> 0:45:09.403
+type of pretraining in currently all state.
+
+0:45:12.632 --> 0:45:14.302
+The other one is okay now.
+
+0:45:14.302 --> 0:45:18.260
+It's always like how much of the model do
+you plea track a bit?
+
+0:45:18.658 --> 0:45:22.386
+To the other one you can do contextural word
+embedded.
+
+0:45:22.386 --> 0:45:28.351
+That is something like bird or Roberta where
+you train already a sequence model and the
+
+0:45:28.351 --> 0:45:34.654
+embeddings you're using are no longer specific
+for word but they are also taking the context
+
+0:45:34.654 --> 0:45:35.603
+into account.
+
+0:45:35.875 --> 0:45:50.088
+The embedding you're using is no longer depending
+on the word itself but on the whole sentence,
+
+0:45:50.088 --> 0:45:54.382
+so you can use this context.
+
+0:45:55.415 --> 0:46:02.691
+You can use similar things also in the decoder
+just by having layers which don't have access
+
+0:46:02.691 --> 0:46:12.430
+to the source, but there it still might have
+and these are typically models like: And finally
+
+0:46:12.430 --> 0:46:14.634
+they will look at the end.
+
+0:46:14.634 --> 0:46:19.040
+You can also have models which are already
+sequenced.
+
+0:46:19.419 --> 0:46:28.561
+So you may be training a sequence to sequence
+models.
+
+0:46:28.561 --> 0:46:35.164
+You have to make it a bit challenging.
+
+0:46:36.156 --> 0:46:43.445
+But the idea is really you're pre-training
+your whole model and then you'll find tuning.
+
+0:46:47.227 --> 0:46:59.614
+But let's first do a bit of step back and
+look into what are the different things.
+
+0:46:59.614 --> 0:47:02.151
+The first thing.
+
+0:47:02.382 --> 0:47:11.063
+The wooden bettings are just this first layer
+and you can train them with feedback annual
+
+0:47:11.063 --> 0:47:12.028
+networks.
+
+0:47:12.212 --> 0:47:22.761
+But you can also train them with an N language
+model, and by now you hopefully have also seen
+
+0:47:22.761 --> 0:47:27.699
+that you cannot transform a language model.
+
+0:47:30.130 --> 0:47:37.875
+So this is how you can train them and you're
+training them.
+
+0:47:37.875 --> 0:47:45.234
+For example, to speak the next word that is
+the easiest.
+
+0:47:45.525 --> 0:47:55.234
+And that is what is now referred to as South
+Supervised Learning and, for example, all the
+
+0:47:55.234 --> 0:48:00.675
+big large language models like Chad GPT and
+so on.
+
+0:48:00.675 --> 0:48:03.129
+They are trained with.
+
+0:48:03.823 --> 0:48:15.812
+So that is where you can hopefully learn how
+a word is used because you always try to previct
+
+0:48:15.812 --> 0:48:17.725
+the next word.
+
+0:48:19.619 --> 0:48:27.281
+Word embedding: Why do you keep the first
+look at the word embeddings and the use of
+
+0:48:27.281 --> 0:48:29.985
+word embeddings for our task?
+
+0:48:29.985 --> 0:48:38.007
+The main advantage was it might be only the
+first layer where you typically have most of
+
+0:48:38.007 --> 0:48:39.449
+the parameters.
+
+0:48:39.879 --> 0:48:57.017
+Most of your parameters already on the large
+data, then on your target data you have to
+
+0:48:57.017 --> 0:48:59.353
+train less.
+
+0:48:59.259 --> 0:49:06.527
+Big difference that your input size is so
+much bigger than the size of the novel in size.
+
+0:49:06.626 --> 0:49:17.709
+So it's a normally sign, maybe like, but your
+input and banning size is something like.
+
+0:49:17.709 --> 0:49:20.606
+Then here you have to.
+
+0:49:23.123 --> 0:49:30.160
+While here you see it's only like zero point
+five times as much in the layer.
+
+0:49:30.750 --> 0:49:36.534
+So here is where most of your parameters are,
+which means if you already replace the word
+
+0:49:36.534 --> 0:49:41.739
+embeddings, they might look a bit small in
+your overall and in key architecture.
+
+0:49:41.739 --> 0:49:47.395
+It's where most of the things are, and if
+you're doing that you already have really big
+
+0:49:47.395 --> 0:49:48.873
+games and can do that.
+
+0:49:57.637 --> 0:50:01.249
+The thing is we have seen these were the bettings.
+
+0:50:01.249 --> 0:50:04.295
+They can be very good use for other types.
+
+0:50:04.784 --> 0:50:08.994
+You learn some general relations between words.
+
+0:50:08.994 --> 0:50:17.454
+If you're doing this type of language modeling
+cast, you predict: The one thing is you have
+
+0:50:17.454 --> 0:50:24.084
+a lot of data, so the one question is we want
+to have data to trade a model.
+
+0:50:24.084 --> 0:50:28.734
+The other thing, the tasks need to be somehow
+useful.
+
+0:50:29.169 --> 0:50:43.547
+If you would predict the first letter of the
+word, then you wouldn't learn anything about
+
+0:50:43.547 --> 0:50:45.144
+the word.
+
+0:50:45.545 --> 0:50:53.683
+And the interesting thing is people have looked
+at these wood embeddings.
+
+0:50:53.954 --> 0:50:58.550
+And looking at the word embeddings.
+
+0:50:58.550 --> 0:51:09.276
+You can ask yourself how they look and visualize
+them by doing dimension reduction.
+
+0:51:09.489 --> 0:51:13.236
+Don't know if you and you are listening to
+artificial intelligence.
+
+0:51:13.236 --> 0:51:15.110
+Advanced artificial intelligence.
+
+0:51:15.515 --> 0:51:23.217
+We had on yesterday there how to do this type
+of representation, but you can do this time
+
+0:51:23.217 --> 0:51:29.635
+of representation, and now you're seeing interesting
+things that normally.
+
+0:51:30.810 --> 0:51:41.027
+Now you can represent a here in a three dimensional
+space with some dimension reduction.
+
+0:51:41.027 --> 0:51:46.881
+For example, the relation between male and
+female.
+
+0:51:47.447 --> 0:51:56.625
+So this vector between the male and female
+version of something is always not the same,
+
+0:51:56.625 --> 0:51:58.502
+but it's related.
+
+0:51:58.718 --> 0:52:14.522
+So you can do a bit of maths, so you do take
+king, you subtract this vector, add this vector.
+
+0:52:14.894 --> 0:52:17.591
+So that means okay, there is really something
+stored.
+
+0:52:17.591 --> 0:52:19.689
+Some information are stored in that book.
+
+0:52:20.040 --> 0:52:22.621
+Similar, you can do it with Bob Hansen.
+
+0:52:22.621 --> 0:52:25.009
+See here swimming slam walking walk.
+
+0:52:25.265 --> 0:52:34.620
+So again these vectors are not the same, but
+they are related.
+
+0:52:34.620 --> 0:52:42.490
+So you learn something from going from here
+to here.
+
+0:52:43.623 --> 0:52:49.761
+Or semantically, the relations between city
+and capital have exactly the same sense.
+
+0:52:51.191 --> 0:52:56.854
+And people had even done that question answering
+about that if they showed the diembeddings
+
+0:52:56.854 --> 0:52:57.839
+and the end of.
+
+0:52:58.218 --> 0:53:06.711
+All you can also do is don't trust the dimensions
+of the reaction because maybe there is something.
+
+0:53:06.967 --> 0:53:16.863
+You can also look into what happens really
+in the individual space.
+
+0:53:16.863 --> 0:53:22.247
+What is the nearest neighbor of the.
+
+0:53:22.482 --> 0:53:29.608
+So you can take the relationship between France
+and Paris and add it to Italy and you'll.
+
+0:53:30.010 --> 0:53:33.078
+You can do big and bigger and you have small
+and smaller and stuff.
+
+0:53:33.593 --> 0:53:49.417
+Because it doesn't work everywhere, there
+is also some typical dish here in German.
+
+0:53:51.491 --> 0:54:01.677
+You can do what the person is doing for famous
+ones, of course only like Einstein scientists
+
+0:54:01.677 --> 0:54:06.716
+that find midfielders not completely correct.
+
+0:54:06.846 --> 0:54:10.134
+You see the examples are a bit old.
+
+0:54:10.134 --> 0:54:15.066
+The politicians are no longer they am, but
+of course.
+
+0:54:16.957 --> 0:54:26.759
+What people have done there, especially at
+the beginning training our end language model,
+
+0:54:26.759 --> 0:54:28.937
+was very expensive.
+
+0:54:29.309 --> 0:54:38.031
+So one famous model was, but we are not really
+interested in the language model performance.
+
+0:54:38.338 --> 0:54:40.581
+Think something good to keep in mind.
+
+0:54:40.581 --> 0:54:42.587
+What are we really interested in?
+
+0:54:42.587 --> 0:54:45.007
+Do we really want to have an R&N no?
+
+0:54:45.007 --> 0:54:48.607
+In this case we are only interested in this
+type of mapping.
+
+0:54:49.169 --> 0:54:55.500
+And so successful and very successful was
+this word to vet.
+
+0:54:55.535 --> 0:54:56.865
+The idea is okay.
+
+0:54:56.865 --> 0:55:03.592
+We are not training real language one, making
+it even simpler and doing this, for example,
+
+0:55:03.592 --> 0:55:05.513
+continuous peck of words.
+
+0:55:05.513 --> 0:55:12.313
+We're just having four input tokens and we're
+predicting what is the word in the middle and
+
+0:55:12.313 --> 0:55:15.048
+this is just like two linear layers.
+
+0:55:15.615 --> 0:55:21.627
+So it's even simplifying things and making
+the calculation faster because that is what
+
+0:55:21.627 --> 0:55:22.871
+we're interested.
+
+0:55:23.263 --> 0:55:32.897
+All this continuous skip ground models with
+these other models which refer to as where
+
+0:55:32.897 --> 0:55:34.004
+to where.
+
+0:55:34.234 --> 0:55:42.394
+Where you have one equal word and the other
+way around, you're predicting the four words
+
+0:55:42.394 --> 0:55:43.585
+around them.
+
+0:55:43.585 --> 0:55:45.327
+It's very similar.
+
+0:55:45.327 --> 0:55:48.720
+The task is in the end very similar.
+
+0:55:51.131 --> 0:56:01.407
+Before we are going to the next point, anything
+about normal weight vectors or weight embedding.
+
+0:56:04.564 --> 0:56:07.794
+The next thing is contexture.
+
+0:56:07.794 --> 0:56:12.208
+Word embeddings and the idea is helpful.
+
+0:56:12.208 --> 0:56:19.206
+However, we might even be able to get more
+from one lingo layer.
+
+0:56:19.419 --> 0:56:31.732
+And now in the word that is overlap of these
+two meanings, so it represents both the meaning
+
+0:56:31.732 --> 0:56:33.585
+of can do it.
+
+0:56:34.834 --> 0:56:40.410
+But we might be able to in the pre-trained
+model already disambiguate this because they
+
+0:56:40.410 --> 0:56:41.044
+are used.
+
+0:56:41.701 --> 0:56:53.331
+So if we can have a model which can not only
+represent a word but can also represent the
+
+0:56:53.331 --> 0:56:58.689
+meaning of the word within the context,.
+
+0:56:59.139 --> 0:57:03.769
+So then we are going to context your word
+embeddings.
+
+0:57:03.769 --> 0:57:07.713
+We are really having a representation in the.
+
+0:57:07.787 --> 0:57:11.519
+And we have a very good architecture for that
+already.
+
+0:57:11.691 --> 0:57:23.791
+The hidden state represents what is currently
+said, but it's focusing on what is the last
+
+0:57:23.791 --> 0:57:29.303
+one, so it's some of the representation.
+
+0:57:29.509 --> 0:57:43.758
+The first one doing that is something like
+the Elmo paper where they instead of this is
+
+0:57:43.758 --> 0:57:48.129
+the normal language model.
+
+0:57:48.008 --> 0:57:50.714
+Within the third, predicting the fourth, and
+so on.
+
+0:57:50.714 --> 0:57:53.004
+So you are always predicting the next work.
+
+0:57:53.193 --> 0:57:57.335
+The architecture is the heaven words embedding
+layer and then layers.
+
+0:57:57.335 --> 0:58:03.901
+See you, for example: And now instead of using
+this one in the end, you're using here this
+
+0:58:03.901 --> 0:58:04.254
+one.
+
+0:58:04.364 --> 0:58:11.245
+This represents the meaning of this word mainly
+in the context of what we have seen before.
+
+0:58:11.871 --> 0:58:18.610
+We can train it in a language model style
+always predicting the next word, but we have
+
+0:58:18.610 --> 0:58:21.088
+more information trained there.
+
+0:58:21.088 --> 0:58:26.123
+Therefore, in the system it has to learn less
+additional things.
+
+0:58:27.167 --> 0:58:31.261
+And there is one Edendang which is done currently
+in GPS.
+
+0:58:31.261 --> 0:58:38.319
+The only difference is that we have more layers,
+bigger size, and we're using transformer neurocell
+
+0:58:38.319 --> 0:58:40.437
+potential instead of the RNA.
+
+0:58:40.437 --> 0:58:45.095
+But that is how you train like some large
+language models at the.
+
+0:58:46.746 --> 0:58:55.044
+However, if you look at this contextual representation,
+they might not be perfect.
+
+0:58:55.044 --> 0:59:02.942
+So if you think of this one as a contextual
+representation of the third word,.
+
+0:59:07.587 --> 0:59:16.686
+Is representing a three in the context of
+a sentence, however only in the context of
+
+0:59:16.686 --> 0:59:18.185
+the previous.
+
+0:59:18.558 --> 0:59:27.413
+However, we have an architecture which can
+also take both sides and we have used that
+
+0:59:27.413 --> 0:59:30.193
+already in the ink holder.
+
+0:59:30.630 --> 0:59:34.264
+So we could do the iron easily on your, also
+in the backward direction.
+
+0:59:34.874 --> 0:59:42.826
+By just having the states the other way around
+and then we couldn't combine the forward and
+
+0:59:42.826 --> 0:59:49.135
+the forward into a joint one where we are doing
+this type of prediction.
+
+0:59:49.329 --> 0:59:50.858
+So you have the word embedding.
+
+0:59:51.011 --> 1:00:02.095
+Then you have two in the states, one on the
+forward arm and one on the backward arm, and
+
+1:00:02.095 --> 1:00:10.314
+then you can, for example, take the cocagenation
+of both of them.
+
+1:00:10.490 --> 1:00:23.257
+Now this same here represents mainly this
+word because this is what both puts in it last
+
+1:00:23.257 --> 1:00:30.573
+and we know is focusing on what is happening
+last.
+
+1:00:31.731 --> 1:00:40.469
+However, there is a bit of difference when
+training that as a language model you already
+
+1:00:40.469 --> 1:00:41.059
+have.
+
+1:00:43.203 --> 1:00:44.956
+Maybe There's Again This Masking.
+
+1:00:46.546 --> 1:00:47.748
+That is one solution.
+
+1:00:47.748 --> 1:00:52.995
+First of all, why we can't do it is the information
+you leak it, so you cannot just predict the
+
+1:00:52.995 --> 1:00:53.596
+next word.
+
+1:00:53.596 --> 1:00:58.132
+If we just predict the next word in this type
+of model, that's a very simple task.
+
+1:00:58.738 --> 1:01:09.581
+You know the next word because it's influencing
+this hidden state predicting something is not
+
+1:01:09.581 --> 1:01:11.081
+a good task.
+
+1:01:11.081 --> 1:01:18.455
+You have to define: Because in this case what
+will end with the system will just ignore these
+
+1:01:18.455 --> 1:01:22.966
+estates and what will learn is copy this information
+directly in here.
+
+1:01:23.343 --> 1:01:31.218
+So it would be representing this word and
+you would have nearly a perfect model because
+
+1:01:31.218 --> 1:01:38.287
+you only need to find encoding where you can
+encode all words somehow in this.
+
+1:01:38.458 --> 1:01:44.050
+The only thing can learn is that turn and
+encode all my words in this upper hidden.
+
+1:01:44.985 --> 1:01:53.779
+Therefore, it's not really useful, so we need
+to find a bit of different ways out.
+
+1:01:55.295 --> 1:01:57.090
+There is a masking one.
+
+1:01:57.090 --> 1:02:03.747
+I'll come to that shortly just a bit that
+other things also have been done, so the other
+
+1:02:03.747 --> 1:02:06.664
+thing is not to directly combine them.
+
+1:02:06.664 --> 1:02:13.546
+That was in the animal paper, so you have
+them forward R&M and you keep them completely
+
+1:02:13.546 --> 1:02:14.369
+separated.
+
+1:02:14.594 --> 1:02:20.458
+So you never merged to state.
+
+1:02:20.458 --> 1:02:33.749
+At the end, the representation of the word
+is now from the forward.
+
+1:02:33.873 --> 1:02:35.953
+So it's always the hidden state before the
+good thing.
+
+1:02:36.696 --> 1:02:41.286
+These two you join now to your to the representation.
+
+1:02:42.022 --> 1:02:48.685
+And then you have now a representation also
+about like the whole sentence for the word,
+
+1:02:48.685 --> 1:02:51.486
+but there is no information leakage.
+
+1:02:51.486 --> 1:02:58.149
+One way of doing this is instead of doing
+a bidirection along you do a forward pass and
+
+1:02:58.149 --> 1:02:59.815
+then join the hidden.
+
+1:03:00.380 --> 1:03:05.960
+So you can do that in all layers.
+
+1:03:05.960 --> 1:03:16.300
+In the end you do the forwarded layers and
+you get the hidden.
+
+1:03:16.596 --> 1:03:19.845
+However, it's a bit of a complicated.
+
+1:03:19.845 --> 1:03:25.230
+You have to keep both separate and merge things
+so can you do.
+
+1:03:27.968 --> 1:03:33.030
+And that is the moment where like the big.
+
+1:03:34.894 --> 1:03:39.970
+The big success of the burnt model was used
+where it okay.
+
+1:03:39.970 --> 1:03:47.281
+Maybe in bite and rich case it's not good
+to do the next word prediction, but we can
+
+1:03:47.281 --> 1:03:48.314
+do masking.
+
+1:03:48.308 --> 1:03:56.019
+Masking mainly means we do a prediction of
+something in the middle or some words.
+
+1:03:56.019 --> 1:04:04.388
+So the idea is if we have the input, we are
+putting noise into the input, removing them,
+
+1:04:04.388 --> 1:04:07.961
+and then the model we are interested.
+
+1:04:08.048 --> 1:04:15.327
+Now there can be no information leakage because
+this wasn't predicting that one is a big challenge.
+
+1:04:16.776 --> 1:04:19.957
+Do any assumption about our model?
+
+1:04:19.957 --> 1:04:26.410
+It doesn't need to be a forward model or a
+backward model or anything.
+
+1:04:26.410 --> 1:04:29.500
+You can always predict the three.
+
+1:04:30.530 --> 1:04:34.844
+There's maybe one bit of a disadvantage.
+
+1:04:34.844 --> 1:04:40.105
+Do you see what could be a bit of a problem
+this?
+
+1:05:00.000 --> 1:05:06.429
+Yes, so yeah, you can of course mask more,
+but to see it more globally, just first assume
+
+1:05:06.429 --> 1:05:08.143
+you're only masked one.
+
+1:05:08.143 --> 1:05:13.930
+For the whole sentence, we get one feedback
+signal, like what is the word three.
+
+1:05:13.930 --> 1:05:22.882
+So we have one training example: If you do
+the language modeling taste, we predicted here,
+
+1:05:22.882 --> 1:05:24.679
+we predicted here.
+
+1:05:25.005 --> 1:05:26.735
+So we have number of tokens.
+
+1:05:26.735 --> 1:05:30.970
+For each token we have a feet pad and say
+what is the best correction.
+
+1:05:31.211 --> 1:05:43.300
+So in this case this is less efficient because
+we are getting less feedback signals on what
+
+1:05:43.300 --> 1:05:45.797
+we should predict.
+
+1:05:48.348 --> 1:05:56.373
+So and bird, the main ideas are that you're
+doing this bidirectional model with masking.
+
+1:05:56.373 --> 1:05:59.709
+It's using transformer architecture.
+
+1:06:00.320 --> 1:06:06.326
+There are two more minor changes.
+
+1:06:06.326 --> 1:06:16.573
+We'll see that this next word prediction is
+another task.
+
+1:06:16.957 --> 1:06:30.394
+You want to learn more about what language
+is to really understand following a story or
+
+1:06:30.394 --> 1:06:35.127
+their independent tokens into.
+
+1:06:38.158 --> 1:06:42.723
+The input is using word units as we use it.
+
+1:06:42.723 --> 1:06:50.193
+It has some special token that is framing
+for the next word prediction.
+
+1:06:50.470 --> 1:07:04.075
+It's more for classification task because
+you may be learning a general representation
+
+1:07:04.075 --> 1:07:07.203
+as a full sentence.
+
+1:07:07.607 --> 1:07:19.290
+You're doing segment embedding, so you have
+an embedding for it.
+
+1:07:19.290 --> 1:07:24.323
+This is the first sentence.
+
+1:07:24.684 --> 1:07:29.099
+Now what is more challenging is this masking.
+
+1:07:29.099 --> 1:07:30.827
+What do you mask?
+
+1:07:30.827 --> 1:07:35.050
+We already have the crush enough or should.
+
+1:07:35.275 --> 1:07:42.836
+So there has been afterwards eating some work
+like, for example, a bearer.
+
+1:07:42.836 --> 1:07:52.313
+It's not super sensitive, but if you do it
+completely wrong then you're not letting anything.
+
+1:07:52.572 --> 1:07:54.590
+That's Then Another Question There.
+
+1:07:56.756 --> 1:08:04.594
+Should I mask all types of should I always
+mask the footwork or if I have a subword to
+
+1:08:04.594 --> 1:08:10.630
+mask only like a subword and predict them based
+on the other ones?
+
+1:08:10.630 --> 1:08:14.504
+Of course, it's a bit of a different task.
+
+1:08:14.894 --> 1:08:21.210
+If you know three parts of the words, it might
+be easier to guess the last because they here
+
+1:08:21.210 --> 1:08:27.594
+took the easiest selection, so not considering
+words anymore at all because you're doing that
+
+1:08:27.594 --> 1:08:32.280
+in the preprocessing and just taking always
+words and like subwords.
+
+1:08:32.672 --> 1:08:36.089
+Think in group there is done differently.
+
+1:08:36.089 --> 1:08:40.401
+They mark always the full words, but guess
+it's not.
+
+1:08:41.001 --> 1:08:46.044
+And then what to do with the mask word in
+eighty percent of the cases.
+
+1:08:46.044 --> 1:08:50.803
+If the word is masked, they replace it with
+a special token thing.
+
+1:08:50.803 --> 1:08:57.197
+This is a mask token in ten percent they put
+in some random other token in there, and ten
+
+1:08:57.197 --> 1:08:59.470
+percent they keep it on change.
+
+1:09:02.202 --> 1:09:10.846
+And then what you can do is also this next
+word prediction.
+
+1:09:10.846 --> 1:09:14.880
+The man went to Mass Store.
+
+1:09:14.880 --> 1:09:17.761
+He bought a gallon.
+
+1:09:18.418 --> 1:09:24.088
+So may you see you're joining them, you're
+doing both masks and prediction that you're.
+
+1:09:24.564 --> 1:09:29.449
+Is a penguin mask or flyless birds.
+
+1:09:29.449 --> 1:09:41.390
+These two sentences have nothing to do with
+each other, so you can do also this type of
+
+1:09:41.390 --> 1:09:43.018
+prediction.
+
+1:09:47.127 --> 1:09:57.043
+And then the whole bird model, so here you
+have the input here to transform the layers,
+
+1:09:57.043 --> 1:09:58.170
+and then.
+
+1:09:58.598 --> 1:10:17.731
+And this model was quite successful in general
+applications.
+
+1:10:17.937 --> 1:10:27.644
+However, there is like a huge thing of different
+types of models coming from them.
+
+1:10:27.827 --> 1:10:38.709
+So based on others these supervised molds
+like a whole setup came out of there and now
+
+1:10:38.709 --> 1:10:42.086
+this is getting even more.
+
+1:10:42.082 --> 1:10:46.640
+With availability of a large language model
+than the success.
+
+1:10:47.007 --> 1:10:48.436
+We have now even larger ones.
+
+1:10:48.828 --> 1:10:50.961
+Interestingly, it goes a bit.
+
+1:10:50.910 --> 1:10:57.847
+Change the bit again from like more the spider
+action model to uni directional models.
+
+1:10:57.847 --> 1:11:02.710
+Are at the moment maybe a bit more we're coming
+to them now?
+
+1:11:02.710 --> 1:11:09.168
+Do you see one advantage while what is another
+event and we have the efficiency?
+
+1:11:09.509 --> 1:11:15.901
+Is one other reason why you are sometimes
+more interested in uni-direction models than
+
+1:11:15.901 --> 1:11:17.150
+in bi-direction.
+
+1:11:22.882 --> 1:11:30.220
+It depends on the pass, but for example for
+a language generation pass, the eccard is not
+
+1:11:30.220 --> 1:11:30.872
+really.
+
+1:11:32.192 --> 1:11:40.924
+It doesn't work so if you want to do a generation
+like the decoder you don't know the future
+
+1:11:40.924 --> 1:11:42.896
+so you cannot apply.
+
+1:11:43.223 --> 1:11:53.870
+So this time of model can be used for the
+encoder in an encoder model, but it cannot
+
+1:11:53.870 --> 1:11:57.002
+be used for the decoder.
+
+1:12:00.000 --> 1:12:05.012
+That's a good view to the next overall cast
+of models.
+
+1:12:05.012 --> 1:12:08.839
+Perhaps if you view it from the sequence.
+
+1:12:09.009 --> 1:12:12.761
+We have the encoder base model.
+
+1:12:12.761 --> 1:12:16.161
+That's what we just look at.
+
+1:12:16.161 --> 1:12:20.617
+They are bidirectional and typically.
+
+1:12:20.981 --> 1:12:22.347
+That Is the One We Looked At.
+
+1:12:22.742 --> 1:12:34.634
+At the beginning is the decoder based model,
+so see out in regressive models which are unidirective
+
+1:12:34.634 --> 1:12:42.601
+like an based model, and there we can do the
+next word prediction.
+
+1:12:43.403 --> 1:12:52.439
+And what you can also do first, and there
+you can also have a special things called prefix
+
+1:12:52.439 --> 1:12:53.432
+language.
+
+1:12:54.354 --> 1:13:05.039
+Because we are saying it might be helpful
+that some of your input can also use bi-direction.
+
+1:13:05.285 --> 1:13:12.240
+And that is somehow doing what it is called
+prefix length.
+
+1:13:12.240 --> 1:13:19.076
+On the first tokens you directly give your
+bidirectional.
+
+1:13:19.219 --> 1:13:28.774
+So you somehow merge that and that mainly
+works only in transformer based models because.
+
+1:13:29.629 --> 1:13:33.039
+There is no different number of parameters
+in our end.
+
+1:13:33.039 --> 1:13:34.836
+We need a back foot our end.
+
+1:13:34.975 --> 1:13:38.533
+Transformer: The only difference is how you
+mask your attention.
+
+1:13:38.878 --> 1:13:44.918
+We have seen that in the anchoder and decoder
+the number of parameters is different because
+
+1:13:44.918 --> 1:13:50.235
+you do cross attention, but if you do forward
+and backward or union directions,.
+
+1:13:50.650 --> 1:13:58.736
+It's only like you mask your attention to
+only look at the bad past or to look into the
+
+1:13:58.736 --> 1:13:59.471
+future.
+
+1:14:00.680 --> 1:14:03.326
+And now you can of course also do mixing.
+
+1:14:03.563 --> 1:14:08.306
+So this is a bi-directional attention matrix
+where you can attend to everything.
+
+1:14:08.588 --> 1:14:23.516
+There is a uni-direction or causal where you
+can look at the past and you can do the first
+
+1:14:23.516 --> 1:14:25.649
+three words.
+
+1:14:29.149 --> 1:14:42.831
+That somehow clear based on that, then of
+course you cannot do the other things.
+
+1:14:43.163 --> 1:14:50.623
+So the idea is we have our anchor to decoder
+architecture.
+
+1:14:50.623 --> 1:14:57.704
+Can we also train them completely in a side
+supervisor?
+
+1:14:58.238 --> 1:15:09.980
+And in this case we have the same input to
+both, so in this case we need to do some type
+
+1:15:09.980 --> 1:15:12.224
+of masking here.
+
+1:15:12.912 --> 1:15:17.696
+Here we don't need to do the masking, but
+here we need to masking that doesn't know ever
+
+1:15:17.696 --> 1:15:17.911
+so.
+
+1:15:20.440 --> 1:15:30.269
+And this type of model got quite successful
+also, especially for pre-training machine translation.
+
+1:15:30.330 --> 1:15:39.059
+The first model doing that is a Bart model,
+which exactly does that, and yes, it's one
+
+1:15:39.059 --> 1:15:42.872
+successful way to pre train your one.
+
+1:15:42.872 --> 1:15:47.087
+It's pretraining your full encoder model.
+
+1:15:47.427 --> 1:15:54.365
+Where you put in contrast to machine translation,
+where you put in source sentence, we can't
+
+1:15:54.365 --> 1:15:55.409
+do that here.
+
+1:15:55.715 --> 1:16:01.382
+But we can just put the second twice in there,
+and then it's not a trivial task.
+
+1:16:01.382 --> 1:16:02.432
+We can change.
+
+1:16:03.003 --> 1:16:12.777
+And there is like they do different corruption
+techniques so you can also do.
+
+1:16:13.233 --> 1:16:19.692
+That you couldn't do in an agricultural system
+because then it wouldn't be there and you cannot
+
+1:16:19.692 --> 1:16:20.970
+predict somewhere.
+
+1:16:20.970 --> 1:16:26.353
+So the anchor, the number of input and output
+tokens always has to be the same.
+
+1:16:26.906 --> 1:16:29.818
+You cannot do a prediction for something which
+isn't in it.
+
+1:16:30.110 --> 1:16:38.268
+Here in the decoder side it's unidirection
+so we can also delete the top and then try
+
+1:16:38.268 --> 1:16:40.355
+to generate the full.
+
+1:16:41.061 --> 1:16:45.250
+We can do sentence permutation.
+
+1:16:45.250 --> 1:16:54.285
+We can document rotation and text infilling
+so there is quite a bit.
+
+1:16:55.615 --> 1:17:06.568
+So you see there's quite a lot of types of
+models that you can use in order to pre-train.
+
+1:17:07.507 --> 1:17:14.985
+Then, of course, there is again for the language
+one.
+
+1:17:14.985 --> 1:17:21.079
+The other question is how do you integrate?
+
+1:17:21.761 --> 1:17:26.636
+And there's also, like yeah, quite some different
+ways of techniques.
+
+1:17:27.007 --> 1:17:28.684
+It's a Bit Similar to Before.
+
+1:17:28.928 --> 1:17:39.068
+So the easiest thing is you take your word
+embeddings or your free trained model.
+
+1:17:39.068 --> 1:17:47.971
+You freeze them and stack your decoder layers
+and keep these ones free.
+
+1:17:48.748 --> 1:17:54.495
+Can also be done if you have this type of
+bark model.
+
+1:17:54.495 --> 1:18:03.329
+What you can do is you freeze your word embeddings,
+for example some products and.
+
+1:18:05.865 --> 1:18:17.296
+The other thing is you initialize them so
+you initialize your models but you train everything
+
+1:18:17.296 --> 1:18:19.120
+so you're not.
+
+1:18:22.562 --> 1:18:29.986
+Then one thing, if you think about Bart, you
+want to have the Chinese language, the Italian
+
+1:18:29.986 --> 1:18:32.165
+language, and the deconer.
+
+1:18:32.165 --> 1:18:35.716
+However, in Bart we have the same language.
+
+1:18:36.516 --> 1:18:46.010
+The one you get is from English, so what you
+can do there is so you cannot try to do some.
+
+1:18:46.366 --> 1:18:52.562
+Below the barge, in order to learn some language
+specific stuff, or there's a masculine barge,
+
+1:18:52.562 --> 1:18:58.823
+which is trained on many languages, but it's
+trained only on like the Old Coast Modern Language
+
+1:18:58.823 --> 1:19:03.388
+House, which may be trained in German and English,
+but not on German.
+
+1:19:03.923 --> 1:19:08.779
+So then you would still need to find June
+and the model needs to learn how to better
+
+1:19:08.779 --> 1:19:10.721
+do the attention cross lingually.
+
+1:19:10.721 --> 1:19:15.748
+It's only on the same language but it mainly
+only has to learn this mapping and not all
+
+1:19:15.748 --> 1:19:18.775
+the rest and that's why it's still quite successful.
+
+1:19:21.982 --> 1:19:27.492
+Now certain thing which is very commonly used
+is what is required to it as adapters.
+
+1:19:27.607 --> 1:19:29.754
+So for example you take and buy.
+
+1:19:29.709 --> 1:19:35.218
+And you put some adapters on the inside of
+the networks so that it's small new layers
+
+1:19:35.218 --> 1:19:40.790
+which are in between put in there and then
+you only train these adapters or also train
+
+1:19:40.790 --> 1:19:41.815
+these adapters.
+
+1:19:41.815 --> 1:19:47.900
+For example, an embryo you could see that
+this learns to map the Sears language representation
+
+1:19:47.900 --> 1:19:50.334
+to the Tiger language representation.
+
+1:19:50.470 --> 1:19:52.395
+And then you don't have to change that luck.
+
+1:19:52.792 --> 1:19:59.793
+You give it extra ability to really perform
+well on that.
+
+1:19:59.793 --> 1:20:05.225
+These are quite small and so very efficient.
+
+1:20:05.905 --> 1:20:12.632
+That is also very commonly used, for example
+in modular systems where you have some adaptors
+
+1:20:12.632 --> 1:20:16.248
+in between here which might be language specific.
+
+1:20:16.916 --> 1:20:22.247
+So they are trained only for one language.
+
+1:20:22.247 --> 1:20:33.777
+The model has some or both and once has the
+ability to do multilingually to share knowledge.
+
+1:20:34.914 --> 1:20:39.058
+But there's one chance in general in the multilingual
+systems.
+
+1:20:39.058 --> 1:20:40.439
+It works quite well.
+
+1:20:40.439 --> 1:20:46.161
+There's one case or one specific use case
+for multilingual where this normally doesn't
+
+1:20:46.161 --> 1:20:47.344
+really work well.
+
+1:20:47.344 --> 1:20:49.975
+Do you have an idea what that could be?
+
+1:20:55.996 --> 1:20:57.536
+It's for Zero Shot Cases.
+
+1:20:57.998 --> 1:21:03.660
+Because having here some situation with this
+might be very language specific and zero shot,
+
+1:21:03.660 --> 1:21:09.015
+the idea is always to learn representations
+view which are more language dependent and
+
+1:21:09.015 --> 1:21:10.184
+with the adaptors.
+
+1:21:10.184 --> 1:21:15.601
+Of course you get in representations again
+which are more language specific and then it
+
+1:21:15.601 --> 1:21:17.078
+doesn't work that well.
+
+1:21:20.260 --> 1:21:37.730
+And there is also the idea of doing more knowledge
+pistolation.
+
+1:21:39.179 --> 1:21:42.923
+And now the idea is okay.
+
+1:21:42.923 --> 1:21:54.157
+We are training it the same, but what we want
+to achieve is that the encoder.
+
+1:21:54.414 --> 1:22:03.095
+So you should learn faster by trying to make
+these states as similar as possible.
+
+1:22:03.095 --> 1:22:11.777
+So you compare the first-hit state of the
+pre-trained model and try to make them.
+
+1:22:12.192 --> 1:22:18.144
+For example, by using the out two norms, so
+by just making these two representations the
+
+1:22:18.144 --> 1:22:26.373
+same: The same vocabulary: Why does it need
+the same vocabulary with any idea?
+
+1:22:34.754 --> 1:22:46.137
+If you have different vocabulary, it's typical
+you also have different sequenced lengths here.
+
+1:22:46.137 --> 1:22:50.690
+The number of sequences is different.
+
+1:22:51.231 --> 1:22:58.888
+If you now have pipe stains and four states
+here, it's no longer straightforward which
+
+1:22:58.888 --> 1:23:01.089
+states compare to which.
+
+1:23:02.322 --> 1:23:05.246
+And that's just easier if you have like the
+same number.
+
+1:23:05.246 --> 1:23:08.940
+You can always compare the first to the first
+and second to the second.
+
+1:23:09.709 --> 1:23:16.836
+So therefore at least the very easy way of
+knowledge destination only works if you have.
+
+1:23:17.177 --> 1:23:30.030
+Course: You could do things like yeah, the
+average should be the same, but of course there's
+
+1:23:30.030 --> 1:23:33.071
+a less strong signal.
+
+1:23:34.314 --> 1:23:42.979
+But the advantage here is that you have a
+diameter training signal here on the handquarter
+
+1:23:42.979 --> 1:23:51.455
+so you can directly make some of the encoder
+already giving a good signal while normally
+
+1:23:51.455 --> 1:23:52.407
+an empty.
+
+1:23:56.936 --> 1:24:13.197
+Yes, think this is most things for today,
+so what you should keep in mind is remind me.
+
+1:24:13.393 --> 1:24:18.400
+The one is a back translation idea.
+
+1:24:18.400 --> 1:24:29.561
+If you have monolingual and use that, the
+other one is to: And mentally it is often helpful
+
+1:24:29.561 --> 1:24:33.614
+to combine them so you can even use both of
+that.
+
+1:24:33.853 --> 1:24:38.908
+So you can use pre-trained walls, but then
+you can even still do back translation where
+
+1:24:38.908 --> 1:24:40.057
+it's still helpful.
+
+1:24:40.160 --> 1:24:45.502
+We have the advantage we are training like
+everything working together on the task so
+
+1:24:45.502 --> 1:24:51.093
+it might be helpful even to backtranslate some
+data and then use it in a real translation
+
+1:24:51.093 --> 1:24:56.683
+setup because in pretraining of course the
+beach challenge is always that you're training
+
+1:24:56.683 --> 1:24:57.739
+it on different.
+
+1:24:58.058 --> 1:25:03.327
+Different ways of how you integrate this knowledge.
+
+1:25:03.327 --> 1:25:08.089
+Even if you just use a full model, so in this.
+
+1:25:08.748 --> 1:25:11.128
+This is the most similar you can get.
+
+1:25:11.128 --> 1:25:13.945
+You're doing no changes to the architecture.
+
+1:25:13.945 --> 1:25:19.643
+You're really taking the model and just fine
+tuning them on the new task, but it still has
+
+1:25:19.643 --> 1:25:24.026
+to completely newly learn how to do the attention
+and how to do that.
+
+1:25:24.464 --> 1:25:29.971
+And that might be, for example, helpful to
+have more back-translated data to learn them.
+
+1:25:32.192 --> 1:25:34.251
+That's for today.
+
+1:25:34.251 --> 1:25:44.661
+There's one important thing that next Tuesday
+there is a conference or a workshop or so in
+
+1:25:44.661 --> 1:25:45.920
+this room.
+
+1:25:47.127 --> 1:25:56.769
+You should get an e-mail if you're in Elias
+that there's a room change for Tuesdays and
+
+1:25:56.769 --> 1:25:57.426
+it's.
+
+1:25:57.637 --> 1:26:03.890
+There are more questions, yeah, have a more
+general position, especially: In computer vision
+
+1:26:03.890 --> 1:26:07.347
+you can enlarge your data center data orientation.
+
+1:26:07.347 --> 1:26:08.295
+Is there any?
+
+1:26:08.388 --> 1:26:15.301
+It's similar to a large speech for text for
+the data of an edge.
+
+1:26:15.755 --> 1:26:29.176
+And you can use this back translation and
+also masking, but back translation is some
+
+1:26:29.176 --> 1:26:31.228
+way of data.
+
+1:26:31.371 --> 1:26:35.629
+So it has also been, for example, even its
+used not only for monolingual data.
+
+1:26:36.216 --> 1:26:54.060
+If you have good MP system, it can also be
+used for parallel data.
+
+1:26:54.834 --> 1:26:59.139
+So would say this is the most similar one.
+
+1:26:59.139 --> 1:27:03.143
+There's ways you can do power phrasing.
+
+1:27:05.025 --> 1:27:12.057
+But for example there is very hard to do this
+by rules like which words to replace because
+
+1:27:12.057 --> 1:27:18.936
+there is not a coup like you cannot always
+say this word can always be replaced by that.
+
+1:27:19.139 --> 1:27:27.225
+Mean, although they are many perfect synonyms,
+normally they are good in some cases, but not
+
+1:27:27.225 --> 1:27:29.399
+in all cases, and so on.
+
+1:27:29.399 --> 1:27:36.963
+And if you don't do a rule based, you have
+to train your model and then the freshness.
+
+1:27:38.058 --> 1:27:57.236
+The same architecture as the pre-trained mount.
+
+1:27:57.457 --> 1:27:59.810
+Should be of the same dimension, so it's easiest
+to have the same dimension.
+
+1:28:00.000 --> 1:28:01.590
+Architecture.
+
+1:28:01.590 --> 1:28:05.452
+We later will learn inefficiency.
+
+1:28:05.452 --> 1:28:12.948
+You can also do knowledge cessulation with,
+for example, smaller.
+
+1:28:12.948 --> 1:28:16.469
+You can learn the same within.
+
+1:28:17.477 --> 1:28:22.949
+Eight layers for it so that is possible, but
+yeah agree it should be of the same.
+
+1:28:23.623 --> 1:28:32.486
+Yeah yeah you need the question then of course
+you can do it like it's an initialization or
+
+1:28:32.486 --> 1:28:41.157
+you can do it doing training but normally it
+most makes sense during the normal training.
+
+1:28:45.865 --> 1:28:53.963
+Do it, then thanks a lot, and then we'll see
+each other again on Tuesday.
+
+0:00:00.981 --> 0:00:20.036
+Today about is how to use some type of additional
+resources to improve the translation.
+
+0:00:20.300 --> 0:00:28.188
+We have in the first part of the semester
+two thirds of the semester how to build some
+
+0:00:28.188 --> 0:00:31.361
+of your basic machine translation.
+
+0:00:31.571 --> 0:00:42.317
+Now the basic components are both for statistical
+and for neural, with the encoded decoding.
+
+0:00:43.123 --> 0:00:46.000
+Now, of course, that's not where it stops.
+
+0:00:46.000 --> 0:00:51.286
+It's still what nearly every machine translation
+system is currently in there.
+
+0:00:51.286 --> 0:00:57.308
+However, there's a lot of challenges which
+you need to address in addition and which need
+
+0:00:57.308 --> 0:00:58.245
+to be solved.
+
+0:00:58.918 --> 0:01:09.858
+And there we want to start to tell you what
+else can you do around this, and partly.
+
+0:01:10.030 --> 0:01:14.396
+And one important question there is on what
+do you train your models?
+
+0:01:14.394 --> 0:01:32.003
+Because like this type of parallel data, it's
+easier in machine translation than in other
+
+0:01:32.003 --> 0:01:33.569
+trusts.
+
+0:01:33.853 --> 0:01:41.178
+And therefore an important question is, can
+we also learn from like other sources and through?
+
+0:01:41.701 --> 0:01:47.830
+Because if you remember strongly right at
+the beginning of the election,.
+
+0:01:51.171 --> 0:01:53.801
+This Is How We Train All Our.
+
+0:01:54.194 --> 0:01:59.887
+Machine learning models from statistical to
+neural.
+
+0:01:59.887 --> 0:02:09.412
+This doesn't have changed so we need this
+type of parallel data where we have a source
+
+0:02:09.412 --> 0:02:13.462
+sentence aligned with a target data.
+
+0:02:13.493 --> 0:02:19.135
+We have now a strong model here, a very good
+model to do that.
+
+0:02:19.135 --> 0:02:22.091
+However, we always rely on this.
+
+0:02:22.522 --> 0:02:28.437
+For languages, high risk language pairs say
+from German to English or other European languages,
+
+0:02:28.437 --> 0:02:31.332
+there is decent amount at least for similarly.
+
+0:02:31.471 --> 0:02:37.630
+But even there if we are going to very specific
+domains it might get difficult and then your
+
+0:02:37.630 --> 0:02:43.525
+system performance might drop because if you
+want to translate now some medical text for
+
+0:02:43.525 --> 0:02:50.015
+example of course you need to also have peril
+data in the medical domain to know how to translate
+
+0:02:50.015 --> 0:02:50.876
+these types.
+
+0:02:51.231 --> 0:02:55.264
+Phrases how to use the vocabulary and so on
+in the style.
+
+0:02:55.915 --> 0:03:04.887
+And if you are going to other languages, there
+is a lot bigger challenge and the question
+
+0:03:04.887 --> 0:03:05.585
+there.
+
+0:03:05.825 --> 0:03:09.649
+So is really this the only resource we can
+use.
+
+0:03:09.889 --> 0:03:19.462
+Can be adapted or training phase in order
+to also make use of other types of models that
+
+0:03:19.462 --> 0:03:27.314
+might enable us to build strong systems with
+other types of information.
+
+0:03:27.707 --> 0:03:35.276
+And that we will look into now in the next
+starting from from just saying the next election.
+
+0:03:35.515 --> 0:03:40.697
+So this idea we already have covered on Tuesday.
+
+0:03:40.697 --> 0:03:45.350
+One very successful idea for this is to do.
+
+0:03:45.645 --> 0:03:51.990
+So that we're no longer doing translation
+between languages, but we can do translation
+
+0:03:51.990 --> 0:03:55.928
+between languages and share common knowledge
+between.
+
+0:03:56.296 --> 0:04:03.888
+You also learned about things like zero shots
+machine translation so you can translate between
+
+0:04:03.888 --> 0:04:06.446
+languages where you don't have.
+
+0:04:06.786 --> 0:04:09.790
+Which is the case for many, many language
+pairs.
+
+0:04:10.030 --> 0:04:16.954
+Like even with German, you have not translation
+parallel data to all languages around the world,
+
+0:04:16.954 --> 0:04:23.450
+or most of them you have it to the Europeans
+once, maybe even for Japanese, so it will get
+
+0:04:23.450 --> 0:04:26.377
+difficult to get a really decent amount.
+
+0:04:26.746 --> 0:04:35.332
+There is quite a lot of data, for example
+English to Japanese, but German to Japanese
+
+0:04:35.332 --> 0:04:37.827
+or German to Vietnamese.
+
+0:04:37.827 --> 0:04:41.621
+There is some data from Multilingual.
+
+0:04:42.042 --> 0:04:54.584
+So there is a very promising direction if
+you want to build translation systems between
+
+0:04:54.584 --> 0:05:00.142
+language peers, typically not English.
+
+0:05:01.221 --> 0:05:05.887
+And the other ideas, of course, we don't have
+to either just search for it.
+
+0:05:06.206 --> 0:05:12.505
+Some work on a data crawling so if I don't
+have a corpus directly or I don't have an high
+
+0:05:12.505 --> 0:05:19.014
+quality corpus like from the European Parliament
+for a TED corpus so maybe it makes sense to
+
+0:05:19.014 --> 0:05:23.913
+crawl more data and get additional sources
+so you can build stronger.
+
+0:05:24.344 --> 0:05:35.485
+There has been quite a big effort in Europe
+to collect really large data sets for parallel
+
+0:05:35.485 --> 0:05:36.220
+data.
+
+0:05:36.220 --> 0:05:40.382
+How can we do this data crawling?
+
+0:05:40.600 --> 0:05:46.103
+There the interesting thing from the machine
+translation point is not just general data
+
+0:05:46.103 --> 0:05:46.729
+crawling.
+
+0:05:47.067 --> 0:05:50.037
+But how can we explicitly crawl data?
+
+0:05:50.037 --> 0:05:52.070
+Which is some of a peril?
+
+0:05:52.132 --> 0:05:58.461
+So there is in the Internet quite a lot of
+data which has been company websites which
+
+0:05:58.461 --> 0:06:01.626
+have been translated and things like that.
+
+0:06:01.626 --> 0:06:05.158
+So how can you extract them parallel fragments?
+
+0:06:06.566 --> 0:06:13.404
+That is typically more noisy than where you
+do more at hands where mean if you have Parliament.
+
+0:06:13.693 --> 0:06:17.680
+You can do some rules how to extract parallel
+things.
+
+0:06:17.680 --> 0:06:24.176
+Here there is more to it, so the quality is
+later maybe not as good, but normally scale
+
+0:06:24.176 --> 0:06:26.908
+is then a possibility to address it.
+
+0:06:26.908 --> 0:06:30.304
+So you just have so much more data that even.
+
+0:06:33.313 --> 0:06:40.295
+The other thing can be used monolingual data
+and monolingual data has a big advantage that
+
+0:06:40.295 --> 0:06:46.664
+we can have a huge amount of that so that you
+can be autocrawed from the Internet.
+
+0:06:46.664 --> 0:06:51.728
+The nice thing is you can also get it typically
+for many domains.
+
+0:06:52.352 --> 0:06:59.558
+There is just so much more magnitude of monolingual
+data so that it might be very helpful.
+
+0:06:59.559 --> 0:07:03.054
+We can do that in statistical machine translation.
+
+0:07:03.054 --> 0:07:06.755
+It was quite easy to integrate using language
+models.
+
+0:07:08.508 --> 0:07:16.912
+In neural machine translation we have the
+advantage that we have this overall architecture
+
+0:07:16.912 --> 0:07:22.915
+that does everything together, but it has also
+the disadvantage.
+
+0:07:23.283 --> 0:07:25.675
+We'll look today at two things.
+
+0:07:25.675 --> 0:07:32.925
+On the one end you can still try to do a bit
+of language modeling in there and add an additional
+
+0:07:32.925 --> 0:07:35.168
+language model into in there.
+
+0:07:35.168 --> 0:07:38.232
+There is some work, one very successful.
+
+0:07:38.178 --> 0:07:43.764
+A way in which I think is used in most systems
+at the moment is to do some scientific data.
+
+0:07:43.763 --> 0:07:53.087
+Is a very easy thing, but you can just translate
+there and use it as training gator, and normally.
+
+0:07:53.213 --> 0:07:59.185
+And thereby you are able to use like some
+type of monolingual a day.
+
+0:08:00.380 --> 0:08:05.271
+Another way to do it is unsupervised and the
+extreme case.
+
+0:08:05.271 --> 0:08:11.158
+If you have a scenario then you only have
+data, only monolingual data.
+
+0:08:11.158 --> 0:08:13.976
+Can you still build translations?
+
+0:08:14.754 --> 0:08:27.675
+If you have large amounts of data and languages
+are not too dissimilar, you can build translation
+
+0:08:27.675 --> 0:08:31.102
+systems without parallel.
+
+0:08:32.512 --> 0:08:36.267
+That we will see you then next Thursday.
+
+0:08:37.857 --> 0:08:50.512
+And then there is now a third type of pre-trained
+model that recently became very successful
+
+0:08:50.512 --> 0:08:55.411
+and now with large language models.
+
+0:08:55.715 --> 0:09:03.525
+So the idea is we are no longer sharing the
+real data, but it can also help to train a
+
+0:09:03.525 --> 0:09:04.153
+model.
+
+0:09:04.364 --> 0:09:11.594
+And that is now a big advantage of deep learning
+based approaches.
+
+0:09:11.594 --> 0:09:22.169
+There you have this ability that you can train
+a model in some task and then apply it to another.
+
+0:09:22.722 --> 0:09:33.405
+And then, of course, the question is, can
+I have an initial task where there's huge amounts
+
+0:09:33.405 --> 0:09:34.450
+of data?
+
+0:09:34.714 --> 0:09:40.251
+And the test that typically you pre train
+on is more like similar to a language moral
+
+0:09:40.251 --> 0:09:45.852
+task either direct to a language moral task
+or like a masking task which is related so
+
+0:09:45.852 --> 0:09:51.582
+the idea is oh I can train on this data and
+the knowledge about words how they relate to
+
+0:09:51.582 --> 0:09:53.577
+each other I can use in there.
+
+0:09:53.753 --> 0:10:00.276
+So it's a different way of using language
+models.
+
+0:10:00.276 --> 0:10:06.276
+There's more transfer learning at the end
+of.
+
+0:10:09.029 --> 0:10:17.496
+So first we will start with how can we use
+monolingual data to do a Yeah to do a machine
+
+0:10:17.496 --> 0:10:18.733
+translation?
+
+0:10:20.040 --> 0:10:27.499
+That: Big difference is you should remember
+from what I mentioned before is.
+
+0:10:27.499 --> 0:10:32.783
+In statistical machine translation we directly
+have the opportunity.
+
+0:10:32.783 --> 0:10:39.676
+There's peril data for the translation model
+and monolingual data for the language model.
+
+0:10:39.679 --> 0:10:45.343
+And you combine your translation model and
+language model, and then you can make use of
+
+0:10:45.343 --> 0:10:45.730
+both.
+
+0:10:46.726 --> 0:10:53.183
+That you can make use of these large large
+amounts of monolingual data, but of course
+
+0:10:53.183 --> 0:10:55.510
+it has also some disadvantage.
+
+0:10:55.495 --> 0:11:01.156
+Because we say the problem is we are optimizing
+both parts a bit independently to each other
+
+0:11:01.156 --> 0:11:06.757
+and we say oh yeah the big disadvantage of
+newer machine translations now we are optimizing
+
+0:11:06.757 --> 0:11:10.531
+the overall architecture everything together
+to perform best.
+
+0:11:10.890 --> 0:11:16.994
+And then, of course, we can't do there, so
+Leo we can can only do a mural like use power
+
+0:11:16.994 --> 0:11:17.405
+data.
+
+0:11:17.897 --> 0:11:28.714
+So the question is, but this advantage is
+not so important that we can train everything,
+
+0:11:28.714 --> 0:11:35.276
+but we have a moral legal data or even small
+amounts.
+
+0:11:35.675 --> 0:11:43.102
+So in data we know it's not only important
+the amount of data we have but also like how
+
+0:11:43.102 --> 0:11:50.529
+similar it is to your test data so it can be
+that this modeling data is quite small but
+
+0:11:50.529 --> 0:11:55.339
+it's very well fitting and then it's still
+very helpful.
+
+0:11:55.675 --> 0:12:02.691
+At the first year of surprisingness, if we
+are here successful with integrating a language
+
+0:12:02.691 --> 0:12:09.631
+model into a translation system, maybe we can
+also integrate some type of language models
+
+0:12:09.631 --> 0:12:14.411
+into our empty system in order to make it better
+and perform.
+
+0:12:16.536 --> 0:12:23.298
+The first thing we can do is we know there
+is language models, so let's try to integrate.
+
+0:12:23.623 --> 0:12:31.096
+There was our language model because these
+works were mainly done before transformer-based
+
+0:12:31.096 --> 0:12:31.753
+models.
+
+0:12:32.152 --> 0:12:38.764
+In general, of course, you can do the same
+thing with transformer baseball.
+
+0:12:38.764 --> 0:12:50.929
+There is nothing about whether: It's just
+that it has mainly been done before people
+
+0:12:50.929 --> 0:13:01.875
+started using R&S and they tried to do
+this more in cases.
+
+0:13:07.087 --> 0:13:22.938
+So what we're happening here is in some of
+this type of idea, and in key system you remember
+
+0:13:22.938 --> 0:13:25.495
+the attention.
+
+0:13:25.605 --> 0:13:29.465
+Gets it was your last in this day that you
+calculate easy attention.
+
+0:13:29.729 --> 0:13:36.610
+We get the context back, then combine both
+and then base the next in state and then predict.
+
+0:13:37.057 --> 0:13:42.424
+So this is our system, and the question is,
+can we send our integrated language model?
+
+0:13:42.782 --> 0:13:49.890
+And somehow it makes sense to take out a neural
+language model because we are anyway in the
+
+0:13:49.890 --> 0:13:50.971
+neural space.
+
+0:13:50.971 --> 0:13:58.465
+It's not surprising that it contrasts to statistical
+work used and grants it might make sense to
+
+0:13:58.465 --> 0:14:01.478
+take a bit of a normal language model.
+
+0:14:01.621 --> 0:14:06.437
+And there would be something like on Tubbles
+Air, a neural language model, and our man based
+
+0:14:06.437 --> 0:14:11.149
+is you have a target word, you put it in, you
+get a new benchmark, and then you always put
+
+0:14:11.149 --> 0:14:15.757
+in the words and get new hidden states, and
+you can do some predictions at the output to
+
+0:14:15.757 --> 0:14:16.948
+predict the next word.
+
+0:14:17.597 --> 0:14:26.977
+So if we're having this type of in language
+model, there's like two main questions we have
+
+0:14:26.977 --> 0:14:34.769
+to answer: So how do we combine now on the
+one hand our system and on the other hand our
+
+0:14:34.769 --> 0:14:35.358
+model?
+
+0:14:35.358 --> 0:14:42.004
+You see that was mentioned before when we
+started talking about ENCODA models.
+
+0:14:42.004 --> 0:14:45.369
+They can be viewed as a language model.
+
+0:14:45.805 --> 0:14:47.710
+The wine is lengthened, unconditioned.
+
+0:14:47.710 --> 0:14:49.518
+It's just modeling the target sides.
+
+0:14:49.970 --> 0:14:56.963
+And the other one is a conditional language
+one, which is a language one conditioned on
+
+0:14:56.963 --> 0:14:57.837
+the Sewer.
+
+0:14:58.238 --> 0:15:03.694
+So how can you combine to language models?
+
+0:15:03.694 --> 0:15:14.860
+Of course, it's like the translation model
+will be more important because it has access
+
+0:15:14.860 --> 0:15:16.763
+to the source.
+
+0:15:18.778 --> 0:15:22.571
+If we have that, the other question is okay.
+
+0:15:22.571 --> 0:15:24.257
+Now we have models.
+
+0:15:24.257 --> 0:15:25.689
+How do we train?
+
+0:15:26.026 --> 0:15:30.005
+Pickers integrated them.
+
+0:15:30.005 --> 0:15:34.781
+We have now two sets of data.
+
+0:15:34.781 --> 0:15:42.741
+We have parallel data where you can do the
+lower.
+
+0:15:44.644 --> 0:15:53.293
+So the first idea is we can do something more
+like a parallel combination.
+
+0:15:53.293 --> 0:15:55.831
+We just keep running.
+
+0:15:56.036 --> 0:15:59.864
+So here you see your system that is running.
+
+0:16:00.200 --> 0:16:09.649
+It's normally completely independent of your
+language model, which is up there, so down
+
+0:16:09.649 --> 0:16:13.300
+here we have just our NMT system.
+
+0:16:13.313 --> 0:16:26.470
+The only thing which is used is we have the
+words, and of course they are put into both
+
+0:16:26.470 --> 0:16:30.059
+systems, and out there.
+
+0:16:30.050 --> 0:16:42.221
+So we use them somehow for both, and then
+we are doing our decision just by merging these
+
+0:16:42.221 --> 0:16:42.897
+two.
+
+0:16:43.343 --> 0:16:53.956
+So there can be, for example, we are doing
+a probability distribution here, and then we
+
+0:16:53.956 --> 0:17:03.363
+are taking the average of post-perability distribution
+to do our predictions.
+
+0:17:11.871 --> 0:17:18.923
+You could also take the output with Steve's
+to be more in chore about the mixture.
+
+0:17:20.000 --> 0:17:32.896
+Yes, you could also do that, so it's more
+like engaging mechanisms that you're not doing.
+
+0:17:32.993 --> 0:17:41.110
+Another one would be cochtrinate the hidden
+states, and then you would have another layer
+
+0:17:41.110 --> 0:17:41.831
+on top.
+
+0:17:43.303 --> 0:17:56.889
+You think about if you do the conqueredination
+instead of taking the instead and then merging
+
+0:17:56.889 --> 0:18:01.225
+the probability distribution.
+
+0:18:03.143 --> 0:18:16.610
+Introduce many new parameters, and these parameters
+have somehow something special compared to
+
+0:18:16.610 --> 0:18:17.318
+the.
+
+0:18:23.603 --> 0:18:37.651
+So before all the error other parameters can
+be trained independent, the language model
+
+0:18:37.651 --> 0:18:42.121
+can be trained independent.
+
+0:18:43.043 --> 0:18:51.749
+If you have a joint layer, of course you need
+to train them because you have now inputs.
+
+0:18:54.794 --> 0:19:02.594
+Not surprisingly, if you have a parallel combination
+of whether you could, the other way is to do
+
+0:19:02.594 --> 0:19:04.664
+more serial combinations.
+
+0:19:04.924 --> 0:19:10.101
+How can you do a similar combination?
+
+0:19:10.101 --> 0:19:18.274
+Your final decision makes sense to do a face
+on the system.
+
+0:19:18.438 --> 0:19:20.996
+So you have on top of your normal and system.
+
+0:19:21.121 --> 0:19:30.678
+The only thing is now you're inputting into
+your system.
+
+0:19:30.678 --> 0:19:38.726
+You're no longer inputting the word embeddings.
+
+0:19:38.918 --> 0:19:45.588
+So you're training your mainly what you have
+your lower layers here which are trained more
+
+0:19:45.588 --> 0:19:52.183
+on the purely language model style and then
+on top your putting into the NMT system where
+
+0:19:52.183 --> 0:19:55.408
+it now has already here the language model.
+
+0:19:55.815 --> 0:19:58.482
+So here you can also view it.
+
+0:19:58.482 --> 0:20:06.481
+Here you have more contextual embeddings which
+no longer depend only on the word but they
+
+0:20:06.481 --> 0:20:10.659
+also depend on the context of the target site.
+
+0:20:11.051 --> 0:20:19.941
+But you have more understanding of the source
+word, so you have a language in the current
+
+0:20:19.941 --> 0:20:21.620
+target sentence.
+
+0:20:21.881 --> 0:20:27.657
+So if it's like the word can, for example,
+will be put in here always the same independent
+
+0:20:27.657 --> 0:20:31.147
+of its user can of beans, or if it's like I
+can do it.
+
+0:20:31.147 --> 0:20:37.049
+However, because you are having your language
+model style, you have maybe disintegrated this
+
+0:20:37.049 --> 0:20:40.984
+already a bit, and you give this information
+directly to the.
+
+0:20:41.701 --> 0:20:43.095
+An empty cyst.
+
+0:20:44.364 --> 0:20:49.850
+You, if you're remembering more the transformer
+based approach, you have some layers.
+
+0:20:49.850 --> 0:20:55.783
+The lower layers are purely languaged while
+the other ones are with attention to the source.
+
+0:20:55.783 --> 0:21:01.525
+So you can view it also that you just have
+lower layers which don't attend to the source.
+
+0:21:02.202 --> 0:21:07.227
+This is purely a language model, and then
+at some point you're starting to attend to
+
+0:21:07.227 --> 0:21:08.587
+the source and use it.
+
+0:21:13.493 --> 0:21:20.781
+Yes, so this is how you combine them in peril
+or first do the language model and then do.
+
+0:21:23.623 --> 0:21:26.147
+Questions for the integration.
+
+0:21:31.831 --> 0:21:35.034
+Not really sure about the input of the.
+
+0:21:35.475 --> 0:21:38.102
+Model, and in this case in the sequence.
+
+0:21:38.278 --> 0:21:54.854
+Case so the actual word that we transferred
+into a numerical lecture, and this is an input.
+
+0:21:56.176 --> 0:22:03.568
+That depends on if you view the word embedding
+as part of the language model.
+
+0:22:03.568 --> 0:22:10.865
+So if you first put the word target word then
+you do the one hot end coding.
+
+0:22:11.691 --> 0:22:13.805
+And then the word embedding there is the r&
+
+0:22:13.805 --> 0:22:13.937
+n.
+
+0:22:14.314 --> 0:22:21.035
+So you can use this together as your language
+model when you first do the word embedding.
+
+0:22:21.401 --> 0:22:24.346
+All you can say is like before.
+
+0:22:24.346 --> 0:22:28.212
+It's more a definition, but you're right.
+
+0:22:28.212 --> 0:22:30.513
+So what's the steps out?
+
+0:22:30.513 --> 0:22:36.128
+You take the word, the one hut encoding, the
+word embedding.
+
+0:22:36.516 --> 0:22:46.214
+What one of these parrots, you know, called
+a language model is definition wise and not
+
+0:22:46.214 --> 0:22:47.978
+that important.
+
+0:22:53.933 --> 0:23:02.264
+So the question is how can you then train
+them and make this this one work?
+
+0:23:02.264 --> 0:23:02.812
+The.
+
+0:23:03.363 --> 0:23:15.201
+So in the case where you combine the language
+one of the abilities you can train them independently
+
+0:23:15.201 --> 0:23:18.516
+and just put them together.
+
+0:23:18.918 --> 0:23:27.368
+Might not be the best because we have no longer
+the stability that we had before that optimally
+
+0:23:27.368 --> 0:23:29.128
+performed together.
+
+0:23:29.128 --> 0:23:33.881
+It's not clear if they really work the best
+together.
+
+0:23:34.514 --> 0:23:41.585
+At least you need to somehow find how much
+do you trust the one model and how much.
+
+0:23:43.323 --> 0:23:45.058
+Still in some cases useful.
+
+0:23:45.058 --> 0:23:48.530
+It might be helpful if you have only data
+and software.
+
+0:23:48.928 --> 0:23:59.064
+However, in MT we have one specific situation
+that at least for the MT part parallel is also
+
+0:23:59.064 --> 0:24:07.456
+always monolingual data, so what we definitely
+can do is train the language.
+
+0:24:08.588 --> 0:24:18.886
+So what we also can do is more like the pre-training
+approach.
+
+0:24:18.886 --> 0:24:24.607
+We first train the language model.
+
+0:24:24.704 --> 0:24:27.334
+The pre-training approach.
+
+0:24:27.334 --> 0:24:33.470
+You first train on the monolingual data and
+then you join the.
+
+0:24:33.933 --> 0:24:41.143
+Of course, the model size is this way, but
+the data size is too bigly the other way around.
+
+0:24:41.143 --> 0:24:47.883
+You often have a lot more monolingual data
+than you have here parallel data, in which
+
+0:24:47.883 --> 0:24:52.350
+scenario can you imagine where this type of
+pretraining?
+
+0:24:56.536 --> 0:24:57.901
+Any Ideas.
+
+0:25:04.064 --> 0:25:12.772
+One example where this might also be helpful
+if you want to adapt to domains.
+
+0:25:12.772 --> 0:25:22.373
+So let's say you do medical sentences and
+if you want to translate medical sentences.
+
+0:25:23.083 --> 0:25:26.706
+In this case it could be or its most probable
+happen.
+
+0:25:26.706 --> 0:25:32.679
+You're learning here up there what medical
+means, but in your fine tuning step the model
+
+0:25:32.679 --> 0:25:38.785
+is forgotten everything about Medicare, so
+you may be losing all the information you gain.
+
+0:25:39.099 --> 0:25:42.366
+So this type of priest training step is good.
+
+0:25:42.366 --> 0:25:47.978
+If your pretraining data is more general,
+very large and then you're adapting.
+
+0:25:48.428 --> 0:25:56.012
+But in the task with moral lingual data, which
+should be used to adapt the system to some
+
+0:25:56.012 --> 0:25:57.781
+general topic style.
+
+0:25:57.817 --> 0:26:06.795
+Then, of course, this is not a good strategy
+because you might forgot about everything up
+
+0:26:06.795 --> 0:26:09.389
+there and you don't have.
+
+0:26:09.649 --> 0:26:14.678
+So then you have to check what you can do
+for them.
+
+0:26:14.678 --> 0:26:23.284
+You can freeze this part and change it any
+more so you don't lose the ability or you can
+
+0:26:23.284 --> 0:26:25.702
+do a direct combination.
+
+0:26:25.945 --> 0:26:31.028
+Where you jointly train both of them, so you
+train the NMT system on the, and then you train
+
+0:26:31.028 --> 0:26:34.909
+the language model always in parallels so that
+you don't forget about.
+
+0:26:35.395 --> 0:26:37.684
+And what you learn of the length.
+
+0:26:37.937 --> 0:26:46.711
+Depends on what you want to combine because
+it's large data and you have a good general
+
+0:26:46.711 --> 0:26:48.107
+knowledge in.
+
+0:26:48.548 --> 0:26:55.733
+Then you normally don't really forget it because
+it's also in the or you use it to adapt to
+
+0:26:55.733 --> 0:26:57.295
+something specific.
+
+0:26:57.295 --> 0:26:58.075
+Then you.
+
+0:27:01.001 --> 0:27:06.676
+Then this is a way of how we can make use
+of monolingual data.
+
+0:27:07.968 --> 0:27:12.116
+It seems to be the easiest one somehow.
+
+0:27:12.116 --> 0:27:20.103
+It's more similar to what we are doing with
+statistical machine translation.
+
+0:27:21.181 --> 0:27:31.158
+Normally always beats this type of model,
+which in some view can be like from the conceptual
+
+0:27:31.158 --> 0:27:31.909
+thing.
+
+0:27:31.909 --> 0:27:36.844
+It's even easier from the computational side.
+
+0:27:40.560 --> 0:27:42.078
+And the idea is OK.
+
+0:27:42.078 --> 0:27:49.136
+We have monolingual data that we just translate
+and then generate some type of parallel data
+
+0:27:49.136 --> 0:27:50.806
+and use that then to.
+
+0:27:51.111 --> 0:28:00.017
+So if you want to build a German-to-English
+system first, take the large amount of data
+
+0:28:00.017 --> 0:28:02.143
+you have translated.
+
+0:28:02.402 --> 0:28:10.446
+Then you have more peril data and the interesting
+thing is if you then train on the joint thing
+
+0:28:10.446 --> 0:28:18.742
+or on the original peril data and on what is
+artificial where you have generated the translations.
+
+0:28:18.918 --> 0:28:26.487
+So you can because you are not doing the same
+era all the times and you have some knowledge.
+
+0:28:28.028 --> 0:28:43.199
+With this first approach, however, there is
+one issue why it might not work the best.
+
+0:28:49.409 --> 0:28:51.177
+Very a bit shown in the image to you.
+
+0:28:53.113 --> 0:28:58.153
+You trade on that quality data.
+
+0:28:58.153 --> 0:29:02.563
+Here is a bit of a problem.
+
+0:29:02.563 --> 0:29:08.706
+Your English style is not really good.
+
+0:29:08.828 --> 0:29:12.213
+And as you're saying, the system always mistranslates.
+
+0:29:13.493 --> 0:29:19.798
+Something then you will learn that this is
+correct because now it's a training game and
+
+0:29:19.798 --> 0:29:23.022
+you will encourage it to make it more often.
+
+0:29:23.022 --> 0:29:29.614
+So the problem with training on your own areas
+yeah you might prevent some areas you rarely
+
+0:29:29.614 --> 0:29:29.901
+do.
+
+0:29:30.150 --> 0:29:31.749
+But errors use systematically.
+
+0:29:31.749 --> 0:29:34.225
+Do you even enforce more and will even do
+more?
+
+0:29:34.654 --> 0:29:40.145
+So that might not be the best solution to
+have any idea how you could do it better.
+
+0:29:44.404 --> 0:29:57.754
+Is one way there is even a bit of more simple
+idea.
+
+0:30:04.624 --> 0:30:10.975
+The problem is yeah, the translations are
+not perfect, so the output and you're learning
+
+0:30:10.975 --> 0:30:12.188
+something wrong.
+
+0:30:12.188 --> 0:30:17.969
+Normally it's less bad if your inputs are
+not bad, but your outputs are perfect.
+
+0:30:18.538 --> 0:30:24.284
+So if your inputs are wrong you may learn
+that if you're doing this wrong input you're
+
+0:30:24.284 --> 0:30:30.162
+generating something correct, but you're not
+learning to generate something which is not
+
+0:30:30.162 --> 0:30:30.756
+correct.
+
+0:30:31.511 --> 0:30:47.124
+So often the case it is that it is more important
+than your target is correct.
+
+0:30:47.347 --> 0:30:52.182
+But you can assume in your application scenario
+you hope that you may only get correct inputs.
+
+0:30:52.572 --> 0:31:02.535
+So that is not harming you, and in machine
+translation we have one very nice advantage:
+
+0:31:02.762 --> 0:31:04.648
+And also the other way around.
+
+0:31:04.648 --> 0:31:10.062
+It's a very similar task, so there's a task
+to translate from German to English, but the
+
+0:31:10.062 --> 0:31:13.894
+task to translate from English to German is
+very similar, and.
+
+0:31:14.094 --> 0:31:19.309
+So what we can do is we can just switch it
+initially and generate the data the other way
+
+0:31:19.309 --> 0:31:19.778
+around.
+
+0:31:20.120 --> 0:31:25.959
+So what we are doing here is we are starting
+with an English to German system.
+
+0:31:25.959 --> 0:31:32.906
+Then we are translating the English data into
+German where the German is maybe not very nice.
+
+0:31:33.293 --> 0:31:51.785
+And then we are training on our original data
+and on the back translated data.
+
+0:31:52.632 --> 0:32:02.332
+So here we have the advantage that our target
+side is human quality and only the input.
+
+0:32:03.583 --> 0:32:08.113
+Then this helps us to get really good.
+
+0:32:08.113 --> 0:32:15.431
+There is one difference if you think about
+the data resources.
+
+0:32:21.341 --> 0:32:27.336
+Too obvious here we need a target site monolingual
+layer.
+
+0:32:27.336 --> 0:32:31.574
+In the first example we had source site.
+
+0:32:31.931 --> 0:32:45.111
+So back translation is normally working if
+you have target size peril later and not search
+
+0:32:45.111 --> 0:32:48.152
+side modeling later.
+
+0:32:48.448 --> 0:32:56.125
+Might be also, like if you think about it,
+understand a little better to understand the
+
+0:32:56.125 --> 0:32:56.823
+target.
+
+0:32:57.117 --> 0:33:01.469
+On the source side you have to understand
+the content.
+
+0:33:01.469 --> 0:33:08.749
+On the target side you have to generate really
+sentences and somehow it's more difficult to
+
+0:33:08.749 --> 0:33:12.231
+generate something than to only understand.
+
+0:33:17.617 --> 0:33:30.734
+This works well if you have to select how
+many back translated data do you use.
+
+0:33:31.051 --> 0:33:32.983
+Because only there's like a lot more.
+
+0:33:33.253 --> 0:33:42.136
+Question: Should take all of my data there
+is two problems with it?
+
+0:33:42.136 --> 0:33:51.281
+Of course it's expensive because you have
+to translate all this data.
+
+0:33:51.651 --> 0:34:00.946
+So if you don't know the normal good starting
+point is to take equal amount of data as many
+
+0:34:00.946 --> 0:34:02.663
+back translated.
+
+0:34:02.963 --> 0:34:04.673
+It depends on the used case.
+
+0:34:04.673 --> 0:34:08.507
+If we have very few data here, it makes more
+sense to have more.
+
+0:34:08.688 --> 0:34:15.224
+Depends on how good your quality is here,
+so the better the more data you might use because
+
+0:34:15.224 --> 0:34:16.574
+quality is better.
+
+0:34:16.574 --> 0:34:22.755
+So it depends on a lot of things, but your
+rule of sum is like which general way often
+
+0:34:22.755 --> 0:34:24.815
+is to have equal amounts of.
+
+0:34:26.646 --> 0:34:29.854
+And you can, of course, do that now.
+
+0:34:29.854 --> 0:34:34.449
+I said already that it's better to have the
+quality.
+
+0:34:34.449 --> 0:34:38.523
+At the end, of course, depends on this system.
+
+0:34:38.523 --> 0:34:46.152
+Also, because the better this system is, the
+better your synthetic data is, the better.
+
+0:34:47.207 --> 0:34:50.949
+That leads to what is referred to as iterated
+back translation.
+
+0:34:51.291 --> 0:34:56.917
+So you play them on English to German, and
+you translate the data on.
+
+0:34:56.957 --> 0:35:03.198
+Then you train a model on German to English
+with the additional data.
+
+0:35:03.198 --> 0:35:09.796
+Then you translate German data and then you
+train to gain your first one.
+
+0:35:09.796 --> 0:35:14.343
+So in the second iteration this quality is
+better.
+
+0:35:14.334 --> 0:35:19.900
+System is better because it's not only trained
+on the small data but additionally on back
+
+0:35:19.900 --> 0:35:22.003
+translated data with this system.
+
+0:35:22.442 --> 0:35:24.458
+And so you can get better.
+
+0:35:24.764 --> 0:35:28.053
+However, typically you can stop quite early.
+
+0:35:28.053 --> 0:35:35.068
+Maybe one iteration is good, but then you
+have diminishing gains after two or three iterations.
+
+0:35:35.935 --> 0:35:46.140
+There is very slight difference because you
+need a quite big difference in the quality
+
+0:35:46.140 --> 0:35:46.843
+here.
+
+0:35:47.207 --> 0:36:02.262
+Language is also good because it means you
+can already train it with relatively bad profiles.
+
+0:36:03.723 --> 0:36:10.339
+It's a design decision would advise so guess
+because it's easy to get it.
+
+0:36:10.550 --> 0:36:20.802
+Replace that because you have a higher quality
+real data, but then I think normally it's okay
+
+0:36:20.802 --> 0:36:22.438
+to replace it.
+
+0:36:22.438 --> 0:36:28.437
+I would assume it's not too much of a difference,
+but.
+
+0:36:34.414 --> 0:36:42.014
+That's about like using monolingual data before
+we go into the pre-train models to have any
+
+0:36:42.014 --> 0:36:43.005
+more crash.
+
+0:36:49.029 --> 0:36:55.740
+Yes, so the other thing which we can do and
+which is recently more and more successful
+
+0:36:55.740 --> 0:37:02.451
+and even more successful since we have this
+really large language models where you can
+
+0:37:02.451 --> 0:37:08.545
+even do the translation task with this is the
+way of using pre-trained models.
+
+0:37:08.688 --> 0:37:16.135
+So you learn a representation of one task,
+and then you use this representation from another.
+
+0:37:16.576 --> 0:37:26.862
+It was made maybe like one of the first words
+where it really used largely is doing something
+
+0:37:26.862 --> 0:37:35.945
+like a bird which you pre trained on purely
+text era and you take it in fine tune.
+
+0:37:36.496 --> 0:37:42.953
+And one big advantage, of course, is that
+people can only share data but also pre-trained.
+
+0:37:43.423 --> 0:37:59.743
+The recent models and the large language ones
+which are available.
+
+0:37:59.919 --> 0:38:09.145
+Where I think it costs several millions to
+train them all, just if you would buy the GPUs
+
+0:38:09.145 --> 0:38:15.397
+from some cloud company and train that the
+cost of training.
+
+0:38:15.475 --> 0:38:21.735
+And guess as a student project you won't have
+the budget to like build these models.
+
+0:38:21.801 --> 0:38:24.598
+So another idea is what you can do is okay.
+
+0:38:24.598 --> 0:38:27.330
+Maybe if these months are once available,.
+
+0:38:27.467 --> 0:38:36.598
+Can take them and use them as an also resource
+similar to pure text, and you can now build
+
+0:38:36.598 --> 0:38:44.524
+models which somehow learn not only from from
+data but also from other models.
+
+0:38:44.844 --> 0:38:49.127
+So it's a quite new way of thinking of how
+to train.
+
+0:38:49.127 --> 0:38:53.894
+We are not only learning from examples, but
+we might also.
+
+0:38:54.534 --> 0:39:05.397
+The nice thing is that this type of training
+where we are not learning directly from data
+
+0:39:05.397 --> 0:39:07.087
+but learning.
+
+0:39:07.427 --> 0:39:17.647
+So the main idea this go is you have a person
+initial task.
+
+0:39:17.817 --> 0:39:26.369
+And if you're working with anLP, that means
+you're training pure taxator because that's
+
+0:39:26.369 --> 0:39:30.547
+where you have the largest amount of data.
+
+0:39:30.951 --> 0:39:35.854
+And then you're defining some type of task
+in order to you do your creek training.
+
+0:39:36.176 --> 0:39:43.092
+And: The typical task you can train on on
+that is like the language waddling task.
+
+0:39:43.092 --> 0:39:50.049
+So to predict the next word or we have a related
+task to predict something in between, we'll
+
+0:39:50.049 --> 0:39:52.667
+see depending on the architecture.
+
+0:39:52.932 --> 0:39:58.278
+But somehow to predict something which you
+have not in the input is a task which is easy
+
+0:39:58.278 --> 0:40:00.740
+to generate, so you just need your data.
+
+0:40:00.740 --> 0:40:06.086
+That's why it's called self supervised, so
+you're creating your supervised pending data.
+
+0:40:06.366 --> 0:40:07.646
+By yourself.
+
+0:40:07.646 --> 0:40:15.133
+On the other hand, you need a lot of knowledge
+and that is the other thing.
+
+0:40:15.735 --> 0:40:24.703
+Because there is this idea that the meaning
+of a word heavily depends on the context that.
+
+0:40:25.145 --> 0:40:36.846
+So can give you a sentence with some giverish
+word and there's some name and although you've
+
+0:40:36.846 --> 0:40:41.627
+never heard the name you will assume.
+
+0:40:42.062 --> 0:40:44.149
+And exactly the same thing.
+
+0:40:44.149 --> 0:40:49.143
+The models can also learn something about
+the world by just using.
+
+0:40:49.649 --> 0:40:53.651
+So that is typically the mule.
+
+0:40:53.651 --> 0:40:59.848
+Then we can use this model to train the system.
+
+0:41:00.800 --> 0:41:03.368
+Course we might need to adapt the system.
+
+0:41:03.368 --> 0:41:07.648
+To do that we have to change the architecture
+we might use only some.
+
+0:41:07.627 --> 0:41:09.443
+Part of the pre-trained model.
+
+0:41:09.443 --> 0:41:14.773
+In there we have seen that a bit already in
+the R&N case you can also see that we have
+
+0:41:14.773 --> 0:41:17.175
+also mentioned the pre-training already.
+
+0:41:17.437 --> 0:41:22.783
+So you can use the R&N as one of these
+approaches.
+
+0:41:22.783 --> 0:41:28.712
+You train the R&M language more on large
+pre-train data.
+
+0:41:28.712 --> 0:41:32.309
+Then you put it somewhere into your.
+
+0:41:33.653 --> 0:41:37.415
+So this gives you the ability to really do
+these types of tests.
+
+0:41:37.877 --> 0:41:53.924
+So you can build a system which is knowledge,
+which is just trained on large amounts of data.
+
+0:41:56.376 --> 0:42:01.564
+So the question is maybe what type of information
+so what type of models can you?
+
+0:42:01.821 --> 0:42:05.277
+And we want today to look at briefly at swings.
+
+0:42:05.725 --> 0:42:08.850
+That was what was initially done.
+
+0:42:08.850 --> 0:42:17.213
+It wasn't as famous as in machine translation
+as in other things, but it's also used there
+
+0:42:17.213 --> 0:42:21.072
+and that is to use static word embedding.
+
+0:42:21.221 --> 0:42:28.981
+So we have this mapping from the one hot to
+a small continuous word representation.
+
+0:42:29.229 --> 0:42:38.276
+Using this one in your NG system, so you can,
+for example, replace the embedding layer by
+
+0:42:38.276 --> 0:42:38.779
+the.
+
+0:42:39.139 --> 0:42:41.832
+That is helpful to be a really small amount
+of data.
+
+0:42:42.922 --> 0:42:48.517
+And we're always in this pre-training phase
+and have the thing the advantage is.
+
+0:42:48.468 --> 0:42:52.411
+More data than the trade off, so you can get
+better.
+
+0:42:52.411 --> 0:42:59.107
+The disadvantage is, does anybody have an
+idea of what might be the disadvantage of using
+
+0:42:59.107 --> 0:43:00.074
+things like.
+
+0:43:04.624 --> 0:43:12.175
+What was one mentioned today giving like big
+advantage of the system compared to previous.
+
+0:43:20.660 --> 0:43:25.134
+Where one advantage was the enter end training,
+so you have the enter end training so that
+
+0:43:25.134 --> 0:43:27.937
+all parameters and all components play optimal
+together.
+
+0:43:28.208 --> 0:43:33.076
+If you know pre-train something on one fast,
+it may be no longer optimal fitting to everything
+
+0:43:33.076 --> 0:43:33.384
+else.
+
+0:43:33.893 --> 0:43:37.862
+So what do pretending or not?
+
+0:43:37.862 --> 0:43:48.180
+It depends on how important everything is
+optimal together and how important.
+
+0:43:48.388 --> 0:43:51.874
+Is a iquality of large amount.
+
+0:43:51.874 --> 0:44:00.532
+The pre-change one is so much better that
+it's helpful and the advantage of.
+
+0:44:00.600 --> 0:44:11.211
+Getting everything optimal together, yes,
+we would use random instructions for raising.
+
+0:44:11.691 --> 0:44:26.437
+The problem is you might be already in some
+area where it's not easy to get.
+
+0:44:26.766 --> 0:44:35.329
+But often in some way right, so often it's
+not about your really worse pre trained monolepsy.
+
+0:44:35.329 --> 0:44:43.254
+If you're going already in some direction,
+and if this is not really optimal for you,.
+
+0:44:43.603 --> 0:44:52.450
+But if you're not really getting better because
+you have a decent amount of data, it's so different
+
+0:44:52.450 --> 0:44:52.981
+that.
+
+0:44:53.153 --> 0:44:59.505
+Initially it wasn't a machine translation
+done so much because there are more data in
+
+0:44:59.505 --> 0:45:06.153
+MPs than in other tasks, but now with really
+large amounts of monolingual data we do some
+
+0:45:06.153 --> 0:45:09.403
+type of pretraining in currently all state.
+
+0:45:12.632 --> 0:45:14.302
+The other one is okay now.
+
+0:45:14.302 --> 0:45:18.260
+It's always like how much of the model do
+you plea track a bit?
+
+0:45:18.658 --> 0:45:22.386
+To the other one you can do contextural word
+embedded.
+
+0:45:22.386 --> 0:45:28.351
+That is something like bird or Roberta where
+you train already a sequence model and the
+
+0:45:28.351 --> 0:45:34.654
+embeddings you're using are no longer specific
+for word but they are also taking the context
+
+0:45:34.654 --> 0:45:35.603
+into account.
+
+0:45:35.875 --> 0:45:50.088
+The embedding you're using is no longer depending
+on the word itself but on the whole sentence,
+
+0:45:50.088 --> 0:45:54.382
+so you can use this context.
+
+0:45:55.415 --> 0:46:02.691
+You can use similar things also in the decoder
+just by having layers which don't have access
+
+0:46:02.691 --> 0:46:12.430
+to the source, but there it still might have
+and these are typically models like: And finally
+
+0:46:12.430 --> 0:46:14.634
+they will look at the end.
+
+0:46:14.634 --> 0:46:19.040
+You can also have models which are already
+sequenced.
+
+0:46:19.419 --> 0:46:28.561
+So you may be training a sequence to sequence
+models.
+
+0:46:28.561 --> 0:46:35.164
+You have to make it a bit challenging.
+
+0:46:36.156 --> 0:46:43.445
+But the idea is really you're pre-training
+your whole model and then you'll find tuning.
+
+0:46:47.227 --> 0:46:59.614
+But let's first do a bit of step back and
+look into what are the different things.
+
+0:46:59.614 --> 0:47:02.151
+The first thing.
+
+0:47:02.382 --> 0:47:11.063
+The wooden bettings are just this first layer
+and you can train them with feedback annual
+
+0:47:11.063 --> 0:47:12.028
+networks.
+
+0:47:12.212 --> 0:47:22.761
+But you can also train them with an N language
+model, and by now you hopefully have also seen
+
+0:47:22.761 --> 0:47:27.699
+that you cannot transform a language model.
+
+0:47:30.130 --> 0:47:37.875
+So this is how you can train them and you're
+training them.
+
+0:47:37.875 --> 0:47:45.234
+For example, to speak the next word that is
+the easiest.
+
+0:47:45.525 --> 0:47:55.234
+And that is what is now referred to as South
+Supervised Learning and, for example, all the
+
+0:47:55.234 --> 0:48:00.675
+big large language models like Chad GPT and
+so on.
+
+0:48:00.675 --> 0:48:03.129
+They are trained with.
+
+0:48:03.823 --> 0:48:15.812
+So that is where you can hopefully learn how
+a word is used because you always try to previct
+
+0:48:15.812 --> 0:48:17.725
+the next word.
+
+0:48:19.619 --> 0:48:27.281
+Word embedding: Why do you keep the first
+look at the word embeddings and the use of
+
+0:48:27.281 --> 0:48:29.985
+word embeddings for our task?
+
+0:48:29.985 --> 0:48:38.007
+The main advantage was it might be only the
+first layer where you typically have most of
+
+0:48:38.007 --> 0:48:39.449
+the parameters.
+
+0:48:39.879 --> 0:48:57.017
+Most of your parameters already on the large
+data, then on your target data you have to
+
+0:48:57.017 --> 0:48:59.353
+train less.
+
+0:48:59.259 --> 0:49:06.527
+Big difference that your input size is so
+much bigger than the size of the novel in size.
+
+0:49:06.626 --> 0:49:17.709
+So it's a normally sign, maybe like, but your
+input and banning size is something like.
+
+0:49:17.709 --> 0:49:20.606
+Then here you have to.
+
+0:49:23.123 --> 0:49:30.160
+While here you see it's only like zero point
+five times as much in the layer.
+
+0:49:30.750 --> 0:49:40.367
+So here is where most of your parameters are,
+which means if you already replace the word
+
+0:49:40.367 --> 0:49:48.915
+embeddings they might look a bit small in your
+overall and in key architecture.
+
+0:49:57.637 --> 0:50:01.249
+The thing is we have seen these were the bettings.
+
+0:50:01.249 --> 0:50:04.295
+They can be very good use for other types.
+
+0:50:04.784 --> 0:50:08.994
+You learn some general relations between words.
+
+0:50:08.994 --> 0:50:17.454
+If you're doing this type of language modeling
+cast, you predict: The one thing is you have
+
+0:50:17.454 --> 0:50:24.084
+a lot of data, so the one question is we want
+to have data to trade a model.
+
+0:50:24.084 --> 0:50:28.734
+The other thing, the tasks need to be somehow
+useful.
+
+0:50:29.169 --> 0:50:43.547
+If you would predict the first letter of the
+word, then you wouldn't learn anything about
+
+0:50:43.547 --> 0:50:45.144
+the word.
+
+0:50:45.545 --> 0:50:53.683
+And the interesting thing is people have looked
+at these wood embeddings.
+
+0:50:53.954 --> 0:50:58.550
+And looking at the word embeddings.
+
+0:50:58.550 --> 0:51:09.276
+You can ask yourself how they look and visualize
+them by doing dimension reduction.
+
+0:51:09.489 --> 0:51:13.236
+Don't know if you and you are listening to
+artificial intelligence.
+
+0:51:13.236 --> 0:51:15.110
+Advanced artificial intelligence.
+
+0:51:15.515 --> 0:51:23.217
+We had on yesterday there how to do this type
+of representation, but you can do this time
+
+0:51:23.217 --> 0:51:29.635
+of representation, and now you're seeing interesting
+things that normally.
+
+0:51:30.810 --> 0:51:41.027
+Now you can represent a here in a three dimensional
+space with some dimension reduction.
+
+0:51:41.027 --> 0:51:46.881
+For example, the relation between male and
+female.
+
+0:51:47.447 --> 0:51:56.625
+So this vector between the male and female
+version of something is always not the same,
+
+0:51:56.625 --> 0:51:58.502
+but it's related.
+
+0:51:58.718 --> 0:52:14.522
+So you can do a bit of maths, so you do take
+king, you subtract this vector, add this vector.
+
+0:52:14.894 --> 0:52:17.591
+So that means okay, there is really something
+stored.
+
+0:52:17.591 --> 0:52:19.689
+Some information are stored in that book.
+
+0:52:20.040 --> 0:52:22.492
+Similar, you can do it with bug answers.
+
+0:52:22.492 --> 0:52:25.004
+You see here swimming slang walking walk.
+
+0:52:25.265 --> 0:52:34.620
+So again these vectors are not the same, but
+they are related.
+
+0:52:34.620 --> 0:52:42.490
+So you learn something from going from here
+to here.
+
+0:52:43.623 --> 0:52:49.761
+Or semantically, the relations between city
+and capital have exactly the same sense.
+
+0:52:51.191 --> 0:52:56.854
+And people had even done that question answering
+about that if they showed the diembeddings
+
+0:52:56.854 --> 0:52:57.839
+and the end of.
+
+0:52:58.218 --> 0:53:06.711
+All you can also do is don't trust the dimensions
+of the reaction because maybe there is something.
+
+0:53:06.967 --> 0:53:16.863
+You can also look into what happens really
+in the individual space.
+
+0:53:16.863 --> 0:53:22.247
+What is the nearest neighbor of the.
+
+0:53:22.482 --> 0:53:29.608
+So you can take the relationship between France
+and Paris and add it to Italy and you'll.
+
+0:53:30.010 --> 0:53:33.078
+You can do big and bigger and you have small
+and smaller and stuff.
+
+0:53:33.593 --> 0:53:49.417
+Because it doesn't work everywhere, there
+is also some typical dish here in German.
+
+0:53:51.491 --> 0:54:01.677
+You can do what the person is doing for famous
+ones, of course only like Einstein scientists
+
+0:54:01.677 --> 0:54:06.716
+that find midfielders not completely correct.
+
+0:54:06.846 --> 0:54:10.134
+You see the examples are a bit old.
+
+0:54:10.134 --> 0:54:15.066
+The politicians are no longer they am, but
+of course.
+
+0:54:16.957 --> 0:54:26.759
+What people have done there, especially at
+the beginning training our end language model,
+
+0:54:26.759 --> 0:54:28.937
+was very expensive.
+
+0:54:29.309 --> 0:54:38.031
+So one famous model was, but we are not really
+interested in the language model performance.
+
+0:54:38.338 --> 0:54:40.581
+Think something good to keep in mind.
+
+0:54:40.581 --> 0:54:42.587
+What are we really interested in?
+
+0:54:42.587 --> 0:54:45.007
+Do we really want to have an R&N no?
+
+0:54:45.007 --> 0:54:48.607
+In this case we are only interested in this
+type of mapping.
+
+0:54:49.169 --> 0:54:55.500
+And so successful and very successful was
+this word to vet.
+
+0:54:55.535 --> 0:54:56.865
+The idea is okay.
+
+0:54:56.865 --> 0:55:03.592
+We are not training real language one, making
+it even simpler and doing this, for example,
+
+0:55:03.592 --> 0:55:05.513
+continuous peck of words.
+
+0:55:05.513 --> 0:55:12.313
+We're just having four input tokens and we're
+predicting what is the word in the middle and
+
+0:55:12.313 --> 0:55:15.048
+this is just like two linear layers.
+
+0:55:15.615 --> 0:55:21.627
+So it's even simplifying things and making
+the calculation faster because that is what
+
+0:55:21.627 --> 0:55:22.871
+we're interested.
+
+0:55:23.263 --> 0:55:32.897
+All this continuous skip ground models with
+these other models which refer to as where
+
+0:55:32.897 --> 0:55:34.004
+to where.
+
+0:55:34.234 --> 0:55:42.394
+Where you have one equal word and the other
+way around, you're predicting the four words
+
+0:55:42.394 --> 0:55:43.585
+around them.
+
+0:55:43.585 --> 0:55:45.327
+It's very similar.
+
+0:55:45.327 --> 0:55:48.720
+The task is in the end very similar.
+
+0:55:51.131 --> 0:56:01.407
+Before we are going to the next point, anything
+about normal weight vectors or weight embedding.
+
+0:56:04.564 --> 0:56:07.794
+The next thing is contexture.
+
+0:56:07.794 --> 0:56:12.208
+Word embeddings and the idea is helpful.
+
+0:56:12.208 --> 0:56:19.206
+However, we might even be able to get more
+from one lingo layer.
+
+0:56:19.419 --> 0:56:31.732
+And now in the word that is overlap of these
+two meanings, so it represents both the meaning
+
+0:56:31.732 --> 0:56:33.585
+of can do it.
+
+0:56:34.834 --> 0:56:40.410
+But we might be able to in the pre-trained
+model already disambiguate this because they
+
+0:56:40.410 --> 0:56:41.044
+are used.
+
+0:56:41.701 --> 0:56:53.331
+So if we can have a model which can not only
+represent a word but can also represent the
+
+0:56:53.331 --> 0:56:58.689
+meaning of the word within the context,.
+
+0:56:59.139 --> 0:57:03.769
+So then we are going to context your word
+embeddings.
+
+0:57:03.769 --> 0:57:07.713
+We are really having a representation in the.
+
+0:57:07.787 --> 0:57:11.519
+And we have a very good architecture for that
+already.
+
+0:57:11.691 --> 0:57:23.791
+The hidden state represents what is currently
+said, but it's focusing on what is the last
+
+0:57:23.791 --> 0:57:29.303
+one, so it's some of the representation.
+
+0:57:29.509 --> 0:57:43.758
+The first one doing that is something like
+the Elmo paper where they instead of this is
+
+0:57:43.758 --> 0:57:48.129
+the normal language model.
+
+0:57:48.008 --> 0:57:50.714
+Within the third, predicting the fourth, and
+so on.
+
+0:57:50.714 --> 0:57:53.004
+So you are always predicting the next work.
+
+0:57:53.193 --> 0:57:57.335
+The architecture is the heaven words embedding
+layer and then layers.
+
+0:57:57.335 --> 0:58:03.901
+See you, for example: And now instead of using
+this one in the end, you're using here this
+
+0:58:03.901 --> 0:58:04.254
+one.
+
+0:58:04.364 --> 0:58:11.245
+This represents the meaning of this word mainly
+in the context of what we have seen before.
+
+0:58:11.871 --> 0:58:18.610
+We can train it in a language model style
+always predicting the next word, but we have
+
+0:58:18.610 --> 0:58:21.088
+more information trained there.
+
+0:58:21.088 --> 0:58:26.123
+Therefore, in the system it has to learn less
+additional things.
+
+0:58:27.167 --> 0:58:31.261
+And there is one Edendang which is done currently
+in GPS.
+
+0:58:31.261 --> 0:58:38.319
+The only difference is that we have more layers,
+bigger size, and we're using transformer neurocell
+
+0:58:38.319 --> 0:58:40.437
+potential instead of the RNA.
+
+0:58:40.437 --> 0:58:45.095
+But that is how you train like some large
+language models at the.
+
+0:58:46.746 --> 0:58:55.044
+However, if you look at this contextual representation,
+they might not be perfect.
+
+0:58:55.044 --> 0:59:02.942
+So if you think of this one as a contextual
+representation of the third word,.
+
+0:59:07.587 --> 0:59:16.686
+Is representing a three in the context of
+a sentence, however only in the context of
+
+0:59:16.686 --> 0:59:18.185
+the previous.
+
+0:59:18.558 --> 0:59:27.413
+However, we have an architecture which can
+also take both sides and we have used that
+
+0:59:27.413 --> 0:59:30.193
+already in the ink holder.
+
+0:59:30.630 --> 0:59:34.264
+So we could do the iron easily on your, also
+in the backward direction.
+
+0:59:34.874 --> 0:59:42.826
+By just having the states the other way around
+and then we couldn't combine the forward and
+
+0:59:42.826 --> 0:59:49.135
+the forward into a joint one where we are doing
+this type of prediction.
+
+0:59:49.329 --> 0:59:50.858
+So you have the word embedding.
+
+0:59:51.011 --> 1:00:02.095
+Then you have two in the states, one on the
+forward arm and one on the backward arm, and
+
+1:00:02.095 --> 1:00:10.314
+then you can, for example, take the cocagenation
+of both of them.
+
+1:00:10.490 --> 1:00:23.257
+Now this same here represents mainly this
+word because this is what both puts in it last
+
+1:00:23.257 --> 1:00:30.573
+and we know is focusing on what is happening
+last.
+
+1:00:31.731 --> 1:00:40.469
+However, there is a bit of difference when
+training that as a language model you already
+
+1:00:40.469 --> 1:00:41.059
+have.
+
+1:00:43.203 --> 1:00:44.956
+Maybe There's Again This Masking.
+
+1:00:46.546 --> 1:00:47.748
+That is one solution.
+
+1:00:47.748 --> 1:00:52.995
+First of all, why we can't do it is the information
+you leak it, so you cannot just predict the
+
+1:00:52.995 --> 1:00:53.596
+next word.
+
+1:00:53.596 --> 1:00:58.132
+If we just predict the next word in this type
+of model, that's a very simple task.
+
+1:00:58.738 --> 1:01:09.581
+You know the next word because it's influencing
+this hidden state predicting something is not
+
+1:01:09.581 --> 1:01:11.081
+a good task.
+
+1:01:11.081 --> 1:01:18.455
+You have to define: Because in this case what
+will end with the system will just ignore these
+
+1:01:18.455 --> 1:01:22.966
+estates and what will learn is copy this information
+directly in here.
+
+1:01:23.343 --> 1:01:31.218
+So it would be representing this word and
+you would have nearly a perfect model because
+
+1:01:31.218 --> 1:01:38.287
+you only need to find encoding where you can
+encode all words somehow in this.
+
+1:01:38.458 --> 1:01:44.050
+The only thing can learn is that turn and
+encode all my words in this upper hidden.
+
+1:01:44.985 --> 1:01:53.779
+Therefore, it's not really useful, so we need
+to find a bit of different ways out.
+
+1:01:55.295 --> 1:01:57.090
+There is a masking one.
+
+1:01:57.090 --> 1:02:03.747
+I'll come to that shortly just a bit that
+other things also have been done, so the other
+
+1:02:03.747 --> 1:02:06.664
+thing is not to directly combine them.
+
+1:02:06.664 --> 1:02:13.546
+That was in the animal paper, so you have
+them forward R&M and you keep them completely
+
+1:02:13.546 --> 1:02:14.369
+separated.
+
+1:02:14.594 --> 1:02:20.458
+So you never merged to state.
+
+1:02:20.458 --> 1:02:33.749
+At the end, the representation of the word
+is now from the forward.
+
+1:02:33.873 --> 1:02:35.953
+So it's always the hidden state before the
+good thing.
+
+1:02:36.696 --> 1:02:41.286
+These two you join now to your to the representation.
+
+1:02:42.022 --> 1:02:48.685
+And then you have now a representation also
+about like the whole sentence for the word,
+
+1:02:48.685 --> 1:02:51.486
+but there is no information leakage.
+
+1:02:51.486 --> 1:02:58.149
+One way of doing this is instead of doing
+a bidirection along you do a forward pass and
+
+1:02:58.149 --> 1:02:59.815
+then join the hidden.
+
+1:03:00.380 --> 1:03:05.960
+So you can do that in all layers.
+
+1:03:05.960 --> 1:03:16.300
+In the end you do the forwarded layers and
+you get the hidden.
+
+1:03:16.596 --> 1:03:19.845
+However, it's a bit of a complicated.
+
+1:03:19.845 --> 1:03:25.230
+You have to keep both separate and merge things
+so can you do.
+
+1:03:27.968 --> 1:03:33.030
+And that is the moment where like the big.
+
+1:03:34.894 --> 1:03:39.970
+The big success of the burnt model was used
+where it okay.
+
+1:03:39.970 --> 1:03:47.281
+Maybe in bite and rich case it's not good
+to do the next word prediction, but we can
+
+1:03:47.281 --> 1:03:48.314
+do masking.
+
+1:03:48.308 --> 1:03:56.019
+Masking mainly means we do a prediction of
+something in the middle or some words.
+
+1:03:56.019 --> 1:04:04.388
+So the idea is if we have the input, we are
+putting noise into the input, removing them,
+
+1:04:04.388 --> 1:04:07.961
+and then the model we are interested.
+
+1:04:08.048 --> 1:04:15.327
+Now there can be no information leakage because
+this wasn't predicting that one is a big challenge.
+
+1:04:16.776 --> 1:04:19.957
+Do any assumption about our model?
+
+1:04:19.957 --> 1:04:26.410
+It doesn't need to be a forward model or a
+backward model or anything.
+
+1:04:26.410 --> 1:04:29.500
+You can always predict the three.
+
+1:04:30.530 --> 1:04:34.844
+There's maybe one bit of a disadvantage.
+
+1:04:34.844 --> 1:04:40.105
+Do you see what could be a bit of a problem
+this?
+
+1:05:00.000 --> 1:05:06.429
+Yes, so yeah, you can of course mask more,
+but to see it more globally, just first assume
+
+1:05:06.429 --> 1:05:08.143
+you're only masked one.
+
+1:05:08.143 --> 1:05:13.930
+For the whole sentence, we get one feedback
+signal, like what is the word three.
+
+1:05:13.930 --> 1:05:22.882
+So we have one training example: If you do
+the language modeling taste, we predicted here,
+
+1:05:22.882 --> 1:05:24.679
+we predicted here.
+
+1:05:25.005 --> 1:05:26.735
+So we have number of tokens.
+
+1:05:26.735 --> 1:05:30.970
+For each token we have a feet pad and say
+what is the best correction.
+
+1:05:31.211 --> 1:05:43.300
+So in this case this is less efficient because
+we are getting less feedback signals on what
+
+1:05:43.300 --> 1:05:45.797
+we should predict.
+
+1:05:48.348 --> 1:05:56.373
+So and bird, the main ideas are that you're
+doing this bidirectional model with masking.
+
+1:05:56.373 --> 1:05:59.709
+It's using transformer architecture.
+
+1:06:00.320 --> 1:06:06.326
+There are two more minor changes.
+
+1:06:06.326 --> 1:06:16.573
+We'll see that this next word prediction is
+another task.
+
+1:06:16.957 --> 1:06:30.394
+You want to learn more about what language
+is to really understand following a story or
+
+1:06:30.394 --> 1:06:35.127
+their independent tokens into.
+
+1:06:38.158 --> 1:06:42.723
+The input is using word units as we use it.
+
+1:06:42.723 --> 1:06:50.193
+It has some special token that is framing
+for the next word prediction.
+
+1:06:50.470 --> 1:07:04.075
+It's more for classification task because
+you may be learning a general representation
+
+1:07:04.075 --> 1:07:07.203
+as a full sentence.
+
+1:07:07.607 --> 1:07:19.290
+You're doing segment embedding, so you have
+an embedding for it.
+
+1:07:19.290 --> 1:07:24.323
+This is the first sentence.
+
+1:07:24.684 --> 1:07:29.099
+Now what is more challenging is this masking.
+
+1:07:29.099 --> 1:07:30.827
+What do you mask?
+
+1:07:30.827 --> 1:07:35.050
+We already have the crush enough or should.
+
+1:07:35.275 --> 1:07:42.836
+So there has been afterwards eating some work
+like, for example, a bearer.
+
+1:07:42.836 --> 1:07:52.313
+It's not super sensitive, but if you do it
+completely wrong then you're not letting anything.
+
+1:07:52.572 --> 1:07:54.590
+That's Then Another Question There.
+
+1:07:56.756 --> 1:08:04.594
+Should I mask all types of should I always
+mask the footwork or if I have a subword to
+
+1:08:04.594 --> 1:08:10.630
+mask only like a subword and predict them based
+on the other ones?
+
+1:08:10.630 --> 1:08:14.504
+Of course, it's a bit of a different task.
+
+1:08:14.894 --> 1:08:21.210
+If you know three parts of the words, it might
+be easier to guess the last because they here
+
+1:08:21.210 --> 1:08:27.594
+took the easiest selection, so not considering
+words anymore at all because you're doing that
+
+1:08:27.594 --> 1:08:32.280
+in the preprocessing and just taking always
+words and like subwords.
+
+1:08:32.672 --> 1:08:36.089
+Think in group there is done differently.
+
+1:08:36.089 --> 1:08:40.401
+They mark always the full words, but guess
+it's not.
+
+1:08:41.001 --> 1:08:46.044
+And then what to do with the mask word in
+eighty percent of the cases.
+
+1:08:46.044 --> 1:08:50.803
+If the word is masked, they replace it with
+a special token thing.
+
+1:08:50.803 --> 1:08:57.197
+This is a mask token in ten percent they put
+in some random other token in there, and ten
+
+1:08:57.197 --> 1:08:59.470
+percent they keep it on change.
+
+1:09:02.202 --> 1:09:10.846
+And then what you can do is also this next
+word prediction.
+
+1:09:10.846 --> 1:09:14.880
+The man went to Mass Store.
+
+1:09:14.880 --> 1:09:17.761
+He bought a gallon.
+
+1:09:18.418 --> 1:09:24.088
+So may you see you're joining them, you're
+doing both masks and prediction that you're.
+
+1:09:24.564 --> 1:09:29.449
+Is a penguin mask or flyless birds.
+
+1:09:29.449 --> 1:09:41.390
+These two sentences have nothing to do with
+each other, so you can do also this type of
+
+1:09:41.390 --> 1:09:43.018
+prediction.
+
+1:09:47.127 --> 1:09:56.572
+And then the whole bird model, so here you
+have the in-foot to transform the layers, and
+
+1:09:56.572 --> 1:09:58.164
+you can train.
+
+1:09:58.598 --> 1:10:17.731
+And this model was quite successful in general
+applications.
+
+1:10:17.937 --> 1:10:27.644
+However, there is like a huge thing of different
+types of models coming from them.
+
+1:10:27.827 --> 1:10:38.709
+So based on others these supervised molds
+like a whole setup came out of there and now
+
+1:10:38.709 --> 1:10:42.086
+this is getting even more.
+
+1:10:42.082 --> 1:10:46.640
+With availability of a large language model
+than the success.
+
+1:10:47.007 --> 1:10:48.436
+We have now even larger ones.
+
+1:10:48.828 --> 1:10:50.961
+Interestingly, it goes a bit.
+
+1:10:50.910 --> 1:10:57.847
+Change the bit again from like more the spider
+action model to uni directional models.
+
+1:10:57.847 --> 1:11:02.710
+Are at the moment maybe a bit more we're coming
+to them now?
+
+1:11:02.710 --> 1:11:09.168
+Do you see one advantage while what is another
+event and we have the efficiency?
+
+1:11:09.509 --> 1:11:15.901
+Is one other reason why you are sometimes
+more interested in uni-direction models than
+
+1:11:15.901 --> 1:11:17.150
+in bi-direction.
+
+1:11:22.882 --> 1:11:30.220
+It depends on the pass, but for example for
+a language generation pass, the eccard is not
+
+1:11:30.220 --> 1:11:30.872
+really.
+
+1:11:32.192 --> 1:11:40.924
+It doesn't work so if you want to do a generation
+like the decoder you don't know the future
+
+1:11:40.924 --> 1:11:42.896
+so you cannot apply.
+
+1:11:43.223 --> 1:11:53.870
+So this time of model can be used for the
+encoder in an encoder model, but it cannot
+
+1:11:53.870 --> 1:11:57.002
+be used for the decoder.
+
+1:12:00.000 --> 1:12:05.012
+That's a good view to the next overall cast
+of models.
+
+1:12:05.012 --> 1:12:08.839
+Perhaps if you view it from the sequence.
+
+1:12:09.009 --> 1:12:12.761
+We have the encoder base model.
+
+1:12:12.761 --> 1:12:16.161
+That's what we just look at.
+
+1:12:16.161 --> 1:12:20.617
+They are bidirectional and typically.
+
+1:12:20.981 --> 1:12:22.347
+That Is the One We Looked At.
+
+1:12:22.742 --> 1:12:34.634
+At the beginning is the decoder based model,
+so see out in regressive models which are unidirective
+
+1:12:34.634 --> 1:12:42.601
+like an based model, and there we can do the
+next word prediction.
+
+1:12:43.403 --> 1:12:52.439
+And what you can also do first, and there
+you can also have a special things called prefix
+
+1:12:52.439 --> 1:12:53.432
+language.
+
+1:12:54.354 --> 1:13:05.039
+Because we are saying it might be helpful
+that some of your input can also use bi-direction.
+
+1:13:05.285 --> 1:13:12.240
+And that is somehow doing what it is called
+prefix length.
+
+1:13:12.240 --> 1:13:19.076
+On the first tokens you directly give your
+bidirectional.
+
+1:13:19.219 --> 1:13:28.774
+So you somehow merge that and that mainly
+works only in transformer based models because.
+
+1:13:29.629 --> 1:13:33.039
+There is no different number of parameters
+in our end.
+
+1:13:33.039 --> 1:13:34.836
+We need a back foot our end.
+
+1:13:34.975 --> 1:13:38.533
+Transformer: The only difference is how you
+mask your attention.
+
+1:13:38.878 --> 1:13:44.918
+We have seen that in the anchoder and decoder
+the number of parameters is different because
+
+1:13:44.918 --> 1:13:50.235
+you do cross attention, but if you do forward
+and backward or union directions,.
+
+1:13:50.650 --> 1:13:58.419
+It's only like that you mask your attention
+to only look at the bad past or to look into
+
+1:13:58.419 --> 1:13:59.466
+the future.
+
+1:14:00.680 --> 1:14:03.326
+And now you can of course also do mixing.
+
+1:14:03.563 --> 1:14:08.306
+So this is a bi-directional attention matrix
+where you can attend to everything.
+
+1:14:08.588 --> 1:14:23.516
+There is a uni-direction or causal where you
+can look at the past and you can do the first
+
+1:14:23.516 --> 1:14:25.649
+three words.
+
+1:14:29.149 --> 1:14:42.831
+That somehow clear based on that, then of
+course you cannot do the other things.
+
+1:14:43.163 --> 1:14:50.623
+So the idea is we have our anchor to decoder
+architecture.
+
+1:14:50.623 --> 1:14:57.704
+Can we also train them completely in a side
+supervisor?
+
+1:14:58.238 --> 1:15:09.980
+And in this case we have the same input to
+both, so in this case we need to do some type
+
+1:15:09.980 --> 1:15:12.224
+of masking here.
+
+1:15:12.912 --> 1:15:17.591
+Here we don't need to do the masking, but
+here we need to the masking that doesn't know
+
+1:15:17.591 --> 1:15:17.910
+ever.
+
+1:15:20.440 --> 1:15:30.269
+And this type of model got quite successful
+also, especially for pre-training machine translation.
+
+1:15:30.330 --> 1:15:39.059
+The first model doing that is a Bart model,
+which exactly does that, and yes, it's one
+
+1:15:39.059 --> 1:15:42.872
+successful way to pre train your one.
+
+1:15:42.872 --> 1:15:47.087
+It's pretraining your full encoder model.
+
+1:15:47.427 --> 1:15:54.365
+Where you put in contrast to machine translation,
+where you put in source sentence, we can't
+
+1:15:54.365 --> 1:15:55.409
+do that here.
+
+1:15:55.715 --> 1:16:01.382
+But we can just put the second twice in there,
+and then it's not a trivial task.
+
+1:16:01.382 --> 1:16:02.432
+We can change.
+
+1:16:03.003 --> 1:16:12.777
+And there is like they do different corruption
+techniques so you can also do.
+
+1:16:13.233 --> 1:16:19.692
+That you couldn't do in an agricultural system
+because then it wouldn't be there and you cannot
+
+1:16:19.692 --> 1:16:20.970
+predict somewhere.
+
+1:16:20.970 --> 1:16:26.353
+So the anchor, the number of input and output
+tokens always has to be the same.
+
+1:16:26.906 --> 1:16:29.818
+You cannot do a prediction for something which
+isn't in it.
+
+1:16:30.110 --> 1:16:38.268
+Here in the decoder side it's unidirection
+so we can also delete the top and then try
+
+1:16:38.268 --> 1:16:40.355
+to generate the full.
+
+1:16:41.061 --> 1:16:45.250
+We can do sentence permutation.
+
+1:16:45.250 --> 1:16:54.285
+We can document rotation and text infilling
+so there is quite a bit.
+
+1:16:55.615 --> 1:17:06.568
+So you see there's quite a lot of types of
+models that you can use in order to pre-train.
+
+1:17:07.507 --> 1:17:14.985
+Then, of course, there is again for the language
+one.
+
+1:17:14.985 --> 1:17:21.079
+The other question is how do you integrate?
+
+1:17:21.761 --> 1:17:26.636
+And there's also, like yeah, quite some different
+ways of techniques.
+
+1:17:27.007 --> 1:17:28.684
+It's a Bit Similar to Before.
+
+1:17:28.928 --> 1:17:39.068
+So the easiest thing is you take your word
+embeddings or your free trained model.
+
+1:17:39.068 --> 1:17:47.971
+You freeze them and stack your decoder layers
+and keep these ones free.
+
+1:17:48.748 --> 1:17:54.495
+Can also be done if you have this type of
+bark model.
+
+1:17:54.495 --> 1:18:03.329
+What you can do is you freeze your word embeddings,
+for example some products and.
+
+1:18:05.865 --> 1:18:17.296
+The other thing is you initialize them so
+you initialize your models but you train everything
+
+1:18:17.296 --> 1:18:19.120
+so you're not.
+
+1:18:22.562 --> 1:18:29.986
+Then one thing, if you think about Bart, you
+want to have the Chinese language, the Italian
+
+1:18:29.986 --> 1:18:32.165
+language, and the deconer.
+
+1:18:32.165 --> 1:18:35.716
+However, in Bart we have the same language.
+
+1:18:36.516 --> 1:18:46.010
+The one you get is from English, so what you
+can do there is so you cannot try to do some.
+
+1:18:46.366 --> 1:18:52.562
+Below the barge, in order to learn some language
+specific stuff, or there's a masculine barge,
+
+1:18:52.562 --> 1:18:58.823
+which is trained on many languages, but it's
+trained only on like the Old Coast Modern Language
+
+1:18:58.823 --> 1:19:03.388
+House, which may be trained in German and English,
+but not on German.
+
+1:19:03.923 --> 1:19:08.779
+So then you would still need to find June
+and the model needs to learn how to better
+
+1:19:08.779 --> 1:19:10.721
+do the attention cross lingually.
+
+1:19:10.721 --> 1:19:15.748
+It's only on the same language but it mainly
+only has to learn this mapping and not all
+
+1:19:15.748 --> 1:19:18.775
+the rest and that's why it's still quite successful.
+
+1:19:21.982 --> 1:19:27.492
+Now certain thing which is very commonly used
+is what is required to it as adapters.
+
+1:19:27.607 --> 1:19:29.754
+So for example you take and buy.
+
+1:19:29.709 --> 1:19:35.218
+And you put some adapters on the inside of
+the networks so that it's small new layers
+
+1:19:35.218 --> 1:19:40.790
+which are in between put in there and then
+you only train these adapters or also train
+
+1:19:40.790 --> 1:19:41.815
+these adapters.
+
+1:19:41.815 --> 1:19:47.900
+For example, an embryo you could see that
+this learns to map the Sears language representation
+
+1:19:47.900 --> 1:19:50.334
+to the Tiger language representation.
+
+1:19:50.470 --> 1:19:52.395
+And then you don't have to change that luck.
+
+1:19:52.792 --> 1:19:59.793
+You give it extra ability to really perform
+well on that.
+
+1:19:59.793 --> 1:20:05.225
+These are quite small and so very efficient.
+
+1:20:05.905 --> 1:20:12.632
+That is also very commonly used, for example
+in modular systems where you have some adaptors
+
+1:20:12.632 --> 1:20:16.248
+in between here which might be language specific.
+
+1:20:16.916 --> 1:20:22.247
+So they are trained only for one language.
+
+1:20:22.247 --> 1:20:33.777
+The model has some or both and once has the
+ability to do multilingually to share knowledge.
+
+1:20:34.914 --> 1:20:39.058
+But there's one chance in general in the multilingual
+systems.
+
+1:20:39.058 --> 1:20:40.439
+It works quite well.
+
+1:20:40.439 --> 1:20:46.161
+There's one case or one specific use case
+for multilingual where this normally doesn't
+
+1:20:46.161 --> 1:20:47.344
+really work well.
+
+1:20:47.344 --> 1:20:49.975
+Do you have an idea what that could be?
+
+1:20:55.996 --> 1:20:57.536
+It's for Zero Shot Cases.
+
+1:20:57.998 --> 1:21:03.660
+Because having here some situation with this
+might be very language specific and zero shot,
+
+1:21:03.660 --> 1:21:09.015
+the idea is always to learn representations
+view which are more language dependent and
+
+1:21:09.015 --> 1:21:10.184
+with the adaptors.
+
+1:21:10.184 --> 1:21:15.601
+Of course you get in representations again
+which are more language specific and then it
+
+1:21:15.601 --> 1:21:17.078
+doesn't work that well.
+
+1:21:20.260 --> 1:21:37.730
+And there is also the idea of doing more knowledge
+pistolation.
+
+1:21:39.179 --> 1:21:42.923
+And now the idea is okay.
+
+1:21:42.923 --> 1:21:54.157
+We are training it the same, but what we want
+to achieve is that the encoder.
+
+1:21:54.414 --> 1:22:03.095
+So you should learn faster by trying to make
+these states as similar as possible.
+
+1:22:03.095 --> 1:22:11.777
+So you compare the first-hit state of the
+pre-trained model and try to make them.
+
+1:22:12.192 --> 1:22:18.144
+For example, by using the out two norms, so
+by just making these two representations the
+
+1:22:18.144 --> 1:22:26.373
+same: The same vocabulary: Why does it need
+the same vocabulary with any idea?
+
+1:22:34.754 --> 1:22:46.137
+If you have different vocabulary, it's typical
+you also have different sequenced lengths here.
+
+1:22:46.137 --> 1:22:50.690
+The number of sequences is different.
+
+1:22:51.231 --> 1:22:58.888
+If you now have pipe stains and four states
+here, it's no longer straightforward which
+
+1:22:58.888 --> 1:23:01.089
+states compare to which.
+
+1:23:02.322 --> 1:23:05.246
+And that's just easier if you have like the
+same number.
+
+1:23:05.246 --> 1:23:08.940
+You can always compare the first to the first
+and second to the second.
+
+1:23:09.709 --> 1:23:16.836
+So therefore at least the very easy way of
+knowledge destination only works if you have.
+
+1:23:17.177 --> 1:23:30.030
+Course: You could do things like yeah, the
+average should be the same, but of course there's
+
+1:23:30.030 --> 1:23:33.071
+a less strong signal.
+
+1:23:34.314 --> 1:23:42.979
+But the advantage here is that you have a
+diameter training signal here on the handquarter
+
+1:23:42.979 --> 1:23:51.455
+so you can directly make some of the encoder
+already giving a good signal while normally
+
+1:23:51.455 --> 1:23:52.407
+an empty.
+
+1:23:56.936 --> 1:24:13.197
+Yes, think this is most things for today,
+so what you should keep in mind is remind me.
+
+1:24:13.393 --> 1:24:18.400
+The one is a back translation idea.
+
+1:24:18.400 --> 1:24:29.561
+If you have monolingual and use that, the
+other one is to: And mentally it is often helpful
+
+1:24:29.561 --> 1:24:33.614
+to combine them so you can even use both of
+that.
+
+1:24:33.853 --> 1:24:38.908
+So you can use pre-trained walls, but then
+you can even still do back translation where
+
+1:24:38.908 --> 1:24:40.057
+it's still helpful.
+
+1:24:40.160 --> 1:24:45.502
+We have the advantage we are training like
+everything working together on the task so
+
+1:24:45.502 --> 1:24:51.093
+it might be helpful even to backtranslate some
+data and then use it in a real translation
+
+1:24:51.093 --> 1:24:56.683
+setup because in pretraining of course the
+beach challenge is always that you're training
+
+1:24:56.683 --> 1:24:57.739
+it on different.
+
+1:24:58.058 --> 1:25:03.327
+Different ways of how you integrate this knowledge.
+
+1:25:03.327 --> 1:25:08.089
+Even if you just use a full model, so in this.
+
+1:25:08.748 --> 1:25:11.128
+This is the most similar you can get.
+
+1:25:11.128 --> 1:25:13.945
+You're doing no changes to the architecture.
+
+1:25:13.945 --> 1:25:19.643
+You're really taking the model and just fine
+tuning them on the new task, but it still has
+
+1:25:19.643 --> 1:25:24.026
+to completely newly learn how to do the attention
+and how to do that.
+
+1:25:24.464 --> 1:25:29.971
+And that might be, for example, helpful to
+have more back-translated data to learn them.
+
+1:25:32.192 --> 1:25:34.251
+That's for today.
+
+1:25:34.251 --> 1:25:44.661
+There's one important thing that next Tuesday
+there is a conference or a workshop or so in
+
+1:25:44.661 --> 1:25:45.920
+this room.
+
+1:25:47.127 --> 1:25:56.769
+You should get an e-mail if you're in Elias
+that there's a room change for Tuesdays and
+
+1:25:56.769 --> 1:25:57.426
+it's.
+
+1:25:57.637 --> 1:26:03.890
+There are more questions, yeah, have a more
+general position, especially: In computer vision
+
+1:26:03.890 --> 1:26:07.347
+you can enlarge your data center data orientation.
+
+1:26:07.347 --> 1:26:08.295
+Is there any?
+
+1:26:08.388 --> 1:26:15.301
+It's similar to a large speech for text for
+the data of an edge.
+
+1:26:15.755 --> 1:26:29.176
+And you can use this back translation and
+also masking, but back translation is some
+
+1:26:29.176 --> 1:26:31.228
+way of data.
+
+1:26:31.371 --> 1:26:35.629
+So it has also been, for example, even its
+used not only for monolingual data.
+
+1:26:36.216 --> 1:26:54.060
+If you have good MP system, it can also be
+used for parallel data.
+
+1:26:54.834 --> 1:26:59.139
+So would say this is the most similar one.
+
+1:26:59.139 --> 1:27:03.143
+There's ways you can do power phrasing.
+
+1:27:05.025 --> 1:27:12.057
+But for example there is very hard to do this
+by rules like which words to replace because
+
+1:27:12.057 --> 1:27:18.936
+there is not a coup like you cannot always
+say this word can always be replaced by that.
+
+1:27:19.139 --> 1:27:27.225
+Mean, although they are many perfect synonyms,
+normally they are good in some cases, but not
+
+1:27:27.225 --> 1:27:29.399
+in all cases, and so on.
+
+1:27:29.399 --> 1:27:36.963
+And if you don't do a rule based, you have
+to train your model and then the freshness.
+
+1:27:38.058 --> 1:27:57.236
+The same architecture as the pre-trained mount.
+
+1:27:57.457 --> 1:27:59.810
+Should be of the same dimension, so it's easiest
+to have the same dimension.
+
+1:28:00.000 --> 1:28:01.590
+Architecture.
+
+1:28:01.590 --> 1:28:05.452
+We later will learn inefficiency.
+
+1:28:05.452 --> 1:28:12.948
+You can also do knowledge cessulation with,
+for example, smaller.
+
+1:28:12.948 --> 1:28:16.469
+You can learn the same within.
+
+1:28:17.477 --> 1:28:22.949
+Eight layers for it so that is possible, but
+yeah agree it should be of the same.
+
+1:28:23.623 --> 1:28:32.486
+Yeah yeah you need the question then of course
+you can do it like it's an initialization or
+
+1:28:32.486 --> 1:28:41.157
+you can do it doing training but normally it
+most makes sense during the normal training.
+
+1:28:45.865 --> 1:28:53.963
+Do it, then thanks a lot, and then we'll see
+each other again on Tuesday.
+
+0:00:00.981 --> 0:00:20.036
+Today about is how to use some type of additional
+resources to improve the translation.
+
+0:00:20.300 --> 0:00:28.188
+We have in the first part of the semester
+two thirds of the semester how to build some
+
+0:00:28.188 --> 0:00:31.361
+of your basic machine translation.
+
+0:00:31.571 --> 0:00:42.317
+Now the basic components are both for statistical
+and for neural, with the encoded decoding.
+
+0:00:43.123 --> 0:00:46.000
+Now, of course, that's not where it stops.
+
+0:00:46.000 --> 0:00:51.286
+It's still what nearly every machine translation
+system is currently in there.
+
+0:00:51.286 --> 0:00:57.308
+However, there's a lot of challenges which
+you need to address in addition and which need
+
+0:00:57.308 --> 0:00:58.245
+to be solved.
+
+0:00:58.918 --> 0:01:09.858
+And there we want to start to tell you what
+else can you do around this, and partly.
+
+0:01:10.030 --> 0:01:14.396
+And one important question there is on what
+do you train your models?
+
+0:01:14.394 --> 0:01:32.003
+Because like this type of parallel data, it's
+easier in machine translation than in other
+
+0:01:32.003 --> 0:01:33.569
+trusts.
+
+0:01:33.853 --> 0:01:41.178
+And therefore an important question is, can
+we also learn from like other sources and through?
+
+0:01:41.701 --> 0:01:47.830
+Because if you remember strongly right at
+the beginning of the election,.
+
+0:01:51.171 --> 0:01:53.801
+This Is How We Train All Our.
+
+0:01:54.194 --> 0:01:59.887
+Machine learning models from statistical to
+neural.
+
+0:01:59.887 --> 0:02:09.412
+This doesn't have changed so we need this
+type of parallel data where we have a source
+
+0:02:09.412 --> 0:02:13.462
+sentence aligned with a target data.
+
+0:02:13.493 --> 0:02:19.135
+We have now a strong model here, a very good
+model to do that.
+
+0:02:19.135 --> 0:02:22.091
+However, we always rely on this.
+
+0:02:22.522 --> 0:02:28.395
+For languages, high risk language pairs say
+from German to English or other European languages,
+
+0:02:28.395 --> 0:02:31.332
+there is decent amount, at least for similarly.
+
+0:02:31.471 --> 0:02:37.630
+But even there if we are going to very specific
+domains it might get difficult and then your
+
+0:02:37.630 --> 0:02:43.525
+system performance might drop because if you
+want to translate now some medical text for
+
+0:02:43.525 --> 0:02:50.015
+example of course you need to also have peril
+data in the medical domain to know how to translate
+
+0:02:50.015 --> 0:02:50.876
+these types.
+
+0:02:51.231 --> 0:02:55.264
+Phrases how to use the vocabulary and so on
+in the style.
+
+0:02:55.915 --> 0:03:04.887
+And if you are going to other languages, there
+is a lot bigger challenge and the question
+
+0:03:04.887 --> 0:03:05.585
+there.
+
+0:03:05.825 --> 0:03:09.649
+So is really this the only resource we can
+use.
+
+0:03:09.889 --> 0:03:19.462
+Can be adapted or training phase in order
+to also make use of other types of models that
+
+0:03:19.462 --> 0:03:27.314
+might enable us to build strong systems with
+other types of information.
+
+0:03:27.707 --> 0:03:35.276
+And that we will look into now in the next
+starting from from just saying the next election.
+
+0:03:35.515 --> 0:03:40.697
+So this idea we already have covered on Tuesday.
+
+0:03:40.697 --> 0:03:45.350
+One very successful idea for this is to do.
+
+0:03:45.645 --> 0:03:51.990
+So that we're no longer doing translation
+between languages, but we can do translation
+
+0:03:51.990 --> 0:03:55.928
+between languages and share common knowledge
+between.
+
+0:03:56.296 --> 0:04:03.888
+You also learned about things like zero shots
+machine translation so you can translate between
+
+0:04:03.888 --> 0:04:06.446
+languages where you don't have.
+
+0:04:06.786 --> 0:04:09.790
+Which is the case for many, many language
+pairs.
+
+0:04:10.030 --> 0:04:19.209
+Like even with German, you have not translation
+parallel data to all languages around the world,
+
+0:04:19.209 --> 0:04:26.400
+or most of them you have it to the Europeans
+once, maybe even for Japanese.
+
+0:04:26.746 --> 0:04:35.332
+There is quite a lot of data, for example
+English to Japanese, but German to Japanese
+
+0:04:35.332 --> 0:04:37.827
+or German to Vietnamese.
+
+0:04:37.827 --> 0:04:41.621
+There is some data from Multilingual.
+
+0:04:42.042 --> 0:04:54.584
+So there is a very promising direction if
+you want to build translation systems between
+
+0:04:54.584 --> 0:05:00.142
+language peers, typically not English.
+
+0:05:01.221 --> 0:05:05.887
+And the other ideas, of course, we don't have
+to either just search for it.
+
+0:05:06.206 --> 0:05:12.505
+Some work on a data crawling so if I don't
+have a corpus directly or I don't have an high
+
+0:05:12.505 --> 0:05:19.014
+quality corpus like from the European Parliament
+for a TED corpus so maybe it makes sense to
+
+0:05:19.014 --> 0:05:23.913
+crawl more data and get additional sources
+so you can build stronger.
+
+0:05:24.344 --> 0:05:35.485
+There has been quite a big effort in Europe
+to collect really large data sets for parallel
+
+0:05:35.485 --> 0:05:36.220
+data.
+
+0:05:36.220 --> 0:05:40.382
+How can we do this data crawling?
+
+0:05:40.600 --> 0:05:46.103
+There the interesting thing from the machine
+translation point is not just general data
+
+0:05:46.103 --> 0:05:46.729
+crawling.
+
+0:05:47.067 --> 0:05:50.037
+But how can we explicitly crawl data?
+
+0:05:50.037 --> 0:05:52.070
+Which is some of a peril?
+
+0:05:52.132 --> 0:05:58.461
+So there is in the Internet quite a lot of
+data which has been company websites which
+
+0:05:58.461 --> 0:06:01.626
+have been translated and things like that.
+
+0:06:01.626 --> 0:06:05.158
+So how can you extract them parallel fragments?
+
+0:06:06.566 --> 0:06:13.404
+That is typically more noisy than where you
+do more at hands where mean if you have Parliament.
+
+0:06:13.693 --> 0:06:17.680
+You can do some rules how to extract parallel
+things.
+
+0:06:17.680 --> 0:06:24.176
+Here there is more to it, so the quality is
+later maybe not as good, but normally scale
+
+0:06:24.176 --> 0:06:26.908
+is then a possibility to address it.
+
+0:06:26.908 --> 0:06:30.304
+So you just have so much more data that even.
+
+0:06:33.313 --> 0:06:40.295
+The other thing can be used monolingual data
+and monolingual data has a big advantage that
+
+0:06:40.295 --> 0:06:46.664
+we can have a huge amount of that so that you
+can be autocrawed from the Internet.
+
+0:06:46.664 --> 0:06:51.728
+The nice thing is you can also get it typically
+for many domains.
+
+0:06:52.352 --> 0:06:59.558
+There is just so much more magnitude of monolingual
+data so that it might be very helpful.
+
+0:06:59.559 --> 0:07:03.054
+We can do that in statistical machine translation.
+
+0:07:03.054 --> 0:07:06.755
+It was quite easy to integrate using language
+models.
+
+0:07:08.508 --> 0:07:16.912
+In neural machine translation we have the
+advantage that we have this overall architecture
+
+0:07:16.912 --> 0:07:22.915
+that does everything together, but it has also
+the disadvantage.
+
+0:07:23.283 --> 0:07:25.675
+We'll look today at two things.
+
+0:07:25.675 --> 0:07:32.925
+On the one end you can still try to do a bit
+of language modeling in there and add an additional
+
+0:07:32.925 --> 0:07:35.168
+language model into in there.
+
+0:07:35.168 --> 0:07:38.232
+There is some work, one very successful.
+
+0:07:38.178 --> 0:07:43.764
+A way in which I think is used in most systems
+at the moment is to do some scientific data.
+
+0:07:43.763 --> 0:07:53.087
+Is a very easy thing, but you can just translate
+there and use it as training gator, and normally.
+
+0:07:53.213 --> 0:07:59.185
+And thereby you are able to use like some
+type of monolingual a day.
+
+0:08:00.380 --> 0:08:05.271
+Another way to do it is unsupervised and the
+extreme case.
+
+0:08:05.271 --> 0:08:11.158
+If you have a scenario then you only have
+data, only monolingual data.
+
+0:08:11.158 --> 0:08:13.976
+Can you still build translations?
+
+0:08:14.754 --> 0:08:27.675
+If you have large amounts of data and languages
+are not too dissimilar, you can build translation
+
+0:08:27.675 --> 0:08:31.102
+systems without parallel.
+
+0:08:32.512 --> 0:08:36.267
+That we will see you then next Thursday.
+
+0:08:37.857 --> 0:08:50.512
+And then there is now a third type of pre-trained
+model that recently became very successful
+
+0:08:50.512 --> 0:08:55.411
+and now with large language models.
+
+0:08:55.715 --> 0:09:03.525
+So the idea is we are no longer sharing the
+real data, but it can also help to train a
+
+0:09:03.525 --> 0:09:04.153
+model.
+
+0:09:04.364 --> 0:09:11.594
+And that is now a big advantage of deep learning
+based approaches.
+
+0:09:11.594 --> 0:09:22.169
+There you have this ability that you can train
+a model in some task and then apply it to another.
+
+0:09:22.722 --> 0:09:33.405
+And then, of course, the question is, can
+I have an initial task where there's huge amounts
+
+0:09:33.405 --> 0:09:34.450
+of data?
+
+0:09:34.714 --> 0:09:40.251
+And the test that typically you pre train
+on is more like similar to a language moral
+
+0:09:40.251 --> 0:09:45.852
+task either direct to a language moral task
+or like a masking task which is related so
+
+0:09:45.852 --> 0:09:51.582
+the idea is oh I can train on this data and
+the knowledge about words how they relate to
+
+0:09:51.582 --> 0:09:53.577
+each other I can use in there.
+
+0:09:53.753 --> 0:10:00.276
+So it's a different way of using language
+models.
+
+0:10:00.276 --> 0:10:06.276
+There's more transfer learning at the end
+of.
+
+0:10:09.029 --> 0:10:17.496
+So first we will start with how can we use
+monolingual data to do a Yeah to do a machine
+
+0:10:17.496 --> 0:10:18.733
+translation?
+
+0:10:20.040 --> 0:10:27.499
+That: Big difference is you should remember
+from what I mentioned before is.
+
+0:10:27.499 --> 0:10:32.783
+In statistical machine translation we directly
+have the opportunity.
+
+0:10:32.783 --> 0:10:39.676
+There's peril data for the translation model
+and monolingual data for the language model.
+
+0:10:39.679 --> 0:10:45.343
+And you combine your translation model and
+language model, and then you can make use of
+
+0:10:45.343 --> 0:10:45.730
+both.
+
+0:10:46.726 --> 0:10:53.183
+That you can make use of these large large
+amounts of monolingual data, but of course
+
+0:10:53.183 --> 0:10:55.510
+it has also some disadvantage.
+
+0:10:55.495 --> 0:11:01.156
+Because we say the problem is we are optimizing
+both parts a bit independently to each other
+
+0:11:01.156 --> 0:11:06.757
+and we say oh yeah the big disadvantage of
+newer machine translations now we are optimizing
+
+0:11:06.757 --> 0:11:10.531
+the overall architecture everything together
+to perform best.
+
+0:11:10.890 --> 0:11:16.994
+And then, of course, we can't do there, so
+Leo we can can only do a mural like use power
+
+0:11:16.994 --> 0:11:17.405
+data.
+
+0:11:17.897 --> 0:11:28.714
+So the question is, but this advantage is
+not so important that we can train everything,
+
+0:11:28.714 --> 0:11:35.276
+but we have a moral legal data or even small
+amounts.
+
+0:11:35.675 --> 0:11:43.102
+So in data we know it's not only important
+the amount of data we have but also like how
+
+0:11:43.102 --> 0:11:50.529
+similar it is to your test data so it can be
+that this modeling data is quite small but
+
+0:11:50.529 --> 0:11:55.339
+it's very well fitting and then it's still
+very helpful.
+
+0:11:55.675 --> 0:12:02.691
+At the first year of surprisingness, if we
+are here successful with integrating a language
+
+0:12:02.691 --> 0:12:09.631
+model into a translation system, maybe we can
+also integrate some type of language models
+
+0:12:09.631 --> 0:12:14.411
+into our empty system in order to make it better
+and perform.
+
+0:12:16.536 --> 0:12:23.298
+The first thing we can do is we know there
+is language models, so let's try to integrate.
+
+0:12:23.623 --> 0:12:31.096
+There was our language model because these
+works were mainly done before transformer-based
+
+0:12:31.096 --> 0:12:31.753
+models.
+
+0:12:32.152 --> 0:12:38.764
+In general, of course, you can do the same
+thing with transformer baseball.
+
+0:12:38.764 --> 0:12:50.929
+There is nothing about whether: It's just
+that it has mainly been done before people
+
+0:12:50.929 --> 0:13:01.875
+started using R&S and they tried to do
+this more in cases.
+
+0:13:07.087 --> 0:13:22.938
+So what we're happening here is in some of
+this type of idea, and in key system you remember
+
+0:13:22.938 --> 0:13:25.495
+the attention.
+
+0:13:25.605 --> 0:13:29.465
+Gets it was your last in this day that you
+calculate easy attention.
+
+0:13:29.729 --> 0:13:36.610
+We get the context back, then combine both
+and then base the next in state and then predict.
+
+0:13:37.057 --> 0:13:42.424
+So this is our system, and the question is,
+can we send our integrated language model?
+
+0:13:42.782 --> 0:13:49.890
+And somehow it makes sense to take out a neural
+language model because we are anyway in the
+
+0:13:49.890 --> 0:13:50.971
+neural space.
+
+0:13:50.971 --> 0:13:58.465
+It's not surprising that it contrasts to statistical
+work used and grants it might make sense to
+
+0:13:58.465 --> 0:14:01.478
+take a bit of a normal language model.
+
+0:14:01.621 --> 0:14:06.437
+And there would be something like on Tubbles
+Air, a neural language model, and our man based
+
+0:14:06.437 --> 0:14:11.149
+is you have a target word, you put it in, you
+get a new benchmark, and then you always put
+
+0:14:11.149 --> 0:14:15.757
+in the words and get new hidden states, and
+you can do some predictions at the output to
+
+0:14:15.757 --> 0:14:16.948
+predict the next word.
+
+0:14:17.597 --> 0:14:26.977
+So if we're having this type of in language
+model, there's like two main questions we have
+
+0:14:26.977 --> 0:14:34.769
+to answer: So how do we combine now on the
+one hand our system and on the other hand our
+
+0:14:34.769 --> 0:14:35.358
+model?
+
+0:14:35.358 --> 0:14:42.004
+You see that was mentioned before when we
+started talking about ENCODA models.
+
+0:14:42.004 --> 0:14:45.369
+They can be viewed as a language model.
+
+0:14:45.805 --> 0:14:47.710
+The wine is lengthened, unconditioned.
+
+0:14:47.710 --> 0:14:49.518
+It's just modeling the target sides.
+
+0:14:49.970 --> 0:14:56.963
+And the other one is a conditional language
+one, which is a language one conditioned on
+
+0:14:56.963 --> 0:14:57.837
+the Sewer.
+
+0:14:58.238 --> 0:15:03.694
+So how can you combine to language models?
+
+0:15:03.694 --> 0:15:14.860
+Of course, it's like the translation model
+will be more important because it has access
+
+0:15:14.860 --> 0:15:16.763
+to the source.
+
+0:15:18.778 --> 0:15:22.571
+If we have that, the other question is okay.
+
+0:15:22.571 --> 0:15:24.257
+Now we have models.
+
+0:15:24.257 --> 0:15:25.689
+How do we train?
+
+0:15:26.026 --> 0:15:30.005
+Pickers integrated them.
+
+0:15:30.005 --> 0:15:34.781
+We have now two sets of data.
+
+0:15:34.781 --> 0:15:42.741
+We have parallel data where you can do the
+lower.
+
+0:15:44.644 --> 0:15:53.293
+So the first idea is we can do something more
+like a parallel combination.
+
+0:15:53.293 --> 0:15:55.831
+We just keep running.
+
+0:15:56.036 --> 0:15:59.864
+So here you see your system that is running.
+
+0:16:00.200 --> 0:16:09.649
+It's normally completely independent of your
+language model, which is up there, so down
+
+0:16:09.649 --> 0:16:13.300
+here we have just our NMT system.
+
+0:16:13.313 --> 0:16:26.470
+The only thing which is used is we have the
+words, and of course they are put into both
+
+0:16:26.470 --> 0:16:30.059
+systems, and out there.
+
+0:16:30.050 --> 0:16:42.221
+So we use them somehow for both, and then
+we are doing our decision just by merging these
+
+0:16:42.221 --> 0:16:42.897
+two.
+
+0:16:43.343 --> 0:16:53.956
+So there can be, for example, we are doing
+a probability distribution here, and then we
+
+0:16:53.956 --> 0:17:03.363
+are taking the average of post-perability distribution
+to do our predictions.
+
+0:17:11.871 --> 0:17:18.923
+You could also take the output with Steve's
+to be more in chore about the mixture.
+
+0:17:20.000 --> 0:17:32.896
+Yes, you could also do that, so it's more
+like engaging mechanisms that you're not doing.
+
+0:17:32.993 --> 0:17:41.110
+Another one would be cochtrinate the hidden
+states, and then you would have another layer
+
+0:17:41.110 --> 0:17:41.831
+on top.
+
+0:17:43.303 --> 0:17:56.889
+You think about if you do the conqueredination
+instead of taking the instead and then merging
+
+0:17:56.889 --> 0:18:01.225
+the probability distribution.
+
+0:18:03.143 --> 0:18:16.610
+Introduce many new parameters, and these parameters
+have somehow something special compared to
+
+0:18:16.610 --> 0:18:17.318
+the.
+
+0:18:23.603 --> 0:18:37.651
+So before all the error other parameters can
+be trained independent, the language model
+
+0:18:37.651 --> 0:18:42.121
+can be trained independent.
+
+0:18:43.043 --> 0:18:51.749
+If you have a joint layer, of course you need
+to train them because you have now inputs.
+
+0:18:54.794 --> 0:19:02.594
+Not surprisingly, if you have a parallel combination
+of whether you could, the other way is to do
+
+0:19:02.594 --> 0:19:04.664
+more serial combinations.
+
+0:19:04.924 --> 0:19:10.101
+How can you do a similar combination?
+
+0:19:10.101 --> 0:19:18.274
+Your final decision makes sense to do a face
+on the system.
+
+0:19:18.438 --> 0:19:20.996
+So you have on top of your normal and system.
+
+0:19:21.121 --> 0:19:30.678
+The only thing is now you're inputting into
+your system.
+
+0:19:30.678 --> 0:19:38.726
+You're no longer inputting the word embeddings.
+
+0:19:38.918 --> 0:19:45.588
+So you're training your mainly what you have
+your lower layers here which are trained more
+
+0:19:45.588 --> 0:19:52.183
+on the purely language model style and then
+on top your putting into the NMT system where
+
+0:19:52.183 --> 0:19:55.408
+it now has already here the language model.
+
+0:19:55.815 --> 0:19:58.482
+So here you can also view it.
+
+0:19:58.482 --> 0:20:06.481
+Here you have more contextual embeddings which
+no longer depend only on the word but they
+
+0:20:06.481 --> 0:20:10.659
+also depend on the context of the target site.
+
+0:20:11.051 --> 0:20:19.941
+But you have more understanding of the source
+word, so you have a language in the current
+
+0:20:19.941 --> 0:20:21.620
+target sentence.
+
+0:20:21.881 --> 0:20:27.657
+So if it's like the word can, for example,
+will be put in here always the same independent
+
+0:20:27.657 --> 0:20:31.147
+of its user can of beans, or if it's like I
+can do it.
+
+0:20:31.147 --> 0:20:37.049
+However, because you are having your language
+model style, you have maybe disintegrated this
+
+0:20:37.049 --> 0:20:40.984
+already a bit, and you give this information
+directly to the.
+
+0:20:41.701 --> 0:20:43.095
+An empty cyst.
+
+0:20:44.364 --> 0:20:49.850
+You, if you're remembering more the transformer
+based approach, you have some layers.
+
+0:20:49.850 --> 0:20:55.783
+The lower layers are purely languaged while
+the other ones are with attention to the source.
+
+0:20:55.783 --> 0:21:01.525
+So you can view it also that you just have
+lower layers which don't attend to the source.
+
+0:21:02.202 --> 0:21:07.227
+This is purely a language model, and then
+at some point you're starting to attend to
+
+0:21:07.227 --> 0:21:08.587
+the source and use it.
+
+0:21:13.493 --> 0:21:20.781
+Yes, so this is how you combine them in peril
+or first do the language model and then do.
+
+0:21:23.623 --> 0:21:26.147
+Questions for the integration.
+
+0:21:31.831 --> 0:21:35.034
+Not really sure about the input of the.
+
+0:21:35.475 --> 0:21:38.102
+Model, and in this case in the sequence.
+
+0:21:38.278 --> 0:21:53.199
+Case so the actual word that we transferred
+into a numerical lecture, and this is an input
+
+0:21:53.199 --> 0:21:54.838
+into the.
+
+0:21:56.176 --> 0:22:03.568
+That depends on if you view the word embedding
+as part of the language model.
+
+0:22:03.568 --> 0:22:10.865
+So if you first put the word target word then
+you do the one hot end coding.
+
+0:22:11.691 --> 0:22:13.805
+And then the word embedding there is the r&
+
+0:22:13.805 --> 0:22:13.937
+n.
+
+0:22:14.314 --> 0:22:21.035
+So you can use this together as your language
+model when you first do the word embedding.
+
+0:22:21.401 --> 0:22:24.346
+All you can say is like before.
+
+0:22:24.346 --> 0:22:28.212
+It's more a definition, but you're right.
+
+0:22:28.212 --> 0:22:30.513
+So what's the steps out?
+
+0:22:30.513 --> 0:22:36.128
+You take the word, the one hut encoding, the
+word embedding.
+
+0:22:36.516 --> 0:22:46.214
+What one of these parrots, you know, called
+a language model is definition wise and not
+
+0:22:46.214 --> 0:22:47.978
+that important.
+
+0:22:53.933 --> 0:23:02.264
+So the question is how can you then train
+them and make this this one work?
+
+0:23:02.264 --> 0:23:02.812
+The.
+
+0:23:03.363 --> 0:23:15.201
+So in the case where you combine the language
+one of the abilities you can train them independently
+
+0:23:15.201 --> 0:23:18.516
+and just put them together.
+
+0:23:18.918 --> 0:23:27.368
+Might not be the best because we have no longer
+the stability that we had before that optimally
+
+0:23:27.368 --> 0:23:29.128
+performed together.
+
+0:23:29.128 --> 0:23:33.881
+It's not clear if they really work the best
+together.
+
+0:23:34.514 --> 0:23:41.585
+At least you need to somehow find how much
+do you trust the one model and how much.
+
+0:23:43.323 --> 0:23:45.058
+Still in some cases useful.
+
+0:23:45.058 --> 0:23:48.530
+It might be helpful if you have only data
+and software.
+
+0:23:48.928 --> 0:23:59.064
+However, in MT we have one specific situation
+that at least for the MT part parallel is also
+
+0:23:59.064 --> 0:24:07.456
+always monolingual data, so what we definitely
+can do is train the language.
+
+0:24:08.588 --> 0:24:18.886
+So what we also can do is more like the pre-training
+approach.
+
+0:24:18.886 --> 0:24:24.607
+We first train the language model.
+
+0:24:24.704 --> 0:24:27.334
+The pre-training approach.
+
+0:24:27.334 --> 0:24:33.470
+You first train on the monolingual data and
+then you join the.
+
+0:24:33.933 --> 0:24:41.143
+Of course, the model size is this way, but
+the data size is too bigly the other way around.
+
+0:24:41.143 --> 0:24:47.883
+You often have a lot more monolingual data
+than you have here parallel data, in which
+
+0:24:47.883 --> 0:24:52.350
+scenario can you imagine where this type of
+pretraining?
+
+0:24:56.536 --> 0:24:57.901
+Any Ideas.
+
+0:25:04.064 --> 0:25:12.772
+One example where this might also be helpful
+if you want to adapt to domains.
+
+0:25:12.772 --> 0:25:22.373
+So let's say you do medical sentences and
+if you want to translate medical sentences.
+
+0:25:23.083 --> 0:25:26.706
+In this case it could be or its most probable
+happen.
+
+0:25:26.706 --> 0:25:32.679
+You're learning here up there what medical
+means, but in your fine tuning step the model
+
+0:25:32.679 --> 0:25:38.785
+is forgotten everything about Medicare, so
+you may be losing all the information you gain.
+
+0:25:39.099 --> 0:25:42.366
+So this type of priest training step is good.
+
+0:25:42.366 --> 0:25:47.978
+If your pretraining data is more general,
+very large and then you're adapting.
+
+0:25:48.428 --> 0:25:56.012
+But in the task with moral lingual data, which
+should be used to adapt the system to some
+
+0:25:56.012 --> 0:25:57.781
+general topic style.
+
+0:25:57.817 --> 0:26:06.795
+Then, of course, this is not a good strategy
+because you might forgot about everything up
+
+0:26:06.795 --> 0:26:09.389
+there and you don't have.
+
+0:26:09.649 --> 0:26:14.678
+So then you have to check what you can do
+for them.
+
+0:26:14.678 --> 0:26:23.284
+You can freeze this part and change it any
+more so you don't lose the ability or you can
+
+0:26:23.284 --> 0:26:25.702
+do a direct combination.
+
+0:26:25.945 --> 0:26:31.028
+Where you jointly train both of them, so you
+train the NMT system on the, and then you train
+
+0:26:31.028 --> 0:26:34.909
+the language model always in parallels so that
+you don't forget about.
+
+0:26:35.395 --> 0:26:37.684
+And what you learn of the length.
+
+0:26:37.937 --> 0:26:46.711
+Depends on what you want to combine because
+it's large data and you have a good general
+
+0:26:46.711 --> 0:26:48.107
+knowledge in.
+
+0:26:48.548 --> 0:26:55.733
+Then you normally don't really forget it because
+it's also in the or you use it to adapt to
+
+0:26:55.733 --> 0:26:57.295
+something specific.
+
+0:26:57.295 --> 0:26:58.075
+Then you.
+
+0:27:01.001 --> 0:27:06.676
+Then this is a way of how we can make use
+of monolingual data.
+
+0:27:07.968 --> 0:27:12.116
+It seems to be the easiest one somehow.
+
+0:27:12.116 --> 0:27:20.103
+It's more similar to what we are doing with
+statistical machine translation.
+
+0:27:21.181 --> 0:27:31.158
+Normally always beats this type of model,
+which in some view can be like from the conceptual
+
+0:27:31.158 --> 0:27:31.909
+thing.
+
+0:27:31.909 --> 0:27:36.844
+It's even easier from the computational side.
+
+0:27:40.560 --> 0:27:42.078
+And the idea is OK.
+
+0:27:42.078 --> 0:27:49.136
+We have monolingual data that we just translate
+and then generate some type of parallel data
+
+0:27:49.136 --> 0:27:50.806
+and use that then to.
+
+0:27:51.111 --> 0:28:00.017
+So if you want to build a German-to-English
+system first, take the large amount of data
+
+0:28:00.017 --> 0:28:02.143
+you have translated.
+
+0:28:02.402 --> 0:28:10.446
+Then you have more peril data and the interesting
+thing is if you then train on the joint thing
+
+0:28:10.446 --> 0:28:18.742
+or on the original peril data and on what is
+artificial where you have generated the translations.
+
+0:28:18.918 --> 0:28:26.487
+So you can because you are not doing the same
+era all the times and you have some knowledge.
+
+0:28:28.028 --> 0:28:43.199
+With this first approach, however, there is
+one issue why it might not work the best.
+
+0:28:49.409 --> 0:28:51.177
+Very a bit shown in the image to you.
+
+0:28:53.113 --> 0:28:58.153
+You trade on that quality data.
+
+0:28:58.153 --> 0:29:02.563
+Here is a bit of a problem.
+
+0:29:02.563 --> 0:29:08.706
+Your English style is not really good.
+
+0:29:08.828 --> 0:29:12.213
+And as you're saying, the system always mistranslates.
+
+0:29:13.493 --> 0:29:19.798
+Something then you will learn that this is
+correct because now it's a training game and
+
+0:29:19.798 --> 0:29:23.022
+you will encourage it to make it more often.
+
+0:29:23.022 --> 0:29:29.614
+So the problem with training on your own areas
+yeah you might prevent some areas you rarely
+
+0:29:29.614 --> 0:29:29.901
+do.
+
+0:29:30.150 --> 0:29:31.749
+But errors use systematically.
+
+0:29:31.749 --> 0:29:34.225
+Do you even enforce more and will even do
+more?
+
+0:29:34.654 --> 0:29:40.145
+So that might not be the best solution to
+have any idea how you could do it better.
+
+0:29:44.404 --> 0:29:57.754
+Is one way there is even a bit of more simple
+idea.
+
+0:30:04.624 --> 0:30:10.975
+The problem is yeah, the translations are
+not perfect, so the output and you're learning
+
+0:30:10.975 --> 0:30:12.188
+something wrong.
+
+0:30:12.188 --> 0:30:17.969
+Normally it's less bad if your inputs are
+not bad, but your outputs are perfect.
+
+0:30:18.538 --> 0:30:24.284
+So if your inputs are wrong you may learn
+that if you're doing this wrong input you're
+
+0:30:24.284 --> 0:30:30.162
+generating something correct, but you're not
+learning to generate something which is not
+
+0:30:30.162 --> 0:30:30.756
+correct.
+
+0:30:31.511 --> 0:30:47.124
+So often the case it is that it is more important
+than your target is correct.
+
+0:30:47.347 --> 0:30:52.182
+But you can assume in your application scenario
+you hope that you may only get correct inputs.
+
+0:30:52.572 --> 0:31:02.535
+So that is not harming you, and in machine
+translation we have one very nice advantage:
+
+0:31:02.762 --> 0:31:04.648
+And also the other way around.
+
+0:31:04.648 --> 0:31:10.062
+It's a very similar task, so there's a task
+to translate from German to English, but the
+
+0:31:10.062 --> 0:31:13.894
+task to translate from English to German is
+very similar, and.
+
+0:31:14.094 --> 0:31:19.309
+So what we can do is we can just switch it
+initially and generate the data the other way
+
+0:31:19.309 --> 0:31:19.778
+around.
+
+0:31:20.120 --> 0:31:25.959
+So what we are doing here is we are starting
+with an English to German system.
+
+0:31:25.959 --> 0:31:32.906
+Then we are translating the English data into
+German where the German is maybe not very nice.
+
+0:31:33.293 --> 0:31:51.785
+And then we are training on our original data
+and on the back translated data.
+
+0:31:52.632 --> 0:32:02.332
+So here we have the advantage that our target
+side is human quality and only the input.
+
+0:32:03.583 --> 0:32:08.113
+Then this helps us to get really good.
+
+0:32:08.113 --> 0:32:15.431
+There is one difference if you think about
+the data resources.
+
+0:32:21.341 --> 0:32:27.336
+Too obvious here we need a target site monolingual
+layer.
+
+0:32:27.336 --> 0:32:31.574
+In the first example we had source site.
+
+0:32:31.931 --> 0:32:45.111
+So back translation is normally working if
+you have target size peril later and not search
+
+0:32:45.111 --> 0:32:48.152
+side modeling later.
+
+0:32:48.448 --> 0:32:56.125
+Might be also, like if you think about it,
+understand a little better to understand the
+
+0:32:56.125 --> 0:32:56.823
+target.
+
+0:32:57.117 --> 0:33:01.469
+On the source side you have to understand
+the content.
+
+0:33:01.469 --> 0:33:08.749
+On the target side you have to generate really
+sentences and somehow it's more difficult to
+
+0:33:08.749 --> 0:33:12.231
+generate something than to only understand.
+
+0:33:17.617 --> 0:33:30.734
+This works well if you have to select how
+many back translated data do you use.
+
+0:33:31.051 --> 0:33:32.983
+Because only there's like a lot more.
+
+0:33:33.253 --> 0:33:42.136
+Question: Should take all of my data there
+is two problems with it?
+
+0:33:42.136 --> 0:33:51.281
+Of course it's expensive because you have
+to translate all this data.
+
+0:33:51.651 --> 0:34:00.946
+So if you don't know the normal good starting
+point is to take equal amount of data as many
+
+0:34:00.946 --> 0:34:02.663
+back translated.
+
+0:34:02.963 --> 0:34:04.673
+It depends on the used case.
+
+0:34:04.673 --> 0:34:08.507
+If we have very few data here, it makes more
+sense to have more.
+
+0:34:08.688 --> 0:34:15.224
+Depends on how good your quality is here,
+so the better the more data you might use because
+
+0:34:15.224 --> 0:34:16.574
+quality is better.
+
+0:34:16.574 --> 0:34:22.755
+So it depends on a lot of things, but your
+rule of sum is like which general way often
+
+0:34:22.755 --> 0:34:24.815
+is to have equal amounts of.
+
+0:34:26.646 --> 0:34:29.854
+And you can, of course, do that now.
+
+0:34:29.854 --> 0:34:34.449
+I said already that it's better to have the
+quality.
+
+0:34:34.449 --> 0:34:38.523
+At the end, of course, depends on this system.
+
+0:34:38.523 --> 0:34:46.152
+Also, because the better this system is, the
+better your synthetic data is, the better.
+
+0:34:47.207 --> 0:34:50.949
+That leads to what is referred to as iterated
+back translation.
+
+0:34:51.291 --> 0:34:56.917
+So you play them on English to German, and
+you translate the data on.
+
+0:34:56.957 --> 0:35:03.198
+Then you train a model on German to English
+with the additional data.
+
+0:35:03.198 --> 0:35:09.796
+Then you translate German data and then you
+train to gain your first one.
+
+0:35:09.796 --> 0:35:14.343
+So in the second iteration this quality is
+better.
+
+0:35:14.334 --> 0:35:19.900
+System is better because it's not only trained
+on the small data but additionally on back
+
+0:35:19.900 --> 0:35:22.003
+translated data with this system.
+
+0:35:22.442 --> 0:35:24.458
+And so you can get better.
+
+0:35:24.764 --> 0:35:28.053
+However, typically you can stop quite early.
+
+0:35:28.053 --> 0:35:35.068
+Maybe one iteration is good, but then you
+have diminishing gains after two or three iterations.
+
+0:35:35.935 --> 0:35:46.140
+There is very slight difference because you
+need a quite big difference in the quality
+
+0:35:46.140 --> 0:35:46.843
+here.
+
+0:35:47.207 --> 0:36:02.262
+Language is also good because it means you
+can already train it with relatively bad profiles.
+
+0:36:03.723 --> 0:36:10.339
+It's a design decision would advise so guess
+because it's easy to get it.
+
+0:36:10.550 --> 0:36:20.802
+Replace that because you have a higher quality
+real data, but then I think normally it's okay
+
+0:36:20.802 --> 0:36:22.438
+to replace it.
+
+0:36:22.438 --> 0:36:28.437
+I would assume it's not too much of a difference,
+but.
+
+0:36:34.414 --> 0:36:42.014
+That's about like using monolingual data before
+we go into the pre-train models to have any
+
+0:36:42.014 --> 0:36:43.005
+more crash.
+
+0:36:49.029 --> 0:36:55.740
+Yes, so the other thing which we can do and
+which is recently more and more successful
+
+0:36:55.740 --> 0:37:02.451
+and even more successful since we have this
+really large language models where you can
+
+0:37:02.451 --> 0:37:08.545
+even do the translation task with this is the
+way of using pre-trained models.
+
+0:37:08.688 --> 0:37:16.135
+So you learn a representation of one task,
+and then you use this representation from another.
+
+0:37:16.576 --> 0:37:26.862
+It was made maybe like one of the first words
+where it really used largely is doing something
+
+0:37:26.862 --> 0:37:35.945
+like a bird which you pre trained on purely
+text era and you take it in fine tune.
+
+0:37:36.496 --> 0:37:42.953
+And one big advantage, of course, is that
+people can only share data but also pre-trained.
+
+0:37:43.423 --> 0:37:59.743
+The recent models and the large language ones
+which are available.
+
+0:37:59.919 --> 0:38:09.145
+Where I think it costs several millions to
+train them all, just if you would buy the GPUs
+
+0:38:09.145 --> 0:38:15.397
+from some cloud company and train that the
+cost of training.
+
+0:38:15.475 --> 0:38:21.735
+And guess as a student project you won't have
+the budget to like build these models.
+
+0:38:21.801 --> 0:38:24.598
+So another idea is what you can do is okay.
+
+0:38:24.598 --> 0:38:27.330
+Maybe if these months are once available,.
+
+0:38:27.467 --> 0:38:36.598
+Can take them and use them as an also resource
+similar to pure text, and you can now build
+
+0:38:36.598 --> 0:38:44.524
+models which somehow learn not only from from
+data but also from other models.
+
+0:38:44.844 --> 0:38:49.127
+So it's a quite new way of thinking of how
+to train.
+
+0:38:49.127 --> 0:38:53.894
+We are not only learning from examples, but
+we might also.
+
+0:38:54.534 --> 0:39:05.397
+The nice thing is that this type of training
+where we are not learning directly from data
+
+0:39:05.397 --> 0:39:07.087
+but learning.
+
+0:39:07.427 --> 0:39:17.647
+So the main idea this go is you have a person
+initial task.
+
+0:39:17.817 --> 0:39:26.369
+And if you're working with anLP, that means
+you're training pure taxator because that's
+
+0:39:26.369 --> 0:39:30.547
+where you have the largest amount of data.
+
+0:39:30.951 --> 0:39:35.857
+And then you're defining some type of task
+in order to do your creek training.
+
+0:39:36.176 --> 0:39:43.092
+And: The typical task you can train on on
+that is like the language waddling task.
+
+0:39:43.092 --> 0:39:50.049
+So to predict the next word or we have a related
+task to predict something in between, we'll
+
+0:39:50.049 --> 0:39:52.667
+see depending on the architecture.
+
+0:39:52.932 --> 0:39:58.278
+But somehow to predict something which you
+have not in the input is a task which is easy
+
+0:39:58.278 --> 0:40:00.740
+to generate, so you just need your data.
+
+0:40:00.740 --> 0:40:06.086
+That's why it's called self supervised, so
+you're creating your supervised pending data.
+
+0:40:06.366 --> 0:40:07.646
+By yourself.
+
+0:40:07.646 --> 0:40:15.133
+On the other hand, you need a lot of knowledge
+and that is the other thing.
+
+0:40:15.735 --> 0:40:24.703
+Because there is this idea that the meaning
+of a word heavily depends on the context that.
+
+0:40:25.145 --> 0:40:36.846
+So can give you a sentence with some giverish
+word and there's some name and although you've
+
+0:40:36.846 --> 0:40:41.627
+never heard the name you will assume.
+
+0:40:42.062 --> 0:40:44.149
+And exactly the same thing.
+
+0:40:44.149 --> 0:40:49.143
+The models can also learn something about
+the world by just using.
+
+0:40:49.649 --> 0:40:53.651
+So that is typically the mule.
+
+0:40:53.651 --> 0:40:59.848
+Then we can use this model to train the system.
+
+0:41:00.800 --> 0:41:03.368
+Course we might need to adapt the system.
+
+0:41:03.368 --> 0:41:07.648
+To do that we have to change the architecture
+we might use only some.
+
+0:41:07.627 --> 0:41:09.443
+Part of the pre-trained model.
+
+0:41:09.443 --> 0:41:14.773
+In there we have seen that a bit already in
+the R&N case you can also see that we have
+
+0:41:14.773 --> 0:41:17.175
+also mentioned the pre-training already.
+
+0:41:17.437 --> 0:41:22.783
+So you can use the R&N as one of these
+approaches.
+
+0:41:22.783 --> 0:41:28.712
+You train the R&M language more on large
+pre-train data.
+
+0:41:28.712 --> 0:41:32.309
+Then you put it somewhere into your.
+
+0:41:33.653 --> 0:41:37.415
+So this gives you the ability to really do
+these types of tests.
+
+0:41:37.877 --> 0:41:53.924
+So you can build a system which is knowledge,
+which is just trained on large amounts of data.
+
+0:41:56.376 --> 0:42:01.564
+So the question is maybe what type of information
+so what type of models can you?
+
+0:42:01.821 --> 0:42:05.277
+And we want today to look at briefly at swings.
+
+0:42:05.725 --> 0:42:08.704
+First, that was what was initially done.
+
+0:42:08.704 --> 0:42:15.314
+It wasn't as famous as in machine translation
+as in other things, but it's also used there
+
+0:42:15.314 --> 0:42:21.053
+and that is to use static word embedding, so
+just the first step we know here.
+
+0:42:21.221 --> 0:42:28.981
+So we have this mapping from the one hot to
+a small continuous word representation.
+
+0:42:29.229 --> 0:42:38.276
+Using this one in your NG system, so you can,
+for example, replace the embedding layer by
+
+0:42:38.276 --> 0:42:38.779
+the.
+
+0:42:39.139 --> 0:42:41.832
+That is helpful to be a really small amount
+of data.
+
+0:42:42.922 --> 0:42:48.517
+And we're always in this pre-training phase
+and have the thing the advantage is.
+
+0:42:48.468 --> 0:42:52.411
+More data than the trade off, so you can get
+better.
+
+0:42:52.411 --> 0:42:59.107
+The disadvantage is, does anybody have an
+idea of what might be the disadvantage of using
+
+0:42:59.107 --> 0:43:00.074
+things like.
+
+0:43:04.624 --> 0:43:12.175
+What was one mentioned today giving like big
+advantage of the system compared to previous.
+
+0:43:20.660 --> 0:43:25.134
+Where one advantage was the enter end training,
+so you have the enter end training so that
+
+0:43:25.134 --> 0:43:27.937
+all parameters and all components play optimal
+together.
+
+0:43:28.208 --> 0:43:33.076
+If you know pre-train something on one fast,
+it may be no longer optimal fitting to everything
+
+0:43:33.076 --> 0:43:33.384
+else.
+
+0:43:33.893 --> 0:43:37.862
+So what do pretending or not?
+
+0:43:37.862 --> 0:43:48.180
+It depends on how important everything is
+optimal together and how important.
+
+0:43:48.388 --> 0:43:50.454
+Of large amount.
+
+0:43:50.454 --> 0:44:00.541
+The pre-change one is so much better that
+it's helpful, and the advantage of that.
+
+0:44:00.600 --> 0:44:11.211
+Getting everything optimal together, yes,
+we would use random instructions for raising.
+
+0:44:11.691 --> 0:44:26.437
+The problem is you might be already in some
+area where it's not easy to get.
+
+0:44:26.766 --> 0:44:35.329
+But often in some way right, so often it's
+not about your really worse pre trained monolepsy.
+
+0:44:35.329 --> 0:44:43.254
+If you're going already in some direction,
+and if this is not really optimal for you,.
+
+0:44:43.603 --> 0:44:52.450
+But if you're not really getting better because
+you have a decent amount of data, it's so different
+
+0:44:52.450 --> 0:44:52.981
+that.
+
+0:44:53.153 --> 0:44:59.505
+Initially it wasn't a machine translation
+done so much because there are more data in
+
+0:44:59.505 --> 0:45:06.153
+MPs than in other tasks, but now with really
+large amounts of monolingual data we do some
+
+0:45:06.153 --> 0:45:09.403
+type of pretraining in currently all state.
+
+0:45:12.632 --> 0:45:14.302
+The other one is okay now.
+
+0:45:14.302 --> 0:45:18.260
+It's always like how much of the model do
+you plea track a bit?
+
+0:45:18.658 --> 0:45:22.386
+To the other one you can do contextural word
+embedded.
+
+0:45:22.386 --> 0:45:28.351
+That is something like bird or Roberta where
+you train already a sequence model and the
+
+0:45:28.351 --> 0:45:34.654
+embeddings you're using are no longer specific
+for word but they are also taking the context
+
+0:45:34.654 --> 0:45:35.603
+into account.
+
+0:45:35.875 --> 0:45:50.088
+The embedding you're using is no longer depending
+on the word itself but on the whole sentence,
+
+0:45:50.088 --> 0:45:54.382
+so you can use this context.
+
+0:45:55.415 --> 0:46:02.691
+You can use similar things also in the decoder
+just by having layers which don't have access
+
+0:46:02.691 --> 0:46:12.430
+to the source, but there it still might have
+and these are typically models like: And finally
+
+0:46:12.430 --> 0:46:14.634
+they will look at the end.
+
+0:46:14.634 --> 0:46:19.040
+You can also have models which are already
+sequenced.
+
+0:46:19.419 --> 0:46:28.561
+So you may be training a sequence to sequence
+models.
+
+0:46:28.561 --> 0:46:35.164
+You have to make it a bit challenging.
+
+0:46:36.156 --> 0:46:43.445
+But the idea is really you're pre-training
+your whole model and then you'll find tuning.
+
+0:46:47.227 --> 0:46:59.614
+But let's first do a bit of step back and
+look into what are the different things.
+
+0:46:59.614 --> 0:47:02.151
+The first thing.
+
+0:47:02.382 --> 0:47:11.063
+The wooden bettings are just this first layer
+and you can train them with feedback annual
+
+0:47:11.063 --> 0:47:12.028
+networks.
+
+0:47:12.212 --> 0:47:22.761
+But you can also train them with an N language
+model, and by now you hopefully have also seen
+
+0:47:22.761 --> 0:47:27.699
+that you cannot transform a language model.
+
+0:47:30.130 --> 0:47:37.875
+So this is how you can train them and you're
+training them.
+
+0:47:37.875 --> 0:47:45.234
+For example, to speak the next word that is
+the easiest.
+
+0:47:45.525 --> 0:47:55.234
+And that is what is now referred to as South
+Supervised Learning and, for example, all the
+
+0:47:55.234 --> 0:48:00.675
+big large language models like Chad GPT and
+so on.
+
+0:48:00.675 --> 0:48:03.129
+They are trained with.
+
+0:48:03.823 --> 0:48:15.812
+So that is where you can hopefully learn how
+a word is used because you always try to previct
+
+0:48:15.812 --> 0:48:17.725
+the next word.
+
+0:48:19.619 --> 0:48:27.281
+Word embedding: Why do you keep the first
+look at the word embeddings and the use of
+
+0:48:27.281 --> 0:48:29.985
+word embeddings for our task?
+
+0:48:29.985 --> 0:48:38.007
+The main advantage was it might be only the
+first layer where you typically have most of
+
+0:48:38.007 --> 0:48:39.449
+the parameters.
+
+0:48:39.879 --> 0:48:57.017
+Most of your parameters already on the large
+data, then on your target data you have to
+
+0:48:57.017 --> 0:48:59.353
+train less.
+
+0:48:59.259 --> 0:49:06.527
+Big difference that your input size is so
+much bigger than the size of the novel in size.
+
+0:49:06.626 --> 0:49:17.709
+So it's a normally sign, maybe like, but your
+input and banning size is something like.
+
+0:49:17.709 --> 0:49:20.606
+Then here you have to.
+
+0:49:23.123 --> 0:49:30.160
+While here you see it's only like zero point
+five times as much in the layer.
+
+0:49:30.750 --> 0:49:36.534
+So here is where most of your parameters are,
+which means if you already replace the word
+
+0:49:36.534 --> 0:49:41.739
+embeddings, they might look a bit small in
+your overall and in key architecture.
+
+0:49:41.739 --> 0:49:47.395
+It's where most of the things are, and if
+you're doing that you already have really big
+
+0:49:47.395 --> 0:49:48.873
+games and can do that.
+
+0:49:57.637 --> 0:50:01.249
+The thing is we have seen these were the bettings.
+
+0:50:01.249 --> 0:50:04.295
+They can be very good use for other types.
+
+0:50:04.784 --> 0:50:08.994
+You learn some general relations between words.
+
+0:50:08.994 --> 0:50:17.454
+If you're doing this type of language modeling
+cast, you predict: The one thing is you have
+
+0:50:17.454 --> 0:50:24.084
+a lot of data, so the one question is we want
+to have data to trade a model.
+
+0:50:24.084 --> 0:50:28.734
+The other thing, the tasks need to be somehow
+useful.
+
+0:50:29.169 --> 0:50:43.547
+If you would predict the first letter of the
+word, then you wouldn't learn anything about
+
+0:50:43.547 --> 0:50:45.144
+the word.
+
+0:50:45.545 --> 0:50:53.683
+And the interesting thing is people have looked
+at these wood embeddings.
+
+0:50:53.954 --> 0:50:58.550
+And looking at the word embeddings.
+
+0:50:58.550 --> 0:51:09.276
+You can ask yourself how they look and visualize
+them by doing dimension reduction.
+
+0:51:09.489 --> 0:51:13.236
+Don't know if you and you are listening to
+artificial intelligence.
+
+0:51:13.236 --> 0:51:15.110
+Advanced artificial intelligence.
+
+0:51:15.515 --> 0:51:23.217
+We had on yesterday there how to do this type
+of representation, but you can do this time
+
+0:51:23.217 --> 0:51:29.635
+of representation, and now you're seeing interesting
+things that normally.
+
+0:51:30.810 --> 0:51:41.027
+Now you can represent a here in a three dimensional
+space with some dimension reduction.
+
+0:51:41.027 --> 0:51:46.881
+For example, the relation between male and
+female.
+
+0:51:47.447 --> 0:51:56.625
+So this vector between the male and female
+version of something is always not the same,
+
+0:51:56.625 --> 0:51:58.502
+but it's related.
+
+0:51:58.718 --> 0:52:14.522
+So you can do a bit of maths, so you do take
+king, you subtract this vector, add this vector.
+
+0:52:14.894 --> 0:52:17.591
+So that means okay, there is really something
+stored.
+
+0:52:17.591 --> 0:52:19.689
+Some information are stored in that book.
+
+0:52:20.040 --> 0:52:22.621
+Similar, you can do it with Bob Hansen.
+
+0:52:22.621 --> 0:52:25.009
+See here swimming slam walking walk.
+
+0:52:25.265 --> 0:52:34.620
+So again these vectors are not the same, but
+they are related.
+
+0:52:34.620 --> 0:52:42.490
+So you learn something from going from here
+to here.
+
+0:52:43.623 --> 0:52:49.761
+Or semantically, the relations between city
+and capital have exactly the same sense.
+
+0:52:51.191 --> 0:52:56.854
+And people had even done that question answering
+about that if they showed the diembeddings
+
+0:52:56.854 --> 0:52:57.839
+and the end of.
+
+0:52:58.218 --> 0:53:06.711
+All you can also do is don't trust the dimensions
+of the reaction because maybe there is something.
+
+0:53:06.967 --> 0:53:16.863
+You can also look into what happens really
+in the individual space.
+
+0:53:16.863 --> 0:53:22.247
+What is the nearest neighbor of the.
+
+0:53:22.482 --> 0:53:29.608
+So you can take the relationship between France
+and Paris and add it to Italy and you'll.
+
+0:53:30.010 --> 0:53:33.078
+You can do big and bigger and you have small
+and smaller and stuff.
+
+0:53:33.593 --> 0:53:49.417
+Because it doesn't work everywhere, there
+is also some typical dish here in German.
+
+0:53:51.491 --> 0:54:01.677
+You can do what the person is doing for famous
+ones, of course only like Einstein scientists
+
+0:54:01.677 --> 0:54:06.716
+that find midfielders not completely correct.
+
+0:54:06.846 --> 0:54:10.134
+You see the examples are a bit old.
+
+0:54:10.134 --> 0:54:15.066
+The politicians are no longer they am, but
+of course.
+
+0:54:16.957 --> 0:54:26.759
+What people have done there, especially at
+the beginning training our end language model,
+
+0:54:26.759 --> 0:54:28.937
+was very expensive.
+
+0:54:29.309 --> 0:54:38.031
+So one famous model was, but we are not really
+interested in the language model performance.
+
+0:54:38.338 --> 0:54:40.581
+Think something good to keep in mind.
+
+0:54:40.581 --> 0:54:42.587
+What are we really interested in?
+
+0:54:42.587 --> 0:54:45.007
+Do we really want to have an R&N no?
+
+0:54:45.007 --> 0:54:48.607
+In this case we are only interested in this
+type of mapping.
+
+0:54:49.169 --> 0:54:55.500
+And so successful and very successful was
+this word to vet.
+
+0:54:55.535 --> 0:54:56.865
+The idea is okay.
+
+0:54:56.865 --> 0:55:03.592
+We are not training real language one, making
+it even simpler and doing this, for example,
+
+0:55:03.592 --> 0:55:05.513
+continuous peck of words.
+
+0:55:05.513 --> 0:55:12.313
+We're just having four input tokens and we're
+predicting what is the word in the middle and
+
+0:55:12.313 --> 0:55:15.048
+this is just like two linear layers.
+
+0:55:15.615 --> 0:55:21.627
+So it's even simplifying things and making
+the calculation faster because that is what
+
+0:55:21.627 --> 0:55:22.871
+we're interested.
+
+0:55:23.263 --> 0:55:32.897
+All this continuous skip ground models with
+these other models which refer to as where
+
+0:55:32.897 --> 0:55:34.004
+to where.
+
+0:55:34.234 --> 0:55:42.394
+Where you have one equal word and the other
+way around, you're predicting the four words
+
+0:55:42.394 --> 0:55:43.585
+around them.
+
+0:55:43.585 --> 0:55:45.327
+It's very similar.
+
+0:55:45.327 --> 0:55:48.720
+The task is in the end very similar.
+
+0:55:51.131 --> 0:56:01.407
+Before we are going to the next point, anything
+about normal weight vectors or weight embedding.
+
+0:56:04.564 --> 0:56:07.794
+The next thing is contexture.
+
+0:56:07.794 --> 0:56:12.208
+Word embeddings and the idea is helpful.
+
+0:56:12.208 --> 0:56:19.206
+However, we might even be able to get more
+from one lingo layer.
+
+0:56:19.419 --> 0:56:31.732
+And now in the word that is overlap of these
+two meanings, so it represents both the meaning
+
+0:56:31.732 --> 0:56:33.585
+of can do it.
+
+0:56:34.834 --> 0:56:40.410
+But we might be able to in the pre-trained
+model already disambiguate this because they
+
+0:56:40.410 --> 0:56:41.044
+are used.
+
+0:56:41.701 --> 0:56:53.331
+So if we can have a model which can not only
+represent a word but can also represent the
+
+0:56:53.331 --> 0:56:58.689
+meaning of the word within the context,.
+
+0:56:59.139 --> 0:57:03.769
+So then we are going to context your word
+embeddings.
+
+0:57:03.769 --> 0:57:07.713
+We are really having a representation in the.
+
+0:57:07.787 --> 0:57:11.519
+And we have a very good architecture for that
+already.
+
+0:57:11.691 --> 0:57:23.791
+The hidden state represents what is currently
+said, but it's focusing on what is the last
+
+0:57:23.791 --> 0:57:29.303
+one, so it's some of the representation.
+
+0:57:29.509 --> 0:57:43.758
+The first one doing that is something like
+the Elmo paper where they instead of this is
+
+0:57:43.758 --> 0:57:48.129
+the normal language model.
+
+0:57:48.008 --> 0:57:50.714
+Within the third, predicting the fourth, and
+so on.
+
+0:57:50.714 --> 0:57:53.004
+So you are always predicting the next work.
+
+0:57:53.193 --> 0:57:57.335
+The architecture is the heaven words embedding
+layer and then layers.
+
+0:57:57.335 --> 0:58:03.901
+See you, for example: And now instead of using
+this one in the end, you're using here this
+
+0:58:03.901 --> 0:58:04.254
+one.
+
+0:58:04.364 --> 0:58:11.245
+This represents the meaning of this word mainly
+in the context of what we have seen before.
+
+0:58:11.871 --> 0:58:18.610
+We can train it in a language model style
+always predicting the next word, but we have
+
+0:58:18.610 --> 0:58:21.088
+more information trained there.
+
+0:58:21.088 --> 0:58:26.123
+Therefore, in the system it has to learn less
+additional things.
+
+0:58:27.167 --> 0:58:31.261
+And there is one Edendang which is done currently
+in GPS.
+
+0:58:31.261 --> 0:58:38.319
+The only difference is that we have more layers,
+bigger size, and we're using transformer neurocell
+
+0:58:38.319 --> 0:58:40.437
+potential instead of the RNA.
+
+0:58:40.437 --> 0:58:45.095
+But that is how you train like some large
+language models at the.
+
+0:58:46.746 --> 0:58:55.044
+However, if you look at this contextual representation,
+they might not be perfect.
+
+0:58:55.044 --> 0:59:02.942
+So if you think of this one as a contextual
+representation of the third word,.
+
+0:59:07.587 --> 0:59:16.686
+Is representing a three in the context of
+a sentence, however only in the context of
+
+0:59:16.686 --> 0:59:18.185
+the previous.
+
+0:59:18.558 --> 0:59:27.413
+However, we have an architecture which can
+also take both sides and we have used that
+
+0:59:27.413 --> 0:59:30.193
+already in the ink holder.
+
+0:59:30.630 --> 0:59:34.264
+So we could do the iron easily on your, also
+in the backward direction.
+
+0:59:34.874 --> 0:59:42.826
+By just having the states the other way around
+and then we couldn't combine the forward and
+
+0:59:42.826 --> 0:59:49.135
+the forward into a joint one where we are doing
+this type of prediction.
+
+0:59:49.329 --> 0:59:50.858
+So you have the word embedding.
+
+0:59:51.011 --> 1:00:02.095
+Then you have two in the states, one on the
+forward arm and one on the backward arm, and
+
+1:00:02.095 --> 1:00:10.314
+then you can, for example, take the cocagenation
+of both of them.
+
+1:00:10.490 --> 1:00:23.257
+Now this same here represents mainly this
+word because this is what both puts in it last
+
+1:00:23.257 --> 1:00:30.573
+and we know is focusing on what is happening
+last.
+
+1:00:31.731 --> 1:00:40.469
+However, there is a bit of difference when
+training that as a language model you already
+
+1:00:40.469 --> 1:00:41.059
+have.
+
+1:00:43.203 --> 1:00:44.956
+Maybe There's Again This Masking.
+
+1:00:46.546 --> 1:00:47.748
+That is one solution.
+
+1:00:47.748 --> 1:00:52.995
+First of all, why we can't do it is the information
+you leak it, so you cannot just predict the
+
+1:00:52.995 --> 1:00:53.596
+next word.
+
+1:00:53.596 --> 1:00:58.132
+If we just predict the next word in this type
+of model, that's a very simple task.
+
+1:00:58.738 --> 1:01:09.581
+You know the next word because it's influencing
+this hidden state predicting something is not
+
+1:01:09.581 --> 1:01:11.081
+a good task.
+
+1:01:11.081 --> 1:01:18.455
+You have to define: Because in this case what
+will end with the system will just ignore these
+
+1:01:18.455 --> 1:01:22.966
+estates and what will learn is copy this information
+directly in here.
+
+1:01:23.343 --> 1:01:31.218
+So it would be representing this word and
+you would have nearly a perfect model because
+
+1:01:31.218 --> 1:01:38.287
+you only need to find encoding where you can
+encode all words somehow in this.
+
+1:01:38.458 --> 1:01:44.050
+The only thing can learn is that turn and
+encode all my words in this upper hidden.
+
+1:01:44.985 --> 1:01:53.779
+Therefore, it's not really useful, so we need
+to find a bit of different ways out.
+
+1:01:55.295 --> 1:01:57.090
+There is a masking one.
+
+1:01:57.090 --> 1:02:03.747
+I'll come to that shortly just a bit that
+other things also have been done, so the other
+
+1:02:03.747 --> 1:02:06.664
+thing is not to directly combine them.
+
+1:02:06.664 --> 1:02:13.546
+That was in the animal paper, so you have
+them forward R&M and you keep them completely
+
+1:02:13.546 --> 1:02:14.369
+separated.
+
+1:02:14.594 --> 1:02:20.458
+So you never merged to state.
+
+1:02:20.458 --> 1:02:33.749
+At the end, the representation of the word
+is now from the forward.
+
+1:02:33.873 --> 1:02:35.953
+So it's always the hidden state before the
+good thing.
+
+1:02:36.696 --> 1:02:41.286
+These two you join now to your to the representation.
+
+1:02:42.022 --> 1:02:48.685
+And then you have now a representation also
+about like the whole sentence for the word,
+
+1:02:48.685 --> 1:02:51.486
+but there is no information leakage.
+
+1:02:51.486 --> 1:02:58.149
+One way of doing this is instead of doing
+a bidirection along you do a forward pass and
+
+1:02:58.149 --> 1:02:59.815
+then join the hidden.
+
+1:03:00.380 --> 1:03:05.960
+So you can do that in all layers.
+
+1:03:05.960 --> 1:03:16.300
+In the end you do the forwarded layers and
+you get the hidden.
+
+1:03:16.596 --> 1:03:19.845
+However, it's a bit of a complicated.
+
+1:03:19.845 --> 1:03:25.230
+You have to keep both separate and merge things
+so can you do.
+
+1:03:27.968 --> 1:03:33.030
+And that is the moment where like the big.
+
+1:03:34.894 --> 1:03:39.970
+The big success of the burnt model was used
+where it okay.
+
+1:03:39.970 --> 1:03:47.281
+Maybe in bite and rich case it's not good
+to do the next word prediction, but we can
+
+1:03:47.281 --> 1:03:48.314
+do masking.
+
+1:03:48.308 --> 1:03:56.019
+Masking mainly means we do a prediction of
+something in the middle or some words.
+
+1:03:56.019 --> 1:04:04.388
+So the idea is if we have the input, we are
+putting noise into the input, removing them,
+
+1:04:04.388 --> 1:04:07.961
+and then the model we are interested.
+
+1:04:08.048 --> 1:04:15.327
+Now there can be no information leakage because
+this wasn't predicting that one is a big challenge.
+
+1:04:16.776 --> 1:04:19.957
+Do any assumption about our model?
+
+1:04:19.957 --> 1:04:26.410
+It doesn't need to be a forward model or a
+backward model or anything.
+
+1:04:26.410 --> 1:04:29.500
+You can always predict the three.
+
+1:04:30.530 --> 1:04:34.844
+There's maybe one bit of a disadvantage.
+
+1:04:34.844 --> 1:04:40.105
+Do you see what could be a bit of a problem
+this?
+
+1:05:00.000 --> 1:05:06.429
+Yes, so yeah, you can of course mask more,
+but to see it more globally, just first assume
+
+1:05:06.429 --> 1:05:08.143
+you're only masked one.
+
+1:05:08.143 --> 1:05:13.930
+For the whole sentence, we get one feedback
+signal, like what is the word three.
+
+1:05:13.930 --> 1:05:22.882
+So we have one training example: If you do
+the language modeling taste, we predicted here,
+
+1:05:22.882 --> 1:05:24.679
+we predicted here.
+
+1:05:25.005 --> 1:05:26.735
+So we have number of tokens.
+
+1:05:26.735 --> 1:05:30.970
+For each token we have a feet pad and say
+what is the best correction.
+
+1:05:31.211 --> 1:05:43.300
+So in this case this is less efficient because
+we are getting less feedback signals on what
+
+1:05:43.300 --> 1:05:45.797
+we should predict.
+
+1:05:48.348 --> 1:05:56.373
+So and bird, the main ideas are that you're
+doing this bidirectional model with masking.
+
+1:05:56.373 --> 1:05:59.709
+It's using transformer architecture.
+
+1:06:00.320 --> 1:06:06.326
+There are two more minor changes.
+
+1:06:06.326 --> 1:06:16.573
+We'll see that this next word prediction is
+another task.
+
+1:06:16.957 --> 1:06:30.394
+You want to learn more about what language
+is to really understand following a story or
+
+1:06:30.394 --> 1:06:35.127
+their independent tokens into.
+
+1:06:38.158 --> 1:06:42.723
+The input is using word units as we use it.
+
+1:06:42.723 --> 1:06:50.193
+It has some special token that is framing
+for the next word prediction.
+
+1:06:50.470 --> 1:07:04.075
+It's more for classification task because
+you may be learning a general representation
+
+1:07:04.075 --> 1:07:07.203
+as a full sentence.
+
+1:07:07.607 --> 1:07:19.290
+You're doing segment embedding, so you have
+an embedding for it.
+
+1:07:19.290 --> 1:07:24.323
+This is the first sentence.
+
+1:07:24.684 --> 1:07:29.099
+Now what is more challenging is this masking.
+
+1:07:29.099 --> 1:07:30.827
+What do you mask?
+
+1:07:30.827 --> 1:07:35.050
+We already have the crush enough or should.
+
+1:07:35.275 --> 1:07:42.836
+So there has been afterwards eating some work
+like, for example, a bearer.
+
+1:07:42.836 --> 1:07:52.313
+It's not super sensitive, but if you do it
+completely wrong then you're not letting anything.
+
+1:07:52.572 --> 1:07:54.590
+That's Then Another Question There.
+
+1:07:56.756 --> 1:08:04.594
+Should I mask all types of should I always
+mask the footwork or if I have a subword to
+
+1:08:04.594 --> 1:08:10.630
+mask only like a subword and predict them based
+on the other ones?
+
+1:08:10.630 --> 1:08:14.504
+Of course, it's a bit of a different task.
+
+1:08:14.894 --> 1:08:21.210
+If you know three parts of the words, it might
+be easier to guess the last because they here
+
+1:08:21.210 --> 1:08:27.594
+took the easiest selection, so not considering
+words anymore at all because you're doing that
+
+1:08:27.594 --> 1:08:32.280
+in the preprocessing and just taking always
+words and like subwords.
+
+1:08:32.672 --> 1:08:36.089
+Think in group there is done differently.
+
+1:08:36.089 --> 1:08:40.401
+They mark always the full words, but guess
+it's not.
+
+1:08:41.001 --> 1:08:46.044
+And then what to do with the mask word in
+eighty percent of the cases.
+
+1:08:46.044 --> 1:08:50.803
+If the word is masked, they replace it with
+a special token thing.
+
+1:08:50.803 --> 1:08:57.197
+This is a mask token in ten percent they put
+in some random other token in there, and ten
+
+1:08:57.197 --> 1:08:59.470
+percent they keep it on change.
+
+1:09:02.202 --> 1:09:10.846
+And then what you can do is also this next
+word prediction.
+
+1:09:10.846 --> 1:09:14.880
+The man went to Mass Store.
+
+1:09:14.880 --> 1:09:17.761
+He bought a gallon.
+
+1:09:18.418 --> 1:09:24.088
+So may you see you're joining them, you're
+doing both masks and prediction that you're.
+
+1:09:24.564 --> 1:09:29.449
+Is a penguin mask or flyless birds.
+
+1:09:29.449 --> 1:09:41.390
+These two sentences have nothing to do with
+each other, so you can do also this type of
+
+1:09:41.390 --> 1:09:43.018
+prediction.
+
+1:09:47.127 --> 1:09:57.043
+And then the whole bird model, so here you
+have the input here to transform the layers,
+
+1:09:57.043 --> 1:09:58.170
+and then.
+
+1:09:58.598 --> 1:10:17.731
+And this model was quite successful in general
+applications.
+
+1:10:17.937 --> 1:10:27.644
+However, there is like a huge thing of different
+types of models coming from them.
+
+1:10:27.827 --> 1:10:38.709
+So based on others these supervised molds
+like a whole setup came out of there and now
+
+1:10:38.709 --> 1:10:42.086
+this is getting even more.
+
+1:10:42.082 --> 1:10:46.640
+With availability of a large language model
+than the success.
+
+1:10:47.007 --> 1:10:48.436
+We have now even larger ones.
+
+1:10:48.828 --> 1:10:50.961
+Interestingly, it goes a bit.
+
+1:10:50.910 --> 1:10:57.847
+Change the bit again from like more the spider
+action model to uni directional models.
+
+1:10:57.847 --> 1:11:02.710
+Are at the moment maybe a bit more we're coming
+to them now?
+
+1:11:02.710 --> 1:11:09.168
+Do you see one advantage while what is another
+event and we have the efficiency?
+
+1:11:09.509 --> 1:11:15.901
+Is one other reason why you are sometimes
+more interested in uni-direction models than
+
+1:11:15.901 --> 1:11:17.150
+in bi-direction.
+
+1:11:22.882 --> 1:11:30.220
+It depends on the pass, but for example for
+a language generation pass, the eccard is not
+
+1:11:30.220 --> 1:11:30.872
+really.
+
+1:11:32.192 --> 1:11:40.924
+It doesn't work so if you want to do a generation
+like the decoder you don't know the future
+
+1:11:40.924 --> 1:11:42.896
+so you cannot apply.
+
+1:11:43.223 --> 1:11:53.870
+So this time of model can be used for the
+encoder in an encoder model, but it cannot
+
+1:11:53.870 --> 1:11:57.002
+be used for the decoder.
+
+1:12:00.000 --> 1:12:05.012
+That's a good view to the next overall cast
+of models.
+
+1:12:05.012 --> 1:12:08.839
+Perhaps if you view it from the sequence.
+
+1:12:09.009 --> 1:12:12.761
+We have the encoder base model.
+
+1:12:12.761 --> 1:12:16.161
+That's what we just look at.
+
+1:12:16.161 --> 1:12:20.617
+They are bidirectional and typically.
+
+1:12:20.981 --> 1:12:22.347
+That Is the One We Looked At.
+
+1:12:22.742 --> 1:12:34.634
+At the beginning is the decoder based model,
+so see out in regressive models which are unidirective
+
+1:12:34.634 --> 1:12:42.601
+like an based model, and there we can do the
+next word prediction.
+
+1:12:43.403 --> 1:12:52.439
+And what you can also do first, and there
+you can also have a special things called prefix
+
+1:12:52.439 --> 1:12:53.432
+language.
+
+1:12:54.354 --> 1:13:05.039
+Because we are saying it might be helpful
+that some of your input can also use bi-direction.
+
+1:13:05.285 --> 1:13:12.240
+And that is somehow doing what it is called
+prefix length.
+
+1:13:12.240 --> 1:13:19.076
+On the first tokens you directly give your
+bidirectional.
+
+1:13:19.219 --> 1:13:28.774
+So you somehow merge that and that mainly
+works only in transformer based models because.
+
+1:13:29.629 --> 1:13:33.039
+There is no different number of parameters
+in our end.
+
+1:13:33.039 --> 1:13:34.836
+We need a back foot our end.
+
+1:13:34.975 --> 1:13:38.533
+Transformer: The only difference is how you
+mask your attention.
+
+1:13:38.878 --> 1:13:44.918
+We have seen that in the anchoder and decoder
+the number of parameters is different because
+
+1:13:44.918 --> 1:13:50.235
+you do cross attention, but if you do forward
+and backward or union directions,.
+
+1:13:50.650 --> 1:13:58.736
+It's only like you mask your attention to
+only look at the bad past or to look into the
+
+1:13:58.736 --> 1:13:59.471
+future.
+
+1:14:00.680 --> 1:14:03.326
+And now you can of course also do mixing.
+
+1:14:03.563 --> 1:14:08.306
+So this is a bi-directional attention matrix
+where you can attend to everything.
+
+1:14:08.588 --> 1:14:23.516
+There is a uni-direction or causal where you
+can look at the past and you can do the first
+
+1:14:23.516 --> 1:14:25.649
+three words.
+
+1:14:29.149 --> 1:14:42.831
+That somehow clear based on that, then of
+course you cannot do the other things.
+
+1:14:43.163 --> 1:14:50.623
+So the idea is we have our anchor to decoder
+architecture.
+
+1:14:50.623 --> 1:14:57.704
+Can we also train them completely in a side
+supervisor?
+
+1:14:58.238 --> 1:15:09.980
+And in this case we have the same input to
+both, so in this case we need to do some type
+
+1:15:09.980 --> 1:15:12.224
+of masking here.
+
+1:15:12.912 --> 1:15:17.696
+Here we don't need to do the masking, but
+here we need to masking that doesn't know ever
+
+1:15:17.696 --> 1:15:17.911
+so.
+
+1:15:20.440 --> 1:15:30.269
+And this type of model got quite successful
+also, especially for pre-training machine translation.
+
+1:15:30.330 --> 1:15:39.059
+The first model doing that is a Bart model,
+which exactly does that, and yes, it's one
+
+1:15:39.059 --> 1:15:42.872
+successful way to pre train your one.
+
+1:15:42.872 --> 1:15:47.087
+It's pretraining your full encoder model.
+
+1:15:47.427 --> 1:15:54.365
+Where you put in contrast to machine translation,
+where you put in source sentence, we can't
+
+1:15:54.365 --> 1:15:55.409
+do that here.
+
+1:15:55.715 --> 1:16:01.382
+But we can just put the second twice in there,
+and then it's not a trivial task.
+
+1:16:01.382 --> 1:16:02.432
+We can change.
+
+1:16:03.003 --> 1:16:12.777
+And there is like they do different corruption
+techniques so you can also do.
+
+1:16:13.233 --> 1:16:19.692
+That you couldn't do in an agricultural system
+because then it wouldn't be there and you cannot
+
+1:16:19.692 --> 1:16:20.970
+predict somewhere.
+
+1:16:20.970 --> 1:16:26.353
+So the anchor, the number of input and output
+tokens always has to be the same.
+
+1:16:26.906 --> 1:16:29.818
+You cannot do a prediction for something which
+isn't in it.
+
+1:16:30.110 --> 1:16:38.268
+Here in the decoder side it's unidirection
+so we can also delete the top and then try
+
+1:16:38.268 --> 1:16:40.355
+to generate the full.
+
+1:16:41.061 --> 1:16:45.250
+We can do sentence permutation.
+
+1:16:45.250 --> 1:16:54.285
+We can document rotation and text infilling
+so there is quite a bit.
+
+1:16:55.615 --> 1:17:06.568
+So you see there's quite a lot of types of
+models that you can use in order to pre-train.
+
+1:17:07.507 --> 1:17:14.985
+Then, of course, there is again for the language
+one.
+
+1:17:14.985 --> 1:17:21.079
+The other question is how do you integrate?
+
+1:17:21.761 --> 1:17:26.636
+And there's also, like yeah, quite some different
+ways of techniques.
+
+1:17:27.007 --> 1:17:28.684
+It's a Bit Similar to Before.
+
+1:17:28.928 --> 1:17:39.068
+So the easiest thing is you take your word
+embeddings or your free trained model.
+
+1:17:39.068 --> 1:17:47.971
+You freeze them and stack your decoder layers
+and keep these ones free.
+
+1:17:48.748 --> 1:17:54.495
+Can also be done if you have this type of
+bark model.
+
+1:17:54.495 --> 1:18:03.329
+What you can do is you freeze your word embeddings,
+for example some products and.
+
+1:18:05.865 --> 1:18:17.296
+The other thing is you initialize them so
+you initialize your models but you train everything
+
+1:18:17.296 --> 1:18:19.120
+so you're not.
+
+1:18:22.562 --> 1:18:29.986
+Then one thing, if you think about Bart, you
+want to have the Chinese language, the Italian
+
+1:18:29.986 --> 1:18:32.165
+language, and the deconer.
+
+1:18:32.165 --> 1:18:35.716
+However, in Bart we have the same language.
+
+1:18:36.516 --> 1:18:46.010
+The one you get is from English, so what you
+can do there is so you cannot try to do some.
+
+1:18:46.366 --> 1:18:52.562
+Below the barge, in order to learn some language
+specific stuff, or there's a masculine barge,
+
+1:18:52.562 --> 1:18:58.823
+which is trained on many languages, but it's
+trained only on like the Old Coast Modern Language
+
+1:18:58.823 --> 1:19:03.388
+House, which may be trained in German and English,
+but not on German.
+
+1:19:03.923 --> 1:19:08.779
+So then you would still need to find June
+and the model needs to learn how to better
+
+1:19:08.779 --> 1:19:10.721
+do the attention cross lingually.
+
+1:19:10.721 --> 1:19:15.748
+It's only on the same language but it mainly
+only has to learn this mapping and not all
+
+1:19:15.748 --> 1:19:18.775
+the rest and that's why it's still quite successful.
+
+1:19:21.982 --> 1:19:27.492
+Now certain thing which is very commonly used
+is what is required to it as adapters.
+
+1:19:27.607 --> 1:19:29.754
+So for example you take and buy.
+
+1:19:29.709 --> 1:19:35.218
+And you put some adapters on the inside of
+the networks so that it's small new layers
+
+1:19:35.218 --> 1:19:40.790
+which are in between put in there and then
+you only train these adapters or also train
+
+1:19:40.790 --> 1:19:41.815
+these adapters.
+
+1:19:41.815 --> 1:19:47.900
+For example, an embryo you could see that
+this learns to map the Sears language representation
+
+1:19:47.900 --> 1:19:50.334
+to the Tiger language representation.
+
+1:19:50.470 --> 1:19:52.395
+And then you don't have to change that luck.
+
+1:19:52.792 --> 1:19:59.793
+You give it extra ability to really perform
+well on that.
+
+1:19:59.793 --> 1:20:05.225
+These are quite small and so very efficient.
+
+1:20:05.905 --> 1:20:12.632
+That is also very commonly used, for example
+in modular systems where you have some adaptors
+
+1:20:12.632 --> 1:20:16.248
+in between here which might be language specific.
+
+1:20:16.916 --> 1:20:22.247
+So they are trained only for one language.
+
+1:20:22.247 --> 1:20:33.777
+The model has some or both and once has the
+ability to do multilingually to share knowledge.
+
+1:20:34.914 --> 1:20:39.058
+But there's one chance in general in the multilingual
+systems.
+
+1:20:39.058 --> 1:20:40.439
+It works quite well.
+
+1:20:40.439 --> 1:20:46.161
+There's one case or one specific use case
+for multilingual where this normally doesn't
+
+1:20:46.161 --> 1:20:47.344
+really work well.
+
+1:20:47.344 --> 1:20:49.975
+Do you have an idea what that could be?
+
+1:20:55.996 --> 1:20:57.536
+It's for Zero Shot Cases.
+
+1:20:57.998 --> 1:21:03.660
+Because having here some situation with this
+might be very language specific and zero shot,
+
+1:21:03.660 --> 1:21:09.015
+the idea is always to learn representations
+view which are more language dependent and
+
+1:21:09.015 --> 1:21:10.184
+with the adaptors.
+
+1:21:10.184 --> 1:21:15.601
+Of course you get in representations again
+which are more language specific and then it
+
+1:21:15.601 --> 1:21:17.078
+doesn't work that well.
+
+1:21:20.260 --> 1:21:37.730
+And there is also the idea of doing more knowledge
+pistolation.
+
+1:21:39.179 --> 1:21:42.923
+And now the idea is okay.
+
+1:21:42.923 --> 1:21:54.157
+We are training it the same, but what we want
+to achieve is that the encoder.
+
+1:21:54.414 --> 1:22:03.095
+So you should learn faster by trying to make
+these states as similar as possible.
+
+1:22:03.095 --> 1:22:11.777
+So you compare the first-hit state of the
+pre-trained model and try to make them.
+
+1:22:12.192 --> 1:22:18.144
+For example, by using the out two norms, so
+by just making these two representations the
+
+1:22:18.144 --> 1:22:26.373
+same: The same vocabulary: Why does it need
+the same vocabulary with any idea?
+
+1:22:34.754 --> 1:22:46.137
+If you have different vocabulary, it's typical
+you also have different sequenced lengths here.
+
+1:22:46.137 --> 1:22:50.690
+The number of sequences is different.
+
+1:22:51.231 --> 1:22:58.888
+If you now have pipe stains and four states
+here, it's no longer straightforward which
+
+1:22:58.888 --> 1:23:01.089
+states compare to which.
+
+1:23:02.322 --> 1:23:05.246
+And that's just easier if you have like the
+same number.
+
+1:23:05.246 --> 1:23:08.940
+You can always compare the first to the first
+and second to the second.
+
+1:23:09.709 --> 1:23:16.836
+So therefore at least the very easy way of
+knowledge destination only works if you have.
+
+1:23:17.177 --> 1:23:30.030
+Course: You could do things like yeah, the
+average should be the same, but of course there's
+
+1:23:30.030 --> 1:23:33.071
+a less strong signal.
+
+1:23:34.314 --> 1:23:42.979
+But the advantage here is that you have a
+diameter training signal here on the handquarter
+
+1:23:42.979 --> 1:23:51.455
+so you can directly make some of the encoder
+already giving a good signal while normally
+
+1:23:51.455 --> 1:23:52.407
+an empty.
+
+1:23:56.936 --> 1:24:13.197
+Yes, think this is most things for today,
+so what you should keep in mind is remind me.
+
+1:24:13.393 --> 1:24:18.400
+The one is a back translation idea.
+
+1:24:18.400 --> 1:24:29.561
+If you have monolingual and use that, the
+other one is to: And mentally it is often helpful
+
+1:24:29.561 --> 1:24:33.614
+to combine them so you can even use both of
+that.
+
+1:24:33.853 --> 1:24:38.908
+So you can use pre-trained walls, but then
+you can even still do back translation where
+
+1:24:38.908 --> 1:24:40.057
+it's still helpful.
+
+1:24:40.160 --> 1:24:45.502
+We have the advantage we are training like
+everything working together on the task so
+
+1:24:45.502 --> 1:24:51.093
+it might be helpful even to backtranslate some
+data and then use it in a real translation
+
+1:24:51.093 --> 1:24:56.683
+setup because in pretraining of course the
+beach challenge is always that you're training
+
+1:24:56.683 --> 1:24:57.739
+it on different.
+
+1:24:58.058 --> 1:25:03.327
+Different ways of how you integrate this knowledge.
+
+1:25:03.327 --> 1:25:08.089
+Even if you just use a full model, so in this.
+
+1:25:08.748 --> 1:25:11.128
+This is the most similar you can get.
+
+1:25:11.128 --> 1:25:13.945
+You're doing no changes to the architecture.
+
+1:25:13.945 --> 1:25:19.643
+You're really taking the model and just fine
+tuning them on the new task, but it still has
+
+1:25:19.643 --> 1:25:24.026
+to completely newly learn how to do the attention
+and how to do that.
+
+1:25:24.464 --> 1:25:29.971
+And that might be, for example, helpful to
+have more back-translated data to learn them.
+
+1:25:32.192 --> 1:25:34.251
+That's for today.
+
+1:25:34.251 --> 1:25:44.661
+There's one important thing that next Tuesday
+there is a conference or a workshop or so in
+
+1:25:44.661 --> 1:25:45.920
+this room.
+
+1:25:47.127 --> 1:25:56.769
+You should get an e-mail if you're in Elias
+that there's a room change for Tuesdays and
+
+1:25:56.769 --> 1:25:57.426
+it's.
+
+1:25:57.637 --> 1:26:03.890
+There are more questions, yeah, have a more
+general position, especially: In computer vision
+
+1:26:03.890 --> 1:26:07.347
+you can enlarge your data center data orientation.
+
+1:26:07.347 --> 1:26:08.295
+Is there any?
+
+1:26:08.388 --> 1:26:15.301
+It's similar to a large speech for text for
+the data of an edge.
+
+1:26:15.755 --> 1:26:29.176
+And you can use this back translation and
+also masking, but back translation is some
+
+1:26:29.176 --> 1:26:31.228
+way of data.
+
+1:26:31.371 --> 1:26:35.629
+So it has also been, for example, even its
+used not only for monolingual data.
+
+1:26:36.216 --> 1:26:54.060
+If you have good MP system, it can also be
+used for parallel data.
+
+1:26:54.834 --> 1:26:59.139
+So would say this is the most similar one.
+
+1:26:59.139 --> 1:27:03.143
+There's ways you can do power phrasing.
+
+1:27:05.025 --> 1:27:12.057
+But for example there is very hard to do this
+by rules like which words to replace because
+
+1:27:12.057 --> 1:27:18.936
+there is not a coup like you cannot always
+say this word can always be replaced by that.
+
+1:27:19.139 --> 1:27:27.225
+Mean, although they are many perfect synonyms,
+normally they are good in some cases, but not
+
+1:27:27.225 --> 1:27:29.399
+in all cases, and so on.
+
+1:27:29.399 --> 1:27:36.963
+And if you don't do a rule based, you have
+to train your model and then the freshness.
+
+1:27:38.058 --> 1:27:57.236
+The same architecture as the pre-trained mount.
+
+1:27:57.457 --> 1:27:59.810
+Should be of the same dimension, so it's easiest
+to have the same dimension.
+
+1:28:00.000 --> 1:28:01.590
+Architecture.
+
+1:28:01.590 --> 1:28:05.452
+We later will learn inefficiency.
+
+1:28:05.452 --> 1:28:12.948
+You can also do knowledge cessulation with,
+for example, smaller.
+
+1:28:12.948 --> 1:28:16.469
+You can learn the same within.
+
+1:28:17.477 --> 1:28:22.949
+Eight layers for it so that is possible, but
+yeah agree it should be of the same.
+
+1:28:23.623 --> 1:28:32.486
+Yeah yeah you need the question then of course
+you can do it like it's an initialization or
+
+1:28:32.486 --> 1:28:41.157
+you can do it doing training but normally it
+most makes sense during the normal training.
+
+1:28:45.865 --> 1:28:53.963
+Do it, then thanks a lot, and then we'll see
+each other again on Tuesday.
+
+0:00:00.981 --> 0:00:17.559
+What we want today about is how to use some
+type of additional resources to improve the
+
+0:00:17.559 --> 0:00:20.008
+translation.
+
+0:00:20.300 --> 0:00:31.387
+We have in the first part of the semester
+how to build some of your basic machine translation.
+
+0:00:31.571 --> 0:00:40.743
+You know now the basic components both for
+statistical and for neural, with the encoder
+
+0:00:40.743 --> 0:00:42.306
+decoder model.
+
+0:00:43.123 --> 0:00:45.950
+Now, of course, that's not where it stops.
+
+0:00:45.950 --> 0:00:51.340
+It's still what in nearly every machine translation
+system is currently in India.
+
+0:00:51.340 --> 0:00:57.323
+However, there is a lot of challenges which
+you need to address in addition and which need
+
+0:00:57.323 --> 0:00:58.243
+to be solved.
+
+0:00:58.918 --> 0:01:03.031
+We want to start with these parts.
+
+0:01:03.031 --> 0:01:07.614
+What else can you do around this part?
+
+0:01:07.614 --> 0:01:09.847
+You can be honest.
+
+0:01:10.030 --> 0:01:14.396
+And one important question there is on what
+do you train your models?
+
+0:01:14.394 --> 0:01:27.237
+Because this type of parallel data is easier
+in machine translation than many other tasks
+
+0:01:27.237 --> 0:01:33.516
+where you have a decent amount of training.
+
+0:01:33.853 --> 0:01:40.789
+And therefore an important question is: Can
+we also learn from other sources and improve
+
+0:01:40.789 --> 0:01:41.178
+our.
+
+0:01:41.701 --> 0:01:47.840
+Because if you remember from quite the beginning
+of the lecture,.
+
+0:01:51.171 --> 0:01:53.801
+This is how we train all our.
+
+0:01:54.194 --> 0:02:01.318
+Machine learning models, all the corpus bases
+from statistical to neural.
+
+0:02:01.318 --> 0:02:09.694
+This doesn't have change, so we need this
+type of parallel data where we have a source
+
+0:02:09.694 --> 0:02:13.449
+sentence aligned with the target data.
+
+0:02:13.493 --> 0:02:19.654
+We have now a strong model here, a very good
+model to do that.
+
+0:02:19.654 --> 0:02:22.099
+However, we always rely.
+
+0:02:22.522 --> 0:02:27.376
+More languages, higher resource languages,
+prayers that say from German to English or
+
+0:02:27.376 --> 0:02:31.327
+other European languages, there is a decent
+amount at least for some.
+
+0:02:31.471 --> 0:02:46.131
+But even there, if we're going to very specific
+domains, it might get difficult and then your
+
+0:02:46.131 --> 0:02:50.966
+system performance might drop.
+
+0:02:51.231 --> 0:02:55.261
+Phrases how to use the vocabulary, and so
+on, and the style.
+
+0:02:55.915 --> 0:03:04.104
+And if you're going to other languages, there
+is of course a lot bigger challenge.
+
+0:03:04.104 --> 0:03:05.584
+Why can't you?
+
+0:03:05.825 --> 0:03:09.647
+So is really this the only resource you can
+use.
+
+0:03:09.889 --> 0:03:20.667
+Or can we adapt our models in order to also
+make use of other types of models that might
+
+0:03:20.667 --> 0:03:27.328
+enable us to build strong systems with other
+types of.
+
+0:03:27.707 --> 0:03:35.283
+And that's what we will look into now in the
+next, starting from Tuesday in the next.
+
+0:03:35.515 --> 0:03:43.437
+So this idea we already have covered on Tuesday,
+so one very successful idea for this is to
+
+0:03:43.437 --> 0:03:45.331
+do more multilingual.
+
+0:03:45.645 --> 0:03:52.010
+So that we're no longer only doing translation
+between two languages, but we can do translation
+
+0:03:52.010 --> 0:03:55.922
+between many languages and share common knowledge
+between.
+
+0:03:56.296 --> 0:04:06.477
+And you also learned about that you can even
+do things like zero shot machine translations.
+
+0:04:06.786 --> 0:04:09.792
+Which is the case for many many language pairs.
+
+0:04:10.030 --> 0:04:17.406
+Even with German, you have not translation
+parallel data to all languages around the world,
+
+0:04:17.406 --> 0:04:22.698
+or most of them you have it to the Europeans,
+maybe for Japanese.
+
+0:04:22.698 --> 0:04:26.386
+But even for Japanese, it will get difficult.
+
+0:04:26.746 --> 0:04:32.862
+There is quite a lot of data, for example
+English to Japanese, but German to Vietnamese.
+
+0:04:32.862 --> 0:04:39.253
+There is some data from Multilingual Corpora
+where you can extract the name, but your amount
+
+0:04:39.253 --> 0:04:41.590
+really is dropping significantly.
+
+0:04:42.042 --> 0:04:54.907
+So that is a very promising direction if you
+want to build translation systems between language
+
+0:04:54.907 --> 0:05:00.134
+pairs, typically not English, because.
+
+0:05:01.221 --> 0:05:05.888
+And the other ideas, of course, we don't have
+data, just search for data.
+
+0:05:06.206 --> 0:05:15.755
+There is some work on data crawling so if
+don't have a corpus directly or don't have
+
+0:05:15.755 --> 0:05:23.956
+a high quality corpus from the European Parliament
+for TED corpus maybe.
+
+0:05:24.344 --> 0:05:35.528
+There has been a big effort in Europe to collect
+data sets for parallel data.
+
+0:05:35.528 --> 0:05:40.403
+How can we do this data crawling?
+
+0:05:40.600 --> 0:05:46.103
+There the interesting thing from the machine
+translation point is not just general data
+
+0:05:46.103 --> 0:05:46.729
+crawling.
+
+0:05:47.067 --> 0:05:52.067
+But how can we explicitly crawl data, which
+is somewhat parallel?
+
+0:05:52.132 --> 0:05:58.538
+So there is in the Internet quite a lot of
+data which has been like company websites which
+
+0:05:58.538 --> 0:06:01.565
+have been translated and things like that.
+
+0:06:01.565 --> 0:06:05.155
+So how can you extract them and then extract
+them?
+
+0:06:06.566 --> 0:06:13.406
+There is typically more noisy than where you
+do more, hence mean if you have your Parliament.
+
+0:06:13.693 --> 0:06:21.305
+You can do some rules how to extract the parallel
+things.
+
+0:06:21.305 --> 0:06:30.361
+Here there is more to it, so the quality is
+later maybe not as good.
+
+0:06:33.313 --> 0:06:39.927
+The other thing is can we use monolingual
+data and monolingual data has a big advantage
+
+0:06:39.927 --> 0:06:46.766
+that we can have a huge amount of that so that
+you can be able to crawl from the internet.
+
+0:06:46.766 --> 0:06:51.726
+The nice thing is you can also get it typically
+for many domains.
+
+0:06:52.352 --> 0:06:58.879
+There is just so much more magnitude more
+of monolingual data so that it might be very
+
+0:06:58.879 --> 0:06:59.554
+helpful.
+
+0:06:59.559 --> 0:07:06.187
+We can do that in statistical machine translation
+was quite easy to integrate using language
+
+0:07:06.187 --> 0:07:06.757
+models.
+
+0:07:08.508 --> 0:07:14.499
+In neural machine translation we have the
+advantage that we have this overall and architecture
+
+0:07:14.499 --> 0:07:18.850
+that does everything together, but it has also
+the disadvantage now.
+
+0:07:18.850 --> 0:07:22.885
+It's more difficult to put in this type of
+information or make.
+
+0:07:23.283 --> 0:07:26.427
+We'll look to two things.
+
+0:07:26.427 --> 0:07:37.432
+You can still try to do a bit of language
+modeling in there and add an additional language
+
+0:07:37.432 --> 0:07:38.279
+model.
+
+0:07:38.178 --> 0:07:43.771
+A way which I think is used in most systems
+at the moment is to do synthetic data.
+
+0:07:43.763 --> 0:07:53.095
+It's a very easy thing, but you can just translate
+there and then use it as training data.
+
+0:07:53.213 --> 0:07:59.192
+And thereby you are able to use like some
+type of moonlighting.
+
+0:08:00.380 --> 0:08:09.521
+Another way to do it is to ensure that some
+are in the extreme case.
+
+0:08:09.521 --> 0:08:14.026
+If you have a scenario that only.
+
+0:08:14.754 --> 0:08:24.081
+The impressive thing is if you have large
+amounts of data and the languages are not too
+
+0:08:24.081 --> 0:08:31.076
+dissimilar, you can even in this case build
+a translation system.
+
+0:08:32.512 --> 0:08:36.277
+That we will see then next Thursday.
+
+0:08:37.857 --> 0:08:55.462
+And then there is now a fourth type of restorer
+that recently became very successful and now.
+
+0:08:55.715 --> 0:09:02.409
+So the idea is we are no longer sharing the
+real data such as text data, but it can also
+
+0:09:02.409 --> 0:09:04.139
+help to train a model.
+
+0:09:04.364 --> 0:09:08.599
+And that is now a big advantage of deep learning
+based approaches.
+
+0:09:08.599 --> 0:09:14.414
+There you have this ability that you can train
+a model on some task and then you can modify
+
+0:09:14.414 --> 0:09:19.913
+it maybe and then apply it to another task
+and you can somewhat transfer the knowledge
+
+0:09:19.913 --> 0:09:22.125
+from the first task to the second.
+
+0:09:22.722 --> 0:09:31.906
+And then, of course, the question is, can
+it have an initial task where it's very easy
+
+0:09:31.906 --> 0:09:34.439
+to train on the second?
+
+0:09:34.714 --> 0:09:53.821
+The task that you pre-train on is more similar
+to a language.
+
+0:09:53.753 --> 0:10:06.293
+A bit of a different way of using language
+malls in this more transfer learning set.
+
+0:10:09.029 --> 0:10:18.747
+So first we will start with how can we use
+monolingual data to do a machine translation?
+
+0:10:20.040 --> 0:10:22.542
+The.
+
+0:10:22.062 --> 0:10:28.924
+This big difference is you should remember
+from what I mentioned before is in statistical
+
+0:10:28.924 --> 0:10:30.525
+machine translation.
+
+0:10:30.525 --> 0:10:33.118
+We directly have the opportunity.
+
+0:10:33.118 --> 0:10:39.675
+There's peril data for a translation model
+and monolingual data for a language model.
+
+0:10:39.679 --> 0:10:45.735
+And you combine your translation model and
+your language model, and then you can make.
+
+0:10:46.726 --> 0:10:54.263
+That has big advantages that you can make
+use of these large amounts of monolingual data,
+
+0:10:54.263 --> 0:10:55.519
+but of course.
+
+0:10:55.495 --> 0:11:02.198
+Because we said the problem is, we are optimizing
+both parts independently to each other, and
+
+0:11:02.198 --> 0:11:09.329
+we say the big advantage of newer machine translation
+is we are optimizing the overall architecture
+
+0:11:09.329 --> 0:11:10.541
+to perform best.
+
+0:11:10.890 --> 0:11:17.423
+And then, of course, we can't do that, so
+here we can only use power there.
+
+0:11:17.897 --> 0:11:25.567
+So the question is, but if this advantage
+is not so important, we can train everything,
+
+0:11:25.567 --> 0:11:33.499
+but we have large amounts of monolingual data
+or small amounts, but they fit perfectly, so
+
+0:11:33.499 --> 0:11:35.242
+they are very good.
+
+0:11:35.675 --> 0:11:41.438
+So in data we know it's not only important
+the amount of data we have but also like how
+
+0:11:41.438 --> 0:11:43.599
+similar it is to your test data.
+
+0:11:43.599 --> 0:11:49.230
+So it can be that this volume is even only
+quite small but it's very well fitting and
+
+0:11:49.230 --> 0:11:51.195
+then it's still very helpful.
+
+0:11:51.195 --> 0:11:55.320
+So the question is if this is the case how
+can we make use of?
+
+0:11:55.675 --> 0:12:03.171
+And the first year of surprisingness, if we
+are here successful with integrating a language
+
+0:12:03.171 --> 0:12:10.586
+model into a translation system, maybe we can
+also integrate some types of language models
+
+0:12:10.586 --> 0:12:14.415
+into our MT system in order to make it better.
+
+0:12:16.536 --> 0:12:19.000
+The first thing we can do is okay.
+
+0:12:19.000 --> 0:12:23.293
+We know there is language models, so let's
+try to integrate.
+
+0:12:23.623 --> 0:12:30.693
+There was mainly used language models because
+these works were mainly done before transformer
+
+0:12:30.693 --> 0:12:31.746
+based models.
+
+0:12:32.152 --> 0:12:41.567
+And generally, of course, you can do the same
+thing with all the Transformers baseballs.
+
+0:12:41.721 --> 0:12:58.900
+It has mainly been done before people started
+using R&S, and they tried to do this more
+
+0:12:58.900 --> 0:13:01.888
+in cases where.
+
+0:13:07.087 --> 0:13:17.508
+So what we're having here is some of this
+type of idea.
+
+0:13:17.508 --> 0:13:25.511
+This is a key system here as you remember.
+
+0:13:25.605 --> 0:13:29.470
+Gets in with your last instinct and calculates
+your attention.
+
+0:13:29.729 --> 0:13:36.614
+We get the context and combine both and then
+based on that and then predict the target.
+
+0:13:37.057 --> 0:13:42.423
+So this is our anti-system, and the question
+is, can we somehow integrate the language?
+
+0:13:42.782 --> 0:13:55.788
+And of course, if someone makes sense to take
+out a neural language model because we're anyway
+
+0:13:55.788 --> 0:14:01.538
+in the neural space, it's not surprising.
+
+0:14:01.621 --> 0:14:15.522
+And there would be something like on top of
+there and you're a language model and you have
+
+0:14:15.522 --> 0:14:17.049
+a target.
+
+0:14:17.597 --> 0:14:27.007
+So if we're having this type of language model,
+there's two main questions we have to answer.
+
+0:14:27.007 --> 0:14:28.108
+How do we?
+
+0:14:28.208 --> 0:14:37.935
+So how do we combine now on the one hand our
+NMT system and on the other hand our RNA you
+
+0:14:37.935 --> 0:14:45.393
+see that was mentioned before when we started
+talking about encoder.
+
+0:14:45.805 --> 0:14:49.523
+The wild is like unconditioned, it's just
+modeling the targets side.
+
+0:14:49.970 --> 0:14:57.183
+And the other one is a conditional language,
+which is a language condition on the sewer
+
+0:14:57.183 --> 0:14:57.839
+center.
+
+0:14:58.238 --> 0:15:03.144
+So the question is how can you not combine
+two language models?
+
+0:15:03.144 --> 0:15:09.813
+Of course, it's like the translation model
+will some will be more important because it
+
+0:15:09.813 --> 0:15:11.806
+has access to the source.
+
+0:15:11.806 --> 0:15:16.713
+We want to generate something which corresponds
+to your source.
+
+0:15:18.778 --> 0:15:20.918
+If we had that, the other question is OK.
+
+0:15:20.918 --> 0:15:22.141
+Now we have two models.
+
+0:15:22.141 --> 0:15:25.656
+If we even have integrated them, the answer
+is how do we train them?
+
+0:15:26.026 --> 0:15:39.212
+Because we have integrated them, we have no
+two sets of data with parallel data where you
+
+0:15:39.212 --> 0:15:42.729
+can do the lower thing.
+
+0:15:44.644 --> 0:15:47.575
+So the first idea is okay.
+
+0:15:47.575 --> 0:15:53.436
+We can do something more like a parallel combination.
+
+0:15:53.436 --> 0:15:55.824
+We just keep running.
+
+0:15:56.036 --> 0:15:59.854
+So a year you see your NMT system that is
+running.
+
+0:16:00.200 --> 0:16:08.182
+First of all, it's normally completely independent
+of your language model, which is up there.
+
+0:16:08.182 --> 0:16:13.278
+So down here we have just our NMT system,
+which is running.
+
+0:16:13.313 --> 0:16:26.439
+The only thing which is used is we have the
+words inputted, and of course they are put
+
+0:16:26.439 --> 0:16:28.099
+into both.
+
+0:16:28.099 --> 0:16:41.334
+We also put: So we use them in parallel, and
+then we are doing our decision just by merging
+
+0:16:41.334 --> 0:16:42.905
+these two.
+
+0:16:43.343 --> 0:16:52.288
+So there can be, for example, we are doing
+a probability distribution here, we are doing
+
+0:16:52.288 --> 0:17:01.032
+a purability distribution here, and then we
+are taking the average of both per ability
+
+0:17:01.032 --> 0:17:03.343
+to do our predictions.
+
+0:17:11.871 --> 0:17:18.929
+You could also take the output which seems
+to be more short about the answer.
+
+0:17:20.000 --> 0:17:23.272
+Yes, you could also do that.
+
+0:17:23.272 --> 0:17:27.222
+It's more like a gating mechanism.
+
+0:17:27.222 --> 0:17:32.865
+You're not doing everything, but you're focusing.
+
+0:17:32.993 --> 0:17:38.927
+Another one would be you could also just concatenate
+the hidden states and then you have another
+
+0:17:38.927 --> 0:17:41.802
+layer on top which based on the concatenation.
+
+0:17:43.303 --> 0:17:58.634
+If you think about it, you do the coordination
+instead of taking the instead and then merging
+
+0:17:58.634 --> 0:18:01.244
+the perability.
+
+0:18:03.143 --> 0:18:15.027
+Yes, in the end you introduce many new parameters
+and these parameters have somehow something
+
+0:18:15.027 --> 0:18:17.303
+special compared.
+
+0:18:23.603 --> 0:18:33.657
+So before all the other parameters can be
+trained independently of each other, the language
+
+0:18:33.657 --> 0:18:42.071
+one can be trained independent and an antisystem
+can be trained independent.
+
+0:18:43.043 --> 0:18:51.198
+If you have a joint layer of course you need
+to train them because you have inputs so you
+
+0:18:51.198 --> 0:19:01.560
+need: Not surprisingly, if you have a parallel
+combination or whether you could, the other
+
+0:19:01.560 --> 0:19:04.664
+way is to do more serial combinations.
+
+0:19:04.924 --> 0:19:10.382
+How can you do a similar combination?
+
+0:19:10.382 --> 0:19:18.281
+Your final decision makes sense to do it based
+on the.
+
+0:19:18.438 --> 0:19:20.997
+So you have on top of your normal an system.
+
+0:19:21.121 --> 0:19:30.826
+The only thing is now your inputting into
+your NIT system.
+
+0:19:30.826 --> 0:19:38.723
+You're no longer inputting the word embeddings.
+
+0:19:38.918 --> 0:19:47.819
+You're training the lower layers here which
+are trained more on the purely language model
+
+0:19:47.819 --> 0:19:55.434
+and on top you're putting into the NMT system
+where it now has the language.
+
+0:19:55.815 --> 0:19:59.003
+So here you can also view it here.
+
+0:19:59.003 --> 0:20:06.836
+You have more contextual embeddings which
+no longer depend on the word, but they also
+
+0:20:06.836 --> 0:20:10.661
+depend on the context of the target site.
+
+0:20:11.051 --> 0:20:21.797
+More understanding of the source word.
+
+0:20:21.881 --> 0:20:34.761
+So if it's like the word can, for example,
+will be put in here always the same, independent
+
+0:20:34.761 --> 0:20:41.060
+of its use of can of beans, or if can do it.
+
+0:20:41.701 --> 0:20:43.165
+Empties.
+
+0:20:44.364 --> 0:20:54.959
+So another view, if you're remembering more
+the transformer based approach, is you have
+
+0:20:54.959 --> 0:21:01.581
+some layers, and the lower layers are purely
+language.
+
+0:21:02.202 --> 0:21:08.052
+This is purely language model and then at
+some point you're starting to attend to the
+
+0:21:08.052 --> 0:21:08.596
+source.
+
+0:21:13.493 --> 0:21:20.774
+Yes, so these are two ways of how you combine
+it, so run them in peril, or first do the language.
+
+0:21:23.623 --> 0:21:26.147
+Questions for the integration.
+
+0:21:31.831 --> 0:21:35.034
+Not really sure about the input of the.
+
+0:21:35.475 --> 0:21:38.123
+And this case with a sequence.
+
+0:21:38.278 --> 0:21:50.721
+Is the input and bedding, the target word
+embedding, or the actual word, and then we
+
+0:21:50.721 --> 0:21:54.821
+transfer it to a numerical.
+
+0:21:56.176 --> 0:22:08.824
+That depends on if you view the word embedding
+as part of the language model, so of course
+
+0:22:08.824 --> 0:22:10.909
+you first put.
+
+0:22:11.691 --> 0:22:13.938
+And then the word embedding there is the r&n.
+
+0:22:14.314 --> 0:22:20.296
+So of course you can view this together as
+your language model when you first do the word
+
+0:22:20.296 --> 0:22:21.027
+embedding.
+
+0:22:21.401 --> 0:22:28.098
+All you can say are the RNAs and this is like
+before.
+
+0:22:28.098 --> 0:22:36.160
+It's more a definition, but you're right,
+so what are the steps?
+
+0:22:36.516 --> 0:22:46.655
+One of these parts, you know, called a language
+model is definitionally not that important,
+
+0:22:46.655 --> 0:22:47.978
+but that's.
+
+0:22:53.933 --> 0:23:02.812
+So the question is how can you then train
+them and make make this this one work?
+
+0:23:03.363 --> 0:23:15.492
+So in the case where you combine the language
+of our abilities you can train them independently
+
+0:23:15.492 --> 0:23:18.524
+and then just put them.
+
+0:23:18.918 --> 0:23:29.623
+It might not be the best because we have no
+longer this ability before that.
+
+0:23:29.623 --> 0:23:33.932
+They optimal perform together.
+
+0:23:34.514 --> 0:23:41.050
+At least you need to summarize how much do
+you trust the one model and how much do you
+
+0:23:41.050 --> 0:23:41.576
+trust.
+
+0:23:43.323 --> 0:23:48.529
+But still in some cases usually it might be
+helpful if you have only data and so on.
+
+0:23:48.928 --> 0:24:06.397
+However, we have one specific situation that
+leads to the pearl leader is always mono legal
+
+0:24:06.397 --> 0:24:07.537
+data.
+
+0:24:08.588 --> 0:24:17.693
+So what we can also do is more the pre-training
+approach.
+
+0:24:17.693 --> 0:24:24.601
+We first train the language model and then.
+
+0:24:24.704 --> 0:24:33.468
+So the pre-training approach you first train
+on the monolingual data and then you join the.
+
+0:24:33.933 --> 0:24:45.077
+Of course, the model size is this way, but
+the data size is of course too big.
+
+0:24:45.077 --> 0:24:52.413
+You often have more monolingual data than
+parallel.
+
+0:24:56.536 --> 0:24:57.901
+Any ideas.
+
+0:25:04.064 --> 0:25:10.108
+Had one example where this might also be helpful
+if you want to adapt to a domain so let's say
+
+0:25:10.108 --> 0:25:16.281
+you do medical sentences and if you want to
+translate medical sentences and you have monolingual
+
+0:25:16.281 --> 0:25:22.007
+data on the target side for medical sentences
+but you only have parallel data for general
+
+0:25:22.007 --> 0:25:22.325
+use.
+
+0:25:23.083 --> 0:25:30.601
+In this case it could be, or it's the most
+probable happen if you're learning out there
+
+0:25:30.601 --> 0:25:38.804
+what medical means, but then in your fine tuning
+step the model is forgetting everything about.
+
+0:25:39.099 --> 0:25:42.340
+So this type of priest training step is good.
+
+0:25:42.340 --> 0:25:47.978
+If your pretraining data is more general,
+very large, and then you're adapting.
+
+0:25:48.428 --> 0:25:55.545
+But in the task we have monolingual data,
+which should be used to adapt the system to
+
+0:25:55.545 --> 0:25:57.780
+some genre of topic style.
+
+0:25:57.817 --> 0:26:08.572
+Then, of course, this is not a good strategy
+because you might forget about everything up
+
+0:26:08.572 --> 0:26:09.408
+there.
+
+0:26:09.649 --> 0:26:17.494
+So then you have to check what you can do
+for them to see.
+
+0:26:17.494 --> 0:26:25.738
+You can freeze this part and you can do a
+direct combination.
+
+0:26:25.945 --> 0:26:33.796
+Where you train both of them, and then you
+train the language more and parallel on their
+
+0:26:33.796 --> 0:26:34.942
+one so that.
+
+0:26:35.395 --> 0:26:37.687
+Eh What You Learn in the Length.
+
+0:26:37.937 --> 0:26:48.116
+So the bit depends on what you want to combine
+is that you use a language model because it's.
+
+0:26:48.548 --> 0:26:56.380
+Then you normally don't really forget it because
+it's also in the or you use it to adapt to
+
+0:26:56.380 --> 0:26:58.083
+something specific.
+
+0:27:01.001 --> 0:27:06.662
+Then there is so this is a way of how we can
+make use of monolingual data.
+
+0:27:07.968 --> 0:27:11.787
+It seems to be the easiest one somehow.
+
+0:27:11.787 --> 0:27:19.140
+It's more similar to what we are doing with
+statistical machine translation.
+
+0:27:19.140 --> 0:27:20.095
+However,.
+
+0:27:21.181 --> 0:27:27.211
+Normally always beats this type of model,
+which in some view can be from the conceptual
+
+0:27:27.211 --> 0:27:27.691
+thing.
+
+0:27:27.691 --> 0:27:31.460
+At least it's even easier from the computational
+side.
+
+0:27:31.460 --> 0:27:36.805
+Sometimes it has a disadvantage that it's
+more problematic or more difficult.
+
+0:27:40.560 --> 0:27:42.576
+And the idea is okay.
+
+0:27:42.576 --> 0:27:45.141
+We have a monolingual data.
+
+0:27:45.141 --> 0:27:50.822
+We just translate it and then generate some
+type of parallel.
+
+0:27:51.111 --> 0:28:00.465
+So if you want to build a German to English
+system, your first trained German to English
+
+0:28:00.465 --> 0:28:02.147
+system on your.
+
+0:28:02.402 --> 0:28:05.217
+Then you have more pearl data.
+
+0:28:05.217 --> 0:28:13.482
+The interesting thing is if you then train
+on the joint thing, on the original pearl data,
+
+0:28:13.482 --> 0:28:18.749
+and on that one is artificial, it even normally
+improves.
+
+0:28:18.918 --> 0:28:26.490
+You can because you're not doing the same
+error all the time and you have some knowledge.
+
+0:28:28.028 --> 0:28:40.080
+With this first approach, however, there's
+one issue: why it might not work the best,
+
+0:28:40.080 --> 0:28:43.163
+so could you imagine?
+
+0:28:49.409 --> 0:28:51.186
+Ready a bit shown in image two.
+
+0:28:53.113 --> 0:29:00.637
+Have a few trains on bad quality data.
+
+0:29:00.637 --> 0:29:08.741
+The system will learn also in the states.
+
+0:29:08.828 --> 0:29:12.210
+And as you're saying, it's a system always
+mistranslates.
+
+0:29:13.493 --> 0:29:14.497
+Something.
+
+0:29:14.497 --> 0:29:23.623
+Then you will learn that this is correct because
+now it's training data and you will even encourage
+
+0:29:23.623 --> 0:29:25.996
+it to make it more often.
+
+0:29:25.996 --> 0:29:29.921
+So the problem on training on your own is.
+
+0:29:30.150 --> 0:29:34.222
+But however, as you systematically do, you
+even enforce more and will even do more.
+
+0:29:34.654 --> 0:29:37.401
+So that might not be the best solution.
+
+0:29:37.401 --> 0:29:40.148
+Do any idea how you could do it better?
+
+0:29:44.404 --> 0:29:57.653
+If you had something else to prevent some
+systematic problems, yes, that is one way.
+
+0:30:04.624 --> 0:30:10.809
+The problem is yeah, the translations are
+not perfect, so the output and you're learning
+
+0:30:10.809 --> 0:30:11.990
+something wrong.
+
+0:30:11.990 --> 0:30:17.967
+Normally it's less bad if your inputs are
+somewhat bad, but your outputs are perfect.
+
+0:30:18.538 --> 0:30:26.670
+So if your inputs are wrong you maybe learn
+that if you're doing this wrong input you're
+
+0:30:26.670 --> 0:30:30.782
+generating something correct but you're not.
+
+0:30:31.511 --> 0:30:40.911
+So often the case is that it's more important
+that your target is correct.
+
+0:30:40.911 --> 0:30:47.052
+If on the source there is something crazy,
+then.
+
+0:30:47.347 --> 0:30:52.184
+But you can assume in your application scenario
+you hope that you mainly get correct input.
+
+0:30:52.572 --> 0:31:02.126
+So that is not harming you as much, and in
+machine translation we have some of these symmetries,
+
+0:31:02.126 --> 0:31:02.520
+so.
+
+0:31:02.762 --> 0:31:04.578
+And also the other way around.
+
+0:31:04.578 --> 0:31:09.792
+It's a very similar task, so there's a task
+to translate from German to English, but the
+
+0:31:09.792 --> 0:31:13.892
+task to translate from English to German is
+very similar and helpful.
+
+0:31:14.094 --> 0:31:19.313
+So what we can do is, we can just switch it
+initially and generate the data the other way
+
+0:31:19.313 --> 0:31:19.777
+around.
+
+0:31:20.120 --> 0:31:25.699
+So what we are doing here is we are starting
+with an English to German system.
+
+0:31:25.699 --> 0:31:32.126
+Then we are translating the English data into
+German, where the German is maybe not really
+
+0:31:32.126 --> 0:31:32.903
+very nice.
+
+0:31:33.293 --> 0:31:46.045
+And then we're training on our original data
+and on the back translated data where only
+
+0:31:46.045 --> 0:31:51.696
+the input is good and it's like human.
+
+0:31:52.632 --> 0:32:01.622
+So here we have now the advantage that always
+our target site is of human quality and the
+
+0:32:01.622 --> 0:32:02.322
+input.
+
+0:32:03.583 --> 0:32:08.998
+And then this helps us to get really good
+form.
+
+0:32:08.998 --> 0:32:15.428
+There's one important difference if you think
+about the.
+
+0:32:21.341 --> 0:32:31.604
+It's too obvious here we need a target side
+monolingual layer and the first.
+
+0:32:31.931 --> 0:32:47.143
+So back translation is normally working if
+you have target size parallel and not search
+
+0:32:47.143 --> 0:32:48.180
+side.
+
+0:32:48.448 --> 0:32:55.493
+Might be also a bit if you think about it
+understandable that it's more important to
+
+0:32:55.493 --> 0:32:56.819
+be like better.
+
+0:32:57.117 --> 0:33:04.472
+On the suicide you have to understand the
+content, on the target side you have to generate
+
+0:33:04.472 --> 0:33:12.232
+really sentences and somehow it's more difficult
+to generate something than to only understand.
+
+0:33:17.617 --> 0:33:29.916
+One other thing, so typically it's shown here
+differently, but typically it's like this works
+
+0:33:29.916 --> 0:33:30.701
+well.
+
+0:33:31.051 --> 0:33:32.978
+Because normally there's like a lot more.
+
+0:33:33.253 --> 0:33:36.683
+So the question is, should really take all
+of my data?
+
+0:33:36.683 --> 0:33:38.554
+There's two problems with it.
+
+0:33:38.554 --> 0:33:42.981
+Of course, it's expensive because you have
+to translate all this data.
+
+0:33:42.981 --> 0:33:48.407
+And secondly, if you had, although now your
+packet site is wrong, it might be that you
+
+0:33:48.407 --> 0:33:51.213
+still have your wrong correlations in there.
+
+0:33:51.651 --> 0:34:01.061
+So if you don't know the normally good starting
+point is to take equal amount of data as many
+
+0:34:01.061 --> 0:34:02.662
+backtranslated.
+
+0:34:02.963 --> 0:34:05.366
+Of course, it depends on the use case.
+
+0:34:05.366 --> 0:34:07.215
+There are very few data here.
+
+0:34:07.215 --> 0:34:08.510
+It makes more sense.
+
+0:34:08.688 --> 0:34:14.273
+It depends on how good your quality is here,
+so the better the model is observable, the
+
+0:34:14.273 --> 0:34:17.510
+more data you might use because quality is
+better.
+
+0:34:17.510 --> 0:34:23.158
+So it depends on a lot of things, but yeah,
+a rule of sample like good general way often
+
+0:34:23.158 --> 0:34:24.808
+is to have equal amounts.
+
+0:34:26.646 --> 0:34:31.233
+And you can of course do that now iteratively.
+
+0:34:31.233 --> 0:34:39.039
+It said already that the quality at the end,
+of course, depends on this system.
+
+0:34:39.039 --> 0:34:46.163
+Also, because the better this system is, the
+better your synthetic data.
+
+0:34:47.207 --> 0:34:50.949
+That leads to what is referred to as iterated
+back translation.
+
+0:34:51.291 --> 0:34:56.911
+So you're playing a model on English to German
+and you translate the data.
+
+0:34:56.957 --> 0:35:03.397
+Then you train a model on German to English
+with the additional data.
+
+0:35:03.397 --> 0:35:11.954
+Then you translate German when you translate
+German data and then you train again your first
+
+0:35:11.954 --> 0:35:12.414
+one.
+
+0:35:12.414 --> 0:35:14.346
+So you iterate that.
+
+0:35:14.334 --> 0:35:19.653
+Because now your system is better because
+it's not only trained on the small data but
+
+0:35:19.653 --> 0:35:22.003
+additionally on back translated data.
+
+0:35:22.442 --> 0:35:24.458
+And so you can get better.
+
+0:35:24.764 --> 0:35:31.739
+However, typically you can stop quite early,
+so maybe one iteration is good, but then you
+
+0:35:31.739 --> 0:35:35.072
+have diminishing gains after two or three.
+
+0:35:35.935 --> 0:35:44.094
+There's very slight difference and then yeah
+because you need of course quite big difference
+
+0:35:44.094 --> 0:35:45.937
+in the quality here.
+
+0:35:45.937 --> 0:35:46.814
+In order.
+
+0:35:47.207 --> 0:35:59.810
+Which is not too good because it means you
+can already have to train it with relatively
+
+0:35:59.810 --> 0:36:02.245
+bad performance.
+
+0:36:03.723 --> 0:36:10.323
+And they don't yeah, a design decision would
+advise so guess because it's easy to get it.
+
+0:36:10.550 --> 0:36:16.617
+Better to replace that because you have a
+higher quality, but you of course keep your
+
+0:36:16.617 --> 0:36:18.310
+high quality real data.
+
+0:36:18.310 --> 0:36:21.626
+Then I think normally it's okay to replace
+it.
+
+0:36:21.626 --> 0:36:24.518
+Of course you can also try to append it.
+
+0:36:24.518 --> 0:36:28.398
+I would assume it's not too much of a difference,
+but.
+
+0:36:34.414 --> 0:36:40.567
+That's about like using monolingual data before
+we go into the pre-train models.
+
+0:36:40.567 --> 0:36:42.998
+Do you have any more questions?
+
+0:36:49.029 --> 0:36:57.521
+Yes, so the other thing we can do and which
+is recently more and more successful and even
+
+0:36:57.521 --> 0:37:05.731
+more successful since we have these really
+large language models where you can even do
+
+0:37:05.731 --> 0:37:08.562
+a translation task with this.
+
+0:37:08.688 --> 0:37:16.132
+So here the idea is you learn a representation
+of one task and then you use this representation.
+
+0:37:16.576 --> 0:37:27.276
+It was made maybe like one of the first where
+it's really used largely is doing something
+
+0:37:27.276 --> 0:37:35.954
+like a bird which you pre-train on purely text
+editor and then you take.
+
+0:37:36.496 --> 0:37:42.952
+And the one big advantage, of course, is that
+people can only share data but also pre-train.
+
+0:37:43.423 --> 0:37:53.247
+So if you think of the recent models and the
+large language models which are available,
+
+0:37:53.247 --> 0:37:59.611
+it is not possible for universities often to
+train them.
+
+0:37:59.919 --> 0:38:09.413
+Think it costs several millions to train the
+model just if you rent the GPS from some cloud
+
+0:38:09.413 --> 0:38:15.398
+company and train that the cost of training
+these models.
+
+0:38:15.475 --> 0:38:21.735
+And guess as a student project you won't have
+the budget to like build these models.
+
+0:38:21.801 --> 0:38:24.630
+So another idea is what you can do is okay.
+
+0:38:24.630 --> 0:38:27.331
+Maybe if these months are once available.
+
+0:38:27.467 --> 0:38:34.723
+You can take them and use them as a resource
+similar to pure text, and you can now build
+
+0:38:34.723 --> 0:38:41.734
+models which some will learn not only from
+from data but also from other models which
+
+0:38:41.734 --> 0:38:44.506
+are maybe trained on other tasks.
+
+0:38:44.844 --> 0:38:48.647
+So it's a quite new way of thinking of how
+to train.
+
+0:38:48.647 --> 0:38:53.885
+So we are not only learning from examples,
+but we might also learn from.
+
+0:38:54.534 --> 0:39:03.937
+The nice thing is that this type of training
+where we are not learning directly from data
+
+0:39:03.937 --> 0:39:07.071
+by learning from other tasks.
+
+0:39:07.427 --> 0:39:15.581
+So the main idea to start with is to have
+a personal initial task, and typically this
+
+0:39:15.581 --> 0:39:24.425
+initial task is for: And if you're working
+with, that means you're training pure taxator
+
+0:39:24.425 --> 0:39:30.547
+because you have the largest amount of data
+from the Internet.
+
+0:39:30.951 --> 0:39:35.857
+And then you're defining some type of task
+in order to do your quick training.
+
+0:39:36.176 --> 0:39:42.056
+And: There's a typical task you can train
+on.
+
+0:39:42.056 --> 0:39:52.709
+That is like the language modeling text, so
+to predict the next word, all we have related.
+
+0:39:52.932 --> 0:40:04.654
+But to predict something which you have not
+in the input is a task which is easy to generate.
+
+0:40:04.654 --> 0:40:06.150
+That's why.
+
+0:40:06.366 --> 0:40:14.005
+By yourself, on the other hand, you need a
+lot of knowledge, and that is the other thing
+
+0:40:14.005 --> 0:40:15.120
+you need to.
+
+0:40:15.735 --> 0:40:23.690
+Because there is this idea that the meaning
+of the word heavily depends on the context
+
+0:40:23.690 --> 0:40:24.695
+it's used.
+
+0:40:25.145 --> 0:40:36.087
+So can give you a sentence with some gibberish
+word and there's some name, and although you've
+
+0:40:36.087 --> 0:40:41.616
+never read the name, you will just assume that.
+
+0:40:42.062 --> 0:40:48.290
+Exactly the same thing, the models can also
+learn something about the words in there by
+
+0:40:48.290 --> 0:40:49.139
+just using.
+
+0:40:49.649 --> 0:40:53.246
+So that is typically the new.
+
+0:40:53.246 --> 0:40:59.839
+Then we can use this model, use our data to
+train the.
+
+0:41:00.800 --> 0:41:04.703
+Of course, it might need to adapt the system.
+
+0:41:04.703 --> 0:41:07.672
+To do that we might use only some.
+
+0:41:07.627 --> 0:41:16.326
+Part of the pre-train model in there is that
+we have seen that a bit already in the RNA
+
+0:41:16.326 --> 0:41:17.215
+case is.
+
+0:41:17.437 --> 0:41:22.670
+So you can view the RN as one of these approaches.
+
+0:41:22.670 --> 0:41:28.518
+You train the RN language while on large pre-train
+data.
+
+0:41:28.518 --> 0:41:32.314
+Then you put it somewhere into your.
+
+0:41:33.653 --> 0:41:37.415
+So this gives you the ability to really do
+these types of tests.
+
+0:41:37.877 --> 0:41:49.027
+So that you can build a system which uses
+knowledge, which is just trained on large amounts
+
+0:41:49.027 --> 0:41:52.299
+of data and extracting it.
+
+0:41:52.299 --> 0:41:53.874
+So it knows.
+
+0:41:56.376 --> 0:42:01.561
+So the question is that yeah, what type of
+information so what type of models can you?
+
+0:42:01.821 --> 0:42:05.278
+And we want to today look at briefly at three.
+
+0:42:05.725 --> 0:42:08.474
+Was initially done.
+
+0:42:08.474 --> 0:42:21.118
+It wasn't as famous as in machine translation
+as in other things, but it's also used there.
+
+0:42:21.221 --> 0:42:28.974
+So where you have this mapping from the one
+hot to a small continuous word representation?
+
+0:42:29.229 --> 0:42:37.891
+Using this one in your anthrax you can, for
+example, replace the embedding layer by the
+
+0:42:37.891 --> 0:42:38.776
+trained.
+
+0:42:39.139 --> 0:42:41.832
+That is helpful to be a really small amount
+of data.
+
+0:42:42.922 --> 0:42:48.520
+You're always in this pre training phase and
+have the thing the advantage is.
+
+0:42:48.468 --> 0:42:55.515
+More data, that's the trade off so you can
+get better.
+
+0:42:55.515 --> 0:43:00.128
+Disadvantage is, does anybody have?
+
+0:43:04.624 --> 0:43:12.173
+Was one of the mentioned today, even like
+big advantages of the system compared to previous.
+
+0:43:20.660 --> 0:43:26.781
+Where one advantage was the end to end training
+so that all parameters and all components are
+
+0:43:26.781 --> 0:43:27.952
+optimal together.
+
+0:43:28.208 --> 0:43:33.386
+If you know pre-train something on one pass,
+it's maybe no longer optimal fitting to everything.
+
+0:43:33.893 --> 0:43:40.338
+So that is similar to what should do pretaining
+or not.
+
+0:43:40.338 --> 0:43:48.163
+It depends on how important everything is
+optimal together and how.
+
+0:43:48.388 --> 0:44:00.552
+If the state is a high quality of large amount,
+the pre trained one is just so much better.
+
+0:44:00.600 --> 0:44:11.215
+Standing everything optimal together, we would
+use random actions for amazing vices.
+
+0:44:11.691 --> 0:44:18.791
+Mean, we assume some structures that are trained
+basically.
+
+0:44:18.791 --> 0:44:26.364
+Yes, if you're fine tuning everything, it
+might be the problem.
+
+0:44:26.766 --> 0:44:31.139
+But often yeah, in some way right, so often
+it's not about.
+
+0:44:31.139 --> 0:44:37.624
+You're really worse with some pre-trained
+molecules because you're going already in some
+
+0:44:37.624 --> 0:44:43.236
+direction, and if this is not really optimal
+for you, it might be difficult.
+
+0:44:43.603 --> 0:44:51.774
+But the bigger is, if you're not getting better
+because you have a decent amount of data, it's
+
+0:44:51.774 --> 0:44:52.978
+so different.
+
+0:44:53.153 --> 0:45:04.884
+But mean initially it wasn't a machine translation
+done so much because there was more data in
+
+0:45:04.884 --> 0:45:09.452
+the task, but now it's really large.
+
+0:45:12.632 --> 0:45:14.188
+The other one is then OK.
+
+0:45:14.188 --> 0:45:18.258
+Now it's always like how much of the model
+do your pre-track a bit?
+
+0:45:18.658 --> 0:45:25.057
+The other one you can do is tack contextual
+words and then something like bird or a robota
+
+0:45:25.057 --> 0:45:31.667
+where you train more already as sequence models
+and the embeddings you're using are no longer
+
+0:45:31.667 --> 0:45:35.605
+specific for words but they're also taking
+the context.
+
+0:45:35.875 --> 0:45:54.425
+Embedding you're using is no longer only depending
+on the word itself but on the whole sentence.
+
+0:45:55.415 --> 0:46:03.714
+And of course you can use similar things also
+in the decoder just by having layers which
+
+0:46:03.714 --> 0:46:09.122
+don't have access to the source but there it's
+still not.
+
+0:46:11.451 --> 0:46:19.044
+And finally, and then we'll look at the end,
+you can also have models which are already.
+
+0:46:19.419 --> 0:46:28.605
+So you may be training a sequence model, but
+not a monolingual data.
+
+0:46:28.605 --> 0:46:35.128
+Of course you have to make it a bit challenging.
+
+0:46:36.156 --> 0:46:43.445
+But the idea is really you're pre-training
+your whole model and then you're fine tuning.
+
+0:46:47.227 --> 0:46:59.487
+But let's first do a bit of step back and
+look into what are the differences.
+
+0:46:59.487 --> 0:47:02.159
+The first thing.
+
+0:47:02.382 --> 0:47:06.870
+The word embeddings are just this first layer.
+
+0:47:06.870 --> 0:47:12.027
+You can train them with feed-forward neural
+networks.
+
+0:47:12.212 --> 0:47:25.683
+But you can also train them in language model,
+and by now you hopefully have also seen that
+
+0:47:25.683 --> 0:47:27.733
+you can also.
+
+0:47:30.130 --> 0:47:41.558
+So this is how you can train them, and you
+are training them to predict the next word,
+
+0:47:41.558 --> 0:47:45.236
+the typical language model.
+
+0:47:45.525 --> 0:47:52.494
+And that is what is now referred to as a South
+Supervised Learning, and for example all the
+
+0:47:52.494 --> 0:47:56.357
+big large language models like Chat, gp and
+so on.
+
+0:47:56.357 --> 0:48:03.098
+They are trained at an end or feet, but exactly
+with this objective to predict the next.
+
+0:48:03.823 --> 0:48:12.847
+So that is where you can hopefully learn what
+a word is used because you always try to predict
+
+0:48:12.847 --> 0:48:17.692
+the next word and then you have a ready intuition.
+
+0:48:19.619 --> 0:48:25.374
+In the word embedding, why do people first
+look at the word embeddings and the use of
+
+0:48:25.374 --> 0:48:27.582
+word embeddings for other tasks?
+
+0:48:27.582 --> 0:48:32.600
+The main advantage is it might be only the
+first layer you would think of.
+
+0:48:32.600 --> 0:48:34.474
+What does it really matter?
+
+0:48:34.474 --> 0:48:39.426
+However, it is the layer where you typically
+have most of the parameters.
+
+0:48:39.879 --> 0:48:52.201
+Of course, if you have trained on most of
+your parameters already on the large data,
+
+0:48:52.201 --> 0:48:59.304
+then on your target data you have to train
+less.
+
+0:48:59.259 --> 0:49:05.841
+This big difference that your input size is
+so much bigger than the size of the normal
+
+0:49:05.841 --> 0:49:06.522
+in size.
+
+0:49:06.626 --> 0:49:16.551
+So it's a normal size, maybe two hundred and
+fifty, but your input embedding besides vocabulary
+
+0:49:16.551 --> 0:49:20.583
+size is something like fifty thousand.
+
+0:49:23.123 --> 0:49:30.163
+And bending while here you see, it's only
+like times as much in the layer.
+
+0:49:30.750 --> 0:49:36.747
+So here's where most of your parameters are,
+which means if you already replace the word
+
+0:49:36.747 --> 0:49:41.329
+embeddings, it might look a bit small in your
+overall architecture.
+
+0:49:41.329 --> 0:49:47.056
+It's where most of the things are, and if
+you're doing that, you already have really
+
+0:49:47.056 --> 0:49:48.876
+big games and can do that.
+
+0:49:57.637 --> 0:50:04.301
+The thing is we have seen these wooden beddings
+can be very good used for other taps.
+
+0:50:04.784 --> 0:50:08.921
+Now you learn some relation between words.
+
+0:50:08.921 --> 0:50:14.790
+If you're doing this type of language modeling,
+you predict.
+
+0:50:15.215 --> 0:50:21.532
+The one thing is, of course, you have a lot
+of data, so the one question is we want to
+
+0:50:21.532 --> 0:50:25.961
+have a lot of data to good training models,
+the other thing.
+
+0:50:25.961 --> 0:50:28.721
+The tasks need to be somewhat useful.
+
+0:50:29.169 --> 0:50:41.905
+If you would predict the first letter of the
+word, it has to be a task where you need some
+
+0:50:41.905 --> 0:50:45.124
+syntactic information.
+
+0:50:45.545 --> 0:50:53.066
+The interesting thing is people have looked
+at these world embeddings here in a language
+
+0:50:53.066 --> 0:50:53.658
+model.
+
+0:50:53.954 --> 0:51:04.224
+And you're looking at the word embeddings,
+which are these vectors here.
+
+0:51:04.224 --> 0:51:09.289
+You can ask yourself, do they look?
+
+0:51:09.489 --> 0:51:15.122
+Don't know if your view is listening to artificial
+advance artificial intelligence.
+
+0:51:15.515 --> 0:51:23.994
+We had on yesterday how to do this type of
+representation, but you can do this kind of
+
+0:51:23.994 --> 0:51:29.646
+representation, and now you're seeing interesting
+things.
+
+0:51:30.810 --> 0:51:41.248
+Now you can represent it here in a three dimensional
+space with a dimension reduction.
+
+0:51:41.248 --> 0:51:46.886
+Then you can look into it and the interesting.
+
+0:51:47.447 --> 0:51:57.539
+So this vector between the male and the female
+version of something is not the same, but it's
+
+0:51:57.539 --> 0:51:58.505
+related.
+
+0:51:58.718 --> 0:52:11.256
+So you can do a bit of nuts, you subtract
+this vector, add this vector, and then you
+
+0:52:11.256 --> 0:52:14.501
+look around this one.
+
+0:52:14.894 --> 0:52:19.691
+So that means okay, there is really something
+stored, some information stored in that book.
+
+0:52:20.040 --> 0:52:25.003
+Similar you can do it with Buck and since
+you see here swimming slam walk and walk.
+
+0:52:25.265 --> 0:52:42.534
+So again these vectors are not the same, but
+they're related for going from here to here.
+
+0:52:43.623 --> 0:52:47.508
+Are semantically the relations between city
+and capital?
+
+0:52:47.508 --> 0:52:49.757
+You have exactly the same thing.
+
+0:52:51.191 --> 0:52:57.857
+People having done question answering about
+that if they show these embeddings and.
+
+0:52:58.218 --> 0:53:05.198
+Or you can also, if you don't trust the the
+dimensional reduction because you say maybe
+
+0:53:05.198 --> 0:53:06.705
+there's something.
+
+0:53:06.967 --> 0:53:16.473
+Done you can also look into what happens really
+in the indimensional space.
+
+0:53:16.473 --> 0:53:22.227
+You can look at what is the nearest neighbor.
+
+0:53:22.482 --> 0:53:29.605
+So you can take the relationship between France
+and Paris and add it to Italy and nicely see.
+
+0:53:30.010 --> 0:53:33.082
+You can do big and bigger and you have small
+and small lines.
+
+0:53:33.593 --> 0:53:38.202
+It doesn't work everywhere.
+
+0:53:38.202 --> 0:53:49.393
+There are also some which sometimes work,
+so if you have a typical.
+
+0:53:51.491 --> 0:53:56.832
+You can do what the person is doing for famous
+ones.
+
+0:53:56.832 --> 0:54:05.800
+Of course, only like Einstein, scientist,
+that Messier finds Midfield are not completely
+
+0:54:05.800 --> 0:54:06.707
+correct.
+
+0:54:06.846 --> 0:54:09.781
+You'll see the examples are a bit old.
+
+0:54:09.781 --> 0:54:15.050
+The politicians are no longer there, but the
+first one doesn't learn.
+
+0:54:16.957 --> 0:54:29.003
+What people have done there of courses, especially
+at the beginning.
+
+0:54:29.309 --> 0:54:36.272
+So one famous model was, but we're not really
+interested in the language model performance.
+
+0:54:36.272 --> 0:54:38.013
+We're only interested.
+
+0:54:38.338 --> 0:54:40.634
+Think something good to keep in mind.
+
+0:54:40.634 --> 0:54:42.688
+What are we really interested in?
+
+0:54:42.688 --> 0:54:44.681
+Do we really want to have an RN?
+
+0:54:44.681 --> 0:54:44.923
+No.
+
+0:54:44.923 --> 0:54:48.608
+In this case we are only interested in this
+type of mapping.
+
+0:54:49.169 --> 0:54:55.536
+And so very successful was this word to beg.
+
+0:54:55.535 --> 0:55:02.597
+We are not training real language when making
+it even simpler and doing this for example
+
+0:55:02.597 --> 0:55:04.660
+continuous back of words.
+
+0:55:04.660 --> 0:55:11.801
+We are just having four input tokens and we
+are predicting what is the word in the middle
+
+0:55:11.801 --> 0:55:15.054
+and this is just like two linear layers.
+
+0:55:15.615 --> 0:55:22.019
+It's even simplifying things and making the
+calculation faster because that is what we're
+
+0:55:22.019 --> 0:55:22.873
+interested.
+
+0:55:23.263 --> 0:55:34.059
+All this continues skip ground models of these
+other two models.
+
+0:55:34.234 --> 0:55:38.273
+You have one equal word and it's the other
+way around.
+
+0:55:38.273 --> 0:55:41.651
+You're predicting the four words around them.
+
+0:55:41.651 --> 0:55:43.047
+It's very similar.
+
+0:55:43.047 --> 0:55:48.702
+The task is in the end very similar, but in
+all of them it's about learning.
+
+0:55:51.131 --> 0:56:01.416
+Before we go into the next part, let's talk
+about the normal white vector or white line.
+
+0:56:04.564 --> 0:56:07.562
+The next thing is contextual word embeddings.
+
+0:56:07.562 --> 0:56:08.670
+The idea is yes.
+
+0:56:08.670 --> 0:56:09.778
+This is helpful.
+
+0:56:09.778 --> 0:56:14.080
+However, we might be able to get more from
+just only lingo later.
+
+0:56:14.080 --> 0:56:19.164
+For example, if you think about the word can,
+it can have different meanings.
+
+0:56:19.419 --> 0:56:32.619
+And now in the word embeddings how you have
+an overlap of these two meanings, so it represents
+
+0:56:32.619 --> 0:56:33.592
+those.
+
+0:56:34.834 --> 0:56:40.318
+But we might be able to in the pre-train model
+already disambiguate these because they use
+
+0:56:40.318 --> 0:56:41.041
+completely.
+
+0:56:41.701 --> 0:56:50.998
+So if we can have a model which can not only
+represent the word, but it can also represent
+
+0:56:50.998 --> 0:56:58.660
+the meaning of the word within the context,
+it might be even more helpful.
+
+0:56:59.139 --> 0:57:03.342
+So then we're going to contextual word embeddings.
+
+0:57:03.342 --> 0:57:07.709
+We're really having a representation of the
+context.
+
+0:57:07.787 --> 0:57:11.519
+And we have a very good architecture for that
+already.
+
+0:57:11.691 --> 0:57:20.551
+It's like our base language model where you
+have to do the hidden state.
+
+0:57:20.551 --> 0:57:29.290
+The hidden state represents what is apparently
+said, but it's focusing.
+
+0:57:29.509 --> 0:57:43.814
+The first one doing that is in something like
+the Elmo paper where they instead of like this
+
+0:57:43.814 --> 0:57:48.121
+is a normal language model.
+
+0:57:48.008 --> 0:57:52.735
+Put in the third predicting the fourth and
+so on, so you're always predicting the next
+
+0:57:52.735 --> 0:57:53.007
+one.
+
+0:57:53.193 --> 0:57:57.919
+The architecture of the heaven works embedding
+layer, and then two are an layer here.
+
+0:57:57.919 --> 0:58:04.255
+For example: And now instead of using this
+one in the end you're using here this one.
+
+0:58:04.364 --> 0:58:11.245
+This represents the meaning of this word mainly
+in the context of what we have seen before.
+
+0:58:11.871 --> 0:58:22.909
+We can train it in a language model or predicting
+the next word, but we have more information,
+
+0:58:22.909 --> 0:58:26.162
+train there, and therefore.
+
+0:58:27.167 --> 0:58:31.168
+And there is one even done currently in.
+
+0:58:31.168 --> 0:58:40.536
+The only difference is that we have more layers,
+bigger size, and we're using transform on here
+
+0:58:40.536 --> 0:58:44.634
+or self-attention instead of the R&F.
+
+0:58:44.634 --> 0:58:45.122
+But.
+
+0:58:46.746 --> 0:58:52.737
+However, if you look at this contextual representation,
+they might not be perfect.
+
+0:58:52.737 --> 0:58:58.584
+So what do you think of this one as contextual
+representation of the third word?
+
+0:58:58.584 --> 0:59:02.914
+Do you see anything which is not really considered
+in this?
+
+0:59:07.587 --> 0:59:11.492
+Only one way yes, so that is not a big issue
+here.
+
+0:59:11.492 --> 0:59:18.154
+It's representing a string in the context
+of a sentence, however, only in the context.
+
+0:59:18.558 --> 0:59:28.394
+However, we have an architecture which can
+also take both sides and we have used it in
+
+0:59:28.394 --> 0:59:30.203
+the ink holder.
+
+0:59:30.630 --> 0:59:34.269
+So we could do the and easily only us in the
+backboard direction.
+
+0:59:34.874 --> 0:59:46.889
+By just having the other way around, and then
+we couldn't combine the forward and into a
+
+0:59:46.889 --> 0:59:49.184
+joint one where.
+
+0:59:49.329 --> 0:59:50.861
+So You Have a Word embedding.
+
+0:59:51.011 --> 1:00:03.910
+Then you have two states, one with a forward,
+and then one with a backward.
+
+1:00:03.910 --> 1:00:10.359
+For example, take the representation.
+
+1:00:10.490 --> 1:00:21.903
+Now this same here represents mainly this
+word because this is where what both focuses
+
+1:00:21.903 --> 1:00:30.561
+on is what is happening last but is also looking
+at the previous.
+
+1:00:31.731 --> 1:00:41.063
+However, there is a bit different when training
+that as a language model you already have.
+
+1:00:43.203 --> 1:00:44.956
+Maybe there's again this masking.
+
+1:00:46.546 --> 1:00:47.814
+That is one solution.
+
+1:00:47.814 --> 1:00:53.407
+First of all, why we can't do it is the information
+you leave it, so you cannot just predict the
+
+1:00:53.407 --> 1:00:54.041
+next word.
+
+1:00:54.041 --> 1:00:58.135
+If we just predict the next word in this type
+of model, that's a very.
+
+1:00:58.738 --> 1:01:04.590
+You know the next word because it's influencing
+this hidden stage and then it's very easy so
+
+1:01:04.590 --> 1:01:07.736
+predicting something you know is not a good
+task.
+
+1:01:07.736 --> 1:01:09.812
+This is what I mentioned before.
+
+1:01:09.812 --> 1:01:13.336
+You have to define somehow a task which is
+challenging.
+
+1:01:13.753 --> 1:01:19.007
+Because in this case one would, I mean, the
+system would just ignore the states and what
+
+1:01:19.007 --> 1:01:22.961
+it would learn is that you copy this information
+directly in here.
+
+1:01:23.343 --> 1:01:31.462
+So it would mainly be representing this word
+and you would have a perfect model because
+
+1:01:31.462 --> 1:01:38.290
+you only need to find an encoding where you
+can encode all words somehow.
+
+1:01:38.458 --> 1:01:44.046
+The only thing that will learn is that tenor
+and coat all my words in this upper hidden.
+
+1:01:44.985 --> 1:01:49.584
+And then, of course, it's not really useful.
+
+1:01:49.584 --> 1:01:53.775
+We need to find a bit of different ways.
+
+1:01:55.295 --> 1:01:59.440
+There is a masking one.
+
+1:01:59.440 --> 1:02:06.003
+I'll come to that shortly just a bit.
+
+1:02:06.003 --> 1:02:14.466
+The other thing is not to directly combine
+them.
+
+1:02:14.594 --> 1:02:22.276
+So you never merge the states only at the
+end.
+
+1:02:22.276 --> 1:02:33.717
+The representation of the words is now from
+the forward and the next.
+
+1:02:33.873 --> 1:02:35.964
+So it's always a hidden state before that.
+
+1:02:36.696 --> 1:02:41.273
+And these two you're joined now to your to
+the representation.
+
+1:02:42.022 --> 1:02:50.730
+And then you have now a representation also
+about the whole sentence for the word, but
+
+1:02:50.730 --> 1:02:53.933
+there's no information leakage.
+
+1:02:53.933 --> 1:02:59.839
+One way of doing this is instead of doing
+a bidirectional.
+
+1:03:00.380 --> 1:03:08.079
+You can do that, of course, in all layers.
+
+1:03:08.079 --> 1:03:16.315
+In the end you have different bedding states.
+
+1:03:16.596 --> 1:03:20.246
+However, it's a bit of a complicated.
+
+1:03:20.246 --> 1:03:25.241
+You have to keep up separate and then merge
+things.
+
+1:03:27.968 --> 1:03:33.007
+And that is is the moment where, like the,
+the peak.
+
+1:03:34.894 --> 1:03:42.018
+Idea of the big success of the bird model
+was used, maybe in bidirector case.
+
+1:03:42.018 --> 1:03:48.319
+It's not good to do the next word prediction,
+but we can do masking.
+
+1:03:48.308 --> 1:03:59.618
+And masking maybe means we do a prediction
+of something in the middle or some words.
+
+1:03:59.618 --> 1:04:08.000
+If we have the input, we're just putting noise
+into the input.
+
+1:04:08.048 --> 1:04:14.040
+Now there can be no information leakage because
+this wasn't in the input.
+
+1:04:14.040 --> 1:04:15.336
+Now predicting.
+
+1:04:16.776 --> 1:04:20.524
+So thereby we don't do any assumption again
+about our models.
+
+1:04:20.524 --> 1:04:24.815
+It doesn't need to be a forward model or a
+backward model or anything.
+
+1:04:24.815 --> 1:04:29.469
+You can have any type of architecture and
+you can always predict the street.
+
+1:04:30.530 --> 1:04:39.112
+There is maybe one disadvantage: do you see
+what could be a bit of a problem this type
+
+1:04:39.112 --> 1:04:40.098
+compared?
+
+1:05:00.000 --> 1:05:05.920
+Yes, so yeah mean you cannot cross mass more,
+but to see it more globally just twist assume
+
+1:05:05.920 --> 1:05:07.142
+you only mask one.
+
+1:05:07.142 --> 1:05:12.676
+For the whole sentence we get one feedback
+signal like what is the word street, so we
+
+1:05:12.676 --> 1:05:16.280
+have one training sample, a model for the whole
+center.
+
+1:05:17.397 --> 1:05:19.461
+The language modeling paste.
+
+1:05:19.461 --> 1:05:21.240
+We predicted here three.
+
+1:05:21.240 --> 1:05:22.947
+We predicted here four.
+
+1:05:22.947 --> 1:05:24.655
+We predicted here five.
+
+1:05:25.005 --> 1:05:26.973
+So we have a number of tokens.
+
+1:05:26.973 --> 1:05:30.974
+For each token we have a feet bed and saying
+what is the best.
+
+1:05:31.211 --> 1:05:39.369
+So in this case of course this is a lot less
+efficient because we are getting less feedback
+
+1:05:39.369 --> 1:05:45.754
+signals on what we should predict compared
+to models where we're doing.
+
+1:05:48.348 --> 1:05:54.847
+So in birth the main idea this bidirectional
+model was masking.
+
+1:05:54.847 --> 1:05:59.721
+It was the first large model using transformer.
+
+1:06:00.320 --> 1:06:06.326
+There are two more minor changes.
+
+1:06:06.326 --> 1:06:16.573
+We'll see that this next word prediction is
+another task.
+
+1:06:16.957 --> 1:06:25.395
+Again you want to learn more about what language
+is to really understand.
+
+1:06:25.395 --> 1:06:35.089
+Are these two sentences like following a story
+or they're independent of each other?
+
+1:06:38.158 --> 1:06:43.026
+The input is using subword units as we're
+using it and we're using it.
+
+1:06:43.026 --> 1:06:48.992
+It has some special token, the beginning,
+the CLS token that is straining for the next
+
+1:06:48.992 --> 1:06:50.158
+word prediction.
+
+1:06:50.470 --> 1:06:57.296
+It's more for machine translation.
+
+1:06:57.296 --> 1:07:07.242
+It's more for classification tasks because
+you're.
+
+1:07:07.607 --> 1:07:24.323
+You have two sentences, and then you have
+a position of encoding as we know them in general.
+
+1:07:24.684 --> 1:07:28.812
+Now what is more challenging is masking.
+
+1:07:28.812 --> 1:07:30.927
+So what do you mask?
+
+1:07:30.927 --> 1:07:35.055
+We already have to question like should.
+
+1:07:35.275 --> 1:07:44.453
+So there has been afterwards eating some work
+like, for example, Urbana, which tries to improve.
+
+1:07:44.453 --> 1:07:52.306
+It's not super sensitive, but of course if
+you do it completely wrong then you're.
+
+1:07:52.572 --> 1:07:54.590
+That's then another question there.
+
+1:07:56.756 --> 1:08:03.285
+All types should always mask the poor word.
+
+1:08:03.285 --> 1:08:14.562
+If have a subword, it's good to mask only
+like a subword and predict based.
+
+1:08:14.894 --> 1:08:20.755
+You know, like three parts of the words, it
+might be easier to get the last because they
+
+1:08:20.755 --> 1:08:27.142
+here took the easiest selections, not considering
+words anymore at all because you're doing that
+
+1:08:27.142 --> 1:08:32.278
+in the pre-processing and just taking always
+words like subwords and masking.
+
+1:08:32.672 --> 1:08:36.286
+Their thinking will bear them differently.
+
+1:08:36.286 --> 1:08:40.404
+They mark always the full words, but guess
+it's.
+
+1:08:41.001 --> 1:08:46.969
+And then what to do with the mask work in
+eighty percent of the cases is the word is
+
+1:08:46.969 --> 1:08:47.391
+mask.
+
+1:08:47.391 --> 1:08:50.481
+They replace it with a special token thing.
+
+1:08:50.481 --> 1:08:52.166
+This is the mask token.
+
+1:08:52.166 --> 1:08:58.486
+In ten percent they put in some random other
+token in there, and in ten percent they keep
+
+1:08:58.486 --> 1:08:59.469
+it unchanged.
+
+1:09:02.202 --> 1:09:11.519
+And then what you can do is also this next
+prediction.
+
+1:09:11.519 --> 1:09:17.786
+So if you have the man went to mass.
+
+1:09:18.418 --> 1:09:24.090
+So may you see you're joining that you're
+doing both masks and next prediction that.
+
+1:09:24.564 --> 1:09:34.402
+And if the sentence is pinguine masks are
+flyless birds, then these two sentences have
+
+1:09:34.402 --> 1:09:42.995
+nothing to do with each other, and so in this
+case it's not the next token.
+
+1:09:47.127 --> 1:09:56.184
+And that is the whole bird model, so here
+is the input, here the transformable layers,
+
+1:09:56.184 --> 1:09:58.162
+and you can train.
+
+1:09:58.598 --> 1:10:08.580
+And this model was quite successful in general
+applications.
+
+1:10:08.580 --> 1:10:17.581
+It was not as successful as people are nowadays
+using.
+
+1:10:17.937 --> 1:10:27.644
+However, there is like a huge thing of different
+types of models coming from that.
+
+1:10:27.827 --> 1:10:39.109
+So based on bird and other semi-supervised
+models like a whole setup came out of there
+
+1:10:39.109 --> 1:10:42.091
+and there's different.
+
+1:10:42.082 --> 1:10:46.637
+With the availability of large languages more
+than the success.
+
+1:10:47.007 --> 1:10:48.436
+We have now even larger ones.
+
+1:10:48.828 --> 1:10:50.961
+Interestingly, it goes a bit.
+
+1:10:50.910 --> 1:10:59.321
+Change the bit again from like more this spider
+action model to unidirectional models, or at
+
+1:10:59.321 --> 1:11:03.843
+the moment maybe a bit more we're coming to
+them.
+
+1:11:03.843 --> 1:11:09.179
+Now do you see one advantage,, and we have
+the efficiency.
+
+1:11:09.509 --> 1:11:16.670
+There's one other reason why you sometimes
+are more interested in unidirectional models
+
+1:11:16.670 --> 1:11:17.158
+than.
+
+1:11:22.882 --> 1:11:30.882
+Mean it depends on the task, but for example
+for a language generation task, the task.
+
+1:11:32.192 --> 1:11:34.574
+It's not only interesting, it doesn't work.
+
+1:11:34.574 --> 1:11:39.283
+So if you want to do a generation like the
+decoder so you want to generate a sentence,
+
+1:11:39.283 --> 1:11:42.856
+you don't know the future so you cannot apply
+this type of model.
+
+1:11:43.223 --> 1:11:49.498
+This time off model can be used for the encoder
+in an encoder model but cannot be used for
+
+1:11:49.498 --> 1:11:55.497
+the decoder because it is trained that only
+works and it has information on both sides
+
+1:11:55.497 --> 1:11:56.945
+and if you're doing.
+
+1:12:00.000 --> 1:12:05.559
+Yeah, that's a good view to the next overall
+task of models.
+
+1:12:05.559 --> 1:12:08.839
+We have so if you view it from the.
+
+1:12:09.009 --> 1:12:13.137
+Of you we have the encoder baseball.
+
+1:12:13.137 --> 1:12:16.372
+That's what we just look at.
+
+1:12:16.372 --> 1:12:20.612
+They are bidirectional and typically.
+
+1:12:20.981 --> 1:12:22.347
+That is the one we looked at.
+
+1:12:22.742 --> 1:12:35.217
+At the beginning is the decoder-based model,
+so the outer-regressive mounts which are unit
+
+1:12:35.217 --> 1:12:42.619
+based model, and there we can do the next prediction.
+
+1:12:43.403 --> 1:12:52.421
+And what you can also do first, and there
+you can also have special things called prefix
+
+1:12:52.421 --> 1:12:53.434
+language.
+
+1:12:54.354 --> 1:13:04.079
+Because we are saying it might be helpful
+that some of your inputs you can use by direction
+
+1:13:04.079 --> 1:13:17.334
+because: That is what is called a prefix where
+you say on the first tokens you have bidirectional
+
+1:13:17.334 --> 1:13:19.094
+connections.
+
+1:13:19.219 --> 1:13:28.768
+You somehow merge that mainly works only in
+transformer based models because the uni direction.
+
+1:13:29.629 --> 1:13:34.894
+There is no different number of parameters.
+
+1:13:34.975 --> 1:13:38.533
+Transformer: The only difference is how you
+mask your attention.
+
+1:13:38.878 --> 1:13:47.691
+We have seen that in the encoder, in the decoder,
+the number of parameters is different because
+
+1:13:47.691 --> 1:13:50.261
+you do the cross-attention.
+
+1:13:50.650 --> 1:13:58.389
+It's only like you mask your attention to
+only look at the bad past or also look into
+
+1:13:58.389 --> 1:13:59.469
+the future.
+
+1:14:00.680 --> 1:14:03.323
+And now you can, of course, also do mixing.
+
+1:14:03.563 --> 1:14:08.307
+So this is a bidirectional attention metric
+where you can attend to everything.
+
+1:14:08.588 --> 1:14:23.477
+That is a unidirection or causal where you
+can only look at the past and you can do this
+
+1:14:23.477 --> 1:14:25.652
+with prefix.
+
+1:14:29.149 --> 1:14:42.829
+Some are all clear based on that, then of
+course you can also do the other thing.
+
+1:14:43.163 --> 1:14:54.497
+So the idea is we have our encoder, decoder
+architecture, can we also train them completely
+
+1:14:54.497 --> 1:14:57.700
+in a side supervised way?
+
+1:14:58.238 --> 1:15:06.206
+In this case we have the same input to both,
+so in this case we would have the sentence
+
+1:15:06.206 --> 1:15:08.470
+as input in the decoder.
+
+1:15:08.470 --> 1:15:12.182
+Then we need to do some type of masking.
+
+1:15:12.912 --> 1:15:16.245
+Here we don't need to do the masking, but
+here we need to do.
+
+1:15:16.245 --> 1:15:17.911
+The masking doesn't know ever.
+
+1:15:20.440 --> 1:15:30.269
+And this type of model got quite successful
+also, especially for pre-training machine translation.
+
+1:15:30.330 --> 1:15:45.934
+This is the first model of the BART model,
+which is one successful way to pre-train your
+
+1:15:45.934 --> 1:15:47.162
+model.
+
+1:15:47.427 --> 1:15:52.858
+Where you put in source sentence, we can't
+do that here.
+
+1:15:52.858 --> 1:15:55.430
+We only have one language.
+
+1:15:55.715 --> 1:16:00.932
+But we can just put this twice in there, and
+that is not a trivial task.
+
+1:16:00.932 --> 1:16:08.517
+We can change it in: They do quite a bit of
+different corruption techniques.
+
+1:16:08.517 --> 1:16:12.751
+You can do token masking and you can also.
+
+1:16:13.233 --> 1:16:20.785
+That you couldn't do and go the only system
+because then it wouldn't be there if you cannot
+
+1:16:20.785 --> 1:16:22.345
+predict somewhere.
+
+1:16:22.345 --> 1:16:26.368
+So the number of input and output tokens always.
+
+1:16:26.906 --> 1:16:29.820
+You cannot do a prediction for something which
+isn't it?
+
+1:16:30.110 --> 1:16:39.714
+Here in the decoder side it's uni-direction
+so we can also delete and then generate the
+
+1:16:39.714 --> 1:16:40.369
+full.
+
+1:16:41.061 --> 1:16:48.628
+We can do sentence per rotation where you
+change the sentence.
+
+1:16:48.628 --> 1:16:54.274
+We can document rotation and text and filling.
+
+1:16:55.615 --> 1:17:05.870
+So you see there's quite a lot of types of
+models that you can use in order to pre-train
+
+1:17:05.870 --> 1:17:06.561
+your.
+
+1:17:07.507 --> 1:17:12.512
+And these are the models you can use.
+
+1:17:12.512 --> 1:17:21.072
+Of course, the other question is how do you
+integrate them into?
+
+1:17:21.761 --> 1:17:26.638
+And there's also like yeah quite some different
+ways of techniques.
+
+1:17:27.007 --> 1:17:28.684
+It's a Bit Similar to Before.
+
+1:17:28.928 --> 1:17:39.307
+So the easiest thing is you take your word
+embeddings or your pre-train model.
+
+1:17:39.307 --> 1:17:47.979
+If you're contextual embedding several layers
+you freeze them in.
+
+1:17:48.748 --> 1:17:53.978
+Can also be done if you have a bark model.
+
+1:17:53.978 --> 1:18:03.344
+You freeze your wooden beddings, for example,
+and only train the top layers.
+
+1:18:05.865 --> 1:18:14.965
+The other thing is you initialize them so
+you initialize your models but then you train
+
+1:18:14.965 --> 1:18:19.102
+everything so you're not only training.
+
+1:18:22.562 --> 1:18:32.600
+When you have then one thing, if you think
+about Bart, there's one thing, so you want
+
+1:18:32.600 --> 1:18:35.752
+to have the same language.
+
+1:18:36.516 --> 1:18:46.013
+Typically mean the one you get is from English,
+so you can not try to do some language.
+
+1:18:46.366 --> 1:18:55.165
+Below the barge, in order to learn some language
+specific stuff or there's a multilingual barge
+
+1:18:55.165 --> 1:19:03.415
+which is trained on many languages, it's trained
+only on like it's more or less language.
+
+1:19:03.923 --> 1:19:09.745
+So then you would still need to find June
+and the model needs to learn how to better
+
+1:19:09.745 --> 1:19:12.074
+do the attention cross lingually.
+
+1:19:12.074 --> 1:19:18.102
+It's only on the same language but it mainly
+only has to learn this mapping and not all
+
+1:19:18.102 --> 1:19:18.787
+the rest.
+
+1:19:21.982 --> 1:19:27.492
+A third thing which is is very commonly used
+is what is frequent to it as adapters.
+
+1:19:27.607 --> 1:19:29.749
+So, for example, you take and bark.
+
+1:19:29.709 --> 1:19:35.502
+And you put some adapters on the inside of
+the network so that it's small new layers which
+
+1:19:35.502 --> 1:19:41.676
+are in between put in there and then you only
+train these adapters or also train these adapters.
+
+1:19:41.676 --> 1:19:47.724
+So for example in Embry you could see that
+this learns to map the Seus language representation
+
+1:19:47.724 --> 1:19:50.333
+to the targeted language representation.
+
+1:19:50.470 --> 1:19:52.395
+And then you don't have to change that luck.
+
+1:19:52.792 --> 1:20:04.197
+Ideas that you give it some extra ability
+to really perform well on that, and then it's
+
+1:20:04.197 --> 1:20:05.234
+easier.
+
+1:20:05.905 --> 1:20:15.117
+Is also very commonly used, for example, in
+multilingual systems where the idea is you
+
+1:20:15.117 --> 1:20:16.282
+have some.
+
+1:20:16.916 --> 1:20:23.505
+So they are trained only for one language
+pair, so the model has some of those it once
+
+1:20:23.505 --> 1:20:27.973
+has the abilities to do multilingually to share
+knowledge.
+
+1:20:27.973 --> 1:20:33.729
+But then there is some knowledge which is
+very language specific, and then.
+
+1:20:34.914 --> 1:20:39.291
+But there's one chance in general, the multilingual
+systems.
+
+1:20:39.291 --> 1:20:40.798
+It works quite well.
+
+1:20:40.798 --> 1:20:47.542
+There's one specific use case for multilingual,
+where this normally doesn't really work well.
+
+1:20:47.542 --> 1:20:49.981
+Do you have an idea of what that?
+
+1:20:55.996 --> 1:20:57.534
+It's for Zero Short Cases.
+
+1:20:57.998 --> 1:21:06.051
+Because then you're having to hear some situation
+which might be very language specific again
+
+1:21:06.051 --> 1:21:15.046
+in zero shot, the idea is always to learn representations
+via which are more language dependent and with
+
+1:21:15.046 --> 1:21:17.102
+the adaptors of course.
+
+1:21:20.260 --> 1:21:37.655
+And there's also the idea of doing more like
+a knowledge ventilation setup, so in this.
+
+1:21:39.179 --> 1:21:41.177
+And now the idea is okay.
+
+1:21:41.177 --> 1:21:48.095
+We are training it the same, but what we want
+to achieve is that the hidden stages of the
+
+1:21:48.095 --> 1:21:54.090
+encoder are as similar to the one as the pre-train
+model, just as additional.
+
+1:21:54.414 --> 1:22:07.569
+So you should learn faster by telling the
+model to make these states as similar as possible.
+
+1:22:07.569 --> 1:22:11.813
+You compare the first hidden.
+
+1:22:12.192 --> 1:22:18.549
+For example, by using the L2 norm, so by just
+making these two representations the same.
+
+1:22:20.020 --> 1:22:22.880
+Now here it requires the same vocabulary.
+
+1:22:22.880 --> 1:22:25.468
+Why does it need the same vocabulary?
+
+1:22:25.468 --> 1:22:26.354
+Give me the.
+
+1:22:34.754 --> 1:22:39.132
+You have different vocabulary.
+
+1:22:39.132 --> 1:22:50.711
+You also have different like sequence lengths
+because if you use different these.
+
+1:22:51.231 --> 1:22:55.680
+Then what happens is now we have states here.
+
+1:22:55.680 --> 1:23:01.097
+It's no longer straightforward which states
+to compare.
+
+1:23:02.322 --> 1:23:05.892
+And then it's just easier to have like the
+same number.
+
+1:23:05.892 --> 1:23:08.952
+You can always compare the first to the second.
+
+1:23:09.709 --> 1:23:16.836
+So therefore at least the very easy way of
+knowledge destination only works if you have.
+
+1:23:17.177 --> 1:23:30.871
+Of course you could do things like the average
+should be the same, but of course that's less
+
+1:23:30.871 --> 1:23:33.080
+strong signal.
+
+1:23:34.314 --> 1:23:47.087
+But the advantage here is that you have a
+direct training signal here on the ink corner
+
+1:23:47.087 --> 1:23:52.457
+so you can directly make the signal.
+
+1:23:56.936 --> 1:24:11.208
+Yes, think this is most things for today,
+so what you should keep in mind today is two
+
+1:24:11.208 --> 1:24:18.147
+techniques: The one is a back translation idea.
+
+1:24:18.147 --> 1:24:26.598
+If you have monolingual letters, you back
+translate it and use.
+
+1:24:26.886 --> 1:24:33.608
+And yeah, it is even often helpful to even
+combine them so you can even use both of them.
+
+1:24:33.853 --> 1:24:39.669
+You can do use pre-trained walls, but then
+you can even still do back translation where
+
+1:24:39.669 --> 1:24:40.066
+it's.
+
+1:24:40.160 --> 1:24:47.058
+We have the advantage that we are training
+like everything working together on the tasks
+
+1:24:47.058 --> 1:24:54.422
+so it might be helpful even to backtranslate
+some data and then use it in the real translation
+
+1:24:54.422 --> 1:24:57.755
+because in pre-training the big challenge.
+
+1:24:58.058 --> 1:25:07.392
+You can see there is different ways of integrating
+this knowledge, but even if you use a full
+
+1:25:07.392 --> 1:25:08.087
+model.
+
+1:25:08.748 --> 1:25:11.713
+This is the most similar you can get.
+
+1:25:11.713 --> 1:25:15.224
+You're doing no changes to the architecture.
+
+1:25:15.224 --> 1:25:20.608
+You're really taking the model and just fine
+tuning on the new task.
+
+1:25:20.608 --> 1:25:24.041
+But it still has to completely newly learn.
+
+1:25:24.464 --> 1:25:29.978
+Might be, for example, helpful to have more
+back translated data to learn them.
+
+1:25:32.192 --> 1:25:45.096
+Good, that's important thing that next Tuesday
+there is a conference or a workshop in this
+
+1:25:45.096 --> 1:25:45.947
+room.
+
+1:25:47.127 --> 1:25:54.405
+You should get an email if you're an alias
+that there is a room change for Tuesdays, only
+
+1:25:54.405 --> 1:25:57.398
+for Tuesdays, and it's again normal.
+
+1:25:57.637 --> 1:26:03.714
+Some more questions again have a more general
+perspective, especially: Computer vision.
+
+1:26:03.714 --> 1:26:07.246
+You can enlarge your data set with data augmentation.
+
+1:26:07.246 --> 1:26:08.293
+It's there and.
+
+1:26:08.388 --> 1:26:15.306
+Similarly to a large speech or text, so the
+data orientation.
+
+1:26:15.755 --> 1:26:27.013
+You can use this back translation and also
+the masking, but a bit like that would say
+
+1:26:27.013 --> 1:26:31.201
+that is the most similar thing.
+
+1:26:31.371 --> 1:26:35.632
+So it has also been, for example, it's used
+not only for monolingual data.
+
+1:26:36.216 --> 1:26:40.958
+If you have good MP system, it can also be
+used for parallel data by having like augmenting
+
+1:26:40.958 --> 1:26:46.061
+your data with more data because then you have
+the human translation and the automatic translation
+
+1:26:46.061 --> 1:26:46.783
+is both good.
+
+1:26:46.783 --> 1:26:51.680
+You're just having more data and better feedback
+signal and different ways because there's not
+
+1:26:51.680 --> 1:26:53.845
+only one correct translation but several.
+
+1:26:54.834 --> 1:26:58.327
+Would say this is the most similar one.
+
+1:26:58.327 --> 1:27:00.947
+Just rotate things and so on.
+
+1:27:00.947 --> 1:27:03.130
+There's ways you can do.
+
+1:27:05.025 --> 1:27:07.646
+But for example there's rarely use.
+
+1:27:07.646 --> 1:27:13.907
+It's very hard to do this by by rules like
+which words to replace because there's not
+
+1:27:13.907 --> 1:27:14.490
+a cool.
+
+1:27:14.490 --> 1:27:18.931
+You cannot like always say this word can always
+be replaced.
+
+1:27:19.139 --> 1:27:28.824
+Mean, although they are my perfect synonyms,
+they are good in some cases, but not in all
+
+1:27:28.824 --> 1:27:29.585
+cases.
+
+1:27:29.585 --> 1:27:36.985
+And if you don't do a rule base, you have
+to train the model again.
+
+1:27:38.058 --> 1:27:57.050
+Here we can compare the hidden stages to the
+same architecture as the free train normal.
+
+1:27:57.457 --> 1:27:59.817
+Should be of the same dimension, so it's easiest
+to have the.
+
+1:28:00.000 --> 1:28:03.780
+Architecture: We later will learn in efficiency.
+
+1:28:03.780 --> 1:28:08.949
+You can also do knowledge destillation with,
+for example, smaller.
+
+1:28:08.949 --> 1:28:15.816
+So you can have twelve layers, only five,
+and then you try to learn the same within five
+
+1:28:15.816 --> 1:28:16.433
+layers.
+
+1:28:17.477 --> 1:28:22.945
+Eight layers, so that is possible, but yeah
+agree it should be of the same hidden size.
+
+1:28:23.623 --> 1:28:35.963
+The question then, of course, is you can do
+it as an initialization or you can do it during
+
+1:28:35.963 --> 1:28:37.305
+training?
+
+1:28:37.305 --> 1:28:41.195
+You have some main training.
+
+1:28:45.865 --> 1:28:53.964
+Good, then thanks a lot, and then we'll see
+each other again on Tuesday.
+