Machine Translation PV061 Pavel Rychlý NLP Centre, FI MU 20 Sep 2023 Pavel Rychlý ·Machine Translation ·20 Sep 2023 1 / 37 Technical information History Handling problems Neural Networks Outline of the course Pavel Rychlý ·Machine Translation ·20 Sep 2023 2 / 37 Technical information Technical information Pavel Rychlý ·Machine Translation ·20 Sep 2023 3 / 37 Technical information Technical information Pavel Rychlý head of NLP Centre Natural Language Processing Centre around 10 PhD students you can be part of it (PV173 = 3 credits each semester) Pavel Rychlý ·Machine Translation ·20 Sep 2023 4 / 37 Technical information Technical information Study materials in IS book: Philipp Koehn: Neural Machine Translation (U366) Exam: written – max 10 questions open books (offline) max 60 points 30 point to pass extra points (max 30) for homeworks, projects find good examples, illustrations to improve understanding code, language, pictures exam, homeworks, ... in English, Czech, Slovak Pavel Rychlý ·Machine Translation ·20 Sep 2023 5 / 37 Technical information Previous knowledge no special requirements reading mathematics probabilities examples in Python NumPy, PyTorch (matrix operations) complements PV021: Neural Networks PA153: Natural Language Processing IA161: Natural Language Processing in Practice Pavel Rychlý ·Machine Translation ·20 Sep 2023 6 / 37 History History Pavel Rychlý ·Machine Translation ·20 Sep 2023 7 / 37 History Initial Idea Warren Weaver on translation as code breaking (1947): When I look at an article in Russian, I say: ”This is really written in English, but it has been coded in some strange symbols. I will now proceed to decode”. Pavel Rychlý ·Machine Translation ·20 Sep 2023 8 / 37 History Translation or transcription Pavel Rychlý ·Machine Translation ·20 Sep 2023 9 / 37 History Translation or transcription Pavel Rychlý ·Machine Translation ·20 Sep 2023 9 / 37 History Translation or transcription We need some examples Pavel Rychlý ·Machine Translation ·20 Sep 2023 10 / 37 History Translation or transcription We need some examples Coca-Cola Pavel Rychlý ·Machine Translation ·20 Sep 2023 10 / 37 History Early Efforts Excited research in 1950s and 1960s 1954 - Georgetown experiment Machine could translate 250 words and 6 grammar rules 1966 ALPAC report: only $20 million spent on translation in the US per year no point in machine translation Pavel Rychlý ·Machine Translation ·20 Sep 2023 11 / 37 History Main Idea We can tranlate/transcribe on different levels Pavel Rychlý ·Machine Translation ·20 Sep 2023 12 / 37 History Rule-Based Systems Rule-based systems build dictionaries write transformation rules refine, refine, refine Météo system for weather forecasts (1976) Systran (1968) Pavel Rychlý ·Machine Translation ·20 Sep 2023 13 / 37 History Statistical Machine Translation 1980s: IBM 1990s: increased research Mid 2000s: Phrase-Based MT (Moses, Google) Around 2010: commercial viability Pavel Rychlý ·Machine Translation ·20 Sep 2023 14 / 37 History Neural Machine Translation late 2000s: successful use of neural models for computer vision Since mid 2010s: neural network models for machine translation 2016: Neural machine translation the new state of the art Pavel Rychlý ·Machine Translation ·20 Sep 2023 15 / 37 History Results CUBBITT system for EN to CS UFAL, Faculty of Mathematics and Physics, Charles University Nature Communications paper September 2020 Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals better than human in adequacy in certain circumstances news domain rare phrases, translated literally by human translators Pavel Rychlý ·Machine Translation ·20 Sep 2023 16 / 37 Handling problems Handling problems Pavel Rychlý ·Machine Translation ·20 Sep 2023 17 / 37 Handling problems Word Translation Problems Words are ambiguous How do we find the right meaning, and thus translation? Context should be helpful He deposited money in a bank account with a high interest rate. Sitting on the bank of the Mississippi, a passing ship piqued his interest. Pavel Rychlý ·Machine Translation ·20 Sep 2023 18 / 37 Handling problems Syntactic Translation Problems Languages have different sentence structure Convert from object-verb-subject (OVS) to subject-verb-object (SVO) das behaupten sie wenigstens this claim they at least the she Ambiguities can be resolved through syntactic analysis the meaning the of das not possible (not a noun phrase) the meaning she of sie not possible (subject-verb agreement) Pavel Rychlý ·Machine Translation ·20 Sep 2023 19 / 37 Handling problems Semantic Translation Problems Pronominal anaphora I saw the movie and it is good. How to translate it into German (or French)? it refers to movie movie translates to Film Film has masculine gender ergo: it must be translated into masculine pronoun er Pavel Rychlý ·Machine Translation ·20 Sep 2023 20 / 37 Handling problems Semantic Translation Problems Coreference Whenever I visit my uncle and his daughters, I can’t decide who is my favorite cousin. How to translate cousin into German? Male or female? Complex inference required Pavel Rychlý ·Machine Translation ·20 Sep 2023 21 / 37 Handling problems Semantic Translation Problems Discourse Since you brought it up, I do not agree with you. Since you brought it up, we have been working on it. How to translated since? Temporal or conditional? Analysis of discourse structure — a hard problem Pavel Rychlý ·Machine Translation ·20 Sep 2023 22 / 37 Handling problems Rules hard to find many exceptions, exceptions in exceptions, ... only suitable for cases without data Pavel Rychlý ·Machine Translation ·20 Sep 2023 23 / 37 Handling problems Statistics probabilities/rules learned from data linguistic knowledge about the structure of languages (SVO, VSO, .., ADJ+NN, NN+ADJ, ...) NLP tools (tokenizers, lemmatizers, taggers, ...) sparsity of data, long tail problem Pavel Rychlý ·Machine Translation ·20 Sep 2023 24 / 37 Handling problems Neural Networks (NN) very simple model neurons in layers trained on raw text data (almost no preprocessing) requires many training examples Pavel Rychlý ·Machine Translation ·20 Sep 2023 25 / 37 Neural Networks Neural Networks Pavel Rychlý ·Machine Translation ·20 Sep 2023 26 / 37 Neural Networks Neuron basic element of neural networks many inputs (numbers), weights (numbers) activation (transfer) function (threshold) one output: y = ϕ( m j=0 wjxj + b) Pavel Rychlý ·Machine Translation ·20 Sep 2023 27 / 37 Neural Networks Neural Networks Input/Hidden/Output layer Input/output = vector of numbers hidden layer = matrix of parameters (numbers) yk = ϕ( m j=0 wkjxj) Y = ϕ(WXT ) Pavel Rychlý ·Machine Translation ·20 Sep 2023 28 / 37 Neural Networks Words as vectors continue = [0.286, 0.792, −0.177, −0.107, 0.109, −0.542, 0.349] Pavel Rychlý ·Machine Translation ·20 Sep 2023 29 / 37 Neural Networks Neural Machine Translation encoder-decoder Pavel Rychlý ·Machine Translation ·20 Sep 2023 30 / 37 Neural Networks Tranformer using attention each decoder layer has access to all hidden states from the last encoder use attention to extract important parts (vector) Pavel Rychlý ·Machine Translation ·20 Sep 2023 31 / 37 Neural Networks Why are NNs better than statistics? continues space representation words are not atomic no sparsity problem vectors handles relations many realations, not explicit, unknown NN can represent any function (if deep enough) structure of the function is not pre-defined Pavel Rychlý ·Machine Translation ·20 Sep 2023 32 / 37 Neural Networks Why are NNs used only last 10 years? available big training data powerful hardware matrix processing using specialized hardware GPU, TPU better learning strategies, NN optimizatons ready to use libraries/framewoks, datasets Pavel Rychlý ·Machine Translation ·20 Sep 2023 33 / 37 Outline of the course Outline of the course Pavel Rychlý ·Machine Translation ·20 Sep 2023 34 / 37 Outline of the course Outline 1 statistics, probabilities language models IBM model 1 phrase-base models decoding/generation evaluation Pavel Rychlý ·Machine Translation ·20 Sep 2023 35 / 37 Outline of the course Outline 2 neural networks, computation graphs tokenization, word representaion neural language models neural translation models monolingual data pretrained models Pavel Rychlý ·Machine Translation ·20 Sep 2023 36 / 37 Outline of the course Conclusions Current state MT systems use deep neural networks MT is very good in many areas It can be improved more data bigger models better training strategies Pavel Rychlý ·Machine Translation ·20 Sep 2023 37 / 37