Large qualitative gains were recently observed in Natural Language Processing (NLP) tasks thanks to huge so-called transformers models with hundreds of millions of parameters. However, in practice, these models suffer from high latency, such that they often are hardly usable in practical applications. Specifically, in Neural Machine Translation (NMT), classical approaches iteratively condition each output word on previously generated outputs, effectively aggregating the prediction time. This thesis aims to overview techniques that avoid this property and produce its results in parallel, allowing an order of magnitude lower latency on inference.

The student in this thesis will:

• understand and describe the process of neural machine translation using transformer models
• train different standard (auto-regressive) transformer models for NMT
• measure the performance and inference times of these models
• study the available relevant options of speeding up the inference time and deploy their application for NMT
• evaluate both qualitative and performance impact concerning the inference time on a task of NMT