Neural Language Modelling
PA154 Language Modeling (10.1)
Pavel Rychlý
pary@fi.muni.cz April 30,2024
Deep Learning
■ deep neural networks
■ many Layers
■ trained on big data
■ using advanced hardware: GPU,TPU
■ supervised, semi-supervised or unsupervised
Pavel Rychly • Neural Language Modelling • April 30, 2024
2/26
Neuron
■ basic element of neural networks
■ many inputs (numbers), weights (numbers)
■ activation (transfer) function (threshold)
■ one output: y = ct(Y1T=o wjxj +
3/26
Neuron on an embedding
■ input: embedding (vector of nubmers)
■ output: y = criYljLo wjxj + b)
■ Linear classifier
■ hyperpLane cutting the vector space
■ selects one feature
Pavel Rychly • Neural Language Modelling • April 30, 2024
4/26
Neural Networks
■ Input/Hidden/Output Layer
■ Input/output = vector of numbers
■ hidden Layer = matrix of parameters (numbers)
m
Yk = <t{^2 wkjXj)
7=0
Y = a(WXT)
Hidden
Pavel Rychly • Neural Language Modelling • April 30, 2024
5/26
Activation Functions
■ crucial component of NN
■ non-Linear function
■ many Layers without non-Linear activation functions are equivaLent to single Layer (Linear combination of inputs)
Leaky ReLU
max(0.l£, x)
ReLU
https://en.wikipedia.org/wiki/Activation_function
Pavel Rychly • Neural Language Modelling • April 30, 2024
One-hot representation
■ words/cLasses: [0 0 0 1 0 0 0 0]
■ for each word/cLass one input
■ one input activated (1), others deactivated (0)
■ whole input vector couLd be Large = size of vocabulary (^30k)
■ sequence of words requires sequence of one-hot vectors
■ first Layer transforms words into word embeddings
■ usually not represented explicitly as vectors, using one single number
■ output:
■ one-hot vector during training = expecting one word/class
■ probability distribution during usage
■ cannot be represented using single number
Pavel Rychly • Neural Language Modelling • April 30, 2024
7/26
Training Neural Networks
■ supervised training
■ example: input + result
■ difference between output and expected result (Loss function)
■ adjusts weights according to a Learning rule
■ backpropagation (feedforward neuraL networks)
■ gradient of the Loss function, stochastic gradient descent (SGD)
Pavel Rychly • NeuraL Language Modelling • April 30, 2024
8/26
Training Language Models
■ core function of Language modeLs: predict the foLLowing word:
P(X5|X1JX2JX3JX4)
■ input: context = Xi,x2,X3,x4
■ output: probability distribution of the foLLowing word
■ training:
■ input: xi,X2,X3,X4
■ output: x5 (one-hot vector)
■ training data:
■ get any text (corpus)
■ extract aLL n-grams
Pavel Rychly • Neural Language Modelling • April 30, 2024
9/26
Why are NNs better than statistics
■ continues space representation
■ items (words/tags/classes) are not atomic
■ no sparsity problem
■ don't need to store all observed n-grams
■ vectors handles relations
■ many realations, not explicit, unknown
■ NN can represent any function (if deep enough)
■ structure of the function is not pre-defined
Pavel Rychly • Neural Language Modelling • April 30, 2024
10/26
Why are NNs used only Last 10 years
■ big training data
■ powerful hardware
■ Moore's law: memory size, processor's speed doubles every few years
■ matrix processing using GPU,TPU
■ better Learning strategies, NN optimizatons
■ Adam, AdaFactor optimizers
■ dropout
■ attention
■ ready to use Libraries/framewoks, datasets
Pavel Rychly • Neural Language Modelling • April 30, 2024
11/26
A fixed-window neural Language Model
v = vocabulary size
d = embedding size
h = hidden Layer size
Input: IDs of words (sparse representation of one-hot vector)
E: embeddings (v x d)
W: hidden Layer (4d x h)
U: output Layer (h x v)
books
laptops
1
zoo
u
OOOOOOllltll
w
OOOO   OOOO   OOOO 0000
the     students opened
their
xm       x(2)        x(t) x(4)
Pavel Rychly • Neural Language Modelling • April 30, 2024
12/26
A fixed-window neural Language Model
E: embeddings (v x d)
W: hidden Layer (4d x h)
U: output Layer (h x v)
A = E(X)
B=f(WA)
Z = softmax(UB)
books
laptops
1
ill
zoo
u
OOOOOOllltll
w
OOOO   OOOO   OOOO 0000
the     students opened
their
xm       x(2)        x(t) x(4)
Pavel Rychly • Neural Language Modelling • April 30, 2024
13/26
A fixed-window neural Language Model
books
fixed window is too small
enlarging window enlarges W
window can never be Large enough!
x1 and x2 are multiplied by completely different weights in W.
No symmetry in how the inputs are processed.
laptops
1
ill
zoo
u
OOOOOOllltll
w
OOOO   OOOO   OOOO OOOO
the
students opened
x(2) x(3)
their
Pavel Rychly • Neural Language Modelling • April 30, 2024
14/26
Recurrent Neural Network (RNN)
■ dealing with Long inputs
■ feedforward NN + internal state (memory)
■ finite impulse RNN: unroUto strictly feedforward NN
■ infinite impulse RNN: directed cyclic graph
■ additional storage managed by NN: gated state/memory
■ backpropagation through time
Pavel Rychly • Neural Language Modelling • April 30, 2024
Problem of Long-Term Dependencies
the clouds are in the sky.
©		©		©		©		©
t		t		t		t		t
A	—►	A	—►	A	—►	A	—►	A
X
X
X
/ grew up in France... I speak fluent French .
©		©		©		©				
t		t		t		t		t		t
A	—►	A	—►	A	-►	A	—►	A	—►	A
Pavel Rychly • Neural Language Modelling • April 30, 2024
16/26
Long short-term memory (LSTM)
■ LSTM unit: cell, input gate, output gate and forget gate
■ cell = memeory
■ gates regulate the flow of information into and out of the cell
Pavel Rychly • Neural Language Modelling • April 30, 2024
17/26
LSTM - Gates
Hidden state
H
(-]
Forget
gate
F
Input X
Input
gate
t
Output gate
O
<7
FC layer with activation function
J       Copy —
Concatenate
Pavel Rychly • Neural Language Modelling • April 30, 2024
18/26
LSTM - Input
Hidden state
Lf-]
Forget
gate
F
Input X,
Input gate
1,
i
I
Input node
Output gate
O
		u		tanh		° i
J
(7
FC layer with activation function
j     copy ~Y
Concatenate
Pavel Rychly • Neural Language Modelling • April 30, 2024
19/26
LSTM - Memory
Memory tell internal state
Hidden state H
*(0
Forget gate F
2
•G>
Input gate
Input ^ node T
1
1
Input X,
Output gate O
1
<7		a		tanh		a
FC layer with activation function
o
Elementwise operator
J       Copy —
Concatenate
Pavel Rychly • Neural Language Modelling • April 30, 2024
20/26
LSTM - Hidden state
Memory cell internal stale C
i-1
Hidden state It
MB
Forget gate F
1
•G>
Input gate
Input
©
Output
<7	I,	Ü		tanh		<7
1
1
II
Input XJ
FC layer with activation function
o
Element wise operator
J       Copy —
Concatenate
Pavel Rychly • Neural Language Modelling • April 30, 2024
21/26
GRU, BRNN,...
■ Gated recurrent unit (GRU)
■ fewer parameters than LSTM
■ memory = output
■ Bi-directional RNN
■ two hidden Layers of opposite directions to the same output
■ hierarchical, muLtiLayer
Pavel Rychly • Neural Language Modelling • April 30, 2024
22/26
Encoder-Decoder
variable input/output size, not 1-1 mapping two components
Encoder: variable-Length sequence —>► fixed size state Decoder: fixed size state —>► variable-Length sequence
Encoder
Decoder
Pavel Rychly • Neural Language Modelling • April 30, 2024
Sequence to Sequence
Learning
■ Encoder: Input sequence -> state
■ Decoder: state + output sequence
-> output sequence
Encoder
Decoder
T
They
t r
are watching
J
T
<eos>
lis
i
<bos>
regardent
I
IIS
regardent
<eos>
i
Pavel Rychly • Neural Language Modelling • April 30, 2024
24/26
Sequence to Sequence
Using
Encoder: Input sequence —>► state Decoder: state + sentence delimiter
-> output
Encoder
Decoder
T } t t
They      are watching
T
<eos>
<eos>
lis regardent
i f L_J) L-JT L_>7
<bos>
Pavel Rychly • Neural Language Modelling • April 30, 2024
25/26
Multi-layer encoder/decoder
Encoder
Decoder
n x
Recurrent
Embedding
Sources
Recurrent
I
Embedding
Targets
Pavel Rychly • Neural Language Modelling • April 30, 2024