MUNI
FI
Zero-shot, one-shot, few-shot
PA154 Language Modeling (12.1)
Pavel Rychlý
pary@fi.muni.cz May 14,2024
Usage of Large Models
■ training of big models on huge data is expensive (long training time)
■ fine tuning on small data of target task
■ combining language model with additional NN/layer, training only new layer
■ big modeL is frozen, onLy used
classification    sian |    Text    | Bnraci      Transformer \*\ Linear
Erilalmeril    |   Stun   |   Premise      Dnlirr | Hypslliosi'd | r. nri    - Transformer
Start   1    Textl    | De	m J    Text 2    [ i-f met J ► Transformer |
Stan   1    Text 2 De	m 1     Text 1     1 Exiract |     Transformer |
^avel Rychlý ■ Zero-shot, one-shot, few-shot ■ Mayl4,2024
Usage of LLM without training
Zero-shot
Usage of LLM without fine tuning
■ fine tuning is still expensive
■ models can predict reliably
■ using generation for end tasks
■ zero-shot no task-specific data/training
formulate task in natural language task description + prompt TransLate EngLish to French: cheese =>
Summarize the foLLowing paragraph into one
sentence:
text
^avel Rychlý ■ Zero-shot,one-shot,few-shot ■ Mayl4,2024
^avel Rychlý ■ Zero-shot, one-shot, few-shot ■ Mayl4,2024
One-shot
Few-shot
formulate task in natural language and show one example task description + example + prompt TransLate EngLish to French: seq otter => Loutre de mar cheese =>
■ formulate task in natural language and show a few examples
■ task description + examples + prompt
■ TransLate EngLish to French: seq otter => Loutre de mar peppermint => menthe poivree pLush girafe => girafe peLuche cheese =>
Language Models are Few-Shot Learners
^avel Rychlý ■ Zero-shot,one-shot,few-shot ■ Mayl4,2024
5/8
^avel Rychlý ■ Zero-shot, one-shot, few-shot ■ Mayl4,2024
6/8
Token is MASK
Available LLM
test sentence
template
I will visit Munich next week. Munich is a [MASK]
P retrained Language Model
predicts p(V)
company ORG team
city
country
man girl
TOKEN is a MASK: Few-shot Named Entity Recognition with Pre-trained Language Models
Pythia OPT
GALACTICA TO
^avel Rychlý ■ Zero-shot,one-shot,few-shot ■ Mayl4,2024
^avel Rychlý ■ Zero-shot, one-shot, few-shot ■ Mayl4,2024
67