Learning Optimal Prosody Embedding Codebook based on F0 and
Energy

D 2025

Learning Optimal Prosody Embedding Codebook based on F0 and Energy

PORTEŠ, David a Aleš HORÁK

Základní údaje

Originální název

Learning Optimal Prosody Embedding Codebook based on F0 and Energy

Autoři

PORTEŠ, David a Aleš HORÁK

Vydání

Interspeech 2025. BAIXAS, INTERSPEECH 2025, od s. 4728-4732, 5 s. 2025

Nakladatel

ISCA-INT SPEECH COMMUNICATION ASSOC

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

10201 Computer sciences, information science, bioinformatics

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

elektronická verze "online"

Označené pro přenos do RIV

Ano

Organizační jednotka

Fakulta informatiky

ISSN

Klíčová slova anglicky

Prosody; VQ-VAE; Fundamental frequency; F0; Energy; Embeddings

Štítky

core_A, firank_A

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 1. 4. 2026 10:59, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

Both the Fundamental frequency (F0) and Energy are prominent features of prosody. Together, they have been used across a wide variety of speech-processing tasks. However, there is a lack of freely available pre-trained vector representations of these features. Therefore, in this paper, we provide the research community with high-quality joint embeddings of the frame-level F0 and Energy features, using the VQ-VAE architecture. By converting the F0 and Energy into a single stream of vector embeddings, we make it possible to seamlessly use prosody in modern architectures, such as multimodal LLMs. In order to ensure maximum embedding quality, we conduct a large-scale hyperparameter search, totaling over 150 experiments on the LibriTTS dataset. We outperform previous works on F0 embeddings, reaching FFE error below 1 percent, while simultaneously embedding the additional feature of Energy. We publish our best-performing models on the HuggingFace website.

Návaznosti

MUNI/A/1638/2024, interní kód MU

Název: Umělá inteligence a správa komplexních rozsáhlých dat

Investor: Masarykova univerzita, Umělá inteligence a správa komplexních rozsáhlých dat

MUNI/A/1666/2024, interní kód MU

Název: Zapojení studentů Fakulty informatiky do mezinárodní vědecké komunity 25

Investor: Masarykova univerzita, Zapojení studentů Fakulty informatiky do mezinárodní vědecké komunity 25

Citovat

PORTEŠ, David a Aleš HORÁK. Learning Optimal Prosody Embedding Codebook based on F0 and Energy. Online. In INTERSPEECH 2025. Interspeech 2025. BAIXAS: ISCA-INT SPEECH COMMUNICATION ASSOC, 2025, s. 4728-4732. ISSN 2308-457X. Dostupné z: https://doi.org/10.21437/Interspeech.2025-1020.

@inproceedings{2499737,
   author = {Porteš, David and Horák, Aleš},
   address = {BAIXAS},
   booktitle = {INTERSPEECH 2025},
   doi = {https://doi.org/10.21437/Interspeech.2025-1020},
   edition = {Interspeech 2025},
   keywords = {Prosody; VQ-VAE; Fundamental frequency; F0; Energy; Embeddings},
   howpublished = {elektronická verze "online"},
   language = {eng},
   location = {BAIXAS},
   note = {Accepted for publication.},
   pages = {4728-4732},
   publisher = {ISCA-INT SPEECH COMMUNICATION ASSOC},
   title = {Learning Optimal Prosody Embedding Codebook based on F0 and Energy},
   year = {2025}
}

TY  - CONF
ID  - 2499737
AU  - Porteš, David - Horák, Aleš
PY  - 2025
TI  - Learning Optimal Prosody Embedding Codebook based on F0 and Energy
PB  - ISCA-INT SPEECH COMMUNICATION ASSOC
CY  - BAIXAS
N1  - Accepted for publication.
KW  - Prosody
KW  - VQ-VAE
KW  - Fundamental frequency
KW  - F0
KW  - Energy
KW  - Embeddings
N2  - Both the Fundamental frequency (F0) and Energy are prominent features of prosody. Together, they have been used across a wide variety of speech-processing tasks. However, there is a lack of freely available pre-trained vector representations of these features. Therefore, in this paper, we provide the research community with high-quality joint embeddings of the frame-level F0 and Energy features, using the VQ-VAE architecture. By converting the F0 and Energy into a single stream of vector embeddings, we make it possible to seamlessly use prosody in modern architectures, such as multimodal LLMs. In order to ensure maximum embedding quality, we conduct a large-scale hyperparameter search, totaling over 150 experiments on the LibriTTS dataset. We outperform previous works on F0 embeddings, reaching FFE error below 1 percent, while simultaneously embedding the additional feature of Energy. We publish our best-performing models on the HuggingFace website.
ER  -

PORTEŠ, David a Aleš HORÁK. Learning Optimal Prosody Embedding Codebook based on F0 and Energy. Online. In \textit{INTERSPEECH 2025}. Interspeech 2025. BAIXAS: ISCA-INT SPEECH COMMUNICATION ASSOC, 2025, s.~4728-4732. ISSN~2308-457X. Dostupné z: https://doi.org/10.21437/Interspeech.2025-1020.

Přehled o publikaci