LSTM-Based Real-Time Action Detection and Prediction in Human
Motion Streams

J 2019

LSTM-Based Real-Time Action Detection and Prediction in Human Motion Streams

CARRARA, Fabio, Petr ELIÁŠ, Jan SEDMIDUBSKÝ a Pavel ZEZULA

Základní údaje

Originální název

LSTM-Based Real-Time Action Detection and Prediction in Human Motion Streams

Autoři

CARRARA, Fabio (380 Itálie), Petr ELIÁŠ (203 Česká republika, domácí), Jan SEDMIDUBSKÝ (203 Česká republika, garant, domácí) a Pavel ZEZULA (203 Česká republika, domácí)

Vydání

Multimedia Tools and Applications, Springer US, 2019, 1380-7501

Další údaje

Jazyk

angličtina

Typ výsledku

Článek v odborném periodiku

Obor

10200 1.2 Computer and information sciences

Stát vydavatele

Nizozemské království

Utajení

není předmětem státního či obchodního tajemství

Odkazy

URL

Impakt faktor

Impact factor: 2.313

Kód RIV

RIV/00216224:14330/19:00109721

Organizační jednotka

Fakulta informatiky

DOI

http://dx.doi.org/10.1007/s11042-019-07827-3

UT WoS

000485298000024

Klíčová slova anglicky

motion capture data;stream annotation;action detection and recognition;action prediction;LSTM;recurrent neural network

Štítky

DISA

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 21. 1. 2020 08:21, doc. RNDr. Jan Sedmidubský, Ph.D.

Anotace

V originále

Motion capture data digitally represent human movements by sequences of 3D skeleton configurations. Such spatio-temporal data, often recorded in the stream-based nature, need to be efficiently processed to detect high-interest actions, for example, in human-computer interaction to understand hand gestures in real time. Alternatively, automatically annotated parts of a continuous stream can be persistently stored to become searchable, and thus reusable for future retrieval or pattern mining. In this paper, we focus on multi-label detection of user-specified actions in unsegmented sequences as well as continuous streams. In particular, we utilize the current advances in recurrent neural networks and adopt a unidirectional LSTM model to effectively encode the skeleton frames within the hidden network states. The model learns what subsequences of encoded frames belong to the specified action classes within the training phase. The learned representations of classes are then employed within the annotation phase to infer the probability that an incoming skeleton frame belongs to a given action class. The computed probabilities are finally compared against a learned threshold to automatically determine the beginnings and endings of actions. To further enhance the annotation accuracy, we utilize a bidirectional LSTM model to estimate class probabilities by considering not only the past frames but also the future ones. We extensively evaluate both the models on the three use cases of real-time stream annotation, offline annotation of long sequences, and early action detection and prediction. The experiments demonstrate that our models outperform the state of the art in effectiveness and are at least one order of magnitude more efficient, being able to annotate 10k frames per second.

Návaznosti

EF16_019/0000822, projekt VaV

Název: Centrum excelence pro kyberkriminalitu, kyberbezpečnost a ochranu kritických informačních infrastruktur

Citovat

CARRARA, Fabio, Petr ELIÁŠ, Jan SEDMIDUBSKÝ a Pavel ZEZULA. LSTM-Based Real-Time Action Detection and Prediction in Human Motion Streams. Multimedia Tools and Applications. Springer US, 2019, roč. 78, č. 19, s. 27309-27331. ISSN 1380-7501. Dostupné z: https://dx.doi.org/10.1007/s11042-019-07827-3.

@article{1533577,
   author = {Carrara, Fabio and Eliáš, Petr and Sedmidubský, Jan and Zezula, Pavel},
   article_number = {19},
   doi = {http://dx.doi.org/10.1007/s11042-019-07827-3},
   keywords = {motion capture data;stream annotation;action detection and recognition;action prediction;LSTM;recurrent neural network},
   language = {eng},
   issn = {1380-7501},
   journal = {Multimedia Tools and Applications},
   title = {LSTM-Based Real-Time Action Detection and Prediction in Human Motion Streams},
   url = {http://dx.doi.org/10.1007/s11042-019-07827-3},
   volume = {78},
   year = {2019}
}

TY  - JOUR
ID  - 1533577
AU  - Carrara, Fabio - Eliáš, Petr - Sedmidubský, Jan - Zezula, Pavel
PY  - 2019
TI  - LSTM-Based Real-Time Action Detection and Prediction in Human Motion Streams
JF  - Multimedia Tools and Applications
VL  - 78
IS  - 19
SP  - 27309-27331
EP  - 27309-27331
PB  - Springer US
SN  - 13807501
KW  - motion capture data;stream annotation;action detection and recognition;action prediction;LSTM;recurrent neural network
UR  - http://dx.doi.org/10.1007/s11042-019-07827-3
L2  - http://dx.doi.org/10.1007/s11042-019-07827-3
N2  - Motion capture data digitally represent human movements by sequences of 3D skeleton configurations. Such spatio-temporal data, often recorded in the stream-based nature, need to be efficiently processed to detect high-interest actions, for example, in human-computer interaction to understand hand gestures in real time. Alternatively, automatically annotated parts of a continuous stream can be persistently stored to become searchable, and thus reusable for future retrieval or pattern mining. In this paper, we focus on multi-label detection of user-specified actions in unsegmented sequences as well as continuous streams. In particular, we utilize the current advances in recurrent neural networks and adopt a unidirectional LSTM model to effectively encode the skeleton frames within the hidden network states. The model learns what subsequences of encoded frames belong to the specified action classes within the training phase. The learned representations of classes are then employed within the annotation phase to infer the probability that an incoming skeleton frame belongs to a given action class. The computed probabilities are finally compared against a learned threshold to automatically determine the beginnings and endings of actions. To further enhance the annotation accuracy, we utilize a bidirectional LSTM model to estimate class probabilities by considering not only the past frames but also the future ones. We extensively evaluate both the models on the three use cases of real-time stream annotation, offline annotation of long sequences, and early action detection and prediction. The experiments demonstrate that our models outperform the state of the art in effectiveness and are at least one order of magnitude more efficient, being able to annotate 10k frames per second.
ER  -

CARRARA, Fabio, Petr ELIÁŠ, Jan SEDMIDUBSKÝ a Pavel ZEZULA. LSTM-Based Real-Time Action Detection and Prediction in Human Motion Streams. \textit{Multimedia Tools and Applications}. Springer US, 2019, roč.~78, č.~19, s.~27309-27331. ISSN~1380-7501. Dostupné z: https://dx.doi.org/10.1007/s11042-019-07827-3.

Podrobný výpis o publikaci