Policy Learning for Time-Bounded Reachability in
Continuous-Time Markov Decision Processes via Doubly-Stochastic
Gradient Ascent

D 2016

Policy Learning for Time-Bounded Reachability in Continuous-Time Markov Decision Processes via Doubly-Stochastic Gradient Ascent

BRÁZDIL, Tomáš; Ezio BARTOCCI; Dimitrios MILIOS; Guido SANGUINETTI; Luca BORTOLUSSI et al.

Základní údaje

Originální název

Policy Learning for Time-Bounded Reachability in Continuous-Time Markov Decision Processes via Doubly-Stochastic Gradient Ascent

Autoři

BRÁZDIL, Tomáš; Ezio BARTOCCI; Dimitrios MILIOS; Guido SANGUINETTI a Luca BORTOLUSSI

Vydání

Quebec City, Proceedings of QEST 2016, od s. 244-259, 16 s. 2016

Nakladatel

Springer

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Kanada

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

tištěná verze "print"

Impakt faktor

Impact factor: 0.402 v roce 2005

Označené pro přenos do RIV

Ano

Kód RIV

RIV/00216224:14330/16:00088513

Organizační jednotka

Fakulta informatiky

ISBN

978-3-319-43424-7

ISSN

Klíčová slova anglicky

continuous-time Markov decision processes; reachability; gradient descent

Štítky

firank_B, formela-conference

Změněno: 13. 5. 2020 19:26, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

Continuous-time Markov decision processes are an important class of models in a wide range of applications, ranging from cyber-physical systems to synthetic biology. A central problem is how to devise a policy to control the system in order to maximise the probability of satisfying a set of temporal logic specifications. Here we present a novel approach based on statistical model checking and an unbiased estimation of a functional gradient in the space of possible policies. The statistical approach has several advantages over conventional approaches based on uniformisation, as it can also be applied when the model is replaced by a black box, and does not suffer from state-space explosion. The use of a stochastic gradient to guide our search considerably improves the efficiency of learning policies. We demonstrate the method on a proof-of-principle non-linear population model, showing strong performance in a non-trivial task.

Návaznosti

GA15-17564S, projekt VaV

Název: Teorie her jako prostředek pro formální analýzu a verifikaci počítačových systémů

Investor: Grantová agentura ČR, Teorie her jako prostředek pro formální analýzu a verifikaci počítačových systémů

Citovat

BRÁZDIL, Tomáš; Ezio BARTOCCI; Dimitrios MILIOS; Guido SANGUINETTI a Luca BORTOLUSSI. Policy Learning for Time-Bounded Reachability in Continuous-Time Markov Decision Processes via Doubly-Stochastic Gradient Ascent. In Proceedings of QEST 2016. Quebec City: Springer, 2016, s. 244-259. ISBN 978-3-319-43424-7. Dostupné z: https://doi.org/10.1007/978-3-319-43425-4_17.

@inproceedings{1366655,
   author = {Brázdil, Tomáš and Bartocci, Ezio and Milios, Dimitrios and Sanguinetti, Guido and Bortolussi, Luca},
   address = {Quebec City},
   booktitle = {Proceedings of QEST 2016},
   doi = {https://doi.org/10.1007/978-3-319-43425-4_17},
   keywords = {continuous-time Markov decision processes; reachability; gradient descent},
   howpublished = {tištěná verze "print"},
   language = {eng},
   location = {Quebec City},
   isbn = {978-3-319-43424-7},
   pages = {244-259},
   publisher = {Springer},
   title = {Policy Learning for Time-Bounded Reachability in Continuous-Time Markov Decision Processes via Doubly-Stochastic Gradient Ascent},
   year = {2016}
}

TY  - CONF
ID  - 1366655
AU  - Brázdil, Tomáš - Bartocci, Ezio - Milios, Dimitrios - Sanguinetti, Guido - Bortolussi, Luca
PY  - 2016
TI  - Policy Learning for Time-Bounded Reachability in Continuous-Time Markov Decision Processes via Doubly-Stochastic Gradient Ascent
PB  - Springer
CY  - Quebec City
SN  - 9783319434247
KW  - continuous-time Markov decision processes
KW  - reachability
KW  - gradient descent
N2  - Continuous-time Markov decision processes are an important class of models in a wide range of applications, ranging from cyber-physical systems to synthetic biology. A central problem is how to devise a policy to control the system in order to maximise the probability of satisfying a set of temporal logic specifications. Here we present a novel approach based on statistical model checking and an unbiased estimation of a functional gradient in the space of possible policies. The statistical approach has several advantages over conventional approaches based on uniformisation, as it can also be applied when the model is replaced by a black box, and does not suffer from state-space explosion. The use of a stochastic gradient to guide our search considerably improves the efficiency of learning policies. We demonstrate the method on a proof-of-principle non-linear population model, showing strong performance in a non-trivial task.
ER  -

BRÁZDIL, Tomáš; Ezio BARTOCCI; Dimitrios MILIOS; Guido SANGUINETTI a Luca BORTOLUSSI. Policy Learning for Time-Bounded Reachability in Continuous-Time Markov Decision Processes via Doubly-Stochastic Gradient Ascent. In \textit{Proceedings of QEST 2016}. Quebec City: Springer, 2016, s.~244-259. ISBN~978-3-319-43424-7. Dostupné z: https://doi.org/10.1007/978-3-319-43425-4\_{}17.

Přehled o publikaci