Simulations and study of a new scheduling approach for
distributed data production

D 2016

Simulations and study of a new scheduling approach for distributed data production

MAKATUN, Dzmitry, Jerome LAURET, Hana RUDOVÁ a Michal ŠUMBERA

Základní údaje

Originální název

Simulations and study of a new scheduling approach for distributed data production

Autoři

MAKATUN, Dzmitry (112 Bělorusko), Jerome LAURET (840 Spojené státy), Hana RUDOVÁ (203 Česká republika, garant, domácí) a Michal ŠUMBERA (203 Česká republika)

Vydání

United Kingdom, Journal of Physics: Conference Series, vol. 762, od s. 1-7, 7 s. 2016

Nakladatel

Institute of Physics Publishing

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Velká Británie a Severní Irsko

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

tištěná verze "print"

Kód RIV

RIV/00216224:14330/16:00088810

Organizační jednotka

Fakulta informatiky

ISSN

DOI

http://dx.doi.org/10.1088/1742-6596/762/1/012023

UT WoS

000439689600023

Klíčová slova anglicky

data transfer planning; distributed data processing; Grid; network flows; data production

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 14. 5. 2020 15:33, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

Distributed data processing has found its application in many fields of science (High Energy and Nuclear Physics (HENP), astronomy, biology to name only those). We have focused our research on distributed data production, an essential part of computations in HENP. Using our previous experience, we have recently proposed a new scheduling approach for distributed data production which is based on the network flow maximization model. It has a polynomial complexity providing required scalability with respect to the size of computations. Our approach improves the overall data production throughput due to three factors: transfer input files in advance before their processing (allows to decrease I/O latency); balancing of the network traffic (includes splitting the load between several alternative transfer paths); and transfer files sequentially in a coordinated manner (allows to reduce the influence of possible network bottlenecks). In this contribution, we present the results of our new simulations based on the GridSim framework which is one of the commonly used tools in the field of distributed computations. In these simulations we study the behavior of standard scheduling approaches compared to our recently proposed approach in a realistic environment relying on the data from the STAR and ATLAS experiments and considering the influence of the background traffic. The final goal of the research is to integrate the proposed scheduling approach into the real data production framework. In order to achieve this we are constantly moving our simulations towards real use cases, study scalability of the model and the influence of the scheduling parameters on the quality of the solution.

Návaznosti

GAP202/12/0306, projekt VaV

Název: Dyschnet - Dynamické plánování a rozvrhování výpočetních a síťových zdrojů (Akronym: Dyschnet)

Investor: Grantová agentura ČR, Dyschnet - Dynamické plánování a rozvrhování výpočetních a síťových zdrojů

Citovat

MAKATUN, Dzmitry, Jerome LAURET, Hana RUDOVÁ a Michal ŠUMBERA. Simulations and study of a new scheduling approach for distributed data production. In Journal of Physics: Conference Series, vol. 762. United Kingdom: Institute of Physics Publishing, 2016, s. 1-7. ISSN 1742-6588. Dostupné z: https://dx.doi.org/10.1088/1742-6596/762/1/012023.

@inproceedings{1377625,
   author = {Makatun, Dzmitry and Lauret, Jerome and Rudová, Hana and Šumbera, Michal},
   address = {United Kingdom},
   booktitle = {Journal of Physics: Conference Series, vol. 762},
   doi = {http://dx.doi.org/10.1088/1742-6596/762/1/012023},
   keywords = {data transfer planning; distributed data processing; Grid; network flows; data production},
   howpublished = {tištěná verze "print"},
   language = {eng},
   location = {United Kingdom},
   pages = {1-7},
   publisher = {Institute of Physics Publishing},
   title = {Simulations and study of a new scheduling approach for distributed data production},
   year = {2016}
}

TY  - JOUR
ID  - 1377625
AU  - Makatun, Dzmitry - Lauret, Jerome - Rudová, Hana - Šumbera, Michal
PY  - 2016
TI  - Simulations and study of a new scheduling approach for distributed data production
PB  - Institute of Physics Publishing
CY  - United Kingdom
KW  - data transfer planning
KW  - distributed data processing
KW  - Grid
KW  - network flows
KW  - data production
N2  - Distributed data processing has found its application in many fields of science (High Energy and Nuclear Physics (HENP), astronomy, biology to name only those). We have focused our research on distributed data production, an essential part of computations in HENP. Using our previous experience, we have recently proposed a new scheduling approach for distributed data production which is based on the network flow maximization model. It has a polynomial complexity providing required scalability with respect to the size of computations. Our approach improves the overall data production throughput due to three factors: transfer input files in advance before their processing (allows to decrease I/O latency); balancing of the network traffic (includes splitting the load between several alternative transfer paths); and transfer files sequentially in a coordinated manner (allows to reduce the influence of possible network bottlenecks). In this contribution, we present the results of our new simulations based on the GridSim framework which is one of the commonly used tools in the field of distributed computations. In these simulations we study the behavior of standard scheduling approaches compared to our recently proposed approach in a realistic environment relying on the data from the STAR and ATLAS experiments and considering the influence of the background traffic. The final goal of the research is to integrate the proposed scheduling approach into the real data production framework. In order to achieve this we are constantly moving our simulations towards real use cases, study scalability of the model and the influence of the scheduling parameters on the quality of the solution.
ER  -

MAKATUN, Dzmitry, Jerome LAURET, Hana RUDOVÁ a Michal ŠUMBERA. Simulations and study of a new scheduling approach for distributed data production. In \textit{Journal of Physics: Conference Series, vol. 762}. United Kingdom: Institute of Physics Publishing, 2016, s.~1-7. ISSN~1742-6588. Dostupné z: https://dx.doi.org/10.1088/1742-6596/762/1/012023.

Podrobný výpis o publikaci