Planning of distributed data production for High Energy and
Nuclear Physics

J 2018

Planning of distributed data production for High Energy and Nuclear Physics

MAKATUN, Dzmitry, Jerome LAURET a Hana RUDOVÁ

Základní údaje

Originální název

Planning of distributed data production for High Energy and Nuclear Physics

Autoři

MAKATUN, Dzmitry (112 Bělorusko), Jerome LAURET (840 Spojené státy) a Hana RUDOVÁ (203 Česká republika, garant, domácí)

Vydání

Cluster Computing, 2018, 1386-7857

Další údaje

Jazyk

angličtina

Typ výsledku

Článek v odborném periodiku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Spojené státy

Utajení

není předmětem státního či obchodního tajemství

Odkazy

URL

Impakt faktor

Impact factor: 1.851

Kód RIV

RIV/00216224:14330/18:00100898

Organizační jednotka

Fakulta informatiky

DOI

http://dx.doi.org/10.1007/s10586-018-2834-3

UT WoS

000457276800012

Klíčová slova anglicky

Load balancing; Job scheduling; Planning; Network flow; Distributed computing; Large scale computing; Grid; Data intensive applications; Data production; Big data

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 31. 5. 2022 17:33, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

Modern experiments in High Energy and Nuclear Physics heavily rely on distributed computations using multiple computational facilities across the world. One of the essential types of the computations is a distributed data production where petabytes of raw files from a single source has to be processed once (per production campaign) using thousands of CPUs at distant locations and the output has to be transferred back to that source. The data distribution over a large system does not necessary match the distribution of storage, network and CPU capacity. Therefore, bottlenecks may appear and lead to increased latency and degraded performance. In this paper we propose a new scheduling approach for distributed data production which is based on the network flow maximization model. In our approach a central planner defines how much input and output data should be transferred over each network link in order to maximize the computational throughput. Such plans are created periodically for a fixed planning time interval using up-to-date information on network, storage and CPU resources. The centrally created plans are executed in a distributed manner by dedicated services running at participating sites. Our simulations based on the log records from the data production framework of the experiment STAR (Solenoid Tracker at RHIC) have shown that the proposed model systematically provides a better performance compared to the simulated traditional techniques.

Návaznosti

GAP202/12/0306, projekt VaV

Název: Dyschnet - Dynamické plánování a rozvrhování výpočetních a síťových zdrojů (Akronym: Dyschnet)

Investor: Grantová agentura ČR, Dyschnet - Dynamické plánování a rozvrhování výpočetních a síťových zdrojů

Citovat

MAKATUN, Dzmitry, Jerome LAURET a Hana RUDOVÁ. Planning of distributed data production for High Energy and Nuclear Physics. Cluster Computing. 2018, roč. 21, č. 4, s. 1949-1965. ISSN 1386-7857. Dostupné z: https://dx.doi.org/10.1007/s10586-018-2834-3.

@article{1414263,
   author = {Makatun, Dzmitry and Lauret, Jerome and Rudová, Hana},
   article_number = {4},
   doi = {http://dx.doi.org/10.1007/s10586-018-2834-3},
   keywords = {Load balancing; Job scheduling; Planning; Network flow; Distributed computing; Large scale computing; Grid; Data intensive applications; Data production; Big data},
   language = {eng},
   issn = {1386-7857},
   journal = {Cluster Computing},
   title = {Planning of distributed data production for High Energy and Nuclear Physics},
   url = {http://dx.doi.org/10.1007/s10586-018-2834-3},
   volume = {21},
   year = {2018}
}

TY  - JOUR
ID  - 1414263
AU  - Makatun, Dzmitry - Lauret, Jerome - Rudová, Hana
PY  - 2018
TI  - Planning of distributed data production for High Energy and Nuclear Physics
JF  - Cluster Computing
VL  - 21
IS  - 4
SP  - 1949-1965
EP  - 1949-1965
SN  - 13867857
KW  - Load balancing
KW  - Job scheduling
KW  - Planning
KW  - Network flow
KW  - Distributed computing
KW  - Large scale computing
KW  - Grid
KW  - Data intensive applications
KW  - Data production
KW  - Big data
UR  - http://dx.doi.org/10.1007/s10586-018-2834-3
N2  - Modern experiments in High Energy and Nuclear Physics heavily rely on distributed computations using multiple computational facilities across the world. One of the essential types of the computations is a distributed data production where petabytes of raw files from a single source has to be processed once (per production campaign) using thousands of CPUs at distant locations and the output has to be transferred back to that source. The data distribution over a large system does not necessary match the distribution of storage, network and CPU capacity. Therefore, bottlenecks may appear and lead to increased latency and degraded performance. In this paper we propose a new scheduling approach for distributed data production which is based on the network flow maximization model. In our approach a central planner defines how much input and output data should be transferred over each network link in order to maximize the computational throughput. Such plans are created periodically for a fixed planning time interval using up-to-date information on network, storage and CPU resources. The centrally created plans are executed in a distributed manner by dedicated services running at participating sites. Our simulations based on the log records from the data production framework of the experiment STAR (Solenoid Tracker at RHIC) have shown that the proposed model systematically provides a better performance compared to the simulated traditional techniques.
ER  -

MAKATUN, Dzmitry, Jerome LAURET a Hana RUDOVÁ. Planning of distributed data production for High Energy and Nuclear Physics. \textit{Cluster Computing}. 2018, roč.~21, č.~4, s.~1949-1965. ISSN~1386-7857. Dostupné z: https://dx.doi.org/10.1007/s10586-018-2834-3.

Podrobný výpis o publikaci