Provenance-aware optimization of workload for distributed data
production

D 2017

Provenance-aware optimization of workload for distributed data production

MAKATUN, Dzmitry, Jerome LAURET, Hana RUDOVÁ a Michal ŠUMBERA

Základní údaje

Originální název

Provenance-aware optimization of workload for distributed data production

Autoři

MAKATUN, Dzmitry (112 Bělorusko), Jerome LAURET (840 Spojené státy), Hana RUDOVÁ (203 Česká republika, garant, domácí) a Michal ŠUMBERA (203 Česká republika)

Vydání

United Kingdom, Journal of Physics: Conference Series, vol. 898, od s. 1-8, 8 s. 2017

Nakladatel

Institute of Physics Publishing

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Velká Británie a Severní Irsko

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

tištěná verze "print"

Kód RIV

RIV/00216224:14330/17:00098484

Organizační jednotka

Fakulta informatiky

ISSN

DOI

http://dx.doi.org/10.1088/1742-6596/898/5/052038

Klíčová slova anglicky

data transfer planning; distributed data processing; Grid; network flows; data production

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 27. 8. 2019 12:19, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

Distributed data processing in High Energy and Nuclear Physics (HENP) is a prominent example of big data analysis. Having petabytes of data being processed at tens of computational sites with thousands of CPUs, standard job scheduling approaches either do not address well the problem complexity or are dedicated to one specific aspect of the problem only (CPU, network or storage). Previously we have developed a new job scheduling approach dedicated to distributed data production – an essential part of data processing in HENP (pre- processing in big data terminology). In this contribution, we discuss the load balancing with multiple data sources and data replication, present recent improvements made to our planner and provide results of simulations which demonstrate the advantage against standard scheduling policies for the new use case. Multi-source or provenance is common in computing models of many applications whereas the data may be copied to several destinations. The initial input data set would hence be already partially replicated to multiple locations and the task of the scheduler is to maximize overall computational throughput considering possible data movements and CPU allocation. The studies have shown that our approach can provide a significant gain in overall computational performance in a wide scope of simulations considering realistic size of computational Grid and various input data distribution.

Citovat

MAKATUN, Dzmitry, Jerome LAURET, Hana RUDOVÁ a Michal ŠUMBERA. Provenance-aware optimization of workload for distributed data production. In Journal of Physics: Conference Series, vol. 898. United Kingdom: Institute of Physics Publishing, 2017, s. 1-8. ISSN 1742-6588. Dostupné z: https://dx.doi.org/10.1088/1742-6596/898/5/052038.

@inproceedings{1396960,
   author = {Makatun, Dzmitry and Lauret, Jerome and Rudová, Hana and Šumbera, Michal},
   address = {United Kingdom},
   booktitle = {Journal of Physics: Conference Series, vol. 898},
   doi = {http://dx.doi.org/10.1088/1742-6596/898/5/052038},
   keywords = {data transfer planning; distributed data processing; Grid; network flows; data production},
   howpublished = {tištěná verze "print"},
   language = {eng},
   location = {United Kingdom},
   pages = {1-8},
   publisher = {Institute of Physics Publishing},
   title = {Provenance-aware optimization of workload for distributed data production},
   year = {2017}
}

TY  - JOUR
ID  - 1396960
AU  - Makatun, Dzmitry - Lauret, Jerome - Rudová, Hana - Šumbera, Michal
PY  - 2017
TI  - Provenance-aware optimization of workload for distributed data production
PB  - Institute of Physics Publishing
CY  - United Kingdom
KW  - data transfer planning
KW  - distributed data processing
KW  - Grid
KW  - network flows
KW  - data production
N2  - Distributed data processing in High Energy and Nuclear Physics (HENP) is a prominent example of big data analysis. Having petabytes of data being processed at tens of computational sites with thousands of CPUs, standard job scheduling approaches either do not address well the problem complexity or are dedicated to one specific aspect of the problem only (CPU, network or storage). Previously we have developed a new job scheduling approach dedicated to distributed data production – an essential part of data processing in HENP (pre- processing in big data terminology). In this contribution, we discuss the load balancing with multiple data sources and data replication, present recent improvements made to our planner and provide results of simulations which demonstrate the advantage against standard scheduling policies for the new use case. Multi-source or provenance is common in computing models of many applications whereas the data may be copied to several destinations. The initial input data set would hence be already partially replicated to multiple locations and the task of the scheduler is to maximize overall computational throughput considering possible data movements and CPU allocation. The studies have shown that our approach can provide a significant gain in overall computational performance in a wide scope of simulations considering realistic size of computational Grid and various input data distribution.
ER  -

MAKATUN, Dzmitry, Jerome LAURET, Hana RUDOVÁ a Michal ŠUMBERA. Provenance-aware optimization of workload for distributed data production. In \textit{Journal of Physics: Conference Series, vol. 898}. United Kingdom: Institute of Physics Publishing, 2017, s.~1-8. ISSN~1742-6588. Dostupné z: https://dx.doi.org/10.1088/1742-6596/898/5/052038.

Podrobný výpis o publikaci