Network Flows for Data Distribution and Computation

D 2016

Network Flows for Data Distribution and Computation

MAKATUN, Dzmitry, Jerome LAURET, Hana RUDOVÁ and Michal ŠUMBERA

Basic information

Original name

Network Flows for Data Distribution and Computation

Authors

MAKATUN, Dzmitry (112 Belarus), Jerome LAURET (840 United States of America), Hana RUDOVÁ (203 Czech Republic, guarantor, belonging to the institution) and Michal ŠUMBERA (203 Czech Republic)

Edition

USA, 2016 IEEE Symposium Series on Computational Intelligence (SSCI), p. 1-8, 8 pp. 2016

Publisher

IEEE

Other information

Language

English

Type of outcome

Stať ve sborníku

Field of Study

10201 Computer sciences, information science, bioinformatics

Country of publisher

United States of America

Confidentiality degree

není předmětem státního či obchodního tajemství

Publication form

printed version "print"

RIV identification code

RIV/00216224:14330/16:00094080

Organization unit

Faculty of Informatics

ISBN

978-1-5090-4240-1

DOI

http://dx.doi.org/10.1109/SSCI.2016.7850083

UT WoS

000400488301124

Keywords in English

Production; Processor scheduling; Distributed databases; Optimization; Computational modeling; Planning; Bandwidth

Abstract

V originále

An important class of modern big data applications is distributed data production in High Energy and Nuclear Physics (HENP). Such data intensive computations heavily rely on geographically distributed resources featuring hundreds of thousands CPUs and petabytes of storage. Unfortunately, classical job scheduling approaches either do not address all the aspects of the case or do not scale appropriately. Previously we have developed a new job scheduling approach dedicated to distributed data production, where the load balancing across sites is provided by forwarding data in peer-to-peer manner, but guided by a centrally created and periodically updated plan, aiming to achieve global optimality. Because the many HENP experiments utilize distributed storage, in this work we provide an important generalization of our approach to consider multiple sources of input data. The underlying network flow model is also extended to enable optimization on various additional criteria on top of the flow maximization making it versatile for a wide scope of potential use cases. In this study such additional optimization was used for more efficient reasoning with multiple data sources: balancing their usage and planning of the initial data distribution. Those two considerations allow to reduce an influence of network bottlenecks at early and late stages of data production. The simulations carried out in this work allow to test our approach towards a more general case of networks and servers not limited to specifics of HENP infrastructure. In all of the simulations our planner has shown a significant improvement in both average throughput and makespan against the typically used pull scheduling approach.

Citovat

MAKATUN, Dzmitry, Jerome LAURET, Hana RUDOVÁ and Michal ŠUMBERA. Network Flows for Data Distribution and Computation. In 2016 IEEE Symposium Series on Computational Intelligence (SSCI). USA: IEEE, 2016, p. 1-8. ISBN 978-1-5090-4240-1. Available from: https://dx.doi.org/10.1109/SSCI.2016.7850083.

@inproceedings{1377655,
   author = {Makatun, Dzmitry and Lauret, Jerome and Rudová, Hana and Šumbera, Michal},
   address = {USA},
   booktitle = {2016 IEEE Symposium Series on Computational Intelligence (SSCI)},
   doi = {http://dx.doi.org/10.1109/SSCI.2016.7850083},
   keywords = {Production; Processor scheduling; Distributed databases; Optimization; Computational modeling; Planning; Bandwidth},
   howpublished = {tištěná verze "print"},
   language = {eng},
   location = {USA},
   isbn = {978-1-5090-4240-1},
   pages = {1-8},
   publisher = {IEEE},
   title = {Network Flows for Data Distribution and Computation},
   year = {2016}
}

TY  - JOUR
ID  - 1377655
AU  - Makatun, Dzmitry - Lauret, Jerome - Rudová, Hana - Šumbera, Michal
PY  - 2016
TI  - Network Flows for Data Distribution and Computation
PB  - IEEE
CY  - USA
SN  - 9781509042401
KW  - Production
KW  - Processor scheduling
KW  - Distributed databases
KW  - Optimization
KW  - Computational modeling
KW  - Planning
KW  - Bandwidth
N2  - An important class of modern big data applications is distributed data production in High Energy and Nuclear Physics (HENP). Such data intensive computations heavily rely on geographically distributed resources featuring hundreds of thousands CPUs and petabytes of storage. Unfortunately, classical job scheduling approaches either do not address all the aspects of the case or do not scale appropriately. Previously we have developed a new job scheduling approach dedicated to distributed data production, where the load balancing across sites is provided by forwarding data in peer-to-peer manner, but guided by a centrally created and periodically updated plan, aiming to achieve global optimality. Because the many HENP experiments utilize distributed storage, in this work we provide an important generalization of our approach to consider multiple sources of input data. The underlying network flow model is also extended to enable optimization on various additional criteria on top of the flow maximization making it versatile for a wide scope of potential use cases. In this study such additional optimization was used for more efficient reasoning with multiple data sources: balancing their usage and planning of the initial data distribution. Those two considerations allow to reduce an influence of network bottlenecks at early and late stages of data production. The simulations carried out in this work allow to test our approach towards a more general case of networks and servers not limited to specifics of HENP infrastructure. In all of the simulations our planner has shown a significant improvement in both average throughput and makespan against the typically used pull scheduling approach.
ER  -

MAKATUN, Dzmitry, Jerome LAURET, Hana RUDOVÁ and Michal ŠUMBERA. Network Flows for Data Distribution and Computation. In \textit{2016 IEEE Symposium Series on Computational Intelligence (SSCI)}. USA: IEEE, 2016, p.~1-8. ISBN~978-1-5090-4240-1. Available from: https://dx.doi.org/10.1109/SSCI.2016.7850083.

Detailed Information on Publication Record