Další formáty:
BibTeX
LaTeX
RIS
@inproceedings{1377655,
author = {Makatun, Dzmitry and Lauret, Jerome and Rudová, Hana and Šumbera, Michal},
address = {USA},
booktitle = {2016 IEEE Symposium Series on Computational Intelligence (SSCI)},
doi = {http://dx.doi.org/10.1109/SSCI.2016.7850083},
keywords = {Production; Processor scheduling; Distributed databases; Optimization; Computational modeling; Planning; Bandwidth},
howpublished = {tištěná verze "print"},
language = {eng},
location = {USA},
isbn = {978-1-5090-4240-1},
pages = {1-8},
publisher = {IEEE},
title = {Network Flows for Data Distribution and Computation},
year = {2016}
}
TY - JOUR
ID - 1377655
AU - Makatun, Dzmitry - Lauret, Jerome - Rudová, Hana - Šumbera, Michal
PY - 2016
TI - Network Flows for Data Distribution and Computation
PB - IEEE
CY - USA
SN - 9781509042401
KW - Production
KW - Processor scheduling
KW - Distributed databases
KW - Optimization
KW - Computational modeling
KW - Planning
KW - Bandwidth
N2 - An important class of modern big data applications is distributed data production in High Energy and Nuclear Physics (HENP). Such data intensive computations heavily rely on geographically distributed resources featuring hundreds of thousands CPUs and petabytes of storage. Unfortunately, classical job scheduling approaches either do not address all the aspects of the case or do not scale appropriately. Previously we have developed a new job scheduling approach dedicated to distributed data production, where the load balancing across sites is provided by forwarding data in peer-to-peer manner, but guided by a centrally created and periodically updated plan, aiming to achieve global optimality. Because the many HENP experiments utilize distributed storage, in this work we provide an important generalization of our approach to consider multiple sources of input data. The underlying network flow model is also extended to enable optimization on various additional criteria on top of the flow maximization making it versatile for a wide scope of potential use cases. In this study such additional optimization was used for more efficient reasoning with multiple data sources: balancing their usage and planning of the initial data distribution. Those two considerations allow to reduce an influence of network bottlenecks at early and late stages of data production. The simulations carried out in this work allow to test our approach towards a more general case of networks and servers not limited to specifics of HENP infrastructure. In all of the simulations our planner has shown a significant improvement in both average throughput and makespan against the typically used pull scheduling approach.
ER -
MAKATUN, Dzmitry, Jerome LAURET, Hana RUDOVÁ a Michal ŠUMBERA. Network Flows for Data Distribution and Computation. In \textit{2016 IEEE Symposium Series on Computational Intelligence (SSCI)}. USA: IEEE, 2016, s.~1-8. ISBN~978-1-5090-4240-1. Dostupné z: https://dx.doi.org/10.1109/SSCI.2016.7850083.
|
