Real-life Experience with Major Reconfiguration of Job
Scheduling System

D 2017

Real-life Experience with Major Reconfiguration of Job Scheduling System

KLUSÁČEK, Dalibor; Šimon TÓTH a Gabriela PODOLNÍKOVÁ

Základní údaje

Originální název

Real-life Experience with Major Reconfiguration of Job Scheduling System

Autoři

KLUSÁČEK, Dalibor; Šimon TÓTH a Gabriela PODOLNÍKOVÁ

Vydání

Cham, Switzerland, 20th International Workshop on Job Scheduling Strategies for Parallel Processing, JSSPP 2016, od s. 83-101, 19 s. 2017

Nakladatel

Springer

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Švýcarsko

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

tištěná verze "print"

Odkazy

URL

Impakt faktor

Impact factor: 0.402 v roce 2005

Kód RIV

RIV/00216224:14330/17:00094513

Organizační jednotka

Fakulta informatiky

ISBN

978-3-319-61755-8

ISSN

DOI

https://doi.org/10.1007/978-3-319-61756-5_5

UT WoS

000441213000005

EID Scopus

2-s2.0-85029436587

Klíčová slova anglicky

Scheduling; Fairness; Queue; Workload; Heterogeneity

Štítky

core_B, firank_A

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 13. 5. 2020 19:18, RNDr. Pavel Šmerk, Ph.D.

Anotace

V originále

This work describes the goals and impacts of a large reconfiguration of the job scheduling system, used in the Czech National Grid and Cloud infrastructure MetaCentrum, which was implemented in early 2014. MetaCentrum serves a varied user-base consisting of both individual users and research groups. This imposes strict requirements on the robustness of job scheduling algorithms being employed, as the system must be capable of assigning a highly heterogeneous set of workloads to a similarly heterogeneous set of computational resources. Primary goals for MetaCentrum were always to provide efficient and fair resource utilization with respect to different users in the system. During the last few years, MetaCentrum has gone through a period of rapid growth (1,500 CPU cores in 2009 vs. 10,600 CPU cores in 2014) forcing us to re-evaluate our scheduling approaches, as the ``old" configuration no longer satisfied our utilization and fairness demands. This re-evaluation was supported by a significant body of research, which included the proposal of new scheduling approaches as well as detailed simulations based on real-life complex workload traces. First, a new multi-resource aware fair-sharing algorithm (based on our recent research) was deployed, with the goal of improving fairness with respect to the growing heterogeneity of resources and users' workloads. Second, the queue configuration of the entire system was completely reworked in order to decrease resource fragmentation and improve the utilization and the impact of fairness policies. This paper summarizes the effects of these changes using real data from the production system. Moreover, we publish complex workload from MetaCentrum that was used in this paper, since it represents a valuable source of data concerning a highly heterogeneous production system. Last but not least, we present our advanced job scheduling simulator which is routinely used for testing of new scheduling strategies prior their deployment in the real system.

Návaznosti

GAP202/12/0306, projekt VaV

Název: Dyschnet - Dynamické plánování a rozvrhování výpočetních a síťových zdrojů (Akronym: Dyschnet)

Investor: Grantová agentura ČR, Dyschnet - Dynamické plánování a rozvrhování výpočetních a síťových zdrojů

MUNI/A/1159/2014, interní kód MU

Název: Rozsáhlé výpočetní systémy: modely, aplikace a verifikace IV.

Investor: Masarykova univerzita, Rozsáhlé výpočetní systémy: modely, aplikace a verifikace IV., DO R. 2020_Kategorie A - Specifický výzkum - Studentské výzkumné projekty

Citovat

KLUSÁČEK, Dalibor; Šimon TÓTH a Gabriela PODOLNÍKOVÁ. Real-life Experience with Major Reconfiguration of Job Scheduling System. In Walfredo Cirne and Narayan Desai. 20th International Workshop on Job Scheduling Strategies for Parallel Processing, JSSPP 2016. Cham, Switzerland: Springer, 2017, s. 83-101. ISBN 978-3-319-61755-8. Dostupné z: https://doi.org/10.1007/978-3-319-61756-5_5.

@inproceedings{1309343,
   author = {Klusáček, Dalibor and Tóth, Šimon and Podolníková, Gabriela},
   address = {Cham, Switzerland},
   booktitle = {20th International Workshop on Job Scheduling Strategies for Parallel Processing, JSSPP 2016},
   doi = {https://doi.org/10.1007/978-3-319-61756-5_5},
   editor = {Walfredo Cirne and Narayan Desai},
   keywords = {Scheduling; Fairness; Queue; Workload; Heterogeneity},
   howpublished = {tištěná verze "print"},
   language = {eng},
   location = {Cham, Switzerland},
   isbn = {978-3-319-61755-8},
   pages = {83-101},
   publisher = {Springer},
   title = {Real-life Experience with Major Reconfiguration of Job Scheduling System},
   url = {http://www.cs.huji.ac.il/~feit/parsched/jsspp15/p5-klusacek.pdf},
   year = {2017}
}

TY  - CONF
ID  - 1309343
AU  - Klusáček, Dalibor - Tóth, Šimon - Podolníková, Gabriela
PY  - 2017
TI  - Real-life Experience with Major Reconfiguration of Job Scheduling System
PB  - Springer
CY  - Cham, Switzerland
SN  - 9783319617558
KW  - Scheduling
KW  - Fairness
KW  - Queue
KW  - Workload
KW  - Heterogeneity
UR  - http://www.cs.huji.ac.il/~feit/parsched/jsspp15/p5-klusacek.pdf
L2  - http://www.cs.huji.ac.il/~feit/parsched/jsspp15/p5-klusacek.pdf
N2  - This work describes the goals and impacts of a large reconfiguration of the job scheduling system, used in the Czech National Grid and Cloud infrastructure MetaCentrum, which was implemented in early 2014. MetaCentrum serves a varied user-base consisting of both individual users and research groups. This imposes strict requirements on the robustness of job scheduling algorithms being employed, as the system must be capable of assigning a highly heterogeneous set of workloads to a similarly heterogeneous set of computational resources. Primary goals for MetaCentrum were always to provide efficient and fair resource utilization with respect to different users in the system. During the last few years, MetaCentrum has gone through a period of rapid growth (1,500 CPU cores in 2009 vs. 10,600 CPU cores in 2014) forcing us to re-evaluate our scheduling approaches, as the ``old" configuration no longer satisfied our utilization and fairness demands. This re-evaluation was supported by a significant body of research, which included the proposal of new scheduling approaches as well as detailed simulations based on real-life complex workload traces. First, a new multi-resource aware fair-sharing algorithm (based on our recent research) was deployed, with the goal of improving fairness with respect to the growing heterogeneity of resources and users' workloads. Second, the queue configuration of the entire system was completely reworked in order to decrease resource fragmentation and improve the utilization and the impact of fairness policies. This paper summarizes the effects of these changes using real data from the production system. Moreover, we publish complex workload from MetaCentrum that was used in this paper, since it represents a valuable source of data concerning a highly heterogeneous production system. Last but not least, we present our advanced job scheduling simulator which is routinely used for testing of new scheduling strategies prior their deployment in the real system.
ER  -

KLUSÁČEK, Dalibor; Šimon TÓTH a Gabriela PODOLNÍKOVÁ. Real-life Experience with Major Reconfiguration of Job Scheduling System. In Walfredo Cirne and Narayan Desai. \textit{20th International Workshop on Job Scheduling Strategies for Parallel Processing, JSSPP 2016}. Cham, Switzerland: Springer, 2017, s.~83-101. ISBN~978-3-319-61755-8. Dostupné z: https://doi.org/10.1007/978-3-319-61756-5\_{}5.

Přehled o publikaci