Peer-to-peer Cooperative Scheduling Architecture for National Grid Infrastructure

MATYSKA, Luděk, Miroslav RUDA and Šimon TÓTH. Peer-to-peer Cooperative Scheduling Architecture for National Grid Infrastructure. In Simon C.Lin, Eric Yen. Data Driven e-Science. Academia Sinica Grid Computing Centre, 2011, p. 105-118. ISBN 978-1-4419-8014-4.

Basic information

Original name

Peer-to-peer Cooperative Scheduling Architecture for National Grid Infrastructure

Authors

MATYSKA, Luděk, Miroslav RUDA and Šimon TÓTH

Edition

Data Driven e-Science, p. 105-118, 2011

Publisher

Academia Sinica Grid Computing Centre

---

Other information

Type of outcome

Stať ve sborníku

Confidentiality degree

není předmětem státního či obchodního tajemství

Publication form

printed version "print"

Organization unit

Faculty of Informatics

ISBN

978-1-4419-8014-4

UT WoS

000310340000008

Tags

International impact, Reviewed

---

Abstract

In English

For some ten years, the Czech National Grid Infrastructure MetaCentrum uses a single central PBSPro installation to schedule jobs across the country. This centralized approach keeps a full track about all the clusters, providing support for jobs spanning several sites, implementation for the fair-share policy and better overall control of the grid environment. Despite a steady progress in the increased stability and resilience to intermittent very short network failures, growing number of sites and processors makes this architecture, with a single point of failure and scalability limits, obsolete. As a result, a new scheduling architecture is proposed, which relies on higher autonomy of clusters. It is based on a peer to peer network of semi-independent schedulers for each site or even cluster. Each scheduler accepts jobs for the whole infrastructure, cooperating with other schedulers on implementation of global policies like central job accounting, fair-share, or submission of jobs across several sites. The scheduling system is integrated with the Magrathea system to support scheduling of virtual clusters, including the setup of their internal network, again eventually spanning several sites. On the other hand, each scheduler is local to one of several clusters and is able to directly control and submit jobs to them even if the connection of other scheduling peers is lost. In parallel to the change of the overall architecture, the scheduling system itself is being replaced. Instead of PBSPro, chosen originally for its declared support of large scale distributed environment, the new scheduling architecture is based on the open-source Torque system. The implementation and support for the most desired properties in PBSPro and Torque are discussed and the necessary modifications to Torque to support the MetaCentrum scheduling architecture are presented, too.