Diplomová práce

DevOps and MLOps Pipelines in a Secure Kubernetes Cluster

Bc. David Rusnák
Anotace

Práca sa zameriava na bezpečné a škálovateľné spracovanie lekárskych údajov, čím prispieva k poslaniu projektu AICope zameraného na správu zdravotníckych údajov orientovanú na pacienta. Systém využívajúci Kubernetes a Apache Airflow je nasadený v prostredí Sensitive Cloud. Tento systém predstavuje zásadný krok smerom k futuristickému stavu, ktorý zabezpečuje efektívny prístup k informáciám, prezentuje bezprecedentné poznatky a zachováva dôveru v bezpečné spracovanie údajov.

Abstract

The thesis focuses on the secure and scalable processing of medical data, contributing to the AICope project's mission for patient-centric healthcare data management. A system utilizing Kubernetes and Apache Airflow is deployed in the Sensitive Cloud environment. This system represents a crucial step towards an futuristic state, ensuring efficient information access, presenting unprecedented insights, and maintaining confidence in secure data handling.

Zadání práce
Context:
Patient records are a goldmine for various data analytics and machine learning applications aiming at gaining insights that could lead to more efficient treatments, inform innovative clinical decision support systems and enable patient empowerment. Yet these data are very sensitive and thus they cannot be processed as is. Pseudonymisation (replacing direct patient identifiers with anonymous IDs) can help, but such data still cannot be processed on standard computational infrastructures that are potentially open to attacks, while the secure infrastructure of the data providers (i.e., hospitals) typically does not scale to allow for deployment of modern analytical (e.g., machine learning) pipelines. This problem is further exacerbated in case of multi-site studies, involving multiple clinical data providers where federated analytics may be the only way to gain insights from the separate data silos. Infrastructures with tightly controlled access and clearly defined data transfer and processing protocols with corresponding security guarantees that still support the state of the art analytics, either on-site or federated, are therefore required.
Kubernetes (also K8s for short), a system for automated deployment of containerised applications can enable such infrastructures. And this is the problem the thesis will look into - how to use K8s in a secure HPC environment to deploy advanced, possibly federated analytical pipelines on sensitive patient data, following the state-of-the-art approaches in DevOps and MLOps.

Goals:
- Augment a prototype architecture for deploying machine learning models (including federated ones) and various auxiliary services using a K8s infrastructure in the SensitiveCloud environment maintained by the Institute of Computer Science at MU.
- This will involve contributing to new data and workflow pipelines using for instance the Apache Airflow framework, and implementing a proof-of-concept logging module for reproducibility of the analytical pipelines executed in the environment.
- Validate the work by its deployment as a part of the AIcope research project infrastructure.
- Write up the results in a thesis form.

Requirements:
- Keen interest in the topic.
- At least a minimal knowledge and previous hands-on experience with containerisation, DevOps and MLOps.
- While the thesis can be written and defended in Czech, its elaboration and presentation in English will be supported enthusiastically (the results may be disseminated to and used by partners in EU projects).
- Monthly (or more frequent, if needed) thesis progress review meetings with the supervisor will be expected. Other than that, there will be meetings with broader teams of researchers and developers involved in research projects associated with this topic.
- The student(s) will also be expected to develop and document any related code using the FI MU Gitlab platform, and (if applicable) re-use and interact with other related projects there.
Práce zkontrolována:
18. 12. 2023 13:45, doc. Mgr. Bc. Vít Nováček, PhD, učo 4049
Plný text práce
532,6 KB / soubor PDF
Jazyk práce
angličtina angličtina
Termín obhajoby
13. 2. 2024
Práce byla úspěšně obhájena

Vedoucí

doc. Mgr. Bc. Vít Nováček, PhD, učo 4049
KSUZD FI MU

Oponent

Maroš Lipták
USU Solutions Inc.

Konzultant

Bc. Daniel Plakinger, učo 482849
KSUZD FI MU

  • Přidání souboru

    Soubor nebo složku lze nahrát pomocí tlačítka Přidat.
  • Další operace se soubory

    Podrobnosti lze zjistit označením příslušného řádku.
  • Pohled pro experty

    Pro častou práci je možné zvolit režim Více možností.
  • Vyhledávání souborů

    Vyhledávaný výraz můžete zadat přímo do adresního řádku.
  • Rychlý přístup k souborům

    Pomocí funkce Nedávné je možné se rychle vrátit k právě prohlíženým souborům. Oblíbené soubory je také možné označit Hvězdičkou.