Evaluating Two Approaches to Assessing Student Progress in
Cybersecurity Exercises

D 2022

Evaluating Two Approaches to Assessing Student Progress in Cybersecurity Exercises

ŠVÁBENSKÝ, Valdemar; Richard WEISS; Jack COOK; Jan VYKOPAL; Pavel ČELEDA et. al.

Základní údaje

Originální název

Evaluating Two Approaches to Assessing Student Progress in Cybersecurity Exercises

Autoři

ŠVÁBENSKÝ, Valdemar (703 Slovensko, garant, domácí); Richard WEISS; Jack COOK; Jan VYKOPAL (203 Česká republika, domácí); Pavel ČELEDA (203 Česká republika, domácí); Jens MACHE; Radoslav CHUDOVSKÝ (703 Slovensko, domácí) a Ankur CHATTOPADHYAY

Vydání

New York, NY, USA, Proceedings of the 53rd ACM Technical Symposium on Computer Science Education (SIGCSE '22), od s. 787-793, 7 s. 2022

Nakladatel

ACM

Další údaje

Jazyk

angličtina

Typ výsledku

Stať ve sborníku

Obor

10201 Computer sciences, information science, bioinformatics

Stát vydavatele

Spojené státy

Utajení

není předmětem státního či obchodního tajemství

Forma vydání

elektronická verze "online"

Odkazy

YouTube video Preprint on ArXiv.org

Kód RIV

RIV/00216224:14610/22:00125013

Organizační jednotka

Ústav výpočetní techniky

ISBN

978-1-4503-9070-5

DOI

http://dx.doi.org/10.1145/3478431.3499414

UT WoS

000884263800114

EID Scopus

2-s2.0-85126106998

Klíčová slova anglicky

cybersecurity education; command-line history; educational data mining; learning analytics; assessment; modeling

Štítky

best3, core_A, firank_A, rivok

Příznaky

Mezinárodní význam, Recenzováno

Změněno: 30. 3. 2023 13:22, Mgr. Alena Mokrá

Anotace

V originále

Cybersecurity students need to develop practical skills such as using command-line tools. Hands-on exercises are the most direct way to assess these skills, but assessing students' mastery is a challenging task for instructors. We aim to alleviate this issue by modeling and visualizing student progress automatically throughout the exercise. The progress is summarized by graph models based on the shell commands students typed to achieve discrete tasks within the exercise. We implemented two types of models and compared them using data from 46 students at two universities. To evaluate our models, we surveyed 22 experienced computing instructors and qualitatively analyzed their responses. The majority of instructors interpreted the graph models effectively and identified strengths, weaknesses, and assessment use cases for each model. Based on the evaluation, we provide recommendations to instructors and explain how our graph models innovate teaching and promote further research. The impact of this paper is threefold. First, it demonstrates how multiple institutions can collaborate to share approaches to modeling student progress in hands-on exercises. Second, our modeling techniques generalize to data from different environments to support student assessment, even outside the cybersecurity domain. Third, we share the acquired data and open-source software so that others can use the models in their classes or research.

Návaznosti

EF16_019/0000822, projekt VaV

Název: Centrum excelence pro kyberkriminalitu, kyberbezpečnost a ochranu kritických informačních infrastruktur

MUNI/A/1520/2021, interní kód MU

Název: Aplikovaný výzkum na FI: Bezpečnost počítačových systémů, softwarových architektur kritických infrastruktur a zpracování rozsáhlých senzorových dat

Investor: Masarykova univerzita, Aplikovaný výzkum na FI: Bezpečnost počítačových systémů, softwarových architektur kritických infrastruktur a zpracování rozsáhlých senzorových dat

Přiložené soubory

2022-SIGCSE-evaluating-two-approaches-assessing-student-progress-cybersecurity-exercises-paper.pdf Verze souboru

2022-SIGCSE-evaluating-two-approaches-assessing-student-progress-cybersecurity-exercises-slides.pdf

Citovat

ŠVÁBENSKÝ, Valdemar; Richard WEISS; Jack COOK; Jan VYKOPAL; Pavel ČELEDA; Jens MACHE; Radoslav CHUDOVSKÝ a Ankur CHATTOPADHYAY. Evaluating Two Approaches to Assessing Student Progress in Cybersecurity Exercises. Online. In Proceedings of the 53rd ACM Technical Symposium on Computer Science Education (SIGCSE '22). New York, NY, USA: ACM, 2022, s. 787-793. ISBN 978-1-4503-9070-5. Dostupné z: https://dx.doi.org/10.1145/3478431.3499414.

@inproceedings{1797699,
   author = {Švábenský, Valdemar and Weiss, Richard and Cook, Jack and Vykopal, Jan and Čeleda, Pavel and Mache, Jens and Chudovský, Radoslav and Chattopadhyay, Ankur},
   address = {New York, NY, USA},
   booktitle = {Proceedings of the 53rd ACM Technical Symposium on Computer Science Education (SIGCSE '22)},
   doi = {http://dx.doi.org/10.1145/3478431.3499414},
   keywords = {cybersecurity education; command-line history; educational data mining; learning analytics; assessment; modeling},
   howpublished = {elektronická verze "online"},
   language = {eng},
   location = {New York, NY, USA},
   isbn = {978-1-4503-9070-5},
   pages = {787-793},
   publisher = {ACM},
   title = {Evaluating Two Approaches to Assessing Student Progress in Cybersecurity Exercises},
   url = {https://youtu.be/rzJ0AG_FsvU},
   year = {2022}
}

TY  - CONF
ID  - 1797699
AU  - Švábenský, Valdemar - Weiss, Richard - Cook, Jack - Vykopal, Jan - Čeleda, Pavel - Mache, Jens - Chudovský, Radoslav - Chattopadhyay, Ankur
PY  - 2022
TI  - Evaluating Two Approaches to Assessing Student Progress in Cybersecurity Exercises
PB  - ACM
CY  - New York, NY, USA
SN  - 9781450390705
KW  - cybersecurity education
KW  - command-line history
KW  - educational data mining
KW  - learning analytics
KW  - assessment
KW  - modeling
UR  - https://youtu.be/rzJ0AG_FsvU
N2  - Cybersecurity students need to develop practical skills such as using command-line tools. Hands-on exercises are the most direct way to assess these skills, but assessing students' mastery is a challenging task for instructors. We aim to alleviate this issue by modeling and visualizing student progress automatically throughout the exercise. The progress is summarized by graph models based on the shell commands students typed to achieve discrete tasks within the exercise. We implemented two types of models and compared them using data from 46 students at two universities. To evaluate our models, we surveyed 22 experienced computing instructors and qualitatively analyzed their responses. The majority of instructors interpreted the graph models effectively and identified strengths, weaknesses, and assessment use cases for each model. Based on the evaluation, we provide recommendations to instructors and explain how our graph models innovate teaching and promote further research. The impact of this paper is threefold. First, it demonstrates how multiple institutions can collaborate to share approaches to modeling student progress in hands-on exercises. Second, our modeling techniques generalize to data from different environments to support student assessment, even outside the cybersecurity domain. Third, we share the acquired data and open-source software so that others can use the models in their classes or research.
ER  -

ŠVÁBENSKÝ, Valdemar; Richard WEISS; Jack COOK; Jan VYKOPAL; Pavel ČELEDA; Jens MACHE; Radoslav CHUDOVSKÝ a Ankur CHATTOPADHYAY. Evaluating Two Approaches to Assessing Student Progress in Cybersecurity Exercises. Online. In \textit{Proceedings of the 53rd ACM Technical Symposium on Computer Science Education (SIGCSE '22)}. New York, NY, USA: ACM, 2022, s.~787-793. ISBN~978-1-4503-9070-5. Dostupné z: https://dx.doi.org/10.1145/3478431.3499414.

Přehled o publikaci