Timely Feedback in Unstructured Cybersecurity Exercises
Jan Vykopal, Radek Ošlejšek, Karolína Burská, Kristína Zákopčanová
vykopal@ics.muni.cz, oslejsek@fi.muni.cz, {burska|zakopcanova}@mail.muni.cz
February 22, 2018
Outline
Cyber Range and Defence Exercises
Introduction to Cyber Ranges
Cyber defence exercise
Experiment
Scoring Timeline for Feedback
Post-exercise & Scoring Timeline Survey
Discussion
Vykopal, Ošlejšek, Burská, Zákopčanová ·Timely Feedback in CDX ·February 22, 2018 2 / 10
Cyber Range and Defence Exercises Introduction to Cyber Ranges
Introduction
Global shortage of cybersecurity workers with the skills to prevent or respond to cyber
security attacks ->
Cyber defence exercises (CDX)
unstructured, step-by-step hands-on trainings
enable participants to experience cyber attacks first-hand with real-life limitations
intensive, short-term events lasting several days
... feedback from these exercises?
Vykopal, Ošlejšek, Burská, Zákopčanová ·Timely Feedback in CDX ·February 22, 2018 3 / 10
Cyber Range and Defence Exercises Introduction to Cyber Ranges
Introduction
Figure: A scoring table, the only feedback for the learners
Vykopal, Ošlejšek, Burská, Zákopčanová ·Timely Feedback in CDX ·February 22, 2018 4 / 10
Cyber Range and Defence Exercises Cyber defence exercise
Blue Team Red Team
- attack
- scan
- penetrate
- secure
- monitor
- defense
Green Team
- maintain
- repair
- fix
White Team
- rules
- score
- guide
Figure: Exercise participants, their interactions and tasks.
Vykopal, Ošlejšek, Burská, Zákopčanová ·Timely Feedback in CDX ·February 22, 2018 5 / 10
Cyber Range and Defence Exercises Cyber defence exercise
Exercise Phases
Table: Phases of the exercise with time allocation
Order Phase Duration Day
1 Exercise familirization 3 hrs 1
2 Actual exercise 6 hrs 2
3 Post-exercise survey 5 mins 2
4 Break 25 mins 2
5 Scoring timeline interaction 10 mins 2
6 Scoring timeline survey 5 mins 2
7 Quick exercise debriefing 15 mins 2
Vykopal, Ošlejšek, Burská, Zákopčanová ·Timely Feedback in CDX ·February 22, 2018 5 / 10
Experiment Scoring Timeline for Feedback
Scoring Timeline
Figure: Scoring timeline providing personalized feedback for each team.
Vykopal, Ošlejšek, Burská, Zákopčanová ·Timely Feedback in CDX ·February 22, 2018 6 / 10
Experiment Post-exercise & Scoring Timeline Survey
Post-exercise & Scoring Timeline Survey
E1 My knowledge and skills were sufficient.
E2 I found exercise difficult for me.
E3 Exercise was well organized and struc-
tured.
E4 Exercise was beneficial and useful to me.
F1 The scoring timeline of my team displayed
after the end of the exercise provided useful
feedback.
F2 Do you have any comments on the scoring
timeline?
1 2 3 4 5
F1
E4
E3
E2
E1
Figure: Answers distribution E1 – E4 and F1.
1 = strongly disagree, 5 = completely agree
Vykopal, Ošlejšek, Burská, Zákopčanová ·Timely Feedback in CDX ·February 22, 2018 7 / 10
Experiment Discussion
Discussion
Teams sought out feedback regardless of their achieved score
A need for more detail according to survey question F2 ("Do you have any comments
on the scoring timeline?")
Benefits for instructors which lie in reflection from the learners
Two limitations of the study – small group of participants, simplicity of the timeline
evaluation survey
Vykopal, Ošlejšek, Burská, Zákopčanová ·Timely Feedback in CDX ·February 22, 2018 8 / 10
Experiment Discussion
Conclusion
Motivation – very limited and/or delayed feedback in CDX
-> limited opportunity to learn from the mistakes
Our experiment: an analysis of a simple survey and an interaction with a new tool
Feedback gathering was automatized; only limited level of detail
The new feature was valued by learners
Future work
Learners would welcome more detailed feedback
Not only show the point gain/loss, but also a comment from the instructor
Vykopal, Ošlejšek, Burská, Zákopčanová ·Timely Feedback in CDX ·February 22, 2018 9 / 10
Experiment Discussion
Any questions?
Contact e-mail: vykopal@ics.muni.cz, oslejsek@fi.muni.cz, {burska|zakopcanova}@mail.muni.cz
Vykopal, Ošlejšek, Burská, Zákopčanová ·Timely Feedback in CDX ·February 22, 2018 10 / 10