14/10/2019
1
PV182
Human Computer Interaction
Lecture 13
Brain-Computer Interfaces
Fotis Liarokapis
liarokap@fi.muni.cz
14th October 2019
Aim
• Investigate brain activity in VEs
– Perception
– Immersion
– Cognitive workload
– Presence
• Effectively interact with VEs
– MI, P300
BCI Categories
fMRI
fNIRS
MEG EEG
Brain-Computer Interfaces
• Brain-Computer Interface (BCI) or Brain–
Machine Interface (BMI), is a direct way of
communication between the brain and a
computer system
Invasive Partially-Invasive Non-Invasive
Principles of EEG The 10-20 System
• The international 10-20 system describes the
electrode placement on the scalp for EEG tests
or experiments
14/10/2019
2
EEG-based BCI Feedback
• Event related potential (P300)
– Reflect processes involved in stimulus evaluation or
categorization
• Sensorimotor rhythms (SMR)
– Oscillatory idle rhythm of synchronized
electromagnetic brain activity
• Steady State Visually Evoked Potentials (SSVEP)
– Signals that are natural responses to visual
stimulation at specific freq
BCI Illiteracy
• Around 20 % of BCI users do not obtain reliable
BCI control (Tan and Nijholt, 2010)
• Investigation of BCI illiteracy can lead to:
– Avoid unnecessary training sessions
– Develop co-adaptive learning strategies to improve
BCI illiteracy
– Understand neurophysiological-basis of BCI illiteracy
– Build better BCI systems
Classification Issues
• Differences in brain anatomy may yield very
variable signal quality
• Large muscle artefacts
How to Improve BCI Illiteracy
• Improve classification accuracy
• Change paradigm
• Change neuroimaging technique
• Combine neuroimaging techniques
• Combine paradigms
BCI Interaction in
Games
Commercial BCI headsets
• Non-invasive BCI’s most commonly use EEG:
– Portability
– Low set-up cost
– Easy of use
• Low-cost BCI headsets are used the last 5-7
years
12
14/10/2019
3
BCIs and Computer Games Methodology
• Interaction
– Cognitive functions (brainwaves) are used to move
the forwards/backwards
– Expressive functions are used to steer left/right
• When the user blinks accordingly
• Profile training using Control Panel for 60s
(push/pull actions plus blink calibration)
– Navigating the 3D robot inside the maze to a predefined
waypoint (increasing users cognitive
workload)
• Evaluation with 30 users
Videos
3D Maze Game RomaNova Game
Liarokapis, F., Debattista, K., Vourvopoulos, A., Petridis, P., Ene, A., Comparing interaction techniques for serious games through brain-computer
interfaces: A user perception evaluation study, Entertainment Computing, Elsevier, 5(4): 391-399, 2014.
Comparison of Questionnaires
• No significant differences for the ability to
control, responsiveness, interaction and
naturality of experience were found
– Can be explained by the similar difficulty of the
BCI task
Comparison of Questionnaire & EEG
• Questionnaire
– 16/31 (51%) users have
reported through their
answers that
they were engaged to
the game
• EEG
– 9 out 31 users found
with increased Beta
activity
– That’s 29% of the users
that scored high on the
engagement related
questions
• This could mean that whatever the users think about
their status is different on what actually was
recorded through the EEG
– Taking in good fain that the headset measured accurately
Multimodal BCI
Games
Liarokapis, F., Vourvopoulos, A., Ene, A. Examining User Experiences Through A Multimodal BCI Puzzle, Proc. of the 19th
International Conference on Information Visualisation (IV 2015), IEEE Computer Society, Barcelona, Spain, 21-24 July, 488-
493, 2015. (DOI: 10.1109/iV.2015.87)
14/10/2019
4
Multimodal Games
• The game is multimodal, supporting a “BCI input”
and a “no BCI input” mode
• In the latter, meditation is defaulted at 50% of its
maximum possible value
– Speed is only affected by the number of cleared lines
• An instance of the game depends on:
– Name of the player
– Log’s creation timestamp
– Meditation
Liarokapis, F., Vourvopoulos, A., Ene, A. Examining User Experiences Through A Multimodal BCI Puzzle, Proc. of the 19th International
Conference on Information Visualisation (IV 2015), IEEE Computer Society, Barcelona, Spain, 21-24 July, 488-493, 2015.
Video
20
Evaluation Procedure
• Evaluated by 30 volunteers
– Selected by random sampling
– Duration was approximately 30 minutes
– 73.33% males, 26.67% females
• The dominant age group is 18-25 with 80%
– 10% only aged 26-33
• 83.3% participants reported using the computer
to a very high degree in their daily activities
– However, in terms of gaming experience the
percentage drops to 23.33% 21
EEG Rhythms Log
• Significant correlations were found for attention
• Decreasing Theta (r = -0.2885, p < 0.05)
– Theta is usually linked to inefficiency and daydreaming
• High Alpha (r = -0.1841, p < 0.05)
– Alpha rhythms attenuate with drowsiness, concentration,
stimulation or visual fixation
• High Gamma (r = -0.1589, p < 0.05)
– High gamma oscillations have been observed in a variety
of different purpose neuro-anatomical domains including
informationprocessing
22
Conclusions
• More experienced gamers did not notice the speed
difference because they usually rushed the pace of
the game
• No significant change in terms of meditation was
observed from one game mode to the other
– Participants can get considerably frustrated
• Significant correlations of EEG rhythms with
attention showed that users could possibly be more
concentrated during the session
– Achieving a high degree of relaxation overall during nonBCI
control
Prior Gaming
Experience in MI
Vourvopoulos, A., Liarokapis, F., Chen, M.C. The Effect of Prior Gaming Experience in Motor Imagery Training for Brain-Computer Interfaces: A
Pilot Study, Proc. of VS-Games 2015, IEEE Computer Society, Skovde, Sweden, 16-18 September, 139-146, 2015.
14/10/2019
5
Video Games and the Brain
• People regularly exposed to video-games have
improved :
– Visual and spatial attention (C. S. Green, D. Bavelier,
Nature, 2003)
– Memory (J. Feng et al., Psychol. Sci., 2007)
– Mental rotation abilities
– Enhanced sensorimotor learning (D. G. Gozli, et al.,Hum.
Mov. Sci., 2014)
• Extensive video-game practice has also been shown
to improve the efficiency of:
– Movement control brain networks
– Visuomotor skills (J. A. Granek, et al., Nerv. Syst. Behav.,
2010)
How Used in Current Mental Tasks?
• Mental rotation
• Motor imagery
• Remembering familiar faces
• etc…
Important for
using BCIs
Motor Imagery (MI)
• MI is a mental process by which an individual
rehearses or simulates a given action
– Implies that the subject feels herself/himself
performing the action
– MI is relying on the same brain systems that would be
used for actual performance of the task (Miller et al.,
2010)
Miller, K. J., Schalk, G., Fetz, E. E., Nijs, M. den, Ojemann, J. G., & Rao, R. P. N. (2010). Cortical activity during motor execution and motor imagery.
Proceedings of the National Academy of Sciences of the United States of America, 107(9),4430-5
Movement Imagery
Neurogaming & Brain-Controlled
Virtual Environments
• BCI’s used as primary input
• Excludes the use of traditional controllers
Current Limitations
• Long and repetitive training sessions can
result in user fatigue and declining
performance over time
• No relationship between videogame practice
and BCI training
In this Study
• Neurophysiological correlates of gaming
experience reflected in MI-BCI training
• Designed an experimental setup including:
– A standard BCI training paradigm
– Two different user groups based on their previous
gaming experience
14/10/2019
6
Types of Acquired Data Methodology: Participants
• 12 participants
• Mean age of 28 yrs
• 8 male, 4 female
• 1 left handed
Methodology: Experimental Setup
• 8 Active Electrodes
– Frontal-Central (FC3, FC4)
– Central (C3, C4, C5, C6)
– Central-Parietal (CP3, CP4)
• Frequency: 256Hz
Methodology: Experimental Setup
• Twin 640x480 LCD displays
• 32-degree FOV
Methodology: Experimental Setup
1,5 Sec 3,75 Sec
Video
14/10/2019
7
Methodology: Questionnaires Methodology: Grouping Players
• Clustering based on reported Gamer Dedication
(GD)
– where s = self-ranked score; w = weight
Ernest Adams, Barry Ip, From Casual to Core: A Statistical Mechanism for Studying Gamer Dedication, 2002.
0
10
20
30
40
50
60
70
80
90
0 2 4 6 8 10 12 14
Score%
User ID
ModerateHardcore
Mean
Extracting the EEG Rhythms
• Drowsiness, Concentration,
Visual fixation, Sensorimotor
rhythms (J. M. Stern, 2005)
• Active thinking, Active
attention, Sensorimotor
rhythms (S. Sanei, J. A.
Chambers, 2008)
• Meditative, Relaxed and
Creative states (S. Sanei, J. A.
Chambers, 2008)
• Visual, Auditory, Somatic and
Olfactory perception, Attention
(J. T. Cacioppo et al., 2007)
Rhythms
Seconds
Alpha
Beta
Theta
Gamma
Can different gamer groups modulate
different EEG patterns?
7.7%
-1%
3.7%
-4.3%-4.7% -3.66% -3.4%
-1.13%
Alpha +12.4%
Beta +2.64%
Theta +7.09%
Gamma -3.17%
Hardcore vs Moderate
Can experienced gamers increase their
performance faster?
• Hardcore -> 3.16%
• Moderate ->1.53%
Relationship between demographics
and EEG pattern modulation
• Beta (r=0.7848)
• Theta (r=0.6111)
• Gamma (r=0.6302)
• Alpha (r=-0.5642)
Higher modulation -> women
Gender
Age
Higher modulation -> older
people
14/10/2019
8
Relationship between subjective
reports and brain activity
Alpha
• Game Genre
• Gamer Dedication:
- Discuss games with
friends/bulletin
boards
• Game Addiction:
- Stay up late to play
video games
Beta
• Gender
• Game Addiction :
- Unsuccessful
reduction of video
games duration
- Eat meals while
playing video games
- Headaches, red
eyes, etc. from
playing video games
• Handedness
Theta
• Gamer
Dedication:
- Play games over
many long sessions
Gamma
• Game
Addiction :
- Active member
in activities or
clubs
• Kinesthetic
Imagery:
- Kicking a stone
Overall
• So far, with current results:
– We can distinguish a trend between the two gamer
groups
– A strong gaming profile could possibly enhance the ability
to use a BCI system
– Differences between all EEG bands
– Classification percentages increased performance faster
over time for Hardcore users
• Enhanced sensorimotor capability of experienced
gamers is partially reflected in MI-BCI training
Examining Brain
Activity While
Playing Computer
Games
Bakaoukas, A., Florin, C., Liarokapis, F. Examining Brain Activity While Playing Computer Games, Journal on Multimodal Interfaces, Springer, 1-17,
2015. (DOI: 10.1007/s12193-015-0205-4).
Aim
• Analyse data recorded while
participants were engaged in
playing popular computer
games
• Contribution
– Connection between activities
in the brain and the different
categories of computer games
Three Games
Quake 3TrackMania Nations
Minesweeper
Experiment
• gMOBIlab (g.tec) - 8 channels:
– O1, O2, T7, P3, Cz, P4, T8, Pz
• 21 participants
– 20 males (19 and 26 years old)
– 10 located in a quiet environment
– 11 located in a noisy environment
14/10/2019
9
Different Conditions Results
• Focus on the Alpha and Beta rhythm waves
– Frequencies range of 2–45 Hz
• Results revealed that the highest Alpha and Beta
rhythm magnitude levels are obtained when
engaging with the “Quake3” game
– As expected
• No significant differences between noisy and
quiet environments
– But higher beta from noisy compared to quiet
environment
Video
Understanding Body
Ownership in VR/AR
Aim
• Examining the use of body ownership in real
environment, virtual environment and
augmented reality environment
• Make use of the rubber hand illusion
– Future application in patients with schizophrenia
VR/AR Rubber Hand
• Compared to the classical
experiment where a plastic
rubber hand was used, a
virtual 3D representation
was chosen to create the
same illusion this time in
an immersive VR and AR
environment
14/10/2019
10
Participants & Evaluation
• Experiments were performed on
30 healthy volunteers, aged 19-
49
– 10 female
– 20 male
• Two different questionnaires
– Cognitive workload
• NASA TLX questionnaire
– Rubber Hand
• Ownership, Agency, Ownership
Control, Agency Control
Experimental Setup: Hardware
• Visualisation (Wrap 1200DX AR)
– Twin high-resolution 852 x 480 LCD
displays
– 35 degree diagonal FOV
• BCI (Enobio BCI)
– 32 sensors
– Sampling rate: 500 SPS
– Resolution: 24 bits - 0,05 microvolt
(uV)
Sensor Placement
• Frontal (F3, F4, F7, F8)
• Temporal (T7, T8)
• Central (C3, C4)
• Parietal (P7, P3, P4, P8, P03,
P04)
• Central-Parietal (CP1, CP2, CP5,
CP6)
• Occipital (O1, O2)
• Frontal-Central (FC1, FC2, FC5,
FC6)
• Frontal-Parietal (FP1, FP2)
• Intermediate (AF3, AF4)
• Mid Line (Oz, Pz, Cz, Fz)
Recordings
• EEG signals and head orientation of the
individuals were recorded and stored for
further processing
• Head orientation information is used to
remove artifacts
Video Qualitative Results
• Positive
– It’s fun and interesting
• Negative
– HMD doesn’t cover whole visual area
– HMD has poor resolution, is heavy
– Issues with the AR scene
– Can’t understand the questions
• Suggestions
– ”what would happen if...”
14/10/2019
11
Results - Questionnaires
• ANOVA on questionnaires
• Difference for ownership statements
– I felt as if I was looking at my own hand, sig. p=0.001
– I felt as if the rubber hand was my hand, sig.
p=0.034
• Best-accepted is the rubber hand in the physical
world
• No other significant differences
Results - Analysis of correlations
• Beta and gamma bands correlate positively with
questionnaire outputs
– Pearson r correlation
– Ownership and gamma: r=0.329, p=0.002
– Agency and beta: r=0.346, p=0.001
– More brain wave production for participants
subjectively feeling the illusion
Immersion Results
• Ownership statement rating splits the subjects
• Immersed: 20 in reality, 14 in AR, 13 in VR
– VR and AR ”worked” in less participants
– AR not really different from VR
• AR and VR produced slightly more brain waves
Overall
• Correlation between questionnaires and EEG
– Rubber hand was the preferred medium
– AR subjectively comparable to VR
• Premotor cortex activity linked to higher gamma
production during the illusion
• However AR and VR produced more brain
activity for both gamma and beta waves
User Profiling
for BCIs and Games
Vourvopoulos, A., Niforatos, E., Hlinka, M., Skola, F., Liarokapis, F. Investigating the Effect of User Profile during Training for BCIbased
Games, Proc. of the 9th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games 2017),
IEEE Computer Society, Athens, Greece, 6-8 September, 117-124, 2017. (ISBN: 978-1-5090-5812-9)
Overview
• This research illustrates the importance of:
– User-related effect
– Time-related effect
• The effect of reported workload immersion
during game play
• Difference in training modalities
14/10/2019
12
Experiment
• 34 Participants (17 males)
• 18-33 Age
• 32 EEG channels
Methods
• BCI Training
BCI Game The Protocol
• Motor
Observation
• Motor
Imagery
0 min ~60 min
BCICalibration BCITask QuestionnairesRestSetup & instructions
1s
~25min 5min 5min ~15min5min
Waitfor Cue Cue Pause …Baselin
e
30s 2s 5s 3.5s 282sec
FeedbackWaitfor Cue Cue Pause …Baselin
e
30s 2s 5s 3.5s 306sec
x 24 x 24
(a)
(b)
(c)
0 min ~60 min
BCICalibration BCITask QuestionnairesRestSetup & instructions
1s
~25min 5min 5min ~15min5min
Waitfor Cue Cue Pause …Baselin
e
30s 2s 5s 3.5s 282sec
FeedbackWaitfor Cue Cue Pause …Baselin
e
30s 2s 5s 3.5s 306sec
x 24 x 24
(a)
(b)
(c)
Extracting EEG features
Rhythms
Seconds
Alpha
Beta
Theta
Gamma
Raw EEG
7 - 14 Hz
15 - 30 Hz
4 - 7 Hz
30 - 100 Hz
EEG Data
• Alpha & Beta -> Classifier Input (Lotte, 2014)
• Beta/(Alpha+Theta) -> Engagement Index (A.
T. Pope et al., Biol. Psychol., 1995)
14/10/2019
13
Demographics & Questionnaires
• Demographics
– Gender
– Age
– Role
• Subjective experience
– Presence Questionnaire (PQ)
– Workload (NASA TLX)
– Flow (GEQ)
Results - Effect of Role
• Students vs Employees
• Differences in:
– Reported Workload
– Alpha, Theta bands
– Engagement Index
• Employees -> increased engagement and
decreased workload (mental, temporal
demand)
Results - Effect of Gender
• Differences in:
– EEG bands (Delta, Theta, Alpha, Beta)
– GEQ: Females reported less concentration
Results - Effect of Hour of Day
• Main effect of hour of day on:
– Gamma
– Engagement Index
• Higher at 15:00 than 19:00
Relationship of EEG data with
Reported Experience
• Relationship of Alpha & Theta:
– TLX: effort
– GEQ: Feedback, Time, Experience
• Engagement Index
– PQ: Adjustment in Experience
Motor Imagery vs Motor Observation
• No significant differences
vs
14/10/2019
14
Summary Conclusions
• Overall, this study showcased that gender, role
and time have a significant effect not only on
EEG modulation but also on reported
workload and loss of self-consciousness
during the game play
• This demonstrates how sensitive BCI
interaction can be, easily affected by
insufficient attention due to user distraction
or frustration
Future Work
• Include the analysis of specific electrode
locations, during BCI training, and create
models of user profiles that could be included
in a personalized training together with the
EEG data
User
Data
Brain Chatting using
Augmented Reality
Kerous, B., Liarokapis, F. BrainChat - A Collaborative Augmented Reality Brain Interface for Message Communication, Proc. of
the International Symposium on Mixed and Augmented Reality (ISMAR 2017) Adjunct Proceedings, IEEE Computer Society,
Nantes, France, 279-283, 2017. (DOI: 10.1109/ISMAR-Adjunct.2017.91)
New Communication Ways
• Nowadays we see a number of alternatives for
communication
• May different applications exist
• Ubiquitous computing
Interaction Modalities
• Event Related Potentials
14/10/2019
15
Advantages of ERP
• P300 recommended for mobile uses, as early as
2004 in based on error rates reported in 2003 BCI
competition
• Evaluation of a P300 in a fully mobile
environment
– Moderate drop of performance between sitting and
walking conditions
• The canonical presentation of a the stimuli is
evolving in recent years
Proposed BCI pipeline
• Components:
– Openvibe
– UIVA
– VRPN
– Unity3D
Calibration
• The calibration session was conducted by instructing
the user to count the number of flashes of the target
letter
• Calibration consisted of 10 randomly selected letters
• All rows and columns flashed in random order 12 times
for each letter the user was instructed to spell, with
one second delay between these 12 repetitions
• The flash duration was set at 0.2 seconds, preceded
and followed by a 0.1 second delay
• The user was given 3 second delay before the next
target letter block of flashes was initiated
Experiment
Results – User 100% Accuracy Results – User 25% Accuracy
14/10/2019
16
Future Tasks
• Stimuli changes (motion, size, color, sound)
• Find ways to eliminate multiple layers for
communication
• Embedding the stimuli in a context sensitive
and unimposing way
• Combining more than two users in a shared or
competitive task
Conclusions
• A lot of research is going on in this area
– Bio-feedback: very experimental at this stage
– EEG: ideal for patients and perception studies
• Won’t see many commercial applications soon
– Much more studies are required
– Technology will get better and cheaper
– Better algorithms for cleaning and classification are
needed
Collaborators
• Alina Ene (Coventry University)
• Athanasios Vourvopoulos (Madeira University)
• Panagiotis Petridis (Aston University)
• Anastasios Bakaoukas (University of Northampton)
• Sara de Freitas (Murdoch University)
• Filip Škola (Masaryk University)
• Roman Konečný (Masaryk University)
• Richard Bartecek (Masaryk University)
• Tomas Kasparek (Masaryk University)
Thank you for your Attention
• Personal Page
– http://www.fi.muni.cz/~liarokap/
• HCI Lab
– http://decibel.fi.muni.cz/wiki/index.php/Main_Page
• Scientific Blog
– http://fotisliarokapis.blogspot.com/ 94