2/26/2020
1
PA201
Virtual Environments
Lecture 3
Brain Computer Interfaces for Virtual Environments
Fotis Liarokapis
02nd March 2020
Introduction
Introduction
• Brain-Computer Interface (BCI) or Brain–
Machine Interface (BMI), is a direct way of
communication between the brain and a
computer system
BCI Categories
fMRI
fNIRS
MEG EEG
Functional Magnetic Resonance
Imaging (fMRI)
• fMRI measures brain activity by
detecting changes associated with
blood flow
– Relies on the fact that cerebral blood
flow and neuronal activation are
coupled
– When an area of the brain is in use,
blood flow to that region also increases
• High spatial resolution
– Tells you what is the smallest feature
you can see based on your detector
https://en.wikipedia.org/wiki/Functional_magnetic_resonance_imaging
Functional Near-Infrared Spectroscopy
(fNIRS)
• fNIRS is a non-invasive imaging method for
measuring brain activity through hemodynamic
responses associated with neuron behavior
• fNIR and fMRI are sensitive to similar
physiologic changes and are often comparative
methods
• Studies relating fMRI and fNIR show highly
correlated results in cognitive tasks
https://en.wikipedia.org/wiki/Functional_near-infrared_spectroscopy
2/26/2020
2
Magnetoencephalography (MEG)
• MEG is a functional
neuroimaging technique for
mapping brain activity by
recording magnetic fields
produced by electrical currents
occurring naturally in the brain
– Using very sensitive
magnetometers
• High temporal resolution
– Tells you how quickly you can
measure things
https://en.wikipedia.org/wiki/Magnetoencephalography
The Electroencephalogram (EEG)
• An (EEG) is a measure of
the brain’s voltage
fluctuations as detected
from scalp electrodes
• It is an approximation of
the cumulative electrical
activity of the neurons
• High temporal resolution
Brainwaves and EEG
• The human brain is made up
of billions of interconnected
neurons
• The patterns of interaction
between these neurons are
represented as thoughts and
emotional states
EEG Frequencies
Type Frequency Location Use
Delta (δ) <4 Hz Everywhere Occur during sleep, coma
Theta (θ) 4-7 Hz Temporal and
parietal
Emotional stress (frustration
& disappointment)
Alpha (α) 8-12 Hz Occipital and parietal Sensory stimulation or mental
imagery
Beta (β) 12-36 Hz Parietal and frontal Intense mental activity
Mu (μ) 9-11 Hz Frontal (motor
cortex)
Intention of movement
Brainwaves Graph
http://braintrainers.net/
Principles of EEG
2/26/2020
3
The 10-20 System
• The international 10-20 system describes the
electrode placement on the scalp for EEG tests
or experiments
Types of BCIs
Invasive BCI,
implanted surgically
Partially-Invasive
BCI, implanted
inside the scalp
Non-Invasive BCI,
using electrode cap
EEG-based BCI paradigm
• Three types:
– Event related potential (P300)
– Sensorimotor rhythms (SMR)
– Steady State Visually Evoked Potentials (SSVEP)
Event Related Potential (P300)
• The P300 is thought to
reflect processes involved in
stimulus evaluation or
categorization
• When recorded by EEG,
P300 surfaces as a positive
deflection in voltage with a
latency of roughly 250 to
500 ms
– The signal is typically
measured by the electrodes
covering the parietal lobe
https://en.wikipedia.org/wiki/P300_(neuroscience)
P300
• The presence, magnitude, topography and timing
of this signal are often used as metrics of
cognitive function in decision making processes
• While the neural substrates of this ERP
component still remain hazy, the reproducibility
and ubiquity of this signal makes it a common
choice for psychological tests in both the clinic
and laboratory
https://en.wikipedia.org/wiki/P300_(neuroscience)
P3a and P3b
• Since the initial discovery of the P300,
research has shown that the P300 has two
subcomponents
– P3 or P3a
– P300 which has since been renamed P3b
https://en.wikipedia.org/wiki/P300_(neuroscience)
2/26/2020
4
P3a
• P3a has a positive-going amplitude that displays
maximum amplitude over frontal/central
electrode sites and has a peak latency in the
range of 250-280 ms
• Associated with:
– Brain activity related to the engagement of attention
(especially the orienting, involuntary shifts to
changes in the environment)
– Processing of novelty
https://en.wikipedia.org/wiki/P300_(neuroscience)
P3b
• P3b has a positive-going amplitude that peaks at
around 300 ms, and the peak will vary in latency
from 250-500 ms or more depending upon the task
– Amplitudes are typically highest on the scalp over
parietal brain areas
• Used to study cognitive processes
– Especially psychology research on information processing
• The P3b can also be used to measure how
demanding a task is on cognitive workload
https://en.wikipedia.org/wiki/P300_(neuroscience)
P300 Spellers
• Very popular nowadays
P300 Speller Video
https://www.youtube.com/watch?v=y3lGJVnSSsg
Sensorimotor Rhythms (SMR)
• SMR is an oscillatory idle rhythm of synchronized
electromagnetic brain activity
– It appears in spindles in recordings of EEG, MEG, and
ECoG over the sensorimotor cortex
• The frequency is in the range of 13 to 15 Hz
• SMR is not fully understood
https://en.wikipedia.org/wiki/Sensorimotor_rhythm
How SMR Works
• Brain is producing a stronger SMR amplitude when
the corresponding sensorimotor areas are idle:
– During states of immobility, thus often mixed up with
alpha waves
• SMR typically decrease in amplitude when the
corresponding sensory or motor areas are activated
– i.e. during motor tasks and even during motor imagery
• SMR is very difficult to detect as it is usually
superimposed by the stronger occipital alpha waves
• The feline SMR has been noted as being analogous
to the human mu rhythm
https://en.wikipedia.org/wiki/Sensorimotor_rhythm
2/26/2020
5
SMR Neurofeedback
• Neurofeedback training can be used to gain
control over the SMR activity
– This feedback enables the subject to learn the
regulation of their own SMR
– Some patients may benefit from an increase in SMR
activity via neurofeedback
• i.e. learning difficulties, ADHD, epilepsy and autism
• In BCIs, the SMR amplitude during motor
imagery can be used to control external
applications
https://en.wikipedia.org/wiki/Sensorimotor_rhythm
SMR Speller Video
https://www.youtube.com/watch?v=R-tNE-y2QU0
Two-Dimensional BCI Control
https://www.youtube.com/watch?v=KMxop6xzsKM
Mu Waves
• Mu waves (known as mu rhythms, or sensorimotor
rhythms) are synchronized patterns of electrical
activity involving large numbers of neurons in the
part of the brain that controls voluntary movement
– These patterns repeat at a frequency of 7.5–12.5 (and
primarily 9–11) Hz
– Most prominent when the body is physically at rest
• Measured by:
– Electroencephalography (EEG)
– Magnetoencephalography (MEG)
– Electrocorticography (ECoG)
https://en.wikipedia.org/wiki/Mu_wave
Mu Waves .
• Unlike the alpha wave, which occurs at a similar
frequency over the resting visual cortex at the back
of the scalp, the mu wave is found over the motor
cortex, in a band approximately from ear to ear
• A person suppresses mu wave patterns when he/she
performs a motor action or, with practice, when he
or she visualizes performing a motor action
– This is called desynchronization of the wave because EEG
wave forms are caused by large numbers of neurons firing
in synchrony
https://en.wikipedia.org/wiki/Mu_wave
Steady State Visually Evoked Potentials
(SSVEP)
• SSVEP are signals that are natural responses to
visual stimulation at specific frequencies
• When the retina is excited by a visual stimulus
ranging from 3.5 Hz to 75 Hz, the brain
generates electrical activity at the same (or
multiples of) frequency of the visual stimulus
https://en.wikipedia.org/wiki/Steady_state_visually_evoked_potential
2/26/2020
6
SSVEP Usage
• This technique is used
widely with
electroencephalographic
research regarding vision
• SSVEP's are useful in
research because of the
excellent signal-to-noise
ratio and relative immunity
to artifacts
Open Vibe
https://en.wikipedia.org/wiki/Steady_state_visually_evoked_potential
SSVEP-based Mindspeller
https://www.youtube.com/watch?v=ZupEt1uvcls
SSVEP Chess Video
https://www.youtube.com/watch?v=spIVPw7XbCs
BCI Illiteracy
• Around 20 % of BCI users do not obtain reliable
BCI control (Tan and Nijholt, 2010)
• Investigation of BCI illiteracy can lead to:
– Avoid unnecessary training sessions
– Develop co-adaptive learning strategies to improve
BCI illiteracy
– Understand neurophysiological-basis of BCI illiteracy
– Build better BCI systems
Classification Issues
• Differences in brain anatomy may yield very
variable signal quality
• Large muscle artefacts
How to Improve BCI Illiteracy
• Improve classification accuracy
• Change paradigm
• Change neuroimaging technique
• Combine neuroimaging techniques
• Combine paradigms
2/26/2020
7
EEG Devices
Cheap Commercial BCI Headsets
• Non-invasive BCI’s most commonly use EEG:
– Portability, low set-up cost, easy of use
• Low-cost BCI headsets are used the last 10 years
Neurosky Headset
• NeuroSky MindWave is a simplified version of
the traditional EEG technology
• Attention and Meditation levels are calculated
from raw brainwaves by monitoring:
– Electrical potential between the sensing electrode
• Positioned on the forehead
– Reference electrodes
• Positioned on the left earlobe
http://neurosky.com/
Neurosky Advantages
• Very easy to use
• No calibration is required
– Plug and play!
• Good support is provided
– SDK
Neurosky Drawbacks
• Since there is only one sensor in place,
separating brainwaves becomes a challenge
• Because the headset is not fastened to the
head, pronounced muscle movements, such
as yawning, facial expressions may result in a
momentary decrease in signal quality
Neurosky MindWave Video
https://www.youtube.com/watch?v=1tr4CjtGtvg
2/26/2020
8
Emotiv Epoc Headset
• Emotiv Epoc Headset has 14
wet sensors (and 2
reference sensors) detecting
brain signals and facial
expressions
• Emotiv requires a unique
user profile to be trained to
map users’ brain-activity
Control Panel
https://emotiv.com/
Emotiv Epoc Wheelchair
https://www.youtube.com/watch?v=0at3NzNRySg
gMOBIlab
• gMOBIlab from g.tec
• Available in two versions:
– 8 channel EEG
– Multi-purpose version
• Multiplatform
– Windows and Linux
• Integration
– C or Matlab
http://www.gtec.at/Products/Hardware-and-Accessories/g.MOBIlab-Specs-Features
Enobio BCI
• Wireless, lightweight,
comfortable
• Comes in 8, 20 and 32
channels
• Bandwidth: 0 (DC) to 125
Hz
• Sampling rate: 500 SPS
• Resolution: 24 bits – 0,05
microvolt (uV)
• Triaxial accelerometer data
– For artifact removal
http://www.neuroelectrics.com/products/enobio/enobio-32/
Case Studies
BCIs and Computer Games
2/26/2020
9
Methodology
• Interaction
– Cognitive functions (brainwaves) are used to move
the forwards/backwards
– Expressive functions are used to steer left/right
• When the user blinks accordingly
• Profile training using Control Panel for 60s
(push/pull actions plus blink calibration)
– Navigating the 3D robot inside the maze to a predefined
waypoint (increasing users cognitive
workload)
• Evaluation with 30 users
Videos
3D Maze Game RomaNova Game
Liarokapis, F., Debattista, K., Vourvopoulos, A., Petridis, P., Ene, A., Comparing interaction techniques for serious games through brain-computer
interfaces: A user perception evaluation study, Entertainment Computing, Elsevier, 5(4): 391-399, 2014.
https://www.youtube.com/watch?v=h2_ESAewCaYhttps://www.youtube.com/watch?v=5Y_clGGoO4Y
Comparison of Questionnaires
• No significant differences for the ability to
control, responsiveness, interaction and
naturality of experience were found
– Can be explained by the similar difficulty of the
BCI task
Comparison of Questionnaire & EEG
• Questionnaire
– 16/31 (51%) users have
reported through their
answers that
they were engaged to
the game
• EEG
– 9 out 31 users found
with increased Beta
activity
– That’s 29% of the users
that scored high on the
engagement related
questions
• This could mean that whatever the users think about
their status is different on what actually was
recorded through the EEG
– Taking in good fain that the headset measured accurately
Multimodal BCI
Games
Liarokapis, F., Vourvopoulos, A., Ene, A. Examining User Experiences Through A Multimodal BCI Puzzle, Proc. of the 19th
International Conference on Information Visualisation (IV 2015), IEEE Computer Society, Barcelona, Spain, 21-24 July, 488-
493, 2015. (DOI: 10.1109/iV.2015.87)
Multimodal Games
• The game is multimodal, supporting a “BCI input”
and a “no BCI input” mode
• In the latter, meditation is defaulted at 50% of its
maximum possible value
– Speed is only affected by the number of cleared lines
• An instance of the game depends on:
– Name of the player
– Log’s creation timestamp
– Meditation
Liarokapis, F., Vourvopoulos, A., Ene, A. Examining User Experiences Through A Multimodal BCI Puzzle, Proc. of the 19th International
Conference on Information Visualisation (IV 2015), IEEE Computer Society, Barcelona, Spain, 21-24 July, 488-493, 2015.
2/26/2020
10
Video
https://www.youtube.com/watch?v=OEkWzy09Txg&feature=youtu.be
Evaluation Procedure
• Evaluated by 30 volunteers
– Selected by random sampling
– Duration was approximately 30
minutes
– 73.33% males, 26.67% females
Evaluation Procedure .
• The dominant age group is 18-25 with 80%
– 10% only aged 26-33
• 83.3% participants reported using the computer
to a very high degree in their daily activities
– However, in terms of gaming experience the
percentage drops to 23.33%
EEG Rhythms Log
• Significant correlations were found for attention
• Decreasing Theta (r = -0.2885, p < 0.05)
– Theta is usually linked to inefficiency and daydreaming
• High Alpha (r = -0.1841, p < 0.05)
– Alpha rhythms attenuate with drowsiness, concentration,
stimulation or visual fixation
• High Gamma (r = -0.1589, p < 0.05)
– High gamma oscillations have been observed in a variety
of different purpose neuro-anatomical domains including
informationprocessing
Conclusions
• More experienced gamers did not notice the speed
difference because they usually rushed the pace of
the game
• No significant change in terms of meditation was
observed from one game mode to the other
– Participants can get considerably frustrated
• Significant correlations of EEG rhythms with
attention showed that users could possibly be more
concentrated during the session
– Achieving a high degree of relaxation overall during nonBCI
control
Prior Gaming
Experience in MI
Vourvopoulos, A., Liarokapis, F., Chen, M.C. The Effect of Prior Gaming Experience in Motor Imagery Training for Brain-Computer Interfaces: A
Pilot Study, Proc. of VS-Games 2015, IEEE Computer Society, Skovde, Sweden, 16-18 September, 139-146, 2015.
2/26/2020
11
Video Games and the Brain
• People regularly exposed to video-games have
improved :
– Visual and spatial attention (C. S. Green, D. Bavelier,
Nature, 2003)
– Memory (J. Feng et al., Psychol. Sci., 2007)
– Mental rotation abilities
– Enhanced sensorimotor learning (D. G. Gozli, et al.,Hum.
Mov. Sci., 2014)
• Extensive video-game practice has also been shown
to improve the efficiency of:
– Movement control brain networks
– Visuomotor skills (J. A. Granek, et al., Nerv. Syst. Behav.,
2010)
How Used in Current Mental Tasks?
• Mental rotation
• Motor imagery
• Remembering familiar faces
• etc…
Important for
using BCIs
Motor Imagery (MI)
• MI is a mental process by which an individual
rehearses or simulates a given action
– Implies that the subject feels herself/himself
performing the action
– MI is relying on the same brain systems that would be
used for actual performance of the task (Miller et al.,
2010)
Miller, K. J., Schalk, G., Fetz, E. E., Nijs, M. den, Ojemann, J. G., & Rao, R. P. N. (2010). Cortical activity during motor execution and motor imagery.
Proceedings of the National Academy of Sciences of the United States of America, 107(9),4430-5
Movement Imagery
Neurogaming & Brain-Controlled
Virtual Environments
• BCI’s used as primary input
• Excludes the use of traditional controllers
Current Limitations
• Long and repetitive training sessions can
result in user fatigue and declining
performance over time
• No relationship between videogame practice
and BCI training
In this Study
• Neurophysiological correlates of gaming
experience reflected in MI-BCI training
• Designed an experimental setup including:
– A standard BCI training paradigm
– Two different user groups based on their previous
gaming experience
2/26/2020
12
Types of Acquired Data Methodology: Participants
• 12 participants
• Mean age of 28 yrs
• 8 male, 4 female
• 1 left handed
Methodology: Experimental Setup
• 8 Active Electrodes
– Frontal-Central (FC3, FC4)
– Central (C3, C4, C5, C6)
– Central-Parietal (CP3, CP4)
• Frequency: 256Hz
Methodology: Experimental Setup
• Twin 640x480 LCD displays
• 32-degree FOV
Methodology: Experimental Setup
1,5 Sec 3,75 Sec
Video
https://www.youtube.com/watch?v=FUiGjl3Jsfo&feature=youtu.be
2/26/2020
13
Methodology: Questionnaires Methodology: Grouping Players
• Clustering based on reported Gamer Dedication
(GD)
– where s = self-ranked score; w = weight
Ernest Adams, Barry Ip, From Casual to Core: A Statistical Mechanism for Studying Gamer Dedication, 2002.
0
10
20
30
40
50
60
70
80
90
0 2 4 6 8 10 12 14
Score%
User ID
ModerateHardcore
Mean
Extracting the EEG Rhythms
• Drowsiness, Concentration,
Visual fixation, Sensorimotor
rhythms (J. M. Stern, 2005)
• Active thinking, Active
attention, Sensorimotor
rhythms (S. Sanei, J. A.
Chambers, 2008)
• Meditative, Relaxed and
Creative states (S. Sanei, J. A.
Chambers, 2008)
• Visual, Auditory, Somatic and
Olfactory perception, Attention
(J. T. Cacioppo et al., 2007)
Rhythms
Seconds
Alpha
Beta
Theta
Gamma
Can different gamer groups modulate
different EEG patterns?
7.7%
-1%
3.7%
-4.3%-4.7% -3.66% -3.4%
-1.13%
Alpha +12.4%
Beta +2.64%
Theta +7.09%
Gamma -3.17%
Hardcore vs Moderate
Can experienced gamers increase their
performance faster?
• Hardcore -> 3.16%
• Moderate ->1.53%
Relationship between demographics
and EEG pattern modulation
• Beta (r=0.7848)
• Theta (r=0.6111)
• Gamma (r=0.6302)
• Alpha (r=-0.5642)
Higher modulation -> women
Gender
Age
Higher modulation -> older
people
2/26/2020
14
Relationship between subjective
reports and brain activity
Alpha
• Game Genre
• Gamer Dedication:
- Discuss games with
friends/bulletin
boards
• Game Addiction:
- Stay up late to play
video games
Beta
• Gender
• Game Addiction :
- Unsuccessful
reduction of video
games duration
- Eat meals while
playing video games
- Headaches, red
eyes, etc. from
playing video games
• Handedness
Theta
• Gamer
Dedication:
- Play games over
many long sessions
Gamma
• Game
Addiction :
- Active member
in activities or
clubs
• Kinesthetic
Imagery:
- Kicking a stone
Overall
• So far, with current results:
– We can distinguish a trend between the two gamer
groups
– A strong gaming profile could possibly enhance the ability
to use a BCI system
– Differences between all EEG bands
– Classification percentages increased performance faster
over time for Hardcore users
• Enhanced sensorimotor capability of experienced
gamers is partially reflected in MI-BCI training
Examining Brain
Activity While
Playing Computer
Games
Bakaoukas, A., Florin, C., Liarokapis, F. Examining Brain Activity While Playing Computer Games, Journal on Multimodal Interfaces, Springer, 1-17,
2015. (DOI: 10.1007/s12193-015-0205-4).
Aim
• Analyse data recorded while
participants were engaged in
playing popular computer
games
• Contribution
– Connection between activities
in the brain and the different
categories of computer games
Three Games
Quake 3TrackMania Nations
Minesweeper
Experiment
• gMOBIlab (g.tec) - 8 channels:
– O1, O2, T7, P3, Cz, P4, T8, Pz
• 21 participants
– 20 males (19 and 26 years old)
– 10 located in a quiet environment
– 11 located in a noisy environment
2/26/2020
15
Different Conditions Results
• Focus on the Alpha and Beta rhythm waves
– Frequencies range of 2–45 Hz
• Results revealed that the highest Alpha and Beta
rhythm magnitude levels are obtained when
engaging with the “Quake3” game
– As expected
• No significant differences between noisy and
quiet environments
– But higher beta from noisy compared to quiet
environment
Understanding Body
Ownership in VR/AR
Aim
• Examining the use of body ownership in real
environment, virtual environment and
augmented reality environment
• Make use of the rubber hand illusion
– Future application in patients with schizophrenia
VR/AR Rubber Hand
• Compared to the classical
experiment where a plastic
rubber hand was used, a
virtual 3D representation
was chosen to create the
same illusion this time in
an immersive VR and AR
environment
Participants & Evaluation
• Experiments were performed on
30 healthy volunteers, aged 19-
49
– 10 female
– 20 male
• Two different questionnaires
– Cognitive workload
• NASA TLX questionnaire
– Rubber Hand
• Ownership, Agency, Ownership
Control, Agency Control
2/26/2020
16
Experimental Setup: Hardware
• Visualisation (Wrap 1200DX AR)
– Twin high-resolution 852 x 480 LCD
displays
– 35 degree diagonal FOV
• BCI (Enobio BCI)
– 32 sensors
– Sampling rate: 500 SPS
– Resolution: 24 bits - 0,05 microvolt
(uV)
Sensor Placement
• Frontal (F3, F4, F7, F8)
• Temporal (T7, T8)
• Central (C3, C4)
• Parietal (P7, P3, P4, P8, P03,
P04)
• Central-Parietal (CP1, CP2, CP5,
CP6)
• Occipital (O1, O2)
• Frontal-Central (FC1, FC2, FC5,
FC6)
• Frontal-Parietal (FP1, FP2)
• Intermediate (AF3, AF4)
• Mid Line (Oz, Pz, Cz, Fz)
Recordings
• EEG signals and head orientation of the
individuals were recorded and stored for
further processing
• Head orientation information is used to
remove artifacts
Video
Qualitative Results
• Positive
– It’s fun and interesting
• Negative
– HMD doesn’t cover whole visual area
– HMD has poor resolution, is heavy
– Issues with the AR scene
– Can’t understand the questions
• Suggestions
– ”what would happen if...”
Results - Questionnaires
• ANOVA on questionnaires
• Difference for ownership statements
– I felt as if I was looking at my own hand, sig. p=0.001
– I felt as if the rubber hand was my hand, sig.
p=0.034
• Best-accepted is the rubber hand in the physical
world
• No other significant differences
2/26/2020
17
Results - Analysis of correlations
• Beta and gamma bands correlate positively with
questionnaire outputs
– Pearson r correlation
– Ownership and gamma: r=0.329, p=0.002
– Agency and beta: r=0.346, p=0.001
– More brain wave production for participants
subjectively feeling the illusion
Immersion Results
• Ownership statement rating splits the subjects
• Immersed: 20 in reality, 14 in AR, 13 in VR
– VR and AR ”worked” in less participants
– AR not really different from VR
• AR and VR produced slightly more brain waves
Overall
• Correlation between questionnaires and EEG
– Rubber hand was the preferred medium
– AR subjectively comparable to VR
• Premotor cortex activity linked to higher gamma
production during the illusion
• However AR and VR produced more brain
activity for both gamma and beta waves
User Profiling
for BCIs and Games
Vourvopoulos, A., Niforatos, E., Hlinka, M., Skola, F., Liarokapis, F. Investigating the Effect of User Profile during Training for BCIbased
Games, Proc. of the 9th International Conference on Virtual Worlds and Games for Serious Applications (VS-Games 2017),
IEEE Computer Society, Athens, Greece, 6-8 September, 117-124, 2017. (ISBN: 978-1-5090-5812-9)
Overview
• This research illustrates the importance of:
– User-related effect
– Time-related effect
• The effect of reported workload immersion
during game play
• Difference in training modalities
Experiment
• 34 Participants (17 males)
• 18-33 Age
• 32 EEG channels
2/26/2020
18
Methods
• BCI Training
BCI Game
The Protocol
• Motor
Observation
• Motor
Imagery
0 min ~60 min
BCICalibration BCITask QuestionnairesRestSetup & instructions
1s
~25min 5min 5min ~15min5min
Waitfor Cue Cue Pause …Baselin
e
30s 2s 5s 3.5s 282sec
FeedbackWaitfor Cue Cue Pause …Baselin
e
30s 2s 5s 3.5s 306sec
x 24 x 24
(a)
(b)
(c)
0 min ~60 min
BCICalibration BCITask QuestionnairesRestSetup & instructions
1s
~25min 5min 5min ~15min5min
Waitfor Cue Cue Pause …Baselin
e
30s 2s 5s 3.5s 282sec
FeedbackWaitfor Cue Cue Pause …Baselin
e
30s 2s 5s 3.5s 306sec
x 24 x 24
(a)
(b)
(c)
Extracting EEG features
Rhythms
Seconds
Alpha
Beta
Theta
Gamma
Raw EEG
7 - 14 Hz
15 - 30 Hz
4 - 7 Hz
30 - 100 Hz
EEG Data
• Alpha & Beta -> Classifier Input (Lotte, 2014)
• Beta/(Alpha+Theta) -> Engagement Index (A.
T. Pope et al., Biol. Psychol., 1995)
Demographics & Questionnaires
• Demographics
– Gender
– Age
– Role
• Subjective experience
– Presence Questionnaire (PQ)
– Workload (NASA TLX)
– Flow (GEQ)
2/26/2020
19
Results - Effect of Role
• Students vs Employees
• Differences in:
– Reported Workload
– Alpha, Theta bands
– Engagement Index
• Employees -> increased engagement and
decreased workload (mental, temporal
demand)
Results - Effect of Gender
• Differences in:
– EEG bands (Delta, Theta, Alpha, Beta)
– GEQ: Females reported less concentration
Results - Effect of Hour of Day
• Main effect of hour of day on:
– Gamma
– Engagement Index
• Higher at 15:00 than 19:00
Relationship of EEG data with
Reported Experience
• Relationship of Alpha & Theta:
– TLX: effort
– GEQ: Feedback, Time, Experience
• Engagement Index
– PQ: Adjustment in Experience
Motor Imagery vs Motor Observation
• No significant differences
vs
Summary
2/26/2020
20
Conclusions
• Overall, this study showcased that gender, role
and time have a significant effect not only on
EEG modulation but also on reported
workload and loss of self-consciousness
during the game play
• This demonstrates how sensitive BCI
interaction can be, easily affected by
insufficient attention due to user distraction
or frustration
Future Work
• Include the analysis of specific electrode
locations, during BCI training, and create
models of user profiles that could be included
in a personalized training together with the
EEG data
User
Data
Brain Chatting using
Augmented Reality
Kerous, B., Liarokapis, F. BrainChat - A Collaborative Augmented Reality Brain Interface for Message Communication, Proc. of
the International Symposium on Mixed and Augmented Reality (ISMAR 2017) Adjunct Proceedings, IEEE Computer Society,
Nantes, France, 279-283, 2017. (DOI: 10.1109/ISMAR-Adjunct.2017.91)
New Communication Ways
• Nowadays we see a number of alternatives for
communication
• May different applications exist
• Ubiquitous computing
Interaction Modalities
• Event Related Potentials
Advantages of ERP
• P300 recommended for mobile uses, as early as
2004 in based on error rates reported in 2003 BCI
competition
• Evaluation of a P300 in a fully mobile
environment
– Moderate drop of performance between sitting and
walking conditions
• The canonical presentation of a the stimuli is
evolving in recent years
2/26/2020
21
Proposed BCI pipeline
• Components:
– Openvibe
– UIVA
– VRPN
– Unity3D
Calibration
• The calibration session was conducted by instructing
the user to count the number of flashes of the target
letter
• Calibration consisted of 10 randomly selected letters
• All rows and columns flashed in random order 12 times
for each letter the user was instructed to spell, with
one second delay between these 12 repetitions
• The flash duration was set at 0.2 seconds, preceded
and followed by a 0.1 second delay
• The user was given 3 second delay before the next
target letter block of flashes was initiated
Experiment Results – User 100% Accuracy
Results – User 25% Accuracy Future Tasks
• Stimuli changes (motion, size, color, sound)
• Find ways to eliminate multiple layers for
communication
• Embedding the stimuli in a context sensitive
and unimposing way
• Combining more than two users in a shared or
competitive task
2/26/2020
22
Conclusions
• A lot of research is going on in this area
– Bio-feedback: very experimental at this stage
– EEG: ideal for patients and perception studies
• Won’t see many commercial applications soon
– Much more studies are required
– Technology will get better and cheaper
– Better algorithms for cleaning and classification are
needed
Questions