11/16/2015
1
PA198
Augmented Reality Interfaces
Lecture 7
Brain Computer Interfaces for Virtual and
Augmented Reality
Fotis Liarokapis
16th November 2014
Gizmodo BCI Video
http://www.gizmodo.in/science/Watch-This-Beautiful-10-Minute-Film-on-the-Current-State-of-Neuroscience/articleshow/49272231.cms
Introduction
Introduction
• Brain-Computer Interface (BCI) or Brain–
Machine Interface (BMI), is a direct way of
communication between the brain and a
computer system
BCI Categories
fMRI
fNIRS
MEG EEG
Functional Magnetic Resonance
Imaging (FMRI)
• FMRI measures brain activity by
detecting changes associated with
blood flow
– Relies on the fact that cerebral blood
flow and neuronal activation are
coupled
– When an area of the brain is in use,
blood flow to that region also increases
• High spatial resolution
– Tells you what is the smallest feature
you can see based on your detector
https://en.wikipedia.org/wiki/Functional_magnetic_resonance_imaging
11/16/2015
2
Functional Near-Infrared Spectroscopy
(fNIRS)
• fNIRS is a non-invasive imaging method for
measuring brain activity through hemodynamic
responses associated with neuron behavior
• fNIR and fMRI are sensitive to similar
physiologic changes and are often comparative
methods
• Studies relating fMRI and fNIR show highly
correlated results in cognitive tasks
https://en.wikipedia.org/wiki/Functional_near-infrared_spectroscopy
Magnetoencephalography (MEG)
• MEG is a functional
neuroimaging technique for
mapping brain activity by
recording magnetic fields
produced by electrical currents
occurring naturally in the brain
– Using very sensitive
magnetometers
• High temporal resolution
– Tells you how quickly you can
measure things
https://en.wikipedia.org/wiki/Magnetoencephalography
The Electroencephalogram (EEG)
• An (EEG) is a measure of
the brain’s voltage
fluctuations as detected
from scalp electrodes
• It is an approximation of
the cumulative electrical
activity of the neurons
• High temporal resolution
Brainwaves and EEG
• The human brain is made up
of billions of interconnected
neurons
• The patterns of interaction
between these neurons are
represented as thoughts and
emotional states
EEG Frequencies
Type Frequency Location Use
Delta (δ) <4 Hz Everywhere Occur during sleep, coma
Theta (θ) 4-7 Hz Temporal and
parietal
Emotional stress (frustration
& disappointment)
Alpha (α) 8-12 Hz Occipital and parietal Sensory stimulation or mental
imagery
Beta (β) 12-36 Hz Parietal and frontal Intense mental activity
Mu (μ) 9-11 Hz Frontal (motor
cortex)
Intention of movement
Brainwaves Graph
http://braintrainers.net/
11/16/2015
3
Principles of EEG The 10-20 System
• The international 10-20 system describes the
electrode placement on the scalp for EEG tests
or experiments
Types of BCIs
Invasive BCI,
implanted surgically
Partially-Invasive
BCI, implanted
inside the scalp
Non-Invasive BCI,
using electrode cap
EEG-based BCI paradigm
• Three types:
– Event related potential (P300)
– Sensorimotor rhythms (SMR)
– Steady State Visually Evoked Potentials (SSVEP)
Event Related Potential (P300)
• The P300 is thought to
reflect processes involved in
stimulus evaluation or
categorization
• When recorded by EEG,
P300 surfaces as a positive
deflection in voltage with a
latency of roughly 250 to
500 ms
– The signal is typically
measured by the electrodes
covering the parietal lobe
https://en.wikipedia.org/wiki/P300_(neuroscience)
P300
• The presence, magnitude, topography and timing
of this signal are often used as metrics of
cognitive function in decision making processes
• While the neural substrates of this ERP
component still remain hazy, the reproducibility
and ubiquity of this signal makes it a common
choice for psychological tests in both the clinic
and laboratory
https://en.wikipedia.org/wiki/P300_(neuroscience)
11/16/2015
4
P3a and P3b
• Since the initial discovery of the P300,
research has shown that the P300 has two
subcomponents
• The subcomponents are the novelty P3, or
P3a, and the classic P300, which has since
been renamed P3b
https://en.wikipedia.org/wiki/P300_(neuroscience)
P3a
• P3a has a positive-going amplitude that displays
maximum amplitude over frontal/central
electrode sites and has a peak latency in the
range of 250-280 ms
• Associated with:
– Brain activity related to the engagement of attention
(especially the orienting, involuntary shifts to
changes in the environment)
– Processing of novelty
https://en.wikipedia.org/wiki/P300_(neuroscience)
P3b
• P3b has a positive-going amplitude that peaks at
around 300 ms, and the peak will vary in latency
from 250-500 ms or more depending upon the task
– Amplitudes are typically highest on the scalp over
parietal brain areas
• Used to study cognitive processes
– Especially psychology research on information
processing
• The P3b can also be used to measure how
demanding a task is on cognitive workload
https://en.wikipedia.org/wiki/P300_(neuroscience)
P300 Spellers
• Very popular nowadays
P300 Speller Video
https://www.youtube.com/watch?v=y3lGJVnSSsg
Sensorimotor Rhythms (SMR)
• SMR is an oscillatory idle rhythm of synchronized
electromagnetic brain activity
– It appears in spindles in recordings of EEG, MEG, and
ECoG over the sensorimotor cortex
• The frequency is in the range of 13 to 15 Hz
• SMR is not fully understood
https://en.wikipedia.org/wiki/Sensorimotor_rhythm
11/16/2015
5
How SMR Works
• Brain is producing a stronger SMR amplitude when
the corresponding sensorimotor areas are idle:
– During states of immobility, thus often mixed up with
alpha waves
• SMR typically decrease in amplitude when the
corresponding sensory or motor areas are activated
– i.e. during motor tasks and even during motor imagery
• SMR is very difficult to detect as it is usually
superimposed by the stronger occipital alpha waves
• The feline SMR has been noted as being analogous
to the human mu rhythm
https://en.wikipedia.org/wiki/Sensorimotor_rhythm
SMR Neurofeedback
• Neurofeedback training can be used to gain
control over the SMR activity
– This feedback enables the subject to learn the
regulation of their own SMR
– Some patients may benefit from an increase in SMR
activity via neurofeedback
• i.e. learning difficulties, ADHD, epilepsy and autism
• In BCIs, the SMR amplitude during motor
imagery can be used to control external
applications
https://en.wikipedia.org/wiki/Sensorimotor_rhythm
SMR Speller Video
https://www.youtube.com/watch?v=R-tNE-y2QU0
Two-Dimensional BCI Control
https://www.youtube.com/watch?v=KMxop6xzsKM
Mu Waves
• Mu waves (known as mu rhythms, or sensorimotor
rhythms) are synchronized patterns of electrical
activity involving large numbers of neurons in the
part of the brain that controls voluntary movement
– These patterns repeat at a frequency of 7.5–12.5 (and
primarily 9–11) Hz
– Most prominent when the body is physically at rest
• Measured by:
– Electroencephalography (EEG)
– Magnetoencephalography (MEG)
– Electrocorticography (ECoG)
https://en.wikipedia.org/wiki/Mu_wave
Mu Waves .
• Unlike the alpha wave, which occurs at a similar
frequency over the resting visual cortex at the back
of the scalp, the mu wave is found over the motor
cortex, in a band approximately from ear to ear
• A person suppresses mu wave patterns when he/she
performs a motor action or, with practice, when he
or she visualizes performing a motor action
– This is called desynchronization of the wave because EEG
wave forms are caused by large numbers of neurons firing
in synchrony
https://en.wikipedia.org/wiki/Mu_wave
11/16/2015
6
Steady State Visually Evoked Potentials
(SSVEP)
• SSVEP are signals that are natural responses to
visual stimulation at specific frequencies
• When the retina is excited by a visual stimulus
ranging from 3.5 Hz to 75 Hz, the brain
generates electrical activity at the same (or
multiples of) frequency of the visual stimulus
https://en.wikipedia.org/wiki/Steady_state_visually_evoked_potential
SSVEP Usage
• This technique is used
widely with
electroencephalographic
research regarding vision
• SSVEP's are useful in
research because of the
excellent signal-to-noise
ratio and relative immunity
to artifacts
Open Vibe
https://en.wikipedia.org/wiki/Steady_state_visually_evoked_potential
SSVEP-based Mindspeller
https://www.youtube.com/watch?v=ZupEt1uvcls
SSVEP Chess Video
https://www.youtube.com/watch?v=spIVPw7XbCs
BCI Illiteracy
• Around 20 % of BCI users do not obtain reliable
BCI control (Tan and Nijholt, 2010)
• Investigation of BCI illiteracy can lead to:
– Avoid unnecessary training sessions
– Develop co-adaptive learning strategies to improve
BCI illiteracy
– Understand neurophysiological-basis of BCI illiteracy
– Build better BCI systems
Classification Issues
• Differences in brain anatomy may yield very
variable signal quality
• Large muscle artefacts
11/16/2015
7
How to Improve BCI Illiteracy
• Improve classification accuracy
• Change paradigm
• Change neuroimaging technique
• Combine neuroimaging techniques
• Combine paradigms
EEG Devices
Cheap Commercial BCI Headsets
• Non-invasive BCI’s most commonly use EEG:
– Portability, low set-up cost, easy of use
• Low-cost BCI headsets are used the last 10 years
Neurosky Headset
• NeuroSky MindWave is a simplified version of
the traditional EEG technology
• Attention and Meditation levels are calculated
from raw brainwaves by monitoring:
– Electrical potential between the sensing electrode
• Positioned on the forehead
– Reference electrodes
• Positioned on the left earlobe
http://neurosky.com/
Neurosky Advantages
• Very easy to use
• No calibration is required
– Plug and play!
• Good support is provided
– SDK
Neurosky Drawbacks
• Since there is only one sensor in place,
separating brainwaves becomes a challenge
• Because the headset is not fastened to the
head, pronounced muscle movements, such
as yawning, facial expressions may result in a
momentary decrease in signal quality
11/16/2015
8
Neurosky MindWave Video
https://www.youtube.com/watch?v=1tr4CjtGtvg
Emotiv Epoc Headset
• Emotiv Epoc Headset has 14
wet sensors (and 2
reference sensors) detecting
brain signals and facial
expressions
• Emotiv requires a unique
user profile to be trained to
map users’ brain-activity
Control Panel
https://emotiv.com/
Emotiv Epoc Wheelchair
https://www.youtube.com/watch?v=0at3NzNRySg
gMOBIlab
• gMOBIlab from g.tec
• Available in two versions:
– 8 channel EEG
– Multi-purpose version
• Multiplatform
– Windows and Linux
• Integration
– C or Matlab
http://www.gtec.at/Products/Hardware-and-Accessories/g.MOBIlab-Specs-Features
Enobio BCI
• Wireless, lightweight,
comfortable
• Comes in 8, 20 and 32
channels
• Bandwidth: 0 (DC) to 125
Hz
• Sampling rate: 500 SPS
• Resolution: 24 bits – 0,05
microvolt (uV)
• Triaxial accelerometer data
– For artifact removal
http://www.neuroelectrics.com/products/enobio/enobio-32/
Case Studies
11/16/2015
9
BCIs and Robots
• Main idea
– Move a robot
BCIs and Robots
BCIs and Computer Games 3D Maze Game
• The 3D environment
has been designed
using the ‘Unity 3D’
game engine with a
3D reconstruction of
the robot and a
simple 3D maze
Game Elements
• 3D Robot
– The LEGO NXT Mindstorms
Robot 3D model has been
used with a simple wheel
animation, interacting with
basic physics
• Maze walls
– The maze is made by a set of
hetero-sized blocks
Roma Nova Project
• Seeking to advance information transfer
through immersive ‘living background’
• Partners:
– Serious Games Institute
– University of Toulouse
11/16/2015
10
Brain Computer Interactions
• The Cognitive functions (brainwaves) are used
to move the robot forwards/backwards
• The Expressive functions are used to steer the
robot left/right when the user blinks
accordingly
Roma Nova Video
Initial Evaluation
• An evaluation session has been conducted with
five participants in a laboratory environment
• Feedback was received in direct reply to the
questions, as well as by raising additional issues
• All participants had no previous experience with
BCIs so some time was given to familiarise with
the technology
Initial Evaluation .
• Since all users interacted with a virtual object
using their brainwaves for the first time, it was
necessary to perform repeatable profile training
– So players managed to familiarise with the
prototype system
• At this stage, the system extracts and classifies
more accurately the player’s intentions
Initial Evaluation ..
• All participants had to complete
a small task
– 5 to 10 minutes
• The task was to
– Move an avatar inside the Roma
Nova
– Interact with the agents using just
brainwaves and facial expressions
Positive Feedback
• All participants mentioned that it was a
unique experience to interact with the game
through brainwaves
• Even if it was ‘slower’ to interact with the
game they reported that this way of
interaction is far more enjoyable
– Compared to standard input devices such as the
mouse and the keyboard
11/16/2015
11
Positive Feedback .
• All users enjoyed the graphics quality of the game
– As well as the ‘clever’ dialogues with the intelligent
agents
• The majority of the players mentioned that the
brain computer technologies can be very useful for
interaction of games and it can be combined with
other techniques
– Such as other natural interaction techniques
Negative Feedback
• Some users found it hard to adapt in taking
control of the agent straight away
– They got distracted by external stimuli
• Some mentioned that it was not easy to
concentrate in the game and they would prefer
a more immersive environment
– Even if through time they started to get control and
adapt to the prototype system
Negative Feedback .
• Finally, some participants found the BCI
technology not as accurate as standard input
devices
– Even if in this particular game there were no
significant requirements on accuracy in navigation,
in other computer games that could be
problematic
End User Evaluation
• 31 users have been evaluated the hybrid BCI
architecture providing feedback by interacting with
the two games
• Each user had to complete a set of tasks to evaluate
efficiently the system and the overall interaction
• EEG data from two mental tasks of the user (push,
pull) had been recorded and stored in order to be
analysed and processed
Methodology
• Profile training using Control Panel for 60s
(push/pull actions plus blink calibration)
– Navigating the 3D robot inside the maze to a predefined
waypoint (increasing users cognitive
workload)
• A second training session of 60s
– Interacting with RomaNova
• Evaluation form completion and feedback
interview
Cerebral Palsy User Case
• A user with Cerebral Palsy had been interacted
successfully with the system, being able to move
the virtual objects dispite being affected by
spastic hemiplegia
• Cerebral palsy (CP) is a motor condition that
causes physical disability in human
development in various areas of body
movement
11/16/2015
12
Results
• 16/31 (51%) users have
reported through their
answers that
they were engaged to the
game
• EEG analysis had been
performed to see if their
answers matches their
brain activity
Results .
• Delta and Theta rhythms (0.1-4-8Hz) are lowfrequency
EEG patterns that increase during
sleep in the normal adult
Results ..
• Beta rhythms(12-30Hz) occur in individuals who
are alert and attentive to external stimuli or
exert specific mental effort
Results …
• 9 out 31 users found
with increased Beta
activity
• That’s 29% of the
users that scored
high on the
engagement related
questions
General Finding
• This proves that whatever the users think
about their status is different on what actually
was recorded through the EEG
– Taking in good fain that the headset measured
accurately
BCI Tetris
• Focus on determining the relationship
between user experience enhancement and
cheap commercial BCIs in games
• Aims were two examine:
– Whether a plug-and-play BCI device as an
additional component of the existing interaction
metaphor can be an effective mechanism for
games
– If the gaming experience can be enhanced
11/16/2015
13
System Overview
• A laptop equipped with a 32-bit Intel Core
2 Duo processor P7350 at 2.0 GHz and
3.00 GB of RAM were utilized in
conjunction with the inexpensive noninvasive
commercial BCI device
• In particular, the NeuroSky Mindset
headset was used for this is the only
device that can be used for game
interaction without prior calibration since
it is based on a single dry sensor 73
Tetris Game
• The game is multimodal, supporting a “BCI
input” and a “no BCI input” mode
• In the latter, meditation is defaulted at 50% of its
maximum possible value
– Speed is only affected by the number of cleared lines
• An instance of the game depends on:
– Name of the player
– Log’s creation timestamp
– Meditation
74
Tetris Game .
• The speed of the current falling brick is determined
by the number of milliseconds required for the
shape to traverse one line
– The bigger this time value is, the slower the brick will fall
• The step time will be 150 milliseconds
– Which the double of the current meditation level is added
• As the game progresses, the number of lines cleared
multiplied by 5 is subtracted from the step time
– Whereas for the “non BCI input” mode, the current
meditation value defaults to half of its full potential (50%)
75
BCI Tetris Video
Evaluation Procedure
• Evaluated by 30 volunteers
– Selected by random sampling
– Duration was approximately 30 minutes
– 73.33% males, 26.67% females
• The dominant age group is 18-25 with 80%
– 10% only aged 26-33
• 83.3% participants reported using the computer
to a very high degree in their daily activities
• However, in terms of gaming experience the
percentage drops to 23.33%
77
Qualitative Feedback
• The overall experience was described as
interesting, enjoyable and relaxing
• The BCI mode was generally seen as more
challenging and entertaining
– Compared to the non-BCI mode
• They expressed their confidence that this type
of novel gaming experience has a lot of
potential for becoming industry standard in the
near future
78
11/16/2015
14
Positive Remarks
• The BCI mode was welcomed as a unique and
fascinating concept
• The ability to control the speed of the level in
some instances made the game easier to play
(for some)
• The meditation progress bar helpful and a key
part of the experience which they could use to
their advantage
• Users not noticing much fluctuation in the
speed of the shapes during the two modes
79
Negative Remarks
• User focus shifts from shape control accuracy to
speed only halfway through the game
• Users noticed an increase in speed when
becoming annoyed and impatient
• Some preferred the standard mode as they
discerned a lower speed and allowed more time
left for decision making and made the goal
easier to reach
• The BCI-mode could not be fully experienced by
participants who were tired
80
Recommendations
• Fully control the game using a BCI device
• Expansion of BCIs in competitive multiplayer
environments
• Focus on studying how meditation is controlled in a
multitasking scenario
• Make more use of sound
– i.e. Use sound as an indicator of speed instead of a
progress bar
• Use of more accurate BCI device
• Alternative input methods
– i.e. Voice, eye tracking, etc 81
Results from non-BCI Tetris mode
82
Results from BCI Tetris mode
83
Results Comparison
• 13/30 users thought that the mental demand
quotient for each mode (non-BCI and BCI) should
be of the same value
– Whereas other 13 users saw the BCI controlled game
version more mentally demanding (almost
statistically significant, p = 0.051)
• Other statistically significant differences are for
learnability (p < 0.005) and satisfaction (p <
0.010)
84
11/16/2015
15
Results Comparison .
• 36.6% thought that the meditation-controlled
Tetris required more time to learn how to play it
than the generic mode
• Satisfaction comparison reveals that the same
percentage of participants considered the “BCI”
Tetris mode to be much more enjoyable than the
“non BCI” Tetris mode
• Perceived user performance remains unchanged
from “non BCI” to “BCI”
85
EEG Rhythms Log
• Significant correlations were found for attention
• For increased attention we can see decreasing
Theta oscillations (r = -0.2885, p < 0.05)
• Theta is usually linked to inefficiency and
daydreaming
– In fact, the very lowest waves of theta represent the
fine line between being awake or in a sleep
86
EEG Rhythms Log .
• High Alpha (r = -0.1841, p < 0.05) and high
gamma (r = -0.1589, p < 0.05) oscillations are
observed with increased attention
– Alpha rhythms attenuate with drowsiness,
concentration, stimulation or visual fixation
– High gamma oscillations have been observed in a
variety of different purpose neuroanatomical
domains including information processing
87
EEG Rhythms Log ..
• For the Non-BCI setup, Delta is almost three
times higher, indicating deeper relaxation when
the BCI device was not used as an input
– Delta band is modulated the highest compared with
the rest of the rhythms showing an overall deep
relaxation
88
Conclusions
• 30 participants tested a Tetris-based game in two
modes, normal game play and BCI input
• It was observed that the more experienced gamers
did not notice the speed difference because they
usually rushed the pace of the game
• For the participants who actually managed to
maintain high levels of meditation throughout the
second mode, they were more inclined to notice the
speed difference, achieve more and fully enjoy the
experience
• It is also important to note that after long periods of
use, the device increased user fatigue
89
Games comparison using EEG data
Quake 3TrackMania Nations
Minesweeper
11/16/2015
16
Video BCIs and Games
• The Effect of Prior Gaming Experience in
Motor Imagery Training for Brain-Computer
Interfaces: A Pilot Study
Vourvopoulos, A., Liarokapis, F., Chen, M.C. The Effect of Prior Gaming Experience in Motor Imagery Training for Brain-Computer Interfaces: A
Pilot Study, Proc. of VS-Games 2015, IEEE Computer Society, Skovde, Sweden, 16-18 September, 139-146, 2015.
Video Games and the Brain
• People regularly exposed to video-games have
Improved :
– Visual and Spatial attention (C. S. Green, D.
Bavelier, Nature, 2003)
– Memory (J. Feng et al., Psychol. Sci., 2007)
– Mental rotation abilities
– Enhanced sensorimotor learning (D. G. Gozli, et
al.,Hum. Mov. Sci., 2014)
Video Games and the Brain .
• Extensive video-game practice has also been
shown to improve the efficiency of:
– Movement control brain networks
– Visuomotor skills (J. A. Granek, et al., Nerv. Syst.
Behav., 2010)
How Used in Current Mental Tasks?
• Mental rotation
• Motor imagery
• Remembering familiar faces
• etc…
Important for
using BCIs
Interfacing the Brain with the
Computer
• BCIs are communication systems which
translate brain activity into control signals in
computers or external devices
Signal
Processing
Signal
Acquisition
End
Effector
Control SignalRaw EEG
Motor Imagery
11/16/2015
17
Motor Imagery
• Motor Imagery (MI) is a mental process by
which an individual rehearses or simulates a
given action
• Implies that the subject feels herself/himself
performing the action
Motor Imagery .
• MI is relying on the same brain systems that
would be used for actual performance of the
task (Miller et al., 2010)
Movement Imagery
Miller, K. J., Schalk, G., Fetz, E. E., Nijs, M. den, Ojemann, J. G., & Rao, R. P. N. (2010). Cortical activity during motor execution and motor imagery.
Proceedings of the National Academy of Sciences of the United States of America, 107(9),4430-5
Neurogaming & Brain-Controlled
Virtual Environments
• BCI’s used as primary input
• Excludes the use of traditional controllers
Current Limitations
• Long and repetitive training sessions can
result in user fatigue and declining
performance over time
• No relationship between videogame practice
and BCI training
In this Study
• Neurophysiological correlates of gaming
experience reflected in MI-BCI training
• Designed an experimental setup including:
– A standard BCI training paradigm and
– Two different user groups based on their previous
gaming experience
Types of Acquired Data
11/16/2015
18
Methodology: Participants
• 12 participants
• Mean age of 28 yrs
• 8 male, 4 female
• 1 left handed
Methodology: Experimental Setup
• 8 Active Electrodes
– Frontal-Central (FC3, FC4)
– Central (C3, C4, C5, C6)
– Central-Parietal (CP3, CP4)
• Frequency: 256Hz
Methodology: Experimental Setup
• Twin 640x480 LCD displays
• 32-degree FOV
Methodology: Experimental Setup
1,5 Sec 3,75 Sec
Video Methodology: Questionnaires
11/16/2015
19
Methodology: Grouping Players
• Clustering users based on reported Gamer
Dedication (GD)
• where s = self-ranked score; w = weight
Ernest Adams, Barry Ip, From Casual to Core: A Statistical Mechanism for Studying Gamer Dedication, 2002.
Grouping Players
0
10
20
30
40
50
60
70
80
90
0 2 4 6 8 10 12 14
Score%
User ID
ModerateHardcore
Mean
Extracting the EEG Rhythms Extracting the EEG Rhythms
• Drowsiness, Concentration,
Visual fixation, Sensorimotor
rhythms (J. M. Stern, 2005)
• Active thinking, Active
attention, Sensorimotor
rhythms (S. Sanei, J. A.
Chambers, 2008)
• Meditative, Relaxed and
Creative states (S. Sanei, J. A.
Chambers, 2008)
• Visual, Auditory, Somatic and
Olfactory perception, Attention
(J. T. Cacioppo et al., 2007)
Rhythms
Seconds
Alpha
Beta
Theta
Gamma
Can different gamer groups modulate
different EEG patterns?
7.7%
-1%
3.7%
-4.3%-4.7% -3.66% -3.4%
-1.13%
Alpha +12.4%
Beta +2.64%
Theta +7.09%
Gamma -3.17%
Hardcore vs Moderate
Can experienced gamers increase their
performance faster?
• Hardcore -> 3.16%
• Moderate ->1.53%
11/16/2015
20
Relationship between demographics
and EEG pattern modulation
• Beta (r=0.7848)
• Theta (r=0.6111)
• Gamma (r=0.6302)
• Alpha (r=-0.5642)
Higher modulation -> women
Gender
Age
Higher modulation -> older
people
Relationship between subjective
reports and brain activity
Alpha
• Game Genre
• Gamer Dedication:
- Discuss games with
friends/bulletin
boards
• Game Addiction:
- Stay up late to play
video games
Beta
• Gender
• Game Addiction :
- Unsuccessful
reduction of video
games duration
- Eat meals while
playing video games
- Headaches, red
eyes, etc. from
playing video games
• Handedness
Theta
• Gamer
Dedication:
- Play games over
many long sessions
Gamma
• Game
Addiction :
- Active member
in activities or
clubs
• Kinesthetic
Imagery:
- Kicking a stone
Summarizing: Alpha Activity
• Gaming experience leading to increased Alpha
activity can be the reason that older participants
had higher Alpha
• Gamers with favorite genre the Action games, had
increased Alpha activity
• Increased in Hardcore group
• Past research showed positive correlations
between good game performance (less time to
complete a task) and, on average, the production of
Alpha waves
Summarizing: Beta Activity
• Gamers that unsuccessfully tried to reduce the amount of
time they play video games, and get headaches, red eyes,
etc, have increased Beta activity.
• Participants with higher level of right-handedness and
females had increased modulation.
• Decrease on Moderate group but more stable on
Hardcore
• So far we know that, voluntary movement results in a
desynchronization in alpha and beta band oscillations,
localized over sensorimotor areas
• Therefore high levels of addiction could result into higher
sensorimotor activation
Summarizing: Theta Activity
• Participants which play games over many long
sessions have increased Theta modulation during
training
• Increased in Hardcore group
• Cortical theta is observed frequently during
meditative or relaxing states, but also it has been
shown that the level of theta brainwave activity in
the prefrontal cortex predicts whether people will
be able to overcome ingrained biases - choosing an
action that is counter to habit - when is required to
achieve a goal
Summarizing: Gamma Activity
• High addiction users produce higher Gamma
• Participants with high kinaesthetic imagery related
with “kicking a stone” visualization, have increased
Gamma activity
• Participants of the “Hardcore” group had decreased
activity compared with the “Moderate” group
• Binding of different neurons together into a for
carrying out a certain cognitive or motor function
11/16/2015
21
Overall
• So far, with current results:
– We can distinguish a trend between the two gamer
groups
– A strong gaming profile could possibly enhance the ability
to use a BCI system
– Differences between all EEG bands
– Classification percentages increased performance faster
over time for Hardcore users
• Enhanced sensorimotor capability of experienced
gamers is partially reflected in MI-BCI training
How to Use Current Results
Stimulate the areas
responsible for high
motor control during
training
Use electrophysiological
+ subjective data to classify
gamers
BCIs and VR/AR
• Examining the use of real environment, virtual
environment and augmented reality
environment for body ownership
Filip Škola, Szymon Fiałek, Liarokapis, F. Augmenting the Rubber Hand Illusion, ERCIM News, ERCIM, 103, 25-26, 2015. (ISSN: 0926-4981)
Classical Rubber Hand Illusion
• The traditional rubber hand experiment is a
psychological experiment where participants are
under the illusion that a rubber hand is part of their
own body
• During the experiment, the rubber hand is
positioned in front of the participant while their real
hand is kept hidden from their view
• Synchronous touches are then applied to both their
real hand and the rubber hand and within minutes
participants get the illusion that the rubber hand is
part of their body
Kalckert, A., Ehrsson, H.H. The moving rubber hand illusion revisited: Comparing movements and visuotactile stimulation to induce illusory
ownership, Conscious Cogn. 26:117-32, 2014
VR/AR Rubber Hand
• Compared to the classical
experiment where a plastic
rubber hand was used, a
virtual 3D representation
was chosen to create the
same illusion this time in
an immersive VR and AR
environment
3D Rubber Hand
• Based on photogrammetric techniques
– 15 high definition images were taken
• Processed using medical imaging software to produce the
3D mesh
– Textures were also taken from the high definition
images
• The resulted 3D model consists of 46611
vertices and 93218 triangles offering a very
realistic model of a human hand
11/16/2015
22
Participants
• Experiments were performed on 30 healthy
volunteers, aged 20-35 years old
• Participants were asked to complete two different
questionnaires, one measuring their cognitive
workload (based on the standard NASA TLX
questionnaire) and another one regarding their
experience with the rubber hand illusion
• In addition, EEG signals of the individuals were
recorded and stored for further processing
Finger Sequence & Sensor Placement
• Total: 38 brushes
• Time: 4 secs
• Total Time: 152 secs
• Sequence:
– 3, 1, 4, 3, 1, 4, 1, 2, 1, 3
– 1, 3, 4, 4, 2, 3, 1, 1, 4, 3
– 2, 1, 4, 3, 4, 2, 1, 1, 4, 2
– 1, 2, 1, 3, 2, 4, 3, 4
Video Methodology
• Absolute numbers:
– Average band power from 100 seconds of data in
the later phase of the experiment
• Relative numbers:
– (avg band power from the later phase of the
experiment) minus (avg band power from the
beginning)
– Basically brain wave change for each band
• Positive or negative
Overall Results - Absolute
• Ownership control vs Beta = 0.305, vs Gamma = 0.262
• Agency vs Beta, Gamma = positive correlations
• Ownership Control vs Beta = positive correlations
• Agency Control vs Alpha, Beta, Gamma Theta (Beta
strongest) = positive correlations
• Positive TLX corr.:
– TLX Mental vs Beta
– TLX Physical vs Beta
– TLX Temporal vs Gamma
Rubber Condition - Absolute
• Ownership vs Beta=0.31, Gamma=0.356
• Agency Control vs Beta = 0.554,
Gamma=0.554 strong positive
• Other conditions from both TLX and RH
questionnaire similar as overall
11/16/2015
23
VR Condition - Absolute
• Strongest: Ownership vs Beta = 0.293; Gamma
= 0.186, Delta = 0.368
• Ownership Control vs Theta
• Agency Control vs Beta
AR condition - Absolute
• Strongest: Ownership vs Beta = 0.396; vs
Gamma = 0.313
• Ownership Control vs Beta = 0.448
• Agency Control vs Alpha, Beta
• Weaker Ownership vs Gamma
• also NEGATIVE ownership vs Delta
Overall Results - Relative
• strong correlations: Ownership vs Alpha = -
0.254, Ownership vs Gamma = -0.23
• Owneship Control vs Delta = -0.224
• TLX Physical vs Alpha & TLX Physical vs Theta
(0.211, 0.219)
• TLX Temporal vs Gamma = 0.256
• TLX Performace vs Alpha & TLX Performance
vs Theta (0.22, 0.242)
Conclusions
• A lot of research is going on
• Wont see commercial applications very soon
• More studies are required
• Technology will get better and cheaper soon
Questions