Contents lists available at ScienceDirect
Brain and Cognition
journal homepage: www.elsevier.com/locate/b&c
Perception of direct vs. averted gaze in portrait paintings: An fMRI and eyetracking
study
Ladislav Kesnera,b,⁎
, Dominika Grygarováa,b
, Iveta Fajnerováa
, Jiří Lukavskýa,d
,
Tereza Nekovářováa
, Jaroslav Tintěraa,c
, Yuliya Zaytsevaa
, Jiří Horáčeka
a
National Institute of Mental Health, Topolová 748, 250 67 Klecany, Czech Republic
b
Department of Art History, Masaryk University Brno, Arne Nováka 1, 602 00, Czech Republic
c
Institute of Clinical and Experimental Medicine, Vídeňská 1958/9, 140 21 Prague, Czech Republic
d
Institute of Psychology, Czech Academy of Sciences, Hybernská 8, 110 00 Prague, Czech Republic
A R T I C L E I N F O
Keywords:
Direct gaze
Averted gaze
Artistic portraits
Perception
Eye-tracking
fMRI
A B S T R A C T
In this study, we use separate eye-tracking measurements and functional magnetic resonance imaging to investigate
the neuronal and behavioral response to painted portraits with direct versus averted gaze. We further
explored modulatory eﬀects of several painting characteristics (premodern vs modern period, inﬂuence of style
and pictorial context). In the fMRI experiment, we show that the direct versus averted gaze elicited increased
activation in lingual and inferior occipital and the fusiform face area, as well as in several areas involved in
attentional and social cognitive processes, especially the theory of mind: angular gyrus/temporo-parietal junction,
inferior frontal gyrus and dorsolateral prefrontal cortex. The additional eye-tracking experiment showed
that participants spent more time viewing the portrait’s eyes and mouth when the portrait’s gaze was directed
towards the observer. These results suggest that static and, in some cases, highly stylized depictions of human
beings in artistic portraits elicit brain activation commensurate with the experience of being observed by a
watchful intelligent being. They thus involve observers in implicit inferences of the painted subject’s mental
states and emotions. We further conﬁrm the substantial inﬂuence of representational medium on brain activity.
1. Introduction
1.1. Gaze in art theory and cognitive neuroscience
Gaze represents an important topic in both art history/visual studies
and cognitive neuroscience. Art historians and art critics have written
extensively on both various forms of gaze depicted in ﬁgural representations
in paintings, photography and videoart, as well as on
practices of gazing at works of art (e.g. Bryson, 1983; Wollheim, 1987;
Belting, 2009). They distinguish various dimensions of depicted gazes
(e.g. duration of the gaze, its “power”, or sexual allure) but most of
these qualities cannot be easily objectiﬁed or studied by empirical
methods. One aspect of gaze, however, which features prominently in
both art historical accounts and scientiﬁc examination is the direction
of the gaze.
It is well-established that gaze direction is a critical facial cue,
essential for social interaction and cognition (Argyle & Cook, 1976;
Frischen, Bayliss, & Tipper, 2007; George & Conty, 2008; Gibson &
Pick, 1963; Hamilton, 2016). As an instrument of social
communication, it modiﬁes the perception of emotions and enables
decoding of mental states, related to the process of theory-of-mind or
mentalizing. Gaze direction is a key element of socially relevant
signaling encoded in and decoded from faces. Eye contact, modulates
cognitive processing, particularly enabling to read or to see the
minds of others in direct mutual interaction (Baron-Cohen, 1995;
Nummenmaa & Calder, 2009; Senju & Johnson, 2009; Stawarska,
2006). Depending on the context, it can have various meaning,
ranging from an expression of intimacy to that of dominance or
hierarchy (George & Conty, 2008). In terms of Bayesian theories of
the human brain, humans have prior expectation (priors) that other’s
gaze is directed toward them (Mareschal, Calder, & Cliﬀord, 2013).
Contemporary research thus sheds some light on the practice of visual
artists who for centuries intuitively manipulated the direction of
the gaze of depicted persons to imbue their works with distinct
psychological eﬀects. Eye contact was given prominence in Roman
portrait busts, which often express an awareness of the viewer’s gaze
and initiate – probably for the ﬁrst time in history of art – direct
scopic and thus potentially psychic interaction with beholder
https://doi.org/10.1016/j.bandc.2018.06.004
Received 15 June 2017; Received in revised form 7 June 2018; Accepted 8 June 2018
⁎
Corresponding author at: National Institute of Mental Health, Topolová 748, 250 67 Klecany, Czech Republic.
E-mail address: Ladislav.Kesner@nudz.cz (L. Kesner).
Brain and Cognition 125 (2018) 88–99
0278-2626/ © 2018 Elsevier Inc. All rights reserved.
T
(Nodelman, 1975). This potential has then been further explored in
some of the arguably best Renaissance and modern portraits, where
the gaze of the depicted subject returns the gaze of the viewer, or –
on the other hand – avoids it in various ways. In summary, artists
have been exploring the potential of a painted gaze for centuries,
deploying it in speciﬁc ways and in conjunction with facial expression,
to imbue portraits with distinct meanings and eﬀects and they
continue to do so (Kesner, 2011).
Several neuroimaging experiments have explored the neural correlates
of direct versus averted gaze. The brain areas that respond differently
to eye contact and averted gaze include the posterior superior
temporal sulcus (STS) linked to processing of gaze shifts, the amygdala
engaged in processing of threatening and ambiguous stimuli, the fusiform
gyrus, the orbitofrontal cortex, as well as regions involved in selfrelated
and complex social-cognitive processing and theory of mind/
mentalizing (the paracingulate part of medial prefrontal cortex and
temporo-parietal junction, TPJ) (for a review see Senju & Johnson,
2009; Nummenmaa & Calder, 2009; Hamilton, 2016). However, many
inconsistencies among the ﬁndings prompt further research in this area.
One of the speciﬁc problems concerns the role of speciﬁc visual modality,
which conveys the face and gaze stimulus.
1.2. Aims and hypotheses
Neuroimaging studies on gaze direction conducted so far have used
naturalistic photographs of isolated eyes and eyes in a face, in which
the gaze direction is sometimes digitally manipulated (e.g. Kampe,
Frith, & Frith, 2003; Mason, Tatkow, & Macrae, 2005; Straube, 2010;
Berchio et al., 2016), symbolic line drawings (Friesen & Kingstone,
1998), video clips (Kuzmanovic et al., 2009; Pelphrey, Morris, &
McCarthy, 2004), computer-generated agents (Schilbach et al., 2006;
Wilms et al., 2010) and, most recently, also live people (Cavallo et al.,
2015; Debruille, Brodeur, & Franco Porras, 2012; Myllyneva &
Hietanen, 2015a, 2015b; Pönkänen et al., 2011). Diﬀerent types of
stimuli have diﬀerent levels of ecological validity, social richness and
potential to engage an audience eﬀect (Hamilton, 2016; Risko, Laidlaw,
Freeth, Foulsham, & Kingstone, 2012). So far, there is only scant evidence
of the eﬀect of representational medium in the perception of gaze
direction. Several studies using event-related potential (ERP) N170
showed that the physical and structural characteristics (photographs or
impoverished line-drawn faces) of the stimulus drive and modulate the
response in favor of the photographs (Puce, Smith, & Allison, 2000;
Rossi, Parada, Kolchinsky, & Puce, 2014; Rossi, Parada, Latinus, & Puce,
2015). Congruently, a comparison of responses to a directly gazing live
person, photograph and dummy showed that ERPs in early windows
(125–170 and 170–230 ms) depended on the nature of the stimulus and
in N300 were signiﬁcantly more negative in the case of the dummy
(Debruille et al., 2012). However, it is not clear if the ﬁndings from
these studies are generalizable to brain processing of artistic portraits.
Clearly, the problem if (how) the representational medium modiﬁes the
neurobiological response to human faces and gazes prompts further
investigation.
Our study, consisting of a separate fMRI and eye-tracking experiment,
was designed to explore the neuronal and behavioral response to
painted portraits with direct versus averted gaze. This study addressed
two questions. First, we aimed to identify how the neural and behavioral
response to emotional expressive faces in paintings is modulated
by the direction of gaze. In other words, we asked if direct (vs. averted)
gaze in painterly portraits aﬀects the saliency of the social cognition
brain areas. To identify the behavioral inﬂuence of the gaze direction in
artistic portraits on the beholder’s eye movements and visual scanning,
we complemented the fMRI study by performing separate eye-tracking
measurements. Second, to identify the modulatory eﬀects of painting
characteristics on brain activation, we further resampled our artistic
portraits series according to three contrasting factors: (i) period of the
painting (pre-modernist vs. modernist paintings); (ii) inﬂuence of style
(painterly vs. linear),1
and (iii) pictorial context of the portrait (face only
vs. face and torso vs. face, torso and hand gesture).
We hypothesized that ﬁrstly; direct vs averted gaze will exert different
brain activation similar to previous studies with photographs and
videos. Speciﬁcally, direct gaze will be linked to activity in the fusiform
gyrus and in brain areas processing mentalizing (paracingulate cortex,
MPFC and TPJ). The separate eye-tracking experiment addressed the
question of the behavioral eﬀect of portrait characteristics. We expected
that (i) beholders will spend more time viewing the eyes of the depicted
person with direct gaze, than under an averted gaze condition; (ii) the
representational medium characterized by the modern and painterly
styles (i.e. less realistic) will engage cortical regions responsible for
higher level of visual processing more intensively than realistic pictures
(typically a linear style of painting). We assumed that this eﬀect stems
from the fact that more formalized or stylized/less realistic pictures
need more eﬀort (with larger hemodynamic response) to be recognized
and processed properly. We further assumed that the speciﬁc faceprocessing
brain areas (e.g. the fusiform gyrus) would be activated
more intensively when viewing portraits depicting only faces versus
portraits also depicting body and hand gestures.
2. Materials and methods
2.1. fMRI experiment
2.1.1. Stimuli
A set of artistic portraits was used as stimuli for the fMRI experiment.
The portraits were organized in duplets – each duplet contains
two portraits from the same artist, the ﬁrst portrait being classiﬁed as
“direct gaze” (the portrayed person gazes directly at the observer), the
second one as “averted gaze” (the portrayed person gazes to the side or
behind the observer). The direct gaze (or eye contact) group included
both images with the face directed forward and faces averted at various
degrees. The original set of 110 portraits from 32 artists was ﬁrst assessed
by 11 volunteers in a pilot study validating the set of stimuli. The
evaluators were to decide whether the portrayed person was looking
directly at them (direct gaze) or not (averted gaze). The portraits with a
consensus of a minority of less than seven evaluators were excluded
from the ﬁnal stimuli set together with the paired stimulus (from the
same artist). The ﬁnal set of 72 portraits (36 duplets, 72% with a
consensus of at least 10 evaluators) is from 27 artists from various
periods and provenances, beginning with Flemish Early Renaissance
(Jan van Eyck, Hans Memling, Dieric Bouts, Petrus Christus, anonymous),
Italian Renaissance and Mannerism (Domenico Ghirlandaio,
Agnolo Bronzino), German Renaissance (Albrecht Dürer, Matthias
Grünewald), followed by Baroque paintings (Christian Seybold, Pietro
Antonio Rotari), modernism (Paul Cézanne, Oskar Kokoschka, Egon
Schiele, Max Beckmann, Christian Schad, Charley Toorop, Paula
Modersohn-Becker, Kathe Kollwitz, Vilma Vrbová-Kotrbová, Jan
Preisler, Josef Šíma), and postmodern art: pop art with its typical comic
book style (Roy Lichtenstein), or new ﬁguration with distorted or
1
Here we adopted the historical categories of linear (or tactile) versus painterly style,
introduced by art historians Heinrich Wölﬄin and Alois Riegl. The linear style is characterized
by sharp deﬁnition of the form (e.g. Albrecht Dürer’s portraits): the style emphasizes
contours, it radically diﬀerentiates the ﬁgure from the background or each individual
shapes between each other. The ﬁgure is clearly shaped into a precisely
“graspable” plastic shape, which can even induce “the feeling of touch”, according to
Wölﬄin, such as we somehow touch the precisely sculptured ﬁgures with our eyes
(Wölﬄin, 1950: 21). Conversely, the element of physical touch or grasp is missing in
painterly style, which does not diﬀerentiate shapes and the ﬁgure from the background so
clearly, as it is organically interconnected, having its origin in the same material. Instead
of precise lines and smooth surfaces, the painterly style makes the medium visible and
thus the way the painting has been made (perceptible brushstrokes, blotches etc.; e.g.
Oskar Kokoschka). Although it could invoke an intensive eﬀect of plasticity or depth, it
disrupts the “graspable feeling” of ﬁgures, as Wölﬄin suggested (Wölﬄin, 1950: 21),
therefore paintings rendered in a painterly style engage only vision and lack the intention
to grasp the ﬁgure.
L. Kesner et al. Brain and Cognition 125 (2018) 88–99
89
blurred faces (Francis Bacon, Chuck Close, Marlene Dumas, Jenny
Saville – see Supplementary Material). The pictures were gathered from
various public websites and online databases. The size of the pictures
varied, therefore we used Adobe Photoshop CS5 for the adjustment of
all the stimuli to the resolution of the MRI projector and of the monitor
screen used in the eye-tracking experiment (1024 × 768 pixels).
The portraits were divided into twelve blocks per six portraits (either
“direct gaze” or “averted gaze”). Both sets of portraits were carefully matched
by artist (duplets of direct and averted gaze portraits by the same
artist). The blocks were balanced by average luminance and RGB color
components assessed with ZONER photo studio software (the performed
ANOVA showed no block × luminance/color interaction: p > 0.05;
Luminance: F(1, 11) = 0.389; Red: F(1, 11) = 0.466; Green: F(1,
11) = 0.496; Blue: F(1, 11) = 0.241). In addition, the size of the face, eyes
and mouth areas of individual portraits was compared between direct and
averted gaze stimuli using Mann-Whitney/Wilcoxon test, showing no signiﬁcant
diﬀerences (eyes: Wilcoxon W = 665, p = 0.854; mouth: W = 697,
p = 0.585; face: W = 665, p = 0.8534). Aside from these characteristics, all
of the blocks were also counterbalanced by two experimenters for the following
parameters (meaning each block contains the same amount of
portraits with speciﬁc characteristics): artistic techniques (painting vs.
drawing) and pictorial context of the portraits (detailed face/face only; face
and torso without hands; face and torso with hand gestures) in order to
suppress the possible eﬀect of these characteristics on the measured brain
activity. Notice that the pictorial context characteristic was later applied as a
contrasting factor in the fMRI analysis together with two other characteristics:
period of the painting (pre-modernist vs. modernist paintings) and inﬂuence
of style (painterly vs. linear). All of the three mentioned characteristics
were evaluated ad hoc by 19 art experts and students of art history in
order to classify the portrait categories.
2.1.2. Participants
The fMRI group (Experiment 1) consisted of twenty-four healthy
adult volunteers (11 females, 13 males, all right-handed, with a mean
age of 30 years old (SD = 6), education years 17 (SD = 2.5), and with
no previous expertise in art history.
The eye-tracking sample (Experiment 2) consisted of a diﬀerent
twenty-ﬁve healthy volunteers (mean age of 26 years, range 20–48; 11
women) again naive in art history. Due to inaccurate drift correction in
one of the subjects, only data from 24 participants were analyzed.
All participants had normal or corrected-to-normal vision. The investigation
was carried out in accordance with the latest version of the
Declaration of Helsinki. We obtained written informed consent from all
of the participants, and the local ethics committee approved the study.
2.1.3. fMRI data acquisition
The fMRI experiment was performed using a 3 T Siemens Trio MRI
scanner (Erlangen, Germany). All of the participants had their heads
ﬁrmly immobilized in a 12-channel head coil using foam inserts positioned
on either side of the head. For stimulation by the artistic portraits,
we applied the fMRI paradigm with contrasting ﬁxation versus
direct and averted gaze, and both gaze series between each other.
During the ﬁxation blocks, the subjects looked at a black screen with a
white cross in the middle. The reproductions of 72 portraits (36 duplets)
were displayed in a block design. The participants were instructed
to passively view the portraits displayed by the LCD projector.
The direct and averted gaze portraits were presented in 12 alternating
36-second blocks, 6 with “direct gaze” and 6 with “averted gaze”. Each
portrait was presented for 6 s, with a 3-second inter-scan interval (ISI; 2
scans per portrait). An additional control condition was six ﬁxation
blocks lasting 36 s. All of the three conditions were presented in a
pseudorandom order. In total, 216 brain volumes were acquired. The
total acquisition time of each fMRI was 10 min, 48 s.
The fMRI images (T2*
-weighted gradient echo-planar imaging sequence)
of 45 axial slices (3 mm thickness, no gap) parallel to the
anterior and posterior commissure covering the whole brain were taken
under the following conditions: echo time (TE) 30 ms, repetition time
(TR) 3000 ms, matrix 64 × 64, bandwidth 1594 Hz/pixel, ﬂip angle
90°. The ﬁeld of view (FOV) was 192 mm and the in-plane spatial resolution
was 3 × 3 mm2
. For the fMRI data preprocessing, the participants
were scanned using a T1-weighted 3D-MPRAGE sequence.
2.1.4. fMRI data analysis
The fMRI data were ﬁrst analyzed using Statistical Parametric
Mapping software version 8 (SPM8; www.ﬁl.ion.ucl.ac.uk/spm/
software/spm8) implemented in MATLAB version 7.3 (MathWorks).
Statistical analysis of the data across all of the participants proceeded
by entering each participant’s preprocessed functional data (realigned
to correct for movement, normalized and smoothed using a Gaussian
kernel of 8 mm FWHM) into a generalized linear model. Low-frequency
noise was removed using a highpass ﬁlter (128 s). The ﬁrst-level design
matrix contained factors modelling regressors (in blocs) for the ﬁxation,
direct and averted gaze conditions convolved with a canonical hemodynamic
response function. For the secondary objectives, we estimated
by event-related design approach the eﬀects of painting techniques and
characteristics (one-way ANOVAs) for a) old and modern style of
painting, b) linear and painting style, and c) depicted details of portraits
presented during scanning. For this purpose, the portraits were sorted
into three groups: face only (F), face and torso (FT) and face, torso and
hand gesture (FTG).
The contrast images (linear combinations of β images) were then
subjected to a second-level analysis to determine stimulus-speciﬁc regional
responses at a group level (one sample T- test). For control type I
error, we accepted as signiﬁcant only conservative family-wise error
(FWE) corrected ﬁndings (p ≤ 0.05) with the extent threshold consisting
of 50 voxels per cluster for second level comparisons between
portraits and ﬁxation as a control condition. The coordinates for local
maxima using the MNI (Montreal Neurological Institute) template were
converted to stereotactic Talairach x, y, z coordinates (Lancaster et al.,
2007). The results were displayed using MRIcro V1.4 (www.mricro.
com) and its template.
2.2. Eye-tracking experiment
The set of artistic portrait stimuli used in the previous fMRI study was
extended by an additional 22 stimuli (11 for each category of gaze direction)
which we were speciﬁcally interested in obtaining eye-tracking
data for. These consisted of expressionist and cubist portraits from the
period between 1900 and 1960, by European artists (Pablo Picasso,
Umberto Boccioni, Bohumil Kubišta, Emil Filla, Antonín Procházka, Jan
Zrzavý, Francis Bacon) and applied in the eye-tracking experiment. After
showing four additional portraits as training, the completed set of 94 stimuli
was presented in randomized order in two blocks. The portraits were
each displayed on a 19″ computer screen for 6 s. As previously, the volunteers
were asked to look at the portraits the same as they would in a
gallery, while their gaze-direction was recorded.
The eye movements were measured with the EyeLinkII eye-tracker
(SR Research Ltd) with a head-mounted camera. The participants
viewed portraits from approximately 60 cm (to achieve as natural
viewing conditions as possible, no chinrest was used to enforce constant
distance). We used a 9-dot grid calibration prior to each block and drift
correction prior to each trial. The gaze samples were collected at
250 Hz. We collected data from one eye only (selected based on the
results of the calibration). After ﬁltering the data for blinks, we removed
the trials where a participant spent less than 3 s (50% of time)
looking at the portrait due to blinks or looking beyond the portrait
boundaries. The total dataset for the analysis consisted of 2210 trials
(88–94 per participant, representing 94–100%). We deﬁned three areas
of interest (AOI) manually for each portrait: area of the eyes, mouth,
face (excluding the other two areas, see Fig. 3). We decided to create
eyes and mouth AOI, unlike other local face features such as nose or
forehead, because eyes and mouth are considered not only the core face
L. Kesner et al. Brain and Cognition 125 (2018) 88–99
90
features, but also the main cues for detecting emotions in other people
and for communication (e.g. Morris, Pelphrey, & McCarthy, 2007;
Wegrzyn, Vogt, Kireclioglu, Schneider, & Kissler, 2017). The eyes AOI
covered the area for both eyes and the area around and between them.
We calculated the corresponding dwell times for each AOI. The size of
AOI varied across individual paintings, however the diﬀerences in AOI
size between direct and averted gaze portraits were not signiﬁcant. We
examined whether the extent of AOIs diﬀers between portraits with
direct vs. averted gaze (measured in pixels). The eyes AOI subtended on
average 23,580 pixels (SD = 27,231, median = 11,372, range
2160–154,069), the mouth AOI subtended on average 8560 pixels
(SD = 10,746, median = 4243, range 616–55,759). The face AOI subtended
on average 50,877 pixels (SD = 52,970, median = 28,063,
range 3940–243,599). We compared the areas in portraits with direct
vs. averted gaze and we found no diﬀerence in their extent (eyes:
Wilcoxon W = 1019, p = 0.522; mouth: W = 1018, p = 0.516; face:
W = 1071, p = 0.804; all p values are uncorrected).
2.3. Behavioral measurements and psychometrics (Experiment 1 and 2)
Participants in both the fMRI and eye-tracking experiments were
asked about the characteristics of the individual portraits presented
during the fMRI/eye-tracking paradigm in a debrieﬁng questionnaire
(stimuli were presented on the computer screen). The participants
evaluated the gaze direction of the portrayed persons on a scale from
−5 (averted gaze) to +5 (direct gaze). The subjects were also required
to evaluate the following characteristics of individual stimuli (on a
scales from −5 to +5): (1) Emotional valence (unpleasant/pleasant),
(2) Arousal (calmness/tension). In addition, (3) the Familiarity of the
individual pictures was rated (Yes/Maybe/No). In the fMRI experiment,
this evaluation was done after the scanning using a paper-pencil debrieﬁng
questionnaire (stimuli were presented on the computer screen,
in the same order as during the fMRI measurement), in the eye-tracking
experiment using a computerized debrieﬁng prepared in the PsychoPy
software (pictures presented in a pseudorandomized order). Statistical
analysis of the behavioral results was performed using STATISTICA 11
software (StatSoft). The non-parametric Mann-Whitney U test was used
to analyze the diﬀerences of direct and averted gaze portraits in terms
of the individual characteristics (Gaze direction, Emotional valence,
Arousal, Familiarity). The Spearman Rank Order Correlation analysis
was applied in order to calculate a possible relationship between gaze
direction and/or the individual characteristics.
3. Results
3.1. fMRI experiment
3.1.1. The eﬀect of direct and averted gaze portraits
Both direct and averted gaze portraits (compared to ﬁxation)
showed bilateral activation in occipital lobes including lingual and fusiform
gyrus and the hippocampus. The main diﬀerence in the averted
gaze portraits was the absence of activation in the left superior, inferior
and medial frontal gyri, and the right angular and superior frontal gyri
(FWE corrected p ≤ 0.05, Fig. 1, Table S1). Deactivation by both types
of portraits was found in default mode network regions areas covering
posterior midline structures (precuneus and posterior cingulate), and
bilateral temporo-parietal regions (angular, superior and middle temporal
gyri). The main diﬀerence between the two kinds of gaze was
identiﬁed in the anterior cingulate, which was deactivated for the direct
gaze portraits but not for the averted gaze portraits (FWE corrected
p ≤ 0.05, Fig. 1, Table S1).
Direct gaze portraits contrasted to the averted gaze portraits elicited
an increased BOLD signal in the occipital surfaces covering the calcarine
ﬁssure and surrounding the cortex bilaterally with the local
maxima in the right lingual gyrus and cuneus and this cluster reached
temporal lobe including fusiform gyrus (FWE corrected p ≤ 0.05).
Interestingly, the direct gaze portraits exerted higher activation in the
fusiform gyrus, right angular gyrus and in prefrontal regions (left
middle and the inferior frontal gyrus) but these results did not survive
conservative FWE correction (Fig. 1, Table 1). We did not ﬁnd any
region with higher activity for the averted gaze portraits.
3.1.2. The eﬀect of painting techniques and characteristics on brain activity
(fMRI)
Modernist style contrasted against pre-modern style showed a
greater activation in the occipital lobe bilaterally, notably in the lingual
gyri and in the right fusiform gyrus. The opposite contrast showed no
diﬀerences in neural processing of pre-modern paintings compared to
new ones (Fig. 2, Table 2). Painterly style contrasted against linear style
showed higher activation in the medial and inferior surfaces of the
occipital lobe (calcarine ﬁssure, cuneus, lingual and fusiform gyri) and
the eﬀect was more pronounced on the right side. The opposite contrast
showed no increase in linear compared to painterly style (Fig. 2,
Table 2).
3.1.3. The eﬀect of the pictorial context of paintings
Comparing portraits with faces only (F) and faces with torso (FT),
we found that faces activate more than the FT bilateral occipital cortex
with the local maxima in the right cuneus, the inferior occipital gyrus
and the left middle occipital gyrus (FWE p ≤ 0.05).
Portraits with faces only compared to those showing also torso with
hands (FTG) showed a higher BOLD signal in the lateral occipital cortex
including the fusiform, middle and inferior occipital gyri. The opposite
contrasts (FT > F, and FTG > F) showed no signiﬁcant results. FTG
exerted higher activation then FT bilaterally in the calcarine ﬁssure and
the surrounding cortex (with local maxima in the lingual gyrus) and in
the right thalamus. FT compared to FTG did not show any signiﬁcant
increase in the BOLD signal (Fig. 2, Table 2).
3.2. Eye-tracking experiment
We analyzed the diﬀerences in dwell time with 2 × 2 ANOVA (two
within-subject factors, gaze direction (direct, averted) and area of interest
(eyes, mouth)). The participants preferred looking at eyes rather
than at mouth (F(1, 23) = 131.97, p < 0.001, ηg
2
= 0.763)). When
the portrait’s gaze was directed towards the observer, participants spent
more time viewing its eyes and mouth (F(1, 23) = 149.76, p < 0.001,
ηg
2
= 0.042). The AOI × gaze interaction was signiﬁcant (F(1, 23) =
14.98, p < 0.001, ηg
2
= 0.012), which is caused by a larger eﬀect of
portrait’s gaze on eyes (mean diﬀerence = 212 ms, paired t-test, t(23)
= 8.025, p < 0.001) compared to mouth area (mean diﬀerence
= 68 ms, paired t-test, t(23) = 4.250, p < 0.001) (Fig. 3).
3.3. Behavioral eﬀect and psychometric measurements (Experiment 1 and
2)
The debrieﬁng of tested individuals showed that the subjective
evaluation of the direct and averted gaze directions of the portrayed
persons (scaled from −5 to +5) was in agreement with the two proposed
categories (showing signiﬁcant diﬀerences in the evaluated gaze
direction between the proposed categories averted/direct gaze) in both
the fMRI in Experiment 1 and the eye-tracking sample in Experiment 2
(for the detailed results see Supplementary Table S2). In the post-fMRI
debrieﬁng, no signiﬁcant diﬀerences were observed between the two
gaze direction categories in the two evaluated psychological characteristics
(‘Emotional valence’ and ‘Arousal’) (for the results see
Supplementary Table S2). However, ‘Arousal’ was signiﬁcantly higher
for the direct gaze portraits in the eye-tracking in Experiment 2
(p < 0.001), while no diﬀerence was observed in the ‘Emotional valence’
of the stimuli. Moderate negative correlations were also observed
between the evaluated Emotional valence and Arousal characteristics in
both experiments (1. fMRI: r = −0.45, p < 0.05; 2. Eye-tracking:
L. Kesner et al. Brain and Cognition 125 (2018) 88–99
91
r = −0.55, p < 0.05). Participants rated the portraits as unknown in
70% of all familiarity evaluations (cumulative for all participants and
all stimuli) and only in 11% as known (the remaining 19% of the
evaluations were undecided). Most of the stimuli were rated as known
by less than 2 participants. The familiarity parameter did not diﬀer
between the direct and averted gaze portrait categories in any of the
two experiments (p > 0.05).
4. Discussion
4.1. Direct vs. averted gaze
4.1.1. fMRI experiment
The present work was designed to explore the neuronal and behavioral
responses to painted portraits with a direct versus averted gaze.
We identiﬁed brain areas, which were selectively activated by a direct
versus averted gaze in artistic (painted) portraits. We further identiﬁed
brain areas functionally related to the distinct painting characteristics
such as the period, general style (“linear” vs. “painterly”) and pictorial
context. Not surprisingly, we found that both direct and averted gaze
portraits (compared to ﬁxation) showed bilateral activation in occipital
lobes with local maxima in lingual and fusiform gyrus. This is fully in
accord with inﬂuential models of face perception, which propose that
FFA is involved in expression of invariant processing of faces (Calder &
Young, 2005; Haxby, Hoﬀman, & Gobbini, 2000), as well as with more
recent integrative models of face perception, which argue that the FFA
contributes to the perception of changeable aspects of faces, through a
broad sensitivity to shape information (Bernstein, Erez, & Yovel, 2018;
Duchaine & Yovel, 2015).
Enhanced activity in the fusiform gyrus during direct gaze perception
corresponds to several studies using photographic stimuli (George,
Driver, & Dolan, 2001; Pageler et al., 2003). It has been suggested that a
higher activation of the fusiform gyrus in these studies, which - much
like our own study - involved presentation of many diﬀerent faces, may
reﬂect the need for a “deeper” processing and encoding of novel faces
indicating that the interaction between a beholder and the direct gaze
of the subjects in the painted portraits could be associated with the
increasing feeling of novelty (Boyarskaya, Sebastian, Bauermann,
Hecht, & Tuscher, 2015). However, at present, it is not possible to determine
if diﬀerences in gaze direction have lesser impact on FG activity
compared to the processing of whole face, as some have argued
(Madipakkam, Rothkirch, Guggenmos, Heinz, & Sterzer, 2015), as these
aspects cannot be reliably disentangled.
There are two possible explanations for our ﬁnding of enhanced FG
activity to direct gaze portraits. First, it has been established that face
Fig. 1. The eﬀect of direct and averted gaze
on brain activity. Brain response to portraits
of direct and averted gaze are compared to
ﬁxation cross viewing (two left columns).
Increase (yellow-red) and decrease (blue)
induced by portraits are displayed on corresponding
MRI slices (FWE corrected
p ≤ 0.05, exact statistical results are in
Supplementary Table S1). Two right columns
show the regions in which direct gaze
activated more than averted gaze portraits.
These results are displayed at uncorr. pvalue
p ≤ 0.001. For exact anatomical localization
and description, see Table 1. Legend:
L or R, left or right hemisphere; x, y, z,
Talairach coordinates of the displayed slices
of MRIcro V1.4 (www.mricro.com) template.
(For interpretation of the references to
colour in this ﬁgure legend, the reader is
referred to the web version of this article.)
Table 1
Activation of direct compared to averted gaze portraits.
Cluster vx p(FWE) T Z x y z R or L Local maximum
Activation of direct compared to averted gaze portraits
2926 0.013 6.68 4.97 14 −91 −4 R Lingual Gyrus
0.015 6.62 4.94 6 −93 10 R Cuneus
0.079 5.75 4.51 15 −90 7 R Cuneus
670 0.052 5.96 4.62 −20 −91 −6 L Inferior
Occipital Gyrus
0.248 5.12 4.17 −25 −77 −12 L Fusiform Gyrus
325 0.219 5.19 4.21 −40 −48 −17 L Fusiform Gyrus
0.772 4.27 3.64 −44 −62 −13 L Fusiform Gyrus
0.848 4.14 3.56 −49 −53 −7 L Inferior
Temporal Gyrus
227 0.450 4.74 3.94 −33 26 24 L Middle Frontal
Gyrus
0.608 4.51 3.8 −40 28 24 L Middle Frontal
Gyrus
110 0.814 4.2 3.6 38 16 28 R Middle Frontal
Gyrus
0.941 3.91 3.41 38 9 29 R Inferior Frontal
Gyrus
171 0.854 4.12 3.55 30 −60 42 R Angular gyrus
Signiﬁcant results at uncorrected p ≤ 0.001 for clusters consisting of ≥50
voxels are displayed. Legend: cluster vx, size in voxels; p(FWE), family-wise
error corrected p-value; T, T-value; Z, Z-value equivalent; x, y, z, Talairach
coordinates of voxel of maximum signiﬁcance; R or L, right or left hemisphere.
L. Kesner et al. Brain and Cognition 125 (2018) 88–99
92
processing in this region depends on voluntary attention. Several studies
of face processing in ventral occipitotemporal cortex proved that
FG activation as indexed by BOLD signal or N170 is ampliﬁed by attention
(Pessoa et al., 2002; Srinivasan, Srivastava, Lohani, & Baijal,
2009; Vuilleumier, Armony, Driver, & Dolan, 2001; Wojciulik,
Kanwisher, & Driver, 1998). Morris and colleagues, who examined the
relationship between scanpath variations and FG activity, demonstrated
that activation in the fusiform gyrus correlates with the amount of
ﬁxation in the eyes and mouths (Morris et al., 2007). This amplifying
role of attention on FG is congruent with both our eye-tracking data,
which showed that the participants spent more time viewing both the
person’s eyes and mouth when observing the direct gaze portraits than
the averted ones, as well as with previous eye-tracking studies (for a
review see Pfeiﬀer, Vogeley & Schilbach, 2013). This suggests that a
direct gaze prepares for face-to-face communication more than an
averted gaze even in painterly (ie less realistic) artistic portraits.
However, it should be noted that the individual evaluation of stimuli by
our participants during debrieﬁng showed higher arousal in direct gaze
only after the eye tracking experiment, but not after the fMRI experiment.
This discrepancy between these two behavioural evaluations
could have been caused by the diﬀerent character of the two procedures
applied. We suggest that the MRI measurement itself (exceeding the
length of the eye-tracking study due to morphological sequences measured
consecutively after the fMRI task) could have induced decreased
attention during introspection process of post-scanning debrieﬁng.
Second, FG/FFA region has been increasingly recognized to have
more important set of functions, involving processing of emotions
(Ganel, Valyear, Goshen-Gottstein, & Goodale, 2005; Harry, Williams,
Davis, & Kim, 2013; Kawasaki et al., 2012). Moreover, FG is now often
considered an integral part of social perception and cognition (Schultz
et al., 2003; Stolier & Freeman, 2016; Tso, Rutherford, Fang, Angstadt,
& Taylor, 2018) and it most likely directly contributes to mentalizing
processes (Baetens, Ma, Steen, & Van Overwalle, 2014; Ohnishi et al.,
2004). It is thus possible to speculate that FG activation to direct vs
averted gaze of painted portraits may also in part be interpreted as
contributing to implicit inference of mental states of depicted persons as
further discussed below.
In our study, the regions in the occipital lobe activated by the
perception of the direct gaze included the cuneus and the lingual gyrus.
Activation of these structures has been linked to encoding of complex
images during viewing of the paintings (Vartanian & Skov, 2014). Besides
its role in the visual identiﬁcation, the lingual gyrus is essential for
face recognition (Kozlovskiy et al., 2014). As it was demonstrated by
McCarthy, Puce, Belger, and Allison (1999), the occipito-temporal
cortex, including the lingual gyrus, emits ERP as a response to faces.
The N200 amplitude was evoked by varying faces - colored or grayscale;
normal, blurred or line-drawing faces; or by faces of diﬀerent
sizes.
Most importantly, we also observed parietal (angular gyrus) and
prefrontal cortex brain response to direct but not averted gaze portraits.
The activation in the right angular gyrus suggests the involvement of
social cognitive processes, especially the theory of mind (Seghier,
2013). The angular gyrus is involved in social cognition (as a part of
temporo-parietal junction) on the one hand and attentional selection on
the other, but it subserves other cognitive functions as well (Carter &
Huettel, 2013; Corbetta and Shulman, 2002; Igelström, Webb, &
Graziano, 2015; Krall et al., 2015; Saxe & Kanwisher, 2003; Schurz,
Radua, Aichhorn, Richlan, & Perner, 2014; Seghier, 2013). One can
assume that eye contact with the people depicted in the portraits engages
the viewers in inferring emotions and identiﬁcation of the mental
states of the depicted subjects (we discuss this in more detail below). In
addition to the regions in the occipital and fronto-parietal lobes, we
observed brain activations by direct gaze portraits in two prefrontal
areas: the inferior frontal gyrus (IFG) and the middle frontal gyrus. IFG
has been implicated in a number of speciﬁc cognitive functions related
to the theory of mind. Importantly, a recent meta-analysis (Schurz
et al., 2014, see also Schurz & Tholen, 2016) demonstrated that the left
IFG was activated by the type of experimental tasks including social
animations and “reading mind in the eyes”. Recently, Cavallo and
colleagues have found prominent IFG activation in direct gaze condition
elicited by live experimenter. They interpret this IFG activity as
related to the onset of a communicative intent and a preparation of a
communicative response (Cavallo et al., 2015). IFG also regulates
amygdala activity in emotional face processing (Horáček et al., 2015).
Furthermore, it should be noted that this activation in right IFG probably
anatomically overlaps with right inferior frontal junction, which
Fig. 2. The eﬀect of painting style (left) and
pictorial context on brain activity (right). The
left part shows regions where modernist painting
style portraits activated more than the premodern
ones (upper left), and where painterly
style activated more than the linear one (lower
left). The right part displays the diﬀerences between
portraits of diﬀerent pictorial context of the
portrait. Results are displayed at uncorr. p-value
p ≤ 0.001 for clusters located within the mask of
all portraits eﬀect vs ﬁxation (FWE, p ≤ 0.05).
Anatomical localization and exact statistical results
are in Table 2. Legend: L or R, left or right
hemisphere; x, y, z, Talairach coordinates of the
displayed slices of MRIcro V1.4 (www.mricro.
com) template; F, face only; FT, face and torso;
FTG, torso and hand gesture. (For interpretation
of the references to colour in this ﬁgure legend,
the reader is referred to the web version of this
article.)
L. Kesner et al. Brain and Cognition 125 (2018) 88–99
93
has been recently implicated in face and speciﬁcally eyes processing
(Chan & Downing, 2011).
Rather unexpectedly, we found that direct gaze compared to averted
gaze portraits also bilaterally activated the middle frontal gyrus, an
area co-extensive with the dorsolateral prefrontal cortex (DLPFC, Fig. 1,
Table 1). DLPFC has not generally been involved in gaze processing and
social cognition. However, Bzdok et al. (2012) found that DLPFC also
subserved recruited social judgment (trustworthiness) and age assessment
of naturalistic faces. The authors hypothesize that this activity
might reﬂect the cognitive eﬀort involved in the decision they were
required to make (comparing two faces). Activation of DLPFC was also
found in an aesthetic evaluation of art works (Kirk, Downar, &
Montague, 2011; Kirk, Skov, Hulme, Christensen, & Zeki, 2009). One
way to account for our ﬁnding of activation in DLPFC would be with
reference to Senju and Johnson’s (2009) recent “fast-track modulator
model”, which hypothesizes the key role of DLPFC in the context- and
task-relevant modulation of the key social brain areas involved in the
detection of gaze direction, such as the superior temporal sulcus (STS),
and intentionality, the medial prefrontal cortex (MPFC) and the TPJ.
Another possibility is that the DLPFC activity may be related to the
fact that - contrary to several studies on gaze processing, using
photographs or computer avatars (Kampe et al., 2003; Schilbach et al.,
2006) - we did not ﬁnd activation by direct gaze portraits in the ventromedial
prefrontal cortex (VMPFC). VMPFC is a key region in anterior
component of DMN and has been assigned a central role in generation
of aﬀective meaning, valuation, situated conceptualization of emotion
and self-related processing (for overview see Delgado et al, 2016).
VMPFC has also been associated with social cognition, and is theorized
to play a role in social judgments by simulating mental states that
evaluate one's own and others' behaviours (Flagan & Beer, 2013; Schurz
et al., 2014). The absence of VMPFC activity can be attributed to two
factors. First, the fact that our participants were merely required to
watch paintings as if they were in a gallery and were not explicitly
requested to evaluate the social or emotional content of the portraits
during fMRI paradigm that would evoke activity in more extensive
parts of the social brain network. Incidentally (as observed by Senju &
Johnson, 2009) in studies reporting VMPFC activation in response to a
direct gaze, participants were explicitly asked to decode the intention of
the face to communicate, which may have elicited greater activation for
the direct gaze. Secondly, also relevant is the fact that emotional processing
in VMPFC can be attenuated by DLPFC in evaluation processes
(Hare, Camerer, Rangel, & Hare, 2009; Kirk et al., 2011) and
Table 2
Activation by modernist style portraits comparing to the pre-modern ones.
Cluster vx p(FWE) T Z x y z R or L Local maximum
5519 0.000 8.84 5.84 12 −85 4 R Lingual Gyrus
0.001 8.05 5.55 −7 −86 3 L Lingual Gyrus
0.006 7.21 5.21 −11 −79 2 L Lingual Gyrus
72 0.660 4.53 3.81 26 11 63 R Superior Frontal Gyrus
225 0.683 4.5 3.79 19 −47 −11 R Fusiform Gyrus
0.952 3.98 3.45 25 −36 −15 R Fusiform Gyrus
0.998 3.57 3.16 16 −57 −8 R Lingual Gyrus
Activation by painterly style portraits comparing to the linear ones
Cluster vx p(FWE) T equivZ x y z R or L Local maximum
4871 0.007 7.05 5.14 8 −86 −2 R Lingual Gyrus
0.008 7.02 5.13 10 −97 −3 R Cuneus
0.010 6.89 5.07 −5 −88 3 L Lingual Gyrus
119 0.898 4.12 3.54 23 −47 −12 R Fusiform Gyrus
0.952 3.96 3.44 25 −40 −13 R Fusiform Gyrus
0.996 3.62 3.2 21 −57 −10 R Lingual Gyrus
Depicted details of portraits: F > FT
Cluster vx p(FWE) T Z x y z R or L Local maximum
20,619 0.000 14.74 7.38 15 −92 3 R Cuneus
0.000 12.06 6.79 19 −89 −9 R Inferior Occipital Gyrus
0.000 11.1 6.54 −27 −92 2 L Middle Occipital Gyrus
141 0.275 5.11 4.16 −37 −52 60 L Superior parietal Lobule
0.880 4.13 3.55 −28 −57 51 L Superior Parietal Lobule
115 0.631 4.53 3.81 39 −50 58 R Superior Parietal Lobule
0.937 3.98 3.46 31 −56 61 R Superior Parietal Lobule
Depicted details of portraits: F > FTG
Cluster vx p(FWE) T Z x y z R or L Local maximum
2230 0.017 6.67 4.97 27 −81 −12 R Fusiform Gyrus
0.022 6.55 4.91 17 −89 −6 R Inferior Occipital Gyrus
0.037 6.28 4.78 25 −89 −7 R Inferior Occipital Gyrus
1615 0.027 6.44 4.86 −33 −85 −12 L Inferior Occipital Gyrus
0.083 5.85 4.57 −27 −92 4 L Middle Occipital Gyrus
0.102 5.75 4.51 −20 −92 −14 L Fusiform Gyrus
Depicted details of portraits: FTG > FT
Cluster vx p(FWE) T Z x y z R or L Local maximum
22,994 0.000 15.19 7.47 8 −90 0 R Lingual Gyrus
0.000 12.13 6.81 −9 −82 −7 L Lingual Gyrus
0.000 11.98 6.77 12 −85 −11 R Lingual Gyrus
234 0.047 6.15 4.72 21 −29 9 R Thalamus
Signiﬁcant results at uncorrected p ≤ 0.001 for clusters consisting of ≥50 voxels are displayed. Legend: cluster vx, size in voxels; p(FWE), family-wise error corrected
p-value; T, T-value; Z, Z-value equivalent; x, y, z, Talairach coordinates of voxel of maximum signiﬁcance; R or L, right or left hemisphere; F, face only; FT, face and
torso; FTG, face, torso and hand gesture.
L. Kesner et al. Brain and Cognition 125 (2018) 88–99
94
speciﬁcally in context of art perception (Kirk & Freedberg, 2015; Kirk
et al., 2009). It is therefore possible that our ﬁnding of DLPFC activity
similarly reﬂects implicit evaluation process, which inhibited the
emotional response to portraits in VMPFC. However, these diﬀerences
in parietal and frontal cortices should be interpreted cautiously because
in direct comparison between direct and averted gaze portraits they did
not survive the conservative FWE correction.
4.1.2. Eﬀect of direct and averted gaze on selected AOI dwell time
The complementary eye-tracking measurement conﬁrmed that
participants spent more time viewing depicted person’s eyes when
viewing the direct gaze portraits than the averted ones. This ﬁnding is
in agreement with our neuroimaging results: the activation of the
abovementioned brain regions connected to social cognition and face
processing increased in the direct gaze condition. Heightened attention
to the direct gaze thus supports the neuroimaging ﬁndings and conﬁrms
that the gaze is a crucial feature to driving face processing and social
engagement. The relationship between increased activation of fusiform
region and higher number of ﬁxations on eyes has been previously
shown in patients with autism (Dalton et al., 2005) and in healthy
subjects (Morris, Pelphrey, & McCarthy, 2006). Our eye-tracking ﬁndings,
indicating correlation between the direct gaze condition and
longer dwell time on the eyes area, may be supported also by the individual
evaluations of stimuli by the participants (debrieﬁng after the
experiment) that showed higher arousal after viewing the direct gaze
portraits in line with previous research (e.g. Akechi et al., 2013;
Myllyneva & Hietanen, 2015a, 2015b).
On the other hand, we did not observe any signiﬁcant diﬀerences
between the dwell times spent in the whole face AOI in direct versus
averted gaze portraits. It contradicts the study by Palanica and Itier
(2012), which found that faces with a direct gaze were viewed longer
than faces with an averted gaze regardless of body context, background,
and task demands. The discrepancy might be caused by the character of
our stimuli: whereas the stimuli from the study of Palanica and Itier
(2012) were proportionally balanced in types (portraits, details and full
ﬁgures) and visual correspondence (varying only in gaze direction), our
stimuli were not balanced, and thus the AOI varied in size. The discrepancy
might be thus caused by the visual variability of our stimuli.
Nevertheless, this is an unresolved dilemma regarding the use of authentic
stimuli such as art works. For example, Dukewich, Klein, and
Christie (2008) used original art works for examining the viewers’
preference for following the direction of the depicted gaze, but the gaze
of depicted people was digitally manipulated. The authors achieved the
perfectly balanced triplets varying only in the gaze direction, while
losing authenticity of paintings. We choose the opposite approach,
while sacriﬁcing the exact visual correspondence.
Interestingly, in the direct gaze condition participants also spent
more time viewing mouth, representing a communication signal and
the most important cue for discrimination of both static and dynamic
facial expressions of the depicted person (Blais, Jack, Scheepers, Fiset,
& Caldara, 2008). This may also suggest that direct gaze stimulates
face-to-face communication, or preparedness for such action, in comparison
to averted gaze. Nevertheless, to our knowledge, no study yet
reported, if the gaze direction aﬀects the ﬁxation of mouth area and in
Fig. 3. The illustration of the selected AOI of the face features (left) and the AOI dwell times compared for direct and everted gaze portraits (right). Left: a portrait
with areas of interest: face (yellow), eyes (red), mouth (green). Right: diﬀerences in dwell time for portraits with averted/direct gaze and for each area of interest.
Error bars represent Fisher’s Least Signiﬁcant Diﬀerence. *
p < .05, **
p < .01, ***
p < .001. (For interpretation of the references to colour in this ﬁgure legend, the
reader is referred to the web version of this article.)
L. Kesner et al. Brain and Cognition 125 (2018) 88–99
95
which way. Another recent evidence to consider is the observation of
individual diﬀerences in the style how mouth-eyes stimuli are viewed
(Coutrot, Binetti, Harrison, Mareschal, & Johnston, 2016; Peterson &
Eckstein, 2013; Rogers, Speelman, Guidetti, & Longmuir, 2018). Some
people showed a strong preference for eye gaze, whereas others a strong
preference for mouth gaze, and others distributed their gaze between
the eyes and mouth to varying extents. This shows that both eyes and
mouth are essential for face processing across individuals, and preference
for either of them depends on personal style.
In this context we should also mention that due to the variable
emotions of portrayed persons in the stimuli set, no correlation was
observed between gaze direction and emotional valence of presented
stimuli. Nevertheless, the expected strong relationship between emotional
valence and arousal produced by the emotions depicted in individual
portraits was identiﬁed in both experimental samples, demonstrating
that the depicted emotion was recognised and was
aﬀecting emotional state of tested participants.
4.2. Face processing in general
Direct gaze versus ﬁxation also yielded activation in the hippocampus
bilaterally, an area which has not been reported in other studies
on gaze processing. The medial temporal lobe and particularly the
hippocampus are traditionally involved predominantly in memory
functions and also in higher-order spatial perception and conjunctions
of features that constitute spatial scenes (for a review see Lee, Yeung, &
Barense, 2012). In contrast, some authors demonstrated the role of the
hippocampus both in perception and recollection of objects and faces
(Fried, MacDonald, & Wilson, 1997; Ishai, Schmidt, & Boesiger, 2005).
Our experimental paradigm did not allow for a strict distinction between
perception and memory functions. Nevertheless, during free
viewing of images these two processes cannot be considered to operate
independently, as during the viewing of art portraits beholders probably
implicitly compare representations of socially salient features
(faces, gestures) with memory representations of such objects. This
might also explain the activation of DLPFC, which is involved in processing
and coordination of these processes.
Contrary to other studies, we did not detect the eﬀect of a direct
gaze on amygdala and this ﬁnding thus corresponds to the results of
authors who similarly did not report amygdala response to a direct gaze
(Pageler et al., 2003; Schilbach et al., 2006). The absence of amygdala
activation can be explained by the small number of portraits with
clearly expressed emotions in our set of stimuli (only 14% according to
post-experiment evaluation, in contrast to 66% of portraits with “subtle
emotional expression” and 20% of them labelled as an “inexpressive
mask”). An additional factor could be the variability in emotional expressions
of the portrayed persons in the stimuli set (including only a
few with explicitly negative facial emotions). Nevertheless, the expected
strong relationship between the evaluated emotional valence
and arousal produced by the emotions depicted in the individual portraits
was identiﬁed in both experimental samples, demonstrating that
the depicted emotion was recognized and aﬀected the emotional state
of the tested participants.
4.3. Comparison to other studies on gaze perception: eﬀect of
representational medium
Interestingly, the pattern of brain activation discrimination of direct
and averted gaze portraits in our study (Fig. 1) is more similar to an
imagery task with famous faces than to the gaze experiments using
naturalistic stimuli. Visual imagery of famous faces activates a network
of regions that includes the calcarine, precuneus, hippocampus, intraparietal
sulcus, and inferior frontal gyrus (Ishai, Haxby, &
Ungerleider, 2002). Besides the fact that only a small fraction of pictures
used in our study was known to volunteers, the similarity of our
results with visual imagery ﬁndings suggests the role of mental
imagination in the perception of artistic portraits. In addition, the same
study identiﬁed that focusing attention on features of the imagined
faces (e.g., eyes, lips, or nose) resulted in increased activation in the
right intraparietal sulcus (adjacent to the angular gyrus) and the right
inferior frontal gyrus (Ishai et al., 2002), which responded speciﬁcally
to the direct gaze portraits.
Overall, our ﬁndings (especially activation in the angular gyrus and
the frontal cortex, absence of amygdale and VMPFC activation) seem to
correspond to the general characteristics of a set of painted portraits
used in this experiment compared to standard stimuli used in previous
gaze experiments. Firstly, there is much greater variability of expression
in our set along both essential axes of valence and intensity of
emotions, with some facial conﬁgurations clearly signaling complex
mental states, rather than prototypical basic emotion. On a phenomenological
level, the painted artistic portraits elicit a wide range of
meanings, to signal communicative intent, social interest, dominance,
hostility, and attraction. In general terms, it can be argued that in most
painted portraits the representational medium (compared to naturalistic
photographs or video clips) signiﬁcantly increased an element of
ambiguity of expression, which is present even in naturalistic stimuli
(Back & Jordan, 2014), thus preventing a response in the amygdalae.
4.4. Eﬀect of painting style, period and pictorial context of the portraits
Our results also conﬁrm the substantial inﬂuence of representational
medium on brain activity. Consistent with our hypothesis, images
categorized as being depicted in a painterly (“less realistic”) style exerted
higher activation of the medial and inferior surfaces of the occipital
lobe (calcarine ﬁssure, cuneus, lingual and fusiform gyri) than a
linear style. The most likely explanation is that a more pronounced
BOLD signal reﬂects more intense processing in the visual cortex required
by those portraits, in which faces and gaze are rendered in a less
realistic manner (or must be extracted from the painterly medium). This
assumption is also supported by the comparison of pre-modern and
modernist portraits in our group showing that the modern paintings (in
which the painterly style predominates) activate the speciﬁc visual
areas for face processing (lingual and fusiform gyrus) more intensively
than the premodern paintings.
The ﬁnding that the activity of the visual cortex was substantially inﬂuenced
by the pictorial context of stimuli was consistent with our expectation.
Perception of the portraits featuring faces only (compared to
those depicting faces with torso and hand gestures) activated the visual
cortex and particularly, the fusiform cortex more intensively (Fig. 2),
which suggests an exclusive attention on faces. The increased activity of
the lingual cortex in portraits with visible hand gestures (in contrast to
face and body only) is in concordance with previous studies that identiﬁed
engagement of the lingual gyrus when participants perceived body-only
(as opposed to face or face and body of actors; Morris et al., 2006) or
during face-blurred gestures (Saggar, Shelly, Lepage, Hoeft, & Reiss,
2014). Therefore, participants were attending to visuospatial aspects of
stimuli, especially motion-related information from gestures.
5. Limitations
There are several limitations in our study. First, we have not attempted
a direct comparison between gaze perception of painted portraits
and more naturalistic (photographic) stimuli. This future direct
comparison will require creating a set of photographic portraits, emulating
the posture, emotional expression and gaze behavior, as well as
overall composition of painted models. We were thus unable to conﬁrm
the speciﬁcity of our ﬁndings for artistic portraits. Second, our study
utilized separate fMRI and eye-tracking measurements that required
enrollment of two diﬀerent participant samples in order to avoid repeated
exposure bias. This design did not allow a direct comparison of
separate data sets of the same subjects which could be obtained in simultaneous
fMRI and eye-tracking experiment. Third, our set of stimuli
L. Kesner et al. Brain and Cognition 125 (2018) 88–99
96
was selected (and balanced) to test the inﬂuence of direction of gaze in
painterly portraits. This approach enabled us also to test the eﬀect of
painting techniques and characteristics on fMRI activation. However,
the design (and number of stimuli) did not allow us to test the interactions
between factor of gaze and factors of painting characteristics
under control of possible technical confounds (artist, luminance, and
RGB color components). Hence, the interaction between gaze and
painting techniques should be addressed by future speciﬁcally designed
study. Fourth, to avoid movement artefacts, we did not register direct
behavioral response during scanning, hence the behavioral data collected
during the post-scanning debrieﬁng would be interpreted cautiously.
It is thus not possible to relate the indices of a subjectively felt
response with neuroimaging data and exploring such a link clearly
suggest itself as a topic for future research. Fifth, because the luminance
and some other characteristics of individual portraits were not fully
counterbalanced for some of the secondary tested conditions (date of
the painting; inﬂuence of style, pictorial context of the portrait), a
control fMRI measurement was performed. We analyzed the possible
eﬀect of luminance on reported brain activation. All portraits used in
the original MRI measurement were sorted according to their average
luminance and divided into two sets of blocks containing portraits with
high or low luminance. Their comparison showed luminance speciﬁc
activity only in the primary visual cortex V1/V2 areas. We therefore
argue that luminance of the paintings is not related to the brain activity
reported as speciﬁc for painting style, pictorial context or time period of
the paintings. Another limitation of the study is the quality of art reproductions
presented during the fMRI experiment. To achieve acceptable
comparability of the obtained results from fMRI and eye-tracking
measurements, both the resolution and ﬁeld of view (screen size)
characteristics have been decreased during the eye-tracking measurement.
Finally, while several studies have recently found an eﬀect of
head orientation on the perception of gaze direction (Laube, Kamphuis,
Dicke, & Thier, 2011; Pageler et al., 2003), we did not analyze the eﬀect
of head orientation on gaze direction, as the directional variability of
faces among our stimuli was too great.
6. Conclusion
Our ﬁndings shed some additional light on the vexing problem of
the nature of interaction with depicted human beings. Art historians
have claimed that the imagination enables viewers to react to a depicted
gaze as if they were confronting the live person (e.g. Belting,
2007) and such views have been reiterated in many literary and anecdotal
accounts of viewers establishing some form of emotional (and
even verbal) rapport with the people depicted in the portraits. Within
social neuroscience, an emerging consensus insists on the fundamental
diﬀerence between the ﬁrst-person and second-person perspective in
social interaction, or between social observation and social interaction
(Przyrembel, Smallwood, Pauen, & Singer, 2012; Schilbach et al., 2013;
Tylén, Allen, Hunter, & Roepstorﬀ, 2012). Our results, demonstrating
the activation of angular gyrus/TPJ, middle and inferior frontal gyri by
direct gaze portraits, suggest that static and in some cases highly stylized
depictions of human beings in artistic portraits elicit brain activation
commensurate with the experience of being observed and potentially
addressed by intelligent sentient being and involve observers
in implicit inferences of the painted subject’s mental states and emotions,
and possibly even a preﬁguration of communicative intent. Future
research may thus focus on disentangling the eﬀects of speciﬁc
viewing conditions (instructions) and priming of subjects on mentalizing
and an empathic response to the subjects depicted in portraits.
Acknowledgement
We would like to thank Ondřej Novák for collecting the eye-tracking
data and Peter Adámek for preparation of AOI areas of individual stimuli
for eye-tracking experiment.
Funding
This work has been supported by the project no. LO1611 with a
ﬁnancial support from the MEYS under the NPU I program, and a grant
from the Czech Science Foundation (GAČR) no.15-08577S ‘Aﬀective
response in visual arts: linking art history and neuroscience perspec-
tives.’
Appendix A. Supplementary material
Supplementary data associated with this article can be found, in the
online version, at https://doi.org/10.1016/j.bandc.2018.06.004.
References
Akechi, H., Senju, A., Uibo, H., Kikuchi, Y., Hasegawa, T., & Hietanen, J. K. (2013).
Attention to eye contact in the West and East: Autonomic responses and evaluative
ratings. PLoS One, 8, e59312. http://dx.doi.org/10.1371/journal.pone.0059312.
Argyle, M., & Cook, M. (1976). Gaze and mutual gaze. Cambridge: Cambridge University
Press.
Back, E., & Jordan, T. R. (2014). Revealing variations in perception of mental states from
dynamic facial expressions: A cautionary note. PLoS ONE, 9(1), e84395. http://dx.
doi.org/10.1371/journal.pone.0084395.
Baetens, K., Ma, N., Steen, J., & Van Overwalle, F. (2014). Involvement of the mentalizing
network in social and non-social high construal. Social Cognitive and Aﬀective
Neuroscience, 9(6), 817–824. http://dx.doi.org/10.1093/scan/nst048.
Baron-Cohen, S. (1995). Mindblindness: An essay on autism and theory of mind. Cambridge,
MA: MIT Press.
Belting, H. (2007). Blickwechsel mit Bildern. Die Bilderfrage als Körperfrage. In H. Belting
(Ed.). Bilderfragen: Die Bildwissenschaften im Aufbruch. München: Wilhelm Fink.
Belting, H. (2009). The gaze in the image. A contribution to an iconology of the gaze. In B.
Huppauf, & C. Wulf (Eds.). Dynamics and performativity of imagination. London:
Routledge.
Berchio, C., Rihs, T., Piguet, C., Dayer, A., Aubry, J.-M., & Michel, C. (2016). Early
averted gaze processing in the right Fusiform Gyrus: An EEG source imaging study.
Biological Psychology, 119, 156–170. http://dx.doi.org/10.1016/j.biopsycho.2016.06.
008.
Bernstein, M., Erez, Y., Blank, I., & Yovel, G. (2018). An integrated neural framework for
dynamic and static face processing. Scientiﬁc Reports, 8(1), 7036. http://dx.doi.org/
10.1038/s41598-018-25405-9.
Blais, C., Jack, R. E., Scheepers, C., Fiset, D., & Caldara, R. (2008). Culture shapes how we
look at faces. PLoS ONE, 3, e3022. http://dx.doi.org/10.1371/journal.pone.0003022.
Boyarskaya, E., Sebastian, A., Bauermann, T., Hecht, H., & Tuscher, O. (2015). The Mona
Lisa eﬀect: Neural correlates of centered and oﬀ-centered gaze. Human Brain Mapping,
36(2), 619–632. http://dx.doi.org/10.1002/hbm.22651.
Bryson, N. (1983). Vision and painting. The logic of the Gaze. New Haven: Yale University
Press.
Bzdok, D., Langner, R., Hoﬀstaedter, F., Turetsky, B. I., Zilles, K., & Eickhoﬀ, S. B. (2012).
The modular neuroarchitecture of social judgments on faces. Cerebral Cortex, 22(4),
951–961. http://dx.doi.org/10.1093/cercor/bhr166.
Calder, A. J., & Young, A. W. (2005). Understanding the recognition of facial identity and
facial expression. Nature Reviews Neuroscience, 6, 641–651. http://dx.doi.org/10.
1038/nrn1724.
Carter, R. M., & Huettel, S. A. (2013). A nexus model of the temporo-parietal junction.
Trends in Cognitive Sciences, 17(7), 328–336. http://dx.doi.org/10.1016/j.tics.2013.
05.007.
Cavallo, A., Lungu, O., Becchio, C., Ansuini, C., Rustichini, A., & Fadiga, L. (2015). When
gaze opens the channel for communication: Integrative role of IFG and MPFC.
Neuroimage, 119, 63–69. http://dx.doi.org/10.1016/j.neuroimage.2015.06.025.
Chan, A. W.-Y., & Downing, P. E. (2011). Faces and eyes in human lateral prefrontal
cortex. Frontiers in Human Neuroscience, 5, 51. http://dx.doi.org/10.3389/fnhum.
2011.00051.
Corbetta, M., & Shulman, G. L. (2002). Control of goal-directed and stimulus-driven attention
in the brain. Nature Reviews Neuroscience, 3, 201–215. http://dx.doi.org/10.
1038/nrn755.
Coutrot, A., Binetti, N., Harrison, C., Mareschal, I., & Johnston, A. (2016). Face exploration
dynamics diﬀerentiate men and women. Journal of Vision, 16, 1–19. http://
dx.doi.org/10.3389/fnint.2010.00005.
Dalton, K. M., Nacewicz, B. M., Johnstone, T., Schaefer, H. S., Gernsbacher, M. A.,
Goldsmith, H. H., ... Davidson, R. J., et al. (2005). Gaze ﬁxation and the neural circuitry
of face processing in autism. Nature Neuroscience, 8(4), 519–526. http://dx.doi.
org/10.1038/nn1421.
Debruille, J. B., Brodeur, M. B., & Franco Porras, C. (2012). N300 and social aﬀordances:
A study with a real person and a dummy as stimuli. PLoS ONE, 7(10), e47922. http://
dx.doi.org/10.1371/journal.pone.0047922.
Delgado, M. R., Beer, J. S., Fellows, L. K., Huettel, S. A., Platt, M. L., Quirk, G. J., &
Schiller, D. (2016). Viewpoints: Dialogues on the functional role of the ventromedial
prefrontal cortex. Nature Neuroscience, 19, 1545–1552. http://dx.doi.org/10.1038/
nn.4438.
Duchaine, B., & Yovel, G. (2015). A revised neural framework for face processing. Annual
Review of Vision Science, 1, 393–416. http://dx.doi.org/10.1146/annurev-visionL.
Kesner et al. Brain and Cognition 125 (2018) 88–99
97
082114-035518.
Dukewich, K. R., Klein, R. M., & Christie, J. (2008). The eﬀect of gaze on gaze direction
while looking at art. Psychonomic Bulletin & Review, 15, 1141–1147. http://dx.doi.
org/10.3758/PBR.15.6.1141.
Flagan, T., & Beer, J. S. (2013). Three ways in which midline regions contribute to selfevaluation.
Frontiers in Human Neuroscience, 7, 450. http://dx.doi.org/10.3389/
fnhum.2013.00450.
Fried, I., MacDonald, K. A., & Wilson, C. L. (1997). Single neuron activity in human
hippocampus and amygdala during recognition of faces and objects. Neuron, 18(5),
753–765. http://dx.doi.org/10.1016/S0896-6273(00)80315-3.
Friesen, C. K., & Kingstone, A. (1998). The eyes have it! Reﬂexive orienting is triggered by
nonpredictive gaze. Psychonomic Bulletin & Review, 5(3), 490–495. http://dx.doi.org/
10.3758/BF03208827.
Frischen, A., Bayliss, A. P., & Tipper, S. P. (2007). Gaze cueing of attention: Visual attention,
social cognition, and individual diﬀerences. Psychological Bulletin, 133(4),
694–724. http://dx.doi.org/10.1037/0033-2909.133.4.694.
Ganel, T., Valyear, K. F., Goshen-Gottstein, Y., & Goodale, M. A. (2005). The involvement
of the ‘fusiform face area’ in processing facial expression. Neuropsychologia, 43(11),
1645–1654. http://dx.doi.org/10.1016/j.neuropsychologia.2005.01.012.
George, N., & Conty, L. (2008). Facing the gaze others. Clinical Neurophysiology, 38,
197–207. http://dx.doi.org/10.1016/j.neucli.2008.03.001.
George, N., Driver, J., & Dolan, R. J. (2001). Seen gaze-direction modulates fusiform
activity and its coupling with other brain areas during face processing. NeuroImage,
13(6 Pt 1), 1102–1112. http://dx.doi.org/10.1006/nimg.2001.0769.
Gibson, J., & Pick, A. (1963). Perception of another person’s looking behavior. American
Journal of Psychology, 76, 386–394. http://dx.doi.org/10.2307/1419779.
Hamilton, A. C. (2016). Gazing at me: The importance of social meaning in understanding
direct gaze cues. Philosophical Transactions of the Royal Society of London. Series B,
Biological sciences, 371, 20150080. http://dx.doi.org/10.1098/rstb.2015.0080.
Hare, T. A., Camerer, C. F., Rangel, A., & Hare, C. (2009). Self-control in decision-making
involves modulation of the vmPFC valuation system. Science, 324, 646–648. http://
dx.doi.org/10.1126/science.1168450.
Harry, B., Williams, M. A., Davis, C., & Kim, J. (2013). Emotional expressions evoke a
diﬀerential response in the fusiform face area. Frontiers in Human Neuroscience, 7,
692. http://dx.doi.org/10.3389/fnhum.2013.00692.
Haxby, J. V., Hoﬀman, E. A., & Gobbini, M. I. (2000). The distributed human neural
system for face perception. Trends in Cognitive Sciences, 4(6), 223–233. http://dx.doi.
org/10.1016/S1364-6613(00)01482-0.
Horáček, J., Mikoláš, P., Tintěra, J., Novák, T., Páleníček, T., Brunovský, M., ... Alda, M.
(2015). Sad mood induction has an opposite eﬀect on amygdala response to emotional
stimuli in euthymic patients with bipolar disorder and healthy controls. Journal
of Psychiatry and Neuroscience, 40(2), 134–142. http://dx.doi.org/10.1503/jpn.14.
Igelström, K. M., Webb, T. W., & Graziano, M. S. (2015). Neural processes in the human
temporoparietal cortex separated by localized independent component analysis.
Journal of Neuroscience, 35, 9432–9445. http://dx.doi.org/10.1523/JNEUROSCI.
0551-15.2015.
Ishai, A., Haxby, J. V., & Ungerleider, L. G. (2002). Visual imagery of famous faces: Eﬀects
of memory and attention revealed by fMRI. NeuroImage, 17, 1729–2174. http://dx.
doi.org/10.1006/nimg.2002.1330.
Ishai, A., Schmidt, C. F., & Boesiger, P. (2005). Face perception is mediated by a distributed
cortical network. Brain Research Bulletin, 67(1–2), 87–93. http://dx.doi.org/
10.1016/j.brainresbull.2005.05.027.
Kampe, K. K. W., Frith, C. D., & Frith, U. (2003). ‘Hey John’: Signals conveying communicative
intention toward the self activate brain regions associated with ‘mentalizing’,
regardless of modality. Journal of Neuroscience, 23, 5258–5263. http://dx.doi.
org/10.1006/nimg.2002.1330.
Kawasaki, H., Tsuchiya, N., Kovach, C. K., Nourski, K. V., Oya, H., Howard, M. A., &
Adolphs, R. (2012). Processing of facial emotion in the human fusiform gyrus. Journal
of Cognitive Neuroscience, 24, 1358–1370. http://dx.doi.org/10.1162/jocn_a_00175.
Kesner, L. (2011). Selbst: Geist und Identität. In L. Kesner, & C. Schmitz (Eds.). Images of
the Mind. Bildwelten des Geistes aus Kunst und Wissenschaft. Wallstein: Göttingen.
Kirk, U., Downar, J., & Montague, P. R. (2011). Interoception drives increased rational
decision-making in meditators playing the ultimatum game. Frontiers in Neuroscience,
5, 49. http://dx.doi.org/10.3389/fnins.2011.00049.
Kirk, U., & Freedberg, D. (2015). Contextual bias and insulation against bias during
aesthetic rating. The roles of VMPFC and DLPFC in neural valuation. In J. P. Huston,
M. Nadal, F. Mora, L. Agnati, & C. J. Cela Conde (Eds.). Art, Aesthetics and the brain
(pp. 158–173). Oxford: Oxford University Press.
Kirk, U., Skov, M., Hulme, O., Christensen, M. S., & Zeki, S. (2009). Modulation of aesthetic
value by semantic context: An fMRI study. Neuroimage, 44(3), 1125–1132.
http://dx.doi.org/10.1016/j.neuroimage.2008.10.009.
Kozlovskiy, S. A., Pyasik, M. M., Korotkova, A. V., Vartanov, A. V., Glozman, J. M., &
Kiselnikov, A. A. (2014). Activation of left lingual gyrus related to working memory
for schematic faces. International Journal of Psychophysiology, 94(2), 241. http://dx.
doi.org/10.1016/j.ijpsycho.2014.08.928.
Krall, S. C., Rottschy, C., Oberwelland, E., Bzdok, D., Fox, P. T., Eickhoﬀ, S. B., ... Konrad,
K. (2015). The role of the right temporoparietal junction in attention and social interaction
as revealed by ALE meta-analysis. Brain Structure and Function, 220(2),
587–604. http://dx.doi.org/10.1007/s00429-014-0803-z.
Kuzmanovic, B., Georgescu, A. L., Eickhoﬀ, S. B., Shah, N. J., Bente, G., Fink, G. R., &
Vogeley, K. (2009). Duration matters: Dissociating neural correlates of detection and
evaluation of social gaze. Neuroimage, 46(4), 1154–1163. http://dx.doi.org/10.1016/
j.neuroimage.2009.03.037.
Laube, I., Kamphuis, S., Dicke, P. W., & Thier, P. (2011). Cortical processing of head- and
eye-gaze cues guiding joint social attention. Neuroimage, 54(2), 1643–1653. http://
dx.doi.org/10.1016/j.neuroimage.2010.08.074.
Lancaster, J. L., Tordesillas-Gutiérrez, D., Martinez, M., Salinas, F., Evans, A., Zilles, K.,
Mazziotta, J. C., & Fox, P. T. (2007). Bias between MNI and Talairach coordinates
analyzed using the ICBM-152 brain template. Human Brain Mapping, 28(11),
1194–1205. http://dx.doi.org/10.1002/hbm.20345.
Lee, A. C. H., Yeung, L. K., & Barense, M. D. (2012). The hippocampus and visual perception.
Frontiers in Human Neuroscience, 6, 91. http://dx.doi.org/10.3389/fnhum.
2012.00091.
Madipakkam, A. R., Rothkirch, M., Guggenmos, M., Heinz, A., & Sterzer, P. (2015). Gaze
direction modulates the relation between neural responses to faces and visual
awareness. Journal of Neuroscience, 35, 13287–13299. http://dx.doi.org/10.1523/
jneurosci.0815-15.2015.
Mareschal, I., Calder, A. J., & Cliﬀord, C. W. (2013). Humans have an expectation that
gaze is directed toward them. Current Biology, 22.23(8), 717–721. http://dx.doi.org/
10.1016/j.cub.2013.03.030.
Mason, M. F., Tatkow, E. P., & Macrae, C. N. (2005). The look of love: Gaze shifts and
person perception. Psychological Science, 16(3), 236–239. http://dx.doi.org/10.1111/
j.0956-7976.2005.00809.
McCarthy, G., Puce, A., Belger, A., & Allison, T. (1999). Electrophysiological studies of
human face perception. II: Response properties of face-speciﬁc potentials generated
in occipitotemporal cortex. Cerebral Cortex, 9(5), 431–444. http://dx.doi.org/10.
1093/cercor/9.5.431.
Morris, J. P., Pelphrey, K. A., & McCarthy, G. (2006). Occipitotemporal activation evoked
by the perception of human bodies is modulated by the presence or absence of the
face. Neuropsychologia, 44(10), 1919–1927. http://dx.doi.org/10.1016/j.
neuropsychologia.2006.01.035.
Morris, J. P., Pelphrey, K. A., & McCarthy, G. (2007). Controlled scanpath variation alters
fusiform face activation. Social Cognitive and Aﬀective Neuroscience, 2(1), 31–38.
Myllyneva, A., & Hietanen, J. K. (2015). The dual nature of eye contact: To see and to be
seen. Social Cognitive and Aﬀective Neuroscience pii: nsv075. http://doi.org/10.1093/
scan/nsv075.
Myllyneva, A., & Hietanen, J. K. (2015b). There is more to eye contact than meets the eye.
Cognition, 134, 100–109. http://dx.doi.org/10.1016/j.cognition.2014.09.011.
Nodelman, S. (1975). How to read a roman portrait. Art in America, 63, 27–33.
Nummenmaa, L., & Calder, A. J. (2009). Neural mechanisms of social attention. Trends in
Cognitive Sciences, 13(3), 135–143. http://dx.doi.org/10.1016/j.tics.2008.12.006.
Ohnishi, T., Moriguchi, Y., Matsuda, H., Mori, T., Hirakata, M., Imabayashi, E., ... Uno, A.
(2004). The neural network for the mirror system and mentalizing in normally developed
children: An fMRI study. Neuroreport, 15, 1483–1487. http://dx.doi.org/10.
1097/01.wnr.0000127464.17770.1f.
Pageler, N. M., Menon, V., Merin, N. M., Eliez, S., Brown, W. E., & Reiss, A. L. (2003).
Eﬀect of head orientation on gaze processing in fusiform gyrus and superior temporal
sulcus. NeuroImage, 20, 318–329. http://dx.doi.org/10.1016/S1053-8119(03)
00229-5.
Palanica, A., & Itier, R. J. (2012). Attention capture by direct gaze is robust to context and
task demands. Journal of Nonverbal Behavior, 36(2), 123–134. http://dx.doi.org/10.
1007/s10919-011-0128-z.
Pelphrey, K. A., Morris, J. P., & McCarthy, G. (2004). Grasping the intentions of others:
The perceived intentionality of an action inﬂuences activity in the superior temporal
sulcus during social perception. Journal of Cognitive Neuroscience, 16(10), 1706–1716.
http://dx.doi.org/10.1162/0898929042947900.
Pessoa, L., McKenna, M., Gutierrez, E., & Ungerleider, L. G. (2002). Neural processing of
emotional faces requires attention. Proceedings of the National Academy of Sciences
USA, 99(17), 11458–11463. http://dx.doi.org/10.1073/pnas.172403899.
Peterson, M. F., & Eckstein, M. P. (2013). Individual diﬀerences in eye movements during
face identiﬁcation reﬂect observer-speciﬁc optimal points of ﬁxation. Psychological
Science, 24, 1216–1225. http://dx.doi.org/10.1177/0956797612471684.
Pfeiﬀer, U. J., Vogeley, K., & Schilbach, L. (2013). From gaze cueing to dual eye-tracking:
Novel approaches to investigate the neural correlates of gaze in social interaction.
Neuroscience & Biobehavioral Reviews, 37(10 Pt 2), 2516–2528. http://dx.doi.org/10.
1016/j.neubiorev.2013.07.017.
Pönkänen, L. M., Peltola, M. J., Hietanen, J. K. (2011). The observer observed: Frontal
EEG asymmetry and autonomic responses diﬀerentiate between another person's
direct and averted gaze when the face is seen live. International Journal of
Psychophysiology 82(2), 180i7. http://doi.org/10.1016/j.ijpsycho.2011.08.006.
Przyrembel, M., Smallwood, J., Pauen, M., & Singer, T. (2012). Illuminating the dark
matter of social neuroscience: Considering the problem of social interaction from
philosophical, psychological, and neuroscientiﬁc perspectives. Frontiers in Human
Neuroscience, 6, 190. http://dx.doi.org/10.3389/fnhum.2012.00190.
Puce, A., Smith, A., & Allison, T. (2000). ERPs evoked by viewing facial movements.
Cognitive Neuropsychology, 17(1), 221–239. http://dx.doi.org/10.1080/
026432900380580.
Risko, E. F., Laidlaw, K., Freeth, M., Foulsham, T., & Kingstone, A. (2012). Social attention
with real versus reel stimuli: Toward an empirical approach to concerns about
ecological validity. Frontiers in Human Neuroscience, 6, 143. http://dx.doi.org/10.
3389/fnhum.2012.00143.
Rogers, S. L., Speelman, C. P., Guidetti, O., & Longmuir, M. (2018). Using dual eye
tracking to uncover personal gaze patterns during social interaction. Scientiﬁc Reports,
8. http://dx.doi.org/10.1038/s41598-018-22726-7.
Rossi, A., Parada, F. J., Kolchinsky, A., & Puce, A. (2014). Neural correlates of apparent
motion perception of impoverished facial stimuli: A comparison of ERP and ERSP
activity. NeuroImage, 98, 442–459. http://dx.doi.org/10.1016/j.neuroimage.2014.
04.029.
Rossi, A., Parada, F. J., Latinus, M., & Puce, A. (2015). Photographic but not line-drawn
faces show early perceptual neural sensitivity to eye gaze direction. Frontiers in
Human Neuroscience, 9, 185. http://dx.doi.org/10.3389/fnhum.2015.00185.
Saggar, M., Shelly, E. W., Lepage, J.-F., Hoeft, F., & Reiss, A. L. (2014). Revealing the
L. Kesner et al. Brain and Cognition 125 (2018) 88–99
98
neural networks associated with processing of natural social interaction and the related
eﬀects of actor-orientation and face-visibility. NeuroImage, 84, 648–656. http://
dx.doi.org/10.1016/j.neuroimage.2013.09.046.
Saxe, R., & Kanwisher, N. (2003). People thinking about thinking people: The role of the
temporo-parietal junction in “theory of mind”. NeuroImage, 19(4), 1835–1842.
http://dx.doi.org/10.1016/S1053-8119(03)00230-1.
Schilbach, L., Timmermans, B., Reddy, V., Costall, A., Bente, G., Schlicht, T., & Vogeley,
K. (2013). Toward a second-person neuroscience. Behavioral and Brain Sciences,
36(4), 393–414. http://dx.doi.org/10.1017/S0140525X12000660.
Schilbach, L., Wohlschlaeger, A. M., Kraemer, N. C., Newen, A., Shah, N. J., Fink, G. R., &
Vogeley, K. (2006). Being with virtual others: Neural correlates of social interaction.
Neuropsychologia, 44(5), 718–730. http://dx.doi.org/10.1016/j.neuropsychologia.
2005.07.017.
Schultz, R. T., Grelotti, D. J., Klin, A., Kleinman, J., Van der Gaag, C., Marois, R., &
Skudlarski, P. (2003). The role of the fusiform face area in social cognition:
Implications for the pathobiology of autism. Philosophical Transactions of the Royal
Society of London. Series B, Biological sciences, 358(1430), 415–427. http://dx.doi.org/
10.1098/rstb.2002.1208.
Schurz, M., & Tholen, M. G. (2016). What brain imaging did (not) tell us about the
Inferior Frontal Gyrus in theory of mind: A commentary on Samson et al. (2015).
Cortex, 74, 329–333. http://dx.doi.org/10.1016/j.cortex.2015.08.011.
Schurz, M., Radua, J., Aichhorn, M., Richlan, F., & Perner, J. (2014). Fractionating theory
of mind: A meta-analysis of functional brain imaging studies. Neuroscience &
Biobehavioral Reviews, 42, 9–34. http://dx.doi.org/10.1016/j.neubiorev.2014.01.
009.
Seghier, M. L. (2013). The angular gyrus: Multiple functions and multiple subdivisions.
Neuroscientist, 19(1), 43–61. http://dx.doi.org/10.1177/1073858412440596.
Senju, A., & Johnson, M. H. (2009). The eye contact eﬀect: Mechanisms and development.
Trends in Cognitive Sciences, 13(3), 127–134. http://dx.doi.org/10.1016/j.tics.2008.
11.009.
Srinivasan, N., Srivastava, P., Lohani, M., & Baijal, S. (2009). Focused and distributed
attention. Progress in Brain Research, 176, 87–100. http://dx.doi.org/10.1016/S0079-
6123(09)17606-9.
Stawarska, B. (2006). Mutual gaze and social cognition. Phenomenology and Cognitive
Sciences, 5, 17–30. http://dx.doi.org/10.1007/s11097-005-9009-4.
Stolier, R., & Freeman, J. (2016). The neuroscience of social vision. In J. Absher, & J.
Cloutier (Eds.). Neuroimaging personality, social cognition, and characterr (pp. 140–
157). Elsevier, Amsterdam: Academic Press http://doi.org/10.1016/B978-0-12-
800935-2.00007-5.
Straube, T. (2010). Increased amygdale activation to averted versus direct gaze in humans
is independent of valence of facial expression. NeuroImage, 49, 2680–2686. http://dx.
doi.org/10.1016/j.neuroimage.2009.10.074.
Tso, I. F., Rutherford, S., Fang, Y., Angstadt, M., & Taylor, S. F. (2018). The ‘social brain’
is highly sensitive to the mere presence of social information: An automated metaanalysis
and an independent study. PLoS ONE, 13(5), e0196503. http://dx.doi.org/
10.1371/journal.pone.0196503.
Tylén, K., Allen, M., Hunter, B. K., & Roepstorﬀ, A. (2012). Interaction vs. observation:
Distinctive modes of social cognition in human brain and behavior? A combined fMRI
and eye-tracking study. Frontiers in Human Neuroscience, 6, 331. http://dx.doi.org/10.
3389/fnhum.2012.00331.
Vartanian, O., & Skov, M. (2014). Neural correlates of viewing paintings: Evidence from a
quantitative meta-analysis of functional magnetic resonance imaging data. Brain and
Cognition, 87, 52–56. http://dx.doi.org/10.1016/j.bandc.2014.03.004.
Vuilleumier, P., Armony, J. L., Driver, J., & Dolan, R. J. (2001). Eﬀects of attention and
emotion on face processing in the human brain: An event-related fMRI study. Neuron,
30(3), 829–841. http://dx.doi.org/10.1016/S0896-6273(01)00328-2.
Wegrzyn, M., Vogt, M., Kireclioglu, B., Schneider, J., & Kissler, J. (2017). Mapping the
emotional face. How individual face parts contribute to successful emotion recognition.
PLoS ONE, 12(5), e0177239. http://dx.doi.org/10.1371/journal.pone.0177239.
Wilms, M., Schilbach, L., Pfeiﬀer, U., Bente, G., Fink, G. R., & Vogeley, K. (2010). It's in
your eyes: Using gaze-contingent stimuli to create truly interactive paradigms for
social cognitive and aﬀective neuroscience. Social Cognitive and Aﬀective Neuroscience,
5(1), 98–107. http://dx.doi.org/10.1093/scan/nsq024.
Wojciulik, E., Kanwisher, N., & Driver, J. (1998). Covert visual attention modulates facespeciﬁc
activity in the human fusiform gyrus: fMRI study. Journal of Neurophysiology,
79, 1574–1578. http://dx.doi.org/10.1152/jn.1998.79.3.1574.
Wölﬄin, H. (1950). Principles of art history: The problem of the development of style in later
art. New York: Dover.
Wollheim, R. (1987). Painting as an Art. Andrew M. Mellon Lectures in Fine Arts, National
Gallery of Art. Washington, D.C., Cambridge, Mass: Harvard University Press.
L. Kesner et al. Brain and Cognition 125 (2018) 88–99
99