Fixation-related brain potentials during semantic integration of object-scene information (walk-through the main results)

1 Objectives

In this tutorial, we present the post-processed eye-tracking and EEG data of Coco, Nuthmann & Dimigen (2019) and walk the readers through the descriptive statistics of our eye-tracking data, as well as, the second-level of analysis of our result section.

This tutorial is divided into three main sections. In the first one, we look at the descriptive statistics about the eye-movement behavior and the bounding box surrounding the target object. In the second one, we will be looking at the measures extracted from the eye-tracking data, and in the third one instead, at the EEG data both ERP-stimulus locked and fixation-related potentials of interest. Of the EEG data, we will not present the Threshold-Free cluster enhancement results as they were computed using Matlab.

Before starting though, we need to load the libraries and the data that we are going to use.

library(lme4)     ## for linear-mixed effect modeling
library(lmerTest) ## to do model selection and calculate the p-values 
library(gplots)   ## for plotting
library(reshape)  ## to reshape the data
library(ez)       ## to run classic ANOVA analyses

load("C:/Users/mcoco/Dropbox/Writing/CND/R2/OSF_files/data.RData")

2 Eye-tracking data

In Table 1 of the manuscript, we report some descriptive results (mean and standard deviation) for the eye-movement behavior in the task and properties of the target object.

We will use four different datasets for this section:

• fixation_features: storing characteristics of fixation relative to their type and the target object.
  • fix_dur : fixation duration in ms
  • fixix : ordinal number of first fixation
  • in_sacamp : incoming saccade amplitude in degrees of visual angle
  • target : the type of fixation
    • pst : t + 1
    • pt : t - 1
    • pt2 : t - 2
    • t : t
  • consistency : the semantic consistency of the target object
  • d : the distance from the closest edge of the target object
• nr_refix : re-fixation behavior on the target object
  • nrfix : number of refixations
  • duration : fixation duration in ms
  • consistency : the semantic consistency of the target object
• after_contingency: eye-movement responses after first encountering the target object until the end of the study phase
  • nrfix: number of fixations
  • duration: fixation duration in ms
• boundingbox: low-level properties of the bounding box surrounding the target object
  • scene: the ID of the photograph
  • distcent_vis: distance of the target object center from screen center in degrees of visual angle
  • saliency: mean visual saliency as computed using the AWS model
  • area_vis: area size in degrees of visual angle squared
  • width_vis: the width of the bounding box in degrees of visual angle
  • height_vis: the height of the bounding box in degrees of visual angle
  • consistency: the consistency of the target object

First, let’s look at the very first fixation on the critical target

# extract only the descriptive stats of the fixation on the target object
firstfix = subset(fixation_features, fixation_features$target == "t")

# ordinal number of first fixation on the target
with(firstfix, tapply(fixix, list(consistency), mean))

##        c        i 
## 6.725064 5.176606

with(firstfix, tapply(fixix, list(consistency), sd))

##        c        i 
## 6.024570 5.343164

Now, we look at the duration of each fixation type, the incoming saccade amplitude that brought that fixation, and the distance of such fixations from the bounding box. We also look at the re-fixation behavior on the target.

# duration of the target fixations
with(fixation_features, tapply(fix_dur, list(target, consistency), mean))

##            c        i
## pst 245.3317 243.7477
## pt  207.6553 197.0505
## pt2 220.7263 212.9127
## t   261.6413 263.3079

with(fixation_features, tapply(fix_dur, list(target, consistency), sd))

##             c         i
## pst 148.92642 146.67208
## pt   96.02076  91.83283
## pt2 105.07910  95.06892
## t   146.78624 136.29250

## incoming saccade amplitudes
with(fixation_features, tapply(in_sacamp, list(target, consistency), mean, na.rm = T))

##            c         i
## pst 9.506156 10.163270
## pt  6.149894  5.990658
## pt2 5.880484  6.103110
## t   8.462598  8.277136

with(fixation_features, tapply(in_sacamp, list(target, consistency), sd, na.rm = T))

##            c        i
## pst 5.866331 5.813715
## pt  5.196002 4.850780
## pt2 4.804712 4.717461
## t   5.165782 4.886340

## distances of fixations from bounding box closest edge
with(fixation_features, tapply(d, list(target, consistency), mean))

##             c         i
## pst  7.860970  8.142223
## pt   6.771015  6.302465
## pt2 10.299516  9.386965
## t   -1.180603 -1.441000

with(fixation_features, tapply(d, list(target, consistency), sd))

##            c         i
## pst 6.171721 6.0437775
## pt  5.757710 5.3272929
## pt2 6.324626 6.3745749
## t   2.143717 0.8807079

## refixation behavior
with(nr_refix, tapply(nrfix, list(consistency), mean, na.rm = T))

##        c        i 
## 1.762615 2.211222

with(nr_refix, tapply(nrfix, list(consistency), sd, na.rm = T))

##        c        i 
## 2.029261 2.059923

with(nr_refix, tapply(duration, list(consistency), mean, na.rm = T))

##        c        i 
## 238.9412 250.1554

with(nr_refix, tapply(duration, list(consistency), sd, na.rm = T))

##        c        i 
## 121.8081 135.6975

Let see what happens from when the participants made a saccade out of the target object, and triggered the gaze contingency mechanisms, until the study phase ends.

## Number of fixations after first encountering target object until the end of the study phase
with(after_contingency, tapply(nrfix, list(consistency), mean, na.rm = T))

##        c        i 
## 7.341731 7.255072

with(after_contingency, tapply(nrfix, list(consistency), sd, na.rm = T))

##        c        i 
## 2.144162 1.742696

## after contingency fixation duration
with(after_contingency, tapply(duration, list(consistency), mean, na.rm = T))

##        c        i 
## 254.6405 251.6742

with(after_contingency, tapply(duration, list(consistency), sd, na.rm = T))

##        c        i 
## 120.4519 118.8510

Finally, let’s examine the properties of the bounding box that surrounded the target object.

## distance of bounding box from the center of the screen 
with(boundingbox, tapply(distcent_vis, list(consistency), mean))

##        c        i 
## 12.13071 11.67111

with(boundingbox, tapply(distcent_vis, list(consistency), sd))

##        c        i 
## 2.769445 2.979903

## mean saliency of bounding box (AWS model)
with(boundingbox, tapply(saliency, list(consistency), mean))

##         c         i 
## 0.3595237 0.3718273

with(boundingbox, tapply(saliency, list(consistency), sd))

##         c         i 
## 0.1679568 0.1603509

## width of the bounding box
with(boundingbox, tapply(width_vis, list(consistency), mean))

##        c        i 
## 6.083125 6.136018

with(boundingbox, tapply(width_vis, list(consistency), sd))

##        c        i 
## 1.961227 2.139699

## height of the bounding box
with(boundingbox, tapply(height_vis, list(consistency), mean))

##        c        i 
## 5.137755 5.386191

with(boundingbox, tapply(height_vis, list(consistency), sd))

##        c        i 
## 1.790017 2.166007

## the area size
with(boundingbox, tapply(area_vis, list(consistency), mean))

##        c        i 
## 16.15426 17.33353

with(boundingbox, tapply(area_vis, list(consistency), sd))

##         c         i 
##  8.744652 11.366291

After having investigated the descriptive patters of eye-movement behavior in the task, and some of the target object properties, let’s have a look at the dependent measures of the eye-tracking data (Table 2 and Table 3) that we have further analysed:

## load the data
str(ET)

## 'data.frame':    3318 obs. of  8 variables:
##  $ subj           : int  3 3 3 3 3 3 3 3 3 3 ...
##  $ trial          : int  5 6 7 9 12 13 14 15 16 17 ...
##  $ scene          : Factor w/ 192 levels "bathroom_1","bathroom_10",..: 135 152 126 73 84 127 119 41 103 8 ...
##  $ consistency    : Factor w/ 2 levels "c","i": 1 1 2 1 1 2 2 2 2 1 ...
##  $ FIXNR_FIRSTFIX : int  8 2 1 17 4 2 7 6 30 2 ...
##  $ FIRSTFIX_ON_OBJ: int  0 0 1 0 0 0 0 0 0 0 ...
##  $ TIME_FIRSTFIX  : int  1851 646 388 5183 1138 560 1880 1624 7516 389 ...
##  $ GD             : int  528 313 337 143 640 307 616 653 261 647 ...

The eye-tracking data consists of 3,318 unique trials, 1,567 for the consistent condition and 1,751 for the inconsistent condition. Per participant, this corresponded to an average of 65.3 trials (SD = 6.9, range = 48-78) for consistent and 73.0 trials (SD = 6.9, range = 59-82) for inconsistent items.

inter = with(ET, table(subj, consistency)) ## take frequencies
apply(inter, 2, mean)

##        c        i 
## 65.29167 72.95833

apply(inter, 2, sd)

##        c        i 
## 6.893470 6.855522

apply(inter, 2, range)

##       consistency
##         c  i
##   [1,] 48 59
##   [2,] 78 82

head(ET)

##   subj trial      scene consistency FIXNR_FIRSTFIX FIRSTFIX_ON_OBJ
## 1    3     5   filler_9           c              8               0
## 2    3     6 kitchen_19           c              2               0
## 3    3     7  filler_81           i              1               1
## 4    3     9  filler_33           c             17               0
## 5    3    12  filler_43           c              4               0
## 6    3    13  filler_82           i              2               0
##   TIME_FIRSTFIX  GD
## 1          1851 528
## 2           646 313
## 3           388 337
## 4          5183 143
## 5          1138 640
## 6           560 307

Along the columns of the dataframe, we have organized the following variables:

• subj: the subj ID
• trial: the trial number
• scene: the identifier for the scene that was looked at during the study phase
• consistency: the consistency of the target object (c = consistent, i = inconsistent)
• FIXNR_FIRSTIFX: the ordinal nr. of first fixation on target
• FIRSTFIX_ON_OBJ: a binomial indicating whether the target object was fixated immediately after the scene onset
• TIME_FIRSTFIX: the latency to first fixation on the target object
• GD: the gaze duration on the target object

We will also create a centred version of the consistency variable and code it to range, approximately, between -.5 and .5 that will be used as predictor in the linear-mixed effect models.

ET$consistencyC = as.numeric(as.factor(ET$consistency))
ET$consistencyC = ET$consistencyC - mean(ET$consistencyC, na.rm = T) 
cbind(as.character(ET$consistency[1:10]), ET$consistencyC[1:10])

##       [,1] [,2]                
##  [1,] "c"  "-0.527727546714889"
##  [2,] "c"  "-0.527727546714889"
##  [3,] "i"  "0.472272453285111" 
##  [4,] "c"  "-0.527727546714889"
##  [5,] "c"  "-0.527727546714889"
##  [6,] "i"  "0.472272453285111" 
##  [7,] "i"  "0.472272453285111" 
##  [8,] "i"  "0.472272453285111" 
##  [9,] "i"  "0.472272453285111" 
## [10,] "c"  "-0.527727546714889"

2.1 Cumulative probability of first fixation

We begin by computing and plotting the cumulative distribution of the ordinal number of fixations for consistent and inconsistent objects respectively. We will exclude the values that are higher than the 99% of the distribution.

## get the quantile and look at the ordinal nr. of fixation that is over 99% of the distribution
qnt = quantile(ET$FIXNR_FIRSTFIX, seq(0,1,.01))

mxfix   = as.numeric(qnt[which(names(qnt) == "99%")]) ## maximum number of ordinal fixations considered

cset = subset(ET$FIXNR_FIRSTFIX, ET$consistency == "c")
iset = subset(ET$FIXNR_FIRSTFIX, ET$consistency == "i")

cf  = ecdf(cset)
cf  = cf(1:mxfix)
inf = ecdf(iset)
inf = inf(1:mxfix)

visualize it:

alpha = .05 ## set alpha values for plotting

plot(1:mxfix, cf, ylim = c(0,1), col = 'blue4', lty = 1, 
     lwd = 5, cex = 1.8, pch = 20, type = 'b',
     ylab = 'Cumulative Probability of Target Fixation', 
     xlab = 'Ordinal fixation number')

lines(1:mxfix, inf, ylim = c(0,1), col = 'red4', 
      lwd = 5, cex = 1.8, lty = 2, type = 'b', pch = 15)

legend('bottomright', c('Consistent Object', 'Inconsistent Object'),
       col = c('blue4', 'red4'), lty = c(1,2), pch = c(20,15), 
       cex = 1.5, bty = 'n')
grid()

Then, in order to apply a generalized linear model to the ordinal number of fixations we need to slightly reshape the data, and calculate a frequency count to then compute a probit model. We do this separately for the conditions of object consistency.

cond = unique(ET$consistency)
c = cond[1]; 

freqprob = vector()
for (c in cond){
  print(c)
  fixc = vector()
  sset = subset(ET, ET$consistency == c)
  cump = ecdf(sset$FIXNR_FIRSTFIX)
  for (s in 1:mxfix){
    ixfix = length(which(sset$FIXNR_FIRSTFIX == s))
    fixc = c(fixc, ixfix) 
  }
  
  freqprob = cbind(freqprob, fixc, deparse.level = 0)
}

## [1] "c"
## [1] "i"

freqprob = as.data.frame(freqprob)
colnames(freqprob) = c("Consistent", "Inconsistent")
freqprob$timepoint = 1:mxfix

totn = table(ET$consistency) ## get the total nr. of points 

M           = melt(id = c('timepoint'), data = freqprob)
M$r         = c(rep(totn[1], mxfix),  rep(totn[2], mxfix)) - M$value
M$timepoint = as.numeric(as.matrix(M$timepoint))
M$value     = as.numeric(as.matrix(M$value))
M$x         = M$timepoint/max(M$timepoint)
head(M)

##   timepoint   variable value    r          x
## 1         1 Consistent   148 1419 0.03571429
## 2         2 Consistent   246 1321 0.07142857
## 3         3 Consistent   192 1375 0.10714286
## 4         4 Consistent   172 1395 0.14285714
## 5         5 Consistent   114 1453 0.17857143
## 6         6 Consistent   114 1453 0.21428571

and now, let’s run the model:

model <- glm(cbind(value, r) ~  x*variable, data = M, family = binomial(probit))
summary(model)

## 
## Call:
## glm(formula = cbind(value, r) ~ x * variable, family = binomial(probit), 
##     data = M)
## 
## Deviance Residuals: 
##    Min      1Q  Median      3Q     Max  
## -4.683  -1.588  -0.031   1.863  10.258  
## 
## Coefficients:
##                        Estimate Std. Error z value Pr(>|z|)    
## (Intercept)            -1.04102    0.02073 -50.207  < 2e-16 ***
## x                      -2.02374    0.05708 -35.455  < 2e-16 ***
## variableInconsistent    0.17183    0.02909   5.907 3.49e-09 ***
## x:variableInconsistent -0.71673    0.08891  -8.062 7.53e-16 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 5140.51  on 55  degrees of freedom
## Residual deviance:  389.73  on 52  degrees of freedom
## AIC: 676.24
## 
## Number of Fisher Scoring iterations: 5

confint(model)

##                             2.5 %     97.5 %
## (Intercept)            -1.0814693 -1.0006846
## x                      -2.1356545 -1.9139428
## variableInconsistent    0.1155082  0.2281842
## x:variableInconsistent -0.8877653 -0.5464493

Table 2 of the manuscripts also reports the corresponding GLM model coefficients. We found a significant main effect of Consistency; overall, inconsistent objects were looked at with a higher probability than consistent objects. As expected, the cumulative probability of looking at the critical object increased as a function of the Ordinal Number of Fixation (i.e., x here). There was also a significant interaction between the two variables.

2.2 Probability of immediate fixation

Now, we examine the very first eye-movement during scene exploration to assess whether observers had immediate extrafoveal access to object-scene semantics, i.e., the probability of immediate fixation. The mean probability of immediate object fixation was 12.92

with(ET, mean(FIRSTFIX_ON_OBJ))*100

## [1] 12.92948

Visualize it:

mean <- with(ET, tapply(FIRSTFIX_ON_OBJ, list(consistency, subj), mean))
ymax = max(mean)

M = as.data.frame(cbind(c(mean[1,], mean[2,]), 
                        c(rep("Consistent", ncol(mean)), 
                          rep("Inconsistent", ncol(mean))), 
                        deparse.level = 0),
                  check.names = FALSE,
                  row.names = FALSE)

colnames(M) = c("value", "consistency")

M[,1] = as.numeric(as.matrix(M[,1]))

boxplot(value ~ consistency, data = M, 
        col = c("blue4", "red4"),
        ylab = "Probability of Immediate Fixation",
        cex.lab = 1.3,
        cex.axis = 1.8,
        ylim     = c(0, ymax*1.1))

and now run the LME model:

mod = glmer(FIRSTFIX_ON_OBJ ~ consistencyC
            + (1 | subj) + 
              + (1 | scene), 
            data = ET,
            family = 'binomial')

## there are no model for backward reducing GLMER models.
summary(mod)

## Generalized linear mixed model fit by maximum likelihood (Laplace
##   Approximation) [glmerMod]
##  Family: binomial  ( logit )
## Formula: FIRSTFIX_ON_OBJ ~ consistencyC + (1 | subj) + +(1 | scene)
##    Data: ET
## 
##      AIC      BIC   logLik deviance df.resid 
##   2191.2   2215.6  -1091.6   2183.2     3314 
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -1.8467 -0.3463 -0.2059 -0.1395  5.3468 
## 
## Random effects:
##  Groups Name        Variance Std.Dev.
##  scene  (Intercept) 2.2296   1.4932  
##  subj   (Intercept) 0.2066   0.4545  
## Number of obs: 3318, groups:  scene, 192; subj, 24
## 
## Fixed effects:
##              Estimate Std. Error z value Pr(>|z|)    
## (Intercept)   -2.8194     0.1836 -15.358   <2e-16 ***
## consistencyC   0.2186     0.1582   1.381    0.167    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.033

2.3 Latency to first fixation

The latency to first fixation on the target object is another measure to capture the potency of an object in attracting early attention in extrafoveal vision. This measure is defined as the time elapsed between the onset of the scene image and the first fixation on the critical object.

Visualize:

mean <- with(ET, tapply(TIME_FIRSTFIX, list(consistency, subj), mean, na.rm = TRUE))
ymax = max(mean, na.rm = TRUE)
ymin = min(mean, na.rm = TRUE)

M = as.data.frame(cbind(c(mean[1,], mean[2,]), 
                        c(rep("Consistent", ncol(mean)), 
                          rep("Inconsistent", ncol(mean))), 
                        deparse.level = 0),
                  check.names = FALSE,
                  row.names = FALSE)

## take out the nas
# M = M[-as.numeric(which(is.na(M[,1]) == TRUE)),]

colnames(M) = c("value", "consistency")

M[,1] = as.numeric(as.matrix(M[,1]))

boxplot(value ~ consistency, data = M, 
        col = c("blue4", "red4"),
        ylab = "Latency to First Fixation",
        cex.lab = 1.3,
        cex.axis = 1.8,
        ylim     = c(ymin, ymax))

model it:

## model the time of first fixation
mod = lmer(TIME_FIRSTFIX ~ consistencyC 
           + (1 | subj) + (1 | scene), 
           data = ET)

summary(mod)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: TIME_FIRSTFIX ~ consistencyC + (1 | subj) + (1 | scene)
##    Data: ET
## 
## REML criterion at convergence: 57632.5
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.9543 -0.5315 -0.1950  0.2507  5.3569 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  scene    (Intercept)  929298   964.0  
##  subj     (Intercept)   11979   109.5  
##  Residual             1801006  1342.0  
## Number of obs: 3318, groups:  scene, 192; subj, 24
## 
## Fixed effects:
##              Estimate Std. Error      df t value Pr(>|t|)    
## (Intercept)   1774.44      77.17  160.37   23.00  < 2e-16 ***
## consistencyC  -246.37      63.99 3197.80   -3.85  0.00012 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.014

The latency was significantly shorter for inconsistent as compared to consistent objects (refer to Figure 3C and Table 3 of the manuscript).

2.4 Gaze duration

First-pass gaze duration is a measure of foveal object processing time, defined as the sum of all fixation duration from first entry to first exit on the critical object. On average, participants looked longer at inconsistent (520 msec) than consistent (409 msec) objects.

with(ET, tapply(GD, list(consistency), mean))

##        c        i 
## 408.5175 519.1650

Visualize:

mean <- with(ET, tapply(GD, list(consistency, subj), mean, na.rm = TRUE))
ymax = max(mean, na.rm = TRUE)
ymin = min(mean, na.rm = TRUE)

M = as.data.frame(cbind(c(mean[1,], mean[2,]), 
                        c(rep("Consistent", ncol(mean)), 
                          rep("Inconsistent", ncol(mean))), 
                        deparse.level = 0),
                  check.names = FALSE,
                  row.names = FALSE)

## take out the nas
# M = M[-as.numeric(which(is.na(M[,1]) == TRUE)),]

colnames(M) = c("value", "consistency")

M[,1] = as.numeric(as.matrix(M[,1]))

boxplot(value ~ consistency, data = M, 
        col = c("blue4", "red4"),
        ylab = "Gaze Duration (First Pass)",
        cex.lab = 1.3,
        cex.axis = 1.8,
        ylim     = c(ymin, ymax))

model it:

mod = lmer(GD ~ consistencyC
           + (1 | subj) + (1 | scene), 
           data = ET)

summary(mod)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: GD ~ consistencyC + (1 | subj) + (1 | scene)
##    Data: ET
## 
## REML criterion at convergence: 48345.7
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.0950 -0.5231 -0.1559  0.3106 26.9541 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  scene    (Intercept)  15222   123.38  
##  subj     (Intercept)   6337    79.61  
##  Residual             115310   339.57  
## Number of obs: 3318, groups:  scene, 192; subj, 24
## 
## Fixed effects:
##              Estimate Std. Error      df t value Pr(>|t|)    
## (Intercept)    455.54      19.52   36.55  23.333  < 2e-16 ***
## consistencyC   105.02      14.83 2053.08   7.082 1.94e-12 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.009

On average, participants looked longer at inconsistent (520 ms) than consistent objects (409 ms) before leaving the target object for the first time, and this difference was significant (refer to Table 3 in the manuscript)

In the next section, we are going to be looking whether the prioritization of an inconsistent object over a consistent object manifests also in the fixation-related potentials time-locked to the fixation preceding the target fixation (i.e., t-1), and whether such an effect carries over throughout fixation t. We also look at the rERP locked at the onset of the scene (s), and at the fixation-related potentials of all non-target fixations (i.e., nt)

3 Electro-physiological results

Results in this tutorial will only focus on the second-level of analysis (group statistics). Specifically, we compare the consistent and inconsistent condition in the time windows from 250-350 ms (early effect) and 350-600 ms (late effect) at a mid-central region-of-interest (ROI) of nine electrodes (comprising FC1, FCz, FC2, C1, Cz, C2, CP1, CPz, and CP2), independently for four types of events: stimulus-locked (s), non-target fixations (nt), pre-target fixations (t-1) and target fixations (t).

Because the output provided by the linear deconvolution model (the first-level analysis) is already aggregated at the level of subject-averages, the only predictor included in these LMMs was the Consistency of the object, along with Participant as random effect, which is included both as intercept, and as random slope of Consistency to make the test more conservative. Then, we utilize the step function from the lmerTest() package to compute the level of significance was calculated from an F-test based on the Satterthwaite approximation to the effective degrees of freedom, as well as, evaluate the random structure of the models to maximize their convergence. Additionally, we report a simple ANOVA analysis, which perfectly corroborates the LME results.

We refer the reader to the manuscript for much greater details about the first-level of analysis, and the pipeline that we have followed to compute the fixation-related potentials.

3.1 Structure of the EEG data

Let’s look at the structure of the EEG data file first.

str(EEG)

## 'data.frame':    3456 obs. of  6 variables:
##  $ subj       : int  1 2 3 4 5 6 7 8 9 10 ...
##  $ consistency: Factor w/ 2 levels "consistent","inconsistent": 1 1 1 1 1 1 1 1 1 1 ...
##  $ type_event : Factor w/ 4 levels "nt","pt","s",..: 4 4 4 4 4 4 4 4 4 4 ...
##  $ segment    : Factor w/ 2 levels "0.250-0.350",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ voltage    : num  0.37406 -0.60501 -1.50245 -0.00127 -1.35604 ...
##  $ electrode  : Factor w/ 9 levels "C1","C2","CP1",..: 7 7 7 7 7 7 7 7 7 7 ...

The EEG/FRP data has been structured along the columns in the following way:

• subj = the subject ID
• consistency = the consistency of the target object (consistent, inconsistent)
• type_event = the type of EEG data analysed, with 4 levels (s = stimulus-onset, nt = non-target, pt = preceding the target, t = target)
• segment = the temporal windows of interest (0.250-0.350 = early; 0.350-0.600 = late)
• voltage = the voltage of the EEG recording
• electrode = the nine electrodes of the mid-central region of interest (comprising FC1, FCz, FC2, C1, Cz, C2, CP1, CPz, and CP2).

To this we add the centred version of the consistency variable.

EEG$consistencyC = as.numeric(as.factor(EEG$consistency))
EEG$consistencyC = EEG$consistencyC - mean(EEG$consistencyC, na.rm = T) 
c(as.character(EEG$consistency[1]), EEG$consistencyC[1])

## [1] "consistent" "-0.5"

We will subset and analyse the relevant type of event and window of interests that are reported in the manuscript (Table 4)

3.2 Scene-onset ERP, s

First, we look at the evoked potentials time-locked to the scene onset for the two temporal windows of interest.

s_early = subset(EEG, EEG$type_event == "s" & EEG$segment == "0.250-0.350")
s_late  = subset(EEG, EEG$type_event == "s" & EEG$segment == "0.350-0.600")

print("Early-Window")

## [1] "Early-Window"

s_e = lmer(voltage ~ consistencyC
           + (1 | subj) + (0 + consistency | subj),
           data = s_early, 
           control=lmerControl(check.conv.singular = .makeCC(action = "ignore",  tol = 1e-4)))

print("LME output")

## [1] "LME output"

summary(s_e)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (1 | subj) + (0 + consistency | subj)
##    Data: s_early
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 2489.4
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.1225 -0.6408 -0.1307  0.6284  2.6793 
## 
## Random effects:
##  Groups   Name                    Variance  Std.Dev. Corr
##  subj     (Intercept)             7.819e+00 2.796258     
##  subj.1   consistencyconsistent   1.005e-05 0.003171     
##           consistencyinconsistent 1.454e-05 0.003813 1.00
##  Residual                         1.654e+01 4.067460     
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)   -3.9393     0.6034  22.9999  -6.528 1.16e-06 ***
## consistencyC   0.2802     0.3914 406.9981   0.716    0.475    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.000

s_e_red = step(s_e, ddf = "Satterthwaite", 
                         alpha.random = 0.1, alpha.fixed = 0.08,       
                         reduce.fixed = FALSE, reduce.random = TRUE)

s_e_red    = get_model(s_e_red)


print("LME selected with p-values")

## [1] "LME selected with p-values"

summary(s_e_red)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (1 | subj)
##    Data: s_early
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 2489.4
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.1225 -0.6408 -0.1307  0.6284  2.6793 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  subj     (Intercept)  7.819   2.796   
##  Residual             16.544   4.067   
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)   -3.9393     0.6034  23.0000  -6.528 1.16e-06 ***
## consistencyC   0.2802     0.3914 407.0000   0.716    0.475    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.000

aov_s_e = ezANOVA(
  data = s_early
  , dv = voltage
  , wid = subj
  , within = .(consistency))

print("ANOVA output")

## [1] "ANOVA output"

print(aov_s_e)

## $ANOVA
##        Effect DFn DFd        F          p p<.05         ges
## 2 consistency   1  23 3.450846 0.07606891       0.002301984

print("Now Late-Window")

## [1] "Now Late-Window"

s_l = lmer(voltage ~ consistencyC
           + (1 | subj) + (0 + consistency | subj),
           data = s_late,
           control=lmerControl(check.conv.singular = .makeCC(action = "ignore",  tol = 1e-4)))

print("LME output")

## [1] "LME output"

summary(s_l)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (1 | subj) + (0 + consistency | subj)
##    Data: s_late
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 2482.5
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -2.13624 -0.66651 -0.09792  0.72287  2.84615 
## 
## Random effects:
##  Groups   Name                    Variance  Std.Dev. Corr
##  subj     (Intercept)             4.249e-05 0.006518     
##  subj.1   consistencyconsistent   4.317e+00 2.077808     
##           consistencyinconsistent 4.787e+00 2.187840 1.00
##  Residual                         1.670e+01 4.086668     
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)   -3.5661     0.4777  22.9931  -7.465 1.38e-07 ***
## consistencyC   0.3483     0.3939 366.3213   0.884    0.377    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.052

s_l_red = step(s_l, ddf = "Satterthwaite", 
                         alpha.random = 0.1, alpha.fixed = 0.08,       
                         reduce.fixed = FALSE, reduce.random = TRUE)

s_l_red    = get_model(s_l_red)


print("LME selected with p-values")

## [1] "LME selected with p-values"

summary(s_l_red)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (1 | subj)
##    Data: s_late
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 2482.6
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -2.10818 -0.66949 -0.09964  0.72424  2.84181 
## 
## Random effects:
##  Groups   Name        Variance Std.Dev.
##  subj     (Intercept)  4.547   2.132   
##  Residual             16.705   4.087   
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)   -3.5661     0.4776  23.0000  -7.466 1.37e-07 ***
## consistencyC   0.3483     0.3933 407.0000   0.886    0.376    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.000

aov_s_l = ezANOVA(
  data = s_late
  , dv = voltage
  , wid = subj
  , within = .(consistency))

print("ANOVA output")

## [1] "ANOVA output"

print(aov_s_l)

## $ANOVA
##        Effect DFn DFd        F          p p<.05         ges
## 2 consistency   1  23 3.902595 0.06033019       0.005558076

We do not observe any influence of semantic consistency of the target object on the neural response to the initial presentation of the scene in both windows of analysis.

3.3 Non-target fixation, nt

Now, we look at the non-target fixation.

nt_early = subset(EEG, EEG$type_event == "nt" & EEG$segment == "0.250-0.350")
nt_late  = subset(EEG, EEG$type_event == "nt" & EEG$segment == "0.350-0.600")

print("Early-Window")

## [1] "Early-Window"

nt_e = lmer(voltage ~ consistencyC
            + (1 | subj) + (0 + consistency | subj),
            data = nt_early,
            control=lmerControl(check.conv.singular = .makeCC(action = "ignore",  tol = 1e-4)))

print("LME output")

## [1] "LME output"

summary(nt_e)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (1 | subj) + (0 + consistency | subj)
##    Data: nt_early
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 516.6
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.4588 -0.5841 -0.0784  0.5698  4.1889 
## 
## Random effects:
##  Groups   Name                    Variance Std.Dev. Corr
##  subj     (Intercept)             0.16081  0.4010       
##  subj.1   consistencyconsistent   0.14727  0.3838       
##           consistencyinconsistent 0.08814  0.2969   0.56
##  Residual                         0.14597  0.3821       
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)  -0.52785    0.10397 22.99976  -5.077 3.85e-05 ***
## consistencyC -0.06077    0.07672 23.00228  -0.792    0.436    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC -0.154
## convergence code: 0
## unable to evaluate scaled gradient
## Model failed to converge: degenerate  Hessian with 1 negative eigenvalues

nt_e_red = step(nt_e, ddf = "Satterthwaite", 
                         alpha.random = 0.1, alpha.fixed = 0.08,       
                         reduce.fixed = FALSE, reduce.random = TRUE)

nt_e_red    = get_model(nt_e_red)


print("LME selected with p-values")

## [1] "LME selected with p-values"

summary(nt_e_red)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (0 + consistency | subj)
##    Data: nt_early
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 516.6
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -3.4588 -0.5841 -0.0784  0.5698  4.1889 
## 
## Random effects:
##  Groups   Name                    Variance Std.Dev. Corr
##  subj     consistencyconsistent   0.3081   0.5550       
##           consistencyinconsistent 0.2489   0.4989   0.81
##  Residual                         0.1460   0.3821       
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)  -0.52785    0.10396 23.00101  -5.077 3.85e-05 ***
## consistencyC -0.06077    0.07672 23.00092  -0.792    0.436    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC -0.154

aov_nt_e = ezANOVA(
  data = nt_early
  , dv = voltage
  , wid = subj
  , within = .(consistency))

print("ANOVA output")

## [1] "ANOVA output"

print(aov_nt_e)

## $ANOVA
##        Effect DFn DFd         F         p p<.05         ges
## 2 consistency   1  23 0.6273906 0.4364079       0.003258092

print("Now Late-Window")

## [1] "Now Late-Window"

nt_l      = lmer(voltage ~ consistencyC
                 + (1 | subj) + (0 + consistency | subj),
                 data = nt_late,
                 control=lmerControl(check.conv.singular = .makeCC(action = "ignore",  tol = 1e-4)))

print("LME output")

## [1] "LME output"

summary(nt_l)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (1 | subj) + (0 + consistency | subj)
##    Data: nt_late
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 672.2
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.8719 -0.5579 -0.0300  0.4822  3.3442 
## 
## Random effects:
##  Groups   Name                    Variance Std.Dev. Corr
##  subj     (Intercept)             0.2722   0.5218       
##  subj.1   consistencyconsistent   0.1185   0.3442       
##           consistencyinconsistent 0.0320   0.1789   0.05
##  Residual                         0.2122   0.4606       
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)  -0.45579    0.11602 22.99626  -3.929 0.000672 ***
## consistencyC -0.09661    0.08943 22.99876  -1.080 0.291228    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC -0.174
## convergence code: 0
## Model failed to converge with max|grad| = 0.00370665 (tol = 0.002, component 1)

nt_l_red = step(nt_l, ddf = "Satterthwaite", 
                         alpha.random = 0.1, alpha.fixed = 0.08,       
                         reduce.fixed = FALSE, reduce.random = TRUE)

nt_l_red    = get_model(nt_l_red)


print("LME selected with p-values")

## [1] "LME selected with p-values"

summary(nt_l_red)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (0 + consistency | subj)
##    Data: nt_late
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 672.2
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -2.8719 -0.5579 -0.0300  0.4822  3.3442 
## 
## Random effects:
##  Groups   Name                    Variance Std.Dev. Corr
##  subj     consistencyconsistent   0.3907   0.6250       
##           consistencyinconsistent 0.3042   0.5516   0.80
##  Residual                         0.2122   0.4606       
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error       df t value Pr(>|t|)    
## (Intercept)  -0.45579    0.11602 22.99620  -3.929 0.000672 ***
## consistencyC -0.09661    0.08943 23.00054  -1.080 0.291220    
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC -0.174

aov_nt_l = ezANOVA(
  data = nt_late
  , dv = voltage
  , wid = subj
  , within = .(consistency))

print("ANOVA output")

## [1] "ANOVA output"

print(aov_nt_l)

## $ANOVA
##        Effect DFn DFd        F         p p<.05         ges
## 2 consistency   1  23 1.166996 0.2912174       0.006520055

Consistency had no effect on the FRP for non-target (nt) fixations in both windows of analysis.

3.4 Pre-target fixation, t-1

Moving onto the fixation-related potential time-locked to the pre-target fixation.

pt_early = subset(EEG, EEG$type_event == "pt" & EEG$segment == "0.250-0.350")
pt_late  = subset(EEG, EEG$type_event == "pt" & EEG$segment == "0.350-0.600")

print("Early-Window")

## [1] "Early-Window"

pt_e = lmer(voltage ~ consistencyC
            + (1 | subj) + (0 + consistency | subj),
            data = pt_early,
            control=lmerControl(check.conv.singular = .makeCC(action = "ignore",  tol = 1e-4)))

print("LME output")

## [1] "LME output"

summary(pt_e)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (1 | subj) + (0 + consistency | subj)
##    Data: pt_early
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 1082.8
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -8.4625 -0.5095 -0.0438  0.5212  3.3456 
## 
## Random effects:
##  Groups   Name                    Variance  Std.Dev.  Corr
##  subj     (Intercept)             5.434e-08 0.0002331     
##  subj.1   consistencyconsistent   3.480e-01 0.5898730     
##           consistencyinconsistent 3.455e-01 0.5877536 0.33
##  Residual                         5.838e-01 0.7640382     
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error      df t value Pr(>|t|)    
## (Intercept)   -0.8342     0.1046 22.9994  -7.975 4.53e-08 ***
## consistencyC  -0.2799     0.1576 23.0000  -1.776   0.0889 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC -0.003

pt_e_red = step(pt_e, ddf = "Satterthwaite", 
                         alpha.random = 0.1, alpha.fixed = 0.08,       
                         reduce.fixed = FALSE, reduce.random = TRUE)

pt_e_red    = get_model(pt_e_red)


print("LME selected with p-values")

## [1] "LME selected with p-values"

summary(pt_e_red)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (0 + consistency | subj)
##    Data: pt_early
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 1082.8
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -8.4625 -0.5095 -0.0438  0.5211  3.3456 
## 
## Random effects:
##  Groups   Name                    Variance Std.Dev. Corr
##  subj     consistencyconsistent   0.3479   0.5898       
##           consistencyinconsistent 0.3454   0.5877   0.33
##  Residual                         0.5838   0.7640       
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error      df t value Pr(>|t|)    
## (Intercept)   -0.8342     0.1046 23.0031  -7.975 4.52e-08 ***
## consistencyC  -0.2799     0.1576 23.0024  -1.777   0.0889 .  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC -0.003

aov_pt_e = ezANOVA(
  data = pt_early
  , dv = voltage
  , wid = subj
  , within = .(consistency))


print("ANOVA output")

## [1] "ANOVA output"

print(aov_pt_e)

## $ANOVA
##        Effect DFn DFd        F          p p<.05        ges
## 2 consistency   1  23 3.155887 0.08888493       0.04731496

print("Now Late-Window")

## [1] "Now Late-Window"

pt_l = lmer(voltage ~ consistencyC
            + (1 | subj) + (0 + consistency | subj),
            data = pt_late,
            control=lmerControl(check.conv.singular = .makeCC(action = "ignore",  tol = 1e-4)))

print("LME output")

## [1] "LME output"

summary(pt_l)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (1 | subj) + (0 + consistency | subj)
##    Data: pt_late
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 1150.5
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -10.9245  -0.4486  -0.0203   0.4576   3.4264 
## 
## Random effects:
##  Groups   Name                    Variance Std.Dev. Corr 
##  subj     (Intercept)             0.2663   0.5160        
##  subj.1   consistencyconsistent   0.0287   0.1694        
##           consistencyinconsistent 0.1233   0.3512   -0.49
##  Residual                         0.7080   0.8414        
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error      df t value Pr(>|t|)    
## (Intercept)   -0.7292     0.1171 23.0017  -6.226 2.37e-06 ***
## consistencyC  -0.4662     0.1237 23.0028  -3.769 0.000996 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.136 
## convergence code: 0
## unable to evaluate scaled gradient
## Model failed to converge: degenerate  Hessian with 1 negative eigenvalues

pt_l_red = step(pt_l, ddf = "Satterthwaite", 
                         alpha.random = 0.1, alpha.fixed = 0.08,       
                         reduce.fixed = FALSE, reduce.random = TRUE)

pt_l_red    = get_model(pt_l_red)


print("LME selected with p-values")

## [1] "LME selected with p-values"

summary(pt_l_red)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (0 + consistency | subj)
##    Data: pt_late
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 1150.5
## 
## Scaled residuals: 
##      Min       1Q   Median       3Q      Max 
## -10.9245  -0.4486  -0.0203   0.4576   3.4265 
## 
## Random effects:
##  Groups   Name                    Variance Std.Dev. Corr
##  subj     consistencyconsistent   0.2950   0.5432       
##           consistencyinconsistent 0.3897   0.6242   0.70
##  Residual                         0.7080   0.8414       
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error      df t value Pr(>|t|)    
## (Intercept)   -0.7292     0.1171 22.9986  -6.226 2.37e-06 ***
## consistencyC  -0.4662     0.1237 22.9990  -3.769 0.000997 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.136

aov_pt_l = ezANOVA(
  data = pt_late
  , dv = voltage
  , wid = subj
  , within = .(consistency))

print("ANOVA output")

## [1] "ANOVA output"

print(aov_pt_l)

## $ANOVA
##        Effect DFn DFd        F            p p<.05       ges
## 2 consistency   1  23 14.20578 0.0009964811     * 0.1186879

In the LMM analyses conducted on the mid-central ROI, this effect was marginally significant (p < 0.1) for the early time window (250 to 350 ms), but became highly significant between 350 and 600 ms (p < 0.001, Table 4)

3.5 Target fixation, t

Finally, we look at the fixation-related potential during foveation of the target.

t_early = subset(EEG, EEG$type_event == "t" & EEG$segment == "0.250-0.350")
t_late  = subset(EEG, EEG$type_event == "t" & EEG$segment == "0.350-0.600")

print("Early-Window")

## [1] "Early-Window"

t_e = lmer(voltage ~ consistencyC
           + (1 | subj) + (0 + consistency | subj),
           data = t_early,
           control=lmerControl(check.conv.singular = .makeCC(action = "ignore",  tol = 1e-4)))

print("LME output")

## [1] "LME output"

summary(t_e)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (1 | subj) + (0 + consistency | subj)
##    Data: t_early
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 1278.9
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -9.6219 -0.4435  0.0110  0.4489  3.8891 
## 
## Random effects:
##  Groups   Name                    Variance Std.Dev. Corr
##  subj     (Intercept)             0.0349   0.1868       
##  subj.1   consistencyconsistent   0.2861   0.5349       
##           consistencyinconsistent 0.5691   0.7544   0.41
##  Residual                         0.9434   0.9713       
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error      df t value Pr(>|t|)    
## (Intercept)   -0.7941     0.1265 22.9951  -6.279 2.09e-06 ***
## consistencyC  -0.5298     0.1749 23.0024  -3.030  0.00596 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.267 
## convergence code: 0
## Model failed to converge with max|grad| = 0.00360266 (tol = 0.002, component 1)

t_e_red = step(t_e, ddf = "Satterthwaite", 
                         alpha.random = 0.1, alpha.fixed = 0.08,       
                         reduce.fixed = FALSE, reduce.random = TRUE)

t_e_red    = get_model(t_e_red)

print("LME selected with p-values")

## [1] "LME selected with p-values"

summary(t_e_red)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (0 + consistency | subj)
##    Data: t_early
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 1278.9
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -9.6218 -0.4435  0.0110  0.4489  3.8891 
## 
## Random effects:
##  Groups   Name                    Variance Std.Dev. Corr
##  subj     consistencyconsistent   0.3210   0.5666       
##           consistencyinconsistent 0.6038   0.7771   0.46
##  Residual                         0.9434   0.9713       
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error      df t value Pr(>|t|)    
## (Intercept)   -0.7941     0.1265 22.9942  -6.279 2.09e-06 ***
## consistencyC  -0.5298     0.1748 23.0096  -3.030  0.00595 ** 
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.266

aov_t_e = ezANOVA(
  data = t_early
  , dv = voltage
  , wid = subj
  , within = .(consistency))

print("ANOVA output")

## [1] "ANOVA output"

print(aov_t_e)

## $ANOVA
##        Effect DFn DFd        F           p p<.05      ges
## 2 consistency   1  23 9.179458 0.005958477     * 0.114331

print("Now Late-Window")

## [1] "Now Late-Window"

t_l = lmer(voltage ~ consistencyC
           + (1 | subj) + (0 + consistency | subj),
           data = t_late,
           control=lmerControl(check.conv.singular = .makeCC(action = "ignore",  tol = 1e-4)))

print("LME output")

## [1] "LME output"

summary(t_l)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (1 | subj) + (0 + consistency | subj)
##    Data: t_late
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 1317.1
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -7.7440 -0.4918  0.0128  0.5542  5.3502 
## 
## Random effects:
##  Groups   Name                    Variance Std.Dev. Corr
##  subj     (Intercept)             0.3079   0.5549       
##  subj.1   consistencyconsistent   0.0000   0.0000       
##           consistencyinconsistent 0.3057   0.5529    NaN
##  Residual                         1.0542   1.0267       
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error      df t value Pr(>|t|)    
## (Intercept)   -0.6754     0.1359 21.5286  -4.971 5.99e-05 ***
## consistencyC  -0.3829     0.1500 23.3490  -2.553   0.0177 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.313

t_l_red = step(t_l, ddf = "Satterthwaite", 
                         alpha.random = 0.1, alpha.fixed = 0.08,       
                         reduce.fixed = FALSE, reduce.random = TRUE)

t_l_red    = get_model(t_l_red)

print("LME selected with p-values")

## [1] "LME selected with p-values"

summary(t_l_red)

## Linear mixed model fit by REML. t-tests use Satterthwaite's method [
## lmerModLmerTest]
## Formula: voltage ~ consistencyC + (0 + consistency | subj)
##    Data: t_late
## Control: lmerControl(check.conv.singular = .makeCC(action = "ignore",  
##     tol = 1e-04))
## 
## REML criterion at convergence: 1316.1
## 
## Scaled residuals: 
##     Min      1Q  Median      3Q     Max 
## -7.7036 -0.4948  0.0145  0.5400  5.3023 
## 
## Random effects:
##  Groups   Name                    Variance Std.Dev. Corr
##  subj     consistencyconsistent   0.3517   0.5931       
##           consistencyinconsistent 0.5019   0.7085   0.59
##  Residual                         1.0478   1.0236       
## Number of obs: 432, groups:  subj, 24
## 
## Fixed effects:
##              Estimate Std. Error      df t value Pr(>|t|)    
## (Intercept)   -0.6754     0.1282 23.0037  -5.267 2.41e-05 ***
## consistencyC  -0.3829     0.1574 22.9961  -2.433   0.0232 *  
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Correlation of Fixed Effects:
##             (Intr)
## consistncyC 0.155

aov_t_l = ezANOVA(
  data = t_late
  , dv = voltage
  , wid = subj
  , within = .(consistency))

print("ANOVA output")

## [1] "ANOVA output"

print(aov_t_l)

## $ANOVA
##        Effect DFn DFd        F          p p<.05        ges
## 2 consistency   1  23 5.917444 0.02318162     * 0.06576877

The effect of object consistency is significant in both the early and late window of analysis.

4 Conclusion

Our findings converge to suggest that the visual system accesses semantic features of objects in extrafoveal vision to guide attention towards objects that do not fit to the scene’s overall meaning. They also highlight the utility of investigating attentional and neural mechanisms in parallel to uncover the mechanisms underlying object recognition during the unconstrained exploration of naturalistic scenes.

5 References

Coco, Nuthmann & Dimigen (in press). Fixation-related brain potentials during semantic integration of object-scene information. Journal of Cognitive Neuroscience.