Science of Chess - Eyetracking, board vision, and expertise (Part 2 of 2)

In my last Science of Chess article, I started talking about the idea that understanding how people move their eyes when they try to accomplish some kind of visual task might yield clues regarding the processes that contribute to their success. In visual cognition research more broadly, eyetracking has long been used for this purpose and has had an immense impact on our understanding of visual attention, search, and object recognition. So what about the visual task posed by the chessboard? Can eyetracking reveal anything about how playing strength may relate to chess players' perception of the board?

In that last post, I described the results of a study by Charness et al. (2001) that did reveal some possible perceptual differences between expert and intermediate players based on measurable differences in eye movements. Specifically, we found out that weaker players tended to make more eye movements that were small in magnitude (low saccade amplitude) and also that stronger players tended to both look at salient pieces in a position more frequently, but also that they looked more at empty spaces than intermediate players. I said at the end of that post that this last result could be taken as evidence for a particular hypothesis about how perceptual abilities might differ as a function of playing strength. Here, we'll talk about that hypothesis in more detail and check out a paper from this same group that uses some neat techniques (supported by eyetracking) to investigate the key idea.

Seeing Sidelong - What can you do with peripheral vision?

An important preamble to this discussion is a property of our visual processing you may not have thought about too much, but is very important for thinking about eyetracking results carefully: Where you're looking isn't the only place you see things. It's easy to think about the kinds of scanpaths we get from densely sampled eyetracking data as a trace of what people saw in a particular task, but the reality is more complicated than that. The human retina is structured such that it has a fovea at the center, packed densely with cone photoreceptors that support color vision and tend to project to the next stage of processing with low convergence - the cells next in line tend to only receive signals from a small number of cones. Outside the fovea (which is about 1.5 degrees of visual angle or so), things look different. The high concentration of cones gives way to more and more rod photoreceptors, and these tend to have much higher convergence: each cell receiving signals from these photoreceptors in the periphery may have connections to dozens of cells. Moving your eyes to a particular part of an image means using your fovea to measure what's happening right where you're looking, but you also get to use the periphery to measure the rest of the visual field, too.

Convergence between photoreceptors and ganglion cells is low in the fovea (Cell 1), leading to higher acuity than in the periphery where convergence is much higher. Image credit: Casey Henley, Creative Commons Attribution Non-Commercial Share-Alike (CC BY-NC-SA) 4.0 International License

These structural differences in the retina between the fovea and the periphery have functional consequences: Color discrimination is better in the fovea and declines in the periphery, for example, and the same is true for visual acuity. To give you a sense of how peripheral visual acuity compares to central vision, take a look the figure below in which the letters are scaled to be about equally legible when you're looking at the dot in the center of the image. You can get away with drawing that itsy little "M" in the fovea, but you need a giant K once you're out in the periphery.

You can still see visual structure in the periphery, but subject to important limitations on visual processing. In this figure, the letters are scaled to be approximately equally legible if you're looking at the very center of the image to show off the difference in acuity across the visual field. Stuart M Anstis, CC BY-SA 4.0 <https://creativecommons.org/licenses/by-sa/4.0>, via Wikimedia Commons

Clearly peripheral vision is not the best for some things. Still, that doesn't mean that it isn't there! As you (or a GM, or anyone else) move your eyes around the chessboard, you can see very fine details right where you're looking, AND you get to see coarse, blurry versions of the stuff that's off to the side. OK, to be fair, it might actually be a little bit worse than that due to another limitation on visual processing that I mentioned in another prior article:visual crowding. Crowding refers to the difficulty observers experience trying to recognize an item in peripheral vision when it has other items nearby. Recognizing the object itself tends to be hard (check out the figure below to see for yourself) and subjectively, observers tend to report that items that are subject to crowding look jumbled or scrambled somehow. Observers also sometimes make errors of position in displays where crowding is a problem - you might be convinced that the rightward-looking duck is all the way to the left of the group, for example.

Besides having limited acuity and color discrimination, peripheral vision is also subject to crowding, in which a target in the periphery flanked by other objects becomes difficult to recognize.

All of this makes the result of the first paper I covered interesting and maybe a bit of a puzzle: Stronger players look more at empty squares than weaker players, which means that they aren't looking right at the pieces! Instead, they are getting a peripheral view of those pieces with all of the limitations I just described.

Except...what if those aren't such strong limitations for those stronger players and that's why they're able to look at a board this way? That's the motivation for this second paper I want to talk about, which examines this idea in two separate experiments. I'm actually going to take these in the opposite order from how they are presented in the paper, mostly to give us a chance to remember what kinds of things we can do with eyetracking data via the results of their second experiment.

Looking for checks on a small board

In the previous paper we talked about in this series, the researchers asked participants to look at the board and make a decision about the best possible move. In this experiment, we'll do something different in terms of task, but use the same kinds of descriptors of eye movements to compare experts to intermediate players and novices. Here, rather than look at a chessboard and come up with the best move, participants were asked to look at a board that was only 3x3 squares and contained just a handful of pieces. Their job? To look at this mini-board and answer a simple question: Is the Black King in check? While participants make this determination, their eye movements will be recorded so we can see where they decide to look so they can apprehend the status of the Black King. Besides varying the skill level of the players, the researchers also varied the familiarity of the configurations that participants were asked to evaluate by using both standard symbols for the pieces or letters in place of these symbols. If you've ever tried playing with a weird variety set of pieces here on Lichess, you may have experienced what the researchers were hoping to induce: Quickly understanding the arrangement of pieces might depend a lot on those pieces looking the way you're most used to! Below, you can see what this manipulation looked like along with some simple scanpaths showing off eye movement trajectories with the duration of each fixation added to the diagram.

Figure 2 from Reingold et al. (2001) showing

There are a number of straightforward behavioral results from this experiment that go about the way you think they should: Everybody is pretty good at the task, but the experts (98.7% correct) are significantly better than the intermediate players (97.4% correct) and the novices (96.3%). The experts are also faster than the intermediate players, clocking in at 861ms on average compared to the intermediates' 1087ms and the novices' 1207ms. Manipulating the appearance of the pieces did turn out to matter, too, insofar as it made everyone slower to respond: Average response latencies were about 1036ms for the symbols, but around 1145 for the letters. So far, nothing terribly exciting, but also some good confirmation that key manipulations of playing strength and appearance familiarity are working as intended. The question now is, what about the eye movements?

There turn out to be a number of neat things to see here which I need to walk you through a little carefully so you're not overwhelmed by the data. Remember, eyetracking data is very dense (hundreds of measurements a second potentially) and there is always therefore a need to summarize that large dataset so you can draw conclusions. This is part of why using a mini-board task like this one is helpful: It constrains the task to a small collection of meaningful regions (empty squares and pieces) which can then serve as potential areas over which to condense your eye movement data into handy summaries. To give you an idea of what happened in this task before I talk about the key outcomes, let's take a look at the authors' scattergrams of eye movements across groups (experts, intermediates, and novices).

Figure 4 from Reingold et al. (2001). These are scattergrams of the places where participants in the three player groups made fixations during the task, both in a more or less unedited way (top row) and then with the initial gaze position removed (middle row) and finally only focusing on a key position without initial gaze (bottom row).

I told you it was a lot! We can work our way through this figure from top-to-bottom, however, describing the neat features of the data was we go. Take a look at that top row and I bet there is a conspicuous feature of the scattergram that's apparent to you in all three groups: There is a big old blob right in the middle. You know why? Each trial started with the participants fixating right in the middle of the screen, so this is their initial gaze position - the place where there eyes started. If you look closely at that blob, you might see that it looks darker in the Expert group (rightmost graph) than in the other two, and the Experts also appear to have less well-defined blobs out in the other squares. That visible pattern in the data turns out to be evident in the statistics as well and we can use a very crude descriptor to capture it: On about 16% of trials, Experts didn't even move their eyes! Yup, they just kept 'em glued to the middle of the board and still outdid both of the other two groups on the task for accuracy and speed. Intermediates only kept their eyes still about 3% of the time and Novices only did so about 2% of the time. Already it sure looks like we're seeing something similar to the previous paper we covered: Experts just don't look at the pieces as much as weaker players do, but still get as much (or more) out of the image. We can confirm that by removing the initial fixations (middle and bottom row of the above figure) and looking at what's left to see where players of varying strength look - right at the pieces, or in the spaces between? Again, the way the scattergrams look tells a story that's supported by the stats in this case: Expert players make more fixations on empty squares than both intermediates and novices.

There's even more we can say about this. Besides the fact that Experts look at pieces less than weaker players, they also make fewer fixations overall, which is consistent with this idea that they just aren't moving their eyes around so much. What's neat though, is that this turns out to depend on the familiarity of the board appearance! Remember that weird "Letter" condition up above? It turns out that both Experts and Intermediates are affected by it - making more fixations and looking more often at pieces - but Novices aren't. You can see this interaction effect in both panels of the figure below - eye movements differ as a function of playing strength, but that effect is modulated by how familiar you are with the way the board looks.

Figure 5 from Reingold et al. (2001) - Stronger players make fewer fixations and look less at pieces than weaker players, but changing the board to use letters instead of symbols increases both outcomes (as long as you're not a Novice).

Looking for changes instead of checks - Flicker and Visual Span

Alright, so this first experiment kind of re-established something that we thought we knew from the first paper: Strong players don't bother looking right at pieces so much, and we can see that by looking at distributions of their eye movements across image space and counting fixations. But why? Isn't this a terrible idea if peripheral vision is so bad? Then again, if peripheral vision is so bad, how are the Expert players doing so well at this check detection task when they aren't looking right at the pieces? Now here comes the really cool idea: Maybe Expert players have a larger "Visual Span" than weaker players. To put it another way, maybe Expert players are capable of seeing more visual structure accurately over a larger portion of peripheral vision than weaker players! That is a really neat idea that immediately raises another question: If they can in fact do this while playing chess, can they also do this in other situations? Do Expert players just end up with peripheral vision that can transcend the typical limits imposed by acuity and crowding? That is what we're going to try and find out in this second experiment (which is actually reported first in the paper, mind you!) using a wonderful paradigm from visual cognition research: The flicker paradigm.

Sadly, I don't seem to be able to show off this effect with animated GIF in the Lichess blog interface, so we'll have to make do with what we have. In the flicker paradigm (sometimes referred to as a change detection task or a change blindness assessment), the experiment presents an observer with two images of the same scene that will alternate on-screen with a blank interval in between them. You may see the first image for a quarter of a second, then a blank screen for a bit, then you get the second image for a quarter of second, then the blank, and then back to the first image...and so on and so on. Your task? The experimenter has made some change to the image between image 1 and image 2 - can you find it? Check out the link below and see how you do trying to find the change as quickly as you can. My guess is that it will take at least a little work.

https://upload.wikimedia.org/wikipedia/commons/7/7f/Change_blindness_illusion.gif

<Scroll down when you're ready for the reveal if you need it>

.
.
.
.
.
.
.
.
.
In the image below, it's easy to see in the side-by-side comparison what is different: An entire tree branch has gone missing! Believe it or not, that same kind of large change is often quite difficult to notice under the flickering conditions I just described above with participants taking many, many, alternations to find it.

Even a large change like the absence of the tree branch in the second image can go undetected in a standard flicker paradigm task. Globe_and_high_court.jpg: created by jjronGlobe_and_high_court_fix.jpg: created by jjron, edited by Fir0002derivative work: WikiCantona, CC BY-SA 3.0 <http://creativecommons.org/licenses/by-sa/3.0/>, via Wikimedia Commons

If you really want to annoy your friends, there is one I find particularly fiendish available here: https://sites.socsci.uci.edu/~ddhoff/cb.html - I can post a spoiler in the forum if you want, but give it at least a little while!

Why does this matter for understanding how Expert players see the board? Because we can use the flicker paradigm in a chess position and combine it with eyetracking to measure how much visual area players can meaningfully perceive at one time! What Reingold et al. did in this experiment was present their participants with a chessboard on each trial and asked them to find a change between two images of the same board via the flicker paradigm. If you look at the top row of the figure below, you can see the kind of change they used: A White Rook on f4 changes to a pawn. This kind of thing should be a little tricky to notice based on what we know about how the flicker paradigm works.

Their next step in this experiment is the really cool part, though. They use eyetracking in this study, but for a different purpose than we've seen in our previous discussions. Here, they measure where their participants are looking so they can limit what they allow them to see with something called a gaze-contingent window. What this means is that participants only get to see a spotlight of sorts centered on the place they are looking - the rest of the board is not even put on the screen, so peripheral vision can't do a thing! To see what this was like, you can take a look at the circles superimposed on the board configurations below - these give you an idea of what was visible in the "spotlight" as a player moved their eyes around the screen. If you let your eyes move up to the upper-right corner, those dark spots would be pieces you'd get to see, but the rest of the board would be in darkness.

Why do such a thing? The use of this spotlight (or gaze-contingent window) gives us an important new descriptor of performance: How small can we make the spotlight before it starts to make you worse at the task? If you were really good at using your peripheral vision to see the board accurately, reducing the size of that spotlight even a little might make you worse at this change detection task - you were trying to look at a lot of the board at once, but I cut you off by making the spotlight too small? On the other hand, if you weren't using peripheral vision much at all (like we suspect is the case for our novice players) a small spotlight won't be much worse than a big spotlight because you were just going to move your eyes around anyway. Such a difference in the size of the spotlight window that impairs performance would support the idea that Expert players do have a larger visual span than Novices. But say the experts do have this larger span - does it apply to everything they see or just to this task? Reingold et al., tested this with a classic control - try the same change detection task with randomly shuffled chess pieces on a board and see if the same thing happens. If things turn out the same, then it sure starts to look like Experts have a more general visual capability. If not, then what they have instead is a chess-specific perceptual advantage.

Figure 1 from Reingold et al. (2001) showing off the basic flicker paradigm design and the gaze-contingent window imposed on the display.

You can see their results in the final bar graph below. For my money, I think Figure B really has what you're looking for: Experts' visual span in the change detection task (remember - NOT a best-move or check-detection task!) is more than double that of the Novice and the Intermediate players! Experts are clearly able to integrate visual information across a large chunk of visual space in a way that weaker players can't. That difference vanishes as soon as we start talking about random piece configurations instead of those arising from normal game play, however! That's a fantastic statistical interaction, but honestly a little disappointing for this visual cognition research - I was sort of hoping for a broader change in visual perception. I'll take what I can get, however, and this is really cool. As in my blog post about problem solving, I like seeing researchers use tasks that aren't specifically about playing chess to characterize differences between players of different strength. While it often leads us back to the hypothesis that we're really talking about very domain-specific changes in perception and cognition, these kinds of tasks help establish that we are talking about uniquely perceptual or cognitive mechanisms in some cases.

Figure 3 from Reingold et al., (2001) The visual span of experts is much larger than that of weaker players.

There is still an important question here, though: What does this visual span actually mean? The researchers advocate strongly for De Groot's idea about "perceptual chunking," or an acquired ability to quickly encode groups of pieces distributed across the board as holistic units that end up immune to effects like crowding. That certainly hangs together with some important results about the impact of perceptual organization on visual crowding in general (i.e., not just with chess pieces), but to my mind there is still an alternative account that's worth considering: Is this about stored chunks that can be quickly encoded, or is it about some kind of application of prior expectation on piece identity given location? That is, are strong players really accessing holistic arrangements of multiple pieces, or better able to say "That blurry thing ought to be a rook" based on their extensive experience seeing pieces in different parts of the board. If those sound kind of the same to you, well, maybe they are....I'm still thinking my way through some of these ideas too, but it feels to me like there's room for some alternate accounts of the mechanisms that yielded this data.

For now, however, we'll leave this two-part series on eyetracking here with a neat use of the methodology to characterize perceptual abilities in strong vs. weaker players in what I think is a particularly revealing way. I'm still deciding what topics to tackle next in this series, which means two things: (1) Suggestions are very welcome - send papers, questions, or ideas for future posts in the forum! (2) You should stay tuned to see what we're talking about next. Hope you enjoyed this and see you in the next one!

Support Science of Chess posts!

Thanks for reading! If you're enjoying these Science of Chess posts and would like to send a small donation my way ($1-$5), you can visit my Ko-fi page here: https://ko-fi.com/bjbalas - Never expected, but always appreciated!

References

Charness, N., Reingold, E. M., Pomplun, M., & Stampe, D. M. (2001a). The perceptual aspect of skilled performance in chess: evidence from eye movements. Memory & cognition, 29(8), 1146–1152. https://doi.org/10.3758/bf03206384

Lettvin, J.Y. (1976). On seeing sidelong. The Sciences 16, 10–20.

Reingold, E. M., Charness, N., Pomplun, M., & Stampe, D. M. (2001b). Visual span in expert chess players: evidence from eye movements. Psychological science, 12(1), 48–55. https://doi.org/10.1111/1467-9280.00309

Yarbus A. L. (1967). Eye movements and vision. New York: Plenum.