Input for a chess puzzle experiment needed

Hi all,

I'm designing a hypothetical psychology experiment for a PhD grant, and I could use some input from you guys.

So, the general research question has to do with how we process verbal information versus numerical information when making decisions or evaluating a situation. There has been quite a lot of research on this topic when it comes to how we assess nutrition values when buying groceries and how we evaluate risks of side-effects in medicine (for instance depending on if the label says "very rare" or "0.001%").

My idea is to look into how verbal vs. numerical information affects you cognitively when evaluating a chess position. But how would be a reasonable way of doing this?

One way would be to conduct an experiment where the participants are assigned to solving a set of chess puzzles. One group gets verbal clues that helps them with each puzzle; one group gets numerical clues; and the control group doesn't get any clues at all.

The question is - what would be a clever way of making verbal and numerical clues? The verbal and numerical clues need to be fairly equivalent, i.e conveying the same information, for instance "a move with the rook is 80% likely to be the winning move" versus "a move with the rook is very likely to be the winning move". This would, however, be a bad clue since if there is a single winning move it will always be 100% likely to be the winning move. Even with the clues the puzzle should still require some thinking, and they should still be difficult enough for the participants to fail now and then.

A long and complicated question perhaps, but do you have any thoughts? All suggestions are welcome!

There are two kinds of chess players: verbal and visual. A verbal chess player mentally speaks to himself while thinking about his move. He cannot talk while thinking. A visual chess player just looks at the position while thinking and can speak while thinking.

This is not only a chess thing. Feynman described a little experiment. A person had to mentally keep track of time and call 'stop' after 1 minute. Some people could do it while at the same time performing visual tasks as they silently counted to 60 to keep track of time. Some people could do it while at the same time performing auditive tasks as they imagined a band with numbers 1 to 60 moving before them to keep track of time.

@Dondubbhain "what would be a clever way of making verbal and numerical clues?" 🤔

I think generally the best clues so that the puzzle "should still require some thinking" are verbal clues ("there's a pin you can exploit", "has your opponent any undefended piece?", "how many checks do you have?", "what's your opponent's threat?", etc). I find it hard to reformulate any of them in a numerical way...

However in chess we process numerical information often, and there's especially one kind of numerical information that has an immediate "verbal" correlation: engine evaluation. So maybe not for puzzles clues but for other kind of information we process would it be interesting to compare the difference between how we understand +2.5 versus +- for instance.

You would need to design a different kind of experiment (not sure how exactly). For example, if I'm given a position and the information that move 1 leads to a slight advantage and move 2 leads to a comfortable advantage, would I use that information the same way as if I'm told move 1 is +1.5 and move 2 is +1.9? The key is what is "using" the information in the experiment, which I'm not sure about (I think chess players usually "use" this kind of information to decide whether an opening line is interesting/playable, to decide "who's better", etc, so basically to make different types of decisions that could be maybe evaluated in an experiment).

bump

What about "80% of our tested candidates think the rook is the piece to move".
I don't think that's really a good approach as we all see (or even "get") stats in different ways but I guess it's your PhD not mine :)

@pepellou That's an interesting approach! I haven't really thought about it before, but the engine evaluation really is a perfect numerical way of describing a chess position. I still need to work out the actual design of an experiment, but perhaps it would be possible to set up a form of chess puzzle that makes use of either numerical information (such as +2.0) or verbal information (White is slightly better).
Thanks a lot! I'll give this some more thought.

And if anyone has any more suggestions, I'm all ears.

If external knowledge or training can improve the performance in your experiment, is it truly randomized?

How are you going to divide the people into groups so that you're really assessing verbal vs numeric processing and not simply measuring which group has been exposed to more chess patterns?

I think you need to drop chess from your experiment. Otherwise you don't know if you have a random population in each group.

How would you assess the impact of verbal vs. numeric impact if one group contains only people who have no chess knowledge and the other group contains only grandmasters?

"Well, I'll have the same person perform the same number of puzzles with each type of prompt." But the puzzles themselves won't be the same. I am better at some types of chess puzzles than others, regardless of the type of clue given.

@ericmsd As long as you measure the rating of each participant and have puzzles that adapt according to your rating level, this isn't a problem. You can account for rating differences in the statistical analysis.

But yes, the amount of chess experience you have will obviously have an impact on your performance. This impact is then separated from the impact of the variables you're looking at in the analysis.

@Dondubbhain

How do you measure the rating of each participant? Do you have them do puzzles of various categories first? They probably have to do quite a few in order to get an accurate rating.

When I look here, I'm much stronger in solving puzzles with a sacrifice theme than quiet move. My quiet move just started improving recently because I've been focusing on that. Now, I see that I need to work on the hanging pieces theme.

lichess.org/training/dashboard/30/dashboard

As I am exposed to more patterns, I get stronger in those types of puzzles. It would be inaccurate to assume my blitz/rapid/classic rating corresponds to my puzzle rating.

Obviously, this wouldn't be impossible to do. Otherwise, multivariate analysis wouldn't be a field of study. :) It seems like you would be making your study harder than it would need to be just to include a chess theme.

@Dondubbhain

I hope my comments above are useful. I'm not trying to be argumentative. That just comes naturally. :) I'm hoping that by poking some holes here and there, I can help you.