Do Puzzles Matter? - A Data Analysis

A common recommendation for improving players is to “train tactics.” In practice, this means solving chess puzzles. However, it must first be asked, is this actually useful? Of course, practicing tactics should make you better at finding tactics within games, but what if the puzzles that you do never come up in your games? Or perhaps you did compositions that do not emulate realistic scenarios? Maybe simply analyzing your own games is the way to go, then you will study tactics that are most likely to arise from the positions that your openings produce.

On the flip side, you need repetition to drill tactical patterns into your brain. So much of tactics is pattern recognition, which will come much more quickly through dedicated practice versus game review. Additionally, puzzles present more types of tactics. If you don’t know what a pin is, then you will never pin pieces and you will never get a chance to study tactics related to pinned opponent pieces. Puzzles are necessary to broaden your chess understanding as reviewing and playing games is inherently limited in scope.

Of course, you could test out both theories, or we can see if we can glean any useful insights from the data. Thankfully, Lichess freely provides exactly what we need. However, I still had to choose which players to extract the data from, as there is no way to simply obtain the data for every Lichess user. I ended up using all the players who participated in the “2023 Spring Marathon” tournament. I figured this would give me a good range of players who take the game seriously enough to do something like practice tactics occasionally. With over 24,000 participants, it was also a large sample size. I collected data for each participant’s bullet, blitz, and rapid rating and compared it to their puzzle rating. If they didn’t have one of the ratings then they were not included where applicable.

Here are the results:

All the graphs look very similar. In general, a higher puzzle rating does correspond with a higher in-game rating. However, none of the time controls are strongly correlated, with the R2 values only ranging from .71 to .75. I think there are several reasons for this. First, there are a lot of outliers. A large number of Lichess users only do puzzles or only play games, so people who tried a handful of puzzles, got a rating, and then never tried them again skew the graph due to a disproportionately low puzzle rating for their playing strength. I tried excluding users who had drastically different ratings, but that only had a minimal effect on the R2 value, so the upper bound likely falls in the .7-.8 range. Qualitatively, it makes sense that puzzle strength does not have a 1:1 correlation with playing strength since it is only one aspect of a multifaceted game. In fact, perhaps too much focus on puzzles stifles growth, as it would have to be done at the expense of studying openings, endgames, strategy, etc.. Assuming, of course, you don't have unlimited time.

We can also compare the time controls with their best-fit lines:

We see what we might expect - a greater slope for the faster time controls. Therefore, generally, an increase in puzzle rating has a greater impact when you have less time. This makes sense because drilling puzzles should speed up tactical pattern recognition, granting more precious time in blitz and bullet. However, in the upper echelon of rating the lines intersect. I think this is due to several factors. Firstly, the impact of pattern recognition is lower. The puzzles at this difficulty are so complex that they take several minutes to solve. Even if you noticed a similar pattern in a real game, it might not be worth the risk to spend your time calculating the variations. Secondly, players at this level are so advanced that improvement is not as simple as “do an hour of tactics a day.” In the lower ratings, puzzles serve to teach you new concepts, which will have an outsized impact on your play, whereas the most advanced puzzles usually involve practicing old ideas in different and new ways. Finally, a lot of these players simply hit their peak. Humans have an innate chess ceiling and once you are there, no amount of puzzles will push you past that barrier.

Of course, this analysis has several limitations. Namely, we only looked at correlation, not causation. That would have to be done experimentally. This analysis cannot prove that getting better at puzzles will make you better at playing chess; however, we observe that in general people with high puzzle ratings have high in-game ratings. Also, the data set may not have sufficiently represented the totality of the chess player base. All the data was obtained from lichess.org, which, in general, caters to a less casual audience than chess.com. Additionally, all the ratings were pulled from people who participated in a tournament. Although it was a large, accessible tournament, this likely excluded more casual players. There is some advantage to this approach though - casual players who don’t train puzzles may have an actual puzzle rating below their true puzzle rating, which would make puzzles appear less impactful. This is related to another problem - people who have disproportionately high puzzle ratings due to cheating. Although online chess takes substantial steps to prevent cheating, there are no safeguards in place for puzzles because it does not really affect other users. As a result, there might be individuals with puzzle ratings they did not earn, but the magnitude of this problem is uncertain, albeit likely small.

In conclusion, I do not think this analysis revealed anything new about chess improvement but instead confirmed what is commonly recommended. This data obviously does not definitively prove that tactic training will make you better at chess, but it does show that people who are good at chess are usually good at tactics. Puzzles should be a part of your chess training regimen, but not the only factor. Because puzzle ability is only moderately correlated to in-game rating it cannot be the only aspect of chess skill. If you are serious about chess improvement, dedicate time to puzzles, but do not let it be your study.