A rant about puzzles and a feature request

In the puzzle https://lichess.org/training/kzzgK, the right moves seem quite intuitive, but, unless I missed something, to be able to check that the intuitive moves are actually sound, you need to spot mate in 11. Where the puzzle stops you didn't gain material, and you didn't checkmate the opponent.

Yet because you can just play the intuitive moves, it has a relatively low rating, which in my opinion does not accurately reflect its true difficulty if you try to really understand what's going on rather than just try to find the first few engine moves. There is a fair number of puzzles in a similar situation, and I think Lichess should do something about it.

One option would be to not end the puzzle until it is clear what advantage you gained. Even if that would possibly allow for multiple solutions, which I personally don't see an issue with.

Another option would be to remove from the database those puzzles where it is not completely clear at the end of the puzzle what advantages you gained. I believe this could be automated : if, when the puzzle stops, the player didn't either checkmate or gain material, then the puzzle is eliminated (there might be other possibilities I didn't think of, but hopefully they can be handled programmatically as well).

@polylogarithmique said in #1:

In the puzzle lichess.org/training/kzzgK, the right moves seem quite intuitive, but, unless I missed something, to be able to check that the intuitive moves are actually sound, you need to spot mate in 11. Where the puzzle stops you didn't gain material, and you didn't checkmate the opponent.

Yet because you can just play the intuitive moves, it has a relatively low rating, which in my opinion does not accurately reflect its true difficulty if you try to really understand what's going on rather than just try to find the first few engine moves. There is a fair number of puzzles in a similar situation, and I think Lichess should do something about it.

One option would be to not end the puzzle until it is clear what advantage you gained. Even if that would possibly allow for multiple solutions, which I personally don't see an issue with.

Another option would be to remove from the database those puzzles where it is not completely clear at the end of the puzzle what advantages you gained. I believe this could be automated : if, when the puzzle stops, the player didn't either checkmate or gain material, then the puzzle is eliminated (there might be other possibilities I didn't think of, but hopefully they can be handled programmatically as well).

My understanding is that the point of puzzles is to be instructive. The point of this particular puzzle, as it appears to me, [SPOILER ALERT] is that you COULD take the rook* but you're much better off checking and continuing to deny the King legal moves.

The reason puzzles do not always progress to mate? Well, it could be that it's annoying to fail a puzzle because you played a natural move that leads to mate in four (an approach which will serve you well in actual games, time permitting) but missed a less obvious move that is mate in three. Or it could be that there are several mates, and the puzzle would need to accept all of them which 1) wouldn't be very instructive and 2) would take time (a consideration if you are coding the puzzle manually rather than relying on a bot).

As for rating: this is determined by people attempting the puzzle. Very often I see puzzles rated, say, 1700 where the solution is less intuitive than a 2100 puzzle and I presume the latter is higher rated either because of an immediately tempting yet suboptimal move or because a lot of people do puzzles as though they were doing puzzle storm/racer. Perhaps the most important thing to remember when doing puzzles is that, unlike a game, you can take your time or even just give up and it won't cost you.

*thereby winning material but missing a forced mate that would win the game. Never forget that all the material in the world is useless if the position is wrong.

@Marlonc said in #2:

The point of this particular puzzle, as it appears to me, [SPOILER ALERT] is that you COULD take the rook*
*thereby winning material but missing a forced mate that would win the game. Never forget that all the material in the world is useless if the position is wrong.
Unless I misunderstood what you are talking about, if you take the rook with your rook, the knight takes back.

I mostly agree with the rest of your points though. But I just think it would be better if the rating of the puzzle would accurately reflects its true difficulty, as puzzles shown to you would correspond to your level more accurately. The modifications I suggested are aiming at this purpose.

Ok but as I said, ratings for puzzles (as for players) are an epiphenomenon of others' success. If you feel a puzzle is rated too low or high it may be an indication of an imbalance in the strength of your game which you can address via the puzzle dashboard.

Hum that isn't completely true. Sure, different people have different sets of skills, but puzzle to some extent have an "objective difficulty level". By which I mean not only how difficult it is to come up with the right move, but also and more importantly how difficult it is to understand why the right move is the right move. Or how difficult it is to actually learn something from this puzzle, if you will.

Having puzzle ratings matching at least roughly their objective difficulty allows lichess to recommend you puzzles that are adapted to your level, and from which you will actually learn something.
If a puzzle's rating is way to high for its difficulty, then it will be shown to advanced players who will solve it easily and not learn anything from it, thereby wasting their time.
If a puzzle's rating is way to low for its difficulty, then it will be shown to players that don't have the ability to solve it, thereby wasting their time, and potentially not even learning something from the solution if it's a tricky one.

Now the fact that a puzzle's rating is an epiphenomenon of other people's success does NOT ensure that the rating actually matches the objective difficultly, as illustrated by the case we both mentioned of a puzzle in which the right move is immediately tempting but it's not actually obvious why it is the right move if you think of it. Unfortunately a lot of people will actually play the tempting move without thinking more, thus affecting negatively the puzzle's rating independently of its true difficulty.

If I may say so without causing offence, I regard the notion of a puzzle having and objective difficulty level much as I regard the concept of a player having a true rating. If you play better games you will, in time, have a better rating. The contrapositive is not true though, tempting as it is to believe otherwise.

You might get a puzzle rated 100 points above your rating that has been played 5000 times, or another that has been played 500 times. The rating of the latter may rise or fall depending on how you & other people do with it. But it could still be easier/harder for you because of your playing style.

Incidentally, I do remember that when lichess overhauled its puzzles my puzzle rating ballooned for a while. One of the features that was added was a drop-down menu allowing one to toggle difficulty. If you try harder or hardest, I am confident that you will see the difference.

[UPDATE: I thought that taking the rook would be where most people had gone wrong (and perhaps it was) but the player with the black pieces in the actual game played Rf6 before Bg3+, hanging a piece. Though the principle of don't hang your pieces is so elementary I confess that it would surprise me if that were the only lesson illustrated by a puzzle that was rated over 1700.]

@Marlonc said in #6:

If I may say so without causing offence, I regard the notion of a puzzle having and objective difficulty level much as I regard the concept of a player having a true rating. If you play better games you will, in time, have a better rating. The contrapositive is not true though, tempting as it is to believe otherwise.
I mean. Rating is linea thermometer. If it's hot outside, the number will increase. But if youtinker your thermometer to artificially make it display higher numbers, it won't make the temperature outside be higher.
And yes, the outside temperature is subject to change with time.
Yet, that doesn't mean the the temperature outside isn't something real, or that thermometers are useless. It is indeed very useful for plenty lf reasons to be able to measure the outside temperature accurately at a given time.

I did agree that different players have different set of skills. And yes, these skills might evolve over time. That doesn't mean that, at a given time, some players aren't objectively better than others. You can try as hard as you want to play better games, I don't think you (or me) will ever be able to defeat Magnus.

So much as a thermometer is a useful tool to know what to wear and to make accurate predictions, rating is a tool that can be use to help players learning better, by showing them, at a given time, puzzles that they are likely to learn something from. And my claim is that the way it is done currently is sub-optimal.

UPDATE: I thought that taking the rook would be where most people had gone wrong (and perhaps it was) but the player with the black pieces in the actual game played Rf6 before Bg3+, hanging a piece. Though the principle of don't hang your pieces is so elementary I confess that it would surprise me if that were the only lesson illustrated by a puzzle that was rated over 1700.
Once again, after Rxf1 white has Nxf1, and black only exchanged rooks (and incidentally the engine says +3).

Once again, after Rxf1 white has Nxf1, and black only exchanged rooks.

Yes...?

I'm not saying it's a good move, I'm saying it may have seemed natural at first glance which may plausibly explain why the puzzle is (as you seem to be arguing) overrated.

No, I am arguing it is under-rated actually. Because to solve it you need to spot a mate in 11.

@polylogarithmique said in #9:

No, I am arguing it is under-rated actually. Because to solve it you need to spot a mate in 11.

I don't think that's true at all. Actually, I think that our exchange demonstrates very clearly that that is not true: all you have to do is harass the king, where the alternatives are hanging a piece in the first instance and exchanging for no benefit in the second.

FWIW one thing I have learned to look for in lichess puzzles is a situation where you would have mate apart from the opponent's Q being able to block, and the solution therefore is to settle for exchanging the Queen for e.g. a Rook.