Why is the analysis board sometimes an authoritarian cheapskate?

Hmm so it is cost after all. Thank you for the informed reply.

When you say secondary memory is too slow, do you mean it would take hours or even days to generate a more in depth game analysis? If that were the case, I'm just wondering.... Suppose some of us were willing to wait that long occasionally for some analyses. I wonder if it would be possible to have a two-tiered system, whereby one could request a quick analysis as at present, with its current lacunae, or a full analysis that would take long (how long are we talking about?) to generate, using secondary memory. Or would that be too expensive to set up?
Lichess has done a wonderful job of encouraging serious study of the game. Now that some serious students are here, I presume we would be prepared to wait hours or days for serious analyses of some of our games.

www.quora.com/Is-the-speed-of-SSD-and-RAM-the-same

Even a decent SSD is anywhere from 20 to 1000 times slower than RAM. So if you took 3 seconds to get to a respectable depth 20, you'd have to wait a full minute, or maybe 10, on SSD. For the equivalent of 3 seconds with RAM. So you would be extending a decent-quality couple-minute analysis into an overnight analysis for 0 benefit in terms of strength.

Plus, if you really want to analyze deeply as you say, it would take much more storage capability. I'm too lazy to calculate, but I wouldn't be surprised if it's in the hundreds of gigabytes for a single position. (Stockfish 10 fills the 1 GB I give it almost every time I get past depth 30 in local Arena.) Considering a 500 gb SSD costs ~80 dollars, there's no way this is possibly feasible.

Plus, SSDs work differently compared to RAM. You would probably have to write a module that manages those issues for many many more man-hours.

If you switch to secondary memory and try to store everything, you would be taking up countless dollars for something that is currently done at almost no cost.

You could always explore the potential lines by actually making the moves. I'm no good at chess, but my guess is that's also a better way to analyze than just looking at a move list.

@nayf Serious students of the game who want advanced depth of analysis should get a personal copy of Stockfish to run on their own machines (it's free !). Then you can leave the analysis running for days on any position, ask for arbitrary depth, and even tune the engine parameters to your liking (e.g. contempt value).

Why are positions discarded, yes, it's a matter of limited capacity, RAM is limited. See the blog post about endgame tablebases (lichess.org/blog/W3WeMyQAACQAdfAL/7-piece-syzygy-tablebases-are-complete) for some discussion along the lines of "storing chess trees is expensive". ("The number of unique legal 7-piece positions is 423,836,835,667,331. Syzygy tablebases store all their information in 18.4 TB, so at around 0.35 bits per position.")

Now, the problem of preferred variations not being reported to a long enough depth isn't a matter of limited analysis, or memory capacity, more like some programmers (Stockfish developers, not related to lichess) not trying very hard to keeping that feature alive, and breaking it while doing other things. groups.google.com/forum/?fromgroups=#!searchin/fishcooking/pv$20from$20hashtable%7Csort:relevance/fishcooking/jFKq-aKJOu0/qsUY6wQK4mIJ discusses the crux of the issue, very technically. There apparently is a tradeoff between playing the very best moves, doing some more work, and remembering a long best line.

My understanding (@Sidonia-ChessEngine have an opinion ?) of the issue is that sometimes, SF looks into a line, and it's okay (say 11. e4, +0.2), and then it will look into another, and it looks better (say 11. Nf3, +0.3). So SF will spend all its capacity exploring 11. Nf3, for a long time, at advanced depths, and forget most of its analysis of e4. But then, it turns out, at depth 28, that Nf3 doesn't work (it's only +0.1 after all). Now what : SF knows what the very best line at depth 28 after 11. Nf3 is, and it *could* show you that (but it's only +0.1), or it remembers that e4 is +0.2, but it forgot why, since it wasn't the main line. So right now, it tells you the best is 11. e4, but it forgot why, and hence the displayed prefered variation is short.

And apparently this happens somewhat often and lasts for long periods of time, so when lichess asks "analyse this position for 5s please", oftentimes when the 5s is over, SF is stuck in that state and can only report so much.

That's a reasonable explanation of what I think might be the cause, yes.

@Sidonia-ChessEngine

In other words, it's not saying as in my OP, "best move Nf6, it's obvious dummy", but rather "best move Nf6, but honestly I forget why! So please don't ask."

Thanks very much for these informed comments, exactly what I was hoping for to satisfy my intellectual curiosity, albeit at the disappointment of learning that nothing can be done about it. I have some further questions but am pouring through the linked discussions.

@lecw
I do have stockfish on my phone but don't know how to generate a lichess type of analysis (with recommendations of best lines).

Download stockfish on your computer (stockfishchess.org/) and use a gui like arena (http://www.playwitharena.com/?Download) to analyze your games.

You'll be able to run much, much faster and deeper analyses than what lichess offers. Running stockfish natively gives roughly a 2 times speed boost vs. lichess browser stockfish for me.

@farmersrice Thanks for this link!! What to you think is the best to download because there are a lot of options

@E101 download the latest arena (should be at the top, 3.51) and stockfish for whatever operating system you have.

@farmersrice
@lecw
@Sidonia-ChessEngine

Ah, it's got to be on the laptop, ok.

While I have you knowledgeable guys here, may I ask about a different topic? I've been curious about how the program generates puzzles. My original conjecture was that it looks for double blunders - blunder of commission setting up the puzzle vulnerability, and blunder of omission, missing the opportunity by the opposing player. However, while many of the puzzles do turn out to be such, I believe I've seen some where the player in the game got it right (solved the puzzle in the real game, as it were). Yet the program couldn't select just single blunders, since some of those are mere ridiculous hangings of pieces and so forth. I presume the puzzles are entirely computer generated, not human edited, because of the sheer volume. Any ideas?

I don't have any good knowledge on that, but you can always read the source code.

I found this: lichess.org/blog/U4sjakQAAEAAhH9d/how-training-puzzles-are-generated