Deathmatch: Leela vs. Stockfish

6 games are enough to give you a flavour on decent hardware.

@mmahmood31 It's just decent hardware for both engines. For a more serious match there would probably have to be a fixed budget and much more games.

@biscuitfiend It will be the official Stockfish 10, not the multi-variant fork.

@Luetage Good point. Fixed.

Inb4 windows 10 updates and stops Leela. Best to put Leela on Ubuntu or some other Linux distro too.

Let’s go leela

@kenzaburo 6 games are not enough to say which one is stronger. If they are reasonably close in skill even 20 games are not enough. However determining which one is stronger is the job of chess engine rating lists, this match clearly is for entertainment. (which is fine of course)

@JohnJPershing That's not really true, in principle a NN based engine should perform better in long time controls because the search algorithm can take more time to find complicated tactics and the hopefully better strategic eval will favor it in long time controls once it doesn't blunder anymore.
Leela in specific might be stronger in short time controls (though lets first see how this TCEC season goes, Leela is destroying Div2 so far, lets see if she goes all the way to the highest division) however that is probably because its aggressive play style might just work better in short tcs.

@Vetinari_Computer Of course more time improve their performance, but a stockfish would gain more advantage from extra time than a neural net would. Neural nets have distinct diminishing returns on computing power compared to traditional engines. So LCZero vs Stockfish at a fast control would be more advantaged towards LCZero than if they played on a very long time control. Long time controls will improve both engines play, but would improve Stockfish play more.

@JohnJPershing Do you have any source for the claim "Neural nets have distinct diminishing returns on computing power compared to traditional engines." ? I would say the opposite is true, SF very much runs into diminishing returns at very long time control and the Alpha0 paper says the same. The reason SF doesn't gain as much with long tc is because it often already sees the tactics at short tc. (very oversimplified)

More specifically, Minimax based searches are good at distinguishing moves with very different evals (e.g. tactical shots) however they are not so great at telling which of two very similar evaluated moves is better. The UCT that Alpha0 and Leela use is much better at that however they can struggle with finding deep tactics both because they simply calculate less moves and because it can take a while for a difference in eval due to a tactic carry through all the way to the root. (which in minimax based searches happens instantly)
Obviously in practice it's not that clear cut because SF isn't just some alpha-beta engine, it prunes a lot i.e. has a very selective search which can help with the comparing similar moves issue. But it still doesn't solve that nearly as good as UCT does.

Another point, as blatant as it sounds, is the difference in NPS. The difference between calculating 1 vs 1000 nodes at super short tc is (at least in minimax vs minimax) much larger than for 100k vs 100M. However there it is up for debate whether that is also true with a NN based engine involved.

@Vetinari_Computer I'd say the largest experience is that of seeing LCZero in blitz, rapid, and longer time controls against other engines in various competitions, such as the TCEC. Against the same engines, LCZero scores significantly higher on faster games in these matches.

@JohnJPershing So what did you see in TCEC? I see that in Div2 it currently murders the field with 13.5/17, having not even given up a single half point with White. It did not perform as well in the previous season because that still had a much weaker version. But that is beside the point, even if Leela performs better in shorter tc that still doesn't mean that every NN based engine does. (as mentioned, just look in the Alpha0 paper; they even state that at fast tc Stockfish still beats Alpha0 but at long tc it doesn't)