@DoctorFuu said in #10:
> a few years ago I concluded that using the rating delta of the move was the best thing, because it corrected for stronger players choosing a bad line to put the lower rated opponent out of book
What is a "rating delta" of "a move"?
If you mean "Lichess rating gain/loss" of "a game", there is a problem - the rating difference of Glicko/Glicko2 system is related with the uncertainity of a player himself, timed by the performance difference in a game. That would be unfair when some newcomers may gain +162 rating for his first game.
Yes, that's what I mean.
This is part of the data quality issues I was mentioning above. Realistically, because glicko2 converges fast I would expect this problem to be negligible. Like if a player is rated high enough to be above 2k rating, that player will probably play dozens if not hundreds of games after his calibration (low elo percentages are close to meaningless anyways). You also have all the players who calibrated at low elo and then got better over time, they will not introduce this variance in the games. The vast majority of games are not with player undergoing calibration. I would be extremely surprised if this had significatn effects overall. Also, if you have a small sample of games from a position you can also just have a glance at them to see if there are any of these surprising datapoints, and dismiss them if needed (I was using a manual process in an opening explorer, so I was not only looking at the moves, but also having a look at the games played from that move anyways, as sanity checks).
As a statistician, the question often is not whether there are problems with my models, but whether these problems do materialize in the decision that will be taken out of the analysis. I did not perform a sensitivity analysis on that so I can't be 100%, but I would be extremely surprised if that caused a problem.
Also, I was not using the lichess database but a database from irl / master games. So this issue of calibration period wasn't even there. But you're right, that's a point that needed to be adressed.
@n1000 said in #9:
> Yeah, we all know about the branching factor but it's still startling how quickly your sample shrinks when you want to compare positions even just a few moves in.
Yes, I agree that the sample does shrink significantly from its original form, but it is still massive, at least over 10^15