@justaz There is a major problem with the massive machine learning data crunching Ordo methodology you are suggesting but it isn't your fault at all in how you have gone about doing this project. The issue is that there is an unquestioning assumption present in the whole argument that the Elo curve model and bell shaped normal or logistic distribution models for player ratings are correct and suitably statistically consistent with reality, but in actual fact they are not! You are not alone in thinking this way and it's an easy trap to fall into, there is a lot of bureaucratic resistance to changing the rating systems because it's a lot of work and most people would prefer to ignore the obvious signs that the current outdated system doesn't work, they are stuck in a particular paradigm for thinking quantitatively about chess skill without applying the scientific method to test their beliefs. I am in the process of working on an improvement on the status quo of chess rating systems and imo we should be basing ratings on computer analysis of player performance for whole games rather than just the outcome. We have the technology now to analyse and quantify how strong moves are within games so why we haven't switched over to using it in ratings is a mystery. I will be making some blog posts about it soon so keep an eye out for it ;)
Here is an excellent blog post from world leading chess statistician giving his statistical data science take on FIDE ELO and his proposed optimisations and refinements:
en.chessbase.com/post/the-sonas-rating-formula-better-than-elo(tl;dr Elo curve predictor for game outcome is absolutely inconsistent with real world data, simple truncated linear model gives better fit. Optimised k factor and different weightings for different time controls for a more accurate overall rating help too).
Seeing your graphs of the distribution of Lichess ratings alongside the fitted Ordo ratings reminded me of my own rather in depth regression analysis project which analysed how well various theoretical models fit to the distribution of ratings on Lichess in an attempt to test personal beliefs about the impact the recent influx of new players has had on the rating distribution. I think you may find the statistical conclusions raise important questions about some side assumptions included in some implications about how chess ratings "should" be distributed
(
colab.research.google.com/drive/1aRXuT1RJaestpPC9L2lo6InG3OT8w5eD?usp=drive_link)
tl;dr: chess ratings are absolutely NOT normally distributed! A logistic distribution fits better which makes sense as this relates to the logistic function used by Elo/Glicko systems to predict game outcome, if higher percentile players beat lower percentile players you would expect the cumulative distribution function to resemble whatever predictor of game outcome you are using. Even so, a simple single peaked 2 parameter model will not fit to the data within the errors and confidence intervals of the data. It appears you need a superposition of 2 such logistic peaked models close together to get a good fit, which supports the hypothesis that the player population has been split into 2 sub categories: Pre and Post "The Queen's Gambit"/Rise of Chess Youtube and the mainstreaming of chess.
Some constructive criticism about your data science methodology; your software engineering skills are clearly great (I assume this is what your background is in?) but you have to work on your statistical and scientific critical thinking in how you conduct and disseminate results. If you want to test beliefs and get closer to the truth you have to use a more formal approach with statistical hypothesis tests and you have to be more self aware of when your statistical arguments are too vague and unsupported, you have to go deeper. For instance you make a tenuous claim at the start appealing to the law of large numbers that as the number of games tend to infinity each player's rating will eventually tend towards some exact true value but this is not the case (at best the player's expected or average rating will tend to an exact value but not the live rating itself). On average the rating adjustments will push you in the right direction but the fluctuations and uncertainties will never die down to an exact limiting value, even IF the true Elo value remains constant there will always be a non-zero minimum amount of noise (my pure math skills aren't good enough to prove from first principles that this is the case, but if you think about it on a deep intuitive mathematical level you can see that it makes more sense than what you were suggesting. The point is whether this question crossed your mind when you made the claim in the first place?). Another example is when you stated figures about guessing the outcome right 58% of the time and then stating that it is a significant "good" result without stating the sample size, without this information the effect size has no context of statistical significance to compare against. Given the number of data points you seem to be using I will give you the benefit of the doubt as a few percentage points probably is significant in this context, but remember we don't know that from your blog! But besides the issue of whether it is persuasive enough to an attentive reader, there is also the danger that you will succumb to your own self serving biases if you are not disciplined with statistics because ofc you WANT your system you have worked hard developing to be effective and work well, so you may be overly selective seeing patterns and significance when it benefits you or your pride in your work (I can relate). This can damage the objective validity of your own work so be careful when drawing conclusions without first laying out the statistical success/fail criteria of tests BEFORE you collect the data. This is what makes it so easy to lie with statistics when someone goes about it the wrong way, intentionally or unintentionally.
