Analysis of Lichess' cheating detection with Machine Learning (ML) - a mis-use of ML & doesn't work

@Molurus said in #40:
> xkcd.com/386/
>
> And to make sure... this cartoon is sarcasm. It is intended to illustrate how silly you are being.

So you don't think that lies should be exposed and criticized? It doesn't matter where you discover the lie. Well, that says a lot about you. Don't you think so?
Who is being silly now?

@Molurus said in #40:

> Now please, for once, try to be on topic, not on person. No one cares for your ad homs. (Or just be silent, that'll work too.)

Then I would suggest that you stop responding to my posts. You too have started responding to me by giving my posts a thumbs down. Not the other way around. So you are the trigger of the whole thing.
So don't blame me for that! The blame is on your part.

@phobbs5 said in #13:
>
> 1. Moving your mouse off the board does not make the browser window lose focus. This would require actually clicking on another tab or window, which is not happening in Hikaru's case and seems like a clear signal of cheating, especially if the moves which correspond to focus-change are correlated with higher strength.

That depends on your window manager, which in turn may depend on your OS.

On my non-MacOS Unix systems, I've been running a window manager with the setting "FocusFollowsMouse" for decades. Move the mouse to a different window, and that window will get the focus.

@IrwinCaladinResearch said in #21:
> Sorry, I forgot to reply to this one and it is a really good point:
>
>
>
> Disclaimer: I already had a beer, so this may be flawed, but thinking about how to get labels for cheating detection properly is a good thought.
>
> Unless Lichess proves me wrong, I would say the percentage of confessions out of flagged cheaters is negligible. Relying on this to get labels is not going to work.
>
> Here is how I would do it and it's wild - ASK PEOPLE TO CHEAT. No, don't stop reading here and start writing a response to call me crazy, hear me out and let me describe how this could work:
> Lichess could reach out to players to sign-up for a "anti-cheater program". The selection of players would have to be as random as possible (this is important!), but you'd probably have to make some restrictions (e.g., you could use only accounts that have existed for more than a year) and I'd rely on Lichess to make the right call here.

Wouldn't the random selection of players to participate in a cheater program create its won bias? You would now be creating a database of cheated games created mostly by people who normally don't cheat. I find it hard to believe they behave the same as notorious cheaters. Not only are they "beginners" in cheating, but by participating in the program, they're likely to "overcompensate" (for instance, follow an engines recommendation more often than a real cheater would).

Maybe it will all work out fine, but I need a bit more convincing this creates a good training set. (Disclaimer: while I've been working in IT/CS for nearly 4 decades, I don't much expertise in ML (and only a little in ML (the language))).

> The players playing against those "programmatic cheaters" would obviously get a refund on their rating (and the programmatic cheater would not gain rating) as well as a message telling them that Lichess is grateful they played the last game against a cheater to improve Lichess mechanisms against actual cheating.

So, if I play against such a cheater, all I get is a message indicating my time has been wasted? Sure, I won't losing rating points (but I didn't gain them either), but I won't be compensated for the time. Now, most people won't care, but if it happens often enough, at least some people will move on. Also, if I were to participate in that program, and immediately after the game Lichess points out that I participated/cheated to my opponent, how does that change her view of me? Will she be less likely to play against me a next time?

#44, i agree. i previously thought op was lying about everything they were telling about themselves. but this ask-people-to-cheat program is so badly thought through and so unethical, i now fully believe that op is associated with amazon.

@odoaker2015 said in #30:
> Your observation is obviously incomplete. And the fact that your observation doesn't show that there are left and right innocent banned players doesn't mean that there aren't any. This is a wrong conclusion.
>
> @ Cedur216
Well, u haven't proved there being some neither, have u?

@Abigail-III said in #44:
> Wouldn't the random selection of players to participate in a cheater program create its won bias? You would now be creating a database of cheated games created mostly by people who normally don't cheat. I find it hard to believe they behave the same as notorious cheaters. Not only are they "beginners" in cheating, but by participating in the program, they're likely to "overcompensate" (for instance, follow an engines recommendation more often than a real cheater would).
>
> Maybe it will all work out fine, but I need a bit more convincing this creates a good training set. (Disclaimer: while I've been working in IT/CS for nearly 4 decades, I don't much expertise in ML (and only a little in ML (the language))).
>
>
>
> So, if I play against such a cheater, all I get is a message indicating my time has been wasted? Sure, I won't losing rating points (but I didn't gain them either), but I won't be compensated for the time. Now, most people won't care, but if it happens often enough, at least some people will move on. Also, if I were to participate in that program, and immediately after the game Lichess points out that I participated/cheated to my opponent, how does that change her view of me? Will she be less likely to play against me a next time?

There is no reason to employ humans as 'model cheaters', you would use bots tuned to specific discovered ways of cheating. Would be no surprise if LiChess/Chess.com already deploy this method.

I have over 4 decades of designing and testing safety-critical systems - where failure may result in human death - and when I first investigated these anti-cheat detection systems, I very quickly came to the conclusion that 'model cheater bots' was the ONLY known method of testing the strength and quality of such systems.

You might think that usage of such bots was unethical, but the number would be extremely small (relative to the number of actual cheaters) and the benefits considerable - every time a new method of cheating was 'discovered' (or a known false positive 'discovered') you would create a new targeted bot and so refined the quality of the testing. It's pretty much the only way to counter false-positives due to using statistical and human analysis processes as well a fine tuning the whole anti-cheating processes in an objective manner. A small price to pay for the vast majority of customers. 'Secret Shoppers' are a similar idea and is generally considered very effective means of quality control.

For what it's worth, this is essentially the same method used to qualify complex financial security systems - you employ a team of people dedicated to breaking the security, and when they do you fix the system.

Each player should decide who they want to play. If they want to play someone suspected of cheating, fine. Just let the accuser identify himself and if he's wrong then that will affect how others believe him.

Note that lichess requires you to play people, which should NEVER be the case. You should be able to cancel as many games as you wish without repercussions. "Abort" is a loaded term which should be replaced.

@IrwinCaladinResearch Agreeing with what he said, i am a victim of this, i haven't used engine assistance in any of my games, and my recent games have all been rapid games(TC of 10+ mins), just woke up and saw my account banned and i have been accused of using illegal assistance, but yet i have been on a loosing streak of more than 10 games, so i'm sincerely confused, if i cheated why should i be loosing? i didn't purposely lose, and yet i haven't been winning as well, so i'm really confused, is this ban a result of punishment from when i opened my account newly which was about 2 years ago, why is it coming now? i haven't cheated so far, so i'm really lost here, glad for this post, because i know of people who have cheated recently or over time and yet they haven't been banned so why is it me? like he said, maybe i'm just part of the false positives and that's really not far, of course it's going to be hard to build an ML system with 100% precision, recall, accuracy, but lichess should really look into it.