Plans for the 4th Lichess Bots Championship

The 4th Lichess Bots Championship will start on the 1st of December 2019 UTC and will have different rules (unlike the previous Lichess Bots Championships). After the 3rd Lichess Bots Championship has ended (because that's what's important right now) and before the 1st of December 2019 UTC, participants can sign up for the tournament in a thread that will be created by arbiter Ajile (the time control will also be announced then).

From the 1st of December to the 6th of December 2019 UTC, all participants can challenge each other to matches (at any time) and post the results in a thread that will be created by Ajile. Matches are won when one participant gets a 4 point advantage over another (for example a 7-3 win, this total score has to be posted). Because participants can challenge each other at any time, all relevant results in the crosstables between players will count (you don't have to agree on a time to play and the owner of the bot doesn't have to be present). This (and an uncommon time control) will make it easier to check if the results that are posted by the participants are correct. During the entire tournament only your first match against an opponent will count.

On the 7th of December 2019 UTC, Ajile will create four groups based on how many participants each bot has defeated. If bots have defeated the same amount of participants, the bot with the higher 'proficiency score' (not only tournament games, the formula: rated wins / rated losses * rated wins and rated draws / rated losses) has a higher ranking. Group A is the highest group, group D is the lowest group. Participants that played less than 4 tournament games, are removed from the tournament.

From the 8th of December to the 13th of December 2019 UTC, all remaining participants can challenge each other, but only the results against participants from their own group count (including the tournament results they achieved against participants from their group during the previous week, they 'cannot' play more matches against them). On the 14th of December 2019 UTC, Ajile will split all groups in two based on how many participants from their own group each bot has defeated. If bots have defeated the same amount of participants, the bot with the higher 'proficiency score' (not only tournament games, the formula: rated wins / rated losses * rated wins and rated draws / rated losses) has a higher ranking. Group A1 is the highest group, group D2 is the lowest group. Participants that played less than 16 tournament games so far, are removed from the tournament.

From the 15th of December to the 20th of December 2019 UTC, all remaining participants can challenge each other, but only the results against participants from their own group count (including the tournament results they achieved against participants from their group during the previous two weeks, they 'cannot' play more matches against them). Some participants will already have played against all participants from their group before the end of the week. On the 21th of December 2019 UTC, Ajile decides the ranking in the same way as in week 1 and 2, but the final ranking has to have a clear winner (especially in group A1, which decides the winner of the tournament), so if during week 3 it seems like two participants will end up with the same amount of participants from their own group they have defeated, they can play an additional match against each other to determine who of them gets a better final ranking. The winner of group A1 is the winner of the 4th Lichess Bots Championship.

Questions or reactions based on this plan are welcome.

Sounds good. What time control is gonna be used? Will we still play Standard chess?

All games will be Standard Classical chess. I do have some ideas for the 5th and 6th Lichess Bots Championships that include other time controls and variants.

I think that a participant of the A group will not go to the next round while a worse bot from D group can go to the next round. Am I right?

In general all bots play all three weeks, you don't get eliminated (I just split the tournament into more groups so you play against your own strength more often). The exception is when you don't play enough games (so only participants who actually want to play get challenged to matches). Group A is higher than group D, but that doesn't mean that bots in group A will play more matches. So it's different from the 3rd Lichess Bots Championship where stronger bots play more matches.

So it would be kinda similar to a Swiss tournament, but split into chunks so it's easier to manage?

I think it's more like a round-robin tournament without pairings (so you can play as many games on a day as you want), where the tournament is split into chunks to limit how many opponents you have to play against and to make sure you have to win against other bots that are approximately your strength if you want to win your group.

Bots can qualify for group A by winning against much weaker bots (only one match against each bot), but those games won't count in week 2, so they still have to win against other strong bots in week 2 to qualify for group A1. Strong bots can also choose to already play against other strong bots in week 1 and try to qualify for group A1 without even playing in week 2 (because the results against other bots that end up in group A will carry over into week 2), but they'll risk losing too often and not even getting into group A. To make sure they'll at least get into group A, they'll probably play against weaker bots as well.

I like the experiment with a radical change of the format. Two questions:
1. If there is, let's say, a Stockfish engine on 200 cores in the field, would it not be very hard for that engine to score any points because all candidate opponents would choose a weaker opponent to improve their match score? Did I miss any incentive for the weaker engines to match the very strong one, or is this intentional?

2. Could you elaborate on the 'proficiency score' (maybe a formula with brackets to avoid ambiguities) and the range of games to which it applies?

These are all good questions for me to explain it further, thank you all.

How many participants bots have defeated decides in which groups they go, regardless of the amount of losses, so it doesn't hurt to lose against bots that end up in different groups. The best way to get many points is to play many matches (but only one against each bot). You also don't have to agree on a time to play and the owner of the bot doesn't have to be present, so the very strong bots can just challenge the weaker bots that are online whenever they want. Some bots will probably limit who (and how many at the same time) can challenge it, but that will also limit the amount of matches it plays (and might limit its score). It might also not be in the best interest of a weaker bot to avoid a very strong one, because when that very strong bot ends up in the group of the weaker bot (because it didn't have enough opponents to play against; which I don't think will happen), the weaker bot will have a more difficult week 2.

The 'proficiency score' of a bot is calculated with: rated wins / rated losses * (rated wins + rated draws) / rated losses. It is used to make sure there aren't any ties in week 1 and 2 (ties are less likely in week 3 and an additional match is possible then to decide the final ranking) because no bots have played (and won) the exact same amount of games on Lichess (not only the tournament games count, but all rated games a bot has played on Lichess). I don't think it will play a role in determining the winner of the tournament, so it's mostly a practical tool to help determine the groups when there are ties.