Chess engine based on real database

The ovious problem with bots and engines that they don't use human moves. When I practice I'd like to play against real human moves, including mistakes and falling into traps.

Lichess has a quite extensive game database and I thought it would be possible to integrate it with a chess GUI. The "engine" behind the GUI could choose a move randomly from previous games, taking into account the probabilities of moves. Even rating could be adjusted, for example I may not want to practice against GMs.
If there's no move in the database, the "engine" would fall back to a real engine.

As I saw Lichess has an API or maybe the database is downloadable so it seems to be quite possible to implement this. But it's still a lot of work to implement this and maybe somebody just did it previously.

So, the question is: do you know about an existing solution for the aforementioned challenge?

Well, if you want to play against humen moves, play humans! This is possible on this site!

I think there is little chance to get this done after the opening phase.

One reason would be the huge number of games, which makes it impossible to search for a specific position within reasonable constraints (space and time, or money). As of July 2024, there have been close to 6 billion rated games played on lichess. The compressed PGN files are 1.8 TB in size... And indexing everything would be insanely big.

And you probably would want to pick somewhat intelligently with regards to time control or rating...

But it would be interesting to see when a game really shows a new move that hasn't been played here before. :-)

I second @bikkel21's opinion: Why not simply play a human?

@tvik7755 What you want is technically very easy to do (and already exists) for databases of small size. The tool is called polyglot. It takes a database of games in pgn and creates an opening book. You can control how deep the book should go and how frequently a variation has to appear in the database for it to be included. Once you create a book, you configure your favourite GUI to use it. As long as you are in the book, the GUI will play from the book. To create the book run the following command on the command line:

polyglot make-book -min-game 6 -max-ply 40 -pgn database.pgn -bin book.bin

Here a line is included if it has been played at least 6 times in the database (default is 3). Lines will be truncated at 40 plies (half moves). So if you give -min-game 1 and don't use the max-ply option, then you have everything in the database as an opening line in your book.

Now the main problem is the size of the database. If you use a database of 4 million games (something like pgnmentor), and use -min-game 1, and no max-ply, you will already get a very large book that will be difficult to use in most GUIs on common computers. Even to create a book like that will require a lot of RAM. (On my laptop with 4GB RAM, I already have difficulty creating books from databases of more than 1M games.) If you give min-game 6 or 8, and max-ply 40, you will get a small book (a few MB) (but creating it will still require a lot of RAM if the database is big). In such a book not many lines will reach 40 plies because most games become unique by 40 plies (20 full moves).

@bikkel21 said in #2:

Well, if you want to play against humen moves, play humans! This is possible on this site!

Of course I play against humans online, but when I want to practice for my specific weaknesses in openings it's not really possible. I could write to my opponent that "I want you to play the Polish opening against me." but it'd be weird, right?

@nadjarostowa said in #3:

One reason would be the huge number of games, which makes it impossible to search for a specific position within reasonable constraints (space and time, or money). As of July 2024, there have been close to 6 billion rated games played on lichess. The compressed PGN files are 1.8 TB in size... And indexing everything would be insanely big.

If it would be a criteria to be able to use this tool offline that would be indeed a problem.

But if it's enough to use this tool online, Lichess has an awesome REST API for querying past games.
It's possible to query a given (FEN) position by additional parameters (rating, date) and it returns quite some stats and subsequent moves.
https://lichess.org/api#tag/Opening-Explorer/operation/openingExplorerLichess

@tvik7755 said in #5:

Of course I play against humans online, but when I want to practice for my specific weaknesses in openings it's not really possible. I could write to my opponent that "I want you to play the Polish opening against me." but it'd be weird, right?
You can set the desired position when you want to play with a friend.
I see your rating is similar to mine, so send me a correspondence challenge with the position set to the opening you want to practice (database use is allowed in correspondence)

@tvik7755

You can do something like this in the analysis board - just not automated, so you need to select and make the opponent's moves by hand:

• go to the analysis board

• turn off computer analysis and "arrow for the best move" in the settings

• click the "book" icon (in the right panel, bottom left)

• click the "Lichess" tab

• play your first move with white

• then play the top move (or one of the top moves) from the lichess database.

You can even restrict the database to a rating range or time format (click the settings icon)

I guess with some programming skills, this process could be automated in a script or whatever.

But if you want the computer to play random moves (what other people played), you won't get your desired variations either... You will get all kind of moves, many stupid moves among them.

I would go with something similar to @Panagrellus 's suggestion:

Open an analysis board, and look at the opening explorer there. And if you want to continue against the engine (no more "book" moves), you can chose to "Continue from here" from the menu.

@nadjarostowa

Just for fun, I tried my proposed procedure from #8 with the following tweak:
I didn't click on the top database move, but randomly clicked somewhere in the table of top 5 moves ( without looking - keep the eyes on the board!).

It worked surprisingly well, but even with the enormous Lichess database you'll run out of archived games at around move 20 or so, often earlier. There are just so many possible variations. Even with billions of games played, Lichess users are only exploring a tiny portion of all possible games.