Sharing a 3D chess board on the Web

https://youtu.be/WnymDpkIcVg

This is my third post about my 3D chessboard that integrates with Lichess. The other two posts discuss photo-realistic rendering and the UX.

You can play the latest Beta version free online. This is very early testing, so please expect bugs and help by reporting issues and suggestions on the Chessboard Discord.

The most recent version now allows you to play against human opponents, who are either using the normal Lichess 2D interface, and, if they are also using this app, you both interact with the same physical chessboard, dragging and knocking pieces together. The newest version is also about 1.5x faster on the fast path used on mobile and low-end GPUs.

The goal is to make playing chess feel as close to over-the-board (OTB) play in-real-life on a physical chess board. The pieces are simulated with physics, and can move anywhere, off the center of their square, or even onto the wrong squares. Pieces do not “teleport” to new locations. You need to drag them manually. You also need to remove captured pieces from the board manually, and move the rook for castling, unlike every other digital chess game which magically teleports those pieces after you move. When you play with other people, they may accidentally knock the piece they are moving against another piece. So, ideally, when you are playing against another human, you should both be using the same board, and seeing them drag their pieces and knock other pieces around on the same board. Either player can drag pieces at any time, even during their opponent’s move. The app becomes a sandbox that lets two remote players use a shared chessboard sandbox for rigid body physics. The board tries to follow the logical game and shows the last accepted move to indicate the active color.

Matching

Finding people to play would be impossible without Lichess.org which provides an active community of millions of potential participants. To get the full experience, where both players are sharing the same board means that both players need to be using the app. Implementing this with the Lichess.org matching service is a bit tricky. The first version starts by telling you if your Lichess friends are actively using the app using a green “presence” indicator. The round presence indicates whether the friend is active on Lichess.org, and the square icon shows if they are using the 3D chessboard app. When you start a new game, or join an existing game with one of your friends that is actively using the app you are connected (via WebRTC) and share a common board. The best way to get the full experience playing against someone else in the 3D app is to encourage your friends to try the app!

Later, when there are more active users of my app, I have an idea that will help you match against random players who are using the app by selecting unusual time limits for the game. Specifically, adding one minute to the total time allowed for a game, e.g. using 11+5 instead of the more common 10+5, or 31+0 instead of 30+0. I think the Lichess matching algorithm with prefer matches to the same time limits, but eventually fall back to the next closest time interval. Thoughts?

How does it work?

Once you have created a new game on Lichess.org, the app tries to connect directly to the challenger using a PeerJS signaling server. If the other player is using the app, and you are both playing the same game, a connection is established, and we can share the same chessboard. Whenever you or your opponent drags a piece, the dragging position is sent to the other player. Each side runs a separate simulation using the shared drag position actions. This allows the simulation to run without any lag.

Ideally, the simulation run separately on each client would stay in exact sync, so that every piece is in the exact same position for both players. This is impractical. Instead, we allow the physical positions of pieces to be different, as long as the logical game position is consistent. In fact, the game already allowed for this variation as part of its core mission to work like real-life, where pieces might be off-center on their square, or even knocked onto a different square.

The previous version of the app already allowed the 3D positions of pieces to be different from their logical game positions. For example, as long as a captured piece was off the board, it didn’t matter where it was, and pieces that were on the correct square, but not “centered”, were allowed to drift. A convenient “tidy” action is provided that allows you to center the pieces on the board, or line up the captured pieces in a specific off-board ordering.

When I implemented the previous version, I was thinking ahead to how it would work with multiplayer, but I still had to fix a bunch of bits to make all of this work in sync with two players. For example, the position of off-board pieces after the tidy operation must be identical, rather than just “a nice off-board layout”. How could I determine a specific order, where the same captured pawn is in the exact same off-board position? I decided to use the “capture order” to determine the ordering, and knowing which pieces were captured requires analyzing the move list, not just the current board position (FEN).

In fact, there were several places in the old code where I didn’t differentiate between the eight pawns, or the two knights/bishops/rooks and instead chose blindly based on piece type and color. When playing against another player, if you each chose a different black knight, then the board would get all messed up when you dragged different pieces!

Player Labels

The early version of multiplayer did not have labels with the opponent's Lichess name. When we played the first game against a real human, it was obvious that you needed two important pieces of feedback: when they were dragging a piece, and when they moved the camera.

Positioning the player label smoothly around the scene was pretty tricky too. I wanted the label at the top center of the screen when the opponent was on the opposite side of the board, and in the bottom center if they were on the same side of the board as you. And, when they rotated behind, say, the clock, their name should be behind the clock, wherever that is on your screen. It took me two tries to get the math logic right for these calculations. We start with the world space positions of your and the opponent cameras.

My first idea was to project the opponent's camera position onto our camera plane, extended out past the window edges. Then find the intersection of the edge of the camera with the line starting at the center of the camera plane and moving through the projected opponent camera point. This mostly worked, but had brittle behavior when the opponent was behind your camera. It was also sensitive to the vertical rotation of the camera, so when the opponent was on the same side of the board as you, and moved vertically, their label would flip radically from top to bottom.

https://youtu.be/u0zF4zRg4sA

The second ideas was to remove any dependence on the vertical rotation of the camera, and to use the relative angle between the two cameras of their positions projected down onto the table. The relative angle was stable, and could then easily be mapped to the edges of the screen by projecting outward from the unit disc. The final bit of trickery is how to place the text label using CSS, so that it is left justified on the left edge of the screen and smoothly interpolates to right justified on the right edge of the screen, which you can do pretty easily with absolute positioning, once you realize what's necessary.

Tidy

Though the board tries to be as physically correct as possible, it does possess a bit of magic. For example, if you start a new game, the pieces tumble randomly on the board, and, as you start to set up the pieces, the board recognizes that you want to start a new game and magically flies the pieces to their standard starting position, and creates a new game. Similarly, as you drag pieces around and accidentally knock into pieces that cause the board to be messed up, there’s a special “tidy” gesture, clicking on a piece to raise and lower it in the same place, that causes all the pieces to move gently to their square centers. If you click on an off-board piece, the captured pieces are all lined up in their capture order.

Similarly, if you capture a piece, but fail to remove it from the board, the app will wait until the very last moment, just before your next turn, before it is magically animated off the board in order to maintain the logical positions. The idea is to give you time to drag it manually, or use the fancy “capture exchange” gesture discussed in the previous blog post.

The goal is to allow the board to get “messy”. The 3D pieces are allowed to vary as much as possible, preserving the core “logical” position for the current game.

The tidy mechanism has a bonus in multiplayer: it synchronizes the two remote boards so that the pieces that are tidied are in the exact same locations. I found I was using the tidy gesture all the time when playing the AI, and it was nice to see that it has a benefit for multiplayer simulation.

In addition to sending the “tidy” event to the opponent, we also had to support a few other special events, like resignation and the tipping of the king action. You can also switch to another active game with the same friend, and both your and your opponent’s boards will be updated to the current position in the new game.

https://youtu.be/nWnq589AXR0

UI Tweaks

While testing multiplayer, a friend mentioned that they accidentally missed clicking a piece, clicking instead on the board, and that caused the camera to rotate which was disorientating. So, similar to the Apple Chess program, we now only rotate the camera when you click on the table or frame around the actual board squares. Also, the cursor changes from the default arrow pointer into the hand when hovering over a piece that can be dragged, providing visual re-enforcement that also minimizes mistaken drags.

We also noticed that the tidy gesture was a bit too easy to trigger, so we made it much harder to trigger accidentally. Now you need to click on one of your own pieces, releasing immediately without moving the piece, to trigger the tidy action. This felt much more intentional than the old way, which allowed you to drop the piece anywhere within one square of the previous position. It also allows you to nudge a piece around on its own square gently, without moving it off the square to make a move, which felt more natural than always tidying up as you nudged around a piece on its own square.

Failed Attempts

The most important decision for multiplayer was how to synchronize the physics simulations on each client? Should each client do their own simulation? If so, how do we keep those in sync since simulations are very different if the positions or forces are different in the slightest. Big multiplayer game engines often use a shared server to synchronize the physics, sending out position updates and making each client lag slightly behind due to network overhead. But this game only has two players, not hundreds.

The original idea for this app came from my friend, Drew Olbrich, when we worked together decades ago and Drew implemented a chessboard with rigid body physics that allowed two people to play together on our SGIs. When I asked Drew how he approached the synchronization issue, he responded:

I prioritized it something like this:

• It's most important that the two simulations look good and feel right and have no lag or anything like that.
• The two players are in remote locations and cannot visually compare the two simulations, so making the simulations perfectly synchronized and consistent is not a priority. It's OK if each player sees a slightly different board.
• Do whatever is necessary to make the two configurations of pieces logically plausible in support of the game play, but nothing more than that.

Prior to Drew’s suggestions, I had been planning on having a “single source of truth” for the shared physics simulation. That is, one of the two players was going to simulate everything and broadcast the results of the simulation to the other client. This means that one of the two players would see everything immediately, and the other would lag behind by whatever network delay existed between the two players. The network lag might be small, perhaps 50-100 ms, or much longer, depending on the locations of each player in the world and their network connection. But even a small lag of 50ms would still be “noticeable”. Based on Drew’s suggestions, I decided to simulate each side separately and allow deviation in the actual board positions, as long as the important “logical” game position was preserved.

Getting remote dragging was the very first thing to work on for multiplayer. What data should be sent? The screen position? The piece’s 3D world space position? What other data do we need? How do we uniquely identify the eight different pawns? The very first thing I tried was to send over any dragged piece, and then try to update each side if any of the pieces were moved by the physics system. This quickly led to feedback, like you can see here:

https://youtu.be/vr78JMkwPVM

Internally, the app was not using unique identifiers for each piece. Instead, the pieces were just kept in an array. In general, the array order was mostly consistent. But for multiplayer, I needed to fix the ordering so that the index into the array became a unique identifier, and we never accidentally switched which of the black pawns was placed on, for example, A7. There were a few specific places in the app where we would change the order, especially for captured pieces that needed fixing. There are still a few bugs lurking deep in this area that occasionally cause the board to get messed up, but, now that I know the architecture, I hope to refactor the last bits of brittle logic.

The next set of issues when implementing multiplayer was fixing up all the “magical” animations, like tidy, castling, and capture, so that they were aware of a remote player. For example, here’s an early test where the castling animation starts fighting with the remote player dragging the rook:

https://youtu.be/Q4GflpssGv0

There were other funny things to fix, including a bypass of the normal “pause” effect in the app to prevent the app from hogging the machine when it is inactive. When two players are connected, it wakes up from pausing on any event from the remote client, so the remote user dragging will wake up the local client and show the action, rather than staying frozen.

https://youtu.be/6FYDBJbERCM

Limitations

There are a few limitations in this initial release of multiplayer support:

• No rated games. The app allows you to create games against your friends, but all of the games are unrated for now, because I do not want game bugs to lower your rating.
• No hints. When you play against the computer, the app can show you the “best move” predicted using a local stockfish worker. This is disabled entirely when playing humans to prevent cheating.
• No random matching. You can only create new games against your Lichess friends (or the AI), but you cannot match against a random player yet. After a bit more testing, I’ll add support for random matching.

Next Steps

I expect that multiplayer will require lots of tweaks and fixes, which will be my primary near-term focus. I need to implement random opponent matching, which may be tricky. Now that the app is mostly fleshed out, I desperately need to add testing to allow development without regressions.

There are a few good candidates for my next major task:

• Adding more visualization of the tactics and strategy using transparent overlays, like glass walls showing pawn structures, poorly defended pieces, possible forks and skewers, and similar.
• Improving rendering, both the quality at the high end, and the performance at the low end.
• Adding a “TV” mode for watching Lichess games without being an active player.

Please join us on the Chessboard Discord to discuss these future development directions!