Today at 16:14 UTC, while performing routine maintenance on the main database server, I accidentally deleted everything. 3.2 terabytes of Lichess history, gone.
Fortunately, we made plans for this scenario, and we have a backup server that mirrors the main database, with one hour of delay.
Our sysadmin successfully restored the database from the backup, and Lichess is back online.
However, we lost 38 minutes of data, from 15:36 to 16:14 UTC. This includes games, account creations, and other activities.
Studies and puzzles are not affected, as they are stored in a separate database.
We're deeply sorry for the downtime and data loss.
And we're grateful for the hard work of our sysadmin, who saved the day, once again.
Today at 16:14 UTC, while performing routine maintenance on the main database server, I accidentally deleted everything. 3.2 terabytes of Lichess history, gone.
Fortunately, we made plans for this scenario, and we have a backup server that mirrors the main database, with one hour of delay.
Our sysadmin successfully restored the database from the backup, and Lichess is back online.
However, we lost 38 minutes of data, from 15:36 to 16:14 UTC. This includes games, account creations, and other activities.
Studies and puzzles are not affected, as they are stored in a separate database.
We're deeply sorry for the downtime and data loss.
And we're grateful for the hard work of our sysadmin, who saved the day, once again.
These things happen, and the quick recovery shows how solid the backups and processes are. Well handled!
These things happen, and the quick recovery shows how solid the backups and processes are. Well handled!
ohhhhhh... that's why my really good game got deleted... I'm sry that happened @thibault
ohhhhhh... that's why my really good game got deleted... I'm sry that happened @thibault
That's unfortunate. But hopefully not too much data was lost.
That's unfortunate. But hopefully not too much data was lost.
humans make mistakes anyway... even if 1 good game of mine got deleted i'll just play another good game XD
don't take any large guilt...
humans make mistakes anyway... even if 1 good game of mine got deleted i'll just play another good game XD
don't take any large guilt...
Hey, can I get a rating refund for my game?
https://lichess.org/Q9RrmIQCEAOG
I won it but it scored like it was lost on time somehow
Hey, can I get a rating refund for my game? https://lichess.org/Q9RrmIQCEAOG
I won it but it scored like it was lost on time somehow
<Comment deleted by user>
Great job on restoring everything so quickly. Thanks for everything Thibault, this can happen to anybody!
Great job on restoring everything so quickly. Thanks for everything Thibault, this can happen to anybody!
You are doing a great job, thank you for that!!!
You are doing a great job, thank you for that!!!
Thanks for the transparency and quick fix! Grateful to you, the sysadmin and rest of the team for resolving the issue.
Thanks for the transparency and quick fix! Grateful to you, the sysadmin and rest of the team for resolving the issue.