Strategic Test Suite (STS): Introduction

The Strategic Test Suite (STS) is a series of themed test suites by Dann Corbit and Swaminathan Natarajan initially created circa 2008. Ferdinand Mosca updated the suite and wrote an STS-rating system. That code was last updated in 2019.

The positions were designed to aid chess engine programmers in evaluating their program's understanding of strategical concepts. The positions are also interesting for evaluation by humans!

The EPD file from Mosca has 1,500 positions organized in the following themes with 100 positions per theme:

I recently ran the STS-rating system with the latest versions of Stockfish, Lc0, and Komodo 12.1.1. The results were:

In the EPD file, the positions have a best move (scored 10 on an arbitrary scale) and several alternative moves with a smaller score. I modified the STS-rating code so that it would output how many of the positions the engine failed to find the best move. I then ran Stockfish again, but this time gave it 3 secs per position. That is almost 40 times the allowed time per the STS-rating allowed time for my machine! With that setting, Stockfish missed entirely 67 positions. Stockfish failed to get the supposed best move on 220 positions.

I decided these positions Stockfish failed would be interesting to investigate from a human perspective. I'll describe my findings on this in coming blog posts.

Links

- The Strategic Test Suite (STS) home page.
https://sites.google.com/site/strategictestsuite/

- The STS-rating code.
https://github.com/fsmosca/STS-Rating

- STS discussion and links on the chessprogramming.org site.
https://www.chessprogramming.org/Strategic_Test_Suite