Dev Builds » 20150403-0240

You are viewing an old NCM Stockfish dev build test. You may find the most recent dev build tests using Stockfish 15 as the baseline here.

Use this dev build

NCM plays each Stockfish dev build 20,000 times against Stockfish 14. This yields an approximate Elo difference and establishes confidence in the strength of the dev builds.

Summary

Host Duration Avg Base NPS Games WLD Standard Elo Ptnml(0-2) Gamepair Elo

Test Detail

ID Host Base NPS Games WLD Standard Elo Ptnml(0-2) Gamepair Elo CLI PGN

Commit

Commit ID 926f215061311392bc26c7bc4bde5b719dbab4e5
Author Marco Costalba
Date 2015-04-03 02:40:55 UTC
Add support for playing in 'nodes as time' mode When running more games in parallel, or simply when running a game with a background process, due to how OS scheduling works, there is no guarantee that the CPU resources allocated evenly between the two players. This introduces noise in the result that leads to unreliable result and in the worst cases can even invalidate the result. For instance in SF test framework we avoid running from clouds virtual machines because are a known source of very unstable CPU speed. To overcome this issue, without requiring changes to the GUI, the idea is to use searched nodes instead of time, and to convert time to available nodes upfront, at the beginning of the game. When nodestime UCI option is set at a given nodes per milliseconds (npmsec), at the beginning of the game (and only once), the engine reads the available time to think, sent by the GUI with 'go wtime x' UCI command. Then it translates time in available nodes (nodes = npmsec * x), then feeds available nodes instead of time to the time management logic and starts the search. During the search the engine checks the searched nodes against the available ones in such a way that all the time management logic still fully applies, and the game mimics a real one played on real time. When the search finishes, before returning best move, the total available nodes are updated, subtracting the real searched nodes. After the first move, the time information sent by the GUI is ignored, and the engine fully relies on the updated total available nodes to feed time management. To avoid time losses, the speed of the engine (npms) must be set to a value lower than real speed so that if the real TC is for instance 30 secs, and npms is half of the real speed, the game will last on average 15 secs, so much less than the TC limit, providing for a safety 'time buffer'. There are 2 main limitations with this mode. 1. Engine speed should be the same for both players, and this limits the approach to mainly parameter tuning patches. 2. Because npms is fixed while, in real engines, the speed increases toward endgame, this introduces an artifact that is equivalent to an altered time management. Namely it is like the time management gives less available time than what should be in standard case. May be the second limitation could be mitigated in a future with a smarter 'dynamic npms' approach. Tests shows that the standard deviation of the results with 'nodestime' is lower than in standard TC, as is expected because now all the introduced noise due the random speed variability of the engines during the game is fully removed. Original NIT idea by Michael Hoffman that shows how to play in NIT mode without requiring changes to the GUI. This implementation goes a bit further, the key difference is that we read TC from GUI only once upfront instead of re-reading after every move as in Michael's implementation. No functional change.
Copyright 2011–2024 Next Chess Move LLC