Dev Builds » 20240820-1859

Use this dev build

NCM plays each Stockfish dev build 20,000 times against Stockfish 15. This yields an approximate Elo difference and establishes confidence in the strength of the dev builds.

Summary

Host Duration Avg Base NPS Games WLD Standard Elo Ptnml(0-2) Gamepair Elo
ncm-dbt-01 06:51:55 571649 4010 1511 557 1942 +84.27 ± 4.99 2 116 825 1050 12 +177.58 ± 11.84
ncm-dbt-02 06:50:35 587747 4000 1523 599 1878 +81.73 ± 4.94 1 113 860 1013 13 +171.02 ± 11.57
ncm-dbt-03 06:51:47 576436 4012 1558 612 1842 +83.49 ± 5.05 3 118 832 1036 17 +174.82 ± 11.79
ncm-dbt-04 06:51:46 544309 4000 1533 572 1895 +85.14 ± 4.99 3 107 832 1042 16 +179.0 ± 11.78
ncm-dbt-05 06:51:28 581633 3978 1495 550 1933 +84.14 ± 4.97 1 111 832 1032 13 +176.81 ± 11.78
20000 7620 2890 9490 +83.75 ± 2.23 10 565 4181 5173 71 +175.84 ± 5.25

Test Detail

ID Host Base NPS Games WLD Standard Elo Ptnml(0-2) Gamepair Elo CLI PGN
373644 ncm-dbt-01 467684 168 63 23 82 +84.34 ± 24.72 1 3 35 45 0 +185.38 ± 58.22 ↓
373643 ncm-dbt-04 454735 168 64 21 83 +90.95 ± 26.44 0 8 25 51 0 +196.41 ± 68.36 ↓
373642 ncm-dbt-03 482821 176 66 27 83 +78.29 ± 25.04 1 4 39 43 1 +165.44 ± 54.95 ↓
373641 ncm-dbt-05 581361 478 160 68 250 +67.71 ± 13.77 0 14 120 104 1 +139.3 ± 30.72 ↓
373640 ncm-dbt-02 589240 500 182 79 239 +72.61 ± 14.69 0 20 110 117 3 +147.19 ± 32.57 ↓
373639 ncm-dbt-04 554591 332 132 45 155 +93.22 ± 16.7 0 7 66 92 1 +199.32 ± 42.12 ↓
373638 ncm-dbt-03 589027 336 131 61 144 +73.46 ± 18.83 1 15 66 85 1 +154.14 ± 42.17 ↓
373637 ncm-dbt-01 584034 342 133 51 158 +84.95 ± 15.63 0 5 80 85 1 +178.86 ± 37.49 ↓
373636 ncm-dbt-05 581610 500 195 68 237 +90.22 ± 14.6 0 16 94 137 3 +189.0 ± 35.33 ↓
373635 ncm-dbt-02 589411 500 192 68 240 +88.0 ± 12.86 0 8 110 132 0 +189.0 ± 32.17 ↓
373634 ncm-dbt-04 555804 500 201 62 237 +99.2 ± 13.83 1 7 97 142 3 +213.85 ± 34.58 ↓
373633 ncm-dbt-03 586604 500 198 76 226 +86.52 ± 13.0 0 8 113 128 1 +183.51 ± 31.65 ↓
373632 ncm-dbt-01 583824 500 177 72 251 +74.06 ± 15.01 0 24 98 127 1 +153.86 ± 34.52 ↓
373631 ncm-dbt-05 580157 500 186 65 249 +85.78 ± 12.82 0 8 113 129 0 +183.51 ± 31.65 ↓
373630 ncm-dbt-02 585801 500 184 70 246 +80.63 ± 13.94 0 14 110 124 2 +167.53 ± 32.45 ↓
373629 ncm-dbt-04 556413 500 187 77 236 +77.71 ± 13.71 0 15 110 125 0 +164.07 ± 32.48 ↓
373628 ncm-dbt-03 587707 500 195 77 228 +83.57 ± 14.6 1 15 101 131 2 +176.33 ± 34.04 ↓
373627 ncm-dbt-01 586477 500 197 70 233 +90.22 ± 15.17 1 16 92 137 4 +189.0 ± 35.7 ↓
373626 ncm-dbt-05 583572 500 206 57 237 +106.77 ± 14.05 0 10 85 151 4 +230.16 ± 37.2 ↓
373625 ncm-dbt-02 587155 500 198 80 222 +83.57 ± 13.86 1 11 108 129 1 +178.11 ± 32.72 ↓
373624 ncm-dbt-03 589240 500 186 78 236 +76.25 ± 14.39 0 19 105 125 1 +158.93 ± 33.37 ↓
373623 ncm-dbt-01 584537 500 195 60 245 +96.19 ± 13.16 0 9 97 144 0 +209.91 ± 34.63 ↓
373622 ncm-dbt-04 555311 500 187 65 248 +86.52 ± 13.63 0 12 105 132 1 +183.51 ± 33.24 ↓
373621 ncm-dbt-05 582527 500 182 66 252 +82.1 ± 13.83 0 15 104 131 0 +174.55 ± 33.5 ↓
373620 ncm-dbt-02 586435 500 177 79 244 +68.99 ± 14.56 0 21 112 115 2 +140.62 ± 32.26 ↓
373619 ncm-dbt-01 586604 500 185 77 238 +76.25 ± 14.39 0 20 102 128 0 +160.64 ± 33.88 ↓
373618 ncm-dbt-03 587452 500 195 67 238 +90.96 ± 15.47 0 20 87 138 5 +187.16 ± 36.61 ↓
373617 ncm-dbt-04 555462 500 190 78 232 +79.17 ± 13.75 0 14 111 124 1 +165.8 ± 32.28 ↓
373616 ncm-dbt-02 588728 500 206 71 223 +96.19 ± 13.32 0 9 98 142 1 +207.95 ± 34.43 ↓
373615 ncm-dbt-05 579826 500 191 77 232 +80.63 ± 14.23 0 16 106 126 2 +167.53 ± 33.17 ↓
373614 ncm-dbt-01 583321 500 179 70 251 +76.97 ± 14.27 0 18 106 125 1 +160.64 ± 33.2 ↓
373613 ncm-dbt-04 555728 500 188 78 234 +77.71 ± 14.44 1 16 106 126 1 +164.07 ± 33.19 ↓
373612 ncm-dbt-03 588217 500 195 81 224 +80.63 ± 13.94 0 14 110 124 2 +167.53 ± 32.45 ↓
373611 ncm-dbt-02 589240 500 195 77 228 +83.57 ± 14.45 0 15 106 125 4 +171.02 ± 33.15 ↓
373610 ncm-dbt-05 582318 500 185 79 236 +74.79 ± 14.9 1 19 105 123 2 +155.54 ± 33.38 ↓
373609 ncm-dbt-01 583154 500 193 65 242 +90.96 ± 13.72 0 11 102 135 2 +192.71 ± 33.75 ↓
373608 ncm-dbt-04 554251 500 188 83 229 +74.06 ± 14.88 0 19 112 114 5 +147.19 ± 32.24 ↓
373607 ncm-dbt-03 587834 500 197 71 232 +89.48 ± 13.38 0 10 105 134 1 +190.85 ± 33.16 ↓
373606 ncm-dbt-02 585970 500 189 75 236 +80.63 ± 13.79 0 15 106 129 0 +171.02 ± 33.15 ↓
373605 ncm-dbt-05 581693 500 190 70 240 +85.04 ± 13.75 0 13 105 131 1 +179.9 ± 33.27 ↓
373604 ncm-dbt-01 585211 500 189 69 242 +85.04 ± 13.6 0 10 113 124 3 +176.33 ± 31.78 ↓
373603 ncm-dbt-04 556489 500 196 63 241 +94.7 ± 14.24 1 9 100 136 4 +200.24 ± 34.09 ↓
373602 ncm-dbt-03 589027 500 195 74 231 +85.78 ± 14.07 0 13 106 128 3 +178.11 ± 33.1 ↓

Commit

Commit ID 5d81071953bd304e57613140b694b03a8241eac9
Author Linmiao Xu
Date 2024-08-20 18:59:36 UTC
Update default main net to nn-1111cefa1111.nnue Created from 2 distinct spsa tunes of the latest main net (nn-31337bea577c.nnue) and applying the params to the prior main net (nn-e8bac1c07a5a.nnue). This effectively reverts the modifications to output weights and biases in https://github.com/official-stockfish/Stockfish/pull/5509 SPSA: A: 6000, alpha: 0.602, gamma: 0.101 1st - 437 feature transformer biases where values are < 25 54k / 120k games at 180+1.8 https://tests.stockfishchess.org/tests/view/66af98ac4ff211be9d4edad0 nn-808259761cca.nnue 2nd - 208 L2 weights where values are zero 112k / 120k games at 180+1.8 https://tests.stockfishchess.org/tests/view/66b0c3074ff211be9d4edbe5 nn-a56cb8c3d477.nnue When creating the above 2 nets (nn-808259761cca.nnue, nn-a56cb8c3d477.nnue), spsa params were unintentionally applied to nn-e8bac1c07a5a.nnue rather than nn-31337bea577c.nnue due to an issue in a script that creates nets by applying spsa results to base nets. Since they both passed STC and were neutral or slightly positive at LTC, they were combined to see if the elo from each set of params was additive. The 2 nets can be merged on top of nn-e8bac1c07a5a.nnue with: https://github.com/linrock/nnue-tools/blob/90942d3/spsa/combine_nnue.py ``` python3 combine_nnue.py \ nn-e8bac1c07a5a.nnue \ nn-808259761cca.nnue \ nn-a56cb8c3d477.nnue ``` Merging yields nn-87caa003fc6a.nnue which was renamed to nn-1111cefa1111.nnue with an updated nnue-namer around 10x faster than before by: - using a prefix trie for efficient prefix matches - modifying 4 non-functional bytes near the end of the file instead of 2 https://github.com/linrock/nnue-namer Thanks to @MinetaS for pointing out in #nnue-dev what the non-functional bytes are: L3 is 32, 4 bytes for biases, 32 bytes for weights. (fc_2) So -38 and -37 are technically -2 and -1 of fc_1 (type AffineTransform<30, 32>) And since InputDimension is padded to 32 there are total 32 of 2 adjacent bytes padding. So yes, it's non-functional whatever values are there. It's possible to tweak bytes at -38 - 32 * N and -37 - 32 * N given N = 0 ... 31 The net renamed with the new method passed non-regression STC vs. the original net: https://tests.stockfishchess.org/tests/view/66c0f0a821503a509c13b332 To print the spsa params with nnue-pytorch: ``` import features from serialize import NNUEReader feature_set = features.get_feature_set_from_name("HalfKAv2_hm") with open("nn-31337bea577c.nnue", "rb") as f: model = NNUEReader(f, feature_set).model c_end = 16 for i,ft_bias in enumerate(model.input.bias.data[:3072]): value = int(ft_bias * 254) if abs(value) < 25: print(f"ftB[{i}],{value},-1024,1024,{c_end},0.0020") c_end = 6 for i in range(8): for j in range(32): for k in range(30): value = int(model.layer_stacks.l2.weight.data[32 * i + j, k] * 64) if value == 0: print(f"twoW[{i}][{j}][{k}],{value},-127,127,{c_end},0.0020") ``` New params found with the same method as: https://github.com/official-stockfish/Stockfish/pull/5459 Passed STC: https://tests.stockfishchess.org/tests/view/66b4d4464ff211be9d4edf6e LLR: 2.94 (-2.94,2.94) <0.00,2.00> Total: 136416 W: 35753 L: 35283 D: 65380 Ptnml(0-2): 510, 16159, 34416, 16597, 526 Passed LTC: https://tests.stockfishchess.org/tests/view/66b76e814ff211be9d4ee1cc LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 159336 W: 40753 L: 40178 D: 78405 Ptnml(0-2): 126, 17497, 43864, 18038, 143 closes https://github.com/official-stockfish/Stockfish/pull/5534 bench 1613043
Copyright 2011–2024 Next Chess Move LLC
Feedback & Support Terms of Service Privacy Policy Desktop Version