Dev Builds » 20230701-1134

You are viewing an old NCM Stockfish dev build test. You may find the most recent dev build tests using Stockfish 15 as the baseline here.

Use this dev build

NCM plays each Stockfish dev build 20,000 times against Stockfish 14. This yields an approximate Elo difference and establishes confidence in the strength of the dev builds.

Summary

Host Duration Avg Base NPS Games WLD Standard Elo Ptnml(0-2) Gamepair Elo
ncm-dbt-01 09:54:19 1198287 3332 1411 304 1617 +119.98 ± 5.37 0 46 508 1071 41 +263.33 ± 15.12
ncm-dbt-02 09:51:04 1233717 3322 1408 316 1598 +118.61 ± 5.45 0 51 510 1057 43 +258.49 ± 15.09
ncm-dbt-03 09:55:50 1227105 3364 1420 294 1650 +120.95 ± 5.32 0 50 492 1104 36 +268.19 ± 15.37
ncm-dbt-04 09:55:42 1218432 3330 1387 300 1643 +117.72 ± 5.45 2 51 508 1066 38 +258.3 ± 15.13
ncm-dbt-05 09:51:55 1227572 3314 1408 299 1607 +120.92 ± 5.57 1 55 483 1070 48 +263.96 ± 15.51
ncm-dbt-06 09:55:18 1226549 3338 1386 297 1655 +117.65 ± 5.5 1 48 531 1039 50 +253.66 ± 14.78
20000 8420 1810 9770 +119.31 ± 2.22 4 301 3032 6407 256 +260.96 ± 6.18

Test Detail

ID Host Base NPS Games WLD Standard Elo Ptnml(0-2) Gamepair Elo CLI PGN
195340 ncm-dbt-05 1232347 314 127 28 159 +113.4 ± 18.77 0 8 46 99 4 +243.6 ± 50.81
195339 ncm-dbt-02 1232858 322 134 22 166 +126.1 ± 17.05 0 3 48 105 5 +278.3 ± 49.81
195338 ncm-dbt-04 1200912 330 140 38 152 +111.02 ± 18.41 1 4 58 96 6 +234.32 ± 45.09
195337 ncm-dbt-01 1177307 332 141 27 164 +124.35 ± 17.06 0 4 49 108 5 +273.38 ± 49.36
195336 ncm-dbt-06 1214005 338 133 38 167 +100.35 ± 17.46 0 8 61 97 3 +212.06 ± 44.02
195335 ncm-dbt-03 1210056 364 155 28 181 +126.53 ± 15.43 0 3 53 122 4 +285.38 ± 47.33
195334 ncm-dbt-05 1234000 500 215 45 240 +123.02 ± 14.97 1 11 61 171 6 +275.45 ± 43.72
195333 ncm-dbt-02 1249139 500 215 46 239 +122.24 ± 14.17 0 7 75 160 8 +265.78 ± 39.69
195332 ncm-dbt-01 1194991 500 212 56 232 +112.14 ± 13.89 0 9 80 157 4 +245.2 ± 38.39
195331 ncm-dbt-04 1222031 500 197 41 262 +112.14 ± 13.22 0 6 85 156 3 +247.41 ± 37.09
195330 ncm-dbt-06 1227536 500 221 47 232 +126.17 ± 13.42 0 3 78 161 8 +277.93 ± 38.66
195329 ncm-dbt-03 1234231 500 213 46 241 +120.67 ± 13.67 0 7 74 164 5 +268.17 ± 39.97
195328 ncm-dbt-05 1226052 500 220 49 231 +123.81 ± 13.63 0 6 73 165 6 +275.45 ± 40.24
195327 ncm-dbt-02 1234646 500 215 59 226 +112.14 ± 14.22 0 9 82 153 6 +240.82 ± 37.9
195326 ncm-dbt-01 1203042 500 211 47 242 +118.33 ± 14.36 0 8 78 156 8 +254.16 ± 38.89
195325 ncm-dbt-04 1223219 500 212 40 248 +124.6 ± 13.27 0 7 67 173 3 +285.49 ± 42.08
195324 ncm-dbt-06 1210181 500 206 37 257 +122.24 ± 15.29 0 9 76 152 13 +254.16 ± 39.42
195323 ncm-dbt-03 1210197 500 212 42 246 +123.02 ± 14.49 0 8 73 160 9 +265.78 ± 40.25
195322 ncm-dbt-05 1227410 500 207 43 250 +118.33 ± 13.35 0 6 78 162 4 +263.42 ± 38.85
195321 ncm-dbt-02 1223711 500 207 46 247 +116.0 ± 13.88 0 8 78 159 5 +254.16 ± 38.89
195320 ncm-dbt-01 1193141 500 219 47 234 +124.6 ± 13.97 0 6 74 162 8 +273.0 ± 39.95
195319 ncm-dbt-04 1217096 500 202 43 255 +114.45 ± 15.6 1 13 70 158 8 +245.2 ± 40.83
195318 ncm-dbt-06 1245740 500 214 47 239 +120.67 ± 13.67 0 5 80 158 7 +263.42 ± 38.27
195317 ncm-dbt-03 1237464 500 213 44 243 +122.24 ± 14.82 0 10 70 161 9 +263.42 ± 41.06
195316 ncm-dbt-05 1237044 500 218 42 240 +127.76 ± 14.6 0 7 71 161 11 +275.45 ± 40.84
195315 ncm-dbt-02 1232647 500 200 51 249 +106.77 ± 13.56 0 9 85 154 2 +234.38 ± 37.19
195314 ncm-dbt-04 1240151 500 213 44 243 +122.24 ± 13.66 0 6 75 163 6 +270.57 ± 39.67
195313 ncm-dbt-01 1207171 500 210 40 250 +123.02 ± 13.82 0 6 75 162 7 +270.57 ± 39.67
195312 ncm-dbt-06 1225946 500 202 34 264 +121.45 ± 13.14 0 5 76 165 4 +273.0 ± 39.35
195311 ncm-dbt-03 1232780 500 214 49 237 +119.11 ± 14.03 0 9 72 164 5 +263.42 ± 40.52
195310 ncm-dbt-05 1214722 500 204 48 248 +112.14 ± 14.38 0 8 86 148 8 +236.51 ± 36.93
195309 ncm-dbt-02 1229704 500 217 46 237 +123.81 ± 13.63 0 8 67 171 4 +280.42 ± 42.05
195308 ncm-dbt-04 1211699 500 211 47 242 +118.33 ± 13.7 0 6 80 158 6 +258.75 ± 38.32
195307 ncm-dbt-01 1209927 500 209 47 244 +116.77 ± 13.36 0 6 80 160 4 +258.75 ± 38.32
195306 ncm-dbt-06 1236655 500 208 43 249 +119.11 ± 15.46 1 11 69 160 9 +256.44 ± 41.25
195305 ncm-dbt-03 1225592 500 209 48 243 +116.0 ± 12.84 0 6 78 165 1 +263.42 ± 38.85
195304 ncm-dbt-06 1225785 500 202 51 247 +108.3 ± 13.89 0 7 91 146 6 +230.16 ± 35.78
195303 ncm-dbt-02 1233319 500 220 46 234 +126.17 ± 14.95 0 7 75 155 13 +265.78 ± 39.69
195302 ncm-dbt-05 1221432 500 217 44 239 +125.38 ± 14.63 0 9 68 164 9 +273.0 ± 41.7
195301 ncm-dbt-01 1202435 500 209 40 251 +122.24 ± 13.66 0 7 72 166 5 +273.0 ± 40.54
195300 ncm-dbt-03 1239417 500 204 37 259 +120.67 ± 13.32 0 7 72 168 3 +273.0 ± 40.54
195299 ncm-dbt-04 1213918 500 212 47 241 +119.11 ± 14.19 0 9 73 162 6 +261.07 ± 40.24

Commit

Commit ID 915532181f11812c80ef0b57bc018de4ea2155ec
Author Linmiao Xu
Date 2023-07-01 11:34:30 UTC
Update NNUE architecture to SFNNv7 with larger L1 size of 2048 Creating this net involved: - a 5-step training process from scratch - greedy permuting L1 weights with https://github.com/official-stockfish/Stockfish/pull/4620 - leb128 compression with https://github.com/glinscott/nnue-pytorch/pull/251 - greedy 2- and 3- cycle permuting with https://github.com/official-stockfish/Stockfish/pull/4640 The 5 training steps were: 1. 400 epochs, lambda 1.0, lr 9.75e-4 UHOx2-wIsRight-multinet-dfrc-n5000-largeGensfen-d9.binpack (178G) nodes5000pv2_UHO.binpack data_pv-2_diff-100_nodes-5000.binpack wrongIsRight_nodes5000pv2.binpack multinet_pv-2_diff-100_nodes-5000.binpack dfrc_n5000.binpack large_gensfen_multipvdiff_100_d9.binpack ep399 chosen as start model for step2 2. 800 epochs, end-lambda 0.75, skip 16 LeelaFarseer-T78juntoaugT79marT80dec.binpack (141G) T60T70wIsRightFarseerT60T74T75T76.binpack test78-junjulaug2022-16tb7p.no-db.min.binpack test79-mar2022-16tb7p.no-db.min.binpack test80-dec2022-16tb7p.no-db.min.binpack ep559 chosen as start model for step3 3. 800 epochs, end-lambda 0.725, skip 20 leela96-dfrc99-v2-T80dectofeb-sk20-mar-v6-T77decT78janfebT79apr.binpack (223G) leela96-filt-v2.min.binpack dfrc99-16tb7p-eval-filt-v2.min.binpack test80-dec2022-16tb7p-filter-v6-sk20.min-mar2023.binpack test80-jan2023-16tb7p-filter-v6-sk20.min-mar2023.binpack test80-feb2023-16tb7p-filter-v6-sk20.min-mar2023.binpack test80-mar2023-2tb7p-filter-v6.min.binpack test77-dec2021-16tb7p.no-db.min.binpack test78-janfeb2022-16tb7p.no-db.min.binpack test79-apr2022-16tb7p.no-db.min.binpack ep499 chosen as start model for step4 4. 800 epochs, end-lambda 0.7, skip 24 0dd1cebea57 dataset https://github.com/official-stockfish/Stockfish/pull/4606 ep599 chosen as start model for step5 5. 800 epochs, end-lambda 0.7, skip 28 same dataset as step4 ep619 became nn-1b951f8b449d.nnue For the final step5 training: python3 easy_train.py \ --experiment-name L1-2048-S5-sameData-sk28-S4-0dd1cebea57-shuffled-S3-leela96-dfrc99-v2-T80dectofeb-sk20-mar-v6-T77decT78janfebT79apr-sk20-S2-LeelaFarseerT78T79T80-ep399-S1-UHOx2-wIsRight-multinet-dfrc-n5000-largeGensfen-d9 \ --training-dataset /data/leela96-dfrc99-T60novdec-v2-T80juntonovjanfebT79aprmayT78jantosepT77dec-v6dd-T80apr.binpack \ --early-fen-skipping 28 \ --nnue-pytorch-branch linrock/nnue-pytorch/misc-fixes-L1-2048 \ --engine-test-branch linrock/Stockfish/L1-2048 \ --start-from-engine-test-net False \ --start-from-model /data/experiments/experiment_L1-2048-S4-0dd1cebea57-shuffled-S3-leela96-dfrc99-v2-T80dectofeb-sk20-mar-v6-T77decT78janfebT79apr-sk20-S2-LeelaFarseerT78T79T80-ep399-S1-UHOx2-wIsRight-multinet-dfrc-n5000-largeGensfen-d9/training/run_0/nn-epoch599.nnue --max_epoch 800 \ --lr 4.375e-4 \ --gamma 0.995 \ --start-lambda 1.0 \ --end-lambda 0.7 \ --tui False \ --seed $RANDOM \ --gpus 0 SF training data components for the step1 dataset: https://drive.google.com/drive/folders/1yLCEmioC3Xx9KQr4T7uB6GnLm5icAYGU Leela training data for steps 2-5 can be found at: https://robotmoon.com/nnue-training-data/ Due to larger L1 size and slower inference, the speed penalty loses elo at STC. Measurements from 100 bench runs at depth 13 with x86-64-modern on Intel Core i5-1038NG7 2.00GHz: sf_base = 1240730 +/- 3443 (95%) sf_test = 1153341 +/- 2832 (95%) diff = -87388 +/- 1616 (95%) speedup = -7.04330% +/- 0.130% (95%) Local elo at 25k nodes per move (vs. L1-1536 nn-fdc1d0fe6455.nnue): nn-epoch619.nnue : 21.1 +/- 3.2 Failed STC: https://tests.stockfishchess.org/tests/view/6498ee93dc7002ce609cf979 LLR: -2.95 (-2.94,2.94) <0.00,2.00> Total: 11680 W: 3058 L: 3299 D: 5323 Ptnml(0-2): 44, 1422, 3149, 1181, 44 LTC: https://tests.stockfishchess.org/tests/view/649b32f5dc7002ce609d20cf Elo: 0.68 ± 1.5 (95%) LOS: 80.5% Total: 40000 W: 10887 L: 10809 D: 18304 Ptnml(0-2): 36, 3938, 11958, 4048, 20 nElo: 1.50 ± 3.4 (95%) PairsRatio: 1.02 Passed VLTC 180+1.8: https://tests.stockfishchess.org/tests/view/64992b43dc7002ce609cfd20 LLR: 3.06 (-2.94,2.94) <0.00,2.00> Total: 38086 W: 10612 L: 10338 D: 17136 Ptnml(0-2): 9, 3316, 12115, 3598, 5 Passed VLTC SMP 60+0.6 th 8: https://tests.stockfishchess.org/tests/view/649a21fedc7002ce609d0c7d LLR: 2.95 (-2.94,2.94) <0.50,2.50> Total: 38936 W: 11091 L: 10820 D: 17025 Ptnml(0-2): 1, 2948, 13305, 3207, 7 closes https://github.com/official-stockfish/Stockfish/pull/4646 Bench: 2505168
Copyright 2011–2024 Next Chess Move LLC