Dev Builds » 20210815-1005

You are viewing an old NCM Stockfish dev build test. You may find the most recent dev build tests using Stockfish 15 as the baseline here.

Use this dev build

NCM plays each Stockfish dev build 20,000 times against Stockfish 14. This yields an approximate Elo difference and establishes confidence in the strength of the dev builds.

Summary

Host Duration Avg Base NPS Games WLD Standard Elo Ptnml(0-2) Gamepair Elo
ncm-dbt-01 11:52:28 1231552 3988 1113 885 1990 +19.88 ± 5.36 0 373 1036 569 16 +37.08 ± 10.57
ncm-dbt-02 11:46:53 1263066 3928 1088 822 2018 +23.56 ± 5.44 3 349 1006 591 15 +45.19 ± 10.73
ncm-dbt-03 11:52:22 1266997 3980 1095 847 2038 +21.68 ± 5.38 6 354 1026 594 10 +42.82 ± 10.62
ncm-dbt-04 00:30:36 1488282 166 49 34 83 +31.48 ± 26.3 0 13 43 26 1 +59.17 ± 52.23
ncm-dbt-05 11:47:15 1256416 3958 1072 874 2012 +17.4 ± 5.5 2 401 986 577 13 +32.93 ± 10.85
ncm-dbt-06 11:51:48 1261418 3980 1113 892 1975 +19.31 ± 5.52 8 376 1011 577 18 +36.98 ± 10.71
20000 5530 4354 10116 +20.45 ± 2.42 19 1866 5108 2934 73 +39.15 ± 4.76

Test Detail

ID Host Base NPS Games WLD Standard Elo Ptnml(0-2) Gamepair Elo CLI PGN
209829 ncm-dbt-02 1241938 262 78 53 131 +33.25 ± 20.72 0 20 67 43 1 +64.38 ± 41.75 ↓
209828 ncm-dbt-05 1233549 292 72 65 155 +8.33 ± 21.24 0 38 63 45 0 +16.67 ± 42.72 ↓
209827 ncm-dbt-06 1226509 312 75 72 165 +3.34 ± 18.92 0 36 81 39 0 +6.68 ± 37.95 ↓
209826 ncm-dbt-03 1233241 316 90 65 161 +27.55 ± 18.8 2 21 85 50 0 +59.96 ± 36.86 ↓
209825 ncm-dbt-01 1210925 318 88 61 169 +29.57 ± 18.87 0 26 81 51 1 +57.33 ± 37.95 ↓
209824 ncm-dbt-02 1219534 500 138 106 256 +22.26 ± 14.6 0 42 135 72 1 +43.3 ± 29.24 ↓
209823 ncm-dbt-06 1226083 500 137 111 252 +18.08 ± 15.78 1 50 123 74 2 +34.86 ± 30.78 ↓
209822 ncm-dbt-05 1237611 500 135 110 255 +17.39 ± 15.24 1 47 129 72 1 +34.86 ± 30.03 ↓
209821 ncm-dbt-03 1227938 500 145 124 231 +14.6 ± 15.88 1 52 125 69 3 +26.46 ± 30.53 ↓
209820 ncm-dbt-01 1191617 500 142 110 248 +22.26 ± 15.84 0 52 115 82 1 +43.3 ± 31.77 ↓
209819 ncm-dbt-02 1226582 500 134 103 263 +21.57 ± 15.67 1 49 118 82 0 +44.72 ± 31.41 ↓
209818 ncm-dbt-06 1227828 500 134 105 261 +20.17 ± 14.45 0 42 138 69 1 +39.08 ± 28.85 ↓
209817 ncm-dbt-05 1227782 500 148 110 242 +26.45 ± 15.64 0 45 126 75 4 +47.55 ± 30.4 ↓
209816 ncm-dbt-03 1264450 500 140 95 265 +31.35 ± 15.72 0 45 117 86 2 +60.36 ± 31.55 ↓
209815 ncm-dbt-01 1197914 500 130 103 267 +18.78 ± 15.1 0 48 128 73 1 +36.26 ± 30.15 ↓
209814 ncm-dbt-02 1239393 500 133 112 255 +14.6 ± 15.28 1 49 129 70 1 +29.25 ± 30.03 ↓
209813 ncm-dbt-06 1239426 500 140 106 254 +23.66 ± 16.4 3 45 120 79 3 +47.55 ± 31.16 ↓
209812 ncm-dbt-05 1225341 500 153 119 228 +23.66 ± 16.62 0 55 110 81 4 +41.89 ± 32.37 ↓
209811 ncm-dbt-03 1234769 500 133 100 267 +22.96 ± 15.76 1 47 122 78 2 +44.72 ± 30.91 ↓
209810 ncm-dbt-01 1194821 500 135 107 258 +19.48 ± 14.26 0 41 141 67 1 +37.67 ± 28.45 ↓
209809 ncm-dbt-02 1230711 500 134 111 255 +15.99 ± 15.38 0 52 124 73 1 +30.65 ± 30.66 ↓
209808 ncm-dbt-05 1215833 500 119 99 282 +13.91 ± 15.22 1 50 127 72 0 +29.25 ± 30.28 ↓
209807 ncm-dbt-06 1235008 500 143 106 251 +25.76 ± 15.47 1 41 132 72 4 +47.55 ± 29.63 ↓
209806 ncm-dbt-03 1234707 500 138 103 259 +24.36 ± 15.5 1 45 123 80 1 +48.96 ± 30.78 ↓
209805 ncm-dbt-01 1167445 500 136 99 265 +25.76 ± 14.98 0 42 131 75 2 +48.96 ± 29.75 ↓
209804 ncm-dbt-05 1214929 500 129 111 260 +12.51 ± 15.36 0 53 128 67 2 +22.27 ± 30.16 ↓
209803 ncm-dbt-02 1219136 500 157 97 246 +41.89 ± 15.12 1 31 128 87 3 +82.1 ± 30.08 ↓
209802 ncm-dbt-06 1237682 500 144 117 239 +18.78 ± 15.71 1 48 127 71 3 +34.86 ± 30.28 ↓
209801 ncm-dbt-03 1229069 500 135 111 254 +16.69 ± 14.04 1 38 148 62 1 +33.46 ± 27.51 ↓
209800 ncm-dbt-01 1217085 500 137 123 240 +9.73 ± 15.14 0 52 135 60 3 +15.3 ± 29.27 ↓
209799 ncm-dbt-05 1237688 500 132 110 258 +15.3 ± 14.32 0 44 141 64 1 +29.25 ± 28.47 ↓
209798 ncm-dbt-06 1236333 500 148 119 233 +20.17 ± 14.96 1 43 133 72 1 +40.48 ± 29.51 ↓
209797 ncm-dbt-02 1247837 500 127 98 275 +20.17 ± 14.32 0 40 143 65 2 +37.67 ± 28.18 ↓
209796 ncm-dbt-03 1234679 500 134 115 251 +13.21 ± 15.05 0 51 130 68 1 +25.06 ± 29.91 ↓
209795 ncm-dbt-01 1202064 500 135 120 245 +10.43 ± 14.82 0 52 131 67 0 +20.87 ± 29.78 ↓
209794 ncm-dbt-02 1253710 500 138 104 258 +23.66 ± 15.81 0 46 130 68 6 +39.08 ± 29.9 ↓
209793 ncm-dbt-06 1227369 500 143 113 244 +20.87 ± 16.21 1 51 118 77 3 +39.08 ± 31.4 ↓
209792 ncm-dbt-05 1224242 500 127 113 260 +9.73 ± 14.89 0 53 130 67 0 +19.48 ± 29.91 ↓
209791 ncm-dbt-01 1210015 500 158 116 226 +29.25 ± 15.22 0 39 135 71 5 +51.8 ± 29.22 ↓
209790 ncm-dbt-03 1247909 500 140 101 259 +27.15 ± 14.18 0 37 137 76 0 +54.65 ± 28.94 ↓
178030 ncm-dbt-06 1496532 168 49 43 76 +12.41 ± 28.05 0 20 39 24 1 +20.71 ± 54.85 ↓
178029 ncm-dbt-02 1488754 166 49 38 79 +23.06 ± 29.07 0 20 32 31 0 +46.32 ± 59.38 ↓
178028 ncm-dbt-03 1496216 164 40 33 91 +14.84 ± 27.14 0 18 39 25 0 +29.73 ± 54.94 ↓
178027 ncm-dbt-04 1488282 166 49 34 83 +31.48 ± 26.3 0 13 43 26 1 +59.17 ± 52.23 ↓
178026 ncm-dbt-05 1490770 166 57 37 72 +42.06 ± 29.26 0 16 32 34 1 +80.97 ± 59.81 ↓
178025 ncm-dbt-01 1492083 170 52 46 72 +12.26 ± 28.87 0 21 39 23 2 +16.36 ± 54.8 ↓

Commit

Commit ID d61d38586ee35fd4d93445eb547e4af27cc86e6b
Author Tomasz Sobczyk
Date 2021-08-15 10:05:43 UTC
New NNUE architecture and net Introduces a new NNUE network architecture and associated network parameters The summary of the changes: * Position for each perspective mirrored such that the king is on e..h files. Cuts the feature transformer size in half, while preserving enough knowledge to be good. See https://docs.google.com/document/d/1gTlrr02qSNKiXNZ_SuO4-RjK4MXBiFlLE6jvNqqMkAY/edit#heading=h.b40q4rb1w7on. * The number of neurons after the feature transformer increased two-fold, to 1024x2. This is possibly mostly due to the now very optimized feature transformer update code. * The number of neurons after the second layer is reduced from 16 to 8, to reduce the speed impact. This, perhaps surprisingly, doesn't harm the strength much. See https://docs.google.com/document/d/1gTlrr02qSNKiXNZ_SuO4-RjK4MXBiFlLE6jvNqqMkAY/edit#heading=h.6qkocr97fezq The AffineTransform code did not work out-of-the box with the smaller number of neurons after the second layer, so some temporary changes have been made to add a special case for InputDimensions == 8. Also additional 0 padding is added to the output for some archs that cannot process inputs by <=8 (SSE2, NEON). VNNI uses an implementation that can keep all outputs in the registers while reducing the number of loads by 3 for each 16 inputs, thanks to the reduced number of output neurons. However GCC is particularily bad at optimization here (and perhaps why the current way the affine transform is done even passed sprt) (see https://docs.google.com/document/d/1gTlrr02qSNKiXNZ_SuO4-RjK4MXBiFlLE6jvNqqMkAY/edit# for details) and more work will be done on this in the following days. I expect the current VNNI implementation to be improved and extended to other architectures. The network was trained with a slightly modified version of the pytorch trainer (https://github.com/glinscott/nnue-pytorch); the changes are in https://github.com/glinscott/nnue-pytorch/pull/143 The training utilized 2 datasets. dataset A - https://drive.google.com/file/d/1VlhnHL8f-20AXhGkILujnNXHwy9T-MQw/view?usp=sharing dataset B - as described in https://github.com/official-stockfish/Stockfish/commit/ba01f4b95448bcb324755f4dd2a632a57c6e67bc The training process was as following: train on dataset A for 350 epochs, take the best net in terms of elo at 20k nodes per move (it's fine to take anything from later stages of training). convert the .ckpt to .pt --resume-from-model from the .pt file, train on dataset B for <600 epochs, take the best net. Lambda=0.8, applied before the loss function. The first training command: python3 train.py \ ../nnue-pytorch-training/data/large_gensfen_multipvdiff_100_d9.binpack \ ../nnue-pytorch-training/data/large_gensfen_multipvdiff_100_d9.binpack \ --gpus "$3," \ --threads 1 \ --num-workers 1 \ --batch-size 16384 \ --progress_bar_refresh_rate 20 \ --smart-fen-skipping \ --random-fen-skipping 3 \ --features=HalfKAv2_hm^ \ --lambda=1.0 \ --max_epochs=600 \ --default_root_dir ../nnue-pytorch-training/experiment_$1/run_$2 The second training command: python3 serialize.py \ --features=HalfKAv2_hm^ \ ../nnue-pytorch-training/experiment_131/run_6/default/version_0/checkpoints/epoch-499.ckpt \ ../nnue-pytorch-training/experiment_$1/base/base.pt python3 train.py \ ../nnue-pytorch-training/data/michael_commit_b94a65.binpack \ ../nnue-pytorch-training/data/michael_commit_b94a65.binpack \ --gpus "$3," \ --threads 1 \ --num-workers 1 \ --batch-size 16384 \ --progress_bar_refresh_rate 20 \ --smart-fen-skipping \ --random-fen-skipping 3 \ --features=HalfKAv2_hm^ \ --lambda=0.8 \ --max_epochs=600 \ --resume-from-model ../nnue-pytorch-training/experiment_$1/base/base.pt \ --default_root_dir ../nnue-pytorch-training/experiment_$1/run_$2 STC: https://tests.stockfishchess.org/tests/view/611120b32a8a49ac5be798c4 LLR: 2.97 (-2.94,2.94) <-0.50,2.50> Total: 22480 W: 2434 L: 2251 D: 17795 Ptnml(0-2): 101, 1736, 7410, 1865, 128 LTC: https://tests.stockfishchess.org/tests/view/611152b32a8a49ac5be798ea LLR: 2.93 (-2.94,2.94) <0.50,3.50> Total: 9776 W: 442 L: 333 D: 9001 Ptnml(0-2): 5, 295, 4180, 402, 6 closes https://github.com/official-stockfish/Stockfish/pull/3646 bench: 5189338
Copyright 2011–2024 Next Chess Move LLC
Feedback & Support Terms of Service Privacy Policy Desktop Version