Update network: faster training with AdamW optimizer
The resulting net is less sparse and thus a little slower than before, but the better eval prevails at VVLTC. The reduction in training time aids future net experiments.
The training occurred at https://github.com/vondele/nettest/pull/448
...using Tony's AdamW optimizer PR: https://github.com/official-stockfish/nnue-pytorch/pull/511
The main goal was to reduce resource usage during training.
As the currently used ranger21 variant uses lookahead and "stable weight decay", the training with adamw is around 5% faster (no stable weight decay) and uses less memory during training and for checkpoints (no lookahead).
The other effects came as somewhat of a surprise.
It is unclear what exactly causes the lower sparsity, around 50-52% vs 60+% which does cause a significant slowdown. It is also unclear what exactly causes the apparent better scaling.
Failed STC nonreg: https://tests.stockfishchess.org/tests/view/6a6e1580c054e285ae029c2e
LLR: -2.94 (-2.94,2.94) <-1.75,0.25>
Total: 376800 W: 97403 L: 98082 D: 181315
Ptnml(0-2): 1226, 44985, 96614, 44392, 1183
Nodestime STC nonreg: https://tests.stockfishchess.org/tests/view/6a72310d2cf557d58a2e1ced
LLR: 2.96 (-2.94,2.94) <-1.75,0.25>
Total: 22812 W: 6130 L: 5891 D: 10791
Ptnml(0-2): 89, 2537, 5920, 2766, 94
LTC: https://tests.stockfishchess.org/tests/view/6a7205ef2cf557d58a2e1cb8
Ptnml(0-2): 135, 31150, 81387, 31468, 145
LLR -0.64 <0.00,2.00>
LLR: 7.72 <-1.75,0.25>
Passed VLTC: https://tests.stockfishchess.org/tests/view/6a7a7b595ebead9b2e786ee7
LLR: 2.95 (-2.94,2.94) <0.00,2.00>
Total: 139242 W: 36650 L: 36248 D: 66344
Ptnml(0-2): 13, 13245, 42711, 13631, 21
Passed VVLTC: https://tests.stockfishchess.org/tests/view/6a72cbb02cf557d58a2e1e40
LLR: 2.95 (-2.94,2.94) <0.50,2.50>
Total: 71720 W: 18989 L: 18641 D: 34090
Ptnml(0-2): 6, 6229, 23046, 6569, 10
closes https://github.com/official-stockfish/Stockfish/pull/7043
Bench: 2522345