−143.3
I trained a chess evaluator, then made it play the handcrafted one.
The learned evaluator lost to PeSTO.
I published the loss.
30.5%
2 wins · 74 draws · 52 losses
01 / 06
One search,two evaluators.
HANDCRAFTEDPeSTO piece-square tables, by Ronald Friederich. Each column is what a knight is worth on that square.
LEARNEDA small net I trained: 768 inputs, two 256-wide halves, 32, then one number.
The same alpha-beta search on both sides. Only the judgement at the end of each line changes.
HANDCRAFTEDPeSTO piece-square tables, by Ronald Friederich: what a knight is worth on each square.
LEARNEDA small net I trained: 768 inputs, two 256-wide halves, 32, then one number. The same search runs on both sides.
02 / 06
20,000,000
THE DATAPositions labelled with public Stockfish evaluations, from the Lichess evaluation database (CC0). No Stockfish weights are copied.
WHAT FELL AWAYAnything a leaf evaluation should not be asked to judge: 15.4 million positions, 43%.
TO SCALE35,405,130 read. 20,040,000 kept. 40,000 of those held out for testing.
THE DATAPositions labelled with public Stockfish evaluations from the Lichess database (CC0). No Stockfish weights are copied. 15.4 million fell away through the eight filters below.
35,405,130 read · 20,040,000 kept · 40,000 held out
03 / 06
Trainedtwice.
THE SCORE UNDER EACHHow closely the net's judgement matches Stockfish on 40,000 positions it never saw. 1.0 would be perfect.
HOWPyTorch. Quantisation-aware from the second epoch, exported to int8, run in WebAssembly inside a worker. 410 KB.
NOT A COPYA net that copies PeSTO can never beat it. This one agrees with PeSTO at 0.70, well short of 1.0.
THE SCORE BESIDE EACHAgreement with Stockfish on 40,000 unseen positions; 1.0 is perfect. With PeSTO it is only 0.70: not a copy.
04 / 06
0.0 Elo
FIRST, CHECK THE REFEREEHandcrafted against itself: 100 games at 50,000 nodes a move. The same 50 openings, each played once from each side.
DEAD LEVEL17 wins · 66 draws · 17 losses. The harness is fair, so any later gap belongs to the evaluator.
FIRST, CHECK THE REFEREEHandcrafted against itself: 100 games, 50,000 nodes a move, the same 50 openings from each side.
DEAD LEVEL17 wins · 66 draws · 17 losses. The harness is fair, so any later gap belongs to the evaluator.
50.0%
05 / 06
Then thematch.
LEARNED AGAINST HANDCRAFTEDSame search, 50,000 nodes a move, the same 50 openings from both sides. Set to stop the moment the evidence was decisive.
STOPPED AT GAME 128The evidence crossed the line: LLR −2.99, past the −2.94 bound. The learned net lost.
LEARNED AGAINST HANDCRAFTEDSame search, 50,000 nodes a move, the same 50 openings from both sides. Set to stop once the evidence was decisive.
STOPPED AT GAME 128LLR −2.99, past the −2.94 bound. The learned net lost.
−143.3
±35.4 EloControlled benchmark
06 / 06
±60
WHAT FAILED“Learned” was the material count plus the net, and the net was clamped to ±60 centipawns. It could never move a position by more than 0.6 of a pawn.
TWO CAUSES, NEITHER ISOLATEDThe quality of the training data, or this integration. I cannot yet say which.
NEXTDo not rematch this net. A new one passes the data and progress gates first.
WHAT FAILED“Learned” was the material count plus the net, clamped to ±60 centipawns: never more than 0.6 of a pawn.
TWO CAUSES, NEITHER ISOLATEDThe training data, or this integration. Next: no rematch; a new net passes the data and progress gates first.
A loss at the spec cap is still a result.
plinths to scale, in centipawnsplinths to scale
Playit.
THE SEAM IS THE EVALIt sits where the learned net scores the position, and moves with every move.
Your move. At 50,000 nodes the learned net rates this +0.31 for White, and would play e4.Your move. The learned net rates this +0.31 and would play e4. The seam sits at its score.
Loads 410 KB only when you press Start. Runs in a worker and pauses when the tab is hidden.