5…Bb6
+0.27, the engine’s eval after 5…Bb6. The seam rests there.
Result and evidence
- 70B → 3B studentControlled evaluationNo speed-up claimed.
What was built
- A 70B teacher distilled into a 3B student using QLoRA, DeepSpeed and FlashAttention.
- Limitations
- No speed-up is claimed, because tokens/second, hardware and batch size were not measured.
PyTorch · QLoRA · DeepSpeed · FlashAttention
- The source is in a private repository.
- Back to all work