Solo, May 2025

Large-scale data mining pipeline

100M-event capacity benchmark

Capacity benchmark

2…d5 +0.51

2…d5

+0.51, the engine’s eval after 2…d5. The seam rests there.

Result and evidence

  • 100M-event capacity benchmarkCapacity benchmarkPipeline capacity, not sustained traffic.

The case

Decision
PySpark for features, and a Deep Interest Network on PyTorch DDP for scoring.
Constraint
A failed hour resumes from the last completed slice.
Considered and rejected
Restarting the whole hour.

Architecture

  1. PySparkfeatures
  2. PyTorch DDPDeep Interest Network
Limitations
Cluster size, input distribution and runtime were not recorded.

PySpark · PyTorch DDP · Deep Interest Network

Next, 4…Nf6MirrorFi