Launching today
LoRA Speedrun
A verified speedrun leaderboard for LLM fine-tuning
8 followers
A verified speedrun leaderboard for LLM fine-tuning
8 followers
Fine-tune the same model to the same score on the same GPU — fastest training run wins. Every record is re-run 3x with fresh seeds by a referee before it counts, so no self-reported numbers. The baseline took 11:57; three days and three verified records later, the community has it at 1:44. Attempting is free.
A public log of every submission attempt would be great, even just the timestamps and which configs were tried even if they failed verification. Right now you only see the records that stuck, but seeing the misses is honestly where half the learning happens for people trying to figure out what actually works.
@cafertrf7 Agree that the misses are half the value. What exists today: every official verification runs publicly in the PR thread, pass or fail, and the report stays up either way. And record notes are expected to document rejected variants — record #3's notes list four things that didn't work, with numbers (1.25 epochs overfits the pruned
subset, Liger wasn't needed at that batch size, etc). What I can't capture is people's private iteration on their own GPUs before they submit. But a dedicated verification-log page listing every official attempt including failures is a good idea — adding it as the fails accumulate (so far, embarrassingly, everyone's passed).
One thing that would be great is seeing the actual hyperparameter differences between top runs, like what learning rate or batch size changes helped people cut their time down. Right now the leaderboard just shows the numbers but not what got them there, which makes it harder to iterate.
@eray3okc Fair hit — the data was in the repo but the leaderboard hid it. Fixed: every row now links the exact config and notes. The short version of the chain so far: baseline (2e-4, 2 epochs, full data) → sequence packing + completion-only masking → 1 epoch
at 4e-4 on 3k examples → shortest-4k subset + a chunked cross-entropy that skips materializing the logits. Honestly it's been mostly data pipeline and schedule, not classic hyperparameter tuning — which surprised me too.
honestly love that the runs get verified by a referee, that takes out all the guesswork. one thing that would be cool is a public log of what configs each record used, like the actual hyperparameters and dataset choices, so others can try to beat them instead of just chasing a number
@necdetmjbg That log actually exists for every record — each one ships its full config.yaml (every hyperparameter, including dataset subset choices) and a NOTES.md that has to explain what changed vs the previous record before I'll merge it. But you're right it was buried in the repo. Just added config + notes links to every row on the leaderboard, refresh and they're there.