Launched this week
Soup CLI
Fine-tune an 8B LLM on a 4 GB laptop GPU
91 followers
Fine-tune an 8B LLM on a 4 GB laptop GPU
91 followers
LoRA keeps the base model frozen: read, never written. So Soup keeps it in system RAM and streams it into the GPU one decoder layer at a time. Peak VRAM becomes one layer instead of the whole model. Measured on an RTX 3050 Laptop 4 GB: Llama-3.1-8B trains at 119.6 tok/s in 3.32 GB peak. One YAML, one command. SFT, DPO, GRPO, KTO, plus eval, gating and export. Apache-2.0. Every number is published, including the ones I measured and threw away.
Soup CLI
Publishing the ones that turned out wrong is the detail that stands out. Ran into a smaller version of that this month, checked AI Overview traffic on three real sites expecting some signal and got zero across the board, and the honest move was publishing zero instead of only writing up the wins. Question on the streaming approach: does the correctness check, streamed logits matching a resident run, hold up against an already 4-bit quantized base model, or is that validated against full precision only so far?
Soup CLI
@abdullah_javaid3 Yep, the 4-bit case is validated too. The streamed logits match the resident run with the same 4-bit quantized base model, not just full precision. That check is part of the correctness protocol.
@makazhanalpamys The H100 bug is the same shape I keep hitting with browser automation this week. A click reports success, or a loss curve looks healthy, and the state underneath is wrong anyway. Only forcing an exact match against a known good baseline catches it, spot checking the metric that is supposed to prove correctness does not. What made you check logits at all instead of trusting the loss curve, was there a tell before the H100 run or did it only show up once you had the compute to actually compare runs at that scale?
@makazhanalpamys The layer-streaming idea is neat, but the correctness protocol is the part that won me over — requiring streamed-run logits to exactly match a resident run, because "cut the autograd path and the loss still goes down" is exactly the kind of silent failure most tools never check for.
And publishing the H100-found bug (gradients silently wrong above a layer size while the loss curve looks healthy) with a reproducer, in your own released code, is rarer than the optimization itself. That's how benchmarks earn trust.
119.6 tok/s in 3.32 GB on a 3050 laptop is a genuinely useful floor for people who want to iterate locally before paying for cloud GPUs 👌
Soup CLI
@akbar_b Appreciate it! That was exactly the goal: make the memory optimization useful without sacrificing correctness, and publish the failure cases when we find them.