What would you optimize first to get an in-browser visual-code scanner to 5 FPS?

I’m building TLcube, an open-source 3D barcode — technically 2.5D — that stores data in the relative luminance order of three faces in each cube-like cell.

The generator is working. The harder problem is making the browser-based scanner feel responsive on a phone.

Our practical target is 5 FPS, or one useful decoding attempt every 200 ms. That is a target, not a performance claim.

The first thing we learned was that optimizing a single decode can be misleading.

In an early live test on one Android phone, a Type Y image took only 0.10 seconds in a one-shot static benchmark, while Type O took 1.80 seconds. But median time to first live lock went in the opposite direction: 38.5 seconds for Y versus 3.1 seconds for O.

It was a small test — one device, 38 scan attempts — but it changed our model of the problem. Live experience depends less on the fastest successful decode and more on:

• how many frames fail before one succeeds,

• how expensive those failed frames are,

• and whether the scanner has to rediscover the code from scratch every time.

Here is what we have tried so far:

1. Optimizing each frame in JavaScript

We removed repeated sorting and allocation, replaced an O(n²) clustering path, added cheap-first/full-retry decoding, cached repeated three-face samples, and reused image pyramids between search passes.

The latest cache and pyramid changes reduced median failed-frame cost by roughly 11% on our desktop photo corpus. Useful, but not enough to change the interaction by itself.

2. Measuring the actual pipeline

In one 24-image profiling run, geometry proposal — finding where the code might be — accounted for 97.5% of processing time. Format parsing, error correction, and payload decoding together were comparatively cheap.

In other words, the expensive part is not reading the data after we find the code. It is repeatedly asking, “Where is the code, what is its scale, and how is it rotated?”

3. Testing WebAssembly instead of assuming it would solve the problem

A small Rust/WASM sampling kernel ran about 2.3× faster than its JavaScript equivalent, but the complete pipeline improved by 0.04% or less.

A broader browser experiment produced only about 1.0–1.12× end-to-end improvement while adding another build surface and approximately 58 kB.

We therefore postponed WASM integration. It accelerated a kernel, but not enough of the real bottleneck.

4. Removing an artificial FPS ceiling

The scanner previously waited at least 320 ms between attempts, which imposed a hard ceiling of about 3.1 FPS even when decoding became faster.

We replaced that with an adaptive interval:

max(100 ms, previous frame cost)

That removes the scheduler ceiling and allows up to 10 FPS on a sufficiently fast path. It does not make full detection fast, however.

5. Reusing the previous frame’s pose

This has been the strongest result so far.

A full geometry search took a median of about 2.17 seconds in our corpus. When a valid pose from the previous frame could be reused, the median dropped to 53 ms, with p90 at 92 ms and a measured maximum of 147 ms.

That is comfortably inside the 200 ms budget for 5 FPS.

The remaining problem is continuity. Camera movement, rotation, scale changes, perspective, or a weak previous hypothesis can invalidate the pose. When that happens, the scanner falls back to the expensive full search.

So the question is no longer simply “How do we make every decode faster?”

It is:

How should a browser scanner avoid rediscovering the same code from scratch on every frame — without trusting a stale or incorrect pose for too long?

If you have worked on mobile computer vision, barcode scanning, AR tracking, or browser-based image processing, which direction would you test first?

• A two-loop design: a cheap tracker every frame and a full detector periodically

• Optical flow or corner tracking around the previous finder geometry

• Re-detection inside a predicted ROI instead of propagating the full homography

• A motion model combining the previous homography with device-motion data

• Worker/OffscreenCanvas/WebGPU acceleration after the tracking structure is settled

• A different approach entirely

I’m also interested in how you would define the benchmark.

Would you optimize for time to first lock, sustained FPS after lock, p90 recovery after losing the code, accuracy across rotation and perspective, or thermal and battery cost on real phones?

What would you investigate next, and what failure mode would you guard against first?

The implementation and format are open source if you want to inspect the current approach:

https://github.com/SoliEstre/TLcube