Launched this week
Velrim
Extract data from documents. Know which fields to trust.
2 followers
Extract data from documents. Know which fields to trust.
2 followers
Extraction APIs return valid JSON even when the value isn't in the document. Nothing throws. The wrong number just lands in your database. Velrim gives every field a confidence score you can check, so you know what to use as-is and what to send to a human. Zod or Pydantic schema in, typed data with bounding boxes out. We benchmarked ourselves against Gemini, ChatGPT and Mistral on 124 real documents and published all outputs, including where we lose. $0.02/page. ~500 pages free, no card.
Hi PH, founder here. Velrim is a document extraction API that returns a confidence score per field. It runs on Gemini, so we put ourselves in the table next to the model we run on and four other setups to see how we hold up.
The question was whether the confidence layer measures anything. I.e. when the confidence score says 80, are about 80 out of 100 such fields right, and does the score drop when we invent a value.
Writeup: https://velrim.com/research/fabrication-on-absent-fields
Repo with every raw output and the scorer: https://github.com/velrimhq/velrim-eval
Archive: https://doi.org/10.5281/zenodo.22233430
Notes on the method:
1. 124 real documents from CORD, DeepForm and VRDU. Every model has seen these in pretraining, and that cuts against every setup equally. It's listed in the disclosures in the repo.
2. Before the benchmark ran, I hand-checked all 142 "this field is absent" labels against the pages. 40 were wrong, the value is printed right there. 46 cells struck for every system, so the fabrication numbers are out of 96 absent fields, not 142.
3. Gaps under 4 to 10 points per document type are noise with 124 documents.
4. The run cost about $39. The manifest from each start is in the repo, except the second one, which got overwritten before we archived it.
5. Two vendors that return a confidence number aren't in the table. Their terms ban benchmarking. We asked for consent on July 12 and heard nothing since.
Next up: round two with those two vendors if consent arrives, and a confidence column for OpenAI's model built from its token probabilities.
Rescoring the published outputs is free and needs no API keys. Happy to answer questions.