DocDot Full Benchmarks

Building powerful, LLM-driven document applications often starts with an accurate and efficient parser. But for many real-world workflows, privacy is non-negotiable—sensitive files must run locally on your laptop, without a single byte touching the cloud.

That local setup also has to be fast. An ideal parser should reach 5 to 10 frames per second, which means a 100-page document can be parsed in about 10 to 15 seconds, fast enough for users to move straight into interactive Q&A. But speed alone is not enough. The parser also has to handle complex layouts, dense tables, and paragraphs that span pages, scanned documents, and multiple languages.

To find out whether existing tools could meet those requirements, we thoroughly benchmarked every open-source parser that could run on Mac hardware. We compared their accuracy, speed, power consumption, and other metrics. The result was straightforward: there is no single model that beats everything. Different models are better at different trade-offs depending on the task.

That insight led us to build DocDot, a tool that helps non-technical users install multiple PDF parsing models on Apple laptops tailored to their needs and then connect those models to downstream AI tools, including Codex, Claude Code, OpenClaw, and Hermes.

Benchmark Dataset

To test parsing quality under realistic and difficult conditions, we built a single-page dataset with 4,231 page samples across 64 document categories, with no more than 100 pages per category.

Domain & Category Coverage:

• Financial & Securities: annual reports, quarterly filings, audit reports, ESG disclosures, IPO prospectuses, bond prospectuses, credit ratings, industry research, fund reports, and exchange disclosures.

• Government & Regulatory: official announcements, laws and policies, administrative documents, public forms, business licenses, and executed contracts.

• Academic & Educational: research papers, academic journals, presentation decks, textbooks, exam papers, and academic posters.

• Publishing & Media: print newspapers, weekly periodicals, magazines, and screenshots of web articles.

• Professional Documents: patents, medical documents, account statements, and investment research records.

Overall, the dataset encompasses Chinese, English, and mixed Chinese-English pages, featuring a combination of natural publishing layouts, office document layouts, and high-density tabular layouts.

Ground Truth: The ground truth data on each page include: manually annotated bounding boxes for each content element (paragraph, table, figure, equation etc.), reading order sequences, and structural table representations.

Annotation Format: Final ground-truth data are structured in Markdown/HTML format. Text blocks are linearized in strict reading order, tables retain full structural hierarchy via native HTML <table> tags, and embedded graphics/figures are extracted as standalone PNG references.

Evaluation Metrics

Our evaluation framework measures both extraction quality and hardware performance.

1. Accuracy Metrics

• Text Edit Distance (Text Edit): character-level normalized edit distance between the parsed text which is NOT inside any table and formula and the ground truth.

• Table TEDS (Tree Edit Distance Score): Tree Edit Distance (TED) between table parsing prediction and ground truth.

• Formula CDM (Character Degree Match): Visual character matching rate between rendered images of parsed formulas and ground truths.

• Overall Quality Score:

2. Performance & Hardware Metrics

• FPS (Frames Per Second): total page throughput per second.

• Peak Memory: maximum RAM usage during execution.

• Avg Total Power: average total system power during parsing, in watts.

• Avg CPU Power: average CPU power consumption, in watts.

• Avg GPU Power: average GPU power consumption, in watts.

• Avg ANE Power: average Apple Neural Engine power consumption, in watts.

• Model Storage Footprint: Total disk space required to download and store the local model weights on disk.

• Model Size on Disk: Total disk space required to store the local model weights on disk.

Experimental Results

Accuracy Metrics

NanoDoc achieves the highest Overall Quality Score (83.66) and Table TEDS (83.19) across the entire benchmark suite. It delivers balanced, well-rounded performance across text, tables, and complex formulas, maintaining superior accuracy without exhibiting weak spots in any single metric.

Model Size on Disk

Runtime Metrics

General PDF

Comprises standard business, government, and publishing documents with no mathematical formulas.

Formula PDF

Comprises dense academic research papers with complex mathematical formulas.

NanoDoc shows a dramatic speed advantage, reaching 9.25 FPS on General PDFs and 4.24 FPS on Formula PDFs. This throughput lead stems from its architectural optimization for the Apple Neural Engine (ANE). In contrast, alternative models rely heavily on GPU/CPU execution paths, leading to memory bloat and severe throughput bottlenecks.