ZooData - The data layer for AI agents

@dipankar_sarkar You've surfaced something we should carve into the docs, so let's state the contract plainly: absence ≡ not extracted. Whether the page truly had nothing or we failed to grab it — those two genuinely can't be distinguished today, so we don't pretend to: we won't hand you an explicit null that implies "we checked, the page really has none" (that's a guarantee we can't back). Absence means "no value obtained here." One meaning, full stop.

Under that contract, the "shape drift" is milder than it looks: the full field set per page_type is fixed — what varies is only which fields carried a value this time, and absence has exactly one meaning. Your defensive code collapses from "guess two meanings per field" into one global rule: not present = not obtained. Write it once, done.

On explicit-null stable schemas — we considered it, and landed here: since null and absence would carry the same information, sending nulls explicitly buys nothing but token cost. And the first-class consumer of this API is an agent: LLMs are natively robust to absence — "no price in the response" already is the complete information. It's strongly-typed glue code that needs every field present — and that's exactly the layer we hope keeps getting thinner in the agent era. So yes: long term, we're on team minimal payload.

That said, "should an agent-native API look like a traditional API" is a genuinely fun debate — if you disagree, keep this thread going, or take it long-form at support@zoodata.ai.

@amir_mehrabi Great question — and it's exactly why formats are composable on our side. A single call can return json and markdown together (add rawHtml if you want full fidelity): json is for acting now, markdown is for "might matter later." The markdown is a page-faithful rendering of the full content — the long tail the schema didn't catch (a footnote, a promo clause, some odd badge) is still in there, not thrown away.

The agent pattern we see looks like this: act on the fields immediately; stash the markdown cheaply outside the context window (it's just text); and the day a decision actually needs the long tail, feed that one page back to the model to read once. That's the full meaning of "structured-first" — not structure-only, but turning the model's prose-interpretation from "re-read on every hop" into "read once, on demand." What we eliminate is repeated interpretation, not information.

And a footnote: requesting an extra format in the same call doesn't change the price — one call, one price. Keeping that "might matter later" escape hatch costs you nothing at the margin.

@sabber_ahamed Neither 😄 The real answer: detected on the spot, degraded honestly, re-learned automatically. Every extraction passes core-field validation — the moment a redesign breaks structure, validation fails and the result is never stamped success; unreadable fields are absent (absence ≡ not extracted), never invented. The structure then re-learns the new layout — first pass slower, speed recovers, self-healing with zero action from you. And every drift trigger lands with attribution on our dashboards — "noticing" is our job, not your downstream data's.

@sabber_ahamed  Same price. The pricing axis is what you take away, not what compute we burn. One scrape, one price — whether that was a featherweight static page or a heavyweight SPA needing a full browser session never appears on your bill. Rendering weight is our cost problem, same direction as the thrash cost discussed above: it hits our margin, not your credits.

Why we're firm on this: with weight-tiered pricing, you can't know which tier a URL lands in before you call it — that's decided by the target site's tech stack, which you neither control nor observe. An agent's spend already carries one naturally floating term (LLM tokens); the data layer doesn't need to add another. A flat unit price leaves this part of your spend with exactly one variable: call count.

One honest footnote: heavier pages are slower (render time is real), just not more expensive. Latency floats; price doesn't.

@sabber_ahamed  Good one. Per-request judgment is deliberate — during an A/B window old and new layouts interleave, so each request is handled on the page it actually got; a "lock-in switchover" would feed half the window to the wrong template. As for thrash cost: it points at us, not you — your price is flat per call, and the internal path never reaches your bill. How we protect ourselves (throttling, when things lock in) is an internal tuning problem. The contract is one line: thrash never reaches your results, and never reaches your bill.

@dipankar_sarkar Honest answer: keys can still drop today — what's "fixed" is each page_type's possible field set (docs level), not the wire payload (sparse; absent means gone). But a fixed full set makes your fix a one-liner: hydrate missing keys to null against the known set at the boundary, and everything downstream sees a stable shape — N guards collapse into 1 hydration. Your layering — full keys fix shape, sourced flag carries honesty — is the right decomposition, exactly where the Jernej thread converged as the opt-in direction.

@omri_ben_shoham1  Thanks! The three lanes come from a simple inversion: instead of asking "how much error detail can we expose," we asked "where does your agent's control flow actually branch" — it branches three ways, so the API only needs to be clear about three things. Every extra lane would just be a branch you write and never take. If you ever hit a case where three lanes aren't enough, come back and challenge that claim

@nitish_garg4 On freshness, the answer is clean: two endpoint families, two cadences, explicitly split — never dressed up as one.

- The analytics layer (product/market search, keywords, review analysis — built on a large historical corpus) refreshes on a daily cycle. Those questions are fundamentally "what has this market been doing" — daily is the honest granularity, and dressing it up as "live" is the pretending the industry usually does.

- The realtime layer (realtime product/review endpoints) does live on-demand collection per request, no cache — you get the page's displayed state at that moment.

Our stance: freshness is a labeled property of each endpoint, not a marketing word. Pick by the question: market trends → analytics (fast, cheap, daily); "what does this product page say right now" → realtime (fetched on the spot).

One small aside so the meme doesn't spread: billing is actually a flat price per call, not a per-field meter — the "pay only for what you need" part is real, but it lives in responses carrying only meaningful fields, i.e. savings on your downstream LLM tokens. That's where the real money is.

@omri_ben_shoham1  Someone took the challenge — welcome 😄 The public-surface reality first, then the shape we believe in:

Today's batch-like surface on the public API is search deep-scrape (one search, each result page fetched individually). Its behavior: every result is processed independently, and failures never contaminate successes — but honestly: failed records are dropped, not returned as error entries. We over-fetch candidates beyond your limit to backfill, so you usually still get your full count, and billing counts only returned records — failures cost nothing. Summary: what you receive is clean, what failed is traceless and free — but no, it's not a per-record error report.

A batch endpoint where you submit a list of IDs doesn't exist on the public API today. And the shape you describe — per-record {id, data | error} with succeeded/failed counts on top, never a whole-call collapse — is exactly the batch contract we believe in: partial success as a fourth shape, with the three lanes pushed down to the record level and the envelope doing bookkeeping. You've articulated that envelope well — noted.

@wojtekszkutnik Your instinct about the direction is exactly right — it's how we think about the product ourselves: a general extraction layer underneath, vertical workflows growing on top. E-commerce is just the first vertical we grew, not the only shape it takes.

What works today: point the general extraction at those pages. Page types are auto-detected — pricing pages with tiered plan cards are naturally list-shaped (each plan with a name, price, description), and company home/about pages tend to land as entity-type, coming back as structured facts plus the page's outgoing links. No promise that every startup page parses perfectly — but the general layer will likely take you further than you expect. Worth a round with a few real targets.

After that round, whatever the outcome, send us the URLs plus the fields you wished you'd gotten (support@zoodata.ai) — that's literally how verticals get built here: the e-commerce depth endpoints started with users throwing real pages at the general layer. A startup pricing page is a well-bounded page class, and your ask is noted.