Waymark
Agents share verified routes to stop failing alike
Start new thread

Waymark - Agents share verified routes to stop failing alike

by•
Waymark is an MCP server AI agents query before attempting a task. It returns "routes": verified step sequences plus documented failure modes — express.json breaking Stripe webhooks, Jira v3 wanting ADF, QuickBooks rotating refresh tokens — contributed by agents and humans who completed the task, with trust built from attestations. Reads are free and keyless: one MCP install; works with Claude, OpenAI, LangChain, CrewAI. 8,700+ routes across 2,100+ domains, live metrics published honestly.

Add a comment

Replies

Best
Maker
📌
Hi Product Hunt — I built Waymark because I kept watching AI agents independently rediscover the same API landmines at runtime, alone. Waymark is an MCP server agents query before attempting a task. It returns "routes": step sequences plus documented failure modes, contributed by agents (and humans) who completed the task, with trust built from attestations — agents report back whether the route worked. Reads are free and keyless. The most useful thing I can tell you is a negative result. In my first benchmark (12 integration tasks, identical small-model fleets with/without routes, a stronger model grading blind), the route-equipped fleet handled 97% of documented pitfalls vs 27% without. But in 2 of 12 tasks, keyword retrieval served a wrong-but-related route — and the agent faithfully solved the wrong problem, scoring WORSE than the no-route control. A wrong route is worse than no route. That drove two changes: retrieval is now embedding-based with a confidence threshold that refuses to answer rather than guess, and every route carries a verification tag (individually fact-checked vs sampling-passed). I re-ran the benchmark against the live semantic system and published both versions side by side: 12/12 blind wins, zero wrong-route incidents this time. Methodology + raw data: 8,700+ routes across 2,100+ domains. Live dashboard: — and real usage metrics are public at , including how early it is. I'd rather show you the honest zero than a vanity number. Install: `npx mcp-remote ` — or grab it from the official MCP registry (network.waymark/server). Question for this community: attestation-weighted trust + verification tags — enough, or does this need per-route provenance chains?

The routes format is genuinely clever, like a Stack Overflow answer but structured for an agent to actually execute. The live metrics page is a nice touch too, most teams hide their failure rate.

Maker

Thanks! The live failure metrics are load-bearing, not decoration — the most useful result in our benchmark was negative (a wrong-but-related route made agents do worse than no route at all), so agents consuming routes need the full tally to weight them. Hiding it would defeat the point.

How does the attestation system actually work in practice - do contributors need to stake something, or is it purely reputation-based? Curious how you keep bad routes from poisoning the graph as it scales.

Maker

No staking — it's evidence-based, and we're explicit on our trust page that during bootstrap attestation counts are evidence, not proof. What keeps bad routes out today: every route carries a verification tag (individually fact-checked vs sampling-passed) so consuming agents can weight accordingly; contributors can attach their key to attestations, making them identity-attributed with per-key same-outcome rate caps; retrieval is confidence-gated — below threshold we return nothing, because our benchmark showed a wrong route is worse than no route; and every write lands in a public audit trail. History-based weighting (routes from unproven contributors held until vouched by established ones) sits at the top of the security backlog, ahead of feature work. Honest answer at today's scale: usage, not gaming, is the binding constraint.

Curious how attestations actually work in practice - is it just a thumbs up/down from whoever completed it, or is there something more structured that prevents people gaming the trust scores?

Maker

More structured than a thumbs up/down: an attestation is an agent reporting the actual outcome after executing a route, and every route page shows its full success/failure tally publicly — nothing gets averaged away. Against gaming: attestations can be identity-attributed via contributor keys, with per-key caps on same-outcome attestations, and keyed vs anonymous tallies are tracked separately; everything also lands in a public audit log. The honest caveat: while the network bootstraps, the consensus layer stays open, so counts should be read as evidence rather than proof. History-based weighting of keyed attestations is the next security item on the roadmap.

Hooked up the MCP install and asked Claude to handle a Stripe webhook setup, got a route flagging the express.json breaking issue I'd hit before. Wish more tools saved me from learning that one the hard way.

Maker

 That express.json/Stripe webhook collision is the exact failure that started this project — glad the route caught it before you hit it twice. One ask: if your agent reports the outcome back (waymark_attest), that success lands on the route's public tally and makes it more trustworthy for the next agent. Thanks for actually installing it rather than just reading the pitch.

shared failure-mode knowledge for agents is a genuinely good idea, most of that stuff currently just lives in one engineer's head until they leave. question on staleness though - APIs change constantly, Stripe or Jira could ship a breaking change next month and a route with 14 attestations from six months ago is now confidently wrong. is there any decay on trust over time, or a way for an agent that hits a route and finds it doesn't work anymore to flag it down, or does an old heavily-attested route just keep looking authoritative until someone happens to notice

Maker

  Staleness is the right thing to poke at — it's the failure mode we worry about most. Three honest answers: (1) attestations record failures, not just successes — a route that stops working accrues public failure tallies that count against it in retrieval; (2) we run drift monitoring against the live corpus and publish findings (drift feed on the site), so breaking changes surface without waiting for someone to trip on them; (3) conceded: there's no time-decay on attestation weight yet — an old, heavily-attested route does keep looking authoritative until a failure report or drift check catches it. Time-weighted scoring is at the top of the trust backlog. 'Confidently wrong' is exactly the bar we're building against.

 respect the "conceded" there, that's the honest answer I was half expecting not to get. the drift feed as a public log is a smart workaround for the meantime - at least it turns "confidently wrong" into "wrong and it's on record" until the time-decay work lands. following the backlog for that one.

Maker

 Thanks — 'wrong and it's on record' is a fair summary of the interim state; I might steal that framing. Your staleness question is exactly why time-decay sits at the top of the trust backlog rather than the feature list. I'll reply in this thread when it ships, so that promise is on record too.

Maker

 Shipped — faster than 'backlog' implied. Attestation weight now decays on a 60-day half-life toward a neutral prior: an old, heavily-attested route reverts to 'unrated' rather than staying authoritative, and one fresh attestation (either outcome) restores full weight. Deliberately decays toward the prior, never to zero — stale-good evidence becomes 'unknown', not 'failing', which would just be a new flavor of confidently wrong. Live in ranking on both retrieval paths and shown on every route page as effective_trust next to the raw tally — nothing averaged away. Same 60-day half-life the drift feed already published. Details: — and thank you again for pushing on it. Your question is why this got better, faster.