OrderTrace Eval is a lightweight offline lab for evaluating agent behavior beyond the final answer. It checks task success, tool choice, arguments, grounding, output format, and constraints from replayable traces, with explainable PASS/FAIL results. Built and release-hardened with GPT-6 Astra.

What real task does your product handle with GPT-6 Astra?

GPT-6 Astra was used through Codex to audit an existing agent-evaluation project, identify release gaps, harden trace and raw-output integrity checks, improve CLI and failure semantics, prevent artifact collisions, expand regression coverage, and prepare a sanitized public release. Astra also helped refine release documentation and validation workflows. Astra is used for development and release hardening, not at runtime; the product itself runs locally and offline with a deterministic MockAgent and rule-based evaluator.

Hi Product Hunt, I’m Arvin. I started building OrderTrace Eval because I found that checking only an agent’s final answer is often not enough. An answer can look correct while the tool choice, arguments, evidence, or failure handling behind it are still wrong. OrderTrace Eval is a small offline evaluation lab that uses replayable traces to check task success, tool selection, tool arguments, grounding, output format, and constraints. For this release, I used GPT-6 Astra to audit the project, harden edge cases, add regression coverage, and prepare the public release. Astra is not used at runtime — the current project runs locally and offline. The project is still intentionally small: it uses a rule-based MockAgent and synthetic order data, with no LLM judge or multi-turn memory yet. I’d really appreciate feedback on the evaluation approach, especially what agent failure cases you think are worth testing next.

What real task does your product handle with GPT-6 Astra?

OrderTrace Eval

Trace-first evaluation for reliable AI agents

Trace-first evaluation for reliable AI agents