Agent classes / Eval and judge agent / LLM/agent tracing
LLM/agent tracing for a eval and judge agent
Updated 2026-10-27 (newest stored date on this page) · score asOf 2026-09-14 (stale)
The class: Scores a trace or a patch against a rubric, without the gold label in its prompt, and records the harness version.
- Gold labels kept out of the judge prompt
- Harness version stored next to the score
Depot Pick: Langfuse
When not to use it: Deep LangChain shop — LangSmith. Want proxy in 10 minutes — Helicone. OSS-only — Phoenix. Python/PydanticAI traces + app OTel together — Logfire. Free eval+trace SKU — HoneyHive.
Runner-up: Helicone.
No single-unit sourced price comparison for this job (no_comparable_unit_for_job).
Failure modes this class and job share (6)
- Observability gap: Alert on a drop in trace ingestion and on certificate expiry for the trace host. Do not treat a missing trace as a successful quiet run.
- Benchmark protocol mismatch: Store assisted-or-not, subset, and harness version beside every cited score. Refuse a ranking that mixes those fields.
- Eval contamination: Keep gold labels out of the agent prompt. Store the harness version beside the score. Do not compare a number across a benchmark revision without saying the revision changed.
- Inter-agent misalignment: Log the assigned subtask next to the worker's stated result. The lead rejects a result that does not match the assignment.
- Non-determinism: Run the task more than once and compare the end state, not a single transcript. Record the trial count next to any success rate you cite.
- Weak verification: Success requires an end-state check the agent does not grade for itself: tests, a database diff, or a separate judge that cannot see the gold label.
Free options this class uses (1)
- Langfuse self-host (dev_tool, oss) — source, asOf 2026-10-25