Reading note · Agent systems

Towards automating eval engineering

LangChain’s Eval Engineering Skill turns a repository and its agent traces into proposed, executable evals. The useful part is that the process still makes room for the builder to decide what is actually worth measuring.

Read Viv’s original article on X ↗

From agent behavior to an eval

The skill inspects the agent surface, uses traces as observed contracts, proposes abilities, and iterates with the user before producing the final task.

  1. 01 Map the agent

    Read prompts, models, tools, skills, hooks, data, and services.

  2. 02 Mine traces

    Observe real arguments, results, errors, requests, and failures.

  3. 03 Interview the builder

    Choose the abilities that matter and decide what runs live or simulated.

  4. 04 Build and rerun

    Create the task, inspect the result, revise the verifier, and compare again.

The output is a reproducible Harbor task

evals/<task-id>/
├── task.toml
├── instruction.md
├── environment/
└── tests/
  • Instruction

    The task the agent is asked to complete.

  • Environment

    A controlled setup with the tools, data, permissions, and state the task needs.

  • Verifier

    The scoring logic that decides whether the intended capability was demonstrated.

The verifier is part of the experiment

A first passing score is not enough. The article recommends inspecting both sides of the result to see whether the task measured the thing the builder actually cared about.

Agent trajectory

Messages, tool calls, actions, artifacts, and the path taken to complete the task.

Verifier trajectory

Evidence, reasoning, scoring, and the assumptions that turned behavior into a reward.

What sticks with me

The strongest part of this approach is that it does not pretend the agent can discover the right measurement in one pass. Repository structure and traces provide evidence, but the builder still has to say which capability matters and whether the environment represents the real work.

An eval becomes useful when it captures a real capability in a reproducible environment, not when it simply produces a clean score.

I keep coming back to the maintenance question. I want failures and unexpected behavior to become durable tests, but I do not want another layer of configuration that grows faster than the work it is protecting. The interesting version is a small loop where real traces identify the few evals that have earned a permanent place.

What I want to watch