The Agent Engineering Lifecycle
Build
You have an agent running, but you can’t see what is doing.
Observe (pre-production)
When it fails, you get bad output without explanation. We need visibility (tracing): capturing every LLM call, tool invocation and decision the agent makes.
You run your agent against scenarios, find failures and debug them using traces. It’s the highest ROI activity in agent development
Offline Evals
Automated evaluations that run against a dataset of inputs and expected behaviours. They catch regressions and track progress (is the agent getting better over time?).
Deploy
The agent passes its evals so we deploy to prod. Real users bring real diversity: queries never anticipated, edge cases we never thought, and usage patterns that stress the agent in new ways.
Observe (in production)
- Online Evals and Insights: online evals run automatically on prod traces as they come in, giving dashboards. Insights takes this further, analyzing batches of traces to tell how people are using the agent and where it’s struggling
- Annotations: not every error can be caught by automated evaluation. Sometimes we need to look at traces yourself, verify failures..Annotations let you attach human judgments to traces, then move those annotated traces into datasets to create new online or offline evals.
1. Observability
Tracing: understand what the agent did. We can’t see how a model reasons internally, but we can observe what it does. Tracing captures the complete sequence of steps for a single agent run: we can observe the model’s reasoning, which tools were called with what params, the outputs, timings and cost.
Threads: each user message and agent reply generates its own trace. Those traces are then grouped into a thread, which represents the full conversation history.
Create your Agent Product Requirements Document (PRD): a single page doc that defines what your agent should do, what scenarios should handle, and what success looks like
Playground: your agent routinely fails on the 30th step of a 2min long run. Running it repeatedly is painful and time consuming. Ideally we want to freeze the first 29 steps in place as we iteratively debug the 30th step. We can do this with Playground (LangSmith).
