Evaluating agents is different from evaluating traditional applications. To some extent you can verify the behavior of a traditional application since it is deterministic, but for agent evals, the most value you can extract is after the fact, by looking at how the agent behaves in the real world. Of course there are things you can do to begin with, but they will not give you much confidence in the application. So to begin with, you need to see if your application is instrumented well enough for you to take advantage of your agent being out there in the wild.
Before you can begin improving your agent application, you have to make sure that it is ready for manual review. You need to be able to look at chat logs for a session, look at each turn in its context, what tool was used, what the output was, so that you can make use of failure cases. If automated agent evals is the final destination, then proper instrumentation is the foundation.