Automated LLM evals are the shiny new object everyone wants to include in their AI applications. We come up with sophisticated rubrics to judge the application behavior and spend time and effort building the harness structure into the continuous delivery pipeline.
But in my opinion, the precursor to automated LLM evals is the same thing that powered good machine learning models, which is looking at the data. Before you jump into creating complex harnesses, you have to look at how your AI application is performing in the real world. Only by looking at the traces of your application will you be able to refine the failure taxonomies. Spending time creating or onboarding evaluation frameworks will be for nothing if the work has not been put into understanding the various failure scenarios of the application.
When building an AI application my focus in the first few weeks of development is robust instrumentation. Capturing the input to the LLM workflow, the preprocessing before the LLM call, tool call inputs/outputs, LLM outputs among other things. As I approach a beta release I also set up a protocol for manual verification of the traces. It would be as simple as processing all the sampled conversations into an annotation queue and having the team review and rate the conversations.
So, prioritise manual review over spending time building complex eval harnesses that you are not confident about.