The author developed a custom tool, named llm-judge, to evaluate and select language models for specific tasks. The tool is implemented as a Go binary and functions by running benchmarks as separate subprocesses, allowing for the collection of metrics and comparison of model performance across different configurations.
Benchmarks are categorized into three groups: generative, embedding-based, and decision-based. Generative tasks use metrics like F1 and ROUGE to compare model output against reference texts. Code generation is evaluated via pass_rate, which checks if the code executes successfully against expected test cases.
Decision-based benchmarks utilize metrics such as accuracy, Brier score, and Mean Absolute Error (MAE) to assess choices, confidence levels, and scale-based ratings. For embedding tasks, the tool measures semantic similarity using Spearman correlation and nDCG for ranking relevance.
Beyond primary scores, the system tracks operational metrics including response latency, token consumption, and output stability across multiple samples. The author emphasizes that these metrics help transition from selecting a 'best' model to optimizing the combination of model, task, cost, and speed.
The agent component of the tool is used for final analysis and interpreting results rather than acting as a judge itself. The author notes that subjective evaluation remains important, as models can exhibit noise due to temperature settings, and suggests prioritizing deterministic solutions over LLMs whenever possible.
