Summary
Two candidates both produce perfectly schema-conforming output, but one confuses the service-period start with the invoice date roughly once in ten cases while the other gets dates right and costs many times more per call — so schema conformance cannot separate them, and a recommendation reporting only “accuracy” ignores the cost.
Remodel sits on each Agent node and begins with a task profiler that reads the node’s actual prompts, schemas, knowledge domain, policies and constraints, so the evaluation is of this extraction task rather than of extraction in general. A candidate selector filters frontier, open-weight, off-the-shelf small and fine-tuned models, taking prior preferences into account, and evaluation cases come from generated input-output pairs (which make evaluation possible before a workflow has history) plus human-reviewed traces (the more valuable source, since a reviewer’s date correction is exactly what exposes the weak candidate).
Grading is deliberately split between deterministic validators, which answer exactly specified conditions such as schema conformance and line-item sums, and a model judge that compares against a stronger teacher model’s reference for qualities rules cannot capture — with the honest caveat that neither establishes correctness: valid formatting is not accuracy, and a judge is only as good as its reference, so a teacher that shares the date confusion will reward candidates that repeat it.
The recommendation is made interpretable in plain terms — task fit and confidence in it, cost per call, the team’s own cost of error and willingness to pay per unit of quality — and then ranked, naming recommended, best-quality, best-value and cheapest-viable picks; a technical inset adds bootstrap confidence intervals, a probabilistic Pareto frontier over coupled cost, and a pure-function engine that yields reproducible, explainable rankings.
Fine-tuning is offered under narrower conditions than the offer suggests: a bounded recurring task, enough suitable data, enough volume, and a comparison that includes training compute, evaluation, serving and maintenance. Versioned datasets, separated training and held-out splits, and the same acceptance process for new checkpoints are what keep the evaluation from grading its own homework.