Back to AI Recipes

Evaluator-Optimizer

WorkflowLoops

Workflow

Generator and evaluator models refine outputs through feedback loops.

Summary

The Evaluator-Optimizer pattern implements an iterative improvement loop where a generator LLM creates solutions, an evaluator LLM assesses them against specific criteria, and then the generator refines the solution based on feedback. This cycle continues until the solution meets the desired quality threshold or reaches a maximum number of iterations.

How it works

  1. Generate: Producer model creates initial solution
  2. Evaluate: Critic model scores against criteria
  3. Feedback Loop: If insufficient quality, return feedback to generator
  4. Refine: Generator produces improved version
  5. Terminate: Stop when threshold met or max iterations reached

Evaluation dimensions

  • Accuracy: Factual correctness, mathematical precision
  • Style: Tone consistency, voice adherence
  • Completeness: Coverage of required elements
  • Safety: No harmful content, bias detection

Use cases

  • Content creation where quality and adherence to specific criteria are important
  • Problem-solving tasks that benefit from iterative refinement and critical feedback
  • Creative writing with specific style, tone, or structural requirements
  • Code generation that needs to meet specific performance or style guidelines
  • Educational content that requires accurate information and appropriate difficulty levels
Code View

Evaluator-Optimizer Implementation

// Evaluator-Optimizer recipe using OpenAI
// Install: bun add openai

import OpenAI from "openai";

async function main() {
  const input = "Add your prompt here.";
  const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
  const system = "You are a senior AI engineer and technical writer. Apply the agent recipe to structure the reasoning and produce a useful result. Recipe: Evaluator-Optimizer. Description: Generator and evaluator models refine outputs through feedback loops. Focus: Workflow Provide actionable, implementation-ready guidance.";
  const user = `Request: ${input}`;

  const openaiResponse = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [
      { role: "system", content: system },
      { role: "user", content: user },
    ],
  });

  const openaiText = openaiResponse.choices[0]?.message?.content?.trim() ?? "";

  console.log(openaiText);
}

main().catch((error) => {
  console.error(error);
  process.exitCode = 1;
});