LangSmith 支持评估 **现有** 以比较方式评估实验。与其一次评估一个输出，您可以对多个实验的输出进行相互评分。在本指南中，您将使用 evaluate() 与两个现有实验 定义一个评估器 和 运行配对评估。最后，您将使用 LangSmith UI 来 查看配对实验.

前提条件

• * 如果您尚未创建要比较的实验，请参阅 快速入门 或 操作指南 开始评估。
• * 本指南需要 langsmith Python 版本 >=0.2.0 或 JS 版本 >=0.2.9.

evaluate() 比较参数

简单来说， evaluate / aevaluate 函数接受以下参数：

除了这些之外，您还可以传入以下可选参数：

定义配对评估器

配对评估器只是具有预期签名的函数。

评估器参数

自定义评估器函数必须具有特定的参数名称。它们可以接受以下任意子集参数：

• * inputs: dict: 对应数据集中单个样本的输入字典。
• * outputs: list[dict]: 每个实验在给定输入上生成的字典输出的两项目列表。
• * reference_outputs / referenceOutputs: dict: 与样本关联的参考输出字典（如果有）。
• * runs: list[Run]: 完整的 Run 两个实验在给定样本上生成的对象。如果需要访问每个运行的中间步骤或元数据，请使用此参数。
• * example: Example: 完整的数据集 Example，包括样本输入、输出（如果有）和元数据（如果有）。

在大多数使用场景中，你只需要 inputs, outputs和 reference_outputs / referenceOutputs. runs 和 example 仅在你需要应用程序实际输入和输出之外的额外追踪或样本元数据时才有用。

评估器输出

自定义评估器应返回以下类型之一：

Python and JS/TS

• * dict: 带有关键字的字典：
• * key，表示将被记录的反馈键
• * scores，即从运行 ID 到该运行得分的映射。
• * comment，它是一个字符串。最常用于模型推理。

目前仅支持 Python

• * list[int | float | bool]: 一个包含两项得分的列表。该列表假定与 runs / outputs 评估器参数具有相同的顺序。评估器函数名用作反馈键。

请注意，你应选择一个与运行标准反馈不同的反馈键。我们建议在配对反馈键前加上 pairwise_ or ranked_.

运行配对评估

以下示例使用 一个提示词 让 LLM 决定哪个 AI 助手回复更好。它使用结构化输出来解析 AI 的回复：0、1 或 2。

• - Python：需要 langsmith>=0.2.0
• - TypeScript：需要 langsmith>=0.2.9

from langchain_classic import hub
from langchain.chat_models import init_chat_model
from langsmith import evaluate

# See the prompt: https://smith.langchain.com/hub/langchain-ai/pairwise-evaluation-2
prompt = hub.pull("langchain-ai/pairwise-evaluation-2")
model = init_chat_model("gpt-5.5")
chain = prompt | model

def ranked_preference(inputs: dict, outputs: list[dict]) -> list:
    # Assumes example inputs have a 'question' key and experiment
    # outputs have an 'answer' key.
    response = chain.invoke({
        "question": inputs["question"],
        "answer_a": outputs[0].get("answer", "N/A"),
        "answer_b": outputs[1].get("answer", "N/A"),
    })
    if response["Preference"] == 1:
        scores = [1, 0]
    elif response["Preference"] == 2:
        scores = [0, 1]
    else:
        scores = [0, 0]
    return scores

evaluate(
    ("experiment-1", "experiment-2"),  # Replace with the names/IDs of your experiments
    evaluators=[ranked_preference],
    randomize_order=True,
    max_concurrency=4,
)

const openai = wrapOpenAI(new OpenAI());

async function rankedPreference({
  inputs,
  runs,
}: {
  inputs: Record<string, any>;
  runs: Run[];
}) {
  const scores: Record<string, number> = {};
  const [runA, runB] = runs;
  if (!runA || !runB) throw new Error("Expected at least two runs");

  const payload = {
    question: inputs.question,
    answer_a: runA?.outputs?.output ?? "N/A",
    answer_b: runB?.outputs?.output ?? "N/A",
  };

  const output = await openai.chat.completions.create({
    model: "gpt-4-turbo",
    messages: [
      {
        role: "system",
        content: [
          "Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below.",
          "You should choose the assistant that follows the user's instructions and answers the user's question better.",
          "Your evaluation should consider factors such as the helpfulness, relevance, accuracy, depth, creativity, and level of detail of their responses.",
          "Begin your evaluation by comparing the two responses and provide a short explanation.",
          "Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision.",
          "Do not allow the length of the responses to influence your evaluation. Do not favor certain names of the assistants. Be as objective as possible.",
        ].join(" "),
      },
      {
        role: "user",
        content: [
          `[User Question] ${payload.question}`,
          `[The Start of Assistant A's Answer] ${payload.answer_a} [The End of Assistant A's Answer]`,
          `The Start of Assistant B's Answer] ${payload.answer_b} [The End of Assistant B's Answer]`,
        ].join("\n\n"),
      },
    ],
    tool_choice: {
      type: "function",
      function: { name: "Score" },
    },
    tools: [
      {
        type: "function",
        function: {
          name: "Score",
          description: [
            `After providing your explanation, output your final verdict by strictly following this format:`,
            `Output "1" if Assistant A answer is better based upon the factors above.`,
            `Output "2" if Assistant B answer is better based upon the factors above.`,
            `Output "0" if it is a tie.`,
          ].join(" "),
          parameters: {
            type: "object",
            properties: {
              Preference: {
                type: "integer",
                description: "Which assistant answer is preferred?",
              },
            },
          },
        },
      },
    ],
  });

  const { Preference } = z
    .object({ Preference: z.number() })
    .parse(
      JSON.parse(output.choices[0].message.tool_calls[0].function.arguments)
    );

  if (Preference === 1) {
    scores[runA.id] = 1;
    scores[runB.id] = 0;
  } else if (Preference === 2) {
    scores[runA.id] = 0;
    scores[runB.id] = 1;
  } else {
    scores[runA.id] = 0;
    scores[runB.id] = 0;
  }

  return { key: "ranked_preference", scores };
}

await evaluate(["earnest-name-40", "reflecting-pump-91"], {
  evaluators: [rankedPreference],
});

查看配对实验

从数据集页面导航到"配对实验"标签页：

!配对实验标签页

点击您想要检查的配对实验，您将进入比较视图：

!配对比较视图

You may filter to runs where the first experiment was better or vice versa by clicking the thumbs up/thumbs down buttons in the table header:

!配对筛选