LLM 应用可能难以评估，因为它们通常生成对话文本，没有单一的标准答案。

本指南展示如何定义 LLM-as-a-judge 评估器 用于 离线评估 使用 LangSmith SDK.

创建您自己的 LLM-as-a-judge 评估器

要完全控制评估器逻辑，请创建您自己的 LLM-as-a-judge 评估器并使用 LangSmith SDK 运行它（Python / TypeScript).

前提条件 langsmith>=0.2.0

LLM-as-a-judge 评估器由三个关键组件组成：

1. **评估器函数**：一个接收示例输入和应用输出的函数，然后使用 LLM 对质量进行评分。该函数应返回布尔值、数字、字符串或包含评分信息的字典。
2. **目标函数**：您正在评估的应用逻辑（使用 @traceable 包装以实现可观测性）。
3. **数据集和评估**：测试示例数据集以及 evaluate() 对每个示例运行目标函数并应用评估器的函数。

示例

from langsmith import evaluate, traceable, wrappers, Client
from openai import OpenAI
from pydantic import BaseModel

# Wrap the OpenAI client to automatically trace all LLM calls
oai_client = wrappers.wrap_openai(OpenAI())

# 1. Define your evaluator function
# This function receives the inputs and outputs from each test example
def valid_reasoning(inputs: dict, outputs: dict) -> bool:
    """Use an LLM to judge if the reasoning and the answer are consistent."""
    # Define the evaluation criteria
    instructions = """
Given the following question, answer, and reasoning, determine if the reasoning
for the answer is logically valid and consistent with the question and the answer."""

    # Use structured output to get a boolean score
    class Response(BaseModel):
        reasoning_is_valid: bool

    # Construct the prompt with the actual inputs and outputs
    msg = f"Question: {inputs['question']}\nAnswer: {outputs['answer']}\nReasoning: {outputs['reasoning']}"

    # Call the LLM to judge the output
    response = oai_client.beta.chat.completions.parse(
        model="gpt-4o",
        messages=[{"role": "system", "content": instructions}, {"role": "user", "content": msg}],
        response_format=Response
    )

    # Return the boolean score
    return response.choices[0].message.parsed.reasoning_is_valid

# 2. Define your target function (the application being evaluated)
# The @traceable decorator logs traces to LangSmith for debugging
@traceable
def dummy_app(inputs: dict) -> dict:
    return {"answer": "hmm i'm not sure", "reasoning": "i didn't understand the question"}

# 3. Create a dataset with test examples
ls_client = Client()
dataset = ls_client.create_dataset("big questions")
examples = [
    {"inputs": {"question": "how will the universe end"}},
    {"inputs": {"question": "are we alone"}},
]
ls_client.create_examples(dataset_id=dataset.id, examples=examples)

# 4. Run the evaluation
# This runs dummy_app on each example and applies the valid_reasoning evaluator
results = evaluate(
    dummy_app,              # Your application function
    data=dataset,           # Dataset to evaluate on
    evaluators=[valid_reasoning]  # List of evaluator functions
)

有关如何编写自定义评估器的更多信息，请参阅 如何定义代码评估器（SDK）.