在本指南中，我们将为聊天机器人设置评估。这些评估允许您衡量应用程序在一组数据上的表现。能够快速可靠地获得这些洞察将使您能够充满信心地进行迭代。

从高层来看，在本教程中我们将：

• * *创建一个初始黄金数据集来衡量性能*
• * *定义用于衡量性能的指标*
• * *在几个不同的提示或模型上运行评估*
• * *手动比较结果*
• * *跟踪结果随时间的变化*
• * *Set up automated testing to run in CI/CD*

有关 LangSmith 支持的评估工作流程的更多信息，请查看 操作指南，或查看 evaluate 的参考文档及其异步对应物 aevaluate。

内容很多，让我们开始吧！

设置

首先安装本教程所需的依赖项。我们恰好使用 OpenAI，但 LangSmith 可以与任何模型一起使用：

pip install -U langsmith openai

uv add langsmith openai

并设置环境变量以启用 LangSmith 跟踪：

创建数据集

在准备测试和评估应用程序时，第一步是定义要评估的数据点。这里有几个方面需要考虑：

• * 每个数据点的架构应该是什么？
• * 我应该收集多少个数据点？
• * 我应该如何收集这些数据点？

Schema: 每个数据点至少应包含应用程序的输入。如果可以的话，定义预期输出也很有帮助——这些代表您期望正常运行的应用程序会输出的内容。通常您无法定义完美的输出——没关系！评估是一个迭代过程。有时您可能还想为每个示例定义更多信息——比如 RAG 中要获取的预期文档，或代理要采取的预期步骤。LangSmith 数据集非常灵活，允许您定义任意架构。

数量： 关于应该收集多少，没有硬性规定。主要的是确保您对可能需要防范的边缘情况有适当的覆盖。即使是 10-50 个示例也能提供很多价值！不要担心一开始获得大量数据——您可以（也应该）随着时间不断添加！

如何获取： 这可能是最棘手的部分。一旦您知道要收集数据集...您实际上如何进行？对于大多数刚开始新项目的团队，我们通常看到他们从手动收集前 10-20 个数据点开始。在从这些数据点开始后，这些数据集通常是 *动态的* constructs and grow over time. They generally grow after seeing how real users will use your application, seeing the pain points that exist, and then moving a few of those datapoints into this set. There are also methods like synthetically generating data that can be used to augment your dataset. To start, we recommend not worrying about those and just hand labeling \~10-20 examples.

一旦您有了数据集，有几种不同的方式可以将它们上传到 LangSmith。对于本教程，我们将使用客户端，但您也可以通过 UI 上传（甚至在 UI 中创建它们）。

对于本教程，我们将创建 5 个数据点来评估。我们将评估一个问答应用程序。输入将是一个问题，输出将是一个答案。由于这是一个问答应用程序，我们可以定义预期的答案。让我们展示如何创建并上传此数据集到 LangSmith！

from langsmith import Client

client = Client()

# Define dataset: these are your test cases
dataset_name = "QA Example Dataset"
dataset = client.create_dataset(dataset_name)

client.create_examples(
    dataset_id=dataset.id,
    examples=[
        {
            "inputs": {"question": "What is LangChain?"},
            "outputs": {"answer": "A framework for building LLM applications"},
        },
        {
            "inputs": {"question": "What is LangSmith?"},
            "outputs": {"answer": "A platform for observing and evaluating LLM applications"},
        },
        {
            "inputs": {"question": "What is OpenAI?"},
            "outputs": {"answer": "A company that creates Large Language Models"},
        },
        {
            "inputs": {"question": "What is Google?"},
            "outputs": {"answer": "A technology company known for search"},
        },
        {
            "inputs": {"question": "What is Mistral?"},
            "outputs": {"answer": "A company that creates Large Language Models"},
        }
    ]
)

现在，如果我们转到 LangSmith UI 并查找 QA Example Dataset 在 Datasets & Testing 页面中，当我们点击进入时，我们应该会看到五个新示例。

!测试教程数据集

定义指标

创建数据集后，我们现在可以定义一些指标来评估我们的响应。由于我们有预期的答案，我们可以将其作为评估的一部分进行比较。但是，我们并不期望我们的应用程序输出这些 **完全一致** 答案，而是相似的答案。这使得我们的评估变得稍微复杂一些。

除了评估正确性，我们还要确保答案简短精炼。这会更容易一些——我们可以定义一个简单的Python函数来测量响应的长度。

让我们开始定义这两个指标。

对于第一个指标，我们将使用LLM来 **判断** 输出是否正确（相对于预期输出）。这 **LLM-as-a-judge** 对于太复杂而无法用简单函数衡量的案例来说，这相对常见。我们可以在此处定义自己的提示和LLM用于评估：

from langsmith import wrappers

openai_client = wrappers.wrap_openai(openai.OpenAI())

eval_instructions = "You are an expert professor specialized in grading students' answers to questions."

def correctness(inputs: dict, outputs: dict, reference_outputs: dict) -> bool:
    user_content = f"""You are grading the following question:
{inputs['question']}
Here is the real answer:
{reference_outputs['answer']}
You are grading the following predicted answer:
{outputs['response']}
Respond with CORRECT or INCORRECT:
Grade:"""
    response = openai_client.chat.completions.create(
        model="gpt-5.4-mini",
        temperature=0,
        messages=[
            {"role": "system", "content": eval_instructions},
            {"role": "user", "content": user_content},
        ],
    ).choices[0].message.content
    return response == "CORRECT"

对于评估响应的长度，这要简单得多！我们只需定义一个简单的函数来检查实际输出是否小于预期结果长度的2倍。

def concision(outputs: dict, reference_outputs: dict) -> bool:
    return int(len(outputs["response"]) < 2 * len(reference_outputs["answer"]))

运行评估

很好！现在如何运行评估？既然我们有了数据集和评估器，我们需要的只是应用程序！我们将构建一个简单的应用程序，它只需要一条系统消息，其中包含响应指令，然后将其传递给LLM。我们将直接使用OpenAI SDK构建这个：

default_instructions = "Respond to the users question in a short, concise manner (one short sentence)."

def my_app(question: str, model: str = "gpt-5.4-mini", instructions: str = default_instructions) -> str:
    return openai_client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[
            {"role": "system", "content": instructions},
            {"role": "user", "content": question},
        ],
    ).choices[0].message.content

在通过LangSmith评估运行之前，我们需要定义一个简单的包装器，将数据集中的输入键映射到我们想要调用的函数，然后也将函数的输出映射到我们期望的输出键。

def ls_target(inputs: str) -> dict:
    return {"response": my_app(inputs["question"])}

很好！现在我们准备好运行评估了。让我们开始吧！

experiment_results = client.evaluate(
    ls_target, # Your AI system
    data=dataset_name, # The data to predict and grade over
    evaluators=[concision, correctness], # The evaluators to score the results
    experiment_prefix="openai-4o-mini", # A prefix for your experiment names to easily identify them
)

这将输出一个URL。如果我们点击它，我们应该能看到评估结果！

!测试教程运行

如果我们返回数据集页面并选择 Experiments 标签，我们现在可以看到一次运行的摘要！

!测试教程单次运行

现在让我们用不同的模型试试！让我们尝试 gpt-4-turbo

def ls_target_v2(inputs: str) -> dict:
    return {"response": my_app(inputs["question"], model="gpt-4-turbo")}

experiment_results = client.evaluate(
    ls_target_v2,
    data=dataset_name,
    evaluators=[concision, correctness],
    experiment_prefix="openai-4-turbo",
)

现在让我们使用GPT-4，但也要更新提示，使其对答案简短的要求更严格。

instructions_v3 = "Respond to the users question in a short, concise manner (one short sentence). Do NOT use more than ten words."

def ls_target_v3(inputs: str) -> dict:
    response = my_app(
        inputs["question"],
        model="gpt-4-turbo",
        instructions=instructions_v3
    )
    return {"response": response}

experiment_results = client.evaluate(
    ls_target_v3,
    data=dataset_name,
    evaluators=[concision, correctness],
    experiment_prefix="strict-openai-4-turbo",
)

如果我们返回 Experiments 数据集页面上的标签，我们应该看到现在所有三次运行都显示出来了！

!测试教程三次运行

比较结果

太棒了，我们已经评估了三次不同的运行。但我们如何比较结果呢？第一种方法是我们可以通过查看 Experiments 标签中的运行来实现。如果我们这样做，我们可以看到每次运行指标的高级视图：

!测试教程比较指标

我们可以看出GPT-4在了解公司身份方面比GPT-3.5更好，而且严格的提示对长度有很大帮助。但如果我们要更详细地探索呢？

为了做到这一点，我们可以选择所有想要比较的运行（在这种情况下是全部三个）并将它们打开在比较视图中。我们立即看到所有三个测试并排显示。某些单元格用颜色编码——这显示了某个指标的 *某个指标* 相对于 *某个基准*的回归情况。我们自动选择基准和指标的默认值，但您可以自己更改这些。您也可以通过使用来选择您看到的列和指标 Display control. You can also automatically filter to only see the runs that have improvements/regressions by clicking on the icons at the top.

!测试教程比较运行

如果我们想查看更多信息，我们也可以选择 Expand 悬停在某一行上时出现的按钮，以打开带有更详细信息的侧边面板：

!测试教程侧边面板

Set up automated testing to run in CI/CD

Now that we've run this in a one-off manner, we can set it to run in an automated fashion. We can do this pretty easily by just including it as a pytest file that we run in CI/CD. As part of this, we can either just log the results OR set up some criteria to determine if it passes or not. For example, if I wanted to ensure that we always got at least 80% of generated responses passing the length 检查，我们可以像这样设置测试：

def test_length_score() -> None:
    """Test that the length score is at least 80%."""
    experiment_results = evaluate(
        ls_target, # Your AI system
        data=dataset_name, # The data to predict and grade over
        evaluators=[concision, correctness], # The evaluators to score the results
    )
    # This will be cleaned up in the next release:
    feedback = client.list_feedback(
        run_ids=[r.id for r in client.list_runs(project_name=experiment_results.experiment_name)],
        feedback_key="concision"
    )
    scores = [f.score for f in feedback]
    assert sum(scores) / len(scores) >= 0.8, "Aggregate score should be at least .8"

随时间追踪结果

现在我们已经让这些实验以自动化方式运行，我们想随时间追踪这些结果。我们可以从数据集页面的总体 Experiments 标签页中进行此操作。默认情况下，我们显示随时间变化的评估指标（红色高亮）。我们还会自动追踪 git 指标，以便轻松地将其与您的代码分支关联（黄色高亮）。

!测试教程随时间变化

结论

本教程到此结束！

We've gone over how to create an initial test set, define some evaluation metrics, run experiments, compare them manually, set up CI/CD, and track results over time. This can help you iterate with confidence.

这只是开始。如前所述，评估是一个持续的过程。例如——您想要评估的数据点可能会随着时间继续变化。您可能希望探索多种类型的评估器。如需了解更多信息，请参阅 操作指南.

此外，还有其他方式来评估数据，而不仅仅是这种"离线"方式（例如您可以评估生产数据）。有关在线评估的更多信息，请参阅 设置 LLM 即评判者的在线评估器.

参考代码

from langsmith import Client, wrappers

# Application code
openai_client = wrappers.wrap_openai(openai.OpenAI())

default_instructions = "Respond to the users question in a short, concise manner (one short sentence)."

def my_app(question: str, model: str = "gpt-5.4-mini", instructions: str = default_instructions) -> str:
    return openai_client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[
            {"role": "system", "content": instructions},
            {"role": "user", "content": question},
        ],
    ).choices[0].message.content

client = Client()

# Define dataset: these are your test cases
dataset_name = "QA Example Dataset"
dataset = client.create_dataset(dataset_name)

client.create_examples(
    dataset_id=dataset.id,
    examples=[
        {
            "inputs": {"question": "What is LangChain?"},
            "outputs": {"answer": "A framework for building LLM applications"},
        },
        {
            "inputs": {"question": "What is LangSmith?"},
            "outputs": {"answer": "A platform for observing and evaluating LLM applications"},
        },
        {
            "inputs": {"question": "What is OpenAI?"},
            "outputs": {"answer": "A company that creates Large Language Models"},
        },
        {
            "inputs": {"question": "What is Google?"},
            "outputs": {"answer": "A technology company known for search"},
        },
        {
            "inputs": {"question": "What is Mistral?"},
            "outputs": {"answer": "A company that creates Large Language Models"},
        }
    ]
)

# Define evaluators
eval_instructions = "You are an expert professor specialized in grading students' answers to questions."

def correctness(inputs: dict, outputs: dict, reference_outputs: dict) -> bool:
    user_content = f"""You are grading the following question:
{inputs['question']}
Here is the real answer:
{reference_outputs['answer']}
You are grading the following predicted answer:
{outputs['response']}
Respond with CORRECT or INCORRECT:
Grade:"""
    response = openai_client.chat.completions.create(
        model="gpt-5.4-mini",
        temperature=0,
        messages=[
            {"role": "system", "content": eval_instructions},
            {"role": "user", "content": user_content},
        ],
    ).choices[0].message.content
    return response == "CORRECT"

def concision(outputs: dict, reference_outputs: dict) -> bool:
    return int(len(outputs["response"]) < 2 * len(reference_outputs["answer"]))

# Run evaluations
def ls_target(inputs: str) -> dict:
    return {"response": my_app(inputs["question"])}

experiment_results_v1 = client.evaluate(
    ls_target, # Your AI system
    data=dataset_name, # The data to predict and grade over
    evaluators=[concision, correctness], # The evaluators to score the results
    experiment_prefix="openai-4o-mini", # A prefix for your experiment names to easily identify them
)

def ls_target_v2(inputs: str) -> dict:
    return {"response": my_app(inputs["question"], model="gpt-4-turbo")}

experiment_results_v2 = client.evaluate(
    ls_target_v2,
    data=dataset_name,
    evaluators=[concision, correctness],
    experiment_prefix="openai-4-turbo",
)

instructions_v3 = "Respond to the users question in a short, concise manner (one short sentence). Do NOT use more than ten words."

def ls_target_v3(inputs: str) -> dict:
    response = my_app(
        inputs["question"],
        model="gpt-4-turbo",
        instructions=instructions_v3
    )
    return {"response": response}

experiment_results_v3 = client.evaluate(
    ls_target_v3,
    data=dataset_name,
    evaluators=[concision, correctness],
    experiment_prefix="strict-openai-4-turbo",
)