LangSmith 与开源 openevals 包集成，提供一套评估工具和提示，可作为评估的起点。

设置

您需要安装 openevals 包以使用 LLM 即评判评估器。

pip install -U openevals

yarn add openevals @langchain/core

您还需要将您的 OpenAI API 密钥设置为环境变量，不过您也可以选择其他提供商：

我们还将使用 LangSmith 的 pytest Python 集成以及 Vitest/Jest 来运行我们的评估。 openevals 也能与 evaluate 方法无缝集成。请参阅 相关指南 了解设置说明。

运行评估器

基本流程很简单：从 openevals导入评估器或工厂函数，然后在测试文件中使用输入、输出和参考输出运行它。LangSmith 会自动记录评估器的结果作为反馈。

请注意，并非所有评估器都需要每个参数（例如精确匹配评估器仅需要输出和参考输出）。此外，如果您的 LLM 即评判提示需要其他变量，将其作为 kwargs 传入即可将其格式化到提示中。

按如下方式设置您的测试文件：

from langsmith import testing as t
from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

correctness_evaluator = create_llm_as_judge(
    prompt=CORRECTNESS_PROMPT,
    feedback_key="correctness",
    model="openai:o3-mini",
)

# Mock standin for your application
def my_llm_app(inputs: dict) -> str:
    return "Doodads have increased in price by 10% in the past year."

@pytest.mark.langsmith
def test_correctness():
    inputs = "How much has the price of doodads changed in the past year?"
    reference_outputs = "The price of doodads has decreased by 50% in the past year."
    outputs = my_llm_app(inputs)

    t.log_inputs({"question": inputs})
    t.log_outputs({"answer": outputs})
    t.log_reference_outputs({"answer": reference_outputs})

    correctness_evaluator(
        inputs=inputs,
        outputs=outputs,
        reference_outputs=reference_outputs
    )

// import * as ls from "langsmith/jest";


const correctnessEvaluator = createLLMAsJudge({
    prompt: CORRECTNESS_PROMPT,
    feedbackKey: "correctness",
    model: "openai:o3-mini",
});

// Mock standin for your application
const myLLMApp = async (_inputs: Record<string, unknown>) => {
    return "Doodads have increased in price by 10% in the past year.";
};

ls.describe("Correctness", () => {
    ls.test("incorrect answer", {
        inputs: {
            question: "How much has the price of doodads changed in the past year?"
        },
        referenceOutputs: {
            answer: "The price of doodads has decreased by 50% in the past year."
        }
    }, async ({ inputs, referenceOutputs }) => {
        const outputs = await myLLMApp(inputs);
        ls.logOutputs({ answer: outputs });
        await correctnessEvaluator({
            inputs,
            outputs,
            referenceOutputs,
        });
    });
});

feedback_key/feedbackKey 参数将用作实验中反馈的名称。

在终端中运行评估会产生类似以下的结果：

!预构建评估器终端结果

如果您已在 LangSmith 中创建了数据集，也可以将评估器直接传入 evaluate 方法。如果使用 Python，这需要 langsmith>=0.3.11:

from langsmith import Client
from openevals.llm import create_llm_as_judge
from openevals.prompts import CONCISENESS_PROMPT

client = Client()
conciseness_evaluator = create_llm_as_judge(
    prompt=CONCISENESS_PROMPT,
    feedback_key="conciseness",
    model="openai:o3-mini",
)

experiment_results = client.evaluate(
    # This is a dummy target function, replace with your actual LLM-based system
    lambda inputs: "What color is the sky?",
    data="Sample dataset",
    evaluators=[
        conciseness_evaluator
    ]
)

const concisenessEvaluator = createLLMAsJudge({
    prompt: CONCISENESS_PROMPT,
    feedbackKey: "conciseness",
    model: "openai:o3-mini",
});

await evaluate((inputs) => "What color is the sky?", {
    data: datasetName,
    evaluators: [concisenessEvaluator],
});

有关可用的评估工具和提示的完整列表，请参阅 openevals 和 agentevals repos.