追踪 LLM 应用程序教程 - 以编程方式使用文档

在本教程中，您将使用检索增强生成（RAG）构建一个客户支持聊天机器人，并在开发的每个阶段添加 LangSmith 可观测性，从早期原型设计到生产环境。

完成本教程后，您将学会：

- 追踪单个 LLM 调用和完整应用程序管道。
- 收集和查询用户反馈。
- Log metadata and use it for filtering and A/B testing.
- 使用监控仪表板追踪生产环境性能。

该应用程序将检索相关文档片段并使用它们来回答用户问题。本教程中的检索器是模拟的；在实际应用程序中，您会用向量搜索或类似方法替换它。

前提条件

在开始之前，请确保您拥有：

一个 LangSmith 账户：在注册或登录 smith.langchain.com.
一个 LangSmith API 密钥：请按照创建 API 密钥 guide.
一个 OpenAI API 密钥：从 OpenAI 仪表板.
LangSmith CLI （可选）：安装后可从终端检查追踪记录。有关说明，请参阅 LangSmith CLI.

安装所需的包：

pip install langsmith openai

npm install langsmith openai
npm install -D typescript tsx

原型设计

从一开始就设置好可观测性可以让您更快地进行迭代。您可以准确看到发送给模型的内容、返回的内容以及时间花费在哪里，而无需添加打印语句或运行调试器。

设置您的环境

在您的 shell 中设置以下环境变量：

追踪 LLM 调用

首先追踪您的 OpenAI 调用，这是模型实际被调用的地方。这让您能够立即了解应用程序发送的提示词和模型返回的响应。

使用 wrap_openai(Python) 或 wrapOpenAI (TypeScript) 包装 OpenAI 客户端。创建一个名为 app.py (or app.ts的文件，包含以下代码：

from openai import OpenAI
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    print(support_bot("How many users can I have on the Starter plan?"))

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

async function supportBot(question: string): Promise<string> {
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
}

(async () => {
    console.log(await supportBot("How many users can I have on the Starter plan?"));
})();

调用 support_bot("How many users can I have on the Starter plan?") 会生成 OpenAI 调用的追踪记录。

追踪整个管道

追踪 LLM 调用很有用，但追踪完整管道（包括检索）可以让您全面了解应用程序的行为。在主函数中添加 @traceable(Python) 或 traceable (TypeScript)：

from openai import OpenAI
from langsmith import traceable
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

@traceable  # [!code highlight]
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    print(support_bot("How many users can I have on the Starter plan?"))

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

const supportBot = traceable(async function supportBot(question: string): Promise<string> {  // [!code highlight]
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
});  // [!code highlight]

(async () => {
    console.log(await supportBot("How many users can I have on the Starter plan?"));
})();

调用 support_bot("How many users can I have on the Starter plan?") 现在会生成完整 RAG 管道的追踪记录。

LangSmith UI 显示一个追踪，包含一个外部应用程序跨度和一个嵌套的 LLM 调用跨度。

从终端检查您的追踪

如果您安装了 LangSmith CLI，无需打开 UI 即可列出您项目的最近追踪：

langsmith trace list --project <your-project> --limit 5

To view the full run hierarchy and inputs/outputs for a specific trace:

langsmith trace get <trace-id> --full

Beta 测试

一旦您的应用在原型设计阶段运行良好，您就可以将其发布给一小部分真实用户。在这个阶段，您通常无法准确了解用户将如何与您的应用互动，因此您需要更丰富的可观测性。您不仅想了解应用做了什么，还想了解用户对它的反馈。

收集反馈

链接用户反馈到特定追踪，让您能够识别哪些响应有帮助或无帮助。更新 app.py (or app.ts从上一个步骤添加一个 run ID 到每次调用，并在之后附加分数：

from openai import OpenAI
from langsmith import traceable, Client, uuid7  # [!code highlight]
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

def retriever(query: str) -> list[str]:
    return docs

@traceable
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    run_id = str(uuid7())  # [!code highlight]
    support_bot(  # [!code highlight]
        "How many users can I have on the Starter plan?",  # [!code highlight]
        langsmith_extra={"run_id": run_id},  # [!code highlight]
    )  # [!code highlight]
    ls_client = Client()  # [!code highlight]
    ls_client.create_feedback(run_id, key="user-score", score=1.0)  # [!code highlight]

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

function retriever(query: string): string[] {
    return docs;
}

let capturedRunId: string; // [!code highlight]

const supportBot = traceable(async function supportBot(question: string): Promise<string> {
    capturedRunId = getCurrentRunTree().id; // [!code highlight]
    const context = retriever(question);
    const systemMessage =
        "You are a helpful customer support agent. " +
        "Answer using only the information provided below:\n\n" +
        context.join("\n");
    const response = await client.chat.completions.create({
        model: "gpt-5.4-mini",
        messages: [
            { role: "system", content: systemMessage },
            { role: "user", content: question },
        ],
    });
    return response.choices[0].message?.content ?? "";
});

(async () => {
    await supportBot("How many users can I have on the Starter plan?"); // [!code highlight]
    const lsClient = new Client(); // [!code highlight]
    await lsClient.createFeedback(capturedRunId, "user-score", { score: 1.0 }); // [!code highlight]
    await lsClient.flush(); // [!code highlight]
})();

反馈显示在 **反馈** 当您在 UI 中检查运行时可以使用过滤控件通过反馈分数过滤运行 **运行** table.

记录元数据

元数据允许您使用对过滤和比较有用的属性标记运行。例如，使用的模型版本或发出请求的用户。

以下示例追踪检索器（使用 run_type="retriever"）和主函数（使用 metadata 模型名称属性）：

from openai import OpenAI
from langsmith import traceable
from langsmith.wrappers import wrap_openai

client = wrap_openai(OpenAI())

docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
]

@traceable(run_type="retriever")  # [!code highlight]
def retriever(query: str) -> list[str]:
    return docs

@traceable(metadata={"llm": "gpt-5.4-mini"})  # [!code highlight]
def support_bot(question: str) -> str:
    context = retriever(question)
    system_message = (
        "You are a helpful customer support agent. "
        "Answer using only the information provided below:\n\n"
        + "\n".join(context)
    )
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
            {"role": "system", "content": system_message},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    support_bot("How many users can I have on the Starter plan?")

const client = wrapOpenAI(new OpenAI());

const docs = [
    "Acme Cloud supports unlimited users on Enterprise plans. Starter plans are limited to 5 users.",
    "To reset your password, click 'Forgot password' on the login page and follow the instructions sent to your email.",
    "API rate limits are 1,000 requests per hour on the Starter plan and 10,000 requests per hour on Enterprise.",
];

const retriever = traceable(  // [!code highlight]
    function retriever(query: string): string[] {  // [!code highlight]
        return docs;  // [!code highlight]
    },  // [!code highlight]
    { run_type: "retriever" }  // [!code highlight]
);  // [!code highlight]

const supportBot = traceable(
    async function supportBot(question: string): Promise<string> {
        const context = await retriever(question);
        const systemMessage =
            "You are a helpful customer support agent. " +
            "Answer using only the information provided below:\n\n" +
            context.join("\n");
        const response = await client.chat.completions.create({
            model: "gpt-5.4-mini",
            messages: [
                { role: "system", content: systemMessage },
                { role: "user", content: question },
            ],
        });
        return response.choices[0].message?.content ?? "";
    },
    { metadata: { llm: "gpt-5.4-mini" } }  // [!code highlight]
);

(async () => {
    await supportBot("How many users can I have on the Starter plan?");
})();

两个元数据值都显示在追踪上。您可以使用 **运行** table.

生产

有了强大的可观测性，您可以自信地发布到生产环境。在生产环境中，流量显著增加，无法逐一检查每个追踪。LangSmith 提供监控工具来帮助您了解聚合行为，并在出现问题时进行深入分析。

监控

在 UI 侧边栏中，选择 **监控**，然后从左上角的下拉菜单中选择一个追踪项目。图表显示项目随时间变化的关键指标，包括追踪数量、延迟、错误率、反馈分数和成本。有关可用指标和图表配置的更多信息，请参阅仪表板.

A/B testing

因为您一直在记录 llm 元数据属性，您可以通过该属性对监控图表进行分组，以比较模型性能随时间的变化。从 **监控** 在 UI 侧边栏中，点击 **分组依据** 在左上角，选择 **元数据** 从下拉菜单中，然后选择 llm。图表将更新为显示按该属性分组的结果。有关分组和自定义图表的更多信息，请参阅仪表板.

深入查看

当监控图表显示意外内容时，点击数据点以冻结工具提示，然后点击指标名称（例如， **输入**）跳转到该时间窗口的筛选运行表。有关搜索和筛选运行的更多信息，请参阅筛选追踪.

LangSmith UI 显示监控页面，其中 Input Tokens 图表上的特定点已突出显示。

总结

在本教程中，您在整个应用开发生命周期中添加了 LangSmith 可观测性。在原型设计期间帮助您快速迭代的相同追踪设置将在生产环境中继续提供价值。您将能够看到单个追踪和聚合性能趋势。

更多信息，请参阅：

- 可观测性概念：术语和核心概念。
- 追踪集成：LangChain、LangGraph、Anthropic 等提供商。
- 自动化：在追踪上自动运行的规则和在线评估。