使用 LangSmith SDK 管理反馈配置和 标注队列 rubrics programmatically. Define reusable feedback schemas at the organization level (like accuracy scores or pass/fail judgments), then assign them to specific queues with custom instructions. This enables version control, automation across projects, and consistency—particularly useful for CI/CD pipelines or replicating evaluation setups across environments.

反馈层

LangSmith 采用三层架构来处理结构化人工反馈：

1. **反馈配置**: Organization-wide definitions of feedback keys that establish the schema for evaluation metrics. For example, you might define "accuracy" as a continuous 0–1 score or "correctness" as a pass/fail categorical choice. These configs are reusable across all annotation queues in your organization.
2. **标注队列评分项**：特定于队列的分配，用于确定标注员在审查 运行 特定队列中必须填写的反馈配置。每个评分项可以包含自定义描述、特定分数值的指导，以及反馈是必需还是可选的。
3. **反馈**：标注员在特定 运行上提交的个人评分和值。这是使用您定义的架构收集的实际评估数据。详细了解 LangSmith 中的反馈.

反馈配置

创建反馈配置

反馈配置定义反馈键的架构——无论是连续分数、分类选择还是自由格式文本。唯一键在组织内标识每个配置，并指定标注员如何提交该指标的反馈。

from langsmith import Client

client = Client()

# Continuous score
client.create_feedback_config(
    "accuracy",
    feedback_config={
        "type": "continuous",
        "min": 0,
        "max": 1,
    },
    is_lower_score_better=False,
)

# Categorical
client.create_feedback_config(
    "correctness",
    feedback_config={
        "type": "categorical",
        "categories": [
            {"value": 1, "label": "Pass"},
            {"value": 0, "label": "Fail"},
        ],
    },
)

# Freeform text
client.create_feedback_config(
    "notes",
    feedback_config={"type": "freeform"},
)

const client = new Client();

// Continuous score
await client.createFeedbackConfig({
  feedbackKey: "accuracy",
  feedbackConfig: { type: "continuous", min: 0, max: 1 },
  isLowerScoreBetter: false,
});

// Categorical
await client.createFeedbackConfig({
  feedbackKey: "correctness",
  feedbackConfig: {
    type: "categorical",
    categories: [
      { value: 1, label: "Pass" },
      { value: 0, label: "Fail" },
    ],
  },
});

// Freeform text
await client.createFeedbackConfig({
  feedbackKey: "notes",
  feedbackConfig: { type: "freeform" },
});

• 连续 ("accuracy"）：定义从 0 到 1 的数值范围。该 is_lower_score_better 参数指示较低值是否代表更好的性能。将连续配置用于评分量表或基于百分比的指标。
• 分类 ("correctness"）：提供带有关联值的预定义选项。每个类别需要一个 value （用于评分和分析）和一个 label （显示给标注员）。将分类配置用于二元选择或多类分类。
• 自由格式 ("notes"）：允许开放式的文本输入，没有预定义的结构。将自由格式配置用于定性观察或说明。

列出反馈配置

使用 @[ 检索反馈配置以查看组织中可用的评估标准list_feedback_configs]。您可以列出所有配置或按特定键过滤。每个返回的配置对象包含键、类型、配置详情（如 min/max or categories）和元数据（如 is_lower_score_better:

# List all configs
for config in client.list_feedback_configs():
    print(f"{config.feedback_key}: {config.feedback_config}")

# Filter by specific keys
for config in client.list_feedback_configs(
    feedback_key=["accuracy", "correctness"]
):
    print(config.feedback_key)

// List all configs
for await (const config of client.listFeedbackConfigs()) {
  console.log(`${config.feedback_key}: ${JSON.stringify(config.feedback_config)}`);
}

// Filter by specific keys
for await (const config of client.listFeedbackConfigs({
  feedbackKeys: ["accuracy", "correctness"],
})) {
  console.log(config.feedback_key);
}

更新反馈配置

使用 @[ 修改现有的反馈配置update_feedback_config]，方法是更新特定字段。该方法只更改你提供的字段——其余字段保持不变。这是一种保留其他配置设置的局部更新：

client.update_feedback_config(
    "accuracy",
    is_lower_score_better=True,
)

await client.updateFeedbackConfig("accuracy", {
  isLowerScoreBetter: true,
});

删除反馈配置

使用 @[ 从你的组织中移除反馈配置delete_feedback_config]。这是软删除，会将配置标记为已删除，但不会永久从系统中移除。如有需要，你可以稍后使用相同的密钥重新创建配置：

client.delete_feedback_config("accuracy")

await client.deleteFeedbackConfig("accuracy");

标注队列评分规则项

评分规则项将反馈配置分配给特定的标注队列。它们控制标注员在审阅 该队列中的 时看到哪些反馈表单，以及每个表单是必填还是可选。

创建带有评分规则项的队列

使用 @[ 创建标注队列create_annotation_queue]并通过评分规则项为其分配反馈配置。每个评分规则项通过其密钥引用一个反馈配置，并自定义该队列中标注员看到的显示方式。

该示例创建了一个包含三个评分规则项的队列。队列级别的 rubric_instructions 提供显示在标注界面顶部的通用指导：

queue = client.create_annotation_queue(
    name="QA Review Queue",
    description="Review LLM outputs for accuracy and correctness",
    rubric_instructions="Score each response. Add notes for anything unusual.",
    rubric_items=[
        {
            "feedback_key": "accuracy",
            "description": "How accurate is the response?",
            "score_descriptions": {
                "0": "Completely wrong",
                "1": "Perfectly accurate",
            },
            "is_required": True,
        },
        {
            "feedback_key": "correctness",
            "description": "Did the response pass or fail?",
            "value_descriptions": {
                "Pass": "Factually correct",
                "Fail": "Contains errors",
            },
            "is_required": True,
        },
        {
            "feedback_key": "notes",
            "description": "Any additional observations",
            "is_required": False,
        },
    ],
)

const queue = await client.createAnnotationQueue({
  name: "QA Review Queue",
  description: "Review LLM outputs for accuracy and correctness",
  rubricInstructions: "Score each response. Add notes for anything unusual.",
  rubricItems: [
    {
      feedback_key: "accuracy",
      description: "How accurate is the response?",
      score_descriptions: { "0": "Completely wrong", "1": "Perfectly accurate" },
      is_required: true,
    },
    {
      feedback_key: "correctness",
      description: "Did the response pass or fail?",
      value_descriptions: { Pass: "Factually correct", Fail: "Contains errors" },
      is_required: true,
    },
    {
      feedback_key: "notes",
      description: "Any additional observations",
      is_required: false,
    },
  ],
});

• - feedback_key：现有反馈配置的密钥（先创建此配置）。
• - description：此指标的队列特定指导说明，供标注员参考。
• - score_descriptions / value_descriptions：可选标签，用于解释特定值的含义（连续型配置使用 score_descriptions ，分类型配置使用 value_descriptions ）。
• - is_required：标注员在提交前是否必须完成此反馈。

更新现有队列的评分规则项

使用 @[ 修改分配给标注队列的评分规则项update_annotation_queue]。此操作会替换整个评分规则项列表，因此你必须包含所有想要保留的项——不包含的项将被移除。

你需要队列 ID，可在创建队列时获取，或通过列出队列获取：

client.update_annotation_queue(
    queue.id,
    rubric_items=[
        {"feedback_key": "accuracy", "is_required": True},
        {"feedback_key": "correctness", "is_required": True},
        {
            "feedback_key": "tone",
            "description": "Is the tone appropriate?",
            "is_required": False,
        },
    ],
)

await client.updateAnnotationQueue(queue.id, {
  rubricItems: [
    { feedback_key: "accuracy", is_required: true },
    { feedback_key: "correctness", is_required: true },
    { feedback_key: "tone", description: "Is the tone appropriate?", is_required: false },
  ],
});

反馈配置类型（详细）

连续型

连续型配置定义具有最小值和最大值的数字评分量表。标注员可以在范围内选择任意值，这使其非常适合准确性、质量或相关性等评分维度的数字评分：

# Simple continuous score
client.create_feedback_config(
    "accuracy",
    feedback_config={
        "type": "continuous",
        "min": 0,
        "max": 1,
    },
)

# Continuous with labeled points on the scale
client.create_feedback_config(
    "quality",
    feedback_config={
        "type": "continuous",
        "min": 1,
        "max": 5,
        "categories": [
            {"value": 1, "label": "Poor"},
            {"value": 3, "label": "Average"},
            {"value": 5, "label": "Excellent"},
        ],
    },
)

await client.createFeedbackConfig({
  feedbackKey: "accuracy",
  feedbackConfig: { type: "continuous", min: 0, max: 1 },
});

await client.createFeedbackConfig({
  feedbackKey: "quality",
  feedbackConfig: {
    type: "continuous",
    min: 1,
    max: 5,
    categories: [
      { value: 1, label: "Poor" },
      { value: 3, label: "Average" },
      { value: 5, label: "Excellent" },
    ],
  },
});

第一个示例展示了一个没有标签的 0-1 量表。第二个示例演示了添加 categories ，在量表上标注锚点（如"差"、"一般"、"优秀"），以帮助标注员理解不同值代表的含义。这些标签是可选的，但可以提高标注员对量表解释的一致性。

分类型

分类型配置为标注员提供预定义的离散选项集。每个类别必须有一个 value （用于评分和分析的数值标识符）和一个 label （显示给标注员的文本）。你必须至少定义 2 个类别。

Use categorical configs for binary decisions (pass/fail, correct/incorrect), multi-class classifications (sentiment, topic categories), or any evaluation with a fixed set of discrete options. Do not set min or max 分类型配置的示例：

# Binary pass/fail
client.create_feedback_config(
    "correctness",
    feedback_config={
        "type": "categorical",
        "categories": [
            {"value": 1, "label": "Pass"},
            {"value": 0, "label": "Fail"},
        ],
    },
)

# Multi-class
client.create_feedback_config(
    "sentiment",
    feedback_config={
        "type": "categorical",
        "categories": [
            {"value": 0, "label": "Negative"},
            {"value": 1, "label": "Neutral"},
            {"value": 2, "label": "Positive"},
        ],
    },
)

await client.createFeedbackConfig({
  feedbackKey: "correctness",
  feedbackConfig: {
    type: "categorical",
    categories: [
      { value: 1, label: "Pass" },
      { value: 0, label: "Fail" },
    ],
  },
});

await client.createFeedbackConfig({
  feedbackKey: "sentiment",
  feedbackConfig: {
    type: "categorical",
    categories: [
      { value: 0, label: "Negative" },
      { value: 1, label: "Neutral" },
      { value: 2, label: "Positive" },
    ],
  },
});

The first example shows a binary pass/fail config. The second example demonstrates a multi-class config for sentiment with three options. The numeric values allow you to compute aggregate scores even for categorical feedback.

自由文本型

自由文本型配置允许标注员提供没有任何预定义结构或约束的开放式文本反馈。此类型没有 min, max, or categories 字段——标注员可以输入任意文本。

自由格式反馈对于捕获细致入微的见解很有价值，但与结构化反馈类型相比，更难汇总和分析：

client.create_feedback_config(
    "notes",
    feedback_config={"type": "freeform"},
)

await client.createFeedbackConfig({
  feedbackKey: "notes",
  feedbackConfig: { type: "freeform" },
});

验证规则

参考

反馈配置类型

评分项目字段