UI

1. 创建带有附件的示例

您可以通过几种不同的方式向数据集添加带有附件的示例。

从现有运行

向 LangSmith 数据集添加运行时，可以有选择地将附件从源运行传播到目标示例。要了解更多，请参阅 在应用中管理数据集.

!将带有附件的追踪添加到数据集

从头创建

您可以直接从 LangSmith UI 创建带有附件的示例。点击数据集 UI + Example 标签页中的 Examples 按钮。然后使用"上传文件"按钮上传附件：

!创建带有附件的示例

上传后，您可以在 LangSmith UI 中查看带有附件的示例。每个附件都将渲染预览以便于检查。 !带有示例的附件

2. 创建多模态提示词

LangSmith UI 允许您在评估多模态模型时在提示词中包含附件：

首先，点击您要添加多模态内容的消息中的文件图标。接下来，为每个示例要包含的附件添加模板变量。

• - 如果要包含特定附件，您可以使用建议的变量名，例如 {{attachment.file_name}}，这会将附件列表中具有 file_name 的文件映射并传递给评估器
• - 如果要包含所有附件，请使用 {{attachments}} variable.

!添加多模态变量

3. 定义自定义评估器

您可以创建使用数据集中示例的多模态内容的评估器。

由于您的数据集已经包含带有附件的示例（已在步骤 1 中添加），您可以直接在评估器中引用它们。为此：

1. 选择 **+ 评估器** 从数据集页面。 1. 在 **模板变量** 编辑器中，添加一个变量用于包含附件： - 如果想包含特定附件，可以使用建议的变量名，例如 {{attachment.file_name}}，这会将文件与 file_name 附件列表中的内容映射以传递给评估器。 - 如果想包含所有附件，请使用 {{attachments}} variable.

评估器随后可以将这些附件与模型的输出结合使用来判断质量。例如，你可以创建一个评估器：

• - 检查图像描述是否与实际图像内容匹配。
• - 验证转录是否准确反映音频内容。
• - 验证从 PDF 中提取的文本是否正确。

你也可以创建不使用附件的纯文本评估器来评估模型的文本输出：

• - OCR → 文本纠正：使用视觉模型从文档中提取文本，然后评估提取输出的准确性。
• - 语音转文本 → 转录质量：使用语音模型将音频转录为文本，然后根据参考评估转录质量。

有关定义自定义评估器的更多信息，请参阅 LLM 作为评判者 guide.

4. 使用附件更新示例

在 UI 中编辑示例时，你可以：

• * 上传新附件
• * 重命名和删除附件
• * 使用快速重置按钮将附件重置为之前的状态

点击提交前更改不会被保存。

!附件编辑

SDK

1. 使用附件创建示例

要使用 SDK 上传带有附件的示例，请使用 创建_示例 / 更新_示例 Python 方法或 uploadExamplesMultipart / updateExamplesMultipart TypeScript 方法。

Python

需要 langsmith>=0.3.13

from pathlib import Path
from langsmith import Client

# Publicly available test files
pdf_url = "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf"
wav_url = "https://openaiassets.blob.core.windows.net/$web/API/docs/audio/alloy.wav"
img_url = "https://www.w3.org/Graphics/PNG/nurbcup2si.png"

# Fetch the files as bytes
pdf_bytes = requests.get(pdf_url).content
wav_bytes = requests.get(wav_url).content
img_bytes = requests.get(img_url).content

# Create the dataset
ls_client = Client()
dataset_name = "attachment-test-dataset"
dataset = ls_client.create_dataset(
  dataset_name=dataset_name,
  description="Test dataset for evals with publicly available attachments",
)

inputs = {
  "audio_question": "What is in this audio clip?",
  "image_question": "What is in this image?",
}

outputs = {
  "audio_answer": "The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years.",
  "image_answer": "A mug with a blanket over it.",
}

# Define an example with attachments
example_id = uuid.uuid4()
example = {
  "id": example_id,
  "inputs": inputs,
  "outputs": outputs,
  "attachments": {
      "my_pdf": {"mime_type": "application/pdf", "data": pdf_bytes},
      "my_wav": {"mime_type": "audio/wav", "data": wav_bytes},
      "my_img": {"mime_type": "image/png", "data": img_bytes},
      # Example of an attachment specified via a local file path:
      # "my_local_img": {"mime_type": "image/png", "data": Path(__file__).parent / "my_local_img.png"},
  },
}

# Create the example
ls_client.create_examples(
  dataset_id=dataset.id,
  examples=[example],
  # Uncomment this flag if you'd like to upload attachments from local files:
  # dangerously_allow_filesystem=True
)

TypeScript

需要版本 >= 0.2.13

您可以使用 uploadExamplesMultipart 方法上传带有附件的示例。

请注意，这是一种与标准方法不同的方法 createExamples 方法，目前不支持附件。每个附件需要 Uint8Array or an ArrayBuffer 作为数据类型。

• * Uint8Array：用于直接处理二进制数据。
• * ArrayBuffer：表示固定长度的二进制数据，可以转换为 Uint8Array 根据需要。

请注意，您无法在 TypeScript SDK 中直接传入文件路径，因为在所有运行时环境中都不支持访问本地文件。

// Publicly available test files
const pdfUrl = "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf";
const wavUrl = "https://openaiassets.blob.core.windows.net/$web/API/docs/audio/alloy.wav";
const pngUrl = "https://www.w3.org/Graphics/PNG/nurbcup2si.png";

// Helper function to fetch file as ArrayBuffer
async function fetchArrayBuffer(url: string): Promise {
  const response = await fetch(url);
  if (!response.ok) {
    throw new Error(`Failed to fetch ${url}: ${response.statusText}`);
  }
  return response.arrayBuffer();
}

// Fetch files as ArrayBuffer
const pdfArrayBuffer = await fetchArrayBuffer(pdfUrl);
const wavArrayBuffer = await fetchArrayBuffer(wavUrl);
const pngArrayBuffer = await fetchArrayBuffer(pngUrl);

// Create the LangSmith client (Ensure LANGSMITH_API_KEY is set in env)
const langsmithClient = new Client();

// Create a unique dataset name
const datasetName = "attachment-test-dataset:" + uuid4().substring(0, 8);

// Create the dataset
const dataset = await langsmithClient.createDataset(datasetName, {
  description: "Test dataset for evals with publicly available attachments",
});

// Define the example with attachments
const exampleId = uuid4();
const example = {
  id: exampleId,
  inputs: {
      audio_question: "What is in this audio clip?",
      image_question: "What is in this image?",
  },
  outputs: {
      audio_answer: "The sun rises in the east and sets in the west. This simple fact has been observed by humans for thousands of years.",
      image_answer: "A mug with a blanket over it.",
  },
  attachments: {
    my_pdf: {
      mimeType: "application/pdf",
      data: pdfArrayBuffer
    },
    my_wav: {
      mimeType: "audio/wav",
      data: wavArrayBuffer
    },
    my_img: {
      mimeType: "image/png",
      data: pngArrayBuffer
    },
  },
};

// Upload the example with attachments to the dataset
await langsmithClient.uploadExamplesMultipart(dataset.id, [example]);

2. 运行评估

定义目标函数

既然我们有了包含附件示例的数据集，我们可以定义一个目标函数来运行这些示例。以下示例简单地使用 OpenAI 的 GPT-4o 模型来回答有关图像和音频片段的问题。

Python

您正在评估的目标函数必须有两个位置参数才能使用与示例关联的附件，第一个参数必须名为 inputs ，第二个参数必须名为 attachments.

• * 函数 inputs 参数是一个字典，包含示例的输入数据，但不包括附件。
• * 该 attachments 参数是一个字典，将附件名称映射到包含预签名URL、mime_类型以及文件字节内容的读取器的字典。您可以使用预签名URL或读取器来获取文件内容。附件字典中的每个值都是一个具有以下结构的字典：

{
    "presigned_url": str,
    "mime_type": str,
    "reader": BinaryIO
}

from langsmith.wrappers import wrap_openai

from openai import OpenAI

client = wrap_openai(OpenAI())

# Define target function that uses attachments
def file_qa(inputs, attachments):
    # Read the audio bytes from the reader and encode them in base64
    audio_reader = attachments["my_wav"]["reader"]
    audio_b64 = base64.b64encode(audio_reader.read()).decode('utf-8')

    audio_completion = client.chat.completions.create(
        model="gpt-4o-audio-preview",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": inputs["audio_question"]
                    },
                    {
                        "type": "input_audio",
                        "input_audio": {
                            "data": audio_b64,
                            "format": "wav"
                        }
                    }
                ]
            }
        ]
    )

    # Most models support taking in an image URL directly in addition to base64 encoded images
    # You can pipe the image pre-signed URL directly to the model
    image_url = attachments["my_img"]["presigned_url"]
    image_completion = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[
          {
            "role": "user",
            "content": [
              {"type": "text", "text": inputs["image_question"]},
              {
                "type": "image_url",
                "image_url": {
                  "url": image_url,
                },
              },
            ],
          }
        ],
    )

    return {
        "audio_answer": audio_completion.choices[0].message.content,
        "image_answer": image_completion.choices[0].message.content,
    }

TypeScript

在 TypeScript SDK 中， config 参数用于将附件传递给目标函数（如果 includeAttachments 设置为 true.

该 config 将包含 attachments 它是一个将附件名称映射到以下形式对象的映射对象：

{
  presigned_url: string,
  mime_type: string,
}

const client: any = wrapOpenAI(new OpenAI());

async function fileQA(inputs: Record<string, any>, config?: Record<string, any>) {
  const presignedUrl = config?.attachments?.["my_wav"]?.presigned_url;
  if (!presignedUrl) {
    throw new Error("No presigned URL provided for audio.");
  }

  const response = await fetch(presignedUrl);
  if (!response.ok) {
    throw new Error(`Failed to fetch audio: ${response.statusText}`);
  }

  const arrayBuffer = await response.arrayBuffer();
  const uint8Array = new Uint8Array(arrayBuffer);
  const audioB64 = Buffer.from(uint8Array).toString("base64");

  const audioCompletion = await client.chat.completions.create({
    model: "gpt-4o-audio-preview",
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: inputs["audio_question"] },
          {
            type: "input_audio",
            input_audio: {
              data: audioB64,
              format: "wav",
            },
          },
        ],
      },
    ],
  });

  const imageUrl = config?.attachments?.["my_img"]?.presigned_url
  const imageCompletion = await client.chat.completions.create({
    model: "gpt-5.4-mini",
    messages: [
      {
        role: "user",
        content: [
          { type: "text", text: inputs["image_question"] },
          {
            type: "image_url",
            image_url: {
              url: imageUrl,
            },
          },
        ],
      },
    ],
  });

  return {
    audio_answer: audioCompletion.choices[0].message.content,
    image_answer: imageCompletion.choices[0].message.content,
  };
}

定义自定义评估器

与上述相同的规则适用于确定评估器是否应接收附件。

以下评估器使用LLM来判断推理和答案是否一致。要了解有关如何定义基于LLM的评估器的更多信息，请参阅 如何定义LLM作为评判者的评估器.

# Assumes you've installed pydantic
from pydantic import BaseModel

def valid_image_description(outputs: dict, attachments: dict) -> bool:
  """Use an LLM to judge if the image description and images are consistent."""
  instructions = """
  Does the description of the following image make sense?
  Please carefully review the image and the description to determine if the description is valid.
  """

  class Response(BaseModel):
      description_is_valid: bool

  image_url = attachments["my_img"]["presigned_url"]
  response = client.beta.chat.completions.parse(
      model="gpt-5.5",
      messages=[
          {
              "role": "system",
              "content": instructions
          },
          {
              "role": "user",
              "content": [
                  {"type": "image_url", "image_url": {"url": image_url}},
                  {"type": "text", "text": outputs["image_answer"]}
              ]
          }
      ],
      response_format=Response
  )
  return response.choices[0].message.parsed.description_is_valid

ls_client.evaluate(
  file_qa,
  data=dataset_name,
  evaluators=[valid_image_description],
)

const DescriptionResponse = z.object({
  description_is_valid: z.boolean(),
});

async function validImageDescription({
  outputs,
  attachments,
}: {
  outputs?: any;
  attachments?: any;
}): Promise<{ key: string; score: boolean}> {
  const instructions = `Does the description of the following image make sense?
Please carefully review the image and the description to determine if the description is valid.`;

  const imageUrl = attachments?.["my_img"]?.presigned_url
  const completion = await client.beta.chat.completions.parse({
      model: "gpt-5.5",
      messages: [
          {
              role: "system",
              content: instructions,
          },
          {
              role: "user",
              content: [
                  { type: "image_url", image_url: { url: imageUrl } },
                  { type: "text", text: outputs?.image_answer },
              ],
          },
      ],
      response_format: zodResponseFormat(DescriptionResponse, 'imageResponse'),
  });

  const score: boolean = completion.choices[0]?.message?.parsed?.description_is_valid ?? false;
  return { key: "valid_image_description", score };
}

const resp = await evaluate(fileQA, {
  data: datasetName,
  // Need to pass flag to include attachments
  includeAttachments: true,
  evaluators: [validImageDescription],
  client: langsmithClient
});

3. 使用附件更新示例

在上面的代码中，我们展示了如何向数据集添加带附件的示例。也可以使用SDK更新这些相同的示例。

与现有示例一样，使用附件更新数据集时会对数据集进行版本控制。因此，您可以导航到数据集版本历史记录来查看对每个示例所做的更改。要了解更多信息，请参阅 在UI中创建和管理数据集.

更新带有附件的示例时，您可以通过几种不同的方式更新附件：

• * 传入新附件
• * 重命名现有附件
• * 删除现有附件

请注意：

• * 任何未明确重命名或保留的现有附件 **将被删除**.
• * 如果传入不存在的附件名称到 retain or rename.
• * 如果相同的附件名称出现在 attachments 和 attachment_operations fields.

example_update = {
  "id": example_id,
  "attachments": {
      # These are net new attachments
      "my_new_file": ("text/plain", b"foo bar"),
  },
  "inputs": inputs,
  "outputs": outputs,
  # Any attachments not in rename/retain will be deleted.
  # In this case, that would be "my_img" if we uploaded it.
  "attachments_operations": {
      # Retained attachments will stay exactly the same
      "retain": ["my_pdf"],
      # Renaming attachments preserves the original data
      "rename": {
          "my_wav": "my_new_wav",
      }
  },
}

ls_client.update_examples(dataset_id=dataset.id, updates=[example_update])

const exampleUpdate: ExampleUpdateWithAttachments = {
  id: exampleId,
  attachments: {
    // These are net new attachments
    "my_new_file": {
      mimeType: "text/plain",
      data: Buffer.from("foo bar")
    },
  },
  attachments_operations: {
    // Retained attachments will stay exactly the same
    retain: ["my_img"],
    // Renaming attachments preserves the original data
    rename: {
      "my_wav": "my_new_wav",
    },
    // Any attachments not in rename/retain will be deleted
    // In this case, that would be "my_pdf"
  },
};

await langsmithClient.updateExamplesMultipart(dataset.id, [exampleUpdate]);