> Azure Cosmos DB for NoSQL 支持使用灵活架构查询数据,并原生支持 JSON。它现在提供向量索引和搜索功能。此功能旨在处理高维向量,实现任意规模下高效准确的向量搜索。您现在可以直接在文档中与数据一起存储向量。数据库中的每个文档不仅可以包含传统的无架构数据,还可以包含高维向量作为文档的其他属性。
了解如何从以下方式利用 Azure Cosmos DB for NoSQL 的向量搜索功能 本页面。如果您没有 Azure 账户,您可以 创建一个免费账户 开始使用。
设置
您首先需要安装 @langchain/azure-cosmosdb package:
npm install @langchain/azure-cosmosdb @langchain/core
您还需要运行一个 Azure Cosmos DB for NoSQL 实例。您可以按照以下指南在 Azure 门户免费部署一个版本, 本指南.
Once you have your instance running, make sure you have the connection string. You can find them in the Azure Portal, under the "Settings / Keys" section of your instance. Then you need to set the following environment variables:
# Use connection string to authenticate
AZURE_COSMOSDB_NOSQL_CONNECTION_STRING=
# Use managed identity to authenticate
AZURE_COSMOSDB_NOSQL_ENDPOINT=
使用 Azure 托管标识
如果您使用 Azure 托管标识,可以按如下方式配置凭据:
// Create Azure Cosmos DB vector store
const store = new AzureCosmosDBNoSQLVectorStore(new OpenAIEmbeddings(), {
// Or use environment variable AZURE_COSMOSDB_NOSQL_ENDPOINT
endpoint: "https://my-cosmosdb.documents.azure.com:443/",
// Database and container must already exist
databaseName: "my-database",
containerName: "my-container",
});
使用过滤器时的安全注意事项
允许用户提供的原始输入被拼接到类似 SQL 的子句中(例如 WHERE ${userFilter} )会引入严重的 SQL 注入攻击风险,可能会暴露意外数据或损害系统完整性。为缓解此风险,请始终使用 Azure Cosmos DB 的参数化查询机制,传入 @param 占位符,这样可以将查询逻辑与用户提供的输入清晰地分离。
以下是不安全代码的示例:
const store = new AzureCosmosDBNoSQLVectorStore(embeddings, {});
// Unsafe: user-controlled input injected into the query
const userId = req.query.userId; // e.g. "123' OR 1=1"
const unsafeQuerySpec = {
query: `SELECT * FROM c WHERE c.metadata.userId = '${userId}'`,
};
await store.delete({ filter: unsafeQuerySpec });
如果攻击者提供 123 OR 1=1,则查询变为 SELECT * FROM c WHERE c.metadata.userId = '123' OR 1=1,这会强制条件始终为真,导致绕过预期的过滤器并删除所有文档。
为防止此注入风险,您可以定义一个占位符,例如 @userId ,Cosmos DB 会将用户输入作为参数单独绑定,确保其被严格视为数据而非可执行的查询逻辑,如下所示。
const safeQuerySpec: SqlQuerySpec = {
query: "SELECT * FROM c WHERE c.metadata.userId = @userId",
parameters: [{ name: "@userId", value: userId }],
};
await store.delete({ filter: safeQuerySpec });
现在,如果攻击者输入 123 OR 1=1,该输入将被视为要匹配的字符串字面值,而不是查询结构的一部分。
请参阅官方文档 Azure Cosmos DB for NoSQL 中的参数化查询 以获取更多使用示例和最佳实践。
使用示例
以下是演示如何将文件中的文档索引到 Azure Cosmos DB for NoSQL、运行向量搜索查询并最终使用链式调用以自然语言回答问题的示例 基于检索到的文档。
// Load documents from file
const loader = new TextLoader("./state_of_the_union.txt");
const rawDocuments = await loader.load();
const splitter = new RecursiveCharacterTextSplitter({
chunkSize: 1000,
chunkOverlap: 0,
});
const documents = await splitter.splitDocuments(rawDocuments);
// Create Azure Cosmos DB vector store
const store = await AzureCosmosDBNoSQLVectorStore.fromDocuments(
documents,
new OpenAIEmbeddings(),
{
databaseName: "langchain",
containerName: "documents",
}
);
// Performs a similarity search
const resultDocuments = await store.similaritySearch(
"What did the president say about Ketanji Brown Jackson?"
);
console.log("Similarity search results:");
console.log(resultDocuments[0].pageContent);
/*
Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections.
Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.
One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.
And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.
*/
// Use the store as part of a chain
const model = new ChatOpenAI({ model: "gpt-3.5-turbo-1106" });
const questionAnsweringPrompt = ChatPromptTemplate.fromMessages([
[
"system",
"Answer the user's questions based on the below context:\n\n{context}",
],
["human", "{input}"],
]);
const combineDocsChain = await createStuffDocumentsChain({
llm: model,
prompt: questionAnsweringPrompt,
});
const chain = await createRetrievalChain({
retriever: store.asRetriever(),
combineDocsChain,
});
const res = await chain.invoke({
input: "What is the president's top priority regarding prices?",
});
console.log("Chain response:");
console.log(res.answer);
/*
The president's top priority is getting prices under control.
*/
// Clean up
await store.delete();
高级搜索选项
所有搜索类型都通过统一的 similaritySearch 和 similaritySearchWithScore 方法,使用 searchType 参数在筛选选项中。搜索类型可作为常量从 AzureCosmosDBNoSQLSearchType:
可用的搜索类型:
- -
AzureCosmosDBNoSQLSearchType.Vector(默认):标准向量相似性搜索 - -
AzureCosmosDBNoSQLSearchType.VectorScoreThreshold:带有最低分数筛选器的向量搜索 - -
AzureCosmosDBNoSQLSearchType.FullTextSearch:使用 FullTextContains 的全文搜索(预览) - -
AzureCosmosDBNoSQLSearchType.FullTextRanking:使用 BM25 排名的全文搜索(预览) - -
AzureCosmosDBNoSQLSearchType.Hybrid:使用 RRF 的混合向量 + 全文搜索(预览) - -
AzureCosmosDBNoSQLSearchType.HybridScoreThreshold:带有分数阈值的混合搜索(预览)
创建存储时,您也可以使用 defaultSearchType 配置选项设置默认搜索类型,这样您就不必在每次查询时指定它:
AzureCosmosDBNoSQLVectorStore,
AzureCosmosDBNoSQLSearchType,
} from "@langchain/azure-cosmosdb";
const store = new AzureCosmosDBNoSQLVectorStore(new OpenAIEmbeddings(), {
databaseName: "langchain",
containerName: "documents",
defaultSearchType: AzureCosmosDBNoSQLSearchType.VectorScoreThreshold,
});
带分数阈值的向量搜索
根据最低相似性分数筛选结果:
AzureCosmosDBNoSQLVectorStore,
AzureCosmosDBNoSQLSearchType,
} from "@langchain/azure-cosmosdb";
const store = new AzureCosmosDBNoSQLVectorStore(new OpenAIEmbeddings(), {
databaseName: "langchain",
containerName: "documents",
});
// Only return results with similarity score >= 0.8
const results = await store.similaritySearchWithScore(
"What is the capital of France?",
10,
{
searchType: AzureCosmosDBNoSQLSearchType.VectorScoreThreshold,
threshold: 0.8,
}
);
for (const [doc, score] of results) {
console.log(`Score: ${score}, Content: ${doc.pageContent}`);
}
最大边际相关性(MMR)搜索
MMR 搜索在结果中平衡相关性和多样性:
const store = new AzureCosmosDBNoSQLVectorStore(new OpenAIEmbeddings(), {
databaseName: "langchain",
containerName: "documents",
});
const results = await store.maxMarginalRelevanceSearch("machine learning", {
k: 5, // Number of results to return
fetchK: 20, // Number of candidates to consider
lambda: 0.5, // 0 = max diversity, 1 = max relevance
});
全文和混合搜索(预览)
要使用全文或混合搜索,请在创建存储时启用它:
AzureCosmosDBNoSQLVectorStore,
AzureCosmosDBNoSQLSearchType,
} from "@langchain/azure-cosmosdb";
const store = new AzureCosmosDBNoSQLVectorStore(new OpenAIEmbeddings(), {
databaseName: "langchain",
containerName: "documents",
fullTextSearchEnabled: true,
fullTextPolicy: {
defaultLanguage: "en-US",
fullTextPaths: [{ path: "/text", language: "en-US" }],
},
indexingPolicy: {
indexingMode: "consistent",
automatic: true,
includedPaths: [{ path: "/*" }],
excludedPaths: [{ path: "/_etag/?" }],
vectorIndexes: [{ path: "/vector", type: "quantizedFlat" }],
fullTextIndexes: [{ path: "/text" }],
},
});
全文搜索
// Full-text search using FullTextContains in the filter clause
const fullTextResults = await store.similaritySearch("", 10, {
searchType: AzureCosmosDBNoSQLSearchType.FullTextSearch,
filterClause: "WHERE FullTextContains(c.text, 'artificial intelligence')",
});
全文排名
// Full-text ranking with BM25 scoring
const rankingResults = await store.similaritySearch("", 10, {
searchType: AzureCosmosDBNoSQLSearchType.FullTextRanking,
fullTextRankFilter: [
{ searchField: "text", searchText: "artificial intelligence" },
],
});
混合搜索
混合搜索使用倒数排名融合(RRF)将向量相似性与全文搜索相结合:
// Hybrid search combining vector and full-text results
const hybridResults = await store.similaritySearchWithScore(
"machine learning",
10,
{
searchType: AzureCosmosDBNoSQLSearchType.Hybrid,
fullTextRankFilter: [
{ searchField: "text", searchText: "machine learning" },
],
}
);
// Hybrid search with score threshold
const filteredResults = await store.similaritySearchWithScore(
"machine learning",
10,
{
searchType: AzureCosmosDBNoSQLSearchType.HybridScoreThreshold,
fullTextRankFilter: [
{ searchField: "text", searchText: "machine learning" },
],
threshold: 0.5,
}
);
实用方法
删除文档
// Delete specific documents by ID
await store.delete({ ids: ["document-id-123"] });
// Delete documents matching a filter
await store.delete({
filter: {
query: "SELECT * FROM c WHERE c.metadata.category = @category",
parameters: [{ name: "@category", value: "old" }],
},
});
// Delete all documents
await store.delete();
访问基础容器
// Get direct access to the Cosmos DB container for advanced operations
const container = store.getContainer();
const { resources } = await container.items
.query("SELECT * FROM c WHERE c.metadata.category = 'tech'")
.fetchAll();