语言模型有 token 限制。你不应该超过 token 限制。当你 分割你的文本 成块时，因此最好计算 token 的数量。有很多分词器。当你在文本中计算 token 时，你应该使用与语言模型中使用的相同分词器。

tiktoken

我们可以使用 tiktoken 来估算使用的 token。它可能对 OpenAI 模型更准确。

1. 文本如何分割：通过传入的字符。
2. 块大小如何测量：通过 tiktoken tokenizer.

CharacterTextSplitter、RecursiveCharacterTextSplitter 和 TokenTextSplitter 可以与 tiktoken directly.

pip install --upgrade --quiet langchain-text-splitters tiktoken

from langchain_text_splitters import CharacterTextSplitter

# This is a long document we can split up.
with open("state_of_the_union.txt") as f:
    state_of_the_union = f.read()

要使用 CharacterTextSplitter 分割文本，然后使用 tiktoken合并块，请使用其 .from_tiktoken_encoder() 方法。请注意，从这个方法得到的分割可能比通过 tiktoken tokenizer.

测量的块大小更大。 .from_tiktoken_encoder() 方法接受 encoding_name 作为参数（例如 cl100k_base），或者 model_name (e.g. gpt-4）。所有额外参数如 chunk_size, chunk_overlap和 separators 都用于实例化 CharacterTextSplitter:

text_splitter = CharacterTextSplitter.from_tiktoken_encoder(
    encoding_name="cl100k_base", chunk_size=100, chunk_overlap=0
)
texts = text_splitter.split_text(state_of_the_union)

print(texts[0])

Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.

Last year COVID-19 kept us apart. This year we are finally together again.

Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans.

With a duty to one another to the American people to the Constitution.

要实现对块大小的硬约束，我们可以使用 RecursiveCharacterTextSplitter.from_tiktoken_encoder，其中每个分割如果大小更大将被递归分割：

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    model_name="gpt-4",
    chunk_size=100,
    chunk_overlap=0,
)

我们还可以加载一个 TokenTextSplitter 分割器，它直接处理 tiktoken 并确保每个分割小于块大小。

from langchain_text_splitters import TokenTextSplitter

text_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0)

texts = text_splitter.split_text(state_of_the_union)
print(texts[0])

Madam Speaker, Madam Vice President, our

某些书面语言（例如中文和日语）有被编码为两个或更多 token 的字符。直接使用 TokenTextSplitter 可能会将字符的 token 分割到两个块中，导致格式错误的 Unicode 字符。使用 RecursiveCharacterTextSplitter.from_tiktoken_encoder or CharacterTextSplitter.from_tiktoken_encoder 来确保块包含有效的 Unicode 字符串。

spaCy

LangChain 实现了基于 分词器的分割器.

1. 文本如何分割：通过 spaCy tokenizer.
2. 块大小如何测量：通过字符数。

pip install --upgrade --quiet  spacy

# This is a long document we can split up.
with open("state_of_the_union.txt") as f:
    state_of_the_union = f.read()

from langchain_text_splitters import SpacyTextSplitter

text_splitter = SpacyTextSplitter(chunk_size=1000)

texts = text_splitter.split_text(state_of_the_union)
print(texts[0])

Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.

Members of Congress and the Cabinet.

Justices of the Supreme Court.

My fellow Americans.



Last year COVID-19 kept us apart.

This year we are finally together again.



Tonight, we meet as Democrats Republicans and Independents.

But most importantly as Americans.



With a duty to one another to the American people to the Constitution.



And with an unwavering resolve that freedom will always triumph over tyranny.



Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.

But he badly miscalculated.



He thought he could roll into Ukraine and the world would roll over.

Instead he met a wall of strength he never imagined.



He met the Ukrainian people.



From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.

SentenceTransformers

该 SentenceTransformersTokenTextSplitter 是一个专门用于sentence-transformer模型的分词器。默认行为是将文本分割成适合您想要使用的sentence-transformer模型token窗口大小的块。

要按照sentence-transformers分词器分割文本并限制token数量，请实例化一个 SentenceTransformersTokenTextSplitter。您可以选择指定：

• - chunk_overlap：token重叠的整数数量；
• - model_name：sentence-transformer模型名称，默认为 "sentence-transformers/all-mpnet-base-v2";
• - tokens_per_chunk：每个块的期望token数量。

from langchain_text_splitters import SentenceTransformersTokenTextSplitter

splitter = SentenceTransformersTokenTextSplitter(chunk_overlap=0)
text = "Lorem "

count_start_and_stop_tokens = 2
text_token_count = splitter.count_tokens(text=text) - count_start_and_stop_tokens
print(text_token_count)

2

token_multiplier = splitter.maximum_tokens_per_chunk // text_token_count + 1

# `text_to_split` does not fit in a single chunk
text_to_split = text * token_multiplier

print(f"tokens in text to split: {splitter.count_tokens(text=text_to_split)}")

tokens in text to split: 514

text_chunks = splitter.split_text(text=text_to_split)

print(text_chunks[1])

lorem

NLTK

Rather than just splitting on "\n\n", we can use NLTK 基于 NLTK分词器.

1. 文本如何分割：按 NLTK tokenizer.
2. 如何测量块大小：按字符数。

# pip install nltk

# This is a long document we can split up.
with open("state_of_the_union.txt") as f:
    state_of_the_union = f.read()

from langchain_text_splitters import NLTKTextSplitter

text_splitter = NLTKTextSplitter(chunk_size=1000)

texts = text_splitter.split_text(state_of_the_union)
print(texts[0])

Madam Speaker, Madam Vice President, our First Lady and Second Gentleman.

Members of Congress and the Cabinet.

Justices of the Supreme Court.

My fellow Americans.

Last year COVID-19 kept us apart.

This year we are finally together again.

Tonight, we meet as Democrats Republicans and Independents.

But most importantly as Americans.

With a duty to one another to the American people to the Constitution.

And with an unwavering resolve that freedom will always triumph over tyranny.

Six days ago, Russia’s Vladimir Putin sought to shake the foundations of the free world thinking he could make it bend to his menacing ways.

But he badly miscalculated.

He thought he could roll into Ukraine and the world would roll over.

Instead he met a wall of strength he never imagined.

He met the Ukrainian people.

From President Zelenskyy to every Ukrainian, their fearlessness, their courage, their determination, inspires the world.

Groups of citizens blocking tanks with their bodies.

KoNLPY

分词涉及将文本分割成称为token的更小、更易于管理的单元。这些token通常是单词、短语、符号或其他对进一步处理和分析至关重要的有意义的元素。在英语等语言中，分词通常涉及按空格和标点符号分隔单词。分词的有效性很大程度上取决于分词器对语言结构的理解，确保生成有意义的token。由于为英语设计的分词器不具备理解其他语言（如韩语）独特语义结构的能力，因此它们不能有效地用于韩语处理。

### 使用KoNLPY的kkma分析器进行韩语分词 对于韩语文本，KoNLPY包含一个名为 Kkma （韩语知识形态分析器）的形态分析器。 Kkma 提供详细的韩语文本形态分析。它将句子分解成单词，将单词分解成各自的形态素，为每个token识别词性。它可以将一段文本分割成单独的句子，这对于处理长文本特别有用。

### 使用注意事项 虽然 Kkma 以其详细分析而闻名，但值得注意的是这种精确性可能会影响处理速度。因此， Kkma 最适合将分析深度置于快速文本处理之上的应用场景。

# pip install konlpy

# This is a long Korean document that we want to split up into its component sentences.
with open("./your_korean_doc.txt") as f:
    korean_document = f.read()

from langchain_text_splitters import KonlpyTextSplitter

text_splitter = KonlpyTextSplitter()

texts = text_splitter.split_text(korean_document)
# The sentences are split with "\n\n" characters.
print(texts[0])

춘향전 옛날에 남원에 이 도령이라는 벼슬아치 아들이 있었다.

그의 외모는 빛나는 달처럼 잘생겼고, 그의 학식과 기예는 남보다 뛰어났다.

한편, 이 마을에는 춘향이라는 절세 가인이 살고 있었다.

춘 향의 아름다움은 꽃과 같아 마을 사람들 로부터 많은 사랑을 받았다.

어느 봄날, 도령은 친구들과 놀러 나갔다가 춘 향을 만 나 첫 눈에 반하고 말았다.

두 사람은 서로 사랑하게 되었고, 이내 비밀스러운 사랑의 맹세를 나누었다.

하지만 좋은 날들은 오래가지 않았다.

도령의 아버지가 다른 곳으로 전근을 가게 되어 도령도 떠나 야만 했다.

이별의 아픔 속에서도, 두 사람은 재회를 기약하며 서로를 믿고 기다리기로 했다.

그러나 새로 부임한 관아의 사또가 춘 향의 아름다움에 욕심을 내 어 그녀에게 강요를 시작했다.

춘 향 은 도령에 대한 자신의 사랑을 지키기 위해, 사또의 요구를 단호히 거절했다.

이에 분노한 사또는 춘 향을 감옥에 가두고 혹독한 형벌을 내렸다.

이야기는 이 도령이 고위 관직에 오른 후, 춘 향을 구해 내는 것으로 끝난다.

두 사람은 오랜 시련 끝에 다시 만나게 되고, 그들의 사랑은 온 세상에 전해 지며 후세에까지 이어진다.

- 춘향전 (The Tale of Chunhyang)

Hugging Face分词器

Hugging Face 有许多分词器。

我们使用Hugging Face分词器，即 GPT2TokenizerFast 来计算文本的token长度。

1. 文本如何分割：按传入的字符。
2. 如何测量块大小：按 Hugging Face tokenizer.

from transformers import GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained("gpt2")

# This is a long document we can split up.
with open("state_of_the_union.txt") as f:
    state_of_the_union = f.read()
from langchain_text_splitters import CharacterTextSplitter

text_splitter = CharacterTextSplitter.from_huggingface_tokenizer(
    tokenizer, chunk_size=100, chunk_overlap=0
)
texts = text_splitter.split_text(state_of_the_union)

print(texts[0])

Madam Speaker, Madam Vice President, our First Lady and Second Gentleman. Members of Congress and the Cabinet. Justices of the Supreme Court. My fellow Americans.

Last year COVID-19 kept us apart. This year we are finally together again.

Tonight, we meet as Democrats Republicans and Independents. But most importantly as Americans.

With a duty to one another to the American people to the Constitution.