Chunking
The critical preprocessing step of splitting massive documents into smaller, semantically coherent text segments for ingestion into a vector search engine.
Think of It Like This
Like slicing a massive wedding cake into individual portions so people can actually eat it, making sure not to cut halfway through a decorative flower.
Since embedding models have hard limits on how many tokens they can process at once, entire books must be broken down before they can be searched. If chunks are too small, they lose critical context; if they are too large, the specific detail a user is searching for gets drowned out by unrelated noise. Advanced techniques use overlap or semantic boundaries like paragraphs instead of fixed character counts.