RAG PDF chunking: full text into the base or extract fields? A counterintuitive conclusion

When doing a RAG project, most people’s attention is on the tech selection: Milvus or Chroma, which Embedding model, what Top-K.

Those are later concerns. The first decision is actually: after the PDF is parsed, what exactly do you store in the base.

This has two completely different paths; choose wrong and you have to tear it all down and restart.

Two paths

Full-text chunking ingestion: PDF → extract text → split into chunks by strategy → vectorize → ingest. You keep the document’s full content, and at query time rely on semantic similarity to hit.

Structured extraction ingestion: don’t store the full text; use an LLM to pull out key fields and ingest them as JSON. For an imaging report, the extracted fields might be four: score, tumor size, location, lymph-node status.

Most tutorials only cover the first path, because it’s general. But the second path is the only correct answer in certain scenarios.

Several full-text chunking strategies

If you go the full-text route, the chunking strategy directly decides retrieval quality. Four common ones:

• Fixed size: split by fixed token count, with overlap. Fast, but may cut a sentence in half
• Semantic splitting: split by sentence or paragraph boundaries, merge similar content. High precision, troublesome to implement
• Recursive splitting: split coarsely first, then subdivide overlong blocks. Good for long documents
• Structure-based splitting: split along the hierarchy of headings and sections. Good for papers, whitepapers — documents that already have structure

In practice recursive splitting is used most, because it has the best cost-performance of the bunch:

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=50,
    separators=["\n\n", "\n", "。", "！", "？", "；", "，", " ", ""]
)
chunks = text_splitter.split_text(pdf_text)

Note chunk_overlap. Newcomers often set it to 0, and then a key piece of info happens to straddle the boundary of two chunks; each chunk has half a sentence, and at query time neither can hit. An overlap of 50 to 100 is the common range.

Another easily-overlooked point: whether to put Chinese punctuation in separators. The config above includes 。！？；， — a must when processing Chinese documents. The default English separators basically don’t work on Chinese paragraphs; in the end you’d get a single 5000-character chunk.

What structured extraction looks like

The key to this path is the prompt. Taking an imaging report as an example:

Extract from the following imaging report text:
1. BI-RADS score
2. Tumor size
3. Location description
4. Lymph-node status

Report text: {report_text}
Return in JSON format.

The output is a clean struct. When a doctor asks “how many patients recently tested EGFR-mutation positive,” only this path can answer — full-text chunking ingestion simply can’t do statistical queries.

A set of counterintuitive data

This is the thing I most want to record in this piece.

A study compared 7,294 imaging reports and 2,154 pathology reports, and the result went against my expected direction:

Report type	RAG effect	Reason
Imaging report (short)	slightly decreased	the report itself is short; retrieval introduces noise
Pathology report (long, complex)	+48%	needs to precisely locate relevant info from long text

That RAG on short reports actually drags things down is worth pausing on.

We assume by default “RAG is always better than just stuffing context,” but the premise is that the document is too long for context to hold, or it fits but with too much noise. When the document itself is short and stuffing the whole thing into context is effortless, every irrelevant chunk the retrieval step introduces is a negative asset.

So before deciding whether to use RAG, first look at your document-length distribution, not at whether everyone else is doing RAG.

What’s special about the medical scenario

Medicine is the domain with the most tension on this issue, because it simultaneously wants two conflicting things: fully preserve context and precisely extract key info.

When a patient asks “what are the side effects of this drug,” what’s needed is an open-ended, context-rich answer — full-text chunking fits. When a doctor asks “how many patients tested EGFR-mutation positive in the last three months,” what’s needed is precise field statistics — only structured extraction can do it.

My current conclusion is that these two paths aren’t either/or, but both, routed by query type. The logic can be simple: queries with statistical intent (how many, ratio, average) go structured; queries with semantic intent (why, how, what side effects) go vector retrieval.

This judgment itself can be handed to an LLM, at the cost of one extra call.

Pitfalls I hit

Tables got shredded. Tables extracted from a PDF with normal text extraction become a string of structureless text; after chunking they’re basically unusable. Later I did table recognition first, converting tables separately into markdown or structured data, and ran the remaining body text through the chunking flow.

Headers and footers leaked into the body. Medical reports have hospital name, page number, and print time on every page. If not cleaned, they pollute the embedding — and because they appear on every page, their presence in vector space is actually strong. The cleaning rules look crude, but you can’t skip them.

Scanned images. Some PDFs are scanned; extracting them directly yields empty or garbled text. First judge whether the text layer is missing; if so, go through OCR. OCR errors propagate all the way down, so OCR quality needs separate spot-checking.

Summary

Compress the decision into three questions:

1. Will users ask statistical questions? If yes → you must have the structured-extraction path
2. Are documents generally short? If yes → first try no retrieval, just stuff context; don’t default to RAG
3. Are there tables and scanned pages in the PDFs? If yes → the parsing layer’s workload will exceed your estimate; schedule it separately

Tech selection (vector DB, Embedding model) certainly matters, but relative to the “what to store” decision, they’re all details.