Smart Markdown Parsing Boosts RAG Accuracy Without Breaking Tables

Retrieval-Augmented Generation (RAG) pipelines depend on clean, complete data chunks to deliver accurate answers. Yet many developers overlook a critical flaw: traditional fixed-size token splitters can bisect markdown tables, orphan list items, or truncate code blocks mid-stream. The result? A RAG system confidently answers questions with half the story, leaving users frustrated and engineers puzzled.

Consider a support engineer querying a documentation bot about a failing webhook retry. The retriever returns the first two rows of a markdown table—header and a partial body—while the rows containing the actual 429 status code vanish into the next chunk. No embedder upgrade or reranker tweak can fix this. The solution starts with a splitter that treats markdown structures as sacred, not as arbitrary text.

Why Fixed-Size Token Splitting Fails on Structured Content

Most RAG tutorials default to simple token-based chunking for one reason: it’s concise and predictable. A single function call divides any document into fixed-length segments, and the output is neatly predictable. Yet this approach ignores the inherent structure of technical documents. Tables are split mid-row, code blocks lose their opening fences, and bulleted lists surrender the final items under headings the retriever never sees.

import tiktoken

def fixed_chunks(text: str, size: int = 512) -> list[str]:
    enc = tiktoken.encoding_for_model("gpt-4o")
    tokens = enc.encode(text)
    return [
        enc.decode(tokens[i : i + size])
        for i in range(0, len(tokens), size)
    ]

This code slices content at fixed 512-token intervals, regardless of context. When token 512 lands in the middle of a markdown table row like | Status Code | Retry After |, half the row vanishes into chunk N, while the other half drifts into chunk N+1. The retriever embeds two fragments of a single idea, and neither half answers the question on its own.

A February 2026 benchmark across 50 academic papers evaluated various chunking strategies. Recursive 512-token splitting achieved 69% retrieval accuracy, while pure semantic chunking lagged at 54%. The recursive approach succeeds because it respects document boundaries—an idea this tokenizer-aware markdown splitter extends specifically to structured documents.

Parsing Markdown into Atomic Structure-Aware Blocks

The first step is to transform raw markdown into a structured representation where each element—headings, paragraphs, code blocks, tables, and lists—is treated as an atomic unit. This prevents internal splits that break meaning.

import re
from dataclasses import dataclass

@dataclass
class Block:
    kind: str  # h1..h6, para, code, table, list
    level: int  # heading level, else 0
    text: str

A lightweight parser walks through the markdown line by line, grouping related lines into blocks. It uses regular expressions to detect headings, code fences, table rows, and list items. Each recognized structure becomes a single Block object, ensuring that tables, code blocks, and lists are never split internally.

HEADING = re.compile(r"^(#{1,6})\s+(.*)$")
FENCE = re.compile(r"^`{3}")
TABLE_ROW = re.compile(r"^\s*\|.*\|\s*$")
LIST_ITEM = re.compile(r"^\s*([-*+]|\d+\.)\s+")

The parser includes helper functions like _consume_code, _consume_table, and _consume_list, each accumulating lines until the run ends. When a block is complete, it’s stored as a single unit. This guarantees that a 10-row table remains intact, a multi-line code block stays together, and a bulleted list isn’t orphaned mid-stream.

Token Budgeting: Soft Limits with Hard Boundaries

Once markdown is parsed into blocks, the next challenge is to assemble chunks that stay within token limits while preserving meaning. A soft budget allows slight flexibility: aim for a target token count per chunk, but never exceed a hard_max. For embedders with 512-token windows, setting target=480 and hard_max=512 strikes a balance. For long-context models like text-embedding-3-large (8,191 tokens), target=800 and hard_max=1024 aligns with production best practices.

import tiktoken

ENC = tiktoken.encoding_for_model("gpt-4o")

def n_tokens(text: str) -> int:
    return len(ENC.encode(text))

Caching token counts per block avoids repeated encoding, a hidden performance bottleneck in naive splitters. For large corpora, precompute and store token counts during parsing to speed up chunking significantly.

Greedy Packing with Priority-Based Splits

The core algorithm assembles blocks into chunks using a greedy packing strategy. It iterates through the parsed blocks, adding each to the current buffer if it fits within the target token limit. If a single block exceeds the hard_max, it’s split recursively by priority: headings first, then paragraphs, then sentences, and finally words. Headings always migrate to the next chunk to avoid orphaned titles.

def pack(
    blocks: list[Block],
    target: int = 480,
    hard_max: int = 512,
) -> list[str]:
    chunks: list[str] = []
    buf: list[Block] = []
    buf_tokens = 0

    def flush():
        nonlocal buf, buf_tokens
        if buf:
            chunks.append("\n\n".join(b.text for b in buf))
        buf, buf_tokens = [], 0

    for blk in blocks:
        bt = n_tokens(blk.text)
        if bt > hard_max:
            flush()
            for piece in split_oversize(blk, hard_max):
                chunks.append(piece)
            continue

        if buf_tokens + bt > target and buf_tokens > 0:
            flush()
        buf.append(blk)
        buf_tokens += bt

    flush()
    return chunks

This ensures no chunk is left empty due to premature flushing, and oversized blocks are handled immediately without disrupting the buffer. A 700-token paragraph won’t be stranded; it’s split logically, preserving readability and structure.

Looking Ahead: Structured Chunking for Smarter RAG

The era of treating documents as plain text is fading. As RAG systems grow more sophisticated, structured parsing and token-aware chunking will become table stakes, not novelties. Tools like this markdown splitter prove that respecting structure doesn’t require heavyweight libraries—just thoughtful design and a clear understanding of how models consume data.

For developers building documentation bots, code assistants, or internal knowledge bases, the shift from blind token slicing to intelligent block-aware chunking can mean the difference between answers that mislead and ones that enlighten. The future of RAG isn’t just about bigger models or better embeddings—it’s about smarter ways to feed them.