Why splitting LLM prompts improves extraction accuracy for long docs

Structured data extraction from lengthy documents using a single large language model (LLM) prompt may appear efficient at first glance, yet this approach frequently encounters hidden pitfalls. Many tutorials suggest feeding entire documents into one prompt, instructing the model to extract multiple fields simultaneously and return structured output. While this works for short inputs, production-scale documents reveal critical flaws in the method.

A shift in strategy—breaking the extraction process into two distinct stages—can resolve these issues while improving accuracy, scalability, and reliability. This approach mirrors an accordion’s structure: expanding to segment the document, then contracting to extract precise data from each segment. Here’s why this method outperforms the traditional single-prompt approach and how to implement it effectively.

The limitations of a single oversized prompt

The instinct to consolidate extraction into one prompt is understandable. For a 50-page report, developers often craft a prompt like:

• Extract: title, sections with headings, purpose, services mentioned, acceptance criteria, and more.
• Return the results as structured JSON.

Initially, this may produce acceptable results. However, as document length increases, three major problems emerge:

• Quality inconsistency. Models struggle to maintain consistency across long documents. Later sections may generate weaker summaries, omit fields, or hallucinate details compared to earlier ones.

• Single-point failure. An error in one field—such as a hallucinated acceptance criterion—can invalidate the entire output. This triggers manual review queues and delays processing pipelines.

• Performance bottlenecks. A 30,000-token prompt consumes substantial time and resources. Parallel processing becomes impossible, as each document must be processed sequentially, increasing latency and reducing throughput.

Attempts to mitigate these issues—such as adding more examples, tightening formatting rules, or expanding the prompt—often yield only marginal gains while inflating engineering overhead. The underlying structural flaw remains unresolved.

The accordion pattern: split extraction into two focused stages

The solution lies in decomposing the extraction process into two specialized stages, each governed by a dedicated prompt. This method leverages the concept of an accordion: it first expands the document into manageable segments, then contracts each segment to extract only the required fields.

Stage 1: Document segmentation

• Input: Full document.
• Prompt task: Identify and return a clean array of logical segments—such as sections, paragraphs, or table rows—without attempting to extract structured data.
• Output: A JSON list where each item contains a title and text for a segment.

Stage 2: Field extraction

• Input: One segment at a time.
• Prompt task: Extract specific fields (e.g., purpose, mentioned services, acceptance criteria) from the provided segment.
• Output: Structured JSON record for that segment.

This approach separates the concerns of segmentation and extraction, allowing each prompt to focus on a single responsibility. The first prompt learns to identify boundaries; the second, to extract schema-compliant fields. Neither needs to juggle multiple objectives simultaneously.

Advantages of the two-stage extraction model

This method introduces several key benefits over the single-prompt alternative:

• Single-purpose prompts improve accuracy. Each prompt receives a clearer directive and fewer distractions. Examples are shorter, more relevant, and easier to maintain. The model can focus on one task at a time, reducing hallucinations and omissions.

• Localized errors reduce operational burden. If Stage 2 fails on segment 7, you only need to reprocess segment 7—not the entire document. Failed records can be isolated, retried, or quarantined without disrupting successful extractions.

• Natural parallelization accelerates processing. Stage 1 produces an array of segments. These can be distributed across multiple workers or API calls, enabling concurrent Stage 2 extractions. A 50-segment document can be processed 50 times faster than a single large call.

• Caching becomes feasible. Repeated segments—such as standard headers, footers, or boilerplate text—can be cached at the Stage 2 level. This reduces redundant LLM calls and improves cost efficiency over time.

• Context window constraints vanish. While Stage 1 must read the full document, its output is compact. Stage 2 only ever receives a single segment, so the model’s context window is no longer a limiting factor—even for extremely long documents.

Trade-offs and design considerations

Implementing this pattern is not without cost. It requires more LLM calls—one for segmentation plus one per segment—compared to a single call. For short inputs like two-paragraph emails, this overhead may outweigh the benefits. The accordion approach is best suited for documents long enough to cause quality drift in single-prompt systems.

Additionally, defining what constitutes a "segment" is a deliberate design choice. It might align with section headings, table rows, or logical units that don’t correspond to visible boundaries. This decision directly impacts extraction quality and must be tailored to the document type and extraction goals.

When to adopt the accordion pattern (and when to avoid it)

Use the two-stage extraction method when:

• The input document is lengthy enough to cause the model to lose coherence midway through.
• The desired output schema includes more than five fields that don’t all depend on the same contextual cues.
• You need the ability to retry failed records without reprocessing successful ones.
• Parallel execution is a priority to reduce wall-clock time and improve throughput.

Stick with a single, comprehensive prompt for:

• Short documents where context is naturally retained throughout.
• Extraction tasks where fields are tightly interdependent (e.g., extracting a value based on another extracted value).
• Prototyping phases where speed of iteration matters more than reliability or scale.

A real-world implementation example

Consider a service like StructFlow, which uses this pattern in production. The workflow is straightforward:

Stage 1: Segmentation

{
  "model": "google/gemini-3-flash-preview",
  "system_prompt": "Split this document into logical sections. Return one JSON record per section.",
  "example_output": {
    "section_title": "Introduction",
    "section_text": "...the first paragraph..."
  },
  "inputs": [
    {
      "id": "doc1",
      "data": { "text": "[Full document text here]" }
    }
  ]
}

The response provides an array of segments. Each segment includes a title and clean text.

Stage 2: Extraction (per segment, parallelized)

{
  "model": "google/gemini-3-flash-preview",
  "system_prompt": "From this section, extract: purpose, mentioned services, acceptance criteria.",
  "example_output": {
    "purpose": "Define user authentication flow",
    "mentioned_services": ["AWS Cognito", "Auth0"],
    "acceptance_criteria": ["User can log in within 3 seconds"]
  },
  "inputs": [
    {
      "id": "sec1",
      "data": { "section_text": "[Text of section 1]" }
    }
  ]
}

This process repeats for every segment, with each call running independently and in parallel.

Final thoughts and open questions

The core insight here isn’t tied to any specific tool or API. Whether you use a custom pipeline, an open-source framework, or a managed platform, the pattern remains the same: long documents require long prompts to fail. A single comprehensive prompt rarely scales gracefully.

While the accordion method introduces added complexity, its benefits in accuracy, fault isolation, and performance are undeniable for large-scale extraction tasks. As LLM-based systems continue to handle increasingly complex documents, architectures that embrace modularity and specialization will define the next generation of robust data pipelines.

Have you experimented with multi-stage extraction? Or perhaps you’ve encountered scenarios where even this pattern breaks down? The conversation is just beginning—and your experience matters.