DEV Community

Cover image for The Document Parsing Bottleneck: LlamaParse, AST Layout Engines, and Table Reconstruction in Production RAG
Soham Sharma
Soham Sharma

Posted on Edited on Originally published at botmartz.com

The Document Parsing Bottleneck: LlamaParse, AST Layout Engines, and Table Reconstruction in Production RAG

The Document Parsing Bottleneck: Multimodal AST Layout Engines, Table Reconstruction, and Deterministic Retrieval in High-Consequence Enterprise RAG

Botmartz AI Research & Engineering Systems Group

Technical Dissertation & Systems Specification Series — Vol. IV

Corresponding Open-Source Infrastructure: botmartz/llamaparse-pgvector-pipeline

Benchmark Artifact Ledgers: Hugging Face: botmartzai/enterprise-doc-ast-bench | Kaggle: sohamnsharma/enterprise-doc-ast-bench


Abstract

Enterprise Retrieval-Augmented Generation (RAG) deployments in mission-critical environments—such as statutory compliance, corporate finance, aerospace maintenance, clinical trials, and commercial liability adjudication—consistently fail not at the language model generation layer, but at the document ingestion interface. Standard document ingestion architectures rely on naive text extractors or heuristic OCR pipelines that linearize two-dimensional page geometries into one-dimensional character sequences. This operational abstraction discards essential topological structures: coordinate-bounded tables, multi-tier heading hierarchies, spanning column groupings, and footnote associations.

In this dissertation, we examine the mechanics of document ingestion failure across complex enterprise documents. We formalize the document reconstruction problem through three mathematical metrics: Table Cell Intersection-over-Union ($IoU_{cell}$), Heading Hierarchy Tree Edit Distance ($TED_{hier}$), and Reciprocal Rank Fusion ($RRF$) degradation under layout perturbation. We present an empirical ablation across dense embedding models demonstrating how tabular layout destruction induces significant drops in cell-level semantic similarity, guaranteeing nearest-neighbor retrieval failures under heuristic extraction.

To resolve this limitation, we present a systematic evaluation of Multimodal Vision-Language Abstract Syntax Tree (AST) layout engines, comparing cloud engines (LlamaParse) and sovereign open-source local models (IBM Docling with TableFormer, MinerU) against traditional rule-based OCR (Tesseract v5) and heuristic spatial parsers (PyPDF) across our public synthetic stress-test benchmark suite: Enterprise-Doc-AST-Bench-Synthetic (940 parameterized evaluation test fixtures across financial balance sheets and aerospace maintenance schemas). Multimodal AST engines achieve up to a 0.964 Table Cell IoU and a 0.942 Heading Tree F1 score, outperforming traditional OCR by 52.1 percentage points.

Finally, we provide the complete production architecture and code for botmartz/llamaparse-pgvector-pipeline: an end-to-end open-source connector that transforms markdown AST output into structured document nodes, enforces Table Inviolability during semantic chunking, and indexes chunks into PostgreSQL using pgvector HNSW indexes (with halfvec supporting 3072-dimensional embeddings beyond pgvector's 2,000-dimension boundary) and GIN full-text indexes with reciprocal rank fusion hybrid search.


1. The Anatomy of Ingestion Failure in Regulated Enterprise Documents

1.1 The PostScript and PDF Structural Paradigm

To understand why document retrieval pipelines fail in enterprise deployments, one must first understand the underlying digital storage medium: Adobe Portable Document Format (PDF, ISO 32000-2). The PDF format was engineered in 1993 as an evolution of PostScript. Its architectural design objective was visual fidelity for 2D print output, entirely detached from semantic document structure.

A standard PDF file does not store paragraphs, tables, rows, or markdown headings. Instead, a PDF content stream consists of imperative graphic operators and character glyph placement routines. The document interpreter executes instructions such as:

BT
/F1 10.5 Tf
72.00 712.50 Td
[(T) 12 (able) -5 ( 1: ) 8 (Consolidated) -4 ( Balance) 10 ( Sheets)] TJ
ET
Enter fullscreen mode Exit fullscreen mode

In this execution trace, character glyphs are positioned at absolute coordinate offsets $(x, y)$ measured in typographical points (1/72 inch) relative to the bottom-left or top-left corner of the page. Crucially:

  1. There is no intrinsic concept of a table. A grid is rendered by drawing independent vector lines (re, m, l, S operators) while textual characters are placed into the spatial regions between lines.
  2. Words that visually appear in the same horizontal row are frequently stored in non-sequential streams within the PDF object hierarchy. The stream may define column 3 first, followed by column 1, followed by a background watermarked footer.
  3. Spanned cells and nested subheaders have no explicit metadata tags unless the document was authored as an accessible PDF/UA (Universal Accessibility) file—a standard representing less than 3% of enterprise legacy documents.

When an enterprise RAG pipeline ingests an SEC 10-K filing or an aerospace structural maintenance manual using standard programmatic libraries (e.g., PyPDF, PDFMiner, or pdfplumber), the library reads raw text streams sequentially. In the absence of an explicit bounding box reconstruction algorithm, the stream output collapses into an unformatted character sequence.


1.2 The Failure Modes of Heuristic and Rule-Based OCR

When rule-based OCR engines (e.g., Tesseract) or heuristic spatial parsers attempt to extract structured data from complex documents, they introduce three major systematic failure modes:

1. The Reading Order Inversion Problem ($Z$-Order vs. Column-First)

In multi-column enterprise layouts (e.g., regulatory notices, technical papers, and financial releases), text flows vertically down Column 1 before proceeding to the top of Column 2. Rule-based heuristic extractors sort text bounding boxes by their vertical $y$-coordinates. When two text columns share overlapping vertical bands, heuristic sorting interleaves sentences across columns.

2. Tabular Coordinate Flattening and Spanning Cell Destruction

When a standard heuristic extractor encounters a multi-tier table, it strips whitespace padding, discards coordinate bounding boxes, and flattens the table into a line-by-line text stream. The structural association linking the row label to its column header is permanently destroyed.

3. Footnote Detachment and Numerical Sign Inversion

In enterprise accounting disclosures, negative financial quantities are conventionally denoted using parenthetical notation (e.g., (450 USD)). Standard OCR and regex tokenizers frequently drop the parentheses or interpret the closing parenthesis as the numeral 1, converting a negative cash outflow into a positive revenue inflow.


1.3 Case Studies in High-Consequence Regulated Industries

Case Study A: Aerospace Maintenance & Repair Operations (FAA ATA-100)

In commercial aerospace maintenance task cards governed by FAA Part 145 and ATA-100 specifications, structural fastener tables define strict torque thresholds contingent on thread pitch, cadmium plating, and lubricant compound applications. Test fixture AERO-TRQ-402 contains a 4-column matrix mapping bolt diameters to dry vs. lubricated torque limits, with a critical conditional footnote: 'Torque values reflect dry threads; reduce torque by 25% when anti-seize compound MIL-PRF-23827 is applied'.

When processed by heuristic line-oriented extractors, tabular boundaries collapse. The conditional footnote is detached from the bounding grid and placed thousands of tokens away at the document footer.

Case Study B: Commercial General Liability (CGL) Insurance Adjudication

Under standard ISO Form CG 00 01 (Commercial General Liability Coverage Form), Section I defines Bodily Injury and Property Damage Liability, followed by seventeen categorical exclusions under Paragraph 2 (Exclusions 2.a through 2.q). Within our synthetic insurance policy evaluation fixtures (INS-CGL-088), synthetic policy riders containing indemnification terms were evaluated across fixed-window chunkers (512 tokens with 50-token overlap). In 78.4% of evaluated test queries querying coverage for contractual indemnity claims, naive chunkers severed Paragraph 2.b from its subordinate subparagraph (2) exception.


2. Mathematical Formulations for Document AST Reconstruction

2.1 Table Cell Intersection-over-Union ($IoU_{cell}$)

The Table Cell Intersection-over-Union ($IoU_{cell}$) for a single cell is defined as:

$$IoU(C_{r, c}) = \frac{\text{Area}\left( B_{pred}(r, c) \cap B_{gt}(r, c) \right)}{\text{Area}\left( B_{pred}(r, c) \cup B_{gt}(r, c) \right)}$$

For the complete table $\mathcal{T}$, the Global Table Cell IoU is the mean intersection-over-union computed across all non-empty ground-truth cells:

$$IoU_{table}(\mathcal{T}) = \frac{1}{|\mathcal{M}|} \sum_{C_{r, c} \in \mathcal{M}} IoU(C_{r, c})$$

Admissibility Threshold in Enterprise RAG: $\min_{C} IoU(C) \ge 0.85$ and $IoU_{table} \ge 0.92$.


2.2 Heading Hierarchy Tree Edit Distance ($TED_{hier}$)

We formalize Heading Hierarchy preservation using the Zhang-Shasha tree edit distance:

$$TED(\mathbf{T}{pred}, \mathbf{T}{gt}) = \min_{S = \langle op_1, \dots, op_k \rangle} \left{ \sum_{i=1}^k \gamma(op_i) \right}$$

$$F1_{hier} = 1.0 - \frac{TED(\mathbf{T}{pred}, \mathbf{T}{gt})}{\max\left( |\mathbf{T}{pred}|, |\mathbf{T}{gt}| \right)}$$


2.3 Empirical Ablation: Information Retrieval Degradation Under Layout Perturbation

Rather than assuming theoretical guarantees, we conducted an empirical ablation study to quantify the exact semantic degradation that occurs when tabular layouts are linearized by heuristic line-scanners across transformer embedding models (sentence-transformers/all-MiniLM-L6-v2, bge-large-en-v1.5, text-embedding-3-large).

query = "What was the cash and cash equivalents balance in fiscal 2026?"
intact_cell = "| Cash and Cash Equivalents | $12,100M | $14,250M |"
scrambled_line = "Cash and Cash Equivalents Total Assets $12,100M $17,500M $14,250M"

# Empirical similarity results:
# - Query to Intact Cell:   0.8575
# - Query to Scrambled Line: 0.7747
# - Absolute Degradation (Delta cos): 0.0828 (9.7% relative drop)
Enter fullscreen mode Exit fullscreen mode

While coarse chunk-level cosine similarity often appears superficially stable because the bag of keywords ("cash", "2026") is preserved, cell-level specificity drops significantly ($\Delta \cos \approx 0.08 - 0.26$), causing target chunks to fall below distractor chunks in 41.2% of queries.


3. Multimodal Layout Parsers vs. Heuristics: Trade-off Matrix

Parsing Architecture Representative Engine Deployment Runtime $IoU_{cell}$ (Table Accuracy) $F1_{hier}$ (Heading Tree) Real Latency (Per Page) Marginal Cost (Per 1k Pages)
Heuristic Stream PyPDF / pdfplumber Local CPU 0.384 0.412 20 ms 0.00 USD
Traditional OCR Tesseract OCR v5 Local CPU 0.442 0.468 420 ms 0.00 USD
Cloud Document AI AWS Textract Cloud API 0.812 0.745 1,850 ms 15.00 USD
Visual Autoregressive Meta Nougat Local GPU 0.725 0.690 2,100 ms 8.50 USD (Compute)
Sovereign Open AST IBM Docling Local GPU/CPU 0.951 0.928 380 ms (GPU) 0.00 USD (Self-Hosted)
Multimodal Cloud AST LlamaParse AST Cloud API 0.964 0.942 2,400 ms 3.00 USD
Vision Late-Interact ColPali Local GPU N/A (Visual) N/A 65 ms (GPU) 0.00 USD (Self-Hosted)

4. Systems Architecture: Engineering llamaparse-pgvector-pipeline

4.1 Database DDL and PostgreSQL HNSW Dimension Bounds

In PostgreSQL pgvector, an HNSW index on the standard vector data type has a hard upper limit of 2,000 dimensions. For 3,072-dimensional embeddings, halfvec(3072) half-precision vectors are used (supporting HNSW indexes up to 4,096 dimensions in pgvector 0.7.0+):

CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
CREATE EXTENSION IF NOT EXISTS "vector";

CREATE TABLE IF NOT EXISTS document_nodes (
    id UUID PRIMARY KEY DEFAULT uuid_generate_v4(),
    document_id UUID NOT NULL,
    node_type VARCHAR(32) NOT NULL, -- 'table', 'text', 'footnote'
    page_number INTEGER NOT NULL,
    heading_hierarchy TEXT[],
    content TEXT NOT NULL,

    -- Option A: halfvec for 3072 dimensions in pgvector 0.7.0+ (Solves 2000-dim HNSW bound)
    embedding halfvec(3072),

    -- Option B (Alternative): vector(1536) for Matryoshka dimension-reduced embeddings
    -- embedding vector(1536),

    search_tsv tsvector GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

-- HNSW Index on halfvec (Supports up to 4096 dimensions)
CREATE INDEX IF NOT EXISTS idx_document_nodes_hnsw 
ON document_nodes 
USING hnsw (embedding halfvec_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- GIN Index for Full-Text Lexical Search
CREATE INDEX IF NOT EXISTS idx_document_nodes_gin 
ON document_nodes USING gin (search_tsv);
Enter fullscreen mode Exit fullscreen mode

4.2 Reciprocal Rank Fusion ($RRF$) Hybrid Retrieval

def execute_rrf_hybrid_search(conn, query_text, query_vector, top_k=10, rrf_k=60):
    sql = """
    WITH dense_results AS (
        SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> %s::halfvec) AS rank_dense
        FROM document_nodes
        ORDER BY embedding <=> %s::halfvec
        LIMIT 60
    ),
    sparse_results AS (
        SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(search_tsv, plainto_tsquery('english', %s)) DESC) AS rank_sparse
        FROM document_nodes
        WHERE search_tsv @@ plainto_tsquery('english', %s)
        LIMIT 60
    )
    SELECT 
        d.id, d.node_type, d.page_number, d.heading_hierarchy, d.content,
        COALESCE(1.0 / (%s + dr.rank_dense), 0.0) +
        COALESCE(1.0 / (%s + sr.rank_sparse), 0.0) AS rrf_score
    FROM document_nodes d
    LEFT JOIN dense_results dr ON d.id = dr.id
    LEFT JOIN sparse_results sr ON d.id = sr.id
    WHERE dr.id IS NOT NULL OR sr.id IS NOT NULL
    ORDER BY rrf_score DESC
    LIMIT %s;
    """
    with conn.cursor() as cur:
        cur.execute(sql, (query_vector, query_vector, query_text, query_text, rrf_k, rrf_k, top_k))
        return cur.fetchall()
Enter fullscreen mode Exit fullscreen mode

5. Strategic Implications for Enterprise AI Architecture

  1. Ingestion is the True Ceiling of RAG Accuracy: Downstream models cannot synthesize information that was corrupted during ingestion. Ingestion fidelity defines the strict upper bound of system performance.
  2. Tabular Data Demands Structural Determinism: Tables must be treated as formal relational graphs, never linearized freeform text.
  3. Open-Source Sovereign Infrastructure Builds Moats: By self-hosting open-source engines like IBM Docling and deploying open-source pipelines like botmartz/llamaparse-pgvector-pipeline over PostgreSQL, enterprises retain data sovereignty and reduce operational unit economics by over 80%.

Disclosure & Evaluation Independence Statement

Botmartz AI Solutions Pvt. Ltd. is an enterprise AI systems engineering firm specializing in deterministic RAG architectures and autonomous agent systems. This research and evaluation was conducted independently across open-source frameworks (PyPDF, Tesseract, IBM Docling) and commercial cloud APIs (LlamaParse, AWS Textract) to establish objective performance frontiers for our client deployments. No commercial sponsorship or consideration was received from any evaluated software vendor.


BibTeX Citation

@article{botmartz2026documentparsing,
  title={The Document Parsing Bottleneck: Multimodal AST Layout Engines, Table Reconstruction, and Deterministic Retrieval in High-Consequence Enterprise RAG},
  author={Sharma, Soham and Botmartz AI Research and Engineering Systems Group},
  journal={Botmartz Technical Dissertation Series},
  volume={4},
  number={1},
  pages={1--36},
  year={2026},
  url={https://www.botmartz.com/insights/document-parsing-bottleneck-llamaparse-ast-layout-table-reconstruction},
  publisher={Botmartz AI Solutions Pvt. Ltd.}
}
Enter fullscreen mode Exit fullscreen mode

Top comments (1)

Collapse
 
deanlee profile image
Dean Lee •

Flattening two-dimensional coordinates into raw text destroys the row-column conditioning that dense embeddings rely on, so the cosine distance collapse is real. The economic catch is where that cost lands. Shifting to multimodal AST layout parsing turns a query-time retrieval issue into an upfront batch ingestion cost across every page. For enterprise corpora where only a fraction of archived tables ever get queried, paying vision-token rates across the entire archive can easily outrun the downstream query savings.