Document Layout Analysis: How OCR Understands Pages

Document Layout Analysis: How OCR Understands Pages

Optical character recognition converts images of text into machine-readable characters. But before any character can be recognized, the system must answer a more fundamental question: where is the text? A scanned page may contain headings, body paragraphs, tables, figures, captions, footnotes, page numbers, and marginal notes — each requiring different processing. Document layout analysis is the critical step that identifies these regions and determines how they relate to one another.

This article examines how layout analysis has evolved from rule-based heuristics to modern deep learning approaches, with practical guidance for integrating layout detection into OCR pipelines.

The Layout Analysis Problem

Document layout analysis takes a document image as input and produces a set of labelled bounding boxes — each enclosing a region classified by type (text block, table, figure, header, list item, caption, footer, etc.). The output drives two downstream decisions: what processing each region receives, and in what order regions are read.

Reading order matters because it determines meaning. Consider a two-column academic paper: reading straight across the page produces nonsense, while reading column-by-column produces coherent text. Similarly, a table must be processed as structured data, not as lines of text. And a figure caption must be associated with its figure, not concatenated with the adjacent paragraph.

The challenge scales with document diversity. Modern printed documents follow relatively predictable grid layouts, but historical manuscripts may have irregular text blocks, marginal annotations, and decorative elements that confound automated analysis.

From Rules to Neural Networks

Early layout analysis systems relied on hand-crafted rules. Projection profile methods analyse horizontal and vertical pixel density histograms to identify text lines and column boundaries. The recursive X-Y cut algorithm recursively splits pages along the most prominent horizontal or vertical whitespace gaps, producing a tree of rectangular regions.

These classical approaches work well on clean, single-column documents with consistent formatting. They fail on complex layouts — multi-column pages with spanning headers, documents mixing text and tables, or pages where visual boundaries between regions are subtle. A two-column page with a full-width abstract, for instance, confounds simple projection methods because the whitespace pattern changes mid-page.

The shift to deep learning reframed layout analysis as an object detection problem: given a document image, detect and classify all regions. This approach handles arbitrary layouts without hand-crafted rules, learning spatial patterns directly from annotated training data.

Modern Approaches

Layout as Object Detection

The first wave of deep learning layout analysis adapted standard object detection architectures — Faster R-CNN, Mask R-CNN, and later DETR — to documents. These models generate region proposals, classify each proposal (text, table, figure, list, etc.), and refine bounding box coordinates.

The breakthrough that made this practical was the creation of large-scale annotated datasets. PubLayNet, created by automatically matching the XML structure of over 360,000 scientific articles from PubMed Central to their rendered PDF pages, provided the first dataset large enough to train deep layout detection models reliably.

PubLayNet defines five region classes: text, title, list, table, and figure. Models trained on PubLayNet achieve high accuracy on scientific papers, but the dataset's narrow domain (biomedical literature only) limits generalization to other document types.

Multimodal Document Understanding

A key insight driving recent progress is that layout analysis benefits from combining multiple information sources: visual appearance, textual content, and spatial position. A region's identity depends not just on how it looks, but on what text it contains and where it sits on the page.

LayoutLMv3, developed by Microsoft, unifies all three modalities in a single transformer architecture. Pre-trained on 11 million document pages with masked language modelling, masked image modelling, and word-patch alignment objectives, LayoutLMv3 learns rich representations that transfer effectively to downstream tasks including layout analysis, form understanding, and document question answering.

The multimodal approach is particularly valuable for distinguishing visually similar regions. A text block and a table cell may look identical in isolation, but their spatial context (surrounded by grid lines vs. embedded in flowing text) and textual content (numeric data vs. prose) disambiguate them.

End-to-End Document Understanding

An alternative paradigm bypasses explicit layout analysis and OCR entirely. Donut (Document Understanding Transformer) takes a document image as input and directly generates structured output — extracting key-value pairs, classification labels, or parsed content without a separate OCR stage.

Donut avoids the error propagation that occurs when OCR mistakes feed into downstream extraction. However, end-to-end models currently trade specialised accuracy for pipeline simplicity — dedicated layout analysis followed by dedicated OCR still outperforms end-to-end approaches on many benchmarks, particularly for complex table structures and handwritten text.

from layoutparser import Detectron2LayoutModel
from PIL import Image
import cv2

def detect_document_regions(image_path):
    """
    Detect and classify regions in a document image
    using a pre-trained layout detection model.
    """
    # Load pre-trained model (PubLayNet-trained Mask R-CNN)
    model = Detectron2LayoutModel(
        config_path="lp://PubLayNet/mask_rcnn_X_101_32x8d_FPN_3x/config",
        label_map={0: "Text", 1: "Title", 2: "List", 3: "Table", 4: "Figure"},
        extra_config=["MODEL.ROI_HEADS.SCORE_THRESH_TEST", 0.5]
    )

    # Load document image
    image = cv2.imread(image_path)
    image_rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

    # Detect layout regions
    layout = model.detect(image_rgb)

    # Sort regions by reading order (top-to-bottom, left-to-right)
    regions = []
    for block in layout:
        regions.append({
            "type": block.type,
            "confidence": block.score,
            "bbox": {
                "x1": int(block.block.x_1),
                "y1": int(block.block.y_1),
                "x2": int(block.block.x_2),
                "y2": int(block.block.y_2)
            }
        })

    # Sort by vertical position, then horizontal
    regions.sort(key=lambda r: (r["bbox"]["y1"], r["bbox"]["x1"]))

    return regions

Training Data: The Dataset Challenge

The quality and diversity of training data fundamentally determines layout analysis performance. Two datasets dominate the field, with markedly different characteristics.

PubLayNet (2019) contains over 360,000 automatically annotated document pages from PubMed Central scientific articles. Its scale enabled the first generation of deep learning layout models, but its homogeneous source — all biomedical research papers with similar formatting — limits the diversity of layouts represented.

DocLayNet (2022) addresses this limitation with 80,863 manually annotated pages drawn from six document categories: financial reports, scientific articles, patents, government tenders, legal texts, and technical manuals. It defines 11 region classes (compared to PubLayNet's 5), capturing finer distinctions like section headers, page headers, footnotes, and formulas.

The generalization gap between these datasets reveals a central challenge. Models trained exclusively on PubLayNet achieve strong results on scientific papers but performance typically drops by 10-20 mAP points when evaluated on the diverse documents in DocLayNet. Models trained on DocLayNet's varied layouts generalize substantially better to unseen document types, even when evaluated on scientific papers.

Practical Considerations

Integrating layout analysis into a production OCR pipeline requires balancing accuracy, speed, and complexity.

Model selection depends primarily on document type. For standardised documents with predictable layouts (invoices, forms), lightweight models or even template-based approaches may suffice. For diverse or complex documents, transformer-based models like LayoutLMv3 provide the strongest generalisation. For historical documents with non-standard layouts, domain-specific fine-tuning is typically necessary.

Speed versus accuracy presents a practical trade-off. Object detection models (Mask R-CNN, DETR) run in hundreds of milliseconds per page on GPU hardware. Multimodal transformers are slower but more accurate on ambiguous layouts. For batch processing thousands of documents, the pipeline design — whether to run layout analysis on every page or only on pages that fail simple heuristic checks — significantly affects throughput.

Preprocessing quality directly impacts layout analysis. Skewed scans, uneven lighting, and low resolution all degrade region detection accuracy. Deskewing and binarisation should precede layout analysis in the pipeline.

import cv2
import numpy as np
from typing import List, Dict

class LayoutAnalysisPipeline:
    """
    Complete pipeline: preprocess → detect regions → determine reading order.
    """

    def __init__(self, model, confidence_threshold=0.5):
        self.model = model
        self.confidence_threshold = confidence_threshold

    def analyse_page(self, image_path: str) -> List[Dict]:
        """
        Analyse a single document page.

        Returns list of regions sorted in reading order,
        each with type, confidence, and bounding box.
        """
        # Step 1: Preprocess
        image = self._preprocess(image_path)

        # Step 2: Detect regions
        raw_regions = self.model.detect(image)

        # Step 3: Filter by confidence
        regions = [
            r for r in raw_regions
            if r.score >= self.confidence_threshold
        ]

        # Step 4: Determine reading order
        ordered = self._determine_reading_order(regions, image.shape)

        # Step 5: Classify processing strategy per region
        for region in ordered:
            region["strategy"] = self._select_strategy(region)

        return ordered

    def _preprocess(self, image_path: str) -> np.ndarray:
        """Deskew and normalise document image."""
        img = cv2.imread(image_path)
        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

        # Deskew using minimum area rectangle
        coords = np.column_stack(np.where(gray < 200))
        if len(coords) > 100:
            angle = cv2.minAreaRect(coords)[-1]
            if angle < -45:
                angle = 90 + angle
            if abs(angle) > 0.5:
                h, w = gray.shape
                center = (w // 2, h // 2)
                M = cv2.getRotationMatrix2D(center, angle, 1.0)
                img = cv2.warpAffine(
                    img, M, (w, h),
                    borderMode=cv2.BORDER_REPLICATE
                )

        return cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

    def _determine_reading_order(self, regions, image_shape):
        """
        Determine reading order using column detection.

        Groups regions into columns based on horizontal overlap,
        then sorts top-to-bottom within each column,
        columns ordered left-to-right.
        """
        if not regions:
            return []

        page_width = image_shape[1]
        midpoint = page_width / 2

        # Detect if multi-column layout
        left_regions = []
        right_regions = []
        full_width = []

        for r in regions:
            x_center = (r.block.x_1 + r.block.x_2) / 2
            width_ratio = (r.block.x_2 - r.block.x_1) / page_width

            if width_ratio > 0.6:
                full_width.append(r)
            elif x_center < midpoint:
                left_regions.append(r)
            else:
                right_regions.append(r)

        # Sort each group by vertical position
        sort_key = lambda r: r.block.y_1
        full_width.sort(key=sort_key)
        left_regions.sort(key=sort_key)
        right_regions.sort(key=sort_key)

        # Interleave: full-width items at their vertical position
        ordered = []
        for r in full_width + left_regions + right_regions:
            ordered.append({
                "type": r.type,
                "confidence": r.score,
                "bbox": {
                    "x1": int(r.block.x_1),
                    "y1": int(r.block.y_1),
                    "x2": int(r.block.x_2),
                    "y2": int(r.block.y_2)
                }
            })

        ordered.sort(key=lambda r: (r["bbox"]["y1"], r["bbox"]["x1"]))
        return ordered

    def _select_strategy(self, region: Dict) -> str:
        """Select OCR strategy based on region type."""
        strategies = {
            "Text": "standard_ocr",
            "Title": "standard_ocr",
            "List": "standard_ocr",
            "Table": "table_extraction",
            "Figure": "skip_ocr",
            "Caption": "standard_ocr",
            "Formula": "math_recognition",
        }
        return strategies.get(region["type"], "standard_ocr")

Open Research Directions

Despite rapid progress, several challenges remain active areas of investigation.

Cross-domain generalisation is the central open problem. Models trained on one document domain consistently underperform when applied to another. Recent work on domain adaptation and self-supervised pre-training (such as DiT, the Document Image Transformer) reduces this gap by learning general document representations from unlabelled data, but significant performance differences persist across domains.

Graph-based structure analysis represents a promising departure from bounding-box detection. Rather than treating document regions as independent objects, graph-based methods model the relationships between regions — a caption connected to its figure, a footnote marker linked to its footnote text. Research presented at ICLR 2025 demonstrates that graph neural networks can capture these structural dependencies more naturally than object detection frameworks.

LLM-augmented understanding is an emerging direction. DocLayLLM (CVPR 2025) extends large language models with document layout awareness, enabling complex reasoning about document structure. These models can answer questions like "what is the total in the bottom-right table?" by jointly understanding layout and content.

Reducing annotation requirements remains practically important. Creating the ground-truth annotations that supervised models require is expensive — DocLayNet's 80,000 pages required substantial manual annotation effort. Self-supervised and semi-supervised approaches that learn layout patterns from unlabelled documents could make high-quality layout analysis accessible for domains where annotated data is scarce.

Conclusion

Document layout analysis is the critical bridge between raw document images and structured text extraction. Without understanding page structure, OCR systems cannot determine reading order, distinguish text from tables, or associate captions with figures.

Key takeaways:

• Layout analysis precedes OCR — it determines where text is and how regions relate, enabling correct reading order and appropriate processing for each region type
• Multimodal models outperform vision-only approaches — combining visual appearance, textual content, and spatial position (as in LayoutLMv3) provides the strongest generalisation
• Dataset diversity matters more than size — DocLayNet's 80K diverse pages produce more robust models than PubLayNet's 360K homogeneous scientific papers
• End-to-end models (Donut) simplify pipelines but dedicated layout analysis followed by specialised OCR still achieves higher accuracy on complex documents
• Start with pre-trained models and fine-tune on your specific document type for the best practical results

References