Add Hybrid and Markdown Chunking Strategies with Automatic Fallback

[dpage]

This PR introduces two new structure-aware chunking strategies inspired by Docling's hybrid chunking approach, significantly improving RAG retrieval accuracy for markdown documents.

Key features:

hybrid strategy: Full two-pass refinement that splits oversized chunks and merges undersized consecutive chunks sharing the same heading context

markdown strategy: Simpler structure-aware chunking without refinement passes

Automatic fallback: Both strategies detect plain text and fall back to token_based chunking to avoid unnecessary overhead

Why This Matters

Traditional token-based chunking destroys semantic context by splitting text at arbitrary boundaries. The new strategies:

Preserve document structure - Headings, code blocks, tables, and lists stay intact
Maintain heading hierarchy - Each chunk includes context like [Context: # Chapter 1 > ## Section 1.1]
Optimize chunk sizes - The hybrid strategy merges small chunks and splits large ones intelligently

Summary by CodeRabbit

• New Features

  • Hybrid chunking strategy for structured documents with heading context preservation and two-pass refinement to improve retrieval accuracy
  • Markdown chunking strategy for markdown content offering structure-aware parsing without refinement passes
  • Automatic fallback detection that uses appropriate strategy based on content type

• Documentation

  • Added configuration guide for chunking strategies with use case recommendations and examples

• Tests

  • Added comprehensive regression test suite for new chunking strategies

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[Copilot]

Pull request overview

This PR introduces two new structure-aware chunking strategies (hybrid and markdown) for markdown documents, with automatic fallback to token-based chunking for plain text. The implementation is inspired by Docling's hybrid chunking approach and aims to improve RAG retrieval accuracy by preserving document structure and heading context.

Changes:

• Added hybrid strategy with two-pass refinement (splits oversized chunks, merges undersized chunks with same heading context)
• Added markdown strategy for simpler structure-aware chunking without refinement
• Implemented automatic fallback detection that switches to token-based chunking for non-markdown content

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/hybrid_chunking.c	Implements hybrid and markdown chunking strategies with markdown parsing, structure detection, and refinement passes
src/pgedge_vectorizer.h	Adds enum types and struct definitions for markdown elements and hybrid chunks
src/chunking.c	Registers new strategies in parse_chunk_strategy and routes them in chunk_text
test/sql/hybrid_chunking.sql	Comprehensive test suite covering markdown parsing, heading context, fallback behavior, and edge cases
test/expected/hybrid_chunking.out	Expected test output for the new chunking strategies
docs/configuration.md	Documents new strategies, automatic fallback, and usage examples
docs/changelog.md	Adds changelog entries for the new features
Makefile	Adds hybrid_chunking.c to build and test targets

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

This PR introduces hybrid and markdown-aware chunking strategies for text content. A new module implements structure-aware parsing of markdown documents with two-pass refinement (splitting oversized, merging undersized chunks) and automatic fallback to token-based chunking for plain text. Documentation and comprehensive SQL tests are included.

Changes

Cohort / File(s)	Summary
Build Configuration Makefile	Added src/hybrid_chunking.o to object list and hybrid_chunking to regression test targets
Documentation Updates docs/changelog.md, docs/configuration.md	Added Unreleased changelog entries describing hybrid and markdown strategies; expanded configuration guide with strategy definitions, use-case mapping, and example usage blocks
Core Chunking Module src/chunking.c	Exported chunk_by_tokens (removed static); added support for CHUNK_STRATEGY_HYBRID and CHUNK_STRATEGY_MARKDOWN in strategy parser and dispatcher
Hybrid Chunking Implementation src/hybrid_chunking.c	New module implementing markdown detection, structure-aware parsing (elements: headings, code blocks, lists, blockquotes, tables), hierarchical heading context tracking, and two-pass chunk refinement (split oversized via token boundaries; merge undersized with identical context)
Public API Extensions src/pgedge_vectorizer.h	Added CHUNK_STRATEGY_HYBRID enum value; new enums: MarkdownElementType; new structs: MarkdownElement, HybridChunk; new function declarations: chunk_by_tokens, chunk_hybrid, chunk_markdown, parse_markdown_structure, free_markdown_elements
Test Suite test/sql/hybrid_chunking.sql	New SQL test file with 20 tests covering hybrid/markdown strategies, heading context preservation, code block integrity, plain text fallback, nested structures, and token-based behavior validation

Sequence Diagram

sequenceDiagram
    participant Client as Application
    participant Dispatcher as chunk_text<br/>(chunking.c)
    participant Detector as is_likely_markdown<br/>(hybrid_chunking.c)
    participant Parser as parse_markdown_structure<br/>(hybrid_chunking.c)
    participant Refiner as Refinement Passes<br/>(hybrid_chunking.c)
    participant Fallback as chunk_by_tokens<br/>(token-based)

    Client->>Dispatcher: chunk_text(content, strategy)
    alt Strategy == HYBRID
        Dispatcher->>Detector: Check if markdown
        alt Is Markdown
            Detector-->>Dispatcher: true
            Dispatcher->>Parser: Parse markdown elements
            Parser-->>Dispatcher: MarkdownElement list
            Dispatcher->>Refiner: Pass 1: Split oversized
            Refiner->>Refiner: Pass 2: Merge undersized
            Refiner-->>Dispatcher: Refined HybridChunk array
        else Not Markdown
            Detector-->>Dispatcher: false
            Dispatcher->>Fallback: Fallback to token-based
            Fallback-->>Dispatcher: Text array
        end
    else Strategy == MARKDOWN
        Dispatcher->>Detector: Check if markdown
        alt Is Markdown
            Dispatcher->>Parser: Parse markdown structure
            Parser-->>Dispatcher: Convert to chunks (no refinement)
        else Not Markdown
            Dispatcher->>Fallback: Fallback to token-based
        end
    end
    Dispatcher-->>Client: ArrayType (text array)

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Poem

🐰 Hop through structures, chunk by chunk,
Headings preserved, no context sunk,
Markdown wisdom in two-pass refine,
Plain text falls back, all chunks align! ✨

✨ Finishing touches

• 📝 Generate docstrings

---

📜 Recent review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 373e3ab and ce4b90a.

⛔ Files ignored due to path filters (1)

• test/expected/hybrid_chunking.out is excluded by !**/*.out

📒 Files selected for processing (7)

• Makefile
• docs/changelog.md
• docs/configuration.md
• src/chunking.c
• src/hybrid_chunking.c
• src/pgedge_vectorizer.h
• test/sql/hybrid_chunking.sql

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}