Design a specialized RAG pipeline for extracting, indexing, and querying information from large PDF document collections.
## CONTEXT PDFs remain the dominant format for critical business documents — contracts, financial reports, research papers, regulatory filings, technical specifications — yet they are notoriously difficult for AI systems to process accurately. Tables lose their structure, multi-column layouts merge incorrectly, headers and footers contaminate text extraction, and scanned documents require OCR that introduces errors. A specialized PDF RAG pipeline that properly handles these complexities enables organizations to unlock insights from document collections that were previously searchable only through painful manual review. ## ROLE You are a document intelligence engineer who has built PDF processing pipelines for Fortune 500 companies, processing over 2 million PDFs annually including financial filings, legal contracts, medical records, and engineering specifications. Your pipelines achieve 97% text extraction accuracy on native PDFs and 93% on scanned documents, and your table parsing system correctly structures 95% of tables on first pass. You specialize in the hardest PDF challenges: multi-column layouts, nested tables, embedded charts, mixed-orientation pages, and document collections with inconsistent formatting. ## RESPONSE GUIDELINES - Design for the hardest PDFs first — a pipeline optimized for clean PDFs fails on real-world documents - Include specific extraction accuracy benchmarks and validation methods for each document element type - Address table extraction as a first-class capability — tables contain some of the most valuable data in PDFs - Include fallback strategies for when automated extraction fails: human review queues, confidence-based routing - Do NOT assume all PDFs are text-based — include robust OCR capabilities for scanned documents - Do NOT treat PDF extraction as a solved problem — include continuous quality monitoring and improvement loops ## TASK CRITERIA 1. **PDF Classification & Routing** — Design the initial classification step that identifies PDF type (native text, scanned, hybrid), document category, page count, and layout complexity. Route each PDF to the appropriate extraction pipeline based on its characteristics. 2. **Text Extraction Engine** — Specify the text extraction approach for native PDFs: library selection (PyMuPDF, pdfplumber, or LlamaParse), layout analysis to handle multi-column text, header/footer detection and removal, reading order determination, and character encoding handling for international documents. 3. **OCR Pipeline for Scanned Documents** — Design the OCR system for scanned PDFs and images: preprocessing steps (deskewing, denoising, contrast enhancement), OCR engine configuration, language detection, and post-OCR text correction using AI-based spell-checking and formatting. 4. **Table Detection & Extraction** — Build the table processing system: table boundary detection, row/column structure recognition, merged cell handling, header row identification, and conversion to structured formats (CSV, JSON, dataframes). Include accuracy validation and fallback to manual extraction for complex tables. 5. **Image & Chart Processing** — Design the visual element handler: detect figures, charts, and diagrams, generate descriptive text using vision AI models, associate images with their captions and surrounding context, and extract data from simple charts and graphs. 6. **Intelligent Chunking** — Specify the PDF-aware chunking strategy: section-based chunking that follows document hierarchy, table preservation (never split mid-table), figure-context linking, cross-reference handling for documents with citations and footnotes, and metadata attachment (page number, section path, document title) for every chunk. 7. **Metadata Extraction** — Design the metadata extraction system: document title, authors, dates, version numbers, document classification, table of contents parsing, and domain-specific metadata relevant to [INSERT PDF DOMAIN] (e.g., case numbers for legal, patient IDs for medical, part numbers for engineering). 8. **Multi-Document Query Processing** — Build the query system that handles different question types: textual questions (semantic search), tabular questions (structured lookup with computation), cross-document questions (aggregate information from multiple PDFs), and numerical questions (extract and calculate from parsed data). 9. **Answer Generation with Page Citations** — Design the generation component that produces answers with exact source references: page numbers, section headers, table identifiers, and the ability to highlight the source passage in the original PDF for verification. 10. **Quality Monitoring & Continuous Improvement** — Specify the quality assurance system: extraction accuracy tracking per document type, automated comparison of extracted text against source, user feedback collection on answer accuracy, and a reprocessing pipeline for documents flagged with quality issues. ## INFORMATION ABOUT ME - My PDF domain: [INSERT DOMAIN — e.g., legal contracts, financial statements, academic papers, engineering specs] - My PDF types: [INSERT TYPES — e.g., native text PDFs, scanned documents, mix of both] - My OCR tool preference: [INSERT OCR TOOL — e.g., Tesseract, Amazon Textract, Google Document AI, Azure Form Recognizer] - My typical document complexity: [INSERT COMPLEXITY — e.g., simple single-column, complex multi-column with tables, heavily formatted] - My collection size: [INSERT SIZE — e.g., 500 PDFs, 50,000 pages total] - My primary use case: [INSERT USE CASE — e.g., contract analysis, financial data extraction, research review] ## RESPONSE FORMAT - Start with a pipeline architecture diagram showing the flow from PDF input to query-ready index - Use labeled sections for each processing stage with library recommendations and configuration details - Include a document complexity classification matrix - Provide extraction accuracy benchmark targets per element type (text, tables, images) - Include a sample processing log showing metrics for a batch of processed documents - End with a technology stack comparison table and implementation cost estimate
Or press ⌘C to copy
Replace these placeholders with your own content before using the prompt.
[INSERT PDF DOMAIN]