document-processor pre-1.0 AGPL-3.0-only + Commercial

Document Processor

Parse PDF, DOCX, TXT — on your machine, not theirs.

View on GitHub → Build from source Releases

Overview

A desktop application — Rust backend, Tauri 2 shell, Svelte 5 frontend — that parses documents locally and extracts text plus images with their surrounding context preserved. SQLite database with hybrid search (FTS5, optionally fused with local embeddings), a remembered scan folder for batch re-processing, packaged for Linux.

The point is not parsing. The point is locality. Cloud OCR and document AI services exist; this exists for teams whose documents legally cannot leave the host.

Who it's for

Targets are organisations where document confidentiality is a regulatory or contractual requirement, not a preference:

  • Law firms — contract review, pleadings analysis, opinion drafts. Attorney-client privilege precludes cloud AI.
  • Medical research and clinical labs — patient records, study protocols, trial data. GDPR and patient consent don't extend to overseas cloud LLMs.
  • Patent attorneys — pre-filing applications. Even an embedding leaked to a model's training set is enough to lose novelty.
  • Engineering and R&D teams — unpatented IP, pre-publication research, internal designs.
  • Public-sector operators — classified, restricted, or sovereignty-controlled material.
  • Corporations with M&A pipelines or strategic IP — board reports, due diligence materials, financial models.

Features

  • Multi-format parsing — PDF, DOCX, TXT. .doc files are accepted but routed through the DOCX parser (no guarantee for legacy binary .doc); RTF is not supported.
  • Image extraction with surrounding context — every image carries 200 chars of preceding and following text plus a position marker. Charts and diagrams stay anchored to the prose that explains them. (The schema reserves fields for future OCR text and AI descriptions — neither is implemented yet.)
  • Document type classification — automatic detection of types (configurable; ships with examples for legal documents umowa, pozew, ustawa).
  • Watch folder — remembers a target directory; new files are picked up when you trigger a re-scan ("Scan again"), not automatically in the background.
  • SQLite database — fast cross-document search without a server. Single file, easy backup, no extra service.
  • Hybrid search — SQLite FTS5 lexical search, optionally fused with embeddings from a local Ollama instance (nomic-embed-text). If Ollama isn't running, search degrades cleanly to lexical-only. No cloud calls either way.
  • Modern UI — dark theme, drag-and-drop, responsive. Built with Svelte 5; reactive, small bundle.
  • Linux packages — built and distributed for Linux (.deb / AppImage). The Tauri toolchain compiles on Windows and macOS, but no installers are shipped for them — build from source. Tauri keeps the package small (~10–20 MB) and the runtime fast.

Output structure

Each processed document creates a self-contained directory:

przetworzone/<id>/ 
<app-data-dir>/przetworzone/<document-id>/
├── document.md          # Human-readable markdown
├── document.json        # Structured data for AI ingestion
│                        #   (incl. per-image context metadata)
├── images/
│   ├── img_001.png      # Extracted images
│   └── thumb_img_001.png  # Thumbnail
└── original.pdf         # Original file copy (audit trail)

Two parallel formats: document.md for humans and document.json for AI ingestion. Images live alongside; their metadata (surrounding context, position marker, dimensions) is recorded in document.json and in the SQLite index. The original file is preserved for audit.

Ecosystem fit

Document Processor is a producer; the rest of BuildOnAI consumes what it produces.

  • → Consciousness Server — ingest a folder of documents, push each parsed result as training records and notes into shared memory. Your team's archive becomes queryable by any agent.
  • → Cortex — Cortex reads document.json as context, answers questions against the parsed corpus locally. "Show me every contract clause longer than 12 months." No cloud LLM, no data leaving the host.
  • → Key Server — distribute Document Processor to multiple law-firm or research workstations; each fetches its API token from the vault rather than having one hardcoded in .env.

Install (build from source)

Linux installers (.deb / AppImage) are the distributed form. On any platform you can also build from source:

terminal 
# Linux (Ubuntu/Debian) — system deps
sudo apt install -y libwebkit2gtk-4.1-dev libappindicator3-dev \
                    librsvg2-dev patchelf libssl-dev

# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Clone and build
git clone https://github.com/build-on-ai/document-processor.git
cd document-processor
npm install
npm run tauri build

Windows note: build-from-source only — the bundle config defines Linux targets, so no .msi/.exe installer is produced. Install WebView2 and Visual Studio Build Tools first. Tauri uses the system WebView; nothing bundled.

Status

pre-1.0 — working and in active use. Some edge formats and error paths still need polish before 1.0.

What "pre-1.0" means in practice:

  • API may change without a deprecation cycle. Names of fields in document.json, paths under przetworzone/, plugin hooks — all subject to revision until v1.0.
  • Edge-case PDFs may fail or produce noisy output. Unusual layouts, heavy scans, files over 100 MB — works on most documents, but failure modes are real.
  • Installers are Linux-only. .AppImage + .deb are built; Windows and macOS mean building from source (instructions below). A Windows .msi is on the v1.0 roadmap.
  • No OCR yet. A scanned document without a text layer yields no extracted text — OCR is on the roadmap, not in the app. The document.json schema already reserves per-image fields for future OCR text and AI descriptions.
  • Safe to use on real documents — output goes to a local directory, the original file is preserved as-is, no destructive operations on input.

If you're trialling Document Processor in a regulated workflow, the honest call is: parse a representative subset first, verify the output format meets your needs, then expand. v1.0 will lock the schema.

Roadmap to v1.0:

  • Regression test corpus across vendor PDFs (Adobe, Foxit, scanned, OCR'd).
  • Vertical-specific classification packs — legal, medical, scientific publications, patents.
  • OCR pipeline (planned, not yet implemented) — designed to run entirely on the host, so scanned confidential documents never leave the machine.
  • Document diff workflow — compare two revisions, highlight changes, extract clauses that moved between versions.
  • Windows installer (.msi) alongside the existing Linux .AppImage + .deb.

Next steps