TAB 01 / LIVE PIPELINE COMPARISON

Same model, same question, two context pipelines.

Both paths ask the exact same local model the same question - Path A feeds raw text extracted from the PDF folder, while Path B uses an index to send only the relevant Markdown section and table.

1. Pick a scenario

5 PRESET QUESTIONS - GROUND-TRUTH FACTS PER QUESTION

Scenario / custom question

Model (used for both paths)

Show question text / write a custom question

No measured result

2. DELTA SCOREBOARD

Path B relative to Path A

AWAITING RESULTS

Baseline Benchmark Range (gemma4:26b - 5 preset questions):79.8% Fewer Tokens (57.0% - 93.9%)5.57x Faster First Token (2.49x - 17.97x)+25.0 pts Fact Coverage (+11.1 to +57.1)

3. Watch both pipelines

LIVE TELEMETRY - 100 MS STOPWATCH - FACT VERIFICATION

TAB 02 / DOCUMENT SANITIZER

What the sanitizer removes, and what it adds.

The sanitizer strips extraction noise (TOC dot-leaders, page numbers, repeated headers and footers, disclaimers, column-scrambled table fragments) and adds structure the router can use. Compare any source PDF with its compiled vault file below.

HOW PATH B PICKS CONTEXT

A two-stage lexical router. No embeddings, no vector database, no GPU.

Path B never sends the whole library. A small CPU-only lookup picks the one or two documents that match the question, then cuts out only the sections that answer it. Same question, same vault, same slice, every time.

IN
Question

Lower-cased; stop words dropped.
baseline - traffic - monitoring - blocking - rules
1
Pick the documents
- Term profile: rare words count more; filename ×3, YAML frontmatter and headings ×2.
- INDEX.md routes: +8 per word shared with a curated route, e.g. “Conduit C5-01 SL-T” → D4#t3.
- Best section: the score of the file’s strongest ## clause.
→ top 1 - 3 of 8 files
2
Cut the sections
- Pinned tables: #t1 - #t6 anchors named by INDEX.md.
- Scored clauses: every ##/### section, scored on exact words, word stems, word pairs and heading hits.
- Header: doc ID, authority, Purdue levels.
→ ≤ 7,000 chars per file, top 2 files
OUT
Routed slice

Header + tables + top clauses.
~2,874 median prompt tokens - 5 preset questions

Fails loud. If no vault file shares a single word with the question, the router refuses instead of guessing. Routing time: ~90 ms on CPU.

Scoring weights

Signal	Pick documents (Stage 1)	Cut sections (Stage 2)
Exact query word in section	+2.5 per hit, max 3 hits	+2.0 per hit, max 3 hits
Stem variant (e.g. `block` → `blocking`)	+1.5	+1.5
Adjacent query words found together (“blocking rules”)	+6.0	+6.0
Query word in the heading	+4.0	+3.0
Shared word with an INDEX.md route	+8.0 per word	anchor pinned
Term profile (TF-IDF-style; filename ×3, frontmatter/headings ×2)	added	-
Cut-offs	≥ 45% of top score - ≤ 3 docs - 12k-token budget	section score ≥ 4.0 - ≤ 7,000 chars/doc

Same clause, two paths

“What baseline traffic monitoring is required before enabling blocking rules?”

Pages Path A reads (1 - 3)Answer (p. 15)

PATH A

Keeps the first 5,500 characters of each PDF so all eight fit in a 16k window. That stops on page 3. The clause is on page 15, so the model never sees it.

PATH B

Stage 1 picks nist_sp_800_82_r3. Stage 2 cuts the section containing:

Baseline traffic for at least thirty days before enabling blocking rules.

NIST SP 800-82r3, Section 11

All 8 PDFs as text - 128 pages

72,918 tok - 100%

Compiled vault at rest - 8 files

35,996 tok - 49.4%

Path A prompt cap (to fit 16k)

~14,200 tok - 19.5%

Path B prompt (routed slice)

~2,874 tok - 3.9%

Two short leaflets grow slightly at rest (+8.1%, +11.5%) because structure costs more than the noise it removes. Long manuals shrink up to 75%.

ARCHITECTURAL ECONOMICS

Raw PDFs vs. Compiled Vault + Deterministic Router

Raw PDFs work when the library fits. On a large-context cloud model, Path A can send all ~73k tokens and usually gets it right, but it pays for the whole library on every question. On a 16k local model (this demo), the library doesn't fit: Path A has to truncate to ~15% of the text, and anything past page ~3 is invisible. Path B costs real engineering up front and pays it back in volume.

PATH A

Send the PDFs

Zero build. Works when the library fits.

Nothing to build or maintain. A new PDF works the moment it lands.
The model reads original wording; nothing is curated away.
Paraphrased questions are fine. The model reads, it doesn't keyword-match.
Every question pays for the whole library: ~73k tokens here, roughly 25× a routed slice.
Prefill grows with every page: about 25 s to the first token on this hardware.
When the library outgrows the window you must truncate, and you can't see what was cut.

PATH B

Compile + Route

Engineering up front. Cheap, fast and inspectable per question.

79.8% fewer prompt tokens (median; range 57 - 94%).
5.6× faster to the first token (median).
A clause on page 15 is found as reliably as one on page 1.
Routing is deterministic and shown in the UI. No embeddings, no vector DB, ~90 ms on CPU.
Build: sanitizer, table reconstruction, YAML schema, INDEX.md routes, router calibration.
Upkeep: recompile when a source changes; curate routes; re-test the router.
Matches words, not meaning: “egress” will not find “outbound”.
Reads at most two documents per question.

Measured: 5 preset questions - gemma4:26b - local Ollama - fact coverage is keyword-based. Raw and vault sizes are estimates.

Path B pays off when the same library gets the same kinds of questions, a lot.

Compile + route when…	Send the PDFs when…
The library is a stable reference: standards, compliance frameworks, SOPs	It's a one-off read of a new document
Agents re-read the corpus on every step of a loop	The library is small, fits easily, and is rarely queried
The model is on-prem or has a small context window	Documents change weekly
Someone has to show what the model was given	Questions are open-ended and need meaning, not keywords

A practical path: start with raw PDFs, log real questions, then compile the documents people ask about most. Provider prompt caching narrows the cost gap for repeated identical prefixes. It doesn't lift the context-window ceiling.

Questions per day

Input price, $ per 1M tokensexample rate: enter your contract price

Build effort, hours

Upkeep, hours / month

Loaded engineering rate, $/h

Scenario

Tokens saved per question -

Net saved per month -

Break-even -

On-prem, the bill is GPU time rather than tokens: 25 s vs 4.6 s to the first token means one box answers roughly five times as many questions.

The 8-document corpus

RAW PDF TOKENS → VAULT TOKENS

PATH A INPUT

Raw PDF text extraction

PATH B INPUT

Sanitized vault Markdown

TAB 03 / HOW THIS WAS BUILT

One overseer, one builder, one red team per work package.

Every work package went through the same loop: a written spec, a builder submittal, an adversarial review by a different vendor's model, and a v2 that answers each finding. The artifacts below are the project's actual submittal records.

Section 1 - The Voltron harness

CONTROL ARCHITECTURE

CONTEXT FIREWALL

Fast Worker Subagents (Gemini Flash)

The Gemini Pro overseer keeps the long-lived project context. File reads, log scans and tool calls go to short-lived Gemini Flash worker subagents; only their conclusions come back, so the overseer's context never fills with raw tool output.

SUPERVISION

Dynamic Supervisory Watchdog Timers

Builder and red-team runs are long and headless. The overseer arms watchdog timers instead of polling: zero tokens spent while dormant, then a supervisory check-in on wake that decides whether the run is progressing, stalled, or needs redirection.

SHARED SOURCE OF TRUTH

BRAIN.md & Markdown Specs

Plain-text Markdown (BRAIN.md, overseer.md, and .submittals/*.md) acted as the vendor-agnostic memory bus coordinating Google, Anthropic, and OpenAI models, preventing context drift and eliminating reliance on shared chat histories.

AUTONOMOUS ENGINEERING

Self-Engineered OT PDFs

In Stage 02, the agents calculated that 300-page government manuals would exceed the Mac Mini's 16k context window, and autonomously wrote a Python PDF publishing engine that generated four multi-column OT documents (61 pages) with 10 deterministic firewall "Table Traps" (ground_truth.json) alongside 4 real downloaded .gov PDFs.

LIFECYCLE QUOTA BURN

Multi-Vendor Quota Efficiency

Gemini (overseer)

~5%

Claude Max (builder / auditor)

ChatGPT Pro (red team)

<1%

Multi-vendor quota efficiency across submittal lifecycle.

Section 2 - Work-package walkthrough

SELECT A WORK PACKAGE

Four catches where review mattered.

CROSS-VENDOR CATCHES

01 - INFRASTRUCTURE

launchd respawn race during rollback

KeepAlive=true respawned the failing binary and held the socket.
Rollback could not swap the stable build back in.
Fix: flock + pinned version + explicit lifecycle control.

02 - CORPUS

Wrong statute, overflowing table

6 U.S.C. § 652 hardcoded; NIST needs 44 U.S.C. § 3551.
Firewall table 620 pt wide in a 520 pt frame.
Fix: per-document citations, all-cell verification.

03 - TABLES & SCORING

0/22 tables, negation-blind scorer

Character interleaving: 0/22 exact tables → 22/22 after clip extraction.
"Write access allowed into SIS" scored 100%.
Fix: polarity-aware scorer with negative controls.

04 - ROUTER CALIBRATION

22.7x TTFT Speedup & Table Preservation

Pass 1 pruned context to 687 tokens (22.7x speedup), but dropped secondary tables.
Calibrated extractor to 1,682 tokens (88.2% reduction) to preserve all ground-truth tables.
Preserved 100% table accuracy while maintaining 10x - 15x TTFT speedup.

Full audit trail: every submittal, finding and dispositionExpand to load

Builder-declared dispositions are listed separately from explicit overseer acceptance records.