I've started Venture Deals three times. I've never finished it.
That's not really a discipline problem, it's a format problem. I care about maybe 15% of a book like that: the definitions, the mechanics, the market norms, the "here's where founders get burned" warnings. That 15% is smeared across 300 pages of anecdote and connective tissue. I read the way I grep. Books don't grep.
So I did the thing engineers do instead of solving the actual problem. I built a tool.
Lens Distill takes a PDF plus a free text "topic lens" and turns it into three things:
Atomic claims. Self contained paraphrased statements, each citing the exact paragraphs it came from.
A canonical concept vocabulary. The book's real nouns, normalized and deduplicated.
A sparse concept graph. Prerequisite, related, and confusable edges between those concepts.
There's a Next.js app wrapped around it so other people can watch it run, but that's just a window. The work is the pipeline, and this post is about the pipeline, mostly about the places it broke. Almost all of them broke quietly.
Seven stages, one line
The whole system is a linear job queue. No DAG, no fan out, no orchestration framework, no cron.
| # |
Stage |
LLM? |
Turns |
Into |
| 0 |
(parse) |
no |
PDF bytes |
paragraphs with global indices |
| 1 |
chunk |
no |
paragraphs |
~1200 token windows |
| 2 |
embed |
embedding model |
chunks |
1536-d vectors |
| 3 |
extract |
Haiku |
chunks + persona |
cited claims |
| 4 |
dedupe |
embeddings + Sonnet |
claims |
claim clusters, survivors |
| 5 |
canonicalize |
embeddings (fallback only) |
raw tags |
raw_tag → canonical_tag map |
| 6 |
concepts |
no |
canonical tags |
graph nodes |
| 7 |
concept_graph |
Opus |
nodes |
typed edges, then ready |
Each stage writes its rows and enqueues exactly one next job, or returns null and the book is done. That linearity is a deliberate call. A boring pipeline is a debuggable pipeline: when a book fails there's exactly one place it failed, and that stage's metrics tell you which numbers looked wrong going in.
Everything lives in Neon Postgres with pgvector (1536 dimensions, HNSW cosine indexes on chunk and claim embeddings), Drizzle for the ORM, pdfjs-dist for parsing, js-tiktoken for token counting, the Anthropic SDK direct for the three generative stages, and text-embedding-3-small through OpenRouter.
Stage 0: PDF structure is the boss fight
I assumed parsing would be the boring part. It's the riskiest stage in the system and the only one where corruption is completely silent. Everything downstream succeeds. The output is just garbage.
Here's the canonical failure from an earlier version: a book whose headings wrapped across two lines produced 157 detected chapters instead of 21. Nothing threw. Chunks hard break on chapter boundaries, so chunking produced hundreds of fragments. Extraction ran on fragments and emitted thousands of context free claims. The concept stage built a cloud of junk tags. Ten dollars of tokens later I had a beautiful graph of nothing.
The detection rules that came out of that:
| Rule |
Detail |
| Body size |
Estimated from line heights across the document |
| Chapter candidate |
Line height ≥ 1.35x body, usually first content on a page, followed by body text |
| Section candidate |
Line height ≥ 1.15x body |
| Wrapped headings |
Merged before classification. This is the 157 chapter fix |
| Demotion |
Any "chapter" with < 15 paragraphs becomes a section of the previous chapter |
| Headers/footers |
Dropped if a normalized key appears on > 30% of pages |
| Two column |
Detected when left/right x-modes are split by a mid page gap |
Then a gate: after demotion, chapter count must land in [5, 40] or the upload throws before a cent is spent. A book with 3 chapters or 157 chapters is a book I parsed wrong.
On page numbers, I fail closed. Printed page numbers and PDF indices disagree constantly. Calibration samples pages and looks for a consistent offset, and I only ship a number when agreement is above 0.8. In the mid confidence band I store page = null. Not a guess, not best effort, null. The UI never renders an empty page chip either, and citations show chapter instead. A wrong page number is worse than a missing one, because a missing one tells you to go look.
Last thing, and it's the one I'd tattoo somewhere: every paragraph gets a para_index that's global and monotonic across the whole book, not per chapter. Primary key is (book_id, para_index). I did per chapter first. Those indices collide the moment you join across chapters, and the join succeeds. It just hands you the wrong paragraph. Everything looks fine and nothing is fine.
Stage 1: chunk
No LLM here, just windowing and gates.
| Constant |
Value |
TARGET |
1200 tokens |
MIN_TOKENS |
600 |
OVERLAP |
150 tokens |
SECTION_SOFT_WINDOW |
300 |
MAX_CHUNKS |
550 |
Grow a window by tokens until near target. Hard break on chapter change. Prefer a soft cut at a section boundary if the trailing segment would otherwise be stubby. Overlap ~150 tokens when advancing. Merge undersized adjacent same chapter slices if the combined size still fits under target.
Three loud gates:
That last one is the 157 chapter canary. If chapter detection exploded, chunks come out tiny, and I'd rather kill the book here than at the graph stage an hour and several dollars later.
Stage 2: embed
|
|
| Batch size |
100 chunks per call |
| Dimensions |
1536 |
| Input |
sectionTitle + "\n\n" + text when a section title exists |
| Billing |
Input tokens only, output always 0 |
The thing I got wrong here was conceptual rather than mechanical. There's one vector per chunk, not per token. The API tokenizes the string internally and returns a single 1536 dimension vector for the whole thing. Obvious in retrospect, but my usage accounting was recording one "call" per chunk instead of per batch, which inflated the call count 100x and made the cost dashboard nonsense.
The stage processes chunks with null embeddings ordered by index and re-enqueues itself with a cursor if work or time runs out.
First generative stage, and where the good idea lives.
|
|
| Model |
Haiku |
| Chunks per invocation |
40 |
| Worker concurrency |
4 over a shared in memory queue |
max_tokens |
4096 |
| Output |
Forced tool emit_claims via tool_choice |
| Claims per chunk |
≤ ~12, statements ≤ 300 chars |
The system prompt is hardcoded in the app and the uploader can't touch it. The user's persona goes in the user message, inside a <persona> fence, framed as a topic preference rather than instructions.
The chunk text carries inline paragraph markers:
[p412] Investors typically negotiate for a 1x non-participating preference...
[p413] The participating preferred, by contrast...
Because every paragraph is labeled with its global index, the model cites by marker and I can verify the citation deterministically, with no LLM judge:
support_paras must be non-empty
every index must fall inside [paraStart, paraEnd] of the source chunk
otherwise drop the claim
A model that hallucinates [p9001] gets that claim discarded and the drop rate lands in the stage metrics. No second model call, no similarity check, no vibes. Just a range check on integers.
I also cap anchor_quote at 15 whitespace separated words and null anything longer. Partly a copyright posture, partly because paraphrase is the entire point. If I wanted the original sentences I'd have read the book.
Each claim carries:
| Field |
Notes |
statement |
Self contained paraphrase |
claim_type |
definition, mechanic, heuristic, warning, negotiation_move, anecdote, market_norm |
favors |
investor, founder, neutral, not_applicable |
anchor_quote |
Optional, ≤ 15 words |
support_paras[] |
Validated paragraph indices |
concepts[] |
Raw tags, canonicalized later |
Restarting extract at cursor 0 deletes the book's prior claims, so the stage is idempotent.
Stage 4: dedupe
Roughly two thirds of the wall clock, and the most interesting algorithm in the pipeline.
| Constant |
Value |
Meaning |
SIM_THRESHOLD |
0.86 |
Cosine at or above this is a cluster candidate |
AUTO_MERGE_THRESHOLD |
0.92 |
Merge deterministically, skip the LLM entirely |
Projection window W |
min(48, n−1) |
How many neighbors each claim compares against |
LLM_MERGE_BATCH |
24 |
Sonnet merges per drain invocation |
LLM_CONCURRENCY |
6 |
Parallel merge calls |
The phases:
Embed every live claim's statement text, batches of 100.
Cluster cheaply. Not O(n²). Project all embeddings onto a deterministic 1D axis (LCG, seed 42), sort claims along it, and only compare each claim to its next 48 neighbors. Cosine above 0.86 unions them. Locality sensitive enough for near duplicates, and linear instead of quadratic.
Auto merge any cluster whose tightness (min cosine to centroid) clears 0.92, with no LLM at all. Canonical wording picked by type rank (definition beats mechanic beats the rest) then longer statement.
LLM merge only the 0.86 to 0.92 band, with Sonnet and an emit_merge tool, re-enqueueing until drained.
Assign a durable cluster_id and compute a 2D PCA projection for the embedding scatter UI.
That 0.92 auto merge is what makes the whole thing affordable. Without it Sonnet sees every cluster and the bill roughly triples for merges that are obviously the same sentence written twice.
Losers get superseded = winner_claim_id, and every downstream read filters superseded IS NULL. Forgetting that filter exactly once is how you get duplicate claims in your concept counts.
Two metrics lessons. I was overwriting cumulative merge counts with per invocation values, so a stage that ran eight times reported the eighth run's numbers. Fixed by computing before/after counts from the database at finish instead of accumulating in memory. And on the Venture Deals run most clusters came out size 1, which made the rainbow cluster colors in the scatter plot pure noise. I'd built a color legend for a variable with ~900 distinct values. Check cardinality before you reach for a color scale.
Stage 5: canonicalize
Raw tags are a mess: liquidation_preference, liquidation preference, the liquidation preference clause. This stage builds the raw_tag → canonical_tag map, and the philosophy is load bearing.
String first, embeddings last. Lowering the cosine threshold is specifically the wrong move.
| Rule |
Detail |
| Normalize |
Lowercase, unify punctuation |
| Strip |
Container nouns only: agreement, clause, provision, covenant, language, section, terms |
| Never strip |
Meaning bearing words: right, cap, multiple, period, threshold |
| Negation |
Hard block on non, un, anti, de, dis, no prefixes. Never merge at any similarity |
| Embed fallback |
Only for near singletons with ≥ 2 claims, cosine ≥ 0.93 |
The negation rule isn't hypothetical. nonparticipating_preferred and participating_preferred embed at very high cosine similarity and mean opposite things. Merging them corrupts the single most important distinction in the book. There's an assertNoNegationCrossMerge gate that throws if one slips through.
Semantic similarity is exactly the wrong tool for antonyms. The fix is a regex, not a better model.
Stage 6: concepts
A canonical tag becomes a graph node only at 8 or more claims. Fewer than 5 nodes total throws, because a book that produces 3 concepts wasn't distilled, it was mangled.
Descriptions get picked heuristically by claim type, with a rule that definitions outrank modal obligation claims when both exist. Otherwise information rights gets described as "the company must deliver audited financials within 90 days," which is a fact about the concept, not the concept.
Each node also gets a primary_chapter, computed as the modal chapter across its claims, not MIN(). I used MIN() first. Almost every important concept gets name dropped once in the intro, so MIN() assigned half the graph to chapter 1. Modal assignment puts each concept where the book actually teaches it.
Stage 7: concept_graph, and the bug that shipped
Terminal stage. Opus, max_tokens: 16000, tool emit_concept_graph, emitting three edge kinds with per concept caps (prereq ≤ 3, related ≤ 5, confusable ≤ 3) to keep the graph sparse on purpose.
An earlier version asked the model for claim_ids alongside every edge, for the whole book, in one call. On a large book the response hit the ceiling. So:
stop_reason came back max_tokens.
The JSON was truncated mid object.
The parser caught the error and returned [].
The code saw an empty edge list and fell back to a tag only graph with zero edges.
The book was marked ready.
The UI rendered a perfectly nice force graph of disconnected dots.
Nothing logged an error. Nothing failed. The product quietly became worthless, and I found it days later only because the graph looked suspiciously clean.
The fixes are all about refusing to be helpful:
A shared requireToolUse helper that throws on stop_reason === "max_tokens" or a missing tool_use block. No parsing, no salvage.
The tool schema no longer asks for per edge claim ids across the entire book. Don't build a call whose success depends on the response being small.
A coverage gate: if fewer than 80% of concepts have at least one edge, throw the stage.
Post processing normalizes concept references to real ids, then breaks cycles in the prerequisite subgraph with a recursive CTE, preferring to delete edges that run backward in chapter order. cyclesBroken is a metric, because a book generating lots of prerequisite cycles is telling you something about your concept extraction.
Then the book goes ready, the handler returns null, and the queue is empty.
Silent fallbacks are how demos lie. A fallback that produces a structurally different result (empty edges, fake pages, a tag only graph) is worse than a crash, because a crash gets fixed and a fallback gets shipped.
Holding it together: the queue
A full book takes about an hour, so none of this fits in one request. The orchestration is deliberately minimal:
Jobs are claimed with SELECT ... FOR UPDATE SKIP LOCKED LIMIT 1, which means the worker needs a real transaction. Reads go over Neon's HTTP driver, the worker needs a WebSocket Pool, and every worker path ends in pool.end() inside a finally or you leak connections.
Long stages don't finish in one job. They re-enqueue themselves with a cursor in jobs.payload.
The drain claims, runs, and inserts the next job until 45 seconds before its budget expires, then setTimeouts itself in process if pending jobs remain. Self chaining instead of cron.
Failures back off 2^attempts * 30 seconds, three attempts max. Jobs stuck running for over 15 minutes get recovered to pending.
A module level mutex means one book, one stage at a time, site wide.
Untrusted personas
The persona field is free text from strangers that ends up in a prompt I pay for. Defense in depth, ordered by how much I actually trust each layer:
Forced tool output. There's no free text channel to hijack.
Deterministic citation validation. Even a successful injection has to emit claims citing in range paragraphs.
A global spend cap. Three books per rolling 7 days, site wide, enforced with an advisory lock and a quota row written in the same transaction as the book insert.
A fixed system prompt that frames the persona as a topic preference.
A soft deny list at upload.
That order is deliberate. The deny list is the layer everyone reaches for first and it's the weakest, because deny lists are always incomplete. The real backstops are the tool schema, the range check, and a bounded worst case.
Also worth saying: the PDF bytes are never stored. Parse to paragraphs, discard the binary. Privacy choice, cost choice, and a "this doesn't become a pirate library" choice.
What the run actually looked like
| Metric |
Value |
| Paragraphs / chapters / chunks |
3,001 / 21 / 109 |
| Claims extracted |
~1,054 |
| Claims live after dedupe |
~978 (76 superseded) |
| Concepts / edges |
~54 / ~318 |
| Wall clock |
~45 to 60 min |
| Modeled cost |
~$1 to $2 |
Per stage: embed ~1 min, extract ~6 min, dedupe ~40 min, canonicalize ~5 min, graph ~2 min. Dedupe is 70% of the runtime. Embeddings cost cents. Opus on the graph is a real fraction of the dollars once the concept set gets large.
Did it work? I now know what a participating preferred with a cap does, why full ratchet anti dilution is a founder's problem, and what a drag along actually drags. I can click any of those claims and expand it to the paragraph it came from. I did not read the book.
I did read about 900 sentences from it, individually, sorted by concept. I'd argue that's reading the book with the connective tissue removed. Reasonable people will disagree.
The cheat sheet
| Area |
Numbers |
| Parse |
25 MB cap, chapters 5 to 40, demote under 15 paragraphs, page offset only above 0.8 agreement |
| Chunk |
1200 target, 600 min, 150 overlap, 550 max |
| Embed |
batch 100, 1536-d, input only billing |
| Extract |
40 chunks per job, concurrency 4, max_tokens 4096, quote ≤ 15 words |
| Dedupe |
sim 0.86, auto merge 0.92, window ≤ 48, LLM batch 24 at concurrency 6 |
| Canonicalize |
embed fallback ≥ 0.93, negation hard block |
| Concepts |
≥ 8 claims per node, ≥ 5 nodes total |
| Graph |
max_tokens 16000, ≥ 80% edge coverage, cycles broken |
| Queue |
12 min budget, 45s claim cutoff, 15 min stale recovery, 3 attempts |
What I'd tell past me
Inline markers turn citation into a range check. Labeling paragraphs [p412] costs a few tokens and buys deterministic verification. Never let a model invent identifiers you plan to join on.
Fail loudly or your pipeline will lie. Every silent fallback here produced output that looked correct.
String first, embeddings last. Embeddings can't do antonyms.
Global identifiers, always. Per chapter indices are a join bug waiting for a quiet afternoon.
Gate before you spend. Chapter count, chunk count, average chunk size. Each one caught a book that would have failed slowly and expensively.
Modal, not minimum. Think about which statistic actually encodes what you mean.
The pipeline stops at the graph on purpose. Claims and prerequisite edges are obviously the substrate for something that schedules retrieval practice, and I've been building that separately for myself. Different product, different failure modes. This one's job was just to make the nodes trustworthy enough to rank.
