Most of the difficulty in retrieval-augmented generation is not the retrieval. It is the decisions that never raise an error when you get them wrong. This is a record of those decisions, the alternatives they beat, one that I got wrong and had to trace back, and the parts that are deliberately still missing.
Two independently deployed applications from one repository, rather than a single framework managing both.
Next.js 16 · React 19
Server-rendered marketing, client-rendered app. Reads the answer stream as it arrives.
Express 5 · TypeScript
Auth, documents, and the retrieval pipeline. Layered routes to services to transport-free logic.
Neon Postgres · pgvector
Documents, chunks and 1536-dimension embeddings in one database, behind an HNSW index.
The API is a real Express service rather than a set of framework route handlers. That was the point of the exercise: to build a backend and be able to defend its structure, not to let a framework own decisions about layering, transport and lifecycle on my behalf. The cost is a second deployment target and a real cross-origin story. Both were worth paying.
Four stages, each with a failure mode that shaped how it is written.
A document is hashed, split into overlapping chunks, and embedded in batches. Re-uploading identical text is deduplicated by the database rather than by application logic, so two concurrent uploads of the same file cannot race.
The question is embedded with the same model, then matched against only the caller's chunks by approximate-nearest-neighbour search over an HNSW index. Ownership is part of the query, never a check performed after it.
The top passages become a numbered context block at temperature 0, and the model must cite them. Sources stream to the browser before the first token of the answer, because retrieval is fast and generation is slow.
A follow-up is rewritten against the conversation before anything is searched, because a question like "how long is it?" carries no meaning an embedding model can use. Instructions about the previous answer rather than requests for new information skip retrieval entirely and reuse the sources already on screen.
Every one of these had a reasonable alternative. What follows is why the alternative loses.
IVFFlat learns its cluster centroids from the data present when the index is built, so building it on a near-empty table produces a permanently bad index that no amount of later inserts repairs. HNSW builds incrementally. For a table that grows one upload at a time, starting empty on day one, it was the only honest option.
A cosine index paired with a Euclidean distance operator never raises an error. It silently stops using the index and falls back to scanning every row. The symptom is not a failure, it is slowness with correct results, which is exactly the class of bug that survives review and reaches production. Verified by forcing the planner to reveal whether it really uses an index scan.
Chunks and the document status flip are written together in one transaction, so a document is never readable in a half-ingested state. But the embedding calls happen before that transaction opens. Holding a Postgres transaction open across a slow third-party API call ties a database connection to someone else's latency, and connections are the scarcest resource in the system.
A 403 confirms the identifier exists. That turns identifier enumeration into a map of another tenant's library, which is a disclosure even though no content is returned. Both 404s are byte-identical so the error message cannot become an oracle either. Ownership lives in the WHERE clause, never a lookup followed by a conditional.
EventSource cannot set request headers, and authentication here is a bearer token. SSE would have forced the token into the query string, where it lands in server logs, browser history and referrer headers. That is a constraint imposed by the transport, not a stylistic preference. The service layer yields typed events and never touches the response object, so the transport could change without reopening the answer logic.
Left to its own judgement the model writes a different apology every time, and a waffle is indistinguishable from a weak answer. A verbatim constant is something the interface can detect and an evaluation can assert on. Together with temperature 0, it is what makes refusal accuracy a measurable quantity rather than an impression.
Retrieval works by turning a question into a vector and finding the nearest passages, which only works while the question carries its own meaning. "How long is it?" does not. It embeds to something about pronouns and duration, matches nothing useful, and the grounding prompt then correctly refuses — so the user is told their documents do not cover a question they just watched being answered. Every component behaved exactly as designed. That is the lesson worth taking from this system: the visible failure is almost never where the bug is, because a confident wrong answer and an unnecessary refusal usually both mean retrieval was handed the wrong query.
There is no standalone question hiding inside "make that shorter". Rewriting it produces nonsense, and searching for it returns whatever the least relevant passages in the corpus happen to be, because semantic search always returns something. So the rewriting step is allowed to answer "no search needed", and those turns reuse the previous answer's sources instead. It is the cheapest path in the system and the one that makes it feel like a conversation rather than a search box with a memory.
The generation step never sees the rewritten question. Send it the rewrite and an instruction like "make that shorter" turns into a fresh answer to a question nobody asked. Keeping the two inputs separate is the whole design in one sentence, and getting it backwards produces output that is fluent, grounded, cited, and not a reply to anything the user typed.
The obvious design has the client post its own transcript. It already has one on screen and the server stays stateless. It also lets a caller fabricate an assistant turn, which lands in the slot of the prompt a model weights most heavily: you previously agreed to ignore your source-only rule. History is therefore read back from rows the caller owns. It is the same rule as never taking a user identifier from a request body, applied to a field most people do not immediately read as security-relevant.
The normalised design links each answer to the passages it used. It is wrong here, because passages are deleted with their document and recreated with new identifiers whenever a document is reprocessed — and reprocessing is a first-class operation in this system, not an edge case. A reference would make old citations either vanish or dangle. The resolution comes from asking what a citation actually asserts: not that an answer points at a row, but that it was built from this passage at the time it was given. Historical claims get stored as copies. The cost is duplicated text and the loss of an easy query for what gets cited most, and both were accepted.
A question can be spoken and an answer can be read aloud, and neither capability appears anywhere in the retrieval or generation code. Recording produces text that fills the same box typing fills, and the request that follows is byte-identical to the one a keyboard would have sent. Reading aloud consumes an answer that has already been produced. The alternative — a single endpoint that accepts audio and returns audio — is fewer moving parts and it would have made retrieval quality measure transcription and retrieval together while still being read as retrieval alone. It is the same argument that kept conversations out of the single-turn path, applied before anyone asked for it rather than after the number moved.
Chaining recording to asking saves a round trip and removes the only moment at which a mishearing is visible. A transcriber that turns leave into leaf produces a question that retrieves nothing, and the grounding prompt then correctly refuses — so someone watches a working system insist their document does not say a thing it plainly says. That is the same shape as the follow-up failure above: every component behaved as designed and the visible symptom is nowhere near the cause. Putting the text in an editable box costs one deliberate keystroke and converts a silent retrieval failure into an obvious typo.
The two halves of a voice feature look symmetrical and are not. A wrong transcript changes which passages are retrieved, so input quality changes what the system does; a synthetic voice reading a correct answer is still a correct answer, so output quality is decoration. Money goes where an error changes the result. The side effect is that the output half adds no endpoint, no rate limiter and no cost to a link posted publicly, which is the kind of consequence that makes a decision easy to defend twice.
The answer carries bracketed citation markers and light formatting, both of which several synthesis engines read out literally. Speech therefore gets its own projection of the same string, in the same way a transcript and a model prompt are two projections of the same stored messages. Nothing is lost, because the cited text is on screen while it is being spoken. The harder half is that the answer arrives as a token stream and a token respects sentence boundaries no more than a network packet respects line boundaries: sentences are buffered and the trailing one is always held back, since a buffer ending in version one looks finished until the next token turns it into version one point five.
Every other paid route here costs a roughly fixed amount per call, so counting calls bounds spend. Transcription is billed by duration, which means a single permitted request can cost whatever the caller chooses to make it. Counting is therefore only half the control: the recording stops itself after a minute in the browser, the upload is capped at a size chosen from what a minute of compressed speech weighs, and only then do per-minute, per-day and global request limits mean anything. The size cap is an imperfect proxy for duration and is documented as one, because the codec decides how many seconds fit in a megabyte.
The rewriting step sits in front of retrieval, so folding it into the existing single-turn endpoint would have made retrieval quality measure rewriting and retrieval together while still being read as retrieval alone. When the number moved there would be no way to attribute it. Chat is a sibling instead: the single-turn path is unchanged down to the system prompt string, and the conversation rules are appended to that prompt rather than edited into it. Both surfaces call the same retrieval and generation code, so the thing being measured is still the thing people use.
Left unset, the rate limiter sees the platform edge and the entire internet shares one bucket. Set to true, the framework trusts the whole forwarded-for chain, and since anyone can send that header an attacker mints a fresh unlimited bucket per request. Both failures report correctly in the response headers while enforcing nothing.
Rate limit counters fall back to in-memory storage when no Redis URL is configured, which is what lets a fresh clone run with nothing but Postgres. On an autoscaling platform that same fallback gives every instance its own counters, making the effective limit the configured limit multiplied by the instance count. It fails silently and reports success, so in production the process refuses to start instead.
Two of my own decisions fought each other, and the instruction lost to the example.
Every answer numbers its sources from one. In a conversation each turn retrieves its own passages and numbers them again, so the second turn’s [2] and the fourth turn’s are different documents entirely. Replaying old answers into the prompt with those markers intact invites the model to reuse a numbering that no longer means anything, so the markers are stripped out of history first. Sound reasoning, and it is the right default.
Then I tested “make it shorter”, which reuses the previous turn’s sources rather than searching again. The answer came back correct and completely uncited. The system prompt has an explicit rule requiring an inline citation on every claim. I added a second rule stating that reformatting requests still need them. It was ignored again.
The cause was the stripping. On a reuse turn the model is shown its own previous answer with every citation removed, and then asked to reproduce it more briefly. It copied what it saw. The demonstration sitting in the context window beat the instruction sitting in the system prompt.
The fix is narrow, because the reasoning behind the stripping was never wrong in general: keep the markers on the final answer, but only on a reuse turn, where the sources being sent genuinely are the same ones the numbers referred to. What I would actually take to the next project is the diagnostic habit rather than the patch. When a model ignores a rule, read what the prompt is showing it before writing another rule, because prompt failures are frequently demonstration failures wearing an instruction’s clothes.
Listed because a project that claims no gaps is not being described honestly.
A unit and HTTP suite runs with no database and no network: validation schemas, chunking, and everything that resolves before the first query. What it cannot reach is anything needing a real row, which unfortunately includes the three highest-value targets — the tenant-scoping predicate in the hand-written vector SQL, the deduplication race, and the stream parser. Those need an integration tier against a real database with a stubbed embedder. The dependency-injection seams for it already exist; the tier does not.
It is the newest code here and it was verified by hand against a real database and a real model, which is not the same thing and is worth saying before anyone asks. Three pure functions belong in the existing suite today. The persistence path, the append race and the guarantee that stopping a stream still saves the partial answer all need the integration tier above.
The test runner and a reproducible corpus builder exist. The golden question set and every metric do not. That is the next piece of work, and the reason several numbers below are described as chosen rather than measured.
Rewriting a follow-up before searching is a real improvement and an unproven one. It also introduces a failure mode that did not exist before: a rewrite that loses the user's intent produces confident retrieval of the wrong passage, which reads exactly like a retrieval regression and is not one. The comparison that would settle it is retrieval accuracy on the rewritten query against the same measure on the raw follow-up. It shipped on a product argument, because multi-turn is unusable without it, and no claim about its quality is made anywhere until that number exists.
Turning an answer into something worth listening to is pure, deterministic logic and the natural thing to cover. It is not covered, because the web package still has no test runner at all — the same gap that already leaves the citation parser untested. The logic was checked by driving a simulated token stream through it, which found a real defect: the sentence segmenter breaks after abbreviations like e.g., producing a pause in the middle of a sentence. That was fixed, and a fix confirmed by a script that no longer exists is a weaker claim than a fix confirmed by a suite that runs on every commit.
Transcription introduces a failure mode that did not exist before and sits upstream of everything: a mishearing produces a well-formed question that retrieves the wrong passages, or none. Keeping it outside the pipeline means the existing retrieval measurements stay attributable, but it does not measure the new step. The comparison that would settle it is retrieval accuracy on transcribed questions against the same measure on the typed originals. Until that exists, the honest claim is that voice is an input method, not that it is an accurate one.
A continuous spoken conversation is the version of this feature people picture, and it was rejected rather than postponed. The deployment platform does not hold long-lived socket connections, so retrieval would have to be called back into from inside the model's own session, which means the grounding prompt, the fixed refusal and the temperature setting stop being what produces the answer. Audio is also billed per minute against an open signup, which no request-counting limit bounds. And citations do not survive being spoken: the numbered markers are the product, and reading them aloud is either noise or nothing.
A PDF with no extractable text layer is detected and rejected with a specific error rather than silently ingesting zero chunks.
Access tokens are short-lived and there is no rotation scheme yet. Known, scoped, and not pretended otherwise.
Burst limits, a daily ceiling and a concurrent-stream cap bound usage today. Attributing actual spend per user needs usage reporting turned on in the streaming responses and a table to persist it.
Both are well-understood improvements to retrieval quality, and both are held until there is an evaluation harness that can show they help this corpus rather than adding them because the literature says they usually help. Query rewriting was on this list until conversations needed it, which is a change of sequencing rather than of principle — and it is why the item above exists.
An evaluation harness, and the reason it is the next thing rather than a nice-to-have.
Three numbers in this system were chosen by judgement rather than measured:
Defending a guess as a measurement is the fastest way to lose credibility, so they are described as guesses until a golden question set can settle them. The harness measures retrieval hit rate at k, mean reciprocal rank, groundedness, and refusal accuracy. Temperature 0 and the fixed refusal string are what make those quantities measurable at all, which is why both were decided early rather than tuned later.
It also settles a known limitation left in on purpose: page-aware chunking makes a page a hard chunk boundary, so a PDF of very short pages produces many very small chunks. Whether that hurts retrieval is a question for the harness, not for my intuition.
Conversations added a fourth question rather than removing any. Rewriting a follow-up before searching sits in front of retrieval, so it needs its own measurement — retrieval accuracy on the rewritten query against the same measure on the raw follow-up. The single-turn endpoint was deliberately left untouched so that the harness still has one path where retrieval is the only variable, which is the reason two ways of asking a question exist at all.
Sign up, paste a document or drop a PDF, and ask it something the document does not cover. Watching it refuse is the fastest way to check that any of this is true. Then ask a follow-up with a pronoun in it and watch what it searches for.
The source repository is private. If you are evaluating this as part of a hiring process and want to read the code, ask and I will grant access. Abhishek Satyam