Frontend LLMs & RAG: running retrieval-augmented AI in the browser

For most of the last decade, "add AI to your product" meant "send your users' text to someone else's servers." The model lived in a data centre, you paid per token, and every question a visitor typed left their device. That was the only option — models were too big and too slow to run anywhere else.

That assumption is now outdated. Quantised open-weight models a few hundred megabytes in size, combined with WebGPU and WebAssembly, run comfortably inside a modern browser tab. Libraries like transformers.js and wllama load a model once and run inference locally — no server, no API key, no per-token meter. Pair that with retrieval-augmented generation and you can build a genuinely useful, grounded assistant that never phones home.

RAG in one paragraph

Retrieval-augmented generation is a simple idea: instead of hoping a model memorised your content, you retrieve the most relevant passages from your own documents at question time, augment the prompt with them, and let the model generate an answer grounded in that context. The model supplies fluency and reasoning; your documents supply the facts. That separation is what keeps answers accurate and lets you update knowledge by changing documents rather than retraining anything.

The usual stack, and where the boxes move

In a conventional setup every box in that pipeline is a server: an embedding API turns text into vectors, a hosted vector database stores them, and a hosted LLM generates the answer. It works, but you rent all three forever, and your users' questions and your documents travel across the network to reach them.

The frontend approach moves those same boxes into the browser. Nothing about the RAG algorithm changes — only where it runs.

Index time — done once, ahead of time

Your documents are split into small overlapping chunks, and each chunk is turned into a vector by an embedding model. Those vectors are stored in a lightweight index that ships to the browser — small enough to keep in memory, or cached in IndexedDB so repeat visits skip the download. This step is offline work; visitors never see it.

Query time — entirely on the device

When a visitor asks something, their question is embedded with the same model, compared against the stored vectors to find the closest few chunks, and those chunks are dropped into the prompt. A local LLM reads that grounded prompt and writes the answer. Every step — embedding, search, generation — happens in the tab. No request leaves the browser.

Why put it on the frontend at all?

• Privacy by construction. Questions and documents stay on the device. There is no third-party API to trust, log, or breach — a claim you can make honestly because the network tab proves it.
• No inference bill. The visitor's device does the compute. There is no per-token cost, so a page that answers ten questions costs the same to run as one that answers ten thousand.
• It scales for free. Every new user brings their own hardware. Traffic spikes don't page anyone.
• Offline-capable. Once the model and index are cached, the assistant keeps working with no connection.

You could build all of this — or skip to the end

None of the pieces above are exotic, but wiring them together — choosing a model, chunking and embedding well, building the in-browser store, managing caching and device fallbacks — is real work. That is exactly what Zupport already does. You point it at your content; it ships a grounded assistant that runs entirely in your visitors' browsers.

And because there is no server doing inference, there is no metered bill to fund on our side — which is why Zupport is a single payment, not a subscription. You buy the agent once and embed it forever.