Ask anything, share nothing: How we built our AI assistant for internal documents

Almost every company is having the same conversation these days: can we just ask ChatGPT or Claude about our internal documents? It’s easy to see why this is attractive. Policies, guidelines, and how-to knowledge are scattered across wikis and PDFs, and people keep asking the same questions. The problem is easy to understand, too: those documents are sensitive. Sending them to a third-party API, along with the questions employees ask about them, carries legal, compliance, and trust implications that most organizations are not willing to accept casually.

At ProductDock, we ran into this exact same problem with our own internal documentation. So we built the assistant ourselves: a retrieval-augmented generation (RAG) chatbot where every component runs on infrastructure we control (a GPU VM in our own AWS account, inside our own security boundary). The language model, the vector database, the session storage, all of it. Not a single token leaves our environment or touches a third-party AI API.

This post walks through what we built, the key decisions we made, and the lessons we learned along the way. While we provide context where necessary, we assume familiarity with common LLM serving, inference, and retrieval concepts. By doing so, we can focus strictly on implementation details rather than foundational explanations.

The constraint that shaped everything: no data leaves our infrastructure

Most RAG tutorials start with an OpenAI or Anthropic API key. We couldn’t, and that single constraint drove the architecture. The LLM is self-hosted. We serve an open-weight model with vLLM on the NVIDIA GPU. vLLM exposes an OpenAI-compatible API, so the rest of the stack is written as if it were talking to a cloud provider. We just point it at our own hardware.

Retrieval is self-hosted. Documents are chunked, embedded, and stored in Qdrant, running as a container next to everything else. Sessions, message history, and user feedback live in Postgres. Even observability stays home: every LLM call, including the tool call decisions the model makes, lands as a trace in a self-hosted Arize Phoenix instance via OpenTelemetry, so we get the visibility of a SaaS observability platform without shipping prompts to one.

Self-hosting has become compatible with the developer experience of cloud-based solutions. Since vLLM speaks the OpenAI API dialect, every library, framework, or code written for cloud providers works seamlessly without modification. It also gave us a free dev/prod split: on a laptop, we develop against Ollama, which talks the same API, and the only difference between “running on my machine” and “running on the GPU server” is a base URL in an env file.

The shape of the system

Three processes share the infrastructure (Image 1).

The indexer is a one-shot job that reads our markdown documents, chunks them, embeds them, and stores them in Qdrant. It’s incremental: it hashes documents between runs, so a rerun only re-embeds what actually changed, and documents deleted at the source get pruned from the index. Getting this right matters more than it sounds. The naive “re-embed everything” approach is either slow or, worse, silently duplicates chunks and degrades subsequent answers.

The backend API is FastAPI sitting in front of vLLM, Qdrant, and Postgres. The design is agentic: instead of always searching before answering, the model itself decides when to call the search tool, meaning a vague follow-up gets a clarifying question instead of a search, while a real question triggers one or more targeted searches.

The frontend widget is where we made a deliberately unfashionable choice. It’s not a React app. It’s a web component built with Lit: one script tag and 

<pd-chat-widget api-url=”…” api-key=”…”>

drops the assistant into an internal portal or any page that lets you add HTML. Shadow DOM keeps the host page’s CSS out and ours in (Image 2).

Retrieval: the boring details are the quality

Two retrieval decisions moved the answer quality more than anything else we tried.

The first: embed the headers, not just the text. A chunk that reads “Employees may take up to 20 days per calendar year” is topically anonymous. The embedding model lacks any contextual understanding of the subject matter, such as , e.g., parental leave policies. But the chunk lives under a heading path like Benefits > Parental Leave, and that context is free. We prepend the heading path and filename to the text the embedding model sees, while the LLM still receives the clean original chunk. For documents with numerous headings, this single adjustment noticeably improved recall, at zero runtime cost.

The second: hybrid search with graceful degradation. Dense vector search handles paraphrasing, so “time off when my baby is born” finds the parental leave section. Traditional sparse keyword search handles the rare proper nouns and internal acronyms that embeddings compress into mush. We fuse both with Qdrant’s score fusion, possibly expand the user’s question into several keyword-style variants, and rerank the merged results with a cross-encoder against the original question (Image 3).

The design rule behind the dotted lines in that diagram: every stage above basic hybrid search is optional. If query expansion fails, we search the original question. If the reranker throws an exception, we fall back to fusion order. Retrieval quality features should be optional enhancements that can be disabled when they fail, not dependencies on which the system relies.

The local LLM lesson: buy parameters with quantization

We cannot overstate this: poor decision-making in small models is the primary reason local-LLM initiatives often fall short. We started with Llama-3.1-8B-Instruct, a sensible-looking choice for a single GPU. Over the following months, our git history reads like a tour of the open-weight model zoo: Qwen3-14B-AWQ, mistral-nemo, a 2B-class Gemma, to name a few. Every one of them worked in the demo sense (fluent answers, plausible citations) and every one of them fell apart when the agentic behavior mattered. Our design leans on the model’s judgment: deciding whether to search, extract keywords, ask a clarifying question, or decline an off-topic request.

We know exactly how they fell apart, because we measured it. One of our eval sets is a routing eval: 42 hand-written messages, each labeled with the top-level decision the model should make: search, ask for clarification, decline, or just respond to a greeting. Here is the scorecard:

The failure mode isn’t stupidity, it’s overeagerness. A small model always knows when to search and almost never knows when to stop: vague questions got a confident answer instead of a clarifying question, and off-topic requests got answered instead of declined. On top of that, only 19 of the 23 correctly-routed search calls had well-formed keyword arguments. The rest passed whole sentences or bare pronouns as search queries. No prompt we wrote moved these numbers, because it isn’t a prompting problem. It’s a judgment problem, and judgment scales with parameters.

The fix was a bigger model, made affordable by quantization.
Qwen2.5-32B-Instruct-AWQ is a 32-billion-parameter model compressed to roughly 20 GB of weights, so it fits our single GPU — an NVIDIA L40S with 48 GB of VRAM — on an AWS g6e.xlarge, with room left over for the KV cache. That’s the smallest GPU instance in its family, around $2/hour on demand, and it runs the entire company’s assistant. Our production vLLM configuration:

command: >
  Qwen/Qwen2.5-32B-Instruct-AWQ
  --quantization awq_marlin
  --max-model-len 8192
  --gpu-memory-utilization 0.90
  --enable-auto-tool-choice
  --tool-call-parser hermes
  --enable-prefix-caching

Two details in there deserve attention. The tool-call parser is part of the model choice: every model family emits tool calls in its own format, and vLLM needs to be told which one to expect. We tried this as our config went from –tool-call-parser llama3_json to gemma4 to hermes as the models changed. Swap the model and forget to swap the parser, and tool calling fails in ways that look exactly like model stupidity: the model emits a perfectly good call in its native format, the server doesn’t recognize it, and the “call” arrives as plain text in the chat. And prefix caching is free performance: every request starts with the same long system prompt, and with –enable-prefix-caching vLLM computes it once and reuses it.

The takeaway for anyone sizing a local deployment: in an agentic RAG setup, a quantized large model beats a full-precision small one. Spend your VRAM on parameters, not precision.

Evals: turning “I think it’s better” into a number

The routing scorecard above didn’t exist on day one, and building it changed how we work. Every tweak in a stack like this (prompt edit, an embedding swap, a reranker toggle, a model change) raised the same question: did that make things better or worse? Without evals, the answer is vibes. Ours has four surfaces, ordered from cheapest to most expensive to run:

• Routing: the 42-message set above. It replays only the first LLM call, no retrieval involved, so a failure points straight at the prompt or the model. It runs in seconds, which makes it the inner loop for prompt iteration.
• Retrieval: purely numerical, no LLM judge. The trick is how the dataset gets built: a script walks every chunk in Qdrant and asks the LLM to generate a question whose answer comes only from that chunk, along with the corresponding answer. This gives every question a ground-truth chunk ID by construction. The runner scores Recall@1, Recall@3, MRR, and nDCG, then runs the evaluation 4 times, toggling the reranker and query expansion on and off.
• End-to-end: the same questions POSTed at the real /chat endpoint, graded by LLM judges for faithfulness (is every claim supported by the retrieved context?), correctness, and citation quality.
• Safety: 20 hand-written adversarial prompts: prompt extraction, jailbreaks, false premises, social engineering. A judge classifies each response as refused, leaked, or derailed. This one runs before releases.

A confession that doubles as advice: we built this harness three times. Once in plain Python, once on DeepEval, once on Ragas, to see which one we’d actually keep using. Plain Python won. The frameworks earn their complexity on larger teams; at our size, a runner script, a JSONL dataset, and a results file you can sort by failures. The row-level failures are the real product anyway.

Keeping the model honest

A self-hosted model protects your data. It doesn’t protect your users from confident nonsense. The system prompt is strict about it: the model may only answer from content it retrieved via the search tool, and must say so when the documents don’t contain the answer, even when the answer “seems obvious.” Hallucinated policy answers are the fastest way to lose an internal tool’s most fragile asset: trust.

We also close the loop with users. Every answer in the widget includes thumbs-up/down feedback (Image 4), and ratings land in Postgres alongside the context that produced the answer. That’s not just a vanity metric. It’s exactly the dataset you need when you later tune chunk sizes, swap embedding models, or build an evaluation set.

What this means beyond our documents

Nothing in this architecture is really specific to documents. The constraint that shaped it, keeping sensitive data inside a boundary we control, applies to any workload you’d rather not hand to a third-party API. Swap the corpus ( HR policies, contracts, financial records, source code) and the pattern holds: incremental indexing, hybrid retrieval, a quantized open-weight model behind vLLM, and a drop-in interface where people already work.

If there’s one lesson worth carrying into your own build, it’s that self-hosting is no longer the compromise it used to be, and that nothing here felt trustworthy until we could measure it. The real payoff isn’t any single component. It’s that internal knowledge becomes instantly queryable while every byte stays inside a boundary you own, and that shift doesn’t require handing your data to anyone else’s API.

Building AI, the right way

At ProductDock, this is how we build AI: around your data, your constraints, and your definition of done, not around whatever a vendor’s API makes easy. Self-hosted when the problem calls for it, cloud when it doesn’t, and the same discipline either way. If you are looking for a partner who takes both the engineering and the trust seriously, you are in the right place. Reach out.