An AI agent is only as good as what it knows. Out of the box, a language model knows a lot about the world and nothing about your project: not your decisions, not your documentation, not last week’s change to the deploy process. Ask it anyway and it will either say it doesn’t know or, worse, confidently make something up.
Retrieval-augmented generation (RAG) fixes that. Before the model answers, the system searches the project’s own knowledge, pulls out the most relevant passages and hands them to the model as context. The model answers from your material, and it can cite where each fact came from.
This is how I set it up, and the decisions that mattered most.
The pipeline at a glance
A RAG system has two halves.
Ingestion runs in the background and keeps the knowledge base current:
- Pull content from the sources the project already uses.
- Clean it and split it into chunks.
- Turn each chunk into an embedding, a list of numbers that captures its meaning.
- Store the chunk, its embedding and its metadata in a vector database.
Retrieval runs every time an agent needs to know something:
- Turn the question into an embedding.
- Find the chunks closest in meaning, plus the best keyword matches.
- Rerank the results and keep the best few.
- Give them to the model with the question, and ask for an answer with citations.
Every step has choices that make the difference between a demo and something agents can rely on.
Sources: go where the knowledge already lives
The knowledge base pulls from the systems of record, not from a separate folder someone has to maintain:
- Confluence for decisions, designs and processes.
- Jira for the history behind features and fixes.
- Google Drive for documents and spreadsheets.
- Code repositories for READMEs, docs and architecture notes.
If knowledge only lives in a copy, the copy goes stale. Syncing from the source means the agents know what the team knows.
Chunking: the most underrated decision
Models don’t search whole documents; they search chunks. How you split content decides what can be found.
- Split on structure, not character counts. Break at headings and sections so each chunk is about one thing. A chunk that starts halfway through one topic and ends halfway through the next matches nothing well.
- Keep chunks moderate. A few hundred words is a good starting point: big enough to carry context, small enough to stay focused.
- Overlap slightly so an idea that spans a boundary isn’t lost.
- Carry the context with the chunk. Each one is stored with its document title and section headings, so a chunk that says “the second step” still makes sense on its own.
Metadata: what makes results trustworthy
Every chunk is stored with metadata alongside its text and embedding:
- the source and a link back to the original,
- the author and last updated date,
- the space, project or folder it came from,
- and, most importantly, who is allowed to see it.
Metadata is what lets the system filter by project, prefer recent content and cite sources. It’s also what keeps the system secure, which comes up again below.
Embeddings: local, for privacy
An embedding model turns text into vectors so that passages with similar meaning end up close together. I run the embedding model locally, the same way I run my agents on local models. Every document in the knowledge base passes through the embedding model, so keeping that step on my own machine means the project’s documents never go to an outside provider just to be indexed.
One rule matters here: the same embedding model must be used for indexing and for queries. If you change models, re-embed everything. Vectors from different models aren’t comparable.
The vector database: Postgres with pgvector
There are plenty of dedicated vector databases, and they’re good. I chose PostgreSQL with the pgvector extension, for practical reasons:
- One database instead of two. Chunks, metadata, permissions and vectors live together, so there’s nothing to keep in sync.
- Filters are just SQL. “Only this project, only documents this user can see, updated in the last year” is an ordinary query.
- Fast similarity search with an approximate nearest-neighbor index (HNSW) that stays quick as the collection grows.
- Boring in the best way. Backups, monitoring and access control work like any other Postgres database.
A dedicated vector database starts to make sense at very large scale or with specialized needs. For a project’s knowledge base, Postgres goes a long way.
Hybrid search: meaning and keywords together
Pure vector search is great at meaning and surprisingly bad at exact terms. Search for an error code, a ticket number or a product name, and semantic similarity can miss the one document that contains it.
So retrieval runs two searches at once: a vector search for meaning and a keyword search for exact terms. The results are merged with a simple ranking method that rewards documents scoring well in both. Then a reranker, a small model that reads the question and each candidate passage together, reorders the shortlist so the best few go to the language model.
Hybrid search plus reranking was the single biggest improvement in answer quality.
Permissions: never retrieve what the user can’t see
This is the part that’s easy to get wrong. If a RAG system indexes everything and then retrieves for anyone, it becomes a way around every access control the company has.
The rule is simple: filter by permission at retrieval time. Every query carries the identity of the person or agent asking, and only chunks they’re allowed to see are searched. An agent working for the support team searches support material, not finance or HR. The model never sees content it shouldn’t, so it can never leak it.
Freshness: keep the index current
Knowledge goes stale fast. The index stays current through incremental sync: when a page, ticket or document changes, only that content is re-chunked and re-embedded, and deleted content is removed. A periodic full check catches anything the incremental sync missed.
Stale answers are worse than no answers, because people trust them.
Answers with citations
The model is instructed to answer only from the retrieved passages and to cite its sources with links back to the originals. If the passages don’t contain the answer, it says so instead of guessing.
Citations do two jobs: people can check the answer in one click, and wrong answers become easy to trace back to the passage that caused them.
Measuring it
A RAG system you can’t measure is one you can’t improve. I keep a set of real questions with known good answers and run it whenever anything changes, whether that’s chunking, the embedding model or the reranker:
- Retrieval: did the right passages come back in the top results?
- Faithfulness: did the answer stick to what the passages actually say?
- Usefulness: did it actually answer the question?
When a change improves one and hurts another, the numbers show it before the agents do.
What I’d tell another team
- Chunk on structure. It matters more than which vector database you pick.
- Use hybrid search and a reranker. Meaning alone misses exact terms.
- Enforce permissions at retrieval time. Otherwise RAG becomes a data leak.
- Keep it fresh and cite everything. Trust comes from current answers people can check.
- Start with Postgres. One database is easier to run than two, and it scales further than you’d expect.