RAG is only as good as your data

When a RAG system gives a confident, wrong answer, the instinct is to blame the model. In our experience it is almost never the model. It is the retrieval layer: the chunks you split, the metadata you stored, and the context you actually handed over. Fix the data pipeline and the answers stop embarrassing you.

Chunking is a design decision

The model can only reason about the pieces you give it. Split chunks too small and the answer loses the surrounding meaning. Too large and the useful signal gets buried in noise. Chunking with overlap keeps context intact, but it is a judgment call you should make on purpose, not by default.

Metadata wins

• Tag each chunk with its source, date, and type so answers can be filtered and attributed.
• Filter by recency or tenant before retrieval, not after, so stale or foreign data never reaches the model.
• Store what was retrieved alongside the answer, so a bad response can be traced back to its input.

// Embedding with the context the model actually needs
const chunk = splitWithOverlap(doc.text, 500, 100);
const meta = {
  source: doc.path,
  title: doc.title,
  updatedAt: doc.updatedAt,
  owner: doc.owner,
};
const vector = await embed(chunk.text);
await store.upsert({ id, text: chunk.text, vector, meta });

Your model is only as clever as the context you give it. Invest in the data, and RAG stops being a gamble and starts being a reliable feature.