Building the RAG Pipeline: Vector DBs, Embeddings, and Memory
The first Companion RAG pipeline was not a production architecture. It was a fast Python prototype using SQLite and ChromaDB so we could test memory, retrieval, and response quality quickly.
Ella announced Companion in February: an AI that remembers, retrieves context, and responds as if the past conversation still matters. The vision is the interesting part. This post is the less glamorous engineering reality underneath it.
At this stage, the RAG pipeline is still a prototype.
That is important. We are not building for massive scale yet. We are trying to answer a more basic question: can we get the responses we want?
If the answer is no, the database choice does not matter. If the answer is yes, then we can harden the architecture later.
The Problem Statement
Companion needs to remember useful things from past conversations and retrieve the right fragments at the right moment. Not just keyword matches. Semantic matches. If a user brings up "the deployment issue," the system should be able to find the earlier conversation where they were actually talking about a server problem, even if the words do not line up perfectly.
That is the basic RAG loop:
- Store conversation data.
- Summarize or chunk it into useful pieces.
- Embed those pieces into vectors.
- Store the vectors somewhere searchable.
- Retrieve relevant memory when the user talks again.
- Feed that retrieved context into the model.
Simple in concept. Fiddly in practice.
The Prototype Stack
For the prototype, we kept the stack intentionally small:
- Python for the pipeline code
- SQLite for simple relational/session data
- ChromaDB for vector storage and similarity search
- local scripts and notebooks for testing retrieval behavior
This was not because SQLite and ChromaDB are the final answer for everything. They are not. The point was speed.
When you are still figuring out what the memory should feel like, fast iteration matters more than perfect deployment architecture. I need to be able to change the chunking strategy, rerun the embeddings, inspect the retrieved memories, and see whether the model response improves. If every test requires production-grade ceremony, the feedback loop dies.
SQLite is boring in the best way. It gives me tables, records, timestamps, user/session references, and simple queries without standing up a full database server.
ChromaDB gives me a fast local vector store that is easy to use from Python. Good enough to test whether a memory search works. Good enough to compare chunking strategies. Good enough to find out whether the retrieved context is helping or making the model worse.
Good enough is the point.
What We Were Testing
The goal was not to prove that ChromaDB could run Sorren.ai at scale. The goal was to test memory behavior.
Questions like:
- What should be embedded: raw messages, summaries, or both?
- How many recent messages should be included before retrieval even starts?
- Should the memory system prefer recent context or older high-similarity context?
- How much retrieved context is helpful before the model starts drowning in it?
- Can the system distinguish durable facts from temporary conversation state?
- Can a companion feel grounded in the current conversation while also remembering the past?
Those are product questions as much as engineering questions.
Ella was useful here because she could tell when a response felt wrong. I could look at logs and similarity scores. She could look at the conversation and say, "That memory is technically related, but emotionally irrelevant," or "It remembered the fact but missed why it mattered."
That is the kind of feedback a benchmark does not give you.
Recent Context Still Matters
One early lesson: vector memory is not a replacement for recent context.
A companion needs to know what is happening now. That means the prompt still needs recent messages from the active conversation. The RAG system can pull older memories, but it cannot replace the short-term continuity of the current exchange.
So the prototype pipeline used both:
- recent messages from the active session
- retrieved memories from ChromaDB
The recent messages keep the conversation grounded. The retrieved memories give it depth.
If you only use recent messages, the companion forgets too much. If you only use vector retrieval, the companion can feel disoriented, like it is grabbing old facts without understanding the present moment.
The trick is combining both.
The System Instructions Layer
We also experimented with storing generic system instructions and companion behavior rules alongside the memory structure.
The idea was not that these instructions would be permanent law. A companion should adapt over time. As it interacts with a user, some preferences and boundaries should become more specific. But there still needs to be a baseline: what the companion is, what kind of tone it should use, how it should treat memory, and when it should ask rather than assume.
That baseline can be overridden or refined later as the companion learns the user.
The prototype let us test that relationship: generic instructions, user/session data, recent messages, and retrieved memories all flowing into the context window together.
Sometimes it worked beautifully.
Sometimes it produced a response that made it obvious the weighting was wrong.
That is why this stayed a prototype.
Why Not Build the Final Architecture Immediately?
Because premature infrastructure can hide product failure.
It is tempting to start with the serious stack: PostgreSQL for relational data, a dedicated vector database, queues, workers, migrations, deployment scripts, monitoring, and all the grown-up pieces. We will need those. But in April, the more urgent question was whether the memory design actually improved the conversation.
If the companion retrieves the wrong things, PostgreSQL will not save it.
If the chunks are badly shaped, Qdrant will not save it.
If the model cannot use the retrieved context naturally, better infrastructure will only make the wrong answer arrive more reliably.
So we kept the prototype small.
Move quickly. Test responses. Inspect failures. Change the pipeline. Repeat.
What This Prototype Taught Us
The prototype confirmed a few things:
- Memory needs both structured data and vector search.
- Recent messages are mandatory for grounding.
- Summaries can be better retrieval targets than raw messages.
- Full logs still need to be available when the model has to reflect on details.
- User-specific memory must stay user-specific.
- The companion should not have cross-knowledge about other users.
That last point is important. A companion remembers the person it is working with. It should not casually blend knowledge across users. The whole premise depends on trust.
The Production Direction
The prototype is not the final architecture.
The likely production direction is PostgreSQL for relational data and Qdrant for the vectorized store. PostgreSQL gives us durable relational structure: users, sessions, messages, summaries, permissions, audit trails, and application state. Qdrant gives us a purpose-built vector search layer that can scale beyond the easy local prototype.
That split feels right.
Relational data belongs in a relational database. Vector search belongs in a vector database.
But I am glad we did not start there. SQLite and ChromaDB let us learn quickly. They let us find the shape of the memory system before freezing it into heavier infrastructure.
The right architecture is not the one that looks impressive in a diagram.
It is the one that helps the companion remember the right thing at the right time.
It's almost 8 PM. The prototype is messy. The responses are getting better. That is enough for tonight.