I implemented RAG-powered vector memory using Pinecone for embeddings search and mem0 for structured agent memory, combined with structured data extraction from conversations - improving AI agent response accuracy by approximately 35% on domain-specific queries compared to baseline prompting alone.
When an AI agent tells a caller the wrong refund policy, the problem is rarely the LLM - it is missing context. At UseCortana, I built a RAG-powered memory layer using Pinecone and mem0 that improved agent response accuracy by ~35% on domain-specific queries compared to baseline prompting.
This post covers the architecture, latency tricks for voice, and how RAG fits alongside Temporal workflows and LiveKit voice infrastructure.
Why baseline prompting fails in production
Foundation models know general knowledge. They do not know your client's return policy, insurance rider exclusions, or resort check-in times. Without retrieval, agents either hallucinate or deflect - both destroy conversion.
RAG solves this by searching a verified knowledge base at inference time and injecting the top chunks into the LLM prompt.
The memory stack
| Layer | Tool | Purpose |
|---|---|---|
| Document retrieval | Pinecone | Product docs, FAQs, SOPs as embeddings |
| Structured memory | mem0 | Caller preferences, past interactions, extracted facts |
| Extraction | LLM function calling | Pull structured fields from conversation transcripts |
| Orchestration | Temporal | Async indexing, re-embedding on doc updates |
Pinecone: document RAG
Client knowledge bases are chunked, embedded (OpenAI text-embedding-3-small or similar), and stored in Pinecone namespaces per tenant. At query time:
- Embed the user utterance (or partial transcript in voice).
- Vector search with metadata filters (product line, region, doc type).
- Rerank top-10 to top-3 with a cross-encoder or lightweight reranker.
- Inject chunks into the system prompt with source IDs for traceability.
mem0: structured agent memory
Vector search alone misses relational facts: "this caller prefers morning appointments" or "already quoted on plan B." mem0 stores extracted entities and relationships across sessions so agents personalize without re-asking.
Structured extraction runs as a post-turn activity - the LLM outputs JSON fields, mem0 persists them, and the next turn's prompt includes relevant memory entries.
RAG on the voice critical path
Voice agents have a ~500ms perceived latency budget. Retrieval cannot block the STT-LLM-TTS loop serially.
Production pattern from 2026 voice RAG guides:
- Trigger on partial transcripts at 70–80% confidence - do not wait for end-of-turn.
- Run retrieval in parallel with LLM time-to-first-token warm-up.
- Cap chunks at 3 with tight token budgets.
- Cache frequent queries per tenant (business hours, pricing FAQs).
This is the same latency discipline I applied building VoiceCake's multi-channel agents and the HVAC calling agent.
Structured data extraction
Beyond Q&A retrieval, agents extract structured data from conversations:
- Appointment preferences → CRM fields
- Property attributes → insurance application pre-fill (see Metropolitan Insurance automation)
- Guest party size and dates → resort booking system
Extraction uses LLM function calling with Zod-validated schemas - invalid extractions retry once, then fall back to clarifying questions.
Evaluation and anti-hallucination
RAG without eval is hope. I measure:
- Faithfulness - does the answer match retrieved chunks?
- Context relevance - did retrieval return the right documents?
- Citation correctness - are source IDs accurate in traces?
Run these on every model or embedding model change. The evaluation discipline mirrors my RLHF work on Gemini at Turing - systematic rubrics, not gut feel.
Results
- ~35% improvement in domain-specific query accuracy vs baseline prompting.
- Fewer "I don't know" deflections on answerable questions.
- Multi-tenant namespace isolation - no cross-client data leakage.
- Retrieval traces in OpenTelemetry for debugging bad answers.
When RAG is not enough
RAG retrieves static documents. It does not replace:
- Tool use for live data (inventory, calendar availability, account balances).
- Temporal workflows for multi-step transactions.
- Human escalation for high-stakes decisions.
Use RAG for knowledge grounding, tools for live systems, workflows for durability.
Related work
Common questions
What is RAG for AI agents?
Retrieval-Augmented Generation (RAG) lets agents search a knowledge base at inference time and inject relevant chunks into the LLM context - grounding responses in verified documents instead of model parametric memory.
Why use Pinecone for agent memory?
Pinecone provides managed vector search with low-latency queries, metadata filtering, and namespace isolation - ideal for multi-tenant SaaS agents where each client has separate knowledge bases.
What is mem0 and how does it differ from Pinecone?
mem0 focuses on structured agent memory - user preferences, past interactions, and extracted facts - while Pinecone handles document retrieval. They complement each other in a full memory stack.
How do you keep RAG fast enough for voice agents?
Trigger retrieval on stable partial transcripts, run vector search in parallel with LLM warm-up, rerank to top-3 chunks, and cap context size. Target under 100ms for retrieval on the critical voice path.
How much does RAG improve agent accuracy?
In my production deployment, RAG with structured extraction improved domain-specific query accuracy by approximately 35% versus baseline prompting. Gains depend on knowledge base quality and eval rigor.