Preparing the next answer while the conversation continues

A spoken conversation creates opportunities to prepare. As a user discusses an integration or works through a setup question, nearby documentation may become relevant to the next turn. Retrieval can use that conversational momentum to gather context before the responder needs it.

VoiceAgentRAG implements this idea with two cooperating agents. The architecture places predictive retrieval in the background and gives the foreground responder a semantic cache of document chunks. Their shared conversation stream coordinates the work.

Let the background agent read ahead

The Slow Thinker agent monitors recent conversation turns, retrieves material for the current utterance, and predicts likely follow-up topics. It expresses those predictions as document-style descriptions, retrieves matching chunks, and places the results in a FAISS-backed cache.

The cache stores each chunk with its document embedding, relevance information, and an expiry time. This gives a future query a semantic search over the material already collected. Near-duplicate handling and eviction keep that working set organized as the conversation changes.

Keep the response path explicit

For each query, the Fast Talker embeds the input and searches the cache. A hit supplies context for response generation. A miss triggers direct retrieval, populates the cache with the result, and sends a priority event to the background agent.

This fallback makes the first question on a topic useful preparation for subsequent turns. Query embedding, document retrieval, and response generation remain distinct stages, so their individual latency can be measured. That separation gives an engineer a clear view of where anticipation changes the response pipeline.

Measure the benefit on real network calls

The study uses a synthetic enterprise CRM knowledge base containing 12 documents and 76 chunks. Its evaluation runs 200 queries across ten conversation scenarios against Qdrant Cloud. The dual-agent system records a 75% overall cache hit rate.

On cache-hit queries, retrieval averages 0.35 ms, compared with 110.4 ms for the direct vector-store lookup. The reported comparison measures the retrieval component using real network round trips. Cache-hit rates also vary with the conversation: a sustained topic gives the background agent a more predictable neighborhood of documents to prepare.

For conversational system design, the useful contribution is a concrete coordination pattern. A background process gathers likely evidence, an interactive process consumes it, and explicit events connect successful lookups and cache misses to the next round of preparation.

Research by Jielin Qiu, Haolin Chen, and collaborators at Salesforce AI Research.

Source: VoiceAgentRAG: Solving the RAG Latency Bottleneck in Real-Time Voice Agents Using Dual-Agent Architectures.