The costly part wasn’t search. It was reading.

Servicing agents resolve customer questions by searching a governed internal knowledge base. The legacy tools returned lists—sometimes hundreds of results—that an agent had to disambiguate, open, read, and translate into plain language during a live conversation.

Research across product managers, agent managers, and front-line associates showed that knowledge lookup represented roughly 5.6–6% of average handle time. Only about 20% of that time was spent searching. Reading consumed the other 80%. That finding changed the product from a better search engine into an answer engine.

“You have to read a novel before you can actually help a customer about something we already know.” — Servicing associate

Start behind the agent

We split the servicing journey at the point of contact and made a deliberate scope choice: improve the agent-facing experience rather than launch a customer-facing assistant. An agent could verify the answer before it reached a customer, giving the organization a lower-risk place to learn how retrieval, generation, evaluation, and governance needed to work.

That risk posture was a design decision. It reduced technical lift, preserved human accountability, and created a credible path for Capital One’s first live generative-AI customer-servicing use case to ship in 2023.

De-risk one hypothesis at a time

I structured discovery and delivery as a series of phases, each designed to retire a different uncertainty before the program spent money on the next:

  • Problem evidence. Interviews across five lines of business established that information retrieval was a real, measurable bottleneck.
  • Technical feasibility. An embeddings-based semantic-search demo was tested with more than 10 Bank agents.
  • Time-to-answer. I designed a timed study comparing semantic and full-text search across experienced agents, newer agents, and non-agents.
  • Generative risk. The team benchmarked hallucination and early response quality against a human-accuracy baseline.
  • Production readiness. Four organizations aligned around a shared architecture, decision log, evaluation model, and October 2023 launch.

Answer with receipts

The vision was simple: an agent should be able to ask a complex question and receive a trustworthy response with receipts. Every answer would be grounded in approved source material, cite and link to the passages it used, and remain explicitly marked as guidance for the agent—not customer-ready copy.

Two capabilities created the path. Semantic search ranked knowledge by meaning instead of keyword overlap. Generative summary passed the question and retrieved passages to a language model, which synthesized an answer beside its sources. Search could deliver value on its own while the harder generative layer was still being evaluated.

A reusable RAG service inside existing tools

The Agent Assist API orchestrated retrieval-augmented generation beneath the tools agents already used: Empath for Card and Pinpoint for Bank. It received a query, retrieved relevant knowledge chunks, assembled the prompt, generated a summary, and returned the answer with citations. At MVP, semantic search took roughly 350 milliseconds and the summary roughly 2.4 seconds.

The experience placed an Agent Assist Summary above the ranked results, followed by source links and a simple helpful/not-helpful control. Compliance-sensitive content could be labeled no-summarize, forcing the interface to return source material instead of a generated answer. Query, answer, source, and correlation data supported monitoring and evaluation.

The pilot became the roadmap

Early use exposed the right problems. Knowledge written in customer voice could make a summary sound as though the model were speaking directly to the customer. Well-formed questions produced better answers. Retrieved passages felt dense. And a response could be technically correct without being useful. Those were not findings to hide; they became the roadmap for prompt work, re-ranking, formatting, confidence scoring, and evaluation.

From two agents to an enterprise product

Agent Assist launched on October 31, 2023 with two Card agents on live calls and five Bank back-office associates. Four of the five Bank pilot participants preferred it to legacy keyword search. The Card pilot recorded 226 uses over roughly two and a half weeks and gave the team direct, call-by-call feedback.

Semantic search reached all ~10,000 Pinpoint Bank servicing associates by December 2024. The generative summary reached that same population by December 2025, when 13,300 summaries were viewed on day one. Work then began to integrate the capability into the next-generation servicing platform.

Search recall consistently exceeded legacy keyword search. After fine-tuning, re-ranking, and confidence scoring, offline summary accuracy reached roughly 92–94%, with ~75–76% on live user feedback. Early production reads showed Bank Voice average handle time down 3–4% and GURU consult rate down ~9%. The October management review carried an estimated $10.4M idea value against a $25M potential.

What made it work

  • Risk posture came first. Keeping an agent in the loop created a responsible place to learn.
  • Sequencing beat ambition. Search delivered standalone value while the team de-risked generation.
  • Grounding beat cleverness. Citations, governed sources, and no-summarize controls turned hallucination into a managed product risk.
  • The UX findings became the roadmap. Voice, density, query quality, and usefulness shaped later models and interfaces.
  • Reusability compounded. The work established a retrieval-and-generation pattern that could extend beyond one servicing tool.

My role

I led the research and design that took Dewey from a provocation to a product: discovery across five lines of business, a parallel research track with knowledge creators and consumers, the timed experiment that tested semantic search against keyword search, the MVP experience in Empath and Pinpoint, and the demonstration that carried the vision to executives.

The decisions that mattered most—starting behind the agent, sequencing search before summary, and answering with receipts—came out of that research. Delivery was deeply cross-functional across The Lab, Card and Bank servicing, Enterprise Data Science, and the GenAI Platform.