Enhancing Financial Recovery Audits with NextGen AI 

In this case study, Mosaic enhanced financial recovery audits with NextGen AI by developing an intelligent search tool for financial processes, using LLMs to aid in compliance during email audits.

Introduction 

In the financial services sector, recovery audits and compliance processes are critical for combating profit leakage and ensuring contractual adherence. A significant portion of this work involves sifting through vast amounts of unstructured email data to identify specific contractual terms, rebates, and promotional materials. According to Deloitte, over 80% of enterprise data is unstructured, such as emails, documents, and social media posts, which presents a significant challenge in data management and analytics. IDC reports that financial services firms are expected to manage 30% more data year-over-year, with a significant portion being unstructured. This growth is driven by increasing regulatory requirements and the need for more comprehensive audit trails. 

Traditional keyword and metadata searching methods are often inadequate, leading to inefficiencies and increased costs. McKinsey found that employees spend nearly 20% of their workweek searching for and gathering information, leading to significant productivity losses. In the context of recovery audits, this inefficiency can lead to missed opportunities for revenue recovery and increased operational costs. Specifically, the financial services industry faces up to $26 billion in lost revenue annually due to inefficiencies in data management and retrieval processes, as reported by Accenture. 

The implementation of advanced AI-driven tools such as Natural Language Processing (NLP) and Large Language Models (LLMs) is becoming increasingly essential to address these challenges. NLP and LLMs can transform the way financial services firms manage and retrieve unstructured data, making the process more efficient and accurate. This is where enhanced financial recovery audits with NextGen AI come into play.

Problem 

A leading financial services firm specializing in recovery audits and compliance faced significant challenges in its email audit process. A large part of their business involved auditing past emails to identify specific previous rebates or promotional materials. This was traditionally done using simple keyword and metadata searching, which proved time-consuming, error-prone, and inefficient. The need to search and return critical insights from millions of emails, with information often buried deep within the text, made the audit recovery process costly and labor-intensive.

Additionally, the firm faced difficulties in general question answering, such as identifying key financial stakeholders, transaction details, and compliance-related communications within their vast email archives, further complicating the recovery audit process. 

Solution 

Mosaic Data Science developed a sophisticated intelligent search tool for financial processes, enhancing financial recovery audits with NextGen AI by leveraging modern Large Language Models (LLMs) and Natural Language Processing (NLP) techniques that make up the company’s award-winning Neural Search solution framework. This tool utilized document retrieval and a vectorized database to perform more accurate and specific email searching.

For maximum security and data control, the LLM solution was deployed directly on an in-house server, unlike a cloud service like AWS, Azure, etc. By integrating seamlessly with existing systems, Mosaic ensured the security, flexibility, and avoidance of vendor lock-in. This approach provided the financial firm with complete control over their data and the freedom to select the best AI tools for their unique needs. 

Process for Enhanced Financial Recovery Audits with NextGen AI

Mosaic began the project by facilitating a kickoff meeting to outline objectives and set clear expectations. The company’s dedicated team of data scientists collaborated closely with the firm to gain access to the necessary email data required for developing the Neural Search Prototype. This phase involved collecting a comprehensive dataset of emails and attachments. Each document was meticulously reviewed and parsed to extract annotations and metadata, which were vital for tuning and evaluating the search engine. 

With the data in hand, Mosaic proceeded to develop an end-to-end NLP and LLM search engine prototype. This development phase included refining candidate models and rigorously testing them against a broader range of queries. The goal was to ensure that the models could accurately and efficiently search through vast amounts of unstructured email data. 

A Neo4j graph database served as the backbone of the vector search, where, by design, each customer had their information stored in separate, disconnected instances. These databases stored emails and attachments that had been pre-processed, cleaned, and vectorized – translated into a numeric vector representation of the content using an LLM. Neo4j was chosen because it made “unconventional” data queries more efficient and “natural” for a human end-user using simple, intuitive queries.

Moreover, the graph structure of the database incorporated and displayed contextual relationships extracted from email conversations. Such capabilities enabled possible “meta” searches of the data (such as quickly understanding how many emails certain people had exchanged, etc.). Neo4j offered a scalable and flexible core framework for downstream development and feature enrichment. 

Fig 1. Graphical structure of the underlying graph database. From left to right, the brown node represents an email-domain host node (e.g. truevalue.com), the blue node represents a unique email address (e.g. bob@truevalue.com), the orange node represents the original email source, the pink node is a specific chunk of text indexed for the vector search from a source document, or an attachment, here shown in green. It should be noted that each node contains a number of properties representing additional information associated with that entry, such as a date, time, email address, message IDs, and various other pieces of information from the original .msg or .eml file.    

Once the prototype was developed, the next step was configuring the deployed LLMs in a specific architecture and location that best suited the firm’s infrastructure. This involved setting up a robust and secure environment for the LLMs to operate effectively. The email search prototype was built on top of open-source components using LangChain, a library that integrated document search and LLMs, and LlamaCpp, a library to host and interact with LLMs at the code level.

These components had been heavily modified, customized, and tuned specifically for email search, retrieval, and QA. The specific architecture was adapted from Mosaic’s previous work in Neural Search and was based on a Retrieval Augmented Generation (RAG) embedding retriever model with LLM functionality. 

The underlying concept of natural language search involved transforming any text into a vector representation, encoding the semantic “meaning” of that text into a format understandable to a computer. In this process, a document pipeline first ingested a set of emails and attachments, parsing them into body information, attachment information, and metadata such as sender/receiver, date range, and mail ID. This parsed text was then passed into an embedding model for vectorization and stored in specialized databases, in this case, Neo4j, designed to store vectors and optimize a search against them. 

When a user executed a query in the front end, an embedding retriever vectorized the query and found the documents in the embedding database most similar to the input query based on the similarity between their respective vector embeddings. This was analogous to a simple keyword search, except that it no longer required matching specific words but instead selected results based on similarity in “meaning.” The number of returned documents could be controlled either by a fixed number of results or by assigning a lower bound confidence threshold. In pure similarity search, each document found by the natural language search, after some filtering for relevance and duplicates, was passed directly back to the front-end API. 

The LLM QA was the last step of the search engine and relied on the natural language search as a pre-processor. After the similarity search returned the most relevant documents, they were then passed, together with the user query and an underlying prompt, to a physically hosted LLM as a large block of text. The LLM then used the provided documents as context to answer the user’s questions. The text output of the LLM and source documents were then provided to the front end and user.

It is important to note that no proprietary information was actually stored or learned in the LLM. Instead, the LLM acted as an interpreter for parsing the large block of document text and prompt instructions. This design allowed hot-swapping different models as LLM capabilities evolved. Similarly, LLM QA performance was driven primarily by the LLM context length, which was a measure of how much text could actually be parsed by the LLM in a single query. Larger models might yield more comprehensive results but also require more computational resources for physical hosting. 

Fig 2. Email search architecture using textual embeddings for document search and LLM QA and summarization.   

Additionally, Mosaic developed APIs to facilitate seamless integration of the Neural Search prototype into the firm’s email intelligence tools, ensuring that the new system worked harmoniously with the existing setup. The API allowed end-users to submit natural language queries and access the question-answering and summarization capabilities of the LLM. This was accessible to users on the financial company’s VPN who also possessed the appropriate login information. The similarity search queries and LLM QA were on separate endpoints, providing users the ability to apply the following additional filters: 

• Emails sent to or by specific email addresses 
• Emails exchanged between specific people or groups of people 
• Excluding emails sent or received by specific email addresses 
• Emails exchanged between specific date ranges 
• Emails with specific subject text 
• Emails with a specific character limit (for example, to avoid short emails like “sounds good!” from being returned) 
• Including or excluding text from attachments 
• Only including text from attachments 
• Custom Neo4j queries (for expert users only) 
• The maximum number of documents to return 

Mosaic documented the enhanced financial recovery audits with NextGen AI solution’s architecture and features in detail and provided comprehensive training sessions for the firm’s personnel. This training was essential for ensuring that the staff could effectively use and deploy the Neural Search prototype, maximizing the tool’s potential benefits.  

The final phase of the project involved evaluating the prototype and planning for the future. Mosaic created a detailed report summarizing the search tool’s design and performance. This report included performance statistics and insights gained during the development and testing phases. Furthermore, Mosaic worked with the firm to scope future phases, features, data, and queries, providing detailed documentation for extending the Neural Search prototype’s capabilities. 

Results 

The customer’s document search engine relied on enhanced financial recovery audits with NextGen AI to be capable of performing sophisticated natural language document searches, with numerous filtering and retrieval options to obtain relevant information. The search engine naturally integrated both state-of-the-art natural language document search and the subject matter expertise of the end-user to locate relevant documents quickly, efficiently, and without the need for specific and potentially limited keyword searches. Specifically, the search engine was capable of the following: 

1. Natural language search over the entire document base, with the option to refine those results based on sender email address, receiver email address, conversations between specific email addresses, subject text, document text, date, whether it was in an attachment, and the type of attachment. 
2. Contextual question answering and summarization via an LLM, which could be further tuned and refined similarly to the above. 

These features and functionalities were realized through the Neo4j graph database and a specific selection of LLM models deployed within the broader framework. 

The implementation of the Neural Search Engine significantly benefitted the financial services firm by reducing the time required to locate and extract critical information from emails. Key highlights included: 

• Speed and Efficiency: Searching over hundreds of thousands of emails and attachments with similarity search and LLM summarization, reducing search times to approximately 5 seconds. 
• Enhanced Accuracy: AI-driven search minimized the likelihood of missing vital details, ensuring a more thorough audit recovery process. 
• Cost Efficiency: Improved productivity and reduced operational overhead resulted in lower audit recovery costs and higher profit margins. 
• Hidden Value Unlock: Efficient extraction of critical contract terms and communications helped identify unclaimed rebates and pricing opportunities, maximizing potential revenue recovery. 

Conclusion 

The collaboration between Mosaic Data Science’s world-class team and the financial services firm proved that Neural Search could automate and improve time-consuming and error-prone manual audit processes. The intelligent search tool for financial processes delivered enhanced financial recovery audits with NextGen AI, not only saving time and resources but also improving accuracy and cost-efficiency, unlocking hidden value for the firm.  

Mosaic developed an LLM-powered intelligent search tool for financial processes. This document search tool for chat-like forensic auditing of emails included local deployment capabilities for maximum internal security and data control. While not deployed on portable devices, this resulted in the processes and familiarity with tools required for later local deployment using the requisite software libraries and key architectural differences compared to cloud LLM hosting. 

Mosaic Neural Search revolutionizes search capabilities by leveraging advanced LLMs to provide contextual insights beyond simple document retrieval. This enables better decision-making through unparalleled accuracy and seamless integration with existing systems. Our customizable approach ensures the solution fits perfectly with your unique requirements. Mosaic’s tailored, Next-Gen LLM Neural Search solution for the financial services firm demonstrated how advanced AI technologies could transcend the limitations of off-the-shelf Generative AI tools, providing the firm with a robust and flexible tool to meet their unique needs and objectives.