Gen AI-Assisted Copilot for Risk Analysts in RS IntelliEdge™

Abstract

As digital payments surge, the complexity of fraud continues to escalate, demanding more sophisticated tools to support risk analysts. This white paper introduces the development of a Gen AI-assisted copilot integrated within the RS IntelliEdge™ fraud detection solution, aimed at optimizing fraud alert investigation. The copilot leverages a Retrieval-Augmented Generation (RAG) model that combines a Random Forest (RF) machine learning model as the retriever and a fine-tuned Large Language Model (LLM) for generating contextual responses. By classifying fraud alerts, measuring similarity with historical data, and predicting the next action, the system automates decision-making while generating natural language recommendations for analysts. This approach significantly reduces manual workload, increases operational efficiency, and allows risk analysts to focus on critical, high-priority cases. The result is a robust system that enhances the speed and accuracy of fraud detection, investigation, and resolution, ultimately improving overall operational effectiveness.

Introduction

The rise of fraud in digital payments necessitates the development of efficient and intelligent tools to aid risk analysts in fraud detection. As fraud becomes more complex, the need for sophisticated tools that can predict fraud, suggest appropriate actions, and provide intelligent recommendations has grown.

This white paper introduces a Gen AI-assisted copilot for risk analysts integrated into the RS IntelliEdge™ fraud detection solution. The copilot employs a Retrieval-Augmented Generation (RAG) model combining a Random Forest machine learning model as the retriever and a fine-tuned Large Language Model (LLM). model. By leveraging this two-step approach, the copilot retrieves relevant historical data using the RF model and generates contextual responses with a fine-tuned LLM. This integration enables the system to predict the next best action based on past fraud patterns while providing well-structured, natural language recommendations to guide risk analysts in managing cases efficiently.

Challenge

The RS IntelliEdge™ fraud detection solution continuously monitors transactions and generates alerts when potential fraud is detected. However, determining the appropriate next steps requires significant human effort, as risk analysts must evaluate the alert to take the next action.

Risk analysts investigate alerts by systematically assessing suspicious activities flagged by automated systems to determine whether they represent a genuine risk, fraud, or a false positive. The investigation process involves reviewing alert details, analysing transaction patterns, verifying customer or entity information, checking for related alerts, and gathering additional information if needed. Based on the findings, the next steps include escalating the case if a significant risk is detected, dismissing it if deemed a false positive, or flagging the account for continued monitoring in cases of moderate  suspicion. The steps taken depend on the severity of the alert, the risk involved, and the findings during the investigation. Fraud analysts spend significant time sifting through data, cross-referencing information, and generating reports, limiting their ability to focus on high-priority cases.

The lack of automation and intelligent assistance in the current system makes it difficult for analysts to handle high volumes of cases efficiently, potentially leading to missed or delayed responses to genuine fraud incidents. Moreover, the repetitive nature of tasks like data entry, alert triaging, and report generation contributes to operational inefficiencies.

Method

To address these challenges, a Co-Pilot has been developed to aid Risk analysts with the alert investigation in the case management system workflow. This AI-driven co-pilot will assist fraud analysts by providing real-time insights and offering intelligent recommendations for alert investigation. The co-pilot will streamline the case management process, enabling faster detection, investigation, and resolution of fraudulent activities while reducing manual effort and improving overall efficiency.

We are utilizing a variation of the Retrieval-Augmented Generation (RAG) model in conjunction with a fine-tuned Large Language Model (LLM) to deliver contextual and accurate responses. In this setup, the retrieval process is handled by a Random Forest (RF) model, which serves as the retriever within the RAG framework. The RF model is refreshed daily with data from the past X days to ensure that it accurately reflects recent patterns and violations. The fine-tuned LLM generates responses by leveraging the augmented prompt, which combines historical data retrieved by the RF model with real-time insights. Fine-tuning of the LLM is only required when new violation types or action steps are introduced, ensuring that the system remains up-to-date and relevant while minimizing unnecessary fine-tuning operations.

To streamline this process, we introduce a Gen AI-driven copilot that automates the decision-making process by:

1. Classifying the violation to determine whether it was model-based or rule-based, and if rule-based, determining the specific rule type.
2. Calculating the percentage of similarity with past fraud alerts.
3. Using this information to predict the optimal ‘Next-Action’ using a Random Forest model.
4. Providing a natural language recommendation to assist the risk analyst, leveraging a fine-tuned Large Language Model (DistilGPT-2).

The Workflow: Automating Risk Analyst Assistance

This process flow diagram outlines the core components of the Gen AI system, divided into two key workflows represented by blue and green boxes:

1. Blue Boxes: RF Model Refresh (Retriever)

The blue boxes depict the daily process of refreshing the Random Forest (RF) model, which acts as the system's retriever. To maintain accuracy and relevance, the RF model is updated with data from the past X days. This ensures that the retriever has the most recent information available, enhancing the retrieval process and keeping the system aligned with any new trends or patterns in violation types and actions.

2. Green Boxes: Online Flow with Fine-Tuned LLM (Augmented Prompt)

The green boxes represent the online flow, which utilizes a fine-tuned Large Language Model (LLM). The LLM is provided with an augmented prompt generated from the RF model's retrieved data to offer more contextual and insightful responses. By combining the retriever's latest data with the LLM's advanced language generation capabilities, the system delivers highly relevant and precise results in real time.

This combination of daily RF model refreshes and the augmented prompts used by the fine-tuned LLM ensures that the system remains both data-driven and responsive, providing timely and context-rich insights.

Step 1: Identifying Violation Type

When a new alert is generated, the first task is to identify the ‘Violation_Type’ as depicted in ① of the above flow. In this step, the co-pilot determines the highest priority rule or model that was involved in the generation of the alert. The violation type can have 4 values:

1. List-rule which are hotlist rules, flagging transactions involving known fraudulent entities or behaviours.
2. Transaction-rule which are filters based on transaction thresholds, involving analysis of specific threshold values and transaction patterns.
3. Models that are driven by machine learning models, where ML scores guide decisions.
4. Profile which involves the utilization of profile data to evaluate transaction risk.

Step 2: Calculating Similarity

Next, the system calculates the ‘Similarity_Percentage’ as depicted in task ②, which quantifies how similar the new alert is to previous fraud alerts generated within the past X days. To calculate this similarity score, the past X days of fraud alerts are accessed to see what percentage of them can be identified as a significant (greater than 50%) match as depicted in task ③.

We have used cosine similarity with Natural Language Processing (NLP) in Python to achieve a percentage match with past fraud data. This method involved representing the description of the violation type as numerical vectors and measuring its similarity with past fraud alerts using cosine similarity. By applying this approach, we have been able to efficiently compare the current alert data with historical fraud alerts, indicating the similarity between the current case summary and known fraud scenarios.

A high match percentage suggests a strong correlation with historical fraud patterns, signalling a potentially fraudulent transaction that requires immediate attention. Conversely, a low match percentage may indicate a lower likelihood of fraud, allowing the case to be processed more efficiently with minimal manual intervention. This similarity score helps the Random Forest model predict the optimal next step.

Step 3: Predicting the Next-Action using Random Forest

Once the ‘Violation_Type’ and ‘Similarity_Percentage’ are determined, they are fed into a pre-trained Random Forest model as depicted in ④ to predict the ‘Next-Action’ as shown in ⑤.

The Random Forest model, trained on historical fraud case data, considers the violation type and the similarity score to predict actions such as:

1. Mark the transaction as Genuine
2. Close the Case and Monitor the Account
3. Request Additional Verification from the Customer
4. Escalate the Case to a Specialized Team
5. Tag the Transaction as Fraud

This prediction aids in automating the decision-making process for risk analysts.

Step 4: Generating Recommendations using DistilGPT-2

After predicting the ‘Next-Action’, the copilot takes the following parameters as input as depicted in ⑥:

1. Violation_Type
2. Similarity_Percentage
3. Next-Action

These parameters are fed into a fine-tuned version of the DistilGPT-2 language model. The goal of the model is to generate a natural language recommendation that assists the risk analyst in understanding the reasoning behind the predicted next step and suggests further actions as illustrated in ⑦. The first prompt in each recommendation is designed in a way that aligns with the analyst's expectations, by being hardcoded with respect to the violation type and similarity percentage, ensuring that the fine-tuned DistilGPT-2 model generates responses in a consistent and expected format.

DistilGPT-2 was chosen for its balance of performance, efficiency, and flexibility. It enables fast, accurate, and cost-effective recommendations tailored to the specific needs of risk analysts in the RS IntelliEdge™ solution. The model's smaller size, high-speed inference, and strong language generation capabilities make it an excellent fit for generating fraud case recommendations, driving operational efficiency, and assisting in the resolution of fraud cases.

Results

Fine-Tuning DistilGPT-2

Fine-tuning DistilGPT-2 involved taking the pre-trained DistilGPT-2 model and training it on a specific dataset to adapt it for a particular task, such as generating recommendations in a fraud detection system. The process began by preparing a custom dataset that included the input parameters and desired output (in this case, ‘Violation_Type’, ‘Similarity_Percentage’, ‘Next-Action’, and the corresponding recommendation). The dataset was tokenized using the same tokenizer as the pre-trained model to ensure compatibility and padding/truncation was applied to ensure consistent sequence lengths. During fine-tuning, the model’s weights were adjusted based on the new data using supervised learning, through gradient descent. This allowed the model to learn the nuances of the new task while retaining the general language understanding from its pre-training.

The fine-tuning process leveraged a small learning rate and a relatively short number of training epochs to avoid overfitting while maintaining high-quality output generation. Once the model was fine-tuned, it was validated on unseen data to ensure it could generate accurate and contextually appropriate recommendations. The result was a specialized version of DistilGPT-2, fine-tuned to provide actionable recommendations based on the specific input parameters from the previous steps.

The model went through the data three times (epochs) to improve accuracy, and a weight decay of 0.01 was applied to prevent overfitting, helping the model generalize well to new data. The model was fine-tuned by evaluating performance at the end of each training round (epoch). A small learning rate of 2e-5 was used to ensure gradual adjustments, with two data samples processed at a time during both training and evaluation.

The results from the training process are as follows:

Training Loss: The substantial drop from 0.865600 to 0.005400 indicated that the model learned efficiently across the epochs and could perform with very low error on the training data by the third epoch.

Validation Loss: The consistent decrease from 0.005962 to 0.002772 showed that the model had improved its generalization capabilities and not overfitting the training data.

Based on the above, we inferred that the model’s performance had improved significantly across the epochs. The large drop in training loss and the continued decrease in validation loss suggested the model had learned effectively and generalizing well to new data. This indicated a well-performing model by the end of the third epoch, with no signs of overfitting.

Creating the data for Fine-Tuning DistilGPT-2

To fine-tune DistilGPT-2, we used a dataset with the following columns:

1. Violation_Type
2. Similarity_Percentage
3. Next-Action
4. Recommendation: A 3-5 sentence recommendation tailored to the input parameters, guiding the risk analyst on the appropriate next steps.

The fine-tuning process was done by tokenizing the ‘Recommendation’ field and training the model using the inputs (Violation_Type, Similarity_Percentage, Next-Action) as context.

For example, a sample recommendation that was used to train the LLM for a Violation_Type of ‘Model’ and Similarity_Percentage of 85% for which the Random Forest model predicted the next action as ‘Tag the Transaction as Fraud’ was:

"A bank risk analyst is investigating a case with an 85% match to cases marked as fraud in the past 30 days. The transaction was flagged by a model rule, suggesting a high likelihood of fraud based on historical data. The analyst should tag the transaction as fraud and initiate immediate response actions to mitigate potential losses."

Samples corresponding to each violation type were created with varying percentages of similarity to get the corresponding values of ‘Next action’ from the Random Forest model which were then fed to the fine-tuned LLM to generate recommendations.

Test Data Creation for LLM Fine-Tuning

The test dataset was created by sampling historical fraud cases and generating additional variations to capture a wide range of fraud scenarios and corresponding recommendations. Each test entry in the dataset was based on combinations of:

1. Violation_Type: selected randomly from 'List-rule', 'Transaction-rule', 'Model', 'Profile'.
2. Similarity_Percentage: A range from 0% to 100%, divided into categories as described above.
3. Next-Action: Predicted by the Random Forest model.

The goal of the test dataset was to simulate a variety of real-world scenarios that a risk analyst would encounter, ensuring the language model could handle diverse situations and recommend appropriate actions.

Model Testing

The fine-tuned DistilGPT-2 model was tested with unseen data to evaluate its ability to:

• Generate accurate and relevant recommendations.
• Adhere to the structure and style of the training data.
• Provide actionable insights that risk analysts can easily understand and implement.

We evaluated the model based on its ability to produce clear and coherent recommendations aligned with the predicted ‘Next-Action’.

Conclusion

The AI-assisted copilot for the RS IntelliEdge™ fraud detection solution is an innovative tool that combines the power of traditional machine learning models with modern language models to streamline the decision-making process for risk analysts. By predicting the next action using a Random Forest model and generating actionable recommendations through a fine-tuned DistilGPT-2 model, the copilot offers significant operational efficiency gains while ensuring that alerts are addressed in a timely and accurate manner.

About RS IntelliEdge™

RS IntelliEdge™ is a comprehensive fraud detection and risk management solution designed to address the dynamic challenges of digital payments. Equipped with advanced machine learning models, rule engines, and AI-driven capabilities, it ensures real-time detection and prevention of fraudulent activities. The platform seamlessly integrates with multiple payment methods in the
Page 8 of 8 existing financial ecosystems, offering scalability, flexibility, and high precision in identifying suspicious transactions.

The integration of the Gen AI-assisted copilot further enhances RS IntelliEdge™ by automating alert investigation and decision-making processes. This innovation reduces manual workloads, improves operational efficiency, and empowers risk analysts to focus on high-priority cases. With its state-of-the-art architecture and commitment to continuous improvement, RS IntelliEdge™ remains at the forefront of fraud prevention technology, delivering security, reliability, and trust in digital transactions.