In many industries, a 90% accuracy rate might be considered a success for an AI pilot. In the mortgage industry, such a margin of error is unacceptable. Rigorous testing is the bedrock of AI implementation, and that testing must account for a staggering variety of real-world scenarios.
To ensure reliability, Curran advocates for the use of "ground truth" databases—repositories containing hundreds or thousands of loans that have already undergone exhaustive manual review. These databases serve as the gold standard against which AI performance is measured. By running an AI model against these verified loans, developers can see exactly where the technology aligns with human judgment and where it falters.
The complexity of income verification provides a perfect example of why this variety is necessary. A standard AI might easily process a straightforward W-2 and a single pay stub for a salaried employee. However, the mortgage world is rarely that simple. A single borrower might have six different income sources, complex tax returns, and K-1 forms from various business interests. A truly effective AI must be able to handle all these variations with the same level of consistency and accuracy as a senior underwriter.
Furthermore, the testing process does not end once a model is deployed. The nature of AI—particularly models hosted by third-party providers—is that they are constantly being updated and refined. This necessitates "regression testing," a continuous process of ensuring that new updates to the model do not break existing prompts or lead to unexpected changes in output. If a model’s underlying logic shifts, the results it produces for a specific loan type could change, potentially introducing errors into a live workflow.
Beyond technical accuracy, bias testing has emerged as a critical component of the development cycle. AI models can inadvertently introduce bias based on proxy data or unexpected correlations within the training sets. Without strict controls and constant monitoring of what the model is and is not allowed to consider, lenders risk running afoul of fair lending laws and ethical standards.
Efficiency has limits when judgment is involved
The primary driver of AI adoption is efficiency—the ability to do more with less. In the mortgage lifecycle, there are numerous areas where AI can meaningfully reduce manual labor. Data entry is the most obvious candidate. Manually keying information into a Loan Origination System (LOS), Point of Sale (POS), or servicing system is a labor-intensive process prone to human error. AI’s ability to accurately extract and port this data can streamline operations and allow staff to focus on higher-value tasks.
However, Curran draws a firm line when it comes to the final credit decision. While AI can support the decision-making process by organizing data and flagging potential issues, it lacks the nuanced judgment required to sign off on a loan.
"I don’t believe AI should make the final credit decision," Curran asserts. Credit decisions are not purely mathematical; they often involve "compensating factors"—contextual details that might explain a dip in credit score or a gap in employment that a rigid algorithm might miss. Guidelines vary between investors, and there are often circumstances that require a human to weigh risk against the specific story of the borrower. In these instances, AI serves as an incredibly powerful assistant, but the final accountability must remain with a human professional.
AI could expose what loan sampling misses
