The Math Says Yes, But Should Your AI Model Lend That Money?

Your data scientists are brilliant at mathematics. They can wrangle neural networks and tune hyperparameters in their sleep. But they might be building credit risk models with variables that predict default based on whether someone watches nature documentaries or lets their phone battery die below 20%.

Welcome to the strange new world of AI model risk, where correlation masquerades as causality and explainability is sophisticated guesswork.

The composite risk problem

Naeem Siddiqi, senior advisor for risk and quantitative solutions at SAS, has spent three decades watching banks build models. His insight: “Model risk starts with data risk, and that is probably the biggest risk.” Most banking data, he notes, is dirty — contaminated with approval-decline bias from the start. You’re essentially training models on a skewed reality where you only know the outcomes of customers you’ve already approved.

But after cleaning the data, you hit another major problem: model opacity. Shapley values, partial dependence plots, and other explainability techniques? “They’re speculations,” Siddiqi says flatly. “They speculate as to what’s in the model. They don’t know exactly what’s in the model.”

For CDOs in the finance industry, this creates a governance nightmare. How do you validate what you can’t see? How do you ensure 20 variables buried in a neural network are the right 20 variables for deciding whether to lend someone USD50,000?

The math wiz vs. the banker

Things can also get cultural. Siddiqi highlights one troubling pattern as a result: highly trained data scientists treating modeling as a purely mathematical exercise. “They're very well trained on maths, but there is some lack of awareness around lending and banking as a profession,” he explains. The results are models with statistically powerful variables that make zero business sense.

His solution is radical for most organizations: concurrent validation. Instead of having data scientists work in isolation for 3 months before handing off to validation teams, put everyone in the same room — modelers, risk managers, and validators. “Validation and governance aren’t about where you’re sitting,” Siddiqi argues. “It’s about mindset.”

The resistance is predictable. Modelers worry about independence. But Siddiqi dismisses this: “I could be sitting in the same room as you. If I’m going to challenge you and ask you difficult questions, that’s perfectly fine. I could be in a different country and just approve everything, and that’s still not independent.”

For AI engineering teams, this means breaking down the “exclusive club with a secret knock” mentality. Your models don’t get better in isolation, and they may get weirder.

Alternative data’s hidden bias

Now, let’s add alternative data to the mix. Merchant codes from credit card transactions can reveal shopping behavior that genuinely predicts credit risk. But buried in those same codes? Donations to mosques, churches, and temples, online marijuana purchases in Canada, and the brand of jeans someone buys.

Siddiqi’s team discovered that men who buy flowers at specific times of year are better credit risks. People who watch nature documentaries have lower default rates. In one project, they found that how you charge your cell phone predicts creditworthiness.

“Would you use that to lend money?” he asks. “That’s the bias we’re looking at.”

Essentially, Siddiqi believes correlation without defensible causality is just a discrimination lawsuit waiting to happen. The teams that survive regulatory scrutiny understand they're not building models but are building business logic that must be explainable to regulators, defendable in court, and justifiable to the public.

The difference isn't really about technical or AI sophistication; it’s the willingness to ask “should we?” after the math already answered “can we?”

Siddiqi also believes we need to stop seeing data quality as a theoretical problem. He recalls a project in Hong Kong in the early 2000s where the numbers looked suspicious. So he went down to the street to investigate the data collection process. What he found: kids grabbing pedestrians to sign up for cell phones, collecting just names and addresses. During their breaks, these same kids sat down and filled out the rest of the application forms using pure imagination.

“You gotta do [the legwork] that now,” Siddiqi says. Even with digital data sources like Facebook and other online platforms, you need to trace back to the “actual origination point.” The volume has multiplied, but the fundamental question has not changed: where does this data really come from?

For data engineers scraping together feature sets from terabytes of behavioral data, the question isn’t whether you can find predictive power. You absolutely can. The question is whether you should use it. Is there enough causality to justify the correlation? That’s not a math problem but an intelligence one.

The agentic AI reckoning

Can Agentic AI help, or will it compound the problem? Siddiqi is unequivocal: “I don’t think the trust level is there.”

For example, the Canadian AI and Data Act classifies lending as high-risk, creating hurdles so steep that “in practice, nobody’s actually going to use it because it’s just too much pain.” The fundamental question that every CDO in finance needs to ask: What’s the price of a hallucination when you’re lending money?

Banks use generative AI to draft credit memos today, but humans still read, approve, and decide. That human-in-the-loop isn’t going anywhere soon, at least not for high-stakes decisions. “I don’t see any regulator allowing this for a while yet,” Siddiqi says. “It’s just too risky for the system.”

The skills that actually matter

So what separates good risk professionals from dangerous ones in the AI era? Siddiqi’s answer is surprisingly traditional: question everything. Question your data sources. Question the business logic. Question whether a statistically significant variable makes sense for deciding someone’s financial future.

The volume of data has exploded. The variety is overwhelming. But the fundamental skill remains unchanged: understanding why your data should be believed.

Because somewhere in your black-box model, there might be a variable that works perfectly in testing but codes for ethnicity, religion, or how often someone forgets to charge their phone. The math will tell you it’s predictive. Your Shapley values will confirm it matters. Your model accuracy will hit 99.2%.

And none of that will matter when the regulator asks why you’re making lending decisions based on TV watching habits or when a discrimination lawsuit reveals that your “neutral” algorithm has been systematically disadvantaging protected classes.

Image credit: iStockphoto/Rustic