Do Mainframes Have a Place In an AI World?

You would think that, as companies focus on GPUs for training, mainframes are becoming “tech dinosaurs”. But in a recent interview with IBM’s Francis Goh, vice president of IBM Z Systems and LinuxONE for Asia Pacific, the answer is more nuanced.

Yes, mainframes have a role in an AI world. But not in training, where GPUs often do well. Rather, it is in solving the next challenge in AI: inference. And here, and in specific use cases involving sub-second AI, mainframes can be a more persuasive solution. 

The quiet revolution

Goh has spent the better part of a decade watching this transformation unfold. Speaking from Jakarta, where he was launching IBM's latest generation of mainframes, He painted a picture that challenges conventional wisdom about where AI belongs in the enterprise stack.

“IBM Z powers 70% of the world's transactions by value,” Goh reveals, a statistic that immediately reframes the conversation. “The majority of critical mission-critical transactions still go through the IBM mainframe. When we talk to customers about AI, it's really about bringing AI to where the data is.”

This changes how we think about AI deployment. While the industry has been obsessing over training massive language models in the cloud, IBM has been quietly solving a different problem: how to make AI inference — the process of using a trained machine learning model to make predictions or decisions based on new, untrained data — happen where it matters most, at the moment of transaction.

Solving the sub-second challenge

The technical challenge is staggering. Take credit card transactions as an example. These must clear within roughly one second, or they're declined. Traditional AI systems, even GPU-accelerated ones, can't retrieve data, process it through machine learning models, and return results within this window when operating across different systems.

“Credit card companies only have like a very short window — it's like one second to clear the card before the service level is breached and the card is declined,” Goh explains. “Most banks will adopt fraud detection for maybe 10% or even less than 10% of transactions because they need transactions to clear within a short amount of time.”

This is where IBM's mainframes with its new enhancements come in. The Z17's Telum processor can execute what Goh calls “predictive AI at blazing fast speed,” processing up to 450 billion AI inference operations per day. More importantly, it does this without the latency penalty of external systems.

“Imagine if you can execute real-time fraud detection scoring with AI models 100% of the time in real-time,” Goh says. “This technology is so effective that it can actually score all transactions — 100% of all payment transactions — for fraud before the payment actually happens."

Great for specific use cases

While fraud detection makes for compelling headlines, the applications for mainframe use extend far beyond financial services. IBM has identified over 250 use cases spanning industries: medical image recognition, geospatial analysis, climate change impact modeling, and insurance underwriting.

The key insight is that these aren't general-purpose AI workloads — they're mission-critical applications where latency, security, and reliability matter more than raw computational throughput. “It's not a general-purpose AI scenario,” Goh acknowledges, “but in certain situations that really matter, this technology is beautiful.”

This specialization extends to IBM's positioning relative to GPU-based systems. “We are not claiming that we are a substitute for [GPUs],” Goh states frankly. However, he points out that mainframes can be more efficient than GPU servers.

The energy efficiency claims are particularly striking: IBM says its AI inference capabilities use five times less energy than comparable x86 systems with GPUs while achieving better latency. In an era where hyperscalers are building data centers primarily constrained by power rather than space, this efficiency advantage could prove decisive.

The skills paradox

Perhaps the most intriguing aspect of IBM's AI strategy is how it addresses the mainframe industry's Achilles' heel: the dwindling pool of COBOL programmers and system administrators. Rather than accepting this as an inevitable decline, IBM is using AI to solve the skills problem.

“One of the feedback that we have from customers is, ‘It's not easy to find COBOL skills,’” Goh admits. “COBOL developers usually are quite senior. There are fewer graduates who learn COBOL.”

IBM's response is watsonx Code Assistant for Z, an AI system that can understand decades-old COBOL applications, document their dependencies, and even refactor monolithic code into reusable services. “With the AI assistant, it can actually understand all the application dependencies, how they are related, be able to understand the code, document the code, and even refactor the code for reuse.”

This creates a fascinating recursive loop: AI systems running on mainframes are being used to modernize and maintain the mainframes themselves, potentially extending their relevance far beyond what seemed possible just a few years ago.

The hybrid reality

Despite the impressive technical capabilities, Goh is realistic about market positioning. “IBM believes in hybrid cloud,” he emphasizes. “Many of IBM's software offerings are available on AWS, Azure, GCP, and IBM Cloud. Our philosophy is fit-for-purpose.”

This pragmatic approach acknowledges that different workloads belong in different places. GPU farms for training, edge devices for interaction, and mainframes for mission-critical transactions. But Goh also reveals an interesting counter-trend: “We actually see evidence of customers repatriating workloads back from cloud to on-premise.”

The reasons are varied: cost control, data sovereignty, and regulatory compliance. But they suggest that the cloud-first orthodoxy of the past decade may be giving way to more nuanced architectural decisions.

The persistence of Big Iron

Mainframes’ enduring presence is recognition that some problems require solutions that have been battle-tested for decades. In an age where AI systems can hallucinate, where cloud costs can spiral out of control, and where security breaches make headlines daily, the mainframe's core virtues of reliability, security, and performance become more valuable, not less.

It also means the future of AI may not be 100% in the cloud after all. It may be in the basement, humming quietly in the same rooms where it has faithfully processed the world's most important transactions for half a century. The only difference is that now, it's learning.

Image credit: iStockphoto/DisobeyArt