AI Ditches the Data Center and Heads to the Wild

Beyond the monstrous data centers, AI is going rogue. Edge AI embeds intelligence at the fringes of networks, changing everything. Those who don't get it risk being left in the dust.

In a recent online panel discussion, “AI’s Next Strategic Shift: The Rise of Edge AI,” Matthew Oostveen, chief technology officer for Asia Pacific & Japan at Pure Storage, joined Allan Song, executive director and head of strategy and transformation, FSS, Financial Markets, at Standard Chartered Bank and Nitin Acharekar, vice president at twimbit to lay out the opportunities. The panel, part of the Pure Leadership Series, also examined the gnarly challenges Edge AI introduces.

Can the real Edge AI stand up?

Let’s be real. Edge AI is going through an identity (a.k.a. definition) crisis. The issue is not “AI” but what “edge” really means.

For some, it’s anywhere outside the data center. Others say it's about running AI as close to the customer as possible—think banks or hospitals. Or, maybe the size of your infrastructure matters.

Admittedly, the definition will evolve as we continue to adopt more and more smart devices. But whatever your flavor, there are definite benefits for edge AI.

The first is speed. You can run those slick new language models without lag and with rock-solid security. Such capabilities are already doing wonders for client servicing, AI assistants, etc.

Next, edge devices—smartphones, smart home devices, or even industrial sensors—are becoming literally mini-brains. This AI/device combo means your home can basically run itself (and talk back to you). So, it only makes sense that AI finds new dwellings at the periphery.

Edge AI is not just limited to hosting and running models on the device. AI-driven interfaces, like natural language typing and voice, can also bring users closer to the data. These can allow consumers and professionals like logistics managers to make killer data-driven decisions on the fly.

Industries can also break free from their physical molds, like hospitals helping to monitor patients at home, saving lives and cash.

Insurers can adjust your rate based on today’s jog, not last year’s. Banks can process sensitive data in places that meet data sovereignty laws. Forget the hype—this is AI that matters.

The hidden beasts of Edge AI

ML models are known for their voracious data appetites and being bottomless pits. Edge AI will be no different.

Where Edge AI has an “edge” is in IT operations and management costs. The good news is that Edge AI might mean less strain on these cloud giants.

Edge AI allows a bank, for example, to use machine learning to squeeze payment settlement times into a competitive advantage.

If you care about Scope 3 emissions—those not produced by the company itself or its assets but indirectly responsible for up and down its value chain—Edge AI can be a secret weapon. By moving the data processing to the edge and closer to the client, a company can more accurately target emission reductions.

Yet, in a poll survey with the audience, cost and emissions were not the top worries. It’s the sheer headache of running a data model.

The choice—which surprised some of the panelists—may indicate a rising unease in processing costs, especially with the tremendous rise in GPU costs.

Panelists warned of the danger of overengineering AI models or using foundational models for all use cases. Edge AI offers an alternative route to using specific use cases or allowing companies to use small language models instead. In many cases, LLMs can be like using a gigantic hammer to hit a small nail.

Edge AI can allow companies to rationalize the number of AI projects they actually run. Ask a company whether they do AI; your answer will be laughter. But with every company or department looking at or experimenting with AI, there can be many. Add shadow AI projects to this number, and infrastructure management can be a nightmare.

A lack of talent offers a different dimension. Trying to centralize AI projects can mean hiring AI engineers at a central location. Finding them is hard; retaining them is worse. Edge AI (especially those run across borders) can allow companies to tap into AI talent overseas at local salaries.

The looming governance meltdown

When it comes to AI governance, Edge AI can be a can of worms. First, moving AI models to the edge can stretch centralized AI governance and responsible AI frameworks. Brittle ones can only break.

You can accidentally add local bias that may impact decision-making. And while modern architecture using containers and Kubernetes can help, it will not improve things if you are drowning in technical debt.

New AI legislation is already making AI a nightmare. Many new AI laws, like the E.U. AI Act, are use-case-driven and specific to jurisdiction. That means your models can run into trouble based on the use case and the regulatory framework in which your Edge AI model operates.

Then there’s the nagging problem of explainability, which almost all regulators require model owners to do. It is not just about showing how an AI model arrived at a decision; equally important is knowing where the information came from. The rising litigious behavior is only going to make everyone nervous.

A related problem is checkpointing, a process where AI teams save the state of a model during training to ensure explainability. But it also means more room to store this information. With Edge AI, the number of checkpoints can increase quickly, which may be a model training performance bottleneck.

Ok, but do you need Edge AI?

Pump the brakes. Before you jump on the bandwagon, get honest with what problem you're actually solving. Edge AI is awesome, but it's not a cure-all.

A first principles-based approach helps. AI teams should look at the most cost-effective way to address the problem using AI and decide where to train the model—at the core or the edge.

Sure, data gravity, which refers to the tendency of data to attract more data and applications, infrastructure readiness, data synchronization across edge points, and other such hurdles are major obstacles.

Yet, as the panelists agreed, Edge AI is still young. It's a land of undiscovered use cases and unforeseen potential. That's precisely why AI teams need to be at the forefront, or they'll find themselves playing catch-up in a world running on models deployed to the very edge of civilization.

Ready to dive deeper? Watch the full panel discussion here. 

Image credit: iStockphoto/PeopleImages