GenAI Progress Could Already Be Slowing

Improvements in GenAI could be slowing down, according to a report on The Information that cited anonymous employees at OpenAI. Specifically, it appears that the next full-fledged model currently being trained is showing a smaller performance jump than the one seen between GPT-3 and GPT-4.

Indeed, the upcoming model "isn't reliably better than its predecessor" in certain tasks, say the insiders. At the heart of the concern is how large language models (LLMs) appear to be hitting a plateau in what can be gained from traditional pre-training.

The end of the scaling law?

Over the last few years, AI firms have dramatically ramped up AI efforts with the “scaling law” in mind. Popularized by OpenAI, this revolves around the idea that training with more data and compute power equals better AI models. The result was exponentially larger models that delivered significant improvements, but now the returns are diminishing.

In May this year, the then CTO of OpenAI, Mira Murati, had openly spoken about how OpenAI is betting on the scaling paradigm. She said: “[We] will continue to push the scaling paradigm so models get more powerful as we put more compute, high-quality data, and more data into them. There is no evidence of the scaling paradigm stopping anytime soon; we should expect AI models to become much more powerful [just by scaling up].”

Indeed, she spoke about how she expects the next generation of GenAI models to deliver another “step change” in the form of new capabilities, similar to how GPT-4 was significantly better than GPT-3.5. If the slowdown is true, then it would represent a dramatic reversal of fortunes.

Not enough data

The issue appears to be the growing difficulty of finding original, quality data for training new AI models. This is a non-trivial problem to overcome, and one that has long been suggested by various AI experts. More than one study has established that using AI-created data for training eventually leads to model collapse.

As reported by Ars Technica, research firm Epoch AI attempted to quantify the data problem earlier this year in a white paper that measured the rate of increase in LLM training data sets against estimated human-generated public text. The researchers had estimated that language models will finish utilizing human-generated public data between 2026 and 2032.

Separately, OpenAI co-founder Ilya Sutskever, who has since left the firm, also told Reuters that “the age of scaling” could be over. “Now we're back in the age of wonder and discovery once again,” Sutskever was reported as saying. "Everyone is looking for the next thing. Scaling the right thing matters more now than ever.”

AI firms are not throwing in the towel but working hard to improve AI using new techniques. For instance, it was reported last week that MIT researchers have developed an efficient way to train more reliable AI agents, promising to make AI systems better at complex tasks that involve variability.

Finally, it is arguable that the world has already been irrevocably changed by AI. Even if AI never gets any better, there is no question that some jobs will eventually disappear, and many others will change. It just hasn't reached everyone – yet.

Image credit: iStock/cagkansayin