New Study: Deep Learning Reaching Computational Limits

We might be reaching the limits of deep learning, according to a new study (pdf) by researchers at the Massachusetts Institute of Technology (MIT), MIT-IBM Watson AI Lab, Underwood International College, and the University of Brasilia (UnB).

Titled “The computational limits of deep learning”, the study analyzed just over a thousand research papers before coming to its conclusion.

Limits of deep learning

Key to its assertion is how computational requirements have escalated rapidly in various deep learning domains around image classification, object detection, question answering, named entity recognition, and machine translation, with performance improvements anchored to increases in computing power.

“If progress continues along current lines, these computational requirements will rapidly become technically and economically prohibitive,” noted the report, alluding to a looming deep learning cliff.

Indeed, deep learning is so computationally expensive not by accident, but by design: “The same flexibility that makes it excellent at modeling diverse phenomena and outperforming expert models also makes it dramatically more computationally expensive.”

While the report conceded that the relationship between performance, model complexity, and computational requirements are not well understood, it noted that deep learning is intrinsically more reliant on computing power than other techniques.

To be clear, the report also noted that the actual computational burden of deep learning models is scaling more rapidly than known theory suggests. If correct, this means that substantial efficiency improvements might be possible.

This won’t be the first time that deep learning becomes computationally constrained, however. The report argues that this has been the case since the creation of the first neural networks, though the limitations were temporarily overcome in recent years due to the shift to specialized hardware (Think GPUs and ASICs), as well as a willingness to invest additional resources (Think cloud) to get better performance.

“But, as we show, the computational needs of deep learning scale so rapidly, that they will quickly become burdensome again.”

Photo credit: iStockphoto/anyaberkut