Deep Learning Has Limits | But Its Commercial Impact Has Just Begun.

by NU Intel Staff
0 comment

There has been increased hand-wringing across the AI community in recent months about the limitations of deep learning.

It was a dominant theme a few months ago at NeurIPS, the world’s premier AI conference. In December, deep learning pioneer Yoshua Bengio and AI researcher Gary Marcus engaged in a high-profile televised debate about whether deep learning was the right path forward for AI. Many of the world’s top AI researchers have taken to Twitter recently to discuss deep learning’s shortcomings, sometimes heatedly.

Even deep learning’s staunchest defenders, Bengio included, are beginning to acknowledge the shortcomings of today’s neural networks.

“We have machines that learn in a very narrow way,” Bengio said in his keynote at NeurIPS in December. “They need much more data to learn a task than human examples of intelligence, and they still make stupid mistakes.”

Without question, deep learning is an imperfect model of intelligence. It cannot reason abstractly, does not understand causation and struggles with out-of-distribution generalization. (For those looking for further reading on deep learning’s shortcomings, Marcus’ influential 2018 paper on the topic does an excellent job summarizing the issues.)

There is an important distinction to draw here, however, which has not been clearly articulated in the recent public discourse. It relates to the relationship—and the lag—between cutting-edge academic research and the successful commercialization of that research.

Theoretical frontiers and commercial applications of deep learning represent two very different problem sets. Artificial intelligence need not match or surpass human capabilities to have tremendous commercial value. Deep learning may be bumping up against conceptual limits as a model of intelligence, but opportunities to apply it to transform industries and enact massive real-world change still abound. And the business opportunities that deep learning enables have barely begun to be developed.

Put differently, if all basic AI research stopped today, and the only methodologies available to entrepreneurs were those already in existence, countless billions of dollars in enterprise value would still be created in the years ahead by applying deep learning to solve business problems in novel ways.

Deep Learning

Human brain digital x-ray on blue background 3D rendering

Bringing a new technology to market at scale is a long and challenging process; getting the technology to work is just the beginning. A concrete business application must be identified. The technology must be “productized” in a way that generates demand from mainstream users. Regulatory hurdles often loom large, particularly in heavily-regulated industries like healthcare, transportation and financial services. Developing a go-to-market strategy, building out a capable sales team and refining a sales motion all take time and effort.

AI capabilities, as they exist today, are sufficiently advanced to enable transformative product innovation and value creation across industries: from agriculture to insurance, from healthcare to education, from transportation to construction, and beyond. For the most part, these transformative opportunities have not yet been operationalized at scale.

Concerns about deep learning’s theoretical limits, then, bear little on near-term commercial opportunities. For entrepreneurs and operators willing to work through challenges related to product development, business model, consumer education and regulation, massive white space exists to bring AI-first products to market.

Such opportunities can be found in nearly every industry. For illustrative purposes we will briefly walk through two: radiology and off-road autonomous vehicles.

Radiology
In 2016 AI luminary Geoff Hinton provocatively declared, “It’s quite obvious that we should stop training radiologists now.”

Hinton’s comments ruffled some feathers in the medical community, but it was hard to deny the data on which they were based. Over the past few years, a series of studies have demonstrated that neural networks can identify medical conditions from X-rays more accurately than can human radiologists.

To give a few recent examples: in May 2019, a team of researchers from Google, Stanford and Northwestern published a study in which a deep learning model outperformed human physicians at detecting lung cancer from CT scans. A few months later, a research team from NYU published a series of studies demonstrating AI’s superior performance detecting breast cancer from mammograms. In January 2020, a research group from Google and top medical research centers released another breast cancer detection study, with the AI system again outperforming humans.

In the January study, which received widespread media attention, the AI system produced a 9.4% reduction in false negatives and a 5.7% reduction in false positives relative to human radiologists.

In many ways, radiology is an ideal use case for deep learning. Examining images for the presence of a medical condition like cancer is an exercise in pattern recognition and object classification—exactly what deep learning excels at.

Yet, several years after Hinton predicted the obscolescence of human radiologists, no clinic in the world has deployed AI-driven radiology tools at scale. At best, a handful of forward-thinking health organizations have begun using it in limited settings.

Why is this? There is a huge gap between publishing academic research and building a real company—with real patients, in real clinics, with real lives on the line—to commercialize that research.

The U.S. healthcare system can be byzantine for startups to navigate, with notoriously long sales cycles and institutional inertia. Reimbursement regimes are complex. Consumer education and acceptance happen only gradually.

In addition, FDA approval must be obtained before clinicians can use algorithms for diagnosis in real-world settings. This is a long and tedious process; the FDA has only recently begun issuing approvals to a small handful of companies. Aidoc, a prominent AI radiology startup, just last month received FDA approval for its deep-learning-based stroke detection technology.

Another thorny and unresolved issue is liability: if a human doctor relies on an AI system to make a diagnosis that turns out to be wrong, who should be accountable?

Finally, these AI models are not yet sufficiently generalizable. Most models from the academic research community are trained using data from only one hospital. They often falter when applied to other populations. In one example, a deep learning model trained to detect pneumonia performed at 93% accuracy when used on patients from the same hospital, but dropped as low as 73% when tested on patients from other locations.

“It didn’t work as well because the patients at the other hospitals were different,” said Eric Oermann, one of the researchers.

This lack of generalizability can have serious implications for minority groups who are underrepresented in historical datasets—for instance, black women.

This problem is solvable. It requires collecting larger and more diverse datasets with which to train AI models. But such an effort will be expensive, time-consuming and operationally intensive.

The net result of these challenges is that AI-based radiology tools are still in only the earliest stages of commercial deployment. For instance, in the Google breast cancer study discussed above, the researchers directly acknowledged that their technology was not yet ready for real-world use.

Deep Learning

Deep Learning

“We will continue to explore and build upon our model, working with additional partners across the world, before considering bringing it into clinical practice,” said Shravya Shetty, a Google researcher who co-authored the paper.

This is not to say that no startups are working to commercialize this technology. Last year, CureMetrix became the first company to receive FDA approval for its AI-based breast cancer technology; the company plans to deploy in several clinical settings this year. Other startups angling to commercialize and scale AI-based radiology in the near term include Arterys, Aidoc, Zebra Technologies and DeepHealth.

But for a category projected to be worth over $3B in just a few years, the AI radiology market today remains surprisingly undeveloped. Expect this to change in the years ahead.

Off-Road Autonomous Vehicles
Autonomous vehicles provide another illustration of the massive unrealized commercial potential of today’s AI.

Vehicle autonomy is one of the most high-impact applications of AI being pursued today. While autonomous passenger cars receive the lion’s share of attention, the first meaningful commercial deployments of autonomous vehicles will likely be in quite different settings. Companies have barely begun to scratch the surface of these opportunities.

Construction ($11T), agriculture ($5T) and mining ($600B) are three of the largest industries in the world. A primary cost driver in each is human labor to operate vehicles in structured environments. Deploying autonomous vehicles in these industries would unlock massive cost, productivity and safety benefits.

Because vehicles in these sectors operate in highly controlled environments on repetitive routes at low speeds, the technological challenges are far simpler than for urban robotaxis. Think, for instance, of a tractor driving up and down rows of crops or a haul truck driving back and forth across a mining site. And because these vehicles generally do not operate on public roads, regulations pose less of an obstacle.

The opportunity here is both massive and actionable. Today’s state-of-the-art autonomous vehicle technology is sufficiently mature to enable the automation of off-road vehicles at scale around the world. Yet the number of autonomous vehicle deployments in construction, agriculture or mining remains vanishingly small.

Just as with the radiology example, while recent advances in AI mean that vast industry transformation has become technologically possible, this transformation has not yet come to full fruition due to the operational complexities of commercializing new technology.

Legacy industries like agriculture are underdigitized and ill-equipped to rapidly adopt new technologies. Introducing autonomous vehicles in these industries will require deep, systems-level adjustments to workflows and processes, creating change management challenges. Distrust of automation and concerns about job loss only further impede the diffusion of these technologies.

Early commercial efforts are underway. A handful of startups have begun to get traction in their efforts to commercialize off-road autonomous vehicles. These include Built Robotics in construction, Bear Flag Robotics in agriculture and SafeAI in mining. Industry incumbents like Rio Tinto, Caterpillar and John Deere are also pursuing this technology.

But these efforts remain nascent. To this day, the overwhelming majority of the billions of miles driven worldwide in construction, agriculture and mining are driven by humans.

None of the executional challenges discussed above are insurmountable. In the long run, they can and will be solved. But the process of operationalizing recent AI research to capitalize on these commercial opportunities will require intensive, years-long efforts from entrepreneurs.

The reward? Many billions of dollars of enterprise value creation. According to McKinsey, AI will generate $13 trillion in total global economic output by 2030.

As AI’s commercial pioneers are learning, getting the technology to work is only the start.

Related Posts