In Gartner’s 2018 CIO Agenda Survey, Gartner predicted that through 2022, 85% of AI projects will deliver erroneous outcomes, showing the high likelihood of Big Data and AI projects failing. More than a year later, VentureBeat said that 87% of data science projects fail to make it into production. What are the reasons for this higher likelihood of failure for data science, Big Data and AI projects? We look at the three reasons that cannot be ignored.
1st reason Big Data and AI projects fail: Strategic dilemma of technology and business
It’s not uncommon for companies to point the finger at bad strategic or leadership directions for Big Data and AI project failures.
This blame is backed by an International Data Corporation (IDC) study of global organisations using AI solutions.
It found that only 25% of them have developed an enterprise-wide AI strategy, despite 2/3 of them emphasizing an “AI First” culture and 1/2 seeing AI as a priority.
Lacking a well-defined strategy can lead to difficulties in meeting the high expectations on projects, which is also another organisational attribute.
Business managers tend to make assumptions about the technology that simply cannot be met, so their expectations are beyond what is realistically achievable.
Although technology has advanced so much in recent years, the technology’s capabilities may not match the business expectations for specific functions.
So, a failure to understand what the technology can and cannot do for the project will result in a failure of the project altogether.
However, the other way around, which is technology-first orientation, is also a problem for Big Data and AI projects.
This refers to going deep into a technology’s or model’s capabilities instead of first articulating the key business imperatives it wants to address.
In other words, placing excessive focus on the technology or model to search for a problem to solve using the tool in question.
This results in a project failure because it ends up solving a problem that no one cares about, which would be a waste of the company’s time and resources.
An annual MIT Sloan Management Review-Boston Consulting Group study justified this reason by revealing that companies that view AI as just a technology opportunity failed to reap its benefits unlike those who also saw AI as a business strategy.
2nd reason Big Data and AI projects fail: Data lacking in quality and quantity
For anyone involved in Big Data and AI projects, the importance of available and organised data is undeniably a no-brainer.
No one can imagine any project succeeding without data.
It’s really obvious that AI needs data to function and does its learning based on the training data sets it’s fed.
So, having no data to learn will get the AI project nowhere.
Even if the data is available, the project could still fail if the data set sucks.
Inaccurate, outdated and incomplete data could result in the failure to provide accurate insights for the company.
Thus, the obvious importance of having the right data quality and quantity can’t be emphasised enough.
3rd reason Big Data and AI projects fail: Shortage of Big Data and AI talent
Even if you had the right data sets for your Big Data and AI project, not having the right talent to deal with the data is another reason to send the project to failure town.
The AI Specialist role placed first in LinkedIn’s 2020 list for the Top 15 Emerging Jobs In The US, proving that AI talent is in high demand.
“Artificial Intelligence and Machine Learning have both become synonymous with innovation, and our data shows that’s more than just buzz.
Hiring growth for this role has grown 74% annually in the past 4 years and encompasses a few different titles within the space that all have a very specific set of skills despite being spread across industries, including artificial intelligence and machine learning engineer.”
LinkedIn’s 2020 Emerging Jobs Report.
However, a Gartner study reported that 56% of its respondents named talent shortage as their number one barrier to adopting AI.
In Deloitte’s third edition of the “State of AI in the Enterprise” survey, 23% of the most mature AI adopters surveyed said that they had a major gap between AI needs and current abilities, a higher percentage than the less mature ones.
What can be done about these reasons for Big Data and AI project failures?
Before the impact of any complex issue on Big Data and AI project failures can be discussed, companies must first ask themselves if any of these reasons apply to their failures:
- bad strategy and unrealistic goals but also excessive focus on technology
- no good data to work with
- no good talent to do the work
Once these reasons have been identified, they must then be addressed with the appropriate solutions.
To address the first reason of strategic dilemma, managers need to be conservative in setting the bar for projects by setting a reasonable goal.
They can start by identifying where AI might solve an existing problem or where others have found AI has added value.
For the second reason of poor data governance, companies should set up an organised and consolidated central data repository-dashboard to ensure that everyone is on the same page about the data that leaves no room for mistakes or duplicates.
And finally, for the third reason of talent shortage, it takes the collective efforts of governments, education systems and companies worldwide to provide incentives for people to pursue careers in Big Data and AI.
The government can provide more financial grants and scholarships to students who apply for data science degrees or programmes.
Meanwhile, schools should start teaching technology and data science in early education to cultivate their interest in tech and prepare the next generation of students for the fourth industrial revolution.
Companies, on the other hand, should find ways to encourage their employees to retrain and take on more relevant roles like data scientists and AI specialists.
Of course, I’m not saying we should force people to study data science but, if there’s no nudge for anyone to pursue it, then there’s no way to identify someone who will.
All in all, it looks like the first step in avoiding failure, or at least minimising it, is to go back to basics.