59% of all large enterprises are deploying data science (DS) and machine learning (ML) today.
Nearly 50% of all organizations have up to 25 or more ML models in use today.
29% of enterprises are refreshing their data science and machine learning models every day.
The higher the data literacy an enterprise can achieve before launching Data Science & Machine Learning initiatives, the higher the probability of success.
These and many other insights defining the state of the data science and machine learning market in 2021 are from Dresner Advisory Services’2021 Data Science and Machine Learning Market Study. The 7th annual report is noteworthy for its depth of analysis and insight into how data science and machine learning adoption is growing stronger in enterprises. In addition, the study explains which factors drive adoption and determine the key success factors that matter the most when deploying data science and machine learning techniques. The methodology uses crowdsourcing techniques to recruit respondents from over 6,000 organizations and vendors’ customer communities. As a result, 52% of respondents are from North America and 34% from EMEA, with the balance from Asia-Pacific and Latin America.
“The perceived importance of data science and machine learning correlates with organizational success with BI, with users that self-report as completely successful with BI almost twice as likely to rate data science as critical,” said Jim Ericson, vice president, and research director at Dresner Advisory. “The perceived level of data literacy also correlates directly and positively with the current or likely future use of data science and machine learning in 2021.”
Key insights from the study include the following:
59% of large enterprises are deploying data science and machine learning in production today. Enterprises with 10K employees or more lead all others in adopting and using DS and ML techniques, most often in R&D and Business Intelligence Competency Center (BICC)-related work. Large-scale enterprises often rely on DS and ML to identify how internal processes and workflows can be streamlined and made more cost-efficient. For example, the CEO of a manufacturing company explained on a recent conference call that DS and ML pilots bring much-needed visibility and control across multiple plants and help troubleshoot inventory management and supply chain allocation problems.
The importance of data science and ML to enterprises has doubled in eight years, jumping from 25% in 2014 to 70% in 2021. The Dresner study notes that a record level of enterprises sees data science and ML as critically important to their business in 2021. Furthermore, 90% of enterprises consider these technologies essential to their operations, rating them critically important or very important. Successful projects in Business Intelligence Competency Centers (BICC) and R&D helped data science and ML gain broad adoption across all organizations. Larger-scale enterprises with over 10K employees are successfully scaling data science and ML to improve visibility, control, and profitability in organizations today.
Enterprises dominate the recruiting and retention of data science and machine learning talent. Large-scale enterprises with over 10K employees are the most likely to have BI experts and data scientists/statisticians on staff. In addition, large-scale enterprises lead hiring and retention in seven of the nine roles included in the survey. It’s understandable how the Business Intelligence (BI) expertise of professionals in these roles is helping remove the roadblocks to getting more business value from data science and machine learning. Enterprises are learning how to scale data science and ML models to take on problems that were too complex to solve with analytics or BI alone.
80% of DS and ML respondents most want model lifecycle management, model performance monitoring, model version control, and model lineage and history at a minimum. Keeping track of the state of each model, including version control, is a challenge for nearly all organizations adopting ML today. Enterprises reach ML scale when they can manage ML models across their lifecycles using an automated system. The next four most popular features of model rollback, searchable model repository, collaborative, model co-creation tools, and model registration and certification are consistent with the feedback from Data Science teams on what they need most in an ML platform.
Financial Services prioritize model lifecycle management and model performance monitoring to achieve greater scale from the tens of thousands of models they’re using today. Consistent with other research that tracks ML adoption by industry, the Dresner study found that Financial Services leads all other industries in their need for the two most valuable features of ML platforms, model lifecycle management and model performance monitoring. Retail and Wholesale are reinventing their business models in real-time to become more virtual while also providing greater real-time visibility across supply chains. ML models in these two industries need automated model version control, model lineage and history, model rollback, collaborative, model co-creation tools, and model registration and certification. In addition, retailers and Wholesalers are doubling down on data science and machine learning to support new digital businesses, improve supply chain performance and increase productivity.
Enterprises need support for their expanding range of regression models, text analytics functions, and ensemble learning. Over the last seven years, text analytics functions and sentiment analysis’ popularity has continually grown. Martech vendors and the marketing technologists driving the market are increasing sentiment analysis’ practicality and importance. Recommendation engines and geospatial analysis are also experiencing greater adoption due to martech changing the nature of customer- and market-driven analysis and predictive modeling.
R, TensorFlow, and PyTorch are considered the three most critical open-source statistical and machine learning frameworks in 2021. Nearly 70% of respondents consider R important to getting work done in data science and ML. The R language has established itself as an industry standard and is well-respected across DevOps, and IT teams in financial services, professional services, consulting, process, and discrete manufacturing. Tensorflow and Pytorch are considered important by the majority of organizations Dresner’s research team interviewed. They’re also among the most in-demand ML frameworks today, with new applicants having experience in all three being recruited actively today.
Data literacy predicts DS and ML program success rates. 64% of organizations say they have extremely high literacy rates, implying that DS and ML have reached mainstream adoption thanks partly to BI literacy rates in the past. Enterprises that prioritize data literacy by providing training, certification, and ongoing education increase success odds with ML. A bonus is that employees will have a chance to learn marketable skills they can use in their current and future positions. Investing in training to improve data literacy is a win/win.
On-database analytics and in-memory analytics (both 91%), and multi-tenant cloud services (88%) are the three most popular technologies enterprises rely on for greater scalability. Dresner’s research team observes that the scalability of data science and machine learning often involves multiple, different requirements to address high data volumes, large numbers of users, data variety while supporting analytic throughput. Apache Spark support continues to grow in enterprises and is the fourth-most relied-on industry support for ML scalability.
Demand for machine learning expertise, as reflected in LinkedIn open positions, also shows strong growth. Increasing from 44,864 available jobs in 2020 to 78,372 in 2021 in the U.S. alone, organizations continue to staff up to support new initiatives quickly. Globally, LinkedIn’s open positions requiring machine-learning expertise grew from 98,371 in 2020 to 191,749 in 2021.
Market forecasts and projections also reflect strong growth for AI and machine learning spending globally for the long term. The following are key takeaways from the machine learning market forecasts from the last year include the following:
Forrester says the AI market will be defined and grow within four software segments, with AI maker platforms growing the fastest, reaching $13 billion by 2025, helping drive the market to $37 billion by 2025. Forrester is defining the four AI software segments as follows: AI maker platforms for general-purpose AI algorithms and data sets; AI facilitator platforms for specific AI functions like computer vision; AI-centric applications and middleware tools built around AI for specialized tasks like medical diagnosis; and AI-infused applications and middleware tools that differentiate through advanced use of AI in an existing app or tool category. New AI-centric apps built on AI functions such as medical diagnosis and risk detection solutions will be the second-largest market, valued at nearly $10 billion by 2025. Source: Sizing The AI Software Market: Not As Big As Investors Expect But Still $37 Billion By 2025, December 10, 2020.
IDC predicts worldwide revenues for the artificial intelligence (AI) market, including software, hardware, and services, will grow from $327.5 billion in 2021 to $554.3 billion in 2024, attaining a five-year compound annual growth rate (CAGR) of 17.5%. IDC further predicts that the AI Software Platforms market will be the strongest, with a five-year CAGR of 32.7%. The slowest will be AI System Infrastructure Software, with a five-year CAGR of 13.7% while accounting for roughly 36% of AI software revenues. IDC found that among the three technology categories, software represented 88% of the total AI market revenues in 2020. It’s the slowest growing category with a five-year CAGR of 17.3%. AI Applications took the largest share of revenue within the AI software category at 50% in 2020. Source: IDC Forecasts Improved Growth for Global AI Market in 2021, February 23, 2021
AI projects continued to accelerate in 2020 in the healthcare, bioscience, manufacturing, financial services, and supply chain sectors despite economic & social uncertainty. Two dominant themes emerge from the combination of 30 diverse AI technologies in this year’s Hype Cycle. The first theme is the democratization or broader adoption of AI across organizations. The greater the democratization of AI, the greater the importance of developers and DevOps to create enterprise-grade applications. The second theme is the industrialization of AI platforms. Reusability, scalability, safety, and responsible use of AI and AI governance are the catalysts contributing to the second theme. The Gartner Hype Cycle for Artificial Intelligence, 2020, is shown below: Source: Software Strategies Blog, What’s New In Gartner’s Hype Cycle For AI, 2020, October 20, 2020.
Capgemini finds that Life Sciences, Retail, Consumer Products, and Automotive industries lead in the percentage of successfully deployed AI use cases today. Life Sciences leads all interviewed industries to AI maturity, with 27% of companies saying they have deployed use cases in production and at scale. Retail is also above the industry average of 13% of companies that have deployed AI in production at scale, with 21% of companies in the industry has adopted AI successfully. 17% of companies in the Consumer Products and Automotive industries now have AI in production, running at scale. Source: Capgemini, Making AI Work For You, (The AI-powered enterprise: Unlocking the potential of AI at scale) 2021
Between 2018 and 2020, there’s been a 76% increase in sales professionals using AI-based apps and tools. Salesforce’s latest State of Sales survey found that 57% of high-performance sales organizations use AI today. High-performing sales organizations are 2.8x more likely to use AI than their peers. High-performing sales organizations rely on AI to gain new insights into customer needs, improve forecast accuracy, gain more significant visibility of rep activity, improve competitive analysis, and more. Source: Salesforce Research, 4th Edition, State of Sales, June 2020
While 24% of companies are currently using AI for recruitment, that number is expected to grow, with 56% reporting they plan to adopt AI next year. In addition, Sage’s recent survey of 500 senior HR and people leaders finds adoption of AI as an enabling technology for talent management increasing. AI is proving effective for evaluating job candidates for potential, improving virtual recruiting events, and reducing biased language in job descriptions. It’s also proving effective in helping to improve career planning and mobility. Josh Bersin, a noted HR industry analyst, educator, and technologist, recently published an interesting report on this area titled The Rise of the Talent Intelligence Platform. Leaders in the field of Talent Intelligence Platforms include Eightfold.ai. Grounded in Equal Opportunity Algorithms, the Eightfold® Talent Intelligence Platform uses deep-learning AI to help each person understand their career potential, and each enterprise understands the potential of their workforce.Sources: VentureBeat, 8 ways AI is transforming talent management in 2021, March 25, 2021, and Eightfold.ai.
84% of marketers are using AI-based apps and platforms today, up from 28% in 2018. Salesforce Research’s latest State of Marketing survey finds that high-performing marketers use an average of seven different applications or use cases. The familiarity high-performing marketers have with AI is a primary factor in 52% of them predicting they will increase their use of AI-based apps in the future. Source: Salesforce Research, 6th Edition, State of Marketing, June 2020
Marketing and Sales lead revenue increases due to AI adoption, yet lag behind other departments on cost savings. 40% of the organizations McKinsey interviewed see between a 6 and 10% increase in revenue from adopting AI in their marketing and sales departments. Adopting Ai to reduce costs delivers the best manufacturing and supply chain management results based on the McKinsey survey results. Revenue increases and cost reductions based on AI adoption are shown in the graphic below. Source: McKinsey & Company, The state of AI in 2020, November 17, 2020
AI sees the most significant adoption by marketers working in $500M to $1B companies, with conversational AI for customer service as the most dominant. Businesses with between $500M to $1B lead all other revenue categories in the number and depth of AI adoption cases. Just over 52% of small businesses with sales of $25M or less use AI for predictive analytics for customer insights. It’s interesting to note that small companies are the leaders in AI spending, at 38.1%, to improve marketing ROI by optimizing marketing content and timing. Source: The CMO Survey: Highlights and Insights Report, February 2019. Duke University, Deloitte, and American Marketing Association. (71 pp., PDF, free, no opt-in).
Three out of four companies are fast-tracking automation initiatives, including AI. Bain & Company found that executives would like to use AI to reduce costs and acquire new customers, but they’re uncertain about the ROI and cannot find the talent or solutions they need. Bain research conducted in 2019 found that 90% of tech executives view AI and machine learning as priorities that they should be incorporating into their product lines and businesses. But nearly as many (87%) also said they were not satisfied with their Company’s current approach to AI. Source: Bain & Company, Will the Pandemic Accelerate Adoption of Artificial Intelligence? May 26, 2020
Gartner’s Magic Quadrant for Data Science and Machine Learning Platforms predicts a continued glut of exciting innovations and visionary roadmaps from competing vendors. Competitors in the Data Science and Machine Learning (DSML) market focus on innovation and rapid product innovation over pure execution. Gartner said key areas of differentiation include UI, augmented DSML (AutoML), MLOps, performance and scalability, hybrid and multicloud support, XAI, and cutting-edge use cases and techniques (such as deep learning, large-scale IoT, and reinforcement learning). Please see my recent article on VentureBeat, Gartner’s 2021 Magic Quadrant cites ‘glut of innovation’ in data science and ML, March 14, 2021.
76% of enterprises are prioritizing AI & machine Learning In 2021 IT Budgets. Algorithmia’s survey finds that six in ten (64%) organizations say AI and ML initiatives’ priorities have increased relative to other IT priorities in the last twelve months. Algorithmia’s survey from last summer found that enterprises began doubling down on AI & ML spending last year. The pandemic created a new sense of urgency regarding getting AI and ML projects completed, a key point made by CIOs across the financial services and tech sectors last year during interviews for comparable research studies. Source: Algorithmia’s Third Annual Survey, 2021 Enterprise Trends in Machine Learning.
Technavio predicts the Artificial Intelligence platforms market will grow to $17.29 billion by 2025, attaining a compound annual growth rate (CAGR) of nearly 35%. The research firm cites the increased levels of AI R&D investments globally combined with accelerating adoption for pilot and proof of concept testing across industries. Technavio predicts Alibaba Group Holding Ltd., Alphabet Inc., and Amazon.com Inc. will emerge as top artificial intelligence platforms vendors by 2025. Source: Artificial Intelligence Platforms Market to grow by $ 17.29 Billion at 35% CAGR during 2021-2025. June 21, 2021
Tractica predicts the AI software market will reach $126 billion in worldwide revenue by 2025. The research firm predicts AI will grow fastest in consumer (Internet services), automotive, financial services, telecommunications, and retail industries. As a result, annual global AI software revenue is forecast to grow from $10.1 billion in 2018 to $126.0 billion by 2025. Source: T&D World, AI Software Market to Reach $126.0 Billion in Annual Worldwide Revenue by 2025.