59% of all large enterprises are deploying data science (DS) and machine learning (ML) today.
Nearly 50% of all organizations have up to 25 or more ML models in use today.
29% of enterprises are refreshing their data science and machine learning models every day.
The higher the data literacy an enterprise can achieve before launching Data Science & Machine Learning initiatives, the higher the probability of success.
These and many other insights defining the state of the data science and machine learning market in 2021 are from Dresner Advisory Services’2021 Data Science and Machine Learning Market Study. The 7th annual report is noteworthy for its depth of analysis and insight into how data science and machine learning adoption is growing stronger in enterprises. In addition, the study explains which factors drive adoption and determine the key success factors that matter the most when deploying data science and machine learning techniques. The methodology uses crowdsourcing techniques to recruit respondents from over 6,000 organizations and vendors’ customer communities. As a result, 52% of respondents are from North America and 34% from EMEA, with the balance from Asia-Pacific and Latin America.
“The perceived importance of data science and machine learning correlates with organizational success with BI, with users that self-report as completely successful with BI almost twice as likely to rate data science as critical,” said Jim Ericson, vice president, and research director at Dresner Advisory. “The perceived level of data literacy also correlates directly and positively with the current or likely future use of data science and machine learning in 2021.”
Key insights from the study include the following:
59% of large enterprises are deploying data science and machine learning in production today. Enterprises with 10K employees or more lead all others in adopting and using DS and ML techniques, most often in R&D and Business Intelligence Competency Center (BICC)-related work. Large-scale enterprises often rely on DS and ML to identify how internal processes and workflows can be streamlined and made more cost-efficient. For example, the CEO of a manufacturing company explained on a recent conference call that DS and ML pilots bring much-needed visibility and control across multiple plants and help troubleshoot inventory management and supply chain allocation problems.
The importance of data science and ML to enterprises has doubled in eight years, jumping from 25% in 2014 to 70% in 2021. The Dresner study notes that a record level of enterprises sees data science and ML as critically important to their business in 2021. Furthermore, 90% of enterprises consider these technologies essential to their operations, rating them critically important or very important. Successful projects in Business Intelligence Competency Centers (BICC) and R&D helped data science and ML gain broad adoption across all organizations. Larger-scale enterprises with over 10K employees are successfully scaling data science and ML to improve visibility, control, and profitability in organizations today.
Enterprises dominate the recruiting and retention of data science and machine learning talent. Large-scale enterprises with over 10K employees are the most likely to have BI experts and data scientists/statisticians on staff. In addition, large-scale enterprises lead hiring and retention in seven of the nine roles included in the survey. It’s understandable how the Business Intelligence (BI) expertise of professionals in these roles is helping remove the roadblocks to getting more business value from data science and machine learning. Enterprises are learning how to scale data science and ML models to take on problems that were too complex to solve with analytics or BI alone.
80% of DS and ML respondents most want model lifecycle management, model performance monitoring, model version control, and model lineage and history at a minimum. Keeping track of the state of each model, including version control, is a challenge for nearly all organizations adopting ML today. Enterprises reach ML scale when they can manage ML models across their lifecycles using an automated system. The next four most popular features of model rollback, searchable model repository, collaborative, model co-creation tools, and model registration and certification are consistent with the feedback from Data Science teams on what they need most in an ML platform.
Financial Services prioritize model lifecycle management and model performance monitoring to achieve greater scale from the tens of thousands of models they’re using today. Consistent with other research that tracks ML adoption by industry, the Dresner study found that Financial Services leads all other industries in their need for the two most valuable features of ML platforms, model lifecycle management and model performance monitoring. Retail and Wholesale are reinventing their business models in real-time to become more virtual while also providing greater real-time visibility across supply chains. ML models in these two industries need automated model version control, model lineage and history, model rollback, collaborative, model co-creation tools, and model registration and certification. In addition, retailers and Wholesalers are doubling down on data science and machine learning to support new digital businesses, improve supply chain performance and increase productivity.
Enterprises need support for their expanding range of regression models, text analytics functions, and ensemble learning. Over the last seven years, text analytics functions and sentiment analysis’ popularity has continually grown. Martech vendors and the marketing technologists driving the market are increasing sentiment analysis’ practicality and importance. Recommendation engines and geospatial analysis are also experiencing greater adoption due to martech changing the nature of customer- and market-driven analysis and predictive modeling.
R, TensorFlow, and PyTorch are considered the three most critical open-source statistical and machine learning frameworks in 2021. Nearly 70% of respondents consider R important to getting work done in data science and ML. The R language has established itself as an industry standard and is well-respected across DevOps, and IT teams in financial services, professional services, consulting, process, and discrete manufacturing. Tensorflow and Pytorch are considered important by the majority of organizations Dresner’s research team interviewed. They’re also among the most in-demand ML frameworks today, with new applicants having experience in all three being recruited actively today.
Data literacy predicts DS and ML program success rates. 64% of organizations say they have extremely high literacy rates, implying that DS and ML have reached mainstream adoption thanks partly to BI literacy rates in the past. Enterprises that prioritize data literacy by providing training, certification, and ongoing education increase success odds with ML. A bonus is that employees will have a chance to learn marketable skills they can use in their current and future positions. Investing in training to improve data literacy is a win/win.
On-database analytics and in-memory analytics (both 91%), and multi-tenant cloud services (88%) are the three most popular technologies enterprises rely on for greater scalability. Dresner’s research team observes that the scalability of data science and machine learning often involves multiple, different requirements to address high data volumes, large numbers of users, data variety while supporting analytic throughput. Apache Spark support continues to grow in enterprises and is the fourth-most relied-on industry support for ML scalability.
Improving revenues using BI is now the most popular objective enterprises are pursuing in 2019.
Reporting, dashboards, data integration, advanced visualization, and end-user self-service are the most strategic BI initiatives underway in enterprises today.
Operations, Executive Management, Finance, and Sales are primarily driving Business Intelligence (BI) adoption throughout enterprises today.
Tech companies’ Operations & Sales teams are the most effective at driving BI adoption across industries surveyed, with Advertising driving BI adoption across Marketing.
These and many other fascinating insights are from Dresner Advisory Associates’ 10th edition of its popular Wisdom of Crowds® Business Intelligence Market Study. The study is noteworthy in that it provides insights into how enterprises are expanding their adoption of Business Intelligence (BI) from centralized strategies to tactical ones that seek to improve daily operations. The Dresner research teams’ broad assessment of the BI market makes this report unique, including their use visualizations that provide a strategic view of market trends. The study is based on interviews with respondents from the firms’ research community of over 5,000 organizations as well as vendors’ customers and qualified crowdsourced respondents recruited over social media. Please see pages 13 – 16 for the methodology.
Key insights from the study include the following:
Operations, Executive Management, Finance, and Sales are primarily driving Business Intelligence (BI) adoption throughout their enterprises today. More than half of the enterprises surveyed see these four departments as the primary initiators or drivers of BI initiatives. Over the last seven years, Operations departments have most increased their influence over BI adoption, more than any other department included in the current and previous survey. Marketing and Strategic Planning are also the most likely to be sponsoring BI pilots and looking for new ways to introduce BI applications and platforms into use daily.
Tech companies’ Operations & Sales teams are the most effective at driving BI adoption across industries surveyed, with Advertising driving BI adoption across Marketing. Retail/Wholesale and Tech companies’ sales leadership is primarily driving BI adoption in their respective industries. It’s not surprising to see the leading influencer among Healthcare respondents is resource-intensive HR. The study found that Executive Management is most likely to drive business intelligence in consulting practices most often.
Reporting, dashboards, data integration, advanced visualization, and end-user self-service are the most strategic BI initiatives underway in enterprises today. Second-tier initiatives include data discovery, data warehousing, data discovery, data mining/advanced algorithms, and data storytelling. Comparing the last four years of survey data, Dresner’s research team found reporting retains all-time high scores as the top priority, and data storytelling, governance, and data catalog hold momentum. Please click on the graphic to expand for easier reading.
BI software providers most commonly rely on executive-level personas to design their applications and add new features. Dresner’s research team found all vertical industries except Business Services target business executives first in their product design and messaging. Given the customer-centric nature of advertising and consulting services business models, it is understandable why the primary focus BI vendors rely on in selling to them are customer personas. The following graphic compares targeted users for BI by industry.
Improving revenues using BI is now the most popular objective in 2019, despite BI initially being positioned as a solution for compliance and risk management. Executive Management, Marketing/Sales, and Operations are driving the focus on improving revenues this year. Nearly 50% of enterprises now expect BI to deliver better decision making, making the areas of reporting, and dashboards must-have features. Interestingly, enterprises aren’t looking to BI as much for improving operational efficiencies and cost reductions or competitive advantages. Over the last 12 to 18 months, more tech manufacturing companies have initiated new business models that require their operations teams to support a shift from products to services revenues. An example of this shift is the introduction of smart, connected products that provide real-time data that serves as the foundation for future services strategies. Please click on the graphic to expand for easier reading.
In aggregate, BI is achieving its highest levels of adoption in R&D, Executive Management, and Operations departments today. The growing complexity of products and business models in tech companies, increasing reliance on analytics and BI in retail/wholesale to streamline supply chains and improve buying experiences are contributing factors to the increasing levels of BI adoption in these three departments. The following graphic compares BI’s level of adoption by function today.
Enterprises with the largest BI budgets this year are investing more heavily into dashboards, reporting, and data integration. Conversely, those with smaller budgets are placing a higher priority on open source-based big data projects, end-user data preparation, collaborative support for group-based decision-making, and enterprise planning. The following graphic provides insights into technologies and initiatives strategic to BI at an enterprise level by budget plans.
Marketing/Sales and Operations are using the greatest variety of BI tools today. The survey shows how conversant Operations professionals are with the BI tools in use throughout their departments. Every one of them knows how many and most likely which types of BI tools are deployed in their departments. Across all industries, Research & Development (R&D), Business Intelligence Competency Center (BICC), and IT respondents are most likely to report they have multiple tools in use.
Salesforce (NYSE:CRM) estimates adding analytics and Business Intelligence (BI) applications will increase their Total Addressable Market (TAM) by $13B in FY2014.
89% of business leaders believe Big Data will revolutionize business operations in the same way the Internet did.
83% have pursued Big Data projects in order to seize a competitive edge.
Despite the varying methodologies used in the studies mentioned in this roundup, many share a common set of conclusions. The high priority in gaining greater insights into customers and their unmet needs, more precise information on how to best manage and simplify sales cycles, and how to streamline service are common themes.
The most successful Big Data uses cases revolve around enterprises’ need to get beyond the constraints that hold them back from being more attentive and responsive to customers.
Presented below is a roundup of recent forecasts and estimates:
Wikibon projects the Big Data market will top $84B in 2026, attaining a 17% Compound Annual Growth Rate (CAGR) for the forecast period 2011 to 2026. The Big Data market reached $27.36B in 2014, up from $19.6B in 2013. These and other insights are from Wikibon’s excellent research of Big Data market adoption and growth. The graphic below provides an overview of their Big Data Market Forecast. Source: Executive Summary: Big Data Vendor Revenue and Market Forecast, 2011-2026.
IBM and SAS are the leaders of the Big Data predictive analytics market according to the latest Forrester Wave™: Big Data Predictive Analytics Solutions, Q2 2015. The latest Forrester Wave is based on an analysis of 13 different big data predictive analytics providers including Alpine Data Labs, Alteryx, Angoss Software, Dell, FICO, IBM, KNIME.com, Microsoft, Oracle, Predixion Software, RapidMiner, SAP, and SAS. Forrester specifically called out Microsoft Azure Learning is an impressive new entrant that shows the potential for Microsoft to be a significant player in this market. Gregory Piatetsky (@KDNuggets) has done an excellent analysis of the Forrester Wave Big Data Predictive Analytics Solutions Q2 2015 report here. Source: Courtesy of Predixion Software: The Forrester Wave™: Big Data Predictive Analytics Solutions, Q2 2015 (free, no opt-in).
IBM, KNIME, RapidMiner and SAS are leading the advanced analytics platform market according to Gartner’s latest Magic Quadrant. Gartner’s latest Magic Quadrant for advanced analytics evaluated 16 leading providers of advanced analytics platforms that are used to building solutions from scratch. The following vendors were included in Gartner’s analysis: Alpine Data Labs, Alteryx, Angoss, Dell, FICO, IBM, KNIME, Microsoft, Predixion, Prognoz, RapidMiner, Revolution Analytics, Salford Systems, SAP, SAS and Tibco Software, Gregory Piatetsky (@KDNuggets) provides excellent insights into shifts in Magic Quadrant for Advanced Platform rankings here. Source: Courtesy of RapidMiner: Magic Quadrant for Advanced Analytics Platforms Published: 19 February 2015 Analyst(s): Gareth Herschel, Alexander Linden, Lisa Kart (reprint; free, no opt-in).
Salesforce estimates adding analytics and Business Intelligence (BI) applications will increase their Total Addressable Market (TAM) by $13B in FY2014. Adding new apps in analytics is projected to increase their TAM to $82B for calendar year (CY) 2018, fueling an 11% CAGR in their total addressable market from CY 2013 to 2018. Source: Building on Fifteen Years of Customer Success Salesforce Analyst Day 2014 Presentation (free, no opt in).
89% of business leaders believe big data will revolutionize business operations in the same way the Internet did. 85% believe that big data will dramatically change the way they do business. 79% agree that ‘companies that do not embrace Big Data will lose their competitive position and may even face extinction.’ 83% have pursued big data projects in order to seize a competitive edge. The top three areas where big data will make an impact in their operations include: impacting customer relationships (37%); redefining product development (26%); and changing the way operations is organized (15%).The following graphic compares the top six areas where big data is projected to have the greatest impact in organizations over the next five years. Source: Accenture, Big Success with Big Data: Executive Summary (free, no opt in).
Customer analytics (48%), operational analytics (21%), and fraud & compliance (21%) are the top three use cases for Big Data. Datameer’s analysis of the market also found that the global Hadoop market will grow from $1.5B in 2012 to $50.2B in 2020, and financial services, technology and telecommunications are the leading industries using big data solutions today. Source: Big Data: A Competitive Weapon for the Enterprise.
37% of Asia Pacific manufacturers are using Big Data and analytics technologies to improve production quality management. IDC found manufacturers in this region are relying on these technologies to reduce costs, increase productivity, and attract new customers. Source: Big Data and Analytics Core to Nex-Gen Manufacturing.
Supply chain visibility (56%), geo-location and mapping data (47%) and product traceability data (42%) are the top three potential areas of Big Data opportunity for supply chain management. Transport management, supply chain planning, & network modeling and optimization are the three most popular applications of Big Data in supply chain initiatives. Source: Supply Chain Report, February 2015.
Finding correlations across multiple disparate data sources (48%), predicting customer behavior (46%) and predicting product or services sales (40%) are the three factors driving interest in Big Data analytics. These and other fascinating findings from InformationWeek’s 2015 Analytics & BI Survey provide a glimpse into how enterprises are selecting analytics applications and platforms. Source: Information Week 2015 Analytics & BI Survey.
Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
From manufacturers looking to gain greater insights into streamlining production, reducing time-to-market and increasing product quality to financial services firms seeking to upsell clients, analytics is now essential for any business looking to stay competitive. Marketing is going through its own transformation, away from traditional tactics to analytics- and data-driven strategies that deliver measurable results.
Analytics and the insights they deliver are changing competitive dynamics daily by delivering greater acuity and focus. The high level of interest and hype surrounding analytics, Big Data and business intelligence (BI) is leading to a proliferation of market projections and forecasts, each providing a different perspective of these markets.
Presented below is a roundup of recent forecasts and market estimates:
The Advanced and Predictive Analytics (APA) software market is projected from grow from $2.2B in 2013 to $3.4B in 2018, attaining a 9.9% CAGR in the forecast period. The top 3 vendors in 2013 based on worldwide revenue were SAS ($768.3M, 35.4% market share), IBM ($370.3M, 17.1% market share) and Microsoft ($64.9M, 3% market share). IDC commented that simplified APA tools that provide less flexibility than standalone statistical models tools yet have more intuitive graphical user interfaces and easier-to-use features are fueling business analysts’ adoption. Source: http://www.idc.com/getdoc.jsp?containerId=249054
A.T. Kearney forecasts global spending on Big Data hardware, software and services will grow at a CAGR of 30% through 2018, reaching a total market size of $114B. The average business expects to spend $8M on big data-related initiatives this year. Source: Beyond Big: The Analytically Powered Organization.
Cloud-based Business Intelligence (BI) is projected to grow from $.75B in 2013 to $2.94B in 2018, attaining a CAGR of 31%.Redwood Capital’s recent Sector Report on Business Intelligence (free, no opt in) provides a thorough analysis of the current and future direction of BI. Redwood Capital segments the BI market into traditional, mobile, cloud and social business intelligence. The following two charts from the Sector Report on Business Intelligence illustrate how Redwood Capital sees the progression of the BI market through 2018.
Enterprises getting the most value out of analytics and BI have leaders that concentrate more on collaboration, instilling confidence in their teams, and creating an active analytics community, while laggards focus on technology alone. A.T. Kearney and Carnegie Mellon University recently surveyed 430 companies around the world, representing a wide range of geographies and industries, for the inaugural Leadership Excellence in Analytic Practices (LEAP) study. You can find the study here. The following is a graphic from the study comparing the characteristics of leaders and laggards’ strategies for building a culture of analytics excellence.
The worldwide market for Big Data related hardware, software and professional services is projected to reach $30B in 2014. Signals and System Telecom forecasts the market will attain a Compound Annual Growth Rate (CAGR) of 17% over the next 6 years. Signals and Systems Telecom’s report forecasts Big Data will be a $76B market by 2020. Source: http://www.researchandmarkets.com/research/s2t239/the_big_data
Big Data is projected to be a $28.5B market in 2014, growing to $50.1B in 2015 according to Wikkbon. Their report, Big Data Vendor Revenue and Market Forecast 2013-2017 is outstanding in its accuracy and depth of analysis. The following is a graphic from the study, illustrating Wikibon’s Big Data market forecast broken down by market component through 2017.
SAP, IBM, SAS, Microsoft, Oracle, Information Builders, MicroStrategy, and Actuate are market leaders in BI according to Forrester’s latest Wave analysis of BI platforms. Their report, The Forrester Wave™: Enterprise Business Intelligence Platforms, Q4 2013 (free PDF, no opt in, courtesy of SAS) provides a thorough analysis of 11 different BI software providers using the research firm’s 72-criteria evaluation methodology.
Amazon Web Services, Cloudera, Hortonworks, IBM, MapR Technologies, Pivotal Software, and Teradata are Big Data Hadoop market leaders according to Forrester’s latest Wave analysis of Hadoop Solutions. Their report, The Forrester Wave™: Big Data Hadoop Solutions, Q1 2014 (free PDF, no opt in, courtesy of MapR Technologies) provides a thorough analysis of nine different Big Data Hadoop software providers using the research firm’s 32-criteria evaluation methodology.
IDC forecasts the server market for high performance data analysis (HPDA) will grow at a 23.5% compound annual growth rate (CAGR) reaching $2.7B by 2018. In the same series of studies IDC forecasts the related storage market will expand to $1.6B also in 2018. HPDA is the term IDC created to describe the formative market for big data workloads using HPC. Source: http://www.idc.com/getdoc.jsp?containerId=prUS24938714
Global Big Data technology and services revenue will grow from $14.26B in 2014 to $23.76B in 2016, attaining a compound annual growth rate of 18.55%. These figures and a complete market analysis are available in IDC’s Worldwide Big Data Technology and Services 2012 – 2016 Forecast. You can download the full report here (free, no opt-in): Worldwide Big Data Technology and Services 2012 – 2016 Forecast.
Financial Services firms are projected to spend $6.4B in Big Data-related hardware, software and services in 2015, growing at a CAGR of 22% through 2020. Software and internet-related companies are projected to spend $2.8B in 2015, growing at a CAGR of 26% through 2020. These and other market forecasts and projections can be found in Bain & Company’s Insights Analysis, Big Data: The Organizational Challenge. An infographic of their research results are shown below.