Skip to content

Posts from the ‘Hadoop’ Category

Oct 14

Using Search Analytics To See Into Gartner’s $232B Big Data Forecast

By combining search analytics and the latest Gartner forecast on big data published last week, it’s possible to get a glimpse into this areas’ highest growth industry sectors.  Big data is consistently a leading search term on Gartner.com, which is the basis of the twelve months of data used for the analysis.

In addition, data from Gartner’s latest report, Big Data Drives Rapid Changes in Infrastructure and $232 Billion in IT Spending Through 2016 by Mark A. Beyer, John-David Lovelock, Dan Sommer, and Merv Adrian is also used.  These authors have done a great job of explaining how big data is rapidly emerging as a market force, not just a single market unto itself.  This distinction pervades their analysis and the following table showing Total IT Spending Driven by Big Data reflects the composite market approach.  Use cases from enterprise software spending, storage management, IT services, social media and search forecasts are the basis of the Enterprise Software Spending for Specified Sub-Markets Forecast.  Social Media Analytics are the basis of the Social Media Revenue Worldwide forecast.

Additional Take-Aways

  • Enterprise software spending for specified sub-markets will attain a 16.65% compound annual growth rate (CAGR) in revenue from 2011 to 2016.
  • Attaining a 96.77% CAGR from 2011 through 2016, Social Media Revenue Is one of the primary use case catalysts of this latest forecast.
  • Big Data IT Services Spending will attain a 10.20% CAGR from 2011 to 2016.
  • $29B will be spent on big data throughout 2012 by IT departments.  Of this figure, $5.5B will be for software sales and the balance for IT services.
  • Gartner is projecting a 45% per year average growth rate for social media, social network analysis and content analysis from 2011 to 2016.
  • Gartner projects a 20 times ratio of IT Services to Software in the short term, dropping as this market matures and more expertise is available.
  • By 2020, big data functionality will be part of the baseline of enterprise software, with enterprise vendors enhancing the value of their applications with it.
  • Organizations are already replacing early implementations of big data solutions – and Gartner is projecting this will continue through 2020.
  • By 2016 spending on Application Infrastructure and Middleware becomes one of the most dominant for big data in Enterprise Software-Specified Sub Markets.

  • $232B is projected to be sold in total across all categories in the forecast from 2011 to 2016. From $24.4B in 2011 to $43.7B in 2016, this presents a 12.42% CAGR in total market growth.

Search Analytics and Big Data

Big data is continually one of the top terms search on Gartner.com, and over the last twelve months, this trend has accelerated.  The following time series graph shows the weekly number of inquiries Gartner clients have made, with the red line being the logarithmic trend.

Banking (25%), Services (15%) and Manufacturing (15%) are the three most active industries in making inquiries about big data to Gartner over the last twelve months.  The majority of these are large organizations (63%) located in North America (59%) and Europe (19%).

What unifies all of these industries from a big data standpoint is how critical the stability of their customer relationships are to their business models.  Banks have become famous for bad service and according to the American Customer Satisfaction Index (ACSI) have shown anemic growth in customer satisfaction in the latest period measured, 2010 to 2011.  The potential for using big data to becoming more attuned to customer expectations and deliver more effective customer experiences in this and all services industries shows great upside.

Bottom line: Companies struggling with flat or dropping rankings on the ACSI need to consider big data strategies based on structured and unstructured customer data.  In adopting this strategy the potential exists to drastically improve customer satisfaction, loyalty, and ultimately profits.

Sep 16

4 Comments

Why CIOs Are Quickly Prioritizing Analytics, Cloud and Mobile

Customers are quickly reinventing how they choose to learn about new products, keep current on existing ones, and stay loyal to those brands they most value.  The best-run companies are all over this, orchestrating their IT strategies to be as responsive as possible.

The luxury of long technology evaluation cycles, introspective analysis of systems, and long deployment timeframes are giving way to rapid deployments and systems designed for accuracy and speed.

CIOs need to be just as strong at strategic planning and execution as they are at technology.  Many are quickly prioritizing analytics, cloud and mobile strategies to stay in step with their rapidly changing customer bases.  This is especially true for those companies with less than $1B in sales, as analytics, cloud computing and mobility can be combined to compete very effectively against their much bigger rivals.

What’s Driving CIOs – A Look At Technology Priorities

Gartner’s annual survey of CIOs includes 2,300 respondents located in 44 countries, competing in all major industries.  As of the last annual survey, the three-highest rated priorities for investment from 2012 to 2015 included Analytics and Business Intelligence (BI), Mobile Technologies and Cloud Computing.

Source: From the Gartner Report Market Insight: Technology Opens Up Opportunities in SMB Vertical Markets September 6, 2012 by Christine Arcaris, Jeffrey Roster

 

How Industries Prioritize Analytics, Cloud and Mobile  

When  these priorities are analyzed across eight key industries, patterns emerge showing how the  communications, media and services (CMS) and manufacturing industries have the highest immediate growth potential for mobility (Next 2 years).  In Big Data/BI, Financial Services is projected to be the fastest-developing industry and in Cloud computing, CMS and Government.

In analyzing this and related data, a profile of early adopter enterprises emerges.  These are companies who are based on knowledge-intensive business models, have created and excel at running virtual organization structures, rely on mobility to connect with and build relationships with customers, and have deep analytics expertise.  In short, their business models take the best of what mobility, Big Data/BI and cloud computing have to offer and align it to their strategic plans and programs.  The following figure, Vertical Industry Growth by Technology Over the Next Five Years, shows the prioritization and relative growth by industry.

Source: From the Gartner Report Market Insight: Technology Opens Up Opportunities in SMB Vertical Markets September 6, 2012 by Christine Arcaris, Jeffrey Roster

How Mobility Could Emerge As the Trojan Horse of Enterprise Software

Bring Your Own Device (BYOD), the rapid ascent of enterprise application stores, and the high expectations customers have of continual mobile app usability and performance improvements are just three of many factors driving mobility growth.

Just as significant is the success many mid-tier companies are having in competing with their larger, more globally known rivals using mobile-based Customer Relationship Management (CRM), warranty management, service and spare parts procurement strategies.  What smaller competitors lack in breadth they are more than making up for in speed and responsiveness.   Gartner’s IT Market Clock for Enterprise Mobility, 2012 captures how mobility is changing the nature of competition.

Source: IT Market Clock for Enterprise Mobility, 2012 Published: 10 September 2012 Analyst(s): Monica Basso

 

Bottom Line – By excelling at the orchestration of analytics, cloud and mobile, enterprises can differentiate where it matters most – by delivering an excellent customer experience.  Mobility can emerge as an enterprise Trojan Horse because it unleashes accuracy, precision and speed into customer-facing processes that larger, complacent competitors may have overlooked.

Aug 15

Roundup of Big Data Forecasts and Market Estimates, 2012

From the best-known companies in enterprise software to start-ups, everyone is jumping on the big data bandwagon.

The potential of big data to bring insights and intelligence into enterprises is a strong motivator, where managers are constantly looking for the competitive edge to win in their chosen  markets.  With so much potential to provide enterprises with enhanced analytics, insights and intelligence, it is understandable why this area has such high expectations – and hype – associated with it.

Given the potential big data has to reorder an enterprise and make it more competitive and profitable, it’s understandable why there are so many forecasts and market analyses being done today.  The following is a roundup of the latest big data forecasts and market estimates recently published:

  • As of last month, Gartner had received 12,000 searches over the last twelve months for the term “big data” with the pace increasing.
  • In Hype Cycle for Big Data, 2012, Gartner states that Column-Store DBMS, Cloud Computing, In-Memory Database Management Systems will be the three most transformational technologies in the next five years.  Gartner goes on to predict that Complex Event Processing, Content Analytics, Context-Enriched Services, Hybrid Cloud Computing, Information Capabilities Framework and Telematics round out the technologies the research firm considers transformational.  The Hype Cycle for Big Data is shown below:

  • Predictive modeling is gaining momentum with property and casualty (P&C) companies who are using them to support claims analysis, CRM, risk management, pricing and actuarial workflows, quoting, and underwriting. Web-based quoting systems and pricing optimization strategies are benefiting from investments in predictive modeling as well.   The Priority Matrix for Big Data, 2012 is shown below:

  • Social content is the fastest growing category of new content in the enterprise and will eventually attain 20% market penetration.   Gartner defines social content as unstructured data created, edited and published on corporate blogs, communication and collaboration platforms, in addition to external platforms including Facebook, LinkedIn, Twitter, YouTube and a myriad of others.
  • Gartner reports that 45% as sales management teams identify sales analytics as a priority to help them understand sales performance, market conditions and opportunities.
  • Over 80% of Web Analytics solutions are delivered via Software-as-a-Service (SaaS).  Gartner goes on to estimate that over 90% of the total available market for Web Analytics are already using some form of tools and that Google reported 10 million registrations for Google Analytics alone.  Google also reports 200,000 active users of their free Analytics application.  Gartner also states that the majority of the customers for these systems use two or more Web analytics applications, and less than 50% use the advanced functions including data warehousing, advanced reporting and higher-end customer segmentation features.
  • In the report Market Trends: Big Data Opportunities in Vertical Industries, the following heat map by industry shows that from a volume of data perspective, Banking and Securities, Communications, Media and Services, Government, and Manufacturing and Natural Resources have the greatest potential opportunity for Big Data.

  • Big data: The next frontier for innovation, competition, and productivity is available for download from the McKinsey Global Institute for free.  This is 156 page document authored by McKinsey researchers is excellent.  While it was published last year (June, 2011), if you’re following big data, download a copy as much of the research is still relevant.  McKinsey includes extensive analysis of how big data can deliver value in a manufacturing value chains for example, which is shown below:

Aug 3

8 Comments

Gartner Releases Their Hype Cycle for Cloud Computing, 2012

Enterprises are beginning to change their buying behaviors based on the deployment speed, economics and customization that cloud-based technologies provide.  Gartner cautions however that enterprises are far from abandoning their on-premise models and applications entirely for the cloud.

Based on an analysis of the Gartner Hype Cycle for Cloud Computing, 2012, the best results are being attained by enterprises that focus on a very specific strategy and look to cloud-based technologies to accelerate their performance.  Leading with a strategic framework of goals and objectives increases the probability of cloud-based platform success. Those enterprises that look to cloud platforms only for cost reduction miss out on their full potential.

The Hype Cycle for Cloud Computing, 2012 is shown below:

Cloudwashing and Inflated Enterprise Expectations

While the hype surrounding cloud computing may have peaked, cloudwashing continues to cause confusion and inflated expectations with enterprise buyers.  This just slows down sales cycles, when more straightforward selling could lead to more pilots, sales and a potentially larger market. Cloud vendors who have the expertise gained from delivering cloud platforms on time, under budget, with customer references showing results are starting to overtake those that using cloudwashing as part of their selling strategies.

Additional take-aways from the Gartner Hype Cycle for Cloud Computing include the following:

  • Cloud Email is expected to have a 10% adoption rate in enterprises by 2014, down from the 20% Gartner had forecasted in previous Hype Cycles.  This represents modest growth as the adoption rate of this category had been between 5 and 6% in 2011.
  • Big Data will deliver transformational benefits to enterprises within 2 to 5 years, and by 2015 will enable enterprises adopting this technology to outperform competitors by 20% in every available financial metric.  Gartner defines Big Data as including large volumes processed in streams, in addition to batch.  Integral to Big Data is an extensible services framework that can deploy processing to the data or bring data to the process workflow itself. Gartner also includes more than one asset type of data in their definition, including structured and unstructured content.  The Priority Matrix for Cloud Computing, 2012 is shown below:

  • Master Data Management (MDM) Solutions in the Cloud and Hybrid IT are included in this hype cycle for the first time in 2012.  Gartner reports that MDM Solutions in the Cloud is getting additional interest from Enterprise buyers as part of a continual upward trend of interest in MDM overall.  Dominant vendors in this emerging area include Cognizant, Data Scout, IBM, Informatica, Oracle and Orchestra Networks, are among those with MDM-in-the-cloud solutions.
  • PaaS continues to be one of the most misunderstood aspects of cloud platforms.  The widening gap between enterprise expectations and experiences is most prevalent in this market.  Gartner claims this is attributable to the relatively narrow middleware functions delivered and the consolidation fo vendors and service providers in this market.
  • By 2014 the Personal Cloud will have replaced the personal computer as the center of user’s digital lives.
  • Private Cloud Computing is among the highest interest areas across all cloud computing according to Gartner, with 75% of respondents in Gartner polls saying they plan to pursue a strategy in this area by 2014.  Pilot and production deployments are in process across many different enterprises today, with one of the major goals being the evaluation of virtualization-driven value and benefits.
  • SaaS is rapidly gaining adoption in enterprises, leading Gartner to forecast more than 50% of enterprises will have some form of SaaS-based application strategy by 2015.  Factors driving this adoption are the high priority enterprises are putting on customer relationships, gaining greater insights through analytics, overcoming IT- and capital budget-based limitations, and aligning IT more efficiently to strategic goals.
  • More than 50% of all virtualization workloads are based on the x86 architecture. This is expected to increase to 75% by 2015.  Gartner reports this is a disruptive innovation which is changing the relationship between IT and enterprise where service levels and usage can be tracked.

Bottom line: Gartner’s latest Hype Cycle for Cloud Computing  shows that when cloud-based platforms are aligned with well-defined strategic initiatives and line-of-business objectives, they deliver valuable contributions to an enterprise.  It also shows how Infrastructure as a Service (IaaS) and Platform as a Service (PaaS) are the catalysts of long-term market growth.  The following slide from the presentation  High-Tech Tuesday Webinar: Gartner Worldwide IT Spending Forecast, 2Q12 Update: Cloud Is the Silver Lining (free for download) also makes this point.

Mar 20

1 Comment

Gartner Search Analytics Shows Spike in Hadoop Inquiries in 2012 – Good News For CRM

Hadoop was one of the most-searched terms on Gartner’s website in 2011 through 2012, spiking to 601.8% over the last twelve months alone.  Additional insights from the Search Analytics on Hadoop include the following:

  • 27% of all inquiries are from banking, finance and insurance industries, followed by manufacturing (14%), government (13%), services (10%) and healthcare (8%).
  • North America (75.9%) and EMEA (13.5%) are the two most dominant geographies in terms of query volume.
  • Here is the trend line from Gartner Search Analytics:

What’s driving Hadoop’s meteoric rise in searches is a combination of industry hype about big data, CIOs getting serious about using Hadoop distributions that minimize time and risk yet deliver value, and the dominant role Amazon is playing in bringing Hadoop into the cloud.  Today Amazon offers Elastic MapReduce as a Web Service that relies on a hosted Hadoop framework running the Elastic Compute Cloud (EC2) in conjunction with Amazon Simple Storage Service (S2).

Microsoft also scored a major hiring win this week announcing that Raghu Ramakrishnan, former chief scientist for three divisions of Yahoo is now with Microsoft. Raghu is now a technical fellow working in the Server and Tools Business (STB).  He’ll focus on big data and integration to STB platforms.  Big Data on Azure will accelerate now with him on-board.

Hadoop’s Potentially Galvanizing Effect on CRM and Social CRM Analytics

The quickening pace of Hadoop adoption in the enterprise is good news for CRM and especially social CRM. Analytics and Business Intelligence (BI) are the “glue” that unify CRM and keep it in context. One of Hadoop’s greatest potential contributions is the analysis, categorization and use of unstructured content.  Marketing and sales won’t have to run three or four systems to gain insights into customer data, they can run a single analytics platform that fuels the entire selling cycle and lifetime customer value chain of their businesses.  Hadoop has the potential to make unstructured content more meaningful while also reporting the impact of customer insights on financial performance, profitability and lifetime customer value.

Translating terabytes of customer, sales, services and partner data into meaningful analytics and business intelligence (BI) is emerging as a priority for CIOs, who are sharing responsibility for driving top-line revenue growth.   Hadoop shows potential to be the “glue” or galvanizing technology base that unifies all CRM and Social CRM strategies.

To get a perspective on how fast Hadoop is being evaluated and adopted it’s useful to look at the Hype Cycle for Data Management, the latest edition published July, 2011.   This is another indicator of how quickly Hadoop and big data are gaining in terms of CIO mindshare.  Big Data and extreme information management are on the technology Trigger area of the hype cycle.  The Hype Cycle for Data Management is shown below:

Bottom line:  CRM and Social CRM will benefit more than any other area of an enterprise as Hadoop’s adoption continues to accelerate.  CIOs are increasingly called upon to be strategists, and with the ability to translate terabytes of data into strategies that deliver dollars, look for Hadoop’s contributions to drive top-line revenue growth.

Dec 22

2 Comments

Data Science Shows Potential To Redefine Cloud-based Analytics

[tweetmeme source=@LouisColumbus only_single=false]

The emerging field of data science is a fascinating one that has major implications on the potential of cloud-based analytics, CRM, search, supply chain management and logistics.

Instead of relying purely on latent semantic indexing or the Google PageRank algorithm to define relevance of a search, data science techniques analyze content and its context to determine relevance.  Google today looks at the content of a page; data science considers its surrounding data and relevance.

Earlier this month TechCrunch published the blog post Marissa Mayer’s Next Big Thing: “Contextual Discovery” — Google Results Without Search.  The techniques of contextual discovery Google is experimenting with rely on a very rapid aggregation and transforming of data, which are part of the methodologies of data science.   When Google moves fully into contextual discovery the potential exists for cloud-based analytics, CRM, search, supply chain management and logistics to be completely revolutionized by solving the big data problems associated with each of these areas.

In CRM, this would mean finally being able to access external and internal content (including the massive amount of data on social networks), aggregate the data, and transform it into meaningful analysis.  The vision of social CRM would be realized once data science serves as the catalyst of contextual search or as Google calls it, contextual discovery.

Exploring Data Science

Two of the best blog posts are both from O’Reilly Radar on the emerging topic of data science.  What is data science? By Mike Loukides and Six months after “What is data science?” by Mac Slocum O’Reilly Radar are worth reading and giving some serious thought to.  O’Reilly also has also created a free report titled What is Data Science, which can be downloaded here.

Authors Mike Loukides and Mac Slocum set the foundation for how transformational data science has the potential of being by concentrating on the nascent area of data products.  A data product is the result of accessing, aggregating and transforming content regardless of its location – and capturing data on its attributes – not just the data itself. Both authors point to reference systems and guided reference engines on e-commerce sites as just the beginning.  Yet after reading their assessments and listening to Roger Magoulas, O’Reilly’s Director of Research, interviewed about data science below there are many more potential uses of this evolving area.

Potential Impact of Data Science on Analytics

The blog posts by Mike Loukides and Mac Slocum go into detail explaining how each area of data science is in varying levels of maturity.  After reading these over and considering the big data problems in cloud-based analytics, CRM, search, supply chain management and logistics, the following methodology starts to make sense:

Access – For data science to realize its full potential there needs to be a technology layer that provides for real-time access to structured and unstructured content both within and outside an enterprise.  More than a traditional Enterprise Application Integration (EAI) layer the technologies driving data access need to selectively pull all available content from every unstructured and structured data source available.  Mike Loukides mentions Google Goggles and how MapReduce has made this application possible.  Hadoop as a means to create greater access across federated content has much potential in this phase as well.

Aggregate – Called data conditioning by Mike Loukides, the aggregation phase is where contextual discovery happens.  This could be accomplished through contextual search filters, taxonomies defined by specific alerts, or the use of the MapReduce and Hadoop query and relevance tools in use today.

Transform – Where Hadoop could be used for driving data analysis and as Mike Loukides calls this level of analysis, data jiujitsu.   Examples are mentioned by both Mike Loukides and Mac Slocum including the Hadoop Online Prototype (HOP), which does real-time stream processing and several others.  The impact of the access, aggregate and transform methodology on visualization is available at Flowing Data, one of the best sites on the Web for seeing how MapReduce, Hadoop and other data science-related techniques are taking on massive amounts of data and delivering insights.

Conclusion

Solving the big data problems of social media monitoring, sentiment analysis, forming a scalable platform for social CRM, integrating CRM, supply chain management and logistics data to demand management – and tying all of these areas to financial performance – is potentially achievable with data science.  Deployed as a cloud-based platform opens up even greater potential for getting the most use of social networks, free data sources, and third-party databases than is possible today.

Be sure to check out the video below of Roger Magoulas, O’Reilly’s Director of Research, where he was interviewed about data science.

Article links:

What is data science? By Mike Loukides  O’Reilly Radar
Six months after “What is data science?”  by Mac Slocum O’Reilly Radar

Nov 12

Hadoop Predicted To Be More Disruptive than Linux

[tweetmeme source=@LouisColumbus only_single=false]

Abhishek Mehta is Managing Director for Big Data and Analytics for Bank of America, and serves as Executive in Residence at MIT Media Lab.  SiliconAngle.tv founder John Furrier and Wikibon co-founder David Vellante interviewed him at Hadoop World last month.  Abhishek sees Hadoop as being more disruptive than Linux, and leading to the formation of data factories.  He also sees Hadoop giving programmers greater freedom to concentrate on creating algorithms that solve much larger, more complex problems than is possible today.

Here is a quote from the interview:
“So these data factories are going to emerge as the new drivers of innovation of a massive revolution that will change fundamentally how business models extract value, because data is going to be, is the core asset in a multitude of industries.”

At just under 30 minutes, this is a fascinating look into the future of Hadoop.

Transcript of the interview by Bert Latamore

Source attribution: SiliconAngle.tv video