It was an interesting week last week. Gartner and Strata had their big smack down with competing trade shows during the same week, and based on our attendance at both events, you could not have had two more different perspectives on the industry. Gartner held their Data & Analytics Summit in Grapevine, Texas while O’Reilly ( with Cloudera) held Strata in San Jose,
To give you a sense of the events if you are not familiar, Gartner’s event is mostly Gartner analysts speaking about their perspective on the market with a few customer case studies and vendor pitches mixed in. It is usually a perspective that is targeting the mainstream of the market and not looking to project too much into the future of technology. It is targeting an executive audience and is therefore more about the business impact of technology
Strata, on the other hand is much more focused about what is coming. The audience is much more technology based with a lot of actual implementers in the audience. Many of the presentations get into the technical detail of how someone actually achieved some specific result for a specific project. In contrast to Gartner, this event is much more about what is coming and and where the future might be heading.
On one hand, at Strata, you would think that everyone already had a fully deployed Hadoop cluster and had it in production given the intense focus on machine learning and AI. Given our experience as a tech-company helping main stream companies automate their big data deployments, this is a bit of an exaggeration. Clearly, the mainstream market needs help to simplify the complexity of big data. However, that perspective was still in stark contrast with Gartner where one analyst put forward the idea that data lakes could/should be built on Teradata. I find this point a little hard to accept given Teradata’s shrinking revenue and the continually shrinking market share of all of the traditional DW vendors. Just like a tiger can’t trade its strips for spots, traditional DW’s can’t magically become highly flexible and agile platforms that support all new kinds of semi-structured and unstructured data types.
We also noticed some progression in thinking as well this year. In past years a lot of the Strata attendees shared a perspective on vendors that they didn’t want or need software tools or platforms to simplify away the complexity of Hadoop because they could just as easily code things by hand. I am happy to say that attitude changed quite a big this year with a large number of discussions about how organizations were looking for automation to reduce the complexity of developing and deploying on Hadoop.
Actually, we also noticed a lot of interest in discussing automation at Gartner as well. In fact, the concept of automation was one area of great consistency across both events. In general however, our perspective at Infoworks was that both events were operating a bit at the extreme. Strata was looking too far in the future, while Gartner was holding on too much to the past.
The reality is somewhere in between. Big data technologies like Hadoop are clearly the forward-looking platforms that will enable machine learning (ML) and AI as part of an analytics technology stack. Legacy data warehouses will not somehow evolve to be a different beast that can handle the new kinds of data nor will they cost effectively deal with ML and AI. At the same time, big data needs to mature to a point where it doesn’t take an army of experts to deploy it (Infoworks can help with that!). Focusing on ML and AI when you can’t even implement big data into a repeatable production environment for even the most basic use cases is also unrealistic. For most organizations, it is getting out over the tips of their skis.
The reality sits somewhere in between. There is simply too much venture capital investment in the big data space to believe that it will just disappear. All of that VC money will ultimately close the complexity and maturity gap and new big data technologies will augment the existing DW systems and may even replace many of them with a more cost-effective approach. But all of this will take time.
In the meantime, if you get to attend both Strata and the Gartner conferences next year when they are not during the same week, you will get two very different, yet interesting perspectives that are at least both worth considering.