Those who know Qlik could clearly see we were highlighting our unique Associative engine. With all of the data and relationships stored in-memory, users can explore in any direction and avoid waiting for SQL queries to run every time there’s another question. There are many great customer stories that show how the associative engine allowed them to see the whole story. I’ve found it interesting that users mention some of the most valuable insights involve uncovering data that unexpectedly was not related (e.g. didn’t realize that this region had no sales of product X in January) or even missing (ex. these cases were never followed-up because the case owner field was mistakenly left blank).
While the Associative Engine with in-memory processing is used daily by millions of users around the world, unlike other vendors, Qlik understood that a different approach was needed when it comes to Big Data sources. It would take more than just in-memory processing to effectively meet the needs of Big Data analytic use cases. That’s what drove creating the Qlik Associative Big Data Index, a product which offers an enhanced, scalable version of the associative engine that works its magic within massive data sources such as Hadoop to eliminate any data size limitations.
I was reminded of this when reading this recent article, which talked about important characteristics a BI tool needs to work with data lakes:
- A Scale-Out Distributed Architecture
- BI Processing Running Natively in the Data Lake
- Support for Multiple Data Sources
- Flexible Deployment Options
I believe there are several BI tools that meet these criteria, including the Qlik Associative Big Data Index. But what’s missing are the criteria of acceptable performance and enabling free-form exploration. Even when moving BI processing to the data lake, an SQL-based query tool must cache/aggregate SQL-based views to have any chance of achieving acceptable performance. And since they are only caching some of the data, one needs to guess what questions a user may ask. Go beyond these pre-defined boundaries and the user is faced with waiting while the BI tool undertakes the slow process of a new query against a massive repository. And even more time-consuming would be using SQL-based queries to uncover the unexpected non-related or missing data. With the democratization of data and more business users being called on to use analytics in their daily tasks – making people wait even longer is plainly unacceptable.
The Qlik Associative Big Data Index solves this problem by running the Qlik Associative Engine at the source, thus providing users with the same powerful and unrestricted data exploration experience of Qlik Sense to data sources of immense size. Only then can every hidden relationship be revealed and all insights be discovered to understand the whole story.
This article was written by Michael Distler and originally appeared here: https://blog.qlik.com/the-whole-big-story