POST CATEGORIES
February 6, 2014
Business Intelligence is evolving, and it’s not all about minor tweaks for today’s large-scale solutions. Current implementations are similar, the BI playing field is fairly level given the current state of products and expertise, but things are changing.
Last year I spent time with some data scientists at a government department, what I uncovered was surprising. The organization had all the traditional tools and struggled to perform the data discovery they wanted to do.
“80% of my day is spent ingesting data, and rarely the source data I truly want”, “the remaining time I have available is where I get to engage my mind and commence exploration of the analytical data.” Not surprising given the current state of solutions.
To start with, the data sources were limited, coming from the legacy BI systems, already ‘cleansed’ by filters, which were anonymously configured long ago. The data scientist had no idea what data was missing, sure it might have had errors a filter rule tripped and was dropped, but was it truly erroneous entry? We already knew that fields for location such as Ottawa could appear as Nepean, Gloucester, Kanata, or other identifiers such as CLLI codes.
The other challenge: adding ad hoc data. It was impossible to do so in a timely manner. Prior to even starting a limited data discovery effort, the project was doomed. Effectiveness could only be measured in statements such as “it’s better than we had before.” Clearly it was not inspiring this data scientist. Best effort analysis was not motivating, nor what upper management expected. Insights founded on facts were critical, and urgently needed to guide departmental decisions.
While this organization was government, the same can be said for industry.
Making BI look real time is time consuming for IT/IM teams. Clients have to know what questions they need to ask up front, carefully crafting an SOW, obtain approval and wait a few weeks for that limited answer, generating more questions.
BI has evolved. At RealDecoy, we enable clients with Oracle’s Endeca Information Discovery suite, breaking current BI paradigms. It’s inexpensive to purchase and deploy, it augments existing BI systems, and it’s a game changer for data scientists, product managers, and the IT/IM organizations.
The key benefits revolve around the BI engine, based on in-memory technology, along with an easy to configure data ingest tool kit and data exploration front end with several ‘out of the box’ visualizations. Current capabilities are impressive, and so is Oracle’s road map for the product. Self-service is a term I hear used to describe Endeca Information Discovery, with some assistance getting the system primed, most analysts can perform ETL / ingest functions with minimal training.
A company can augment their existing BI solution with a stand-alone agile BI information discovery platform, without fears of breaking the big system. Feed Endeca your current data sources, which can include suspected dirty data, add spreadsheets to link tables, unstructured data, and sit back and explore. This is exactly what data scientists dream of, more exploration and less housekeeping.
So, what’s this mean?
Right now your BI systems have you operating in a level playing field with competitors, you are likely staffing with analysts consumed with data ingest issues from legacy systems versus uncovering the subtleties that can change your corporate ROI. If you want to get to the root cause of warranty problems, operational issues, or customer retention and conversion rates in a timely manner to impact next quarters results, agile BI, aka, data discovery and exploration tools like Endeca will get you there.