Are you making your decisions based off evidence or experience?

July 26th, 2018

You may have heard the old saying ‘Curiosity Killed the Cat’ which is commonly used to refer to the dangers of unnecessary investigation or experimentation. Did you know, that that is not the entire saying? The full saying is actually “Curiosity killed the cat, but satisfaction brought it back.” The saying encourages people to be curious. How does this relate to Data Literacy and data-driven decision making? You cannot ask too many questions when making decisions using data. In fact, it is the most important part of the process and the one most often overlooked.







With the vast amount of data available to organizations today, people who are tasked with making decisions need to harness that data for evidence and insights. But data-driven decision making goes beyond just gathering the data and visualizing it. That is where the process really begins. There are many various models out there that can help with this. Three of my favorites are the OODA loop, the DIKW Pyramid, and the Scientific Method. Rather than discussing these models in detail, we will pull out the key elements which are critical to this process.

Step 1: Formulate a Focused Question

Too often, people are asking generic questions like “How successful was my campaign?” While that may be a valid business question, it is not something that you can answer without asking more questions and getting more context. Compared to what? Over what time period? You must first define what is important to your decisions and strategy. Understanding what question(s) you really need to answer is critical before you begin.

Step 2: Search for the best available data

Once you have your question(s), you then need to acquire the evidence to help you answer them and ultimately act on it and make a decision. This evidence typically starts with raw data, and then may go through various stages of transformations, aggregations and similar and probably leads to a visualization or set of visualizations that can be used to support your decision.

One model, the DIKW pyramid, talks about how to go from data to information to knowledge and eventually wisdom. If you connect enough of the raw data together, you will find patterns and turn that data into knowledge and wisdom. Another model, called Knowledge crystallization, explains the desire is to get the most compact description possible for a set of data relative to some task, without removing information critical to its execution. Knowledge that is proven useful and objective is kept and the rest is removed. This process explains how a person will gather data to answer their question, then try to make sense of it by constructing a schema for it, and then package it up into some view, or visualization to help see the insights.

This process of constructing a representational framework of your data into a visualization is one method of augmented intelligence. The visualization is enhancing the user's cognitive abilities and human intelligence. Nowadays, this process needs to go beyond just visualizations to having applications help you find patterns in your data like potential correlations to investigate and provide you with starting points to analyze.

Step 3: Critically appraise the data

Where the process typically fails is organizations stop at the previous step. Data and visualizations are not the end point, but just the starting point in the decision making process. This step is similar to the 'orient' stage of the OODA Loop and the 'hypothesis testing' stage of the Scientific Method. Here you are tasked with appraising the data. Make sure that you really understand the data you have, and what data you may not have. One of the tenants of Data Literacy is to be able to argue with your data. This is where that step comes into practice.

Do I have all the data needed? This is not a simple yes and no, but you need to also make sure you are not working with any unconscious bias about your data. There are many types of bias that impact decision making. Here we will highlight confirmation bias and survivor bias. Confirmation bias is the tendency to seek out information that supports your beliefs/hypothesis and ignore information that contradicts them. This is one reason I like using the scientific method, as it forces you to test your hypothesis and try to disprove it rather than trying to prove it. Survivor bias is another type of cognitive bias which occurs when you are making a decision using only part of the data based off past successes, and not based off past failures. This could include only using data from projects that succeeded rather than all projects, as an example.

Step 4: Integrate the evidence with your professional expertise and apply

While you have used data as your evidence up till this step to build a strong case for your decision, it is important to not forget the human element of this. Experience is just as valuable as evidence. When people make decisions, they are typically constructing mental models of the reality of the situation, including their assumptions, beliefs, experiences, and biases. Many times this mental model is done subconsciously. Since these mental models include people’s perceptions, and not necessarily hard evidence, we need to work on making those models visible to the decision maker so that they are not making incorrect inferences from them

Take time during this process to really understand your mental model and try to contradict your thoughts to poke holes in it. Two people with different mental models can see the same situation very differently. To overcome this at the organization level, you should work to create a culture of continuous learning and knowledge sharing. You should drive collaboration through reflective conversations. You should promote innovation and focus not just inward but also externally at trends in the industry. Many times organizations are so focused inward, they fail to see the external trends that will have a huge impact on their decisions.

Step 5: Monitor the outcome

Once you make your decision, it is important to assess the outcomes and either adjust accordingly, or add this to your previous experiences which will aid you in your decision making in the future. This is at the heart of an organization that fosters continuous learning and innovation.