I recently took a detour in my career, from core IT related work to BI and data analysis.
I still employ a lot of the techniques and skills i’ve learnt in IT, such as SQL and understanding data pipelines, and still tap into some of my more dysfunctional personality traits that have served me well over the years, such as catastrophizing and thinking in permutations. But this new world has introduced me to a different way of thinking and looking at problems thats softening some of my hard (geeky) edges. As you can imagine this does not come naturally to me, but I hope I can use this post to communicate what i’ve learnt so far.
This is primarily written for those who are coming from the back of the business to the front. Hopefully you’ll get to see how those pesky business decisions you hate so much actually get made.
What do the numbers say? … err 4,528?
One of my initial punches to the gut, and big dose of imposter syndrome, was being asked to extract numbers from our database system, my boss taking one look at the mass of numbers and exclaiming “ah yes, this particular change in behaviour has been caused by customers ABC doing XYC”, and i’d take a look and only see.. well a sheet of numbers. How is he able to tell stories from these numbers?
Step 1: Hypothesis
After struggling with this for a while, a colleague pointed out that the individuals making data requests usually already had a hypothesis in mind. The data was simply being requested as a way to validate the hypothesis. Without the hypothesis, a lot of the times, the numbers you extract are just.. well.. numbers.
This really brings out the Artsy side of Business Intelligence. The need to have an imagination, coupled with common sense ofcourse. You need to come up with probable scenarios, and have the confidence to treat them seriously until the data proves them otherwise.
So that sounds easy enough. But how do you go about coming up with a hypothesis?
Data requests usually come from two sources:
- Looking for causes of an observed problem
- Looking at opportunities to exploit
1.0 Causes
1.1 Events
If something bad happens or a bad trend begins, start by looking at the date the trend began.
Look for events that happened at or around that time. Look at internal events (Product / service launches, product sunsets, operational activities, hirings, firings, ad campaigns etc) as well as external events (Government announcements, new entrants in the market, products being released by competition etc)
By finding corresponding events, you will have something to attribute the trend to, or at the very least, a starting point in forming a hypothesis that you can chase down with further data.
1.2 Seasonality
Business, sadly, isn’t a hard science, and a lot of what happens is based on human behaviour. Human lives are.. complicated, but some of their ups and downs are predictable. Stands to say, some trends in your business, both good and bad can simply be attributed to seasonality. Towards the end of the calendar year, customers tend to buy more due to end of year bonuses and the general festive spirit. The same customers spend significantly less at the beginning of the year due to obligations such as school fees. Purchases might be higher at the end of a month than at the beginning. Weekend behaviour might be significantly different from weekday behaviour.
A way to check if a given trend is simply seasonality at play is to check if the same trend/behaviour can be seen in the same season at other points in time.
1.3 Composition
When you’re investigating something going wrong, another approach you can take is to determine their composition.
Isolate the customers or products in the wrong. Collect various attributes about them at an individual level, then check what attributes they have the most in common.
I emphasise on “the most” because people like me coming from 1 or 0 backgrounds will want ALL individual entries to fulfil a requirement in order to conclusively say its a factor. But in the business world 60% or even 50% is sometimes good enough to flag an attribute for further investigation.
e.g. A particular set of customers have stopped spending as much as they usually do. Get a list of these customers and whatever attributes you have of them (Favourite product, location, membership type, touch points used etc), and see which ones they have the most in common. Say 60% of them actually purchase your product from one particular store, or 80% buy one particular product over another. By seeing what the majority of this mass of problematic customers/products have in common, you have the bones to begin forming a hypothesis. Now, your imagination has to take the lead. Are they getting bad service from the store? Is there something wrong with that particular product? Were there any events tied to that attribute recently?
You can now chase down each hypothesis with data to see which one actually sticks. This might mean mining more data, making phone calls, checking with other departments etc.
To Be Continued…
