We have all at sometime done some sort of experiment, from maybe from a young age as to see which cry and actions resulted in the reward of milk to test driving cars to find which is best suited to your needs before you buy it. These are experiments that produced results from things we have tried and may not have thought about it as developing an Experimental Mindset. In this article I am concentrating on how this applies to data.

Here are my notes from my research into the topic.

The main areas for an Experimental Mindset are:

• Learning
• Testing
• Evaluating

In order to constantly learn you need to be open to learning and develop your Growth Mindset. I have covered this in another blog so wont repeat here: Having the Right Digital Mindset: Business (Change, Agility and a Growth Mindset).

Having an Experimental Mindset is one of the key traits in being a Data Analyst or Data Scientist and it is not a new term. This has been around as long as the field of science and research has. These arena have developed methodologies that have been adopted and taken forward by many other areas such as business and computing that can be used for testing and evaluating.

At a high level this methodology can be shown as:

Observations –> Hypothesis –> Scientific Law

Overlaid with the areas for data this can be shown as:

Observations (Learning) –> Hypothesis (Testing) –> Scientific Law (Evaluating)

or as:

Observations (Data) –> Hypothesis (Product/Service) –> Scientific Law (Predictive Model)

Using this methodology, one of the more common types of Hypothesis Testing is A/B Testing. This sets out a framework for a simple controlled experiment against two versions (A and B) to look at the impact of changes to a thing or product. Some useful articles on A/B Testing are listed below that go into the details of it:

• A/B Testing
• A Beginner’s Guide To A/B Testing: An Introduction
• A Refresher on A/B Testing

Udacity host a course by Google on A/B testing.

There are some risks to A/B Testing that should be considered when reviewing the results:

• Sampling Bias
• Study Population
• Target Population
• Segmentation
• World Time Zones
• Target Population
• Data/Privacy Laws

I will go further into the realms of A/B testing in a later blog post.

Further Reading

• 5 Benefits of Adopting an Experimental Mindset
• A/B Testing
• A Beginner’s Guide To A/B Testing: An Introduction
• A Refresher on A/B Testing
• Comparison of Segmentation Approaches
• Design Thinking Mindsets for Human-Centered Design
• Embracing an Experimental Mindset
• Sampling Bias
• Sampling Bias
• Sampling bias: What is it and why does it matter?
• Simpson’s Paradox and segmentation: why analysis is crucial
• The Upside of an Experimental Mindset