New Topic: Designing Business Experiments: A Roadmap
- Maria Alice Maia
- Sep 9, 2024
- 3 min read
Everyone wants to "test and learn." It's a phrase you hear in every boardroom. But the hard truth is that most business "experiments" are not experiments at all. They are unstructured observations, plagued by bias, that produce misleading results and burn capital.
They are a sophisticated form of "Doing Data Wrong."
I've seen this countless times, especially in my work with and for Consulting Firms, where the pressure to deliver data-backed recommendations is immense.
The Wrong Way (The "Fake Experiment"): A consulting team advises a retail client to test a new in-store display to increase sales of a specific product. To "test" it, they roll out the new display in their 10 best-performing urban stores and compare sales to the 10 worst-performing rural stores. They run the "test" for one week. The result? A beautiful PowerPoint deck showing a 50% lift, leading to a multi-million dollar national rollout.
This is analytical malpractice. The two groups of stores were never comparable (selection bias), and a single week of data is meaningless noise. The "insight" is worthless.
A real experiment isn't just a pilot. It's a rigorously designed system for creating truth. From my time as an executive driving growth at major companies to building my own startup, I've learned that you need a simple, non-negotiable roadmap.
The Right Way: A 5-Step Roadmap for Business Experiments
Before you spend the first dollar on a pilot, you must be able to answer these five questions:
What is our one, specific causal question? Not "Does the display work?" but "Does the new display cause a statistically significant increase in the average transaction value for shoppers who walk past it, compared to the old display?" Be precise.
What is our unit of randomization, and how will we randomize? You can't randomize individual shoppers for a store display, so your unit is the store. You must then take your pool of comparable stores and randomly assign half to get the new display (treatment) and half to keep the old one (control). This is the only way to eliminate selection bias.
How will we prevent interference (SUTVA)? Are your control stores right next to your treatment stores? Could social media buzz from one location contaminate another? Acknowledging and designing for these spillovers is critical for a clean result.
Do we have enough statistical power? How long do you need to run the test? How many stores do you need in each group? A quick power analysis will tell you if you even have a chance of detecting the effect you're looking for. Running an underpowered test is the same as not running one at all.
How will we measure and interpret the results? The outcome isn't a simple "yes" or "no." It's an estimate with uncertainty. A good result sounds like: "We are 95% confident the new display causes an increase in average transaction value of between $2.50 and $4.00." That is an answer you can take to the bank.
Demand this level of rigor. As a leader, your job is to ask these five questions. As a data professional or consultant, your job is to have the answers ready before the test begins.
This is how you move from "we think this works" to "we have proven the causal impact." It’s the core of a real data-driven culture, and I am passionate about helping leaders build it. This knowledge isn't mine to keep.
If you’re ready to stop hoping and start proving, join my movement. Subscribe to my email list for a no-nonsense, research-backed guide to creating real value with data.
And if you’re planning an experiment and want to pressure-test your design, book a 20-minute, no-nonsense call with me. Let’s make sure you’re set up for success.
