From Research to ROI: New Topic - Pre-Analysis Plans (PAPs) & Credibility in Causal Inference
- Maria Alice Maia
- Mar 3, 2025
- 2 min read
Your consulting team just delivered a brilliant, data-driven insight that will change your strategy.
But did they find the truth, or did they just torture the data until it confessed?
This is a critical question, and it points to one of the most pervasive forms of "Doing Data Wrong": the post-hoc narrative, often born from a "fishing expedition" in the data. It’s a huge problem in the consulting world and internal analytics teams alike.
The "Doing Data Wrong" Scenario: The Fishing Expedition A client's customer churn is up. The consulting team gets a massive dataset and runs hundreds of tests, slicing the data by every imaginable dimension. Lo and behold, they find a "statistically significant" correlation: customers who use a new app feature are churning at a higher rate. A beautiful deck is built, a compelling story is told, and the recommendation is made: kill the feature.
The problem? With hundreds of tests, you're almost guaranteed to find something significant by pure random chance. The consultants didn't have a hypothesis; they found a p-value and built a story around it. The strategy is based on noise.
The Right Way: The Pre-Analysis Plan (PAP) The antidote to data dredging is a simple, powerful tool borrowed from the scientific method: the Pre-Analysis Plan (PAP).
A PAP is a public commitment made before the analysis begins. It's a time-stamped document that specifies:
Your exact hypothesis.
The primary outcome metrics you will use to test it.
The specific statistical model you will run.
Your rules for handling outliers or missing data.
Why is this so crucial? As recent research by Jonathan Roth on pre-testing shows, even seemingly innocuous choices made after seeing the data can severely bias your results and invalidate your conclusions. A PAP protects your analysis from your own human biases and the immense pressure to "find something interesting." It forces intellectual discipline.
It separates analysis from astrology.
When I was at FALCONI, and now as a managing partner at Auge Growth, credibility is the only currency that matters. A Pre-Analysis Plan is the ultimate mark of professional rigor. It tells the client, "We had a sharp hypothesis, we tested it with a pre-agreed methodology, and here is the honest result."
My mission is to bring this level of scientific integrity to the business world. We have to move beyond just finding insights and commit to a process that ensures those insights are credible. This knowledge is not mine to keep.
The Playbook:
For Managers: When you commission an important analysis, ask for a one-page PAP. It forces your team—and you—to have extreme clarity on the business question before the work starts.
For Tech Pros & Consultants: Propose a PAP for your next major project. It demonstrates confidence and integrity. It will differentiate you from the storytellers and mark you as a true professional.
If you’re ready to build more credible, trustworthy data practices, join my movement. Subscribe to my email list.
And if you want to design a more rigorous analytical plan for a critical business question, book a 20-minute, no-nonsense consultation with me. Let's build your PAP together.
