From Research to ROI: New Method - Difference-in-Differences with Variation in Treatment Timing (Goodman-Bacon Decomposition)
- Maria Alice Maia
- Jan 27
- 2 min read
You launched your new customer loyalty program in the Americas in Year 1 and in Europe in Year 2. You run a standard Difference-in-Differences (DiD) model to measure the "average" impact on customer spending. The result is small and not statistically significant.
You’re about to recommend killing the program.
But what if your model is lying to you?
This is a critical "Doing Data Wrong" scenario that has invalidated countless studies. When you have a staggered rollout—where different groups get a treatment at different times—the standard two-way fixed effects (TWFE) DiD model can be dangerously misleading.
As groundbreaking research by Andrew Goodman-Bacon shows, the single "average" effect from a TWFE model is a Frankenstein's monster—a weighted average of every possible 2-group, 2-period comparison in your data.
This includes an extremely problematic type of comparison. Let's look at our Travel and Tourism example:
The "Forbidden Comparison" Trap
To estimate the program's effect on your European customers in Year 2, the standard DiD model doesn't just use your never-treated customers as a control. It also makes a "forbidden comparison": it compares the newly-treated European customers to the American customers, who have already been in the loyalty program for a year.
This is a logical disaster.
You are using a group whose behavior is already changing due to the treatment as your baseline for what "would have happened anyway." If the loyalty program's effect grows over time (as it should!), then the American customers' spending is already trending upward because of the program. Using them as a "control" will make the European launch look less effective than it truly is.
This can lead to the infamous negative weighting problem, where a genuinely positive effect gets averaged with a contaminated comparison, producing a final estimate that is biased towards zero, or is even negative.
The Right Way: Decompose, then Estimate
The Goodman-Bacon decomposition is a powerful diagnostic tool that opens the black box of your DiD model. It shows you exactly what comparisons are being made and what weight each one gets. If you find that a large portion of your estimate comes from these "forbidden comparisons," you know your result is unreliable.
This diagnostic is the necessary first step before using modern, heterogeneity-robust estimators (from researchers like Callaway & Sant’Anna, Sun & Abraham, etc.) that are designed to only make clean comparisons, ensuring you get a credible result.
As an executive who has managed multi-market strategies at companies like Grupo Águia, I know that a single, flawed "average" is useless. I need to understand the effect on the Year 1 cohort and the Year 2 cohort separately and see how those effects evolve.
My mission is to bring these crucial insights from the research frontier to business practice. We have to stop using broken models. This knowledge is not mine to keep.
If you’re ready to move beyond the simple (and flawed) DiD playbook and learn how to analyze your programs with rigor, join my movement. Subscribe to my email list.
And if you're analyzing a staggered rollout and want to ensure you're not making forbidden comparisons, book a 20-minute, no-nonsense consultation with me. Let's diagnose your model together.
