Beyond the Hype: A Leader's Checklist for AI Trustworthiness

You just invested millions in a new country-wide sales program. The initial data says it's working… on average. But what if that 'average' is hiding a terrible truth about your investment?

This is a scenario I see constantly, and it’s a dangerous form of “Doing Data Wrong.” A consumer goods company rolls out a new, expensive program to increase its market share across multiple domains—let's say, different customer segments. These are multivariate, continuous treatments. After six months, they run an analysis and find a positive, statistically significant "average" impact on project success scores. The leadership team signs off on the budget for next year.

The problem? They've made a multi-million-dollar decision based on a single, misleading number.

Ignoring the heterogeneity of that program's effect is a catastrophic analytical error. As Shin et al. (2025) note, this can "mask substantial effects of the treatment on specific subgroups of the population". You might have a program that’s wildly successful for a high-end customer segment but a complete waste of OPEX for your other customers. The "average" effect hides both the triumph and the failure, leading to profoundly flawed resource allocation.

The pivot isn't to stop measuring. It's to ask a much, much better question: "For WHOM does this program work, and WHY?"

This is the essence of analyzing Treatment Effect Heterogeneity. The "Aha!" moment comes from realizing that we now have the statistical tools to answer this question systematically. The research from Shin et al. introduces a powerful framework for this, including Multivariate Treatment Effect Variable Importance Measures (MTE-VIM). This is a method designed specifically to identify which characteristics of a segment (the covariates) are the most significant drivers of a program's success or failure.

Think of it like a master chef tasting a complex sauce. A novice taster says, "It's good." The master chef, however, can identify every single ingredient—the salt, the acid, the umami—and understands precisely which component is making the dish truly special. MTE-VIM gives your data team that master chef's palate, allowing them to pinpoint the exact "ingredients" of your employee base that interact with your training program to create success.

Let's return to our consulting firm. By applying a model that measures heterogeneity and calculates MTE-VIMs, they look beyond the average. The analysis reveals that the program's success is overwhelmingly driven by a single covariate: customer income.

The program is a game-changer for customer segments with less than two minimum wage incom, but it shows zero to negligible impact on anyone with more than five. The company can now make an intelligent, targeted decision: make the program mandatory and expanded for all low-end sales departments, and make it optional for others, saving thousands of SG&A and millions in opportunity cost. This mirrors the real-world findings from the study, which found the negative health effects of pollutants were significantly exacerbated by factors like socioeconomic status, race, and age. The truth was in the data all along; they just weren't asking the right question.

For Managers: Stop accepting the tyranny of the average. Ask your data teams: "What is the Variance of the Treatment Effect (VTE)? Which segments are driving this result? Show me the Conditional Average Treatment Effect (CATE) function for our key employee demographics, not just the single Average Treatment Effect (ATE)."

For my fellow Tech & Data Leaders: Our value now lies in quantifying and explaining heterogeneity. We must build models that can handle multivariate, continuous treatments and identify the key drivers of their varied effects. This means decomposing our regression surfaces to isolate the main effects of covariates and exposures from their interaction effects, allowing for proper regularization and identification of the true causal modifiers . This is how we move from being data reporters to indispensable strategic advisors.

Finding the average effect is table stakes. The future of data-driven strategy is about precision, personalization, and profound understanding of causality in all its varied forms.

After decades as an executive at firms like Itaú and Ambev and building my own company, I've seen firsthand that the biggest competitive advantages come from the deepest insights. The knowledge being forged in top academic journals isn't meant to stay there. My mission is to translate it into tangible, balance-sheet-level value. This knowledge is not mine to keep.

Let's build organizations that operate with precision, not just power.

Trust is not a feature; it's the foundation. To build AI systems and data strategies that are genuinely reliable, you need a framework for rigor. Subscribe to get evidence-based guides on AI governance and causal inference delivered to your inbox. Ready to build a comprehensive strategy for your team? Schedule a 20-minute consultation.