Is Your AI a Genius or a Savant? Using Adaptive Testing to Find Out

You want to drive adoption of a critical new AI tool. Your data team builds a model and tells you to target the obvious evangelists—the engineers already tagged with "Machine Learning" and "Python." What if the real key to viral adoption lies with a completely different group of people your model can't even see?

This is a classic case of “Doing Data Wrong.” It stems from analyzing people as isolated individuals and ignoring the most powerful force in any organization: the network. A standard classification model will find the direct predictors, the “self-related features,” as Hu et al. (2024) call them. It tells you that AI experts are likely to adopt an AI tool. That’s a true, but shallow, insight.

This approach fails because it's blind to the second, more powerful force of influence: the

“network-related features”. These are attributes that don’t directly predict a behavior, but instead shape the network of connections, which in turn drives the behavior of others. Ignoring the network leads to an "incorrect feature set and suboptimal prediction accuracy".

The pivot is to stop analyzing target segments as if they exist in a vacuum and start using methods designed for the networked reality of modern work. The "Aha!" moment comes when we deploy a method that can untangle these two distinct forces. The Pseudo-Likelihood Ratio Screening (PLR-SIS) procedure developed by Hu et al. is a model-free way to do exactly this, identifying both types of influential features from a sea of ultra-high-dimensional data.

My analogy is understanding a restaurant's popularity. A simple analysis looks at the menu (the "self-related features") and concludes, "The food is good." A network-aware analysis looks at the customers (the "network-related features") and discovers, "The city's most influential food critics eat here." The restaurant's success isn't just about its own attributes; it's about the influence flowing through its network. PLR-SIS lets you analyze both the menu and the network of critics.

Let’s return to our tech company trying to drive adoption of their new AI coding assistant. They re-run their analysis using PLR-SIS. The results are a revelation:

1. Self-Related Features: "Python" and "ML" are confirmed as direct predictors of adoption. No surprise.

2. Network-Related Features: The model uncovers a completely unexpected and far more powerful predictor: the tag “Documentation Enthusiast.”

Why? Engineers with this tag aren't necessarily the top AI experts. But they are meticulous, trusted collaborators whose code reviews are highly sought after. They are central nodes in the company's knowledge-sharing network. When theyadopt the AI tool and write guides on how to use it effectively, their endorsement creates a cascade of adoption through their vast network of colleagues. Their influence is indirect but massive. The company just discovered its true, hidden channel for viral change.

For Managers & Tech Leaders: Your organization is a network, not a list of employees. Stop making strategic decisions with models that are blind to this fact. You need to ask your data teams: "Have we separated the direct drivers of behavior from the network-mediated ones? Our obvious experts are self-related predictors, but who are our hidden, network-related influencers?"

For my fellow Tech & Data Professionals: When you have network data, using a standard screening method that assumes independence is a form of model misspecification. Your feature screening process must be network-aware. Methods like PLR-SIS evaluate a feature's importance based on both its direct relationship with the outcome and its role in predicting the network structure itself. This is how you uncover the non-obvious dynamics of influence.

This isn't just about tech adoption. It’s a more sophisticated way of understanding how ideas, behaviors, and culture truly spread. For decades, I've worked to bridge the gap between the rigor of the academic world and the pragmatism of the P&L. This is a perfect example of where advanced, cutting-edge research provides a direct, actionable, and more profitable path forward. This knowledge isn’t mine to keep.

Let's start mapping the hidden wiring of our organizations to drive real, lasting change.

Evaluating your AI with a static benchmark is like giving every student the same test, regardless of their level. To learn the smarter, more efficient methods for understanding your model's true capabilities, join our email list. If you need to build a robust and nuanced evaluation framework for your AI systems, schedule a 20-minute consultation.