From Research to ROI - New Method: How to Run Surveys (Addressing Response Biases & EDE)

You just surveyed your customers, and they love your new sustainable product idea. 90% say they are willing to pay a premium for it.

You’re about to invest millions in the launch. But can you trust what they told you?

Probably not.

This is a painful, expensive form of “Doing Data Wrong” that I see constantly in Marketing and HR departments. We rely on survey data that is fundamentally flawed, not because of a technical error, but because we ignore basic human psychology. The data isn’t wrong; it’s just not telling you the truth.

As Stefanie Stantcheva masterfully outlines in the Annual Review of Economics, a survey is not just a tool for collecting data; it's a process you design to create data. And if you design it naively, you will create biased, misleading data.

Let's look at our Marketing Department example. The survey about the sustainable product line failed because of two powerful response biases:

1. Social Desirability Bias (SDB): Of course people say they care about sustainability! It’s the socially correct answer. Respondents want to project a favorable image and avoid embarrassment, so they don’t reveal their actual, more price-sensitive attitudes.

2. Experimenter Demand Effect (EDE): The questions signaled what you wanted to hear. Respondents who feel the surveyor wants them to respond in a certain way will often do so to be helpful or agreeable.

   
   

So how do we get closer to the truth? Stantcheva’s work provides a clear playbook. Here are a few best practices:

The Right Way: Designing Surveys to Reveal the Truth

• Avoid Agree/Disagree Questions. This format is a magnet for Acquiescence Bias—the tendency to agree regardless of content. Instead of asking, "Do you agree it's important to buy sustainable products?", force a trade-off. Ask: "Which of the following is more important to you when buying coffee: a lower price or an official fair-trade certification?"

• Don't Reveal Your Purpose. To neutralize the Experimenter Demand Effect, provide limited and neutral information about your study's goal on the landing page. Instead of saying, "A survey about sustainable consumption," say, "A survey about consumer preferences." You can then ask respondents at the end what they thought the survey was about to detect potential bias.

  
  

• Guarantee Anonymity to Fight SDB. Online surveys have an edge because they remove face-to-face social pressure. Double down on this by explicitly and repeatedly reassuring respondents that their answers are confidential and anonymous, especially before sensitive questions.

  
  

My time leading teams at Ambev and launching ventures at Alura taught me that customer and employee feedback is critical, but only if you can trust it. We have to move beyond just asking questions and become architects of a truth-discovery process.

It is my mission to bring this level of academic rigor from top-tier research into the daily practice of business. This knowledge is not mine to keep.

If you are tired of making decisions on "feel-good" data and want to join a movement to get this right, subscribe to my email list. You’ll get more no-nonsense, research-backed insights to build a culture of real data quality.

And if you prefer to discuss your specific case directly and confidentialy, book a 20-minute, no-nonsense consultation with me. Let’s tackle it one-on-one.

#SurveyDesign #MarketResearch #HRAnalytics #ResponseBias #DataQuality #CustomerInsights #EmployeeExperience #MariaAliceMaia