By Dr. Ken Broda-Bahm:
For those of us in the social sciences, data is our currency. And of course, we would prefer that our window into attitudes and trends gives us an unbiased view. The reality, though, is that the "how you collected it" matters as much as the "what you collected." A simple survey, for example, can be administered by an interviewer (as in a telephone survey), or it can be self-administered by the respondent (as in an online survey). That latter "self-service" mode isn't new of course, since a paper survey is also self-administered. But as online survey approaches are swiftly moving to displace the traditional telephone survey, the question of influence exerted by the way the data is collected, the "mode effect," is becoming more important. To the extent that litigators rely on that kind of data -- for venue motions, community attitude surveys, focus groups, and mock trials -- then it's a question that matters to litigators as well.
A new Pew Research Center investigation focuses on that difference between telephone and online data collection. Looking at 3,003 survey respondents who answered the same questions either by telephone (interviewer administered) or online (self-administered), Pew concluded that mode differences are "fairly common, but typically not large." The mean difference in the answers obtained via the two methods averaged 5.5 percent across a broad set of 60 questions. That difference is nothing to sneeze at, and it is worth noting that for some of the questions the mode effect difference ranged as high as 18 percent. That by itself is a big deal. But the more important finding is that the differences weren't random, but followed a pattern. While we are tempted to wonder which, telephone or online, is the true answer and which is skewed, there probably is no good answer to that question. Instead, researchers and those who rely on that research need to keep those differences in mind. In this post, I will report on the Pew research on mode effect, and draw out some implications for litigators, including some advice on the differences between oral voir dire responses (interviewer administered) and supplemental juror questionnaire responses (self administered).