My initial reaction to these numbers was different than most. If these numbers had been produced by my firm, I would immediately assume we had made a mistake and send them back to an analyst to double check.
Take a look at the zig-zaggy pattern on the community involvement question, for instance. That’s the only pro-social item on here that went up in the previous 21 years before the 2019 survey, but it’s declined by more than half in just four years without any clear inciting event explaining why. One could maybe speculate that people locked inside during the pandemic did not go out and do volunteer work, but a drop of 35 points in four years is implausible on its face.
The point here is not that the Wall Street Journal and NORC released bad data. The Journal is one of the more thoughtful media sponsors of polling and NORC is the premier survey data-collection organization in the country. Rather, the dramatically different results we see from 2019 and 2023 are because the data was collected differently. The March 2023 survey was collected via NORC’s Amerispeak, an extremely high-quality online panel. In the fine print below the chart, we can see that data from previous waves was collected via telephone survey.
Why should this matter? After all, panelists on NORC’s Amerispeak panel are recruited probabilistically, using the same random sampling methods as a telephone survey. It’s more expensive, but when when you want online data that looks as close as possible to the old gold-standard telephone survey data, you use NORC’s Amerispeak.
But survey mode still matters. Surveying the exact same types of respondents online and over the phone will yield different results. And it matters most for exactly the kinds of values questions that the Journal asked in its survey.
The basic idea is this: if I’m speaking to another human being over the phone, I am much more likely to answer in ways that make me look like an upstanding citizen, one who is patriotic and values community involvement. My answers to the same questions online will probably be more honest, since the format is impersonal and anonymous. So, the 2023 survey probably does a better job at revealing the true state of patriotism, religiosity, community involvement, and so forth. The problem is that the data from previous waves were inflated by social desirability bias—and can’t be trended with the current data to generate a neat-and-tidy viral chart like this.
