Patrick Ruffini: Why this extremely viral poll result might not be real

Read the full analysis by Patrick Ruffini in The Intersection on Substack

This chart from the March Wall Street Journal/NORC poll was destined to go viral. It shows that the values we think of as defining America—patriotism, having children, religion, community, involvement—are falling off a cliff. And the only thing that people now value more? Money. The decline in old-fashioned values has accelerated over the last few years. In 2019, 61 percent said patriotism was very important to them. Today, that number is 38 percent. Also, just four years ago, 62 percent said the same of community involvement. Now, that number is less than half that: 27 percent.

These findings fit into a declinist narrative we are already predisposed to believe. And that’s what makes this chart so powerful and compelling. It’s exceptionally easy to draw sweeping conclusions from it. For example.

My initial reaction to these numbers was different than most. If these numbers had been produced by my firm, I would immediately assume we had made a mistake and send them back to an analyst to double check.

Take a look at the zig-zaggy pattern on the community involvement question, for instance. That’s the only pro-social item on here that went up in the previous 21 years before the 2019 survey, but it’s declined by more than half in just four years without any clear inciting event explaining why. One could maybe speculate that people locked inside during the pandemic did not go out and do volunteer work, but a drop of 35 points in four years is implausible on its face.

The point here is not that the Wall Street Journal and NORC released bad data. The Journal is one of the more thoughtful media sponsors of polling and NORC is the premier survey data-collection organization in the country. Rather, the dramatically different results we see from 2019 and 2023 are because the data was collected differently. The March 2023 survey was collected via NORC’s Amerispeak, an extremely high-quality online panel. In the fine print below the chart, we can see that data from previous waves was collected via telephone survey.

Why should this matter? After all, panelists on NORC’s Amerispeak panel are recruited probabilistically, using the same random sampling methods as a telephone survey. It’s more expensive, but when when you want online data that looks as close as possible to the old gold-standard telephone survey data, you use NORC’s Amerispeak.

But survey mode still matters. Surveying the exact same types of respondents online and over the phone will yield different results. And it matters most for exactly the kinds of values questions that the Journal asked in its survey.

The basic idea is this: if I’m speaking to another human being over the phone, I am much more likely to answer in ways that make me look like an upstanding citizen, one who is patriotic and values community involvement. My answers to the same questions online will probably be more honest, since the format is impersonal and anonymous. So, the 2023 survey probably does a better job at revealing the true state of patriotism, religiosity, community involvement, and so forth. The problem is that the data from previous waves were inflated by social desirability bias—and can’t be trended with the current data to generate a neat-and-tidy viral chart like this.

Head over to Substack to read more of the analysis by Patrick Ruffini in The Intersection

Contact Us

Let's Talk!