Survey Mode, Population and the Party ID Distribution

It is widely understood that the population sampled, whether adults, registered or likely voters, has some effect on the distribution of partisanship. But mode differences among live interview, automated phone and internet data collection also shift the distributions. Pew polls are shown as a high quality live interview reference compared to all live interview polls.

PIDDensity

The most prominent effect is on the size of independents. Both automated phone and internet modes produce substantially smaller estimates of Independents than do live interview phone polls (which include cell phone strata.)

Independents are on average 34.8% of live phone samples but 28.0% and 27.6% of automated phone and internet samples respectively.  This is in line with what we might expect to find if the smaller response rate to automated polls and the “volunteer” samples of internet polls are correlated with political interest and likely with partisan intensity.

Where the independents go in the different modes varies a bit. Republicans are substantially more numerous in automated samples (33.8% vs 26.7% in live phone) but not so in internet samples (25.8%).  Meanwhile Democrats are more common in both automated phone (38.1% vs 34.0% for live phone) and (more modestly) for internet (35.1%).

One might suspect that mode and population sampled are highly correlated and that this accounts for some of the differences. Happily, there is enough variation in population within mode to estimate the effects of both mode and population.

PartyIDModePopNs

The estimates below are from a linear regression of the partisan make up of each sample on mode and population with a polynomial in time to capture trend effects (not shown.)

The data are from HuffPost/Pollster’s collection of 982 polls that reported their partisan make up between September 3, 2008 and April 27, 2014. (Two “mixed mode” polls were deleted. No account has been taken of changing methodologies, such as reported blending of automated phone with internet samples, the detailed timing of which is not available.)

Estimated effects are given below. All are statistically significant except the Rep/Internet coefficient.

ModePopEffectsonPID

 

While both partisan camps gain when we move away from the live interview mode, there is a modest pro-Republican advantage with larger gains. Independents shrink in each case.

While partisan bias is an obvious potential concern, the higher involvement and less independent nature of automated modes may also have more subtle effects, from over stating turnout and candidate awareness to possibly reducing the apparent impact of dynamic factors as more partisan samples are less likely to be moved by events or advertising.

A final note is to compare the distribution of each partisan category between the Pew polls alone and all live interview polls. The considerably greater spread of the distribution for all live polls is a vivid example of house and question wording effects. The very wide spread of the independent distribution in particular is notable. We may assume there is a clear distribution of partisanship (as in the trends below) but the spread around these trends is quite large. Certainty as to the sizes of each partisan camp is illusory. Central tendency should not ignore spread.

PartyIDwDots