The effects of mode on answers in probability-based mixed mode online panel research: evidence and matching methods for controlling self-selection effect in a quasi-experimental design

Online probability-based panels often apply two or more data collection modes to cover both online and offline populations. They do so with the aim of obtaining results that are more representative of the population of interest, in most cases the general population, than Web mode only. This study investigates mode effects in two different surveys - a probability-based quasi-experimental web-push survey and a probability-based online panel study. For both surveys the same questionnaire including items with nationally representative benchmarks was used. The aim of this study is to identify differences in answers in three different modes, the online and two offline (mail and telephone) modes, to establish the degree of measurement errors due to mixing modes and to present evidence about the most suitable combination of online-offline modes in online panel research from a measurement error perspective.

In this paper, we provide evidence that mail mode is most associated with satisficing - it generates slightly more item nonresponse and non-differentiation, and limited primacy after reducing self-selection bias with matching. We also identified some recency and extreme category responding in telephone surveys, potential social desirability associated with interviewer-administered telephone mode, and indication of the presence of question format effect. After controlling for self-selection to mode using different matching solutions, there were fewer differences which we initially assigned to self-selection, but we could not reduce all bias. With exact and coarsened exact matching, we could reduce slightly more bias, and also identify mode effects which were initially revoked by that self-selection of mode. Propensity scores matching proved to decrease self-selection bias, but it also decreased the ability to identify mode effects. Mahalanobis distance matching was not as successful in reducing bias, but it also did not negatively affect post-matching measurement effect assessment.

Due to the potential presence of self-selection effects in a quasi-experimental design, we also tested five different approaches to control for the absence of random assignment of respondents to modes: using socio-demographic controls in regression models (no matching), propensity score matching, Mahalanobis distance matching, exact matching, and coarsened exact matching. We compared the results on mode-effects after controlling for self-selection with those techniques.

It would appear that mode self-selection effects or sample composition effect were a more significant source of differences in the distributions of response variables than measurement mode effects such as satisficing and social desirability or question format effects. That is why we encourage all researchers studying or adjusting for mode effects in a quasi-experimental design and without access to similar single-mode data to use a matching method, preferably (coarsened) exact matching, to reduce self-selection bias.