Using cognitive interviewing techniques in workplace settings

Surveys are a commonly used measurement method in organisational research. They're popularity can be attributed to their relative economy, especially when self-administered, and the ease with which they can be controlled from remote locations using mail, email or telephone. Further, surveys are the only method of observation that can describe the characteristics of a large population. This increases the likelihood of achieving a large sample, thus increasing the power of any statistical analysis undertaken using the subsequent data, increasing the likelihood of achieving statistically significant results even when analysing multiple variables, and improving the generalizability of those results.

However, surveys are not without their weaknesses. Survey respondents may experience problems with comprehension as they fail to understand what the question Is asking, interpret the questions in divergent ways, experience difficulty recalling information with the desired degree of accuracy, or offer the wrong type of response as the questions fail to elicit the type of information the survey was designed to draw out. Respondents may also feel social pressure to respond in a way which is not the most honest or accurate, or they may consider the survey format cumbersome, leading them to deliberately skip items. These issues have the potential to diminish the survey Instrument's reliability, its ability to yield the same findings on repeated applications, and its validity, or the extent to which it accurately reflects or assesses the concept the researcher is attempting to measure.

For these reasons the initial planning, development' and testing stage of the survey design process Is crucial. One technique that has become Increasingly prominent when designing and developing surveys is cognitive interviewing (Beatty & Willis, 2007). Cognitive interviewing was developed by survey methodologists and psychologists to evaluate and prevent sources of error in survey questionnaires. It does so by focusing on respondent's cognitive processes as they answer survey questions; the covert, usually hidden processes as well as the overt, more readily observable processes. Cognitive interviewing helps identify response errors the respondent may commit by misinterpreting the question, failing to recall crucial information, or reporting with social desirability response bias (Desimone & Kerstin Carlson Le, 2004). Further, cognitive interviewing allows the evaluation of the overall questionnaire for structural or logical problems. This information can then be used to modify the survey Instrument, reduce measurement error, and improve its validity.

In this chapter we will briefly discuss the theory behind the use of cognitive interviewing in survey development and the available techniques before demonstrating, with real data, how this approach improved an existing survey instrument designed to assess the costs of ill-health in the working population. Specifically, cognitive interviewing was used to pilot a survey instrument within a small sample of managers. By providing in-depth information and background theory about the survey being developed, and the importance of the concepts it is attempting to measure, we aim to highlight the value of using cognitive interviewing techniques during survey development. We also aim to demortstrate the value of such techniques within a management sample.