Assessing theoretical conclusions with blinded inference to investigate a potential inference crisis

Starns, JJ; Cataldo, AM; Rotello, CM; Annis, J; Aschenbrenner, A; Broder, A; Cox, G; Criss, A; Curl, RA; Dobbins, IG; Dunn, J; Enam, T; Evans, NJ; Farrell, S; Fraundorf, SH; Gronlund, SD; Heathcote, A; Heck, DW; Hicks, JL; Huff, MJ; Kellen, D; Key, KN; Kilic, A; Klauer, KC; Kraemer, KR; Leite, FP; Lloyd, ME; Malejka, S; Mason, A; McAdoo, RM; McDonough, IM; Michael, RB; Mickes, L; Mizrack, E; Morgan, DP; Mueller, ST; Osth, A; Reynolds, A; Seale-Carlisle, TM; Singmann, H; Sloane, JF; Smith, AM; Tillman, G; van Ravenzwaaij, D; Weidemann, CT; Wells, L; White, CN; Wilson, J

eCite Digital Repository

Assessing theoretical conclusions with blinded inference to investigate a potential inference crisis

Citation

Starns, JJ and Cataldo, AM and Rotello, CM and Annis, J and Aschenbrenner, A and Broder, A and Cox, G and Criss, A and Curl, RA and Dobbins, IG and Dunn, J and Enam, T and Evans, NJ and Farrell, S and Fraundorf, SH and Gronlund, SD and Heathcote, A and Heck, DW and Hicks, JL and Huff, MJ and Kellen, D and Key, KN and Kilic, A and Klauer, KC and Kraemer, KR and Leite, FP and Lloyd, ME and Malejka, S and Mason, A and McAdoo, RM and McDonough, IM and Michael, RB and Mickes, L and Mizrack, E and Morgan, DP and Mueller, ST and Osth, A and Reynolds, A and Seale-Carlisle, TM and Singmann, H and Sloane, JF and Smith, AM and Tillman, G and van Ravenzwaaij, D and Weidemann, CT and Wells, L and White, CN and Wilson, J, Assessing theoretical conclusions with blinded inference to investigate a potential inference crisis, Advances in Methods and Practices in Psychological Science, 2, (4) pp. 335-349. ISSN 2515-2459 (2019) [Refereed Article]

Preview

PDF (Post-print)
967Kb

Preview

PDF (Post-print supplement)
317Kb

Copyright Statement

DOI: doi:10.1177/2515245919869583

Abstract

Scientific advances across a range of disciplines hinge on our ability to make inferences about unobservable theoretical entities based on empirical data patterns. Accurate inferences rely on both a) discovering valid, replicable data patterns, and b) accurately 25 interpreting those patterns in terms of their implications for theoretical constructs. The replication crisis in science has led to widespread efforts to improve the reliability of research findings, but comparatively little attention has been devoted to the validity of inferences based on those findings. Using an example from cognitive psychology, we demonstrate a blinded inference paradigm for assessing the quality of theoretical inferences from data. Our results 30 reveal substantial variability in expert judgements on the very same data, hinting at a possible inference crisis.

Item Details

Item Type:	Refereed Article
Keywords:	memory, modeling, Bayesian methods, bootstrap, metascience, blinded inference, open data
Research Division:	Psychology
Research Group:	Cognitive and computational psychology
Research Field:	Decision making
Objective Division:	Expanding Knowledge
Objective Group:	Expanding knowledge
Objective Field:	Expanding knowledge in psychology
UTAS Author:	Heathcote, A (Professor Andrew Heathcote)
ID Code:	134802
Year Published:	2019
Deposited By:	Psychology
Deposited On:	2019-09-05
Last Modified:	2020-04-28
Downloads:	15 View Download Statistics

Repository Staff Only: item control page