HOW DO SKILLED DECISION MAKERS SOLVE HARD PROBLEMS? A PROTOCOL ANALYSIS OF THE BERLIN NUMERACY TEST

Abstract

The Berlin Numeracy Test is one of the strongest predictors of general decision making skill and risk literacy. Despite hundreds of studies linking test performance with decision quality, there is limited evidence on how people solve problems on the test itself. Are skilled decision makers better abstract reasoners who use calculations to solve hard math problems, or are they more like experts who use heuristics to develop a deeper conceptual understanding that can simplify problem solving? To answer this question, I conducted a Protocol analysis on verbalizations expressed during problem solving on the Berlin Numeracy Test (n = 160). Results revealed that most people only relied on formal calculations for easy problem solving, switching to heuristic-based strategies for difficult problems. Cognitive path modeling indicated that the relationship between skilled decision making and problem solving was fully mediated by heuristic use, which helped people understand problems in a way that eliminated the need for complex calculations. Results suggest that the predictive power of the Berlin Numeracy Test may be partly explained by this special property of its difficult questions (e.g., computational simplification via conceptual understanding). Results are interpreted in accordance with Skilled Decision Theory (Cokely et al., 2018), emphasizing the role of representative understanding in superior decision making and effective problem solving. Limitations and implications for measurement and training are briefly discussed.