The distorting effects of deciding to stop sampling information

This paper asks how strategies of information sampling are affected by a learner’s goal. Based on a theoretical analysis and two behavioral experiments, we show that learning goals have a crucial impact on the decision of when to stop sampling. This decision, in turn, affects the statistical properties (e.g. average values, or standard deviations) of the data collected under different goals. Specifically, we find that sampling with the goal of making a binary choice can introduce a correlation between the average value of a sample and its size (the number of values sampled). Across multiple rounds of sampling, this has the potential of biasing learn- ers’ inferences about the underlying process that generated the samples, specifically if learners ignore sample size when making these inferences. We find that people are indeed biased in this way and make different inferences about the same data-generating process when sampling with different learning goals. These findings highlight yet another danger of inferring general patterns from samples of evidence the learner had a hand in collecting.