Personalized Feedback Versus Money: The Effect On Reliability Of Subjective Data In Online Experimental Platforms

We compared the data reliability on a subjective task from two platforms: Amazon's Mechanical Turk (MTurk) and LabintheWild. MTurk incentivizes participants with financial compensation while LabintheWild provides participants with personalized feedback. LabintheWild was found to produce higher data reliability than MTurk. Our findings suggested that online experiment platforms providing feedback in exchange for study participation could produce more reliable data in subjective preference tasks than those offering financial compensation.