Kiran Lakkaraju, Ph.D., Sandia National Laboratories, Senior Member of Technical Staff, Department of Cognitive Modeling 

Title: The Rewards and Risks of Online, Social Experiments: Development, issues and Current Status of the "Controlled, Large Online Social Experimentation" (CLOSE) Platform

Abstract: The ubiquity of computers, along with the prevalence of access to the Internet, has opened up a new subject pool for psychology studies. In fact, in the last decade there has been a rise in using the Internet as a "Virtual Laboratory" to conduct experiments. Amazon Mechanical Turk is one of the best-known platforms to conduct these types of experiments. The large and diverse potential subject pool available online has several benefits. Large experiments done cheaply, and quickly, with diverse subjects pools are possible. Online experiments can also probe into the rise of online interaction influencing real world behavior, as events like the Arab Spring can attest to. However, online experiments are not without their issues and problems. The lack of environmental controls, the ability for subjects to easily deceive the experimenter, and an incentive structure that rewards rapid completion can subtly and not-so subtly influence the results of an experiment. In this talk, I will discuss online social experimentation and it's promises and pitfalls. I will describe the "Controlled, Large Online Social Experimentation" (CLOSE) system which is a new platform to conduct online, longitudinal, social experiments. I will describe the problems we faced in development of the system and the solutions we came up with to address these issues.

Finally, I will highlight some experiments that are currently being developed for the platform and early results on testing the reliability of subjects obtained from Amazon Mechanical Turk.