Earlier this month, Nature Human Behaviour, a subsidiary of the world's leading scientific publication outlet, published an article on how equilibrium price and quantity predictions from foundational economic theory are highly reproducible across thousands of Competitive Market (Continuous Double Auction) experiments played on MobLab. This was the first time thousands of standardized experiments across different courses, schools, and nations were systematically examined and validated against economic theories. We are proud of this achievement and want to share why this study is important for both educators and scientists and how we got it done.
Background
The field of economics has been increasingly receptive to the experimental approach that the natural sciences have been executing for centuries. Seminal experiments conducted in the 1960s by Nobel laureate Vernon Smith (Chapman University) and Charlie Plott (the William D. Hacker Professor of Economics and Political Science, Caltech) demonstrated that markets do converge to the theoretically predicted equilibrium price, in both classroom and controlled laboratory environments and validated experimental economics. For experiments to be deemed successful, they need to yield consistent results repeatedly, and economics lagged the natural sciences in this respect until recently.
By nature, social and economic interactions often occur in networks which have expanded exponentially in recent decades due to technological growth. From the emergence of computer labs on campuses in the late 1990s to the proliferation of smart mobile devices into almost every household in the past few years, more students than ever before are now participating in some form of online learning. MobLab, established in 2011, has maintained a clear objective to make experiments an integral part of teaching and learning the social sciences. Leveraging the latest educational technology, we have built a robust and scalable experimental platform where standardized experiments can be easily set up, run, monitored, and analyzed.
As the platform continues to gain traction, the different types of games and the number of experiments performed have grown exponentially along with the behavioral data from student participants. Since then, the scientists at MobLab have wondered what the aggregate data would inform us about the field of economics and the social sciences in general. One key feature of the platform that makes meta-analysis of the data particularly compelling is the consistency and stability of the experiment design. Across thousands of classrooms, all participants engaged in the experiment scenario through a standardized graphical interface, and many instructors utilized the standard instructions and parameterization that are provided. This cloud-based approach offers researchers a new level of control and reliability that is closer to that of laboratory experiments than traditional classroom experiments.
Still, there would inevitably be uncontrolled differences arising from instructors’ teaching style and course structures. Additionally, there was an open question of whether behavior would differ significantly between a class where an instructor incentivizes students through course credits that correlate with experiment performance and one that does not. With all these variances, how does our data compare to the published results from the lab? Our scientists wanted to find out, and with those questions in mind, we started looking for a collaborative opportunity to assemble an interdisciplinary team to take a deep dive into our data.
Team Journey
The collaboration started with a casual conversation with Colin Camerer, the Robert Kirby Professor of Behavioral Economics at Caltech, when he visited the MobLab office a few years ago. The large volume and diversity of the dataset immediately caught his attention. Soon after, his post-doc Taisuke Imai, who just completed his Ph.D. at Caltech, initiated the data analysis of the competitive market game with Robert Gazzale, then lead economist at MobLab.
The work was put on hiatus with Imai leaving for an academic position in Germany and Gazzale returning to the University of Toronto but regained traction in 2018 when a cohort of economists visited MobLab. Among this group was Po-Hsuan Lin, a young and aspiring intern who was instrumental in not only reigniting the project but also pushing it across the finish line. Lin, an econ graduate who just finished a year at UC Berkeley as a visiting student, came highly recommended to MobLab by his undergrad advisor, Joseph Tao-yi Wang, the John O. Ledyard Endowed Chair Professor of Experimental Economics at National Taiwan University (NTU). With his arrival, Lin brought a flair for curiosity, confidence, and persistence in addition to his talents in economic analysis and data science. Despite being the youngest, a nucleus of experts quickly formed around Lin, including: Si-Yuan Kong, a behavioral economist at MobLab; Alexander L. Brown of Texas A&M University and visiting senior economist at MobLab; Joseph Tao-yi Wang of NTU and a visiting summer senior economist at MobLab in 2018-19; and Stephanie W. Wang, a professor of economics at the University of Pittsburgh and co-founder of MobLab. With Camerer as the principal investigator, the dream team had been assembled.
Key Results
Among several key findings, one prime result is that this study successfully demonstrated the strength of the underlying economic theory in predicting the dynamics of supply and demand interactions in classroom across the globe. This was most obvious in our Competitive Market (Continuous Double Auction) experiment, where students consistently converged to an “equilibrium” price that maximized buyer-seller profit as predicted by theory. Repeatedly achieving the equilibrium price in classrooms across the globe shows just how robust the principle of the “invisible hand” is across cultures and geography.
Equally as exciting is the finding that markets are able to “discover” such price points very quickly, sometimes even within the first trading period among students with limited or no prior trading experience. In addition, the analysis also sheds light on a classic economic debate: how do price changes occur in a market? The data demonstrate that when there are fewer buyers and sellers, strategic and Bayesian learning theories may be better at explaining the price dynamics.
If the aforementioned N-player markets demonstrate self-interest and market efficiency as hallmarks of competitive trading behavior, our Ultimatum experiment presents a remarkable contrast in a two-player situation where fairness may play a central role in bargaining. Here, the MobLab team shows that when it comes to a 50/50 split, there is a spike in offer frequency, a significant non-linear jump in acceptance rate, and a decrease in reaction time from the responders, clearly demonstrating that this is a focal point with significant behavioral influence.
These results continue to show that economics, as a branch of social sciences, can have experiments with predictable outcomes just as routinely as a chemistry or physics experiment.
The Future
As wireless networks continue to improve and smart devices become increasingly utilized in the education of high schoolers and college students, networked social and economic experiments will play an increasingly important role in both teaching and learning. Additionally, as classroom and online data grows, there will be even more meta-analysis opportunities for basic research in the future.
Acknowledgements
This study would not have been possible without all the instructors and tens of thousands of students who have used the MobLab platform in the past. We are humbled by and grateful for the feedback from instructors and students about their experiences playing and learning from MobLab’s games. We strive to continually improve our education console and our service for our users and partners. Together, we will advance our understanding of key social science theories and topics through experiments while simultaneously enhancing social science education for future generations.