Famous Economics Experiment Reproduced Thousands of Times

In an open marketplace, such as a farmers' market where produce and other goods like candles and flowers are exchanged for money, the ideal prices for both consumers and sellers will quickly emerge. For example, if a seller tries to offer a bag of peaches for $10 but another vendor is willing to sell similar peaches for $5, the lower price will eventually win out and become the norm. This phenomenon, which is related to the law of supply and demand, was demonstrated experimentally starting in the 1960s by Caltech alumnus and Nobel Laureate Vernon Smith (BS '49), now at Chapman University, and later by Caltech's Charlie Plott, the William D. Hacker Professor of Economics and Political Science.

Now, nearly 60 years later, Caltech economists have analyzed data from 2,000 repetitions of these experiments, from researchers around the world, to demonstrate for the first time that the work of Smith, Plott, and others is reproducible on large scales. The research was published August 3 in the journal Nature Human Behaviour.

"Until now, there was nothing in the published experimental economics literature like this, in which a common design is done over and over, or in which somebody went back and meta-analyzed a body of data using a common design," says Colin Camerer, the Robert Kirby Professor of Behavioral Economics and director of the T&C Chen Center for Social and Decision Neuroscience in the Tianqiao and Chrissy Chen Institute for Neuroscience at Caltech. "It is like a chemistry experiment—if you combine the same exact chemicals many times over, do you get the same chemical reaction? We actually did not know. There is very little career incentive to replicate somebody else's experiment because it's not as creative as coming up with your own, but it's important to do."

The research was possible thanks to a collaboration with the economics education company MobLab, which provides online experiments for students across the globe. Camerer, who is a scientific advisor at the company, worked with his graduate student Po Hsuan Lin, who himself worked at MobLab prior to starting graduate school, to collect tens of thousands of records from the company. As was the case with Smith's 1960s experiments, student users of the online tool take part in a simple market game, where imaginary goods are exchanged for actual money or course credits. Within as little time as minutes, an equilibrium price that is ideal for both consumers and sellers is reached.

"We pulled the data of the simple market experiment, done by students in countries around the globe," says Lin, who is also affiliated with National Taiwan University. "Even with the variation inherent in having people from different countries participate in the same experiment, we get nearly the same results every time. The final equilibrium prices differ by only pennies."

Because professors using MobLab tools want to be consistent with other classes, the same experiments have been performed across classrooms. This meant that the Caltech economists could study the MobLab data to, in essence, repeat the same experiment over and over again. For the simple market game, they ended up analyzing 2,000 experiments and always got the same results.

"The idea that buyer-seller markets are highly efficient and converge to a single price is in every textbook. It's called perfect competition. But it is based largely on a theory," says Camerer. "Charlie Plott has done these market experiments all over the world, with younger children and other students, but the designs are not exactlly the same. With MobLab, we were able to use the exact same experimental designs and show how incredibly reproducible these experiments really are."

The study, titled "Evidence of General Economic Principles of Bargaining and Trade from 2000 Classroom Experiments," was funded by Deutsche Forschungsgemeinschaft, CRETA (Center for Research in Economic Theory), National Taiwan University, the Ministry of Science and Technology in Taiwan, Caltech, and a MacArthur Foundation Fellowship. Other authors include Alexander L. Brown (MS '05, PhD '08) of Texas A&M University, Taisuke Imai (PhD '16) of LMU Munich, Joseph Tao-yi Wang, a former Caltech postdoc now at National Taiwan University, and Stephanie W. Wang, also a former Caltech postdoc now at the University of Pittsburgh and MobLab.