Caltech Research Shows Medicare Auction Will Face Severe Difficulties

Medicare's new method for buying medical supplies and equipment—everything from wheelchairs and hospital beds to insulin shots and oxygen tanks—is doomed to face severe difficulties, according to a new study by Caltech researchers.

The Center for Medicare and Medicaid Services (CMS) implemented the purchasing process—a novel type of auction—in nine metropolitan areas across the country last year and plans to expand it to 91 in 2013.

The competitive bidding process was designed to improve the efficiency of Medicare's procurement system, potentially saving taxpayers hundreds of millions of dollars. But many experts have criticized the auction, pointing out fundamental flaws in its design. Now, a series of experiments testing the auction structure show that it does, in fact, fall short of expectations. The results of the study, which was conducted by Caltech seniors Brian Merlob and Yuanjun Zhang, and Charles Plott, the Harkness Professor of Political Science and Economics, were published in the May issue of the Quarterly Journal of Economics.

In principle, auctions are a cheap and efficient way to procure goods and reward companies that can inexpensively produce their goods. In the case of the Medicare auction, various companies make bids to the government that represent their best prices for medical supplies and equipment; the government, in turn, wants the best deal and so chooses the lowest bids. For example, if 1,000 hospital beds are needed and it would take five companies to supply them all, the five lowest bids are chosen.

Unlike standard auctions, however, the CMS auction was designed with two unorthodox rules. First, the eventual selling price is set at the median of all of the winning bids. Second, bids are nonbinding, so companies can change their mind once the prices are set.

Critics say that these rules cultivate a harmful bidding strategy. To ensure a winning bid, each company will make a very low offer; this carries no risk, because the companies can cancel their bid if the median price turns out to be too low. The result is that participants in the auction will tend to make low-ball bids, assuring that the median price will also be very low—so low, in fact, that few of the companies can actually afford it, leading them to cancel their offers. At the extreme, nothing is bought or sold and, Plott says, "the auction crashes. It's just not an effective auction."

And what will happen then, critics warn, is that the government will end up negotiating prices with individual companies—negating the whole point of a competitive-bidding scheme in the first place. "You can see immediately from theoretical arguments that the potential for disaster is built right in the strategic structures," Plott says.

To determine if such theoretical predictions translate into real-world behavior, it is essential to examine experimentally how people behave in an actual auction, says Plott—who, for the last 30 years at Caltech, has been developing experimental techniques to test economic theories. He posed the problem to his experimental economics (EC 160) class. Two of his students, Merlob and Zhang, took up the challenge and spent a year researching, designing, and conducting experiments to test that behavior.

The team used computers at Caltech and the University of Maryland to run a simplified version of the CMS auction and several other auction types; one, for example, followed more standard rules, with binding bids and prices set at the lowest bid that did not win, instead of the median of all winning offers. Each auction involved 12 or 16 bidders (student volunteers from Caltech and the University of Maryland), who first had to pass a quiz showing that they understood how the auctions worked. The volunteers were given just one item to sell—a generic "thing" (since the bidders' behavior should be the same in a given auction type, regardless of the item being sold)—each at a different cost to them.

The Caltech team also examined the effect of other auction features, such as whether the costs of each item for each bidder are public knowledge and the effect of charging bidders to participate.

The results, the researchers say, convincingly support critics of the CMS auction design. "It's pretty disastrous what the bidders ended up doing," Zhang says. In the simulated CMS-type auction, some people bid $0, and the "government" was not able to buy all the items it needed. The experiments also showed that a standard auction is much more efficient and successful: the government was able to buy all the items it needed, and the bidders who had the lowest costs were the winners.

Using this experimental approach, the researchers were able to pinpoint the fundamental problem of the CMS auction design: the two rules. "If you just get rid of one of those two rules, it doesn't help—you still have problems," Plott explains. "So you have to get rid of both of them."

Last summer, 244 economists and auction experts, including Plott and Caltech professors John Ledyard, Thomas Palfrey, and Matthew Shum, signed a letter to urge President Obama to change the CMS auction system; the letter cited the Caltech experiments.

In April, however, a study released by the Department of Health and Human Services reported that the auction saved taxpayers $202.1 million in 2011, with no negative effects on health care. The report also estimates that the auction will save taxpayers and beneficiaries $42.8 billion over 10 years.

A preliminary analysis of the pilot program by Peter Cramton of the University of Maryland, an outspoken critic of the CMS auction and an economist who was not part of the Caltech study, found that the auction did in fact suffer from the problems predicted by theory and experiment. Because of the auction design, prices plunged to unsustainable levels, and suppliers dropped out, forcing Medicare to find new suppliers. Cramton also found that the number of submitted claims for equipment declined, which, he says, led to increased rates of visits to the emergency room and hospitalization. As a result, not only were overall costs higher, but so were health risks.

While Plott says he cannot comment on the report without knowing exactly how the study measured and collected its data, he remains confident in the experimental results and the theoretical arguments against the auction. "The theory gives a rather clear picture about the implications of the auction architecture," he says. "But only the data can tell us how these ideas actually play out in such a complex application with variables too numerous to be considered in the theory."

Regardless of whether the CMS will continue to expand the auction or will heed the critics' warnings, this type of research, Plott says, is a good example for how basic, scientific experiments can have direct impact on society. He considers that an especially important lesson for his students. Over the years, he says, several of his undergraduates have stuck with a project long enough to publish, occasionally leading to several awards and seminal papers. "Taking Professor Plott's class was fantastic," Merlob says. "He's a master experimentalist and an amazing mentor."

Zhang, who is doing an independent study in decision-making and neuroscience, will start graduate school in economics at UCLA in the fall; Merlob, a political science major, is still exploring his options. "This entire experience was pretty awesome," he says. "It's probably one of the best of my Caltech career."