TAKING IDEAS ON A TEST DRIVE

Can the principles and methods used in scientific experiments be applied to business and social policy?

By TREVOR BUTTERWORTH

Uncontrolled

By Jim Manzi
(Basic, 300 pages, $28.99) 

Imagine that you are the chief executive for a chain of 10,000 convenience stores, 8,000 of them called QwikMart, 2,000 of them called FastMart. Strangely, the FastMarts are bringing in 10% more in sales on average than the QwikMarts, and your instinct is that it may have something to do with customer preference for the FastMart name. How do you find out if your hunch is correct? (click below to read more)

Jim Manzi, the chairman of Applied Predictive Technologies, was once asked to address such a question by the owner of a convenience-store chain—a story he relates in "Uncontrolled: The Surprising Payoff of Trial-and-Error for Business, Politics, and Society." Finding the answers was not easy. There were hundreds of variables that could account for the QwikMart-FastMart revenue gap, including distance to nearby highways, number of cash registers, cleanliness of stores and the "exact position of each product on each shelf." Worse, all these variables could mate with each other to produce even more variables.

Nevertheless, after studying mathematics at the Massachusetts Institute of Technology, Mr. Manzi found his dream job as a strategic consultant trying to solve such puzzles for various companies. His approach began with: What makes experiments in science so good at producing reliable knowledge—and could the same principles and methods be applied to business and even social policy? The answer, according to Mr. Manzi, was a qualified yes.

The hero of "Uncontrolled" is the randomized controlled trial or, as Mr. Manzi prefers, the "randomized field trial" (RFT). Such trials are familiar to us from their ubiquitous use in medicine, to judge the effectiveness of drugs and other treatments. They are increasingly applied to business strategy, and especially to e-commerce, given that the Internet has lowered the costs of studying customers' preferences and behavior.

It should come as no surprise that the most successful companies in information technology—Google, Amazon and eBay—are relentless experimenters. Millions of consumers, for example, can be tested at little cost to find out whether pop-up ads are more effective on the left side or the right side of a computer screen. Google alone, says Mr. Manzi, "ran about 12,000 randomized experiments in 2009, with about 10% leading to business changes."

Given that Mr. Manzi's own company specializes in conducting this kind of analysis for corporate clients, it is perhaps too much to ask him to reveal the precise methods by which he solved such specific business problems. Still, "Uncontrolled," even if hesitant about divulging the Applied Predictive Technologies secret sauce, offers much to digest.

It's one thing, for instance, to conduct experiments, but it's another to learn from them. A company, Mr. Manzi says, "is an alliance of individuals, and there are always competing theories, power centers, and knowledge silos within any firm." Amid all the "jockeying for control," the most successful experiments are performed when the experimenters don't have a dog in any strategic fight.

"Uncontrolled" is at its most provocative, however, when Mr. Manzi considers the largely unmet potential of controlled experimentation to improve outcomes in social science and government policy. At present, the contrast is embarrassing: Whereas Capital One bank "does thousands of randomized field experiments every month," he writes, the global tally of "well-structured relevant RFTs of reasonable size" on policy issues is paltry, perhaps no more than a few thousand to date.

This paucity reflects two distinct but related problems. First, controlled experiments are hard to do in social science, because any interesting question you might ask of any policy is going to be packed with causal variables and potential confounders. Plotting the path of an asteroid is, by contrast, easy; unlike society, the universe isn't waiting to mug the astrophysicist with unexpected causes. The second problem is that when social scientists do conduct controlled experiments, the results tend to show that supposedly worthwhile policies turn out to be worthless.

Take criminology. Using data from a study of all the research that had been done in the field, Mr. Manzi analyzed 103 RFTs conducted in the U.S. over the past few decades. He grouped the field trials into 40 "program concepts," including intensive probation and mandatory arrest for domestic violence. Of these 40 concepts, just 22 had more than one trial. And of these 22, only one clearly replicated the initial study's findings—nuisance abatement, or encouraging owners to clean up blighted properties.

In the hard sciences, the key to confirming a hypothesis is replication—and, ideally, replication with bigger and bigger population sizes allied to a plausible explanation for what's happening in the statistics. But such fastidiousness appears nonexistent among policy makers, even though, as Mr. Manzi dryly observes, the poverty of reproducible data leads to an inescapably dismal conclusion: "We should be very skeptical of claims of the effectiveness of new programs."

Mr. Manzi has a clever solution: Create a government agency akin to the Food and Drug Administration but devoted solely to testing policies for their validity and effectiveness. The goal of this Federal Experimental Agency would be to rate the likelihood of success for policy proposals, based on experimental results, much as the nonpartisan Congressional Budget Office scores the budgetary effects of policy proposals. But why go to this effort, you might ask, when social and economic behavior is much more causally dense than astrophysics or medicine? The probability of a payoff is going to be marginal.

Because, says Mr. Manzi, change at the margin adds up. Nuisance abatement may have seemed like a minor experimental victory with little practical promise, but it went on to become a central element in the now-famous "broken windows" approach to policing.

"Uncontrolled," as one might imagine, is a vigorous book, pulsing with ideas. Occasionally, its idea-richness leads the author to dawdle in abstraction or to speed through mathematical thickets. But these are minor flaws, and they are to be expected when your argument is that encountering failure is vital to finding success.

Mr. Butterworth is a columnist for The Daily, a tablet newspaper owned by News Corp.