The problem with real-life experiments

When I hear about expert economists who disagree on policy recommendations, I think, “Surely this could be solved by experiment!” What if, instead of arguing over, say, whether raising the minimum wage would help or hurt the economy, we just try it out somewhere and see what happens?

I would love to see more evidence-based policy decisions, and I think a lot of people agree with me. (If you don’t, let me know so I can have a more accurate view of the world!) So I feel like I need to mention what I consider a big problem with trying to run experiments in real life. Before I tell you the general idea, here’s a classic example: The paradox of the parking space.

Imagine that you’re a homeowner on a street where each house has a beautiful lawn but not enough parking. If you want to increase your house’s value, you could pave over your lawn to create extra parking. Now imagine that everyone on your street makes that same decision to pave an extra spot. With so much more asphalt and so much less green, it turns out that the property values on this hypothetical street have actually gone down from their original prices. How is it possible that everyone could act to increase their own housing price but end up worse off?

Here’s how it might work with four houses, if you adding the parking space raises your property value by $20K and lowers everyone else’s by $10K. At first, the four houses have these values:

House A: $100K, House B: $100K, House C: $100K, House D: $100K

Then House A gets a parking spot, increasing its value to $120K. But the neighborhood is a little uglier and everyone else’s property value goes down to $90K.

House A: $120K, House B: $90K, House C: $90K, House D: $90K

Then House B does the same treatment. Its value gets bumped up to $110K, and everyone else’s value drops again by $10K.

House A: $110K, House B: $110K, House C: $80K, House D: $80K

House C does the same:

House A: $100K, House B: $100K, House C: $100K, House D: $70K

And finally, House D paves a parking spot too:

House A: $90K, House B: $90K, House C: $90K, House D: $90K

So each person’s choice to pave makes sense individually, but the combined effect is that everyone is worse off.

Now if the numbers were different, the final outcome could be different as well. If the impact on your three neighbors’ houses were a decrease in $5K instead of $10K, at the end you’d all have houses worth $105K instead of $90K. (That’s the $20K from your improvement, minus three $5K decreases from your three neighbors.) But if the street had twelve houses instead of three, even an increase of $20K for you and a decrease of $2K for your neighbors would mean a net decline in housing prices once everyone paved their parking space.

So the problem with experimenting in real life is this: even if you observe a large positive effect from your experimental policy tried out in one area (like paving a parking space to improve property value), implementing that policy everywhere could have the opposite effect.

This video shows a fun approach to encouraging park visitors to throw away trash:

The experimenters brag, “During one day 72kg of rubbish was collected in our bin. That’s 41kg more than the normal bin just a small distance away.” Alone, this figure demonstrates that trash is more likely to end up in a fun bottomless trash can than an ordinary boring one. But it doesn’t tell you whether the park should convert all their rubbish bins to the fun model: for that you need to know whether the increase in rubbish collected at the fun bin outweighs the total decrease in the neighboring un-fun bins. If only they had bragged instead, “During one day with our fun bin, the rubbish bin system in the park collected 30kg more than it does on an otherwise similar day!” Or even better—since the fun bin may have attracted extra park visitors or the throwing away of non-rubbish—”At the end of one day with our fun bin, the park workers found 20kg less litter in the park than on an otherwise similar day!” That’s the real measure of whether the trash cans are doing their job.

As Henry Hazlitt says in his book Economics in One Lesson:

“The art of economics consists of looking not merely at the immediate but at the longer effects of any act or policy; it consists in tracing the consequences of that policy not merely for one group but for all groups.”

5 thoughts on “The problem with real-life experiments

What do you think? Leave a comment below:

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s