Hedgehogs, predictions, and peak oil

What's a prediction worth?

Posted by David Roberts at 2:18 PM on 01 Dec 2005

A few days ago, Kevin Drum pointed to a Louis Menand review of a book called Expert Political Judgment: How Good Is It? How Can We Know? by psychologist Philip Tetlock.

I haven't read the book, but the review is fascinating, and I highly recommend you read it before (or instead of) the rambling thoughts that follow.

Tetlock's basic thesis, based on a multi-decade study, is that expert predictions are no better -- and often worse -- than random chance or the predictions of casual news consumers. Just like everyone else, experts display confirmation bias, underestimate their past mistakes, and fall for basic probability errors.

Ah, fuel for anti-elitists everywhere ...

This is not a new psychological finding -- laymen often express surprise at it, but psychologists have known for years that experts are no more reliable than anyone else. But one new insight Tetlock brings to the table relates to Isaiah Berlin's famous distinction between hedgehogs (which know a lot about one thing) and foxes (which know a little about a lot of things). From the book:

Low scorers look like hedgehogs: thinkers who "know one big thing," aggressively extend the explanatory reach of that one big thing into new domains, display bristly impatience with those who "do not get it," and express considerable confidence that they are already pretty proficient forecasters, at least in the long term. High scorers look like foxes: thinkers who know many small things (tricks of their trade), are skeptical of grand schemes, see explanation and prediction not as deductive exercises but rather as exercises in flexible "ad hocery" that require stitching together diverse sources of information, and are rather diffident about their own forecasting prowess.

Now, that description of a hedgehog ... does that bring anyone to mind?

Anyone at all?

This is what concerns me about the notion of peak oil and the almost cultish following its accrued in some quarters. Too many peak oilers view themselves as Prophets who have stumbled on the Theory of Everything and view it as their mission to educate the blind, benighted masses.

People like that, even (nay, especially) people who know a whole lot about the subject, tend to be wrong about what's going to happen. This is not a value judgment, but a simple statement of empirical fact.

I do not question the basic facts of peak oil. Oil's a finite resource; ergo, it's going to run out. Noting that there will be a peak in production is simply math.

But peak soon? Now? This past Thanksgiving? And run civilization headlong into a brick wall of scarcity, privation, and conflict? I don't know. For now, I'm withholding judgment.

But here's the ominous rub:

The upside of being a hedgehog, though, is that when you're right you can be really and spectacularly right.

Hmm ...

For story: Hedgehogs, predictions, and peak oil
17 Comments | Post a Comment

Ad hominem arguments

Tetlock's basic thesis - that most predictions are nonsense - is interesting.

The technique of improving the quality of predictions by sorting good prognostigators from bad prognosticators and putting them into bins labeled "hedgehogs" and "foxes," is probably nonsense as well. Is the practitioner of this technique a hedgehog or a fox? - one might ask.

Pre-judging prognosticators by asking whether they look like hedgehogs or foxes is most definitely nonsense. The specific term for this sort of fallacy is "ad hominem argument," as in, "He says this sort of thing because that's the sort of person he is."

The subject is too important to allow ourselves to make caricatures of those who try to move the discussion forward.

by dima at 7:52 PM on 01 Dec 2005

Why Predictions Often Fail

Predictions are, at best, educated guesses. Regardless of how educated one is, there are so many variables in life that the variable(s) that cause a certain result are often not considered. That said, where is the evidence supporting the assertion made by the author or by Dave? It's hard to believe that an educated guess is no better than an uninformed one.

Jeff Hoffman

by jdhlax at 8:55 PM on 01 Dec 2005

Dima,

Tetlock did a study involving more than 800 people over the course of over 20 years. Forget the labels if you want, but one pretty robust result was that these two different personalty types perform differently in the area of prognostication.

Of course you don't diregard anyone's arguments based purely on a quickie personality assessment, but you do take personality into account. If you have an empirical result indicating that those deeply committed to and confident in one overarching theory are wrong more often, then at the very least you greet their predictions with skepticism.

grist.org

by David Roberts at 11:04 PM on 01 Dec 2005
[ Parent ]

predictions

There are different kinds of predictions, aren't there?

A question like "will humanity use more oil or less in 2006" is easier than "what will be the closing price for a barrel of oil in 2006."

I haven't followed the link, but I expect that the difficult predictions (the unpredictable) will show about the same success with experts and non-experts.

FWIW, there are wrinkles to peak oil that no one is going to get from a quick survey. I think we have peaked on light sweet crude, and are shifting to heavier and more sour sources as a result. The thing is, the people most likely to criticize my "prediction" are probably not going to hang around for the details. They'll just scoff, look at falling pump prices (this week) and move along.

by odograph at 2:26 AM on 02 Dec 2005

BTW,

You should read up on the "winner's curse" in conjunction with prediction. The idea is that any prediction made by a group will result in a distribution of values. Often economic actions are driven not by the median prediction, but by the most optimistic.

We could argue that this is one cause of fishery collapse. Predicitions of cod populations (etc.) were made, and the most optimistic predictions ended up driving policy.

If that method, of choosing optimism, failed for cod, why should it work for oil?

by odograph at 2:34 AM on 02 Dec 2005

Doesn't compute

As a counterexample to David's prediction, I am deeply committed to the theory of gravity, based on which I can predict with supreme confidence how long it will take a penny to drop (on Earth, from a reasonable height). Will you now argue that my prediction is suspect because I am one of "those deeply committed to and confident in one overarching theory"? If a theory is no good, the predictions based on it will be no good as well. Peak Oil is not Newtonian mechanics, but it has a fairly sturdy theoretical foundation, tested on data from the many countries and regions that have already peaked. Debating the timing of a global peak is a matter of making some assumptions (because not all information is available or of sufficiently high quality) and plugging them through some math. It is most certainly not a matter of judging the character of the person weighing the evidence or making the computations.

by dima at 9:31 AM on 02 Dec 2005
[ Parent ]

The judgment of judgment

Does this thread strike anyone as a little odd? We are talking about a book that no one seems to have read. It is not clear if most of us are even willing to look over the highlighted review. In short, this thread (including this post) might be best used as fodder for a wry paragraph or two in Skeptic magazine.

In all events, thanks to David Roberts for pointing out the review. It was interesting and I confess to the hope of getting my hands on the actual book soon.

The review reports that the analysis involved 284 experts in politics and economics (not 800 as asserted elsewhere). These two fields are notoriously fuzzy. Moreover, the experts were tracked across 20 years and more than 82,000 (!) predictions. Thats an average of 290 predictions apiece. That is a scarely believable number of predictions. It seems safe to surmise that only a few of these predictions actually fell into the expert's field of study. If true, then it is no surprise that these predictions were no better than predictions from non-experts. One is left to wonder how helpful it is to do an analysis of experts in fuzzy-studies making predictions outside the area of their expertise. I guess I do need to get ahold of that book.

By the way, my apologies if anyone is offended by my references to the fields of economics as "fuzzy". I don't intend to smear the value of the research, nor do I seek to blame the individuals doing the research for the "fuzzy" quality. It is merely that a geologist can use neighboring outcrops and boreholes to predict what a new borehole will encounter. I know of no equivalent cirmcumstance for a political scientist or economist.

The poster who thought it was caricature to label analysts as hedgehogs or foxes was exactly right.

So I ask, why link this study to Peak Oil? Peak Oil, like every other serious topic, has its own suite of idiot-advocates; but I hope that we are not going to let idiots drive the discussion. Obviously, people with a deep familiarity with the field (such as Deffeyes) should get a critical and open-minded hearing. We are not such fools that we would dismiss the evidence of experts because they have expertise.

by georgenattaner at 10:05 AM on 02 Dec 2005

ah

doesn't that go to show that my prediction about the content was correct? ;-)

I wrote:

"I haven't followed the link, but I expect that the difficult predictions (the unpredictable) will show about the same success with experts and non-experts."

You write:

"The review reports that the analysis involved 284 experts in politics and economics (not 800 as asserted elsewhere). These two fields are notoriously fuzzy. Moreover, the experts were tracked across 20 years and more than 82,000 (!) predictions"

I now predict, based on the number of predictions, that he's using future prices of various financial instruments, and outcomes in future elections. I would classify these things as essenetially "unpredictable" but a good business for "experts" nonetheless.

by odograph at 10:35 AM on 02 Dec 2005

Fuzzy

One of the things that raises red flags for me about some of the more, uh, zealous peak oil folks is their pretension to mathematical certitude. While it is certainly the case that geology is a "hard" science, what mainly interests us about peak oil is its socio-economic consequences. No? And that stuff is decidedly "fuzzy." If you're trying to figure out how things are going to play out in the coming decades, the absolute amount of oil left in the ground is only one factor among thousands. Someone who tells me that oil-reserve numbers explain everything else is going to get the one-raised-eyebrow look.

I should make it clear that I'm not dismissing the notion of peak oil, or the work or expertise of those discussing it. I'm just saying that both our intuitions and now these test results indicate that ideologues -- people who think they've cracked the code for good -- are rightly greeted with skepticism.

grist.org

by David Roberts at 10:53 AM on 02 Dec 2005

Russia?

Did the experts forget about the vast untapped Russian oil fields?

Surely as these areas come into production oil supply will increase. Iraq is at some small fraction of possible production levels too.

Heading off supply/demand mismatch with alternatives is more important than speculating on so-called peak oil. As with most pop culture pseudo science, this is a concept constructed mainly out of fallacy.

It's only useful function is to sell books.

Why attack the messenger? Better to point out the obvious flaws in the whole notion of peak oil.

http://amazngdrx.blogharbor.com/blog John Schneider, Northern Wisconsin

by amazingdrx at 11:19 AM on 02 Dec 2005

Geologists probably aren't out to get you.

I asked a geologist friend of mine (who worked many years ago for BP) about his thoughts on peak oil. He is a hedgehyog. He said "Well, my field is geology, peak oil is simple math and abiotic oil has been extensively studied. In the 80's we were pretty much done exploring it. It hadn't panned out. THe math of peak oil is easy, calculating reserves is hard. But I'd be more worried about global warming if I needed to find something to worry about. Peak oil only means the end of around half our food supply right? But global warming is a really big deal. THe carboniferous stuff we go though (relating to the age) is from an atmosphere that would basically kill us today." ANd then he went on to talk about the carboniferous era. Well, his expertise sort of touches on that kind of thing but I would consider that one of the sort of predictions this article is talking about. He isn't a climate researcher. He basically knows what kind of rocks are likely to be where.

I don't know if that relates now that I think of it and hard sciences are a lot different than soft sciences. THere are a lot fewer variables in hard sciences. ANyway, for what it's worth, there is an example.

by BWE at 1:46 PM on 02 Dec 2005

In praise of Theories of Everything

David writes: "Too many peak oilers view themselves as Prophets who have stumbled on the Theory of Everything and view it as their mission to educate the blind, benighted masses."

Right... and the same is true for climate change, Christianity, Ayn Rand capitalism, socialism, Eastern religions, etc.

These grand theories are great for young people -- they show that ideas can be sexy and important. They kindle an intellectual excitment that can extend to other fields, providing a pattern of curiosity and learning.

The problem comes if you stay within the orthodoxy, if the doctrine is a closed room rather than a springboard.

I'm delighted at my 18-year-old friends who go on and on about Peak Oil. If they were to do the same thing at age 48, I'd be worried.

Bart
Energy Bulletin

by Bart Anderson at 6:53 PM on 02 Dec 2005

the distribution

"I'm just saying that both our intuitions and now these test results indicate that ideologues -- people who think they've cracked the code for good -- are rightly greeted with skepticism."

Are you looking at the median predictions, or the outliers?

FWIW, I think Motley Fool takes a good median position in this article. There's no "we're all going to die" but neither is there "nothing to be seen, move along folks."

by odograph at 5:04 AM on 03 Dec 2005

Liked your comment...............

and sifting through the chaff seems to be one of the few pluses that come with age; myself being 59 in one week. I find myself being as critical and suspicious of the political system as my dad was. I like Ken Deffeys treatment of oil, his book, Beyond Hubert's Peak, was terriffic. He wasn't writing to the times (my biggest complaint of any writer from Hemmingway THE OLD MAN AND THE SEA to Ayn Rand's capitalism in any form is great), just telling it like it is. He is secure in his teaching and soon-to-be retirement so he doesn't have a dog in this fight. Naturally, if you look at our resource depleated environment, we the cows calmly chewing our cud are next exploitable target.
So, are we losing our oil production, yeah, but refineries will be build for gasification of coal which we have, Australia has, China has, Russia has, etc. so lets not consider canibalism just yet but that doesn't mean that enough fear can be generated so that we will once again be engulfed in war if it is profitable enough and when is it isn't?
The future world looks more like sci-fi to me; mega cities mostly modern with huge underclasses mired in not just poverty but something like the untouchables in India. And did you read the NYT article about the new 4 lane highway in India, all for the new sacred cow, the cars. The efficient rail road will be allowed to disintegrate in favor of the road transport; much like how we nipped burgeoning rail here in the US in favor of profitable cars.
We have another generation or two before all the world is used up. Mars next?

by katesisco at 11:19 AM on 03 Dec 2005
[ Parent ]

Fuzzy Recursion

Dave, thanks for your response.

The appearance of disagreement can arise when people agree with each other too loudly. Could this be the current case? I agree that those with pretension to certitude are difficult to live with. I agree, too, that the socio-economic consequences are intriguing and even (I fear) alluring. A closet survivalist is harbored within many of us.

But is there not also an issue of getting horse and cart arranged in useful order? Right now the disagreement between sober, creditable, petro-geologists is still significant (compare USGS projections and Deffreyes). I doubt that we could make useful socio-economic projections (i.e. avoiding mere propaganda) on the basis of this current understanding. Perhaps we should focus instead on setting aside the annoyingly pretentious and demand that work be done on duller, vital things. For example, Simmons makes a sustained effort in "Twilight in the Desert" to make people see that we need better data. As you know, history has numerous examples where this class of boring, difficult and actionable advice was regrettably relegated to back burners.

Alas, I still haven't read "Expert Political Judgment". After this I will refrain from reposting until I can actually contribute.

by georgenattaner at 12:34 PM on 03 Dec 2005

selecting optimism

USGS projections are the result of a "winner's curse."

by odograph at 3:47 PM on 03 Dec 2005
[ Parent ]

loving it

I am having a good run here. I posted first, then followed the links above, quickly scanning. Then posted again based on georgenattaner's comments.

The funny thing is, that in this article about the weakness of predictions, my predictions are holding up pretty well.

A new review has appeared, and it produces a take-away similar to one of my comments above, about median vs. outlier predictions:

My caveat: Assume that the experts are usually wrong in their novel predictions. The consensus views of a science still might be worth listening to. Economists cannot forecast business cycles very well, but you should listen when they tell you that a deflationary shock is bad news. Each new forecast or new theory is an example of individual hubris and in expected value terms it is stupid. But the body of experts as a whole, over time, absorbs what is correct. A large number of predictions creates a Hayekian discovery process with increasing returns to scale. Social knowledge still comes out ahead, and in part because of the self-deceiving vanities put forward every day. You can find that point in Jonathan Swift's Gulliver's Travels.

- more here

Based on the good review at Marginal Revolution, I might actually keep an eye out for a copy (at the library)

by odograph at 7:37 AM on 05 Dec 2005

ADVERTISING POLICY

You are not logged in. Thus, you cannot post a comment. If you have an account, log in. If you don't have an account, well, by all means go make one! Meet you back here in five.