Skip to content
October 11, 2012 / Dan Whipple

The secret of my success

One of my most endearing personal characteristics is that I’m always right. My friends speak admiringly of this trait, often employing a word of one syllable followed by the pronoun “you!”

There have been occasional skeptics among my admirers, but I’m pleased to report that my always-rightness has now been scientifically confirmed in a rigidly structured test, the Good Judgment Project forecasting tournament.

We’re in the second year of this tournament, in which we amateur forecasters are asked to predict the outcome of obscure potential international events or the fate of obscure dictators from small underwater nations or the price of Euros on the open market by a date certain. Will the Nigerian government and Boko Haram commence official talks before 31 December 2012? Will the Vice President of Iraq Tariq al-Hashimi’s death sentence be overturned before 1 November 2012? Will the sentence of any of the three members of the band Pussy Riot who were convicted of hooliganism be reduced, nullified, or suspended before 1 December 2012? Will Moody’s issue a new downgrade on the long-term ratings for any of the eight major French banks between 30 July 2012 and 31 December 2012? Stuff like that.

In year one, I answered 80 (out of a possible 99) of this kind of question, and achieved a Brier score of 0.28. (What, you may ask, is a Brier score? I don’t know. But some people do and they calculate them. In the Brier scoring world, as in golf, a lower score is better.) This 0.28 placed me fifth out of 212 fellow prognosticators on the Good Judgement team. Our Good Judgement group of 212 competed against four other groups, presumably of the same size. We reportedly cleaned their clocks, outperforming by a wide margin not only our human competitors, but also doing better than computer algorithms answering the same questions.

My math may be a little off here, but this puts me fifth in a field of, let’s see, 1,060 competitors, or the top 0.47 percent (plus or minus 0.40 percent) of Boko Haram and Pussy Riot predictors in the whole wide world. Yo. Dude.

Thanks to my performance in year one, for year two I’ve been placed with a newly created Gang of Twelve “superforecasters” who all did well the first year. Every one of these folks appears to be smarter than I am, based on the clearly articulated, rationally composed forecasts they are making in the tournament so far. I had hoped Good Judgment Project was going to have me star in my own comic book and give me a cape and a colorful muscle-emphasizing costume in honor of my elevation to superstatus, but no such luck. Brad Pitt has agreed to play me in the movie, however. There are five superforecaster pods. Our beloved superforecaster group—5b1, if you’re keeping Brier score at home—is currently second among the five groups. And I, gentle reader, am numero uno in 5b1, with a Brier score of 0.22. Boo yah.

I’m doing way better at this than I am at Words With Friends on Facebook.

So, I submit that my always being right is now a fact, established with statistical rigor in the rough-and-tumble give-and-take of scientific inquiry. Argue if you want. Then check the Brier Scoreboard.

The real question—the one that the Good Judgment Project and the tournament sponsor, the vaguely sinister sounding IARPA [U.S. Intelligence Advanced Research Projects Activity] are really trying to answer—is, why am I always right? Let’s face it. I didn’t even know Iraq had a vice-president, never mind that he’d been condemned to death, before they asked me to predict this question. The researchers believe that there is wisdom in the predictions of the crowd. Personally I think if this were true, bookies would be out of business.

Prior to the second year of forecasting, I confidently predicted that I would regress to the mean of the rest of the predictors. That is, I’d do no better—and probably worse—than average. This hasn’t been true so far. So I’m not always right. (Hey! Wait a minute!)

Why am I still doing well? The answer is … I have no idea. With most of the questions, you can predict with some certainty that things will stay the same in the future as they are today. That’s the default position. But everybody in tournament knows that. It’s no advantage. A lot of my teammates assign numerical probabilities to the events we’re asked to predict, then calculate conditional probabilities to come up with their answers. I love statistics. I follow these lines of inquiry with deep admiration.

But—at the risk of introducing the post hoc ergo propter hoc fallacy—in order to assign a probability to a forthcoming event, you have to have some information about it. Virtually all of the information available for these questions comes from news stories written by journalists. I’ve been a journalist myself for more than 35 years. And one thing I’ve learned in that time is that journalists on the scene always know more about a topic than they can put in their story. So maybe—and I emphasize maybe—my journalism experience has enabled me to read between the lines of fact in a news story, to grasp the tone and atmosphere, to discern what the people closest to the event think will happen, or are planning to implement.

I think this is very unlikely to really be an explanation for my success (so far) in this tournament. I offer it only as a hypothesis. But I don’t have any other. There’s no rational reason for the dumbest guy in our Gang of Twelve to be leading the pack (other than, god forbid, luck). I still think that I’ll probably regress to the mean in the long run. Meantime, please Br’er Fox, don’t throw me in that Brier patch.

Advertisements

4 Comments

Leave a Comment
  1. Dan Neal / Oct 11 2012 2:04 pm

    You find the weirdest things to do with your time.

  2. Fellow Prognosticator / Oct 15 2012 6:08 am

    As a fellow GJPer, congrats on your score. I think being a cynical journalist with int’l experience is good background for the types of prediction that GJP asks for.

    Watch out for regression to the mean!

    • Dan Whipple / Oct 15 2012 6:52 am

      Thanks for the note. I’d be interested to hear about what approach you take and how much time you spend on it. Despite my personal skepticism about the ability to predict, I’m finding the whole thing very interesting.

  3. Fellow Prognosticator / Oct 24 2012 10:34 am

    I spend way too much time on it 🙂 I generally spend a lot of time browsing news on the web so now I’ve just focused my reading on topics that are GJP questions.

    My approach is basically to bet against what I believe to be journalist biases so probably similar to yours. Anything too sensational or wishful thinking, I’ll discount. Anyone in denial, I’ll upgrade.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: