Via Stoat, I came across a bit of confusion over when we should say that a probabilistic prediction is “incorrect.” The kerfuffle began with Roger Pielke Jr (RPJ) arguing that 28% of the IPCC findings are “incorrect.”
RPJ’s claim is based on the fact the IPCC reports that some events will “very likely” occur (by which they mean there’s a greater than 90% chance of it happening), other events are only “likely” to occur (66% + chance), and so on. RPJ took an inventory of these findings and calculated that 28% of the events could fail to occur consistent with the IPCC findings. So he titled his post, “How Many Findings of the IPCC AR4 WG I are Incorrect? Answer: 28%.”
James Annan took the time to explain that a single outcome cannot disprove a probabilistic claim. JA’s example was the statement that a die is “likely” to land 1-5. This statement is not shown to be “incorrect” if the die happens to land instead on the six.
RPJ responded by arguing that in the real world when we claim that an event is likely, we’re willing to bet on it. And we only win if the event actually occurs. Claiming that we got the probabilities right is irrelevant; it’s a mere “academic exercise.”
This is where I came in. I tried to explain that getting probabilities right is itself a real-world concern. Indeed, when it comes to rationality, getting the probabilities right is more relevant than is the question of whether a particular event occurs or not.
I’d have just left the comment there, but since Willard thought it was worth posting up on its own, I figure I might as well put it here too:
You should avoid the sports betting and crap shooting and pay a little more attention to poker.
It makes perfect sense for a poker player to say, “Well, I lost a lot of money on that hand, but I played the cards right.”
In this case, the poker player correctly assessed the odds and acted rationally to maximize her expectation of winning — even though her opponent happened to beat the odds and hit his flush.
Now, in this case should we say that the poker player was “incorrect” (to use the word from your title that started this)?
It seems to me that we should say that as long as she recognized the cards, calculated the odds correctly, and bet appropriately, then it would be silly to say that her “finding” was “incorrect”. Yes, she lost, but she knows she’ll lose, say 40% of the time in that situation; in the long run she’s going to come out ahead.
Applying this to climate change, the relevant point is that we need to act rationally based on the probabilities. It might be a mistake to bet that everything that the IPCC says is very likely will happen; because if the IPCC is right, 10% of the time those very likely things won’t happen. (It’s important to remember that if 100% of the events designated “likely” occur, then — given enough such events — the IPCC assessment of likelihood will be proven wrong.)
We need to figure out how maximize our winnings given our best assessment of the odds — that’s what rationality is all about. So it’s counterproductive to treat losing money in a poker hand as evidence of an “incorrect finding.” Instead we need to concentrate on (1) making sure our probability assessments are as good as they can be, and (2) developing a strategy that maximizes our chances of getting what we want given those odds.
This is a very real-world concern, not merely an academic exercise.
Surprisingly, not everyone agreed with me. So my someone-is-wrong-on-the-internet syndrome forced me to post the following follow-up:
@-34- boballab said
“Your statement has a fatal flaw, that poker player only played the cards right based on what he “knew”. Well it turned out that what he “knew” was wrong.”
I don’t see a flaw in my statement (let alone a fatal one), except perhaps a failure to be completely clear about the suggested scenario.
When I said that the poker player “correctly assessed the odds” even though her opponent “hit his flush,” I intended to convey a situation in which she has correctly surmised her opponent’s hole cards (I had a hold ’em game in mind) and is therefore only ignorant about the community cards that have not yet been turned up. For definiteness, let’s say that on the turn she correctly infers that her opponent is drawing to a flush and correctly calculates that the odds will be in her favor if she goes all in.
When the flush comes on the river, she has made no error. She does lose, but — contrary to your claim — she made no false assumptions.
Now one could say (and I take it Roger Pielke Jr would want to say) that she erroneously assumed that the river card wouldn’t complete the flush (otherwise she wouldn’t have put her money in). But our poker player could quite reasonably respond, “No. I never said that I’d win the pot; I just said that the odds were three-to-two in my favor, and that’s a good position to bet.”
Note too (and this speaks to some of RP Jr’s more recent points) that the rational assessment of probability can (and should) be iterated to account for the likelihood that one or another probability distribution “actually” obtains. (The scare quotes are there because all the probabilities we’re interested in here are relative to some set of information. Quantum indeterminacy, for example, is irrelevant.)
So, for example, our poker player might correctly judge that her opponent will bluff in the current circumstances about 25% of the time. If she’s right about this likelihood, and if she can develop a strategy for increasing her odds of winning based on it, then it would be foolish to say that she was “incorrect” when she acts on this rational strategy and ends up losing a hand.
Of course I agree with you that if her decisions are based on flawed assumptions (e.g., she thinks her opponent is drawing to a flush, but he actually has a made full house) then she is just wrong. (She thought she had a 60% chance of winning and really she was drawing dead.)
But my point (which point I gather is also James Annan’s and William Connolley’s) is that losing a single poker hand (or even several) does not indicate that any of her assumptions were wrong.
And it seems to me that Roger Pielke Jr.’s language serves to obscure the question of what evidence would indicate an error in the grounding assumptions. (I found Gerard Harbison’s Bayesian sketch in -35- to be a helpful starting point for a discussion of the real issue.)
Somehow RPJ still wasn’t convinced.
He wrote, “Thanks … but be careful, when you say “an error in the grounding assumptions” that is very different than an incorrect forecast. If I fold and you are bluffing, I may have played the odds correctly but if I had called your bluff I’d have won the hand. In that case I may have been going against the odds (but perhaps I didn’t trust your poker face) and made a better (more correct decision).”
So I had to offer a bit more:
@ -54- Roger Pielke, Jr.
Thanks, and I agree that caution is called for. However, I’m inclined to think that the phrase “incorrect forecast” is horribly vague, and that it’s more likely to be misinterpreted than it is to be understood correctly.
Consider today’s weather forecast: The paper told me there was a 30% chance of precipitation, so I brought my umbrella with me. In fact, it did rain. Now, was that forecast correct or incorrect? It was certainly useful to me, I would have gotten wet if I’d thought there was a negligible chance of rain. Let’s assume, for the sake of argument, that in the past 1,000 times the weather service has told us there was a 30% chance of rain, it actually rained 300 times. Would you want to say some number of those forecasts were “incorrect”? Was today’s forecast incorrect (or even “more incorrect than correct”)?
It seems to me that when most people hear that someone made an “incorrect forecast” they are saying that some mistake was made. Either there was some information that could have been taken into account that wasn’t, or some calculational error was made, or some other cognitive error was committed. The implication of an “incorrect forecast” is that some more correct forecast could and should have been made.
Of course, one might mean something else by the term. Obviously we do need some way to describe whether a particular event under discussion occurs or not. And you are correct that typically when we say, “Event X is likely to occur,” we often mean something like “Even though I’m not completely certain, I am assuming that X will happen, and I’m willing to act on that assumption (and you should too).”
The problem with this reading in the current context is that it falls apart when one is making more fine-grained probability statements. Let’s suppose X happens. Which of the following forecasts are “correct” and which are “incorrect” (or even “more or less correct”)? “There is a 90% chance of X occurring.” “ “There is a 70% probability of X occurring.” ”. . . 50% . . . “ ”. . . 30% . . . “ ”. . . 10% . . . “
When I offer probabilities like this, there is no implicit commitment to the claim that X will happen. Of course, the forecast is about X, and X either happens or it doesn’t. But it’s misleading to say that the forecast is “incorrect” when X doesn’t occur.
From your above comments, I gather that you’re inclined to say that we have to rely on context to decide whether some event counts as evidence in favor of a forecast or against it. It seems to me that what you have in mind here is some sort of Bayesian analysis. We have to consider the prior probabilities in order to decide whether some event confirms or disconfirms a hypothesis. If I think there’s a negligible chance of a meteor hitting Boulder, and you say there’s a 10% chance, then if the meteor strikes, we’ll say your forecast was correct (even though you said the meteor is “unlikely” to hit). But if this is right, then you should put forward a Bayesian argument; it’s a red herring to appeal to the common language of football bets.
Coming back to the real world and the IPCC, it seems that the relevant point is just that we should make sure that people understand what the IPCC is saying and what they aren’t saying. I haven’t read your paper, but I gather that you’re calculating how many events discussed by the IPCC could fail to occur consistent with the report itself. Is that correct?
As I read your abstract, then, you’re saying, “If the IPCC is correct in its assessment, then up to 28% of the events that it discusses will not occur.” Now, this is an important point to communicate in order for us to act appropriately in the face of various degrees of certainty and uncertainty. But it seems to me that this is not well communicated by saying “28% of the findings of the IPCC are incorrect.”
I know it’s hard to imagine, but there could be people who would take this latter statement to mean that the IPCC has made a large number of errors. However, you are not pointing to any errors in IPCC assessment; your claims are perfectly compatible with the IPCC reports’ having captured precisely the actual probability distributions of all the events they discuss given all the knowledge that we human beings could possibly gather. (Of course, no one is making this claim, but such a possibility is in no way undermined by your treatment of the so-called “incorrect findings” of the IPCC).
Two final points:
First, your not trusting my poker face is an example of bringing more information into your assessment of the situation. The odds change based on that information. So, for example, suppose you fold because you think the odds are against you. We can imagine a pro poker player telling you, “No, you didn’t play that hand right. You should have noticed how he was sitting in his chair; that made it pretty likely that he was bluffing.” There you missed some information that was available, so your assessment of the relevant odds was off.
Of course, what we’re doing in both poker and in science is trying to get as much information as we can to all us to assign probabilities that will get us what we want.
Second, let me reiterate that your OP is misleading in suggesting that generating accurate assessments of probability is a mere “academic exercise” that is irrelevant to “real world commitments.” Of course action does require that we place bets on events actually occurring, but we cannot reasonably decide which actions to take unless we first take stock of the probabilities. The probability distributions have (or should have) important real-world consequences. They should shape our policy.
Thus we should be very interested in whether our assessments of those probabilities are correct or not. But you only undermine this effort if you claim that a completely accurate assessment of these probabilities implies that “28% of the findings are incorrect.”
And then I had to respond once more to boballab:
-55- & -56- boballab
As I see it there are two main points of disagreement here. One is a (perhaps trivial) difference about how to use words, the other is (I think) of practical importance.
First the trivial issue. Does betting on something mean that we’re “assuming” that it’s going to happen? I suppose there’s a way of using the word such that this is analytically true; after all, one might say that if I were assuming I’d lose, then I wouldn’t place the bet.
But on the whole, it seems to me that this isn’t what we usually mean when we say that someone “assumes” that X will happen. Consider my taking my umbrella today because there was a 30% chance of rain. Did I assume that it would rain? I’d be more inclined to say that I thought there was a significant chance of rain and that it was worth taking an umbrella. I didn’t assume anything except that the weather forecast was roughly accurate.
Or let’s suppose that our poker player is in a 10-person hand where everyone before her has bet (to keep it simple, let’s say that they’ve gone all in, so there are no further betting possibilities to consider). She calculates that she only has a 20% chance of hitting the winning hand, but since she’s getting nine-to-one on her money, she goes ahead and calls the bet.
Now, is she “assuming” that she’ll win? While I’ve admitted that we could understand the words this way, it seems much more reasonable to say that to say that she’s making no such assumption. If asked, she’s say it’s likely that she’ll lose; the odds are four to one against her, so it would be more natural to say that she’s assuming that she’ll lose.
What she is committed to is that there’s a 20% chance of her winning, and if she does win, then she’ll get nine times the money she bet. She committed to the claim that it’s a reasonable bet for her to make, and that if she follows this strategy in the long run, she’ll come out ahead.
This brings us to the more substantial point of disagreement. You say that “the final outcome is all that matters” and that I and James Annan fail to recognize this.
It is true that whether we win or lose depends on the actual outcomes. But what your gloss misses is the fact that the final outcome is irrelevant to our deciding – beforehand – what we should do.
Let’s change our poker example again to a case where the opponent gets incredibly lucky and wins the pot (he “sucks out”). The opponent goes all in on the flop with a backdoor straight flush draw, and our player calls with four of kind. Then the opponent beats incredible odds and makes the straight flush with the turn and river cards, and wins the pot.
Now, is it the case here that “the final outcome is all that matters”? It’s obviously true that the opponent wins the pot, but it’s also true that he reasoned very poorly – his action was ignorant and/or irrational.
The important point, though, is that if we’re trying to figure out which player to emulate – whose advice to listen to – we should recognize that our poker player did the right thing and the opponent did the wrong thing. The fact that he won the money doesn’t mean that he acted rationally – for the simple fact that he did not (and could not) know that he would win.
Given that we cannot know the future, we should be very very interested in figuring out what procedures to follow given our ignorance. Yes, there is a sense in which the poker player was “incorrect” in that she lost the money. But this sense is completely misses the most important point, which is that her strategy was correct and her assessment of the odds was as accurate as it could possibly be.
This is all that matters from a practical, decision-making point of view.
And it seems to me that you obscure this very important point if you insist on saying that the player’s judgment is often “incorrect” when she loses precisely the number of hands that she expects to lose.
One final point on this: “What is the probability of getting a heads on that just completed coin toss?. . . For completed ones you already know the outcome and for the one in the example the probability of head is 0% and tails 100%.”
The probabilities we’re dealing with here are all epistemic; they depend essentially on the information that is available. (There may be ontologically fundamental uncertainties in quantum mechanics, but that’s irrelevant to the issue at hand.)
The probability of getting heads on a fair toss (landing on the edge means it doesn’t count as a toss) is 50% whether the toss is in the future or the past. Of course, given the information about the outcome, that probability changes to 0 or 1. Likewise, the odds of my getting a particular card on the river will depend on the information that is available. The commentator on TV may calculate a percentage that takes account of all the player’s hole cards. The player herself, of course, doesn’t have that information and will come up with a slightly different number (which will change if she’s able to infer someone else’s cards). If someone has all the relevant information (i.e., if he looks in the deck) then he can be certain of the outcome.
But again, the important issue is how we deal with uncertainty. Because that’s the situation we’re in. Looking back with the certainty of hindsight is completely useless, unless that can be translated into looking forward and making the best decisions possible given our ignorance.
I like that ending.