Perspective on Risk - Sept. 13, 2024 (Forecasting & Prediction)
Statistical Independence; Apophenia; Economic Forecasts; Overconfidence; Skill vs Luck; Good Forecasters; Thinking Critically; Conclusion Is Wrong; Information Hazards; Stupidity; Probability Puzzles
There’s a lot to talk about. Draghi’s comments, ChatGPT-5, the end of the Basel III endgame (cover the over). Will get to those another day perhaps.
Be Careful About Assuming Statistical Independence
Ignoring Probability Theory Is Dangerous (EconLib)
In 1999, Sally Clark, a young British lawyer, was convicted of killing her two newborn babies over a period of two years and she received a life sentence. A pediatrician had testified for the prosecution that the probability that the two boys had died from the Sudden Infant Death Syndrome (SIDS) or “crib death” was about 1 over 73,000,000. This was the only real evidence of the crime.
But the probability estimate, which persuaded the jury, was defective. It assumed that the two deaths were statistically independent events, justifying the multiplication of their respective probabilities for both events to happen: 1/8543 × 1/8543 is approximately equal to 1/73,000,000. In reality, however, two SIDS deaths in the same family are not independent events: one such death increases by 10 the medical probability that a second one will happen.
Apophenia
ap·o·phe·nia: the tendency to perceive a connection or meaningful pattern between unrelated or random things (such as objects or ideas)
My biggest weakness and fear. Comes from too much System 1 thinking (in Danny Kahneman terms). It is why I need to pay particular attention to data that refutes my priors, and perhaps why I am known for changing my opinion.
Why do we find it so hard to accept coincidences for what they are? (FT)
… when we do notice a set of circumstances that seem both highly unlikely and related in some way that we consider significant, our tendency is to look for causation.
When the exact same numbers were drawn two weeks in a row in Bulgaria’s national lottery back in 2009, authorities ordered an investigation, suspecting manipulation, but they came up with nothing. A mathematician put the odds of this happening at one in four million — highly improbable, but at some point, such coincidences will inevitably occur.
Economic Forecasts
Two interesting blog posts for us Bayesians.
The NY Fed’s Marco del Negro writes Are Professional [Economic] Forecasters Overconfident? and Can Professional Forecasters Predict Uncertain Times? (Liberty Street). I’ll quote from both below. These are based on a staff report A Bayesian Approach for Inference on Probabilistic Surveys.
The post-COVID years have not been kind to professional forecasters, whether from the private sector or policy institutions: their forecast errors for both output growth and inflation have increased dramatically relative to pre-COVID
We argue that for certain questions, such as whether forecasters are overconfident, this approach (using Bayesian non-parametric techniques rather than fitting the Philly Fed’s Survey of Professional Forecasters to an assumed distribution) makes a difference.
We find that for long horizons—between two and one years—forecasters are overconfident by a factor ranging from two to four for both output growth and inflation. But the opposite is true for short horizons: on average forecasters overestimate uncertainty, with point estimates lower than one for horizons less than four quarters (recall that one means that ex-post and ex-ante uncertainty are equal, as should be the case under RE).
Does subjective uncertainty map into objective uncertainty, that is, forecast errors? … under rational expectations it better be that forecasters that are more uncertain are actually worse forecasters, in the sense that they make on average worse forecast errors. And that if they feel economic uncertainty has declined, they should be able to actually predict the economy better if RE holds.
For output growth, … we find no relationship between subjective uncertainty and the size of the ex-post forecast error for horizons beyond one year.
Professional forecasters clearly do not behave on average like the noisy rational expectations model suggests. But the evidence also indicates that any proposed theory of deviations from rational expectations better account for the fact that such deviations are horizon dependent. Whatever biases forecasters have, whether due to their past experiences or other factors, seem to go away as the horizon gets closer. In other words, professional forecasters cannot really predict what kind of uncertainty regime will materialize in the future, but they seem to have a good grasp of what regime we are in at the moment.
Abandon Statistical Significance (Columbia)
This seems to be a cry in the wilderness to stop p-hacking.
We … discuss our own proposal, which is to abandon … the NHST paradigm—and the p-value thresholds intrinsic to it—as the default statistical paradigm for research, publication, and discovery in the biomedical and social sciences. Specifically, we propose that the p-value be demoted from its threshold screening role and instead, treated continuously, be considered along with currently subordinate factors (e.g., related prior evidence, plausibility of mechanism, study design and data quality, real world costs and benefits, novelty of finding, and other factors that vary by research domain) as just one among many pieces of evidence.
We have no desire to “ban” p-values or other purely statistical measures. Rather, we believe that such measures should not be thresholded and that, thresholded or not, they should not take priority over the currently subordinate factors. We also argue that it seldom makes sense to calibrate evidence as a function of p-values or other purely statistical measures. We offer recommendations for how our proposal can be implemented in the scientific publication process as well as in statistical decision making more broadly
Overconfident Behavior
Unpacking Overconfident Behavior When Betting on Oneself
This paper touches on a lot of the behavioral biases: overconfidence, Ellsberg's paradox, and Heath and Tversky’s competence effects.
Overconfident behavior, the excessive willingness to bet on one’s performance, may be driven by optimistic beliefs and/or ambiguity attitudes. … Our results showed that task difficulty affected both beliefs and ambiguity attitudes. However, while beliefs were more optimistic for relative performance (rank) and more pessimistic for absolute performance (score) on easy tasks compared to hard tasks, ambiguity attitudes were always more optimistic on easy tasks for both absolute and relative performance. Our findings show the subtle interplay between beliefs and ambiguity attitudes: they can reinforce or offset each other, depending on the context, increasing or lowering overconfident behavior.
From a practical risk management perspective, this has a number of implications.
First, pay extra attention to risk assessments for projects perceived as easy. Team members may be doubly biased here - overestimating their relative ability and being more ambiguity-seeking.
Second, separate absolute and relative performance metrics: Assess risks related to absolute performance (e.g., meeting specific targets) separately from risks related to relative performance (e.g., outperforming competitors). Incorporate objective, external benchmarks in risk assessments to counterbalance the tendency for over-placement in easy tasks.
Third, Recognize that team members may be more willing to take on ambiguous risks for easy projects. This could lead to underestimation of potential challenges or overestimation of the team's ability to handle unforeseen issues.
Finally, this supports having ex-ante “devils advocate” reviews of classification and structured ex-ante pre-mortems ( (imagining the project has failed and working backward) to counteract optimism bias.
Skill vs Luck
Top performers are not the most impressive when extreme performance indicates unreliability (PNAS)
The relationship between performance and ability is a central concern in the social sciences: Are the most successful much more able than others, and are failures unskilled? Prior research has shown that noise and self-reinforcing dynamics make performance unpredictable and lead to a weak association between ability and performance. Here we show that the same mechanisms that generate unpredictability imply that extreme performances can be relatively uninformative about ability. As a result, the highest performers may not have the highest expected ability and should not be imitated or praised. We show that whether higher performance indicates higher ability depends on whether extreme performance could be achieved by skill or requires luck.
Keep this in mind when completing performance reviews.
Characteristics Of Good Forecasters
Thinking Critically
The Data Checks Out, But The Conclusion Is Wrong
Annie Duke, poker player, behavioralist and statistician, uses the below popular post to show how you can draw the wrong conclusions from solid data. Fooled by the truth (Annie Duke)
This is a good example of how easily we can fall misleading conclusions backed by data that 100% will survive a fact check. It’s not the data that’s wrong; it’s the conclusions we reach (or, in this instance, accept) based on the data. And because the data survives a fact check, it makes the conclusion feel much more true, especially if the interpretation supports a belief you already have.
Why is the interpretation that people are getting dumber so unfounded here? Because there is an obvious alternative explanation for this seeming disappearance of people of Nobel-Prize-level intelligence: … The average Nobel Laureate is 59 years old when they receive the Nobel Prize.
It may be the ultimate example of survivor bias. … any cohort based on birth year takes nearly a century for a final count.
Information Hazards
Information Hazards: A Typology of Potential Harms from Knowledge
Ethan Mollick summarized this paper’s findings nicely in a tweet stream
Knowing true information can sometimes cause harm (think of the annoyance of seeing spoilers as a tiny example). This paper on information hazards is a preview to many of the issues we face today.
Ideological hazards: Most people have only a little knowledge about what their ideological belief (whether religious or political) really encompasses. On the web, you can learn that your chosen belief system also includes hazardous elements that you feel you need to adopt. 2/
Evocation hazards: there may be particular information that, when people encounter it, triggers them. This is not just in the common sense of triggering past trauma, but that some conspiracy theories or memes might be unusually tempting to people in particular mental states. 3/
Norm hazards: we share common sets of beliefs about how we, as a society or economy, should operate. If information breaks our belief in these norms, even if the information is true, it can create instability in the overall system. 4/
Distraction hazards. You are reading this on Twitter, enough said. 5/
Psychological reaction hazard: "Information can reduce well-being by causing sadness, disappointment, or some other psychological effect in the receiver." We know this actually happens in social media from experiments 6/
Neuropsychological Hazard, where information will actually cause physical harm. Triggering epileptics is one real example, mentioning MacBeth at a theater a fictional one. …
And then there is this amazing example of an information hazard that the paper didn’t consider, when learning about how an organization really works is like Lovecraftian secret knowledge that drives you mad, as you learn how there is no guiding power, only the uncaring void... 8/
The Importance Of Stupidity
The importance of stupidity in scientific research
Productive stupidity means being ignorant by choice. Focusing on important questions puts us in the awkward position of being ignorant.
The more comfortable we become with being stupid, the deeper we will wade into the unknown and the more likely we are to make big discoveries.
Probability Puzzles
If you’re bored at work (or retired!).
Answers at the end of the article