Featured Developer Sponsor • Zero-Token Protection
Bayesian Probability & Rationality Benchmark
Epistemic Calibration & Overconfidence Benchmark
Measure your epistemic calibration against objective reality. Provide an 80% confidence interval for 10 trivia questions. If your brain is perfectly calibrated, exactly 8 out of 10 answers will fall inside your specified bounds.
Enter Your 80% Confidence Interval
For each question, specify a Lower Bound and Upper Bound such that you are 80% confident the true value lies between them. You should feel comfortable betting $80 against $20 on your interval.
Mathematical Foundations of Epistemic Calibration
In decision theory and subjective probability, calibration measures how closely subjective confidence intervals match empirical reality:
1. Perfect Calibration Condition:
P(True ∈ [L_i, U_i] | Confidence = c) = c
For an 80% interval (c = 0.80), exactly 8 out of 10 bounds must capture truth.
2. Calibration Hit Rate:
Hit_Rate = (1 / N) × ∑ I(True_i ∈ [L_i, U_i])
3. Overconfidence Bias Measure (ΔC):
ΔC = Target_Confidence - Hit_Rate
If ΔC > 0: Overconfidence (intervals too narrow)
If ΔC < 0: Underconfidence / Excess Timidity (intervals too wide)
4. Brier Calibration Penalty:
BS = (1 / N) × ∑ (f_k - o_k)^2
P(True ∈ [L_i, U_i] | Confidence = c) = c
For an 80% interval (c = 0.80), exactly 8 out of 10 bounds must capture truth.
2. Calibration Hit Rate:
Hit_Rate = (1 / N) × ∑ I(True_i ∈ [L_i, U_i])
3. Overconfidence Bias Measure (ΔC):
ΔC = Target_Confidence - Hit_Rate
If ΔC > 0: Overconfidence (intervals too narrow)
If ΔC < 0: Underconfidence / Excess Timidity (intervals too wide)
4. Brier Calibration Penalty:
BS = (1 / N) × ∑ (f_k - o_k)^2
5 Fatal Epistemic Overconfidence Pitfalls
1. The Social Penalty on Expressing Uncertainty
Human social hierarchies reward individuals who project certainty with rapid answers and narrow ranges. Expressing genuine uncertainty is often culturally misinterpreted as incompetence, training the brain to give artificially tight intervals.
2. Anchor-and-Adjust Insufficiency
When asked for an interval, the mind instinctively anchors on a single point guess, then nudges slightly higher and lower. This heuristic systematically fails to account for tail risks and distribution skewness.
3. Confusing Calibration with Trivia Knowledge
Believing that calibration requires encyclopedic knowledge. Perfect calibration does not require knowing the answer; it requires knowing how much you do not know and setting your interval wide enough (e.g. 100 to 100,000,000) to honestly achieve 80% confidence.
4. The Linear Certainty Fallacy
Treating unfamiliar physical quantities linearly rather than exponentially. Many astronomical, historical, and geological variables span multiple orders of magnitude, requiring log-scale bounds to be properly captured.
5. The Unscored Belief Feedback Deprivation
Most everyday opinions, business forecasts, and geopolitical predictions are never scored against empirical numbers. Without quantified scoring and Brier tracking, the brain's overconfidence mechanism never receives corrective feedback.
Frequently Asked Questions
What is Epistemic Calibration in decision science?
Why do most educated adults only hit 30% to 50% on 80% interval tests?
How do professional superforecasters train themselves to give wider intervals?
What is the difference between calibration, resolution, and accuracy?
What is the Brier score and how does it evaluate epistemic humility?
Sponsored Utility
While You're Here
Sponsored Recommendations
Advertisement