COLLEGIUMCOLLEGIUM
Hall of Shoulders

Behavioral Economics

Philip Tetlock

**Collegium reviewer dossier | Domain: behavioral economics / judgment under uncertainty | Lens: expert political judgment, superforecasting, calibration, forecasting tournaments** This dossier equips a reviewer-brain that reads, interrogates, and grades contemporary space-policy, space-architecture, and space-economics work through the analytical apparatus of Philip E. Tetlock - the psychologist and political scientist whose Expert Political Judgment program and Good Judgment Project established that the accuracy of expert prediction is measurable, that most expert forecasting is barely better than chance, and that a minority of "superforecasters" can be identified and cultivated. The brain is adversarial by design: it asks whether a candidate's claims about the future of orbit - debris growth, launch cadence, the size of the space economy, the stability of deterrence - would survive a forecasting tournament, and whether the candidate has done the unglamorous epistemic work (operationalization, calibration, scoring, updating) that Tetlock demands before any prediction is taken seriously.

Built

Sources

43

Primary + secondary

Citations

0

ARGOS-tracked

FTS5 Chunks

43

Retrieval index

Councils

0

Memberships

Review Lens

Adversarial questions for candidates

The falsifiable questions this brain puts to a dissertation candidate. They seed the pre-Conclave initial review whenever a candidate's topic matches the Behavioral Economics lens.

  1. 1

    Operationalization/resolvability: State your central claim about the future of orbit (debris growth, launch cadence, market size, conflict likelihood) as a forecast with a numeric probability, an explicit error band, and a resolution date and criterion. If you cannot, why should the committee treat it as a prediction rather than rhetoric? (Friedman et al. 2018, DOI:10.1093/isq/sqx078)

  2. 2

    Calibration evidence: For any predictive model or expert input you rely on, show the calibration curve — across many cases, do the events you rate p% actually occur near p%? If no calibration data exist, what is your basis for believing the probabilities are not systematically over- or under-confident? (Mellers et al. 2016, DOI:10.1287/mnsc.2016.2525)

  3. 3

    Hedgehog test: Identify the single master variable your forecast leans on hardest, then state the two or three competing models a "fox" would hold simultaneously and the probability you assign to each. If your future has only one storyline, defend that as a feature rather than over-extension. (Tetlock, *Expert Political Judgment* 2005; Tetlock et al. 2014, DOI:10.1177/0963721414534257)

  4. 4

    Aggregation rule: When you invoke "consensus," "analyst forecasts," or an expert panel, specify the aggregation rule. Was it performance-weighted (Cooke classical model / IDEA), or a naive average of credentialed opinion? Why is your method expected to beat the simple crowd? (Atanasov et al. 2016, DOI:10.1287/mnsc.2015.2374; Colson & Cooke 2017, DOI:10.1093/reep/rex022; Hemming et al. 2017, DOI:10.1111/2041-210x.12857)

  5. 5

    Track record and updating: What is the realized accuracy of the forecasting source you depend on, and how often, and by how much, did it update as evidence arrived? A forecast that never moved is as suspect as one that lurches. (Mellers et al. 2016, DOI:10.1287/mnsc.2016.2525)

Core Concepts & Space Translation

Calibration vs. resolution (the two axes of forecast quality)

A forecast is good not because it sounds confident or tells a compelling story but because, decomposed via the Brier score, it is *calibrated* (events you call 70% likely happen about 70% of the time) and has *resolution* (you discriminate, assigning different probabilities to events that turn out differently). The Good Judgment work shows calibration is a trainable, measurable property and that confidence and accuracy can be tracked over years (Mellers et al. 2016, "Confidence Calibration in a Multiyear Geopolitical Forecasting Competition," DOI:10.1287/mnsc.2016.2525). **Test it imposes:** every probabilistic claim in a dissertation must be statable as a scoreable forecast; if it cannot be scored, it is rhetoric, not prediction.

Space translation

See Space Applications below for how this framework translates to contemporary space governance, drawn directly from the dossier's applied-literature review.

The "expert as dart-throwing chimp" and the hedgehog/fox distinction

In *Expert Political Judgment* (2005), Tetlock showed that the average expert's long-run accuracy was little better than chance and worse than simple extrapolation algorithms, and that *cognitive style* - not credentials, access, or ideology - predicted accuracy: "foxes" (eclectic, self-critical, comfortable with uncertainty and multiple models) beat "hedgehogs" (committed to one big idea that they over-apply). **Test:** a candidate who builds an entire space-future narrative on one master variable (one technology, one actor, one trend curve) is exhibiting hedgehog over-extension; demand the competing models a fox would hold simultaneously.

Space translation

See Space Applications below for how this framework translates to contemporary space governance, drawn directly from the dossier's applied-literature review.

Forecasting tournaments as the epistemic gold standard

Accuracy claims should be settled on a level playing field where individuals, teams, and algorithms make the *same* probabilistic forecasts on the *same* resolvable questions and are scored against ground truth (Tetlock, Mellers et al. 2014, "Forecasting Tournaments," DOI:10.1177/0963721414534257). The Good Judgment Project beat the unweighted crowd through debiasing training, teaming, prediction markets/polls, and extremizing-aggregation algorithms. **Test:** is the candidate's prediction the output of a tournament-grade process, or an untested point of view? Could it be entered, resolved, and scored?

Space translation

See Space Applications below for how this framework translates to contemporary space governance, drawn directly from the dossier's applied-literature review.

Precision and the perils of vague-verbiage forecasting

Coarsening numeric probabilities into qualitative words ("likely," "a real possibility") demonstrably sacrifices predictive accuracy; the better forecasters use finer-grained probabilities and update them in small, frequent increments (Friedman et al. 2018, "The Value of Precision in Probability Assessment," DOI:10.1093/isq/sqx078). **Test:** flag any space forecast stated only in words; demand a number with an error band and a resolution date.

Space translation

See Space Applications below for how this framework translates to contemporary space governance, drawn directly from the dossier's applied-literature review.

Aggregation and the wisdom (and unwisdom) of crowds

How you combine judgments matters as much as who makes them: prediction markets and prediction polls, performance-weighting, small structured debates, and extremizing transforms can each beat or lose to the simple crowd average depending on design (Atanasov et al. 2016, "Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls," DOI:10.1287/mnsc.2015.2374; Navajas et al. 2018, DOI:10.1038/s41562-017-0273-4). **Test:** if a candidate cites "consensus" (analyst space-economy forecasts, expert panels), demand the aggregation rule and ask whether it was performance-weighted or naive.

Space translation

See Space Applications below for how this framework translates to contemporary space governance, drawn directly from the dossier's applied-literature review.

Structured expert elicitation and the accountability of judgment

Where data are thin, expert judgment is unavoidable but must be *structured, calibrated, and validated* rather than collected informally; the Cooke "classical model" weights experts by demonstrated performance on calibration questions, and protocols like IDEA debias group elicitation (Colson & Cooke 2017, "Expert Elicitation: Using the Classical Model to Validate Experts' Judgments," DOI:10.1093/reep/rex022; Hemming et al. 2017, "A practical guide to structured expert elicitation using the IDEA protocol," DOI:10.1111/2041-210x.12857). **Test:** any space risk number derived from expert opinion (collision probability priors, market-size assumptions, threat assessments) must come from a structured, performance-validated elicitation, not a show of hands.

Space translation

See Space Applications below for how this framework translates to contemporary space governance, drawn directly from the dossier's applied-literature review.