--- marp: true theme: default paginate: true --- # Retrieval-Accuracy Returns to Instrument Investment in Earth-Science Radiometers A hedonic regression of validated science accuracy on cost drivers **Doctoral defense** **Candidate:** JPL_ASTRO_EARTH_10 COLLEGIUM 1st Battalion | NORTH STAR / JPL Earth Science Missions 2026-06-15 --- ## The contribution **This dissertation pursues** a reduced-form hedonic frontier, estimated across NASA-class passive radiometers, that establishes whether validated retrieval accuracy is concave in instrument development cost with an identifiable over-specification spectral-channel count. - **If H1 holds:** cost-capped mission formulation gains a defensible stopping rule for spectral specification, and a value-aware descope and reallocation argument. - **If H0 holds:** diminishing returns is removed as an unexamined assumption in instrument budgeting. - Both outcomes are decision-relevant. This is a design-stage dissertation: estimator, identification, threats, and decision rule are fixed in advance; no result is executed. --- ## The single falsifiable claim **H1 (contribution):** Validated geophysical-retrieval accuracy for Earth-observing radiometers is a concave function of instrument cost. The marginal accuracy gained per additional dollar declines as cost rises and collapses beyond an estimable spectral-channel count, identifying an over-specification region. **H0 (null):** Validated retrieval accuracy is linear in instrument cost. No diminishing returns, constant marginal accuracy per dollar, no over-specification region. H1 is rejected and H0 stands if the cost term is statistically indistinguishable from a straight line, or if the second derivative of the fitted accuracy-cost function is not reliably negative over common support, or if the marginal channel-count contribution stays positive across the observed channel range. --- ## The problem: cost and accuracy are never on the same axes - One community predicts what an instrument will cost from its design (NICM); a separate community measures how accurate its products turn out (cal/val). No one regresses accuracy on cost. - What the field lacks is a population-level estimate of validated accuracy as a function of instrument cost, with the design attributes that drive both held fixed. - No estimate exists of the shape of that relationship, and so none of where an over-specification region begins. - As a result, mission budgets set spectral specification without evidence on whether the marginal dollar still buys validated science. --- ## The gap: three literatures, none closes the join - **Hedonic pricing (Rosen 1974):** recovers implicit attribute values from a good's price. Never inverted to use a performance metric as the outcome. - **Instrument cost models (NICM; Stahl telescope series):** regress cost on design. Dependent variable is cost, not delivered accuracy. - **Calibration and validation (MODIS aerosol, SST, SMAP, GPCP/GPM):** rigorous but product-specific and instrument-isolated. Each literature holds two of the three axes the question needs; the missing axis differs between the cost side and the accuracy side. Closing the gap is no single community's job. --- ## Theoretical framework: why concavity is the expected result Four anchors compose into one prediction, so concavity is over-determined, not assumed. - **Rosen, inverted:** the hedonic surface gives the relationship its functional form, with cost as the central attribute and a near-sufficient statistic for embodied build quality. - **Simon, bounded rationality and satisficing:** designers search to an aspiration level and stop, so early increments buy large gains and later ones do not. - **Simon, near-decomposability:** beyond a point, added spectral channels carry redundant information; accuracy is bounded by non-spectral error. - **Value-driven design:** aerospace tradespaces already contain dominated, over-specified regions; this work locates one across a population. --- ## The named causal mechanism The over-specification prediction is a mechanism, not a bare correlation. - **Driver:** added spectral channels and calibration elaboration raise instrument development cost. - **Mechanism:** beyond a point, channels carry information redundant with channels already present; achievable accuracy is bounded by calibration drift, geolocation error, and intrinsic retrieval difficulty. - **Observable effect:** the accuracy-versus-cost curve flattens; the marginal channel contribution falls toward zero. - **Operational consequence:** dollars spent past the over-specification edge buy specification, not validated science. - **Strategic implication:** a cost-capped portfolio delivers more total validated accuracy by capping per-instrument specification and reallocating. --- ## Data - **Validated accuracy (dependent):** Level-2 and Level-3 metrics from NASA Earthdata DAACs and the peer-reviewed cal/val record, requirement-normalized and sign-oriented (bias, RMSE, unbiased RMSE, expected-error compliance). Drawn from independent validation, never from design specifications. - **Cost (regressor):** NICM-class development cost, constant-year dollars (not life-cycle cost). - **Design controls:** NTRS specifications, channel count, swath, resolution, calibration approach, mass, power, mission epoch, plus a retrieval-difficulty control. - **Unit of analysis:** the instrument-product pair (one row per validated product; cost shared; instrument-clustered inference). - **Population:** NASA and NASA-partnered passive radiometers, MODIS era to present. --- ## Requirement-normalization makes accuracy comparable - Raw retrieval error is in incommensurable units across families (optical depth, kelvin, volumetric soil moisture, mm/day). - Each product's validated error is divided by that product's stated mission accuracy requirement, then sign-oriented so the variable increases in accuracy. - The construct becomes requirement compliance, which is comparable across families in the units the investment decision itself uses. - Worked per family: aerosol (MODIS vs AERONET), SST (vs in-situ), soil moisture (SMAP vs core sites), precipitation (GPCP/GPM), land-surface temperature. --- ## Research design and the estimator **Estimator:** partially linear semiparametric regression. > accuracy = g(cost) + X·beta + e - The cost term g enters through a flexible, concavity-respecting smoother; design controls X enter linearly to conserve degrees of freedom. - The contribution is the second derivative g'' over common support and the marginal channel-count contribution. - A fully parametric quadratic is too brittle; a fully nonparametric fit overfits a small sample. The partially linear form concentrates flexibility on the one dimension the contribution is about. - Inference is clustered at the instrument level, because the effective sample is the number of instruments, not rows. --- ## Identification (the Abadie discipline) Instruments are not randomly assigned their cost levels, so a naive regression confounds the return to investment with the selection of instruments to cost. The claim is selection-on-observables within common support, built in three steps. 1. **Controls absorb the reasons for cost:** retrieval difficulty and technology epoch are the dominant confounders and are held fixed. 2. **Covariate balance is enforced:** entropy-balancing weights equalize the design attributes across cost strata, rather than assuming linear controls did the work. 3. **Common support:** the estimand is restricted to the cost range where instruments of differing cost coexist with comparable attributes, so curvature is interpolation, not extrapolation. --- ## The over-specification test A sharper, more falsifiable claim than aggregate concavity. - Spectral-channel count enters with a flexible term; the marginal contribution of an additional channel is traced as a function of channel count, holding cost and other attributes fixed. - The over-specification edge is the lower bound of the range where that marginal contribution is statistically indistinguishable from zero. - **Physical grounding:** Rodgers' optimal-estimation theory shows that degrees of freedom for signal saturate as channels are added; operational channel-selection practice (CrIS, TES) confirms a small subset captures nearly all retrievable information. - The statistical edge is predicted to coincide with the physical onset of redundancy, which keeps the test from being a numerical artifact. --- ## Threats to validity - **Internal:** omitted drivers of both cost and accuracy (both directions); reverse linkage; cost-version mismatch. Addressed by the control set, the difficulty control, balancing, common support, version matching, and a formal unobserved-confounding sensitivity analysis. - **External:** the estimand is bounded to NASA-class passive radiometers in the modern era, within common support. - **Construct:** accuracy is drawn from independent validation and requirement-normalized; cost is development cost by definition. - **Statistical-conclusion:** small sample; instrument-clustered standard errors; a held-out predictive test guards against in-sample overfitting; a power analysis calibrates how a null is read. --- ## Analysis plan: five pre-registered steps 1. **Assembly:** match NICM cost records to NTRS design specifications by instrument identity and version; attach cal/val accuracy; freeze the table; log every unmatched or ambiguous instrument. 2. **Balance and support:** check covariate balance across cost terciles; compute weights; trim to common support and report what is trimmed. 3. **Baseline linear (H0) model:** accuracy linear in cost plus controls, clustered errors; record its leave-one-instrument-out cross-validated error as the benchmark. 4. **Semiparametric concave model:** estimate g(cost) under the shape constraint; compare out-of-sample; estimate g'' and its band. 5. **Over-specification test:** locate the channel-count edge; corroborate against the information-content prior. --- ## The decision rule (fixed in advance) - **Aggregate concavity (H1):** supported only if the semiparametric model beats the linear null in leave-one-instrument-out cross-validation AND g'' is reliably negative over a non-trivial portion of common support. - **Over-specification:** supported only if the marginal channel-count contribution reaches an indistinguishable-from-zero region within the observed channel range. - In-sample curvature alone retains H0, because in a small sample it is the expected symptom of overfitting. - Four graded outcomes are reportable, including a partial H1 (the sharper channel prediction holds while aggregate curvature is too imprecise to declare) and a clean, informative H0. --- ## Expected results (illustrative, not yet executed) **No result here is computed on the full assembled dataset. All shapes are design illustrations.** - **Under H1:** g(cost) rises steeply at low cost, then flattens where calibration drift, geolocation error, and retrieval difficulty cap accuracy; the marginal channel contribution declines to zero at an over-specification edge that falls near the information-saturation channel count. - **Under H0:** g(cost) is indistinguishable from a straight line; the marginal channel contribution stays positive across the observed range. - The result tables are pre-registered as shells and left unpopulated by design. --- ## Confidence and uncertainty Stated at design-stage grades, with what would raise or lower each. - **Problem is real / cost is a structured index:** very high (convergent across NICM and Stahl). - **Materiality under fixed toplines:** high (rests on the decision structure, not on the result). - **Concavity (H1) is true:** withheld at the design stage; raised by an out-of-sample win plus a reliably negative g''; lowered by an underpowered held-out test. - **Over-specification edge exists:** physically motivated, withheld statistically; raised if the statistical edge meets the information-saturation prior. - **Reference-ceiling rival distinguishable:** lowest confidence; the corpus is thin on reference-network uncertainty budgets, so it is probed, not dispatched. --- ## How the argument is built **Aim:** estimate the accuracy-cost frontier shape and its over-specification edge, to give cost-capped formulation a stopping rule or to remove diminishing returns as an assumption. | Part of the case | Where it is established | |----------|---------| | That the problem exists | Separate cost vs cal/val literatures (Ch 1, 3) | | That it matters for budgeting | Fixed toplines, distributed architectures (Ch 1, 7) | | That the design captures the mechanism | Partially linear hedonic + balancing + info-content (Ch 2, 5, 6) | | Why this design is preferable | Selection-on-observables + held-out test vs naive/overfit (Ch 5, 6) | | How the remaining risk is bounded | Named, scoped mitigations (Ch 4, 5, 6, 7) | **Residual risk:** the validation-reference ceiling could counterfeit high-cost flattening; unobserved confounding is bounded, not eliminated; the small instrument sample limits resolution. Each is named and either mitigated or scoped. --- ## Scope and what is not claimed - Not a structural causal model of retrieval physics; a reduced-form embodied-investment frontier. - Not a universal over-specification channel count; an edge conditional on the controls and the supported population. - Not a dollar valuation of accuracy; the frontier is the input any valuation would take. - Active sensors, non-NASA cost regimes, and future technologies are out of scope, named as scoped extensions (active-sensor analogue; commercial-radiometer analogue; the physics-versus-cost-regime question). - The work does not produce a systems or capability architecture; the single decision it yields is a plain-language management recommendation. --- ## Contribution restated - The first design for a population-level estimate of the accuracy-cost frontier across Earth-observing radiometers. - It inverts the Rosen hedonic surface onto validated accuracy, draws the cost regressor from NICM, draws the dependent variable from independent cal/val records, identifies the cost effect with the Abadie selection-on-observables discipline, and takes its concavity prior from Simon. - **If H1 holds:** a defensible stopping rule for spectral specification and a value-aware reallocation toward calibration, geolocation, or more instruments. - **If H0 holds:** diminishing returns removed as an unexamined budgeting assumption. - Either result moves the JPL Earth Science portfolio toward converting dollars into validated science by evidence rather than instinct. --- ## Selected references - S. Rosen, Hedonic Prices and Implicit Markets, *J. Polit. Econ.*, 1974. doi:10.1086/260169 - H. Habib-Agahi, J. Mrozinski, G. Fox, Latest NASA Instrument Cost Model (NICM): Version VI, NTRS 20160008251, 2014. - H. P. Stahl, Survey of cost models for space telescopes, *Opt. Eng.*, 2010. doi:10.1117/1.3430603 - G. Imbens, J. Wooldridge, Recent Developments in the Econometrics of Program Evaluation, *J. Econ. Lit.*, 2009. doi:10.1257/jel.47.1.5 - J. Hainmueller, Entropy Balancing for Causal Effects, *Political Analysis*, 2012. doi:10.1093/pan/mpr025 - H. A. Simon, The Architecture of Complexity, in *The Sciences of the Artificial*, 2019. doi:10.7551/mitpress/12107.003.0011 - C. D. Rodgers, Information content and optimisation of high spectral resolution remote measurements, *Adv. Space Res.*, 1998. doi:10.1016/S0273-1177(97)00915-0 - L. Remer et al., The MODIS Aerosol Algorithm, Products, and Validation, *J. Atmos. Sci.*, 2005. doi:10.1175/JAS3385.1 - S. Chan et al., Validation of SMAP surface soil moisture products with core validation sites, *RSE*, 2017. doi:10.1016/j.rse.2017.01.021 Full 141-entry reference list in the dissertation back matter. --- ## Defense questions anticipated - How is the cost variable's version mismatch quantified, not just acknowledged? - Could a validation-reference ceiling, rather than instrument capability, produce the flattening? - Is the sample large enough to estimate g(cost) flexibly, and how is overfitting ruled out? - Why exclude active sensors rather than control for sensor type? - What channel count would you pre-register as the expected over-specification edge, and why? - Under H0, what exactly does the result license, and what does it not? **Thank you. Questions welcome.**