--- marp: true theme: default paginate: true --- # Does Flight Heritage Buy Reliability? A Cross-Mission Regression of Realized On-Orbit Failure Rates Against Heritage and Parts-Class **Candidate:** JPL_MGMT_SMA_TECH_04 COLLEGIUM 1st Battalion North Star / JPL category: Safety, Mission Assurance and Health Defense deck (design-stage) | 2026-06-15 --- ## The answer, first - This dissertation contributes a pre-registered, falsifiable specification that puts claimed flight heritage in direct competition with EEE parts-class and integration-test fidelity as predictors of realized JPL-class subsystem reliability. - The deliverable is the design and the falsification conditions, not estimated coefficients. - The value to the Safety, Mission Assurance and Health portfolio holds under either hypothesis: it tells the portfolio where the next assurance dollar buys the most reliability, and which of its two errors it is more often making. --- ## The contribution as H0 vs H1 A single head-to-head test of competing predictors of on-orbit subsystem failure. - **H0 (null):** Prior-flight heritage depth is the primary driver of delivered reliability for JPL-class spacecraft; heritage retains the dominant, stable association after parts-class and test-fidelity enter. - **H1 (alternative):** Realized subsystem failure rate is predicted more strongly by EEE parts-class and integration-test fidelity than by claimed heritage, once confounders are controlled. Decided by the relative magnitude and stability of the standardized heritage coefficient against the parts-class and test-fidelity coefficients. --- ## The problem - Few judgments in mission assurance carry as much quiet weight, or as little scrutiny, as the appeal to flight heritage at a design review to justify reduced test scope and relaxed parts decisions. - Heritage is a claim about a design's ancestry, not a measured property of the delivered article. - A heritage label can attach to a box rebuilt with different parts lots, flown to a new environment, and given a thinner test campaign than the original. - The decision-relevant failure mode: a weaker heritage claim substituted for a stronger one at the review board. --- ## What "heritage" actually denotes Four distinct claims travel under one word; only the strongest retires environmental risk. - **Design heritage:** the block diagram or algorithm has flown; the article is newly built. - **Build heritage:** same drawings, comparable process, often same supplier. - **Parts heritage:** same EEE parts, same qualified lots or class. - **Same-environment flight heritage:** an identical article has operated in the orbit and radiation environment the new mission imposes. The dissertation codes heritage as an ordinal depth so the regression can distinguish these, not collapse them. --- ## The literature gap - Castet and Saleh built nonparametric and mixture-Weibull reliability curves: substantial infant mortality, subsystem-specific hazards, environment and mass dependence. - Tafazoli and the platform health scorecard showed failures concentrate in a few subsystems. - The radiation and COTS literature shows parts-class and screening are first-order, mechanism-backed reliability drivers. - None of this isolates claimed heritage from parts-class and test-fidelity, holding environment and prominence fixed, on a JPL-class population. That inter-literature gap is the opening. --- ## Theoretical framework - A conceptual split: reliability as a property of the **delivered article** vs reliability as a claim about its **provenance**. They coincide only under a transfer assumption the data often violate. - Heritage is an informal, subsystem-level **maturity** claim; the maturity that matters is the article's maturity in its actual environment, not the design concept's. - The rival theory names two article-property mechanisms: parts-class sets the intrinsic defect and degradation rate; test-fidelity sets the probability latent defects are caught before launch. --- ## The causal-inference scaffold - The estimation is observational: no one randomizes heritage, parts-class, or test fidelity across spacecraft. - **Rubin potential outcomes:** reconstruct and defend the assignment mechanism; demonstrate overlap; complete the design before outcomes are examined. - **Angrist and Pischke design-based discipline:** identify the comparison, the exploited variation, and avoid bad controls (never condition on the test verdict, which is downstream of the regressors). - Strongest defensible claim: a conditional association under stated unconfoundedness, not an identified causal effect. --- ## Data sources - **NASA anomaly records (outcome):** LLIS and problem-reporting (PRACA) data at subsystem level, with time-to-failure and root-cause class. - **NTRS reports (rigor):** reliability, parts-stress, radiation-hardness-assurance, and qualification/acceptance test summaries. - **JPL mission archives (heritage):** heritage assessment matrices, parts control board records, EEE parts lists with class, integration and test plans. - **Unit of analysis:** the mission-by-subsystem cell. --- ## Variable construction - **Outcomes:** time-to-first-failure (right-censored); infant-mortality indicator on a window set from the mixture-Weibull early-subpopulation inflection. - **Heritage depth (ordinal):** none < design-only < design+build < design+build+same-environment flight. - **EEE parts-class (ordinal):** high-reliability (class S) > class B > commercial/COTS, plus a screening subindicator. - **Test-program fidelity (index):** thermal-vacuum, vibration/acoustic, screening/burn-in, system-level functional coverage. - **Controls:** environment, prominence, mass class, launch epoch. All regressors standardized for magnitude comparison. --- ## The specification The hazard of first failure for subsystem cell i in mission m at on-orbit time t: `log h(t) = log h0(t) + b1 * Heritage_i + b2 * PartsClass_i + b3 * TestFidelity_i + g * Controls_im + u_s` - `h0(t)`: flexible (piecewise-constant or Weibull-mixture) baseline. - `u_s`: Gaussian random intercept for subsystem type s. - Regressors standardized so b1, b2, b3 are directly comparable. - A discrete-time complementary-log-log companion estimates the same index for the infant-mortality outcome. --- ## Design and identification - **Estimator:** mixed-effects (frailty) survival model with subsystem random effects, plus a discrete-time cloglog hazard for infant mortality. - **Identification:** selection on observables. Condition on environment, prominence, mass class, epoch, subsystem. - **Overlap:** reported as a first-class result; restrict estimation to the common-support region. - **Bad controls:** never condition on the post-design test verdict. - **The test:** nested Model A (heritage + controls) to Model B (adds parts-class and test-fidelity). --- ## Survivorship correction (built in, not a caveat) - Sampling frame built from **launch manifests**, not surviving-mission documentation, so early-failed missions enter the frame. - **Inverse-probability-of-documentation weighting:** model the probability a cell is fully documented; weight by its inverse. - **Bounded sensitivity analysis:** vary the assumed failure behavior of undocumented cells and report how the heritage-versus-rigor comparison moves. - The falsification rule must hold within these bounds, not only at the point estimate. --- ## Threats to validity - **Internal:** omitted rigor or program competence confounding heritage. Conclusion stated as conditional association, not causation; bounded by sensitivity analysis. - **External:** JPL-class population; transfer to NewSpace and CubeSat bounded by subgroup analysis and offered as a replication hypothesis. - **Construct:** indices built from auditable fields, outcome-blind, with inter-coder reliability reported. - **Statistical-conclusion:** mission-clustered inference, subsystem random effects, minimum detectable difference reported. --- ## Analysis plan (eight steps, outcome-blind by order) 1. Build the mission-by-subsystem frame from manifests. 2. Code heritage, parts-class, test-fidelity (two coders; report agreement). 3. Construct outcomes and censoring. 4. Estimate documentation weights. 5. Restrict to the overlap region. 6. Estimate Model A and Model B for both outcomes (clustered SEs, random effects). 7. Run survivorship sensitivity. 8. Report the nested comparison and the pre-registered falsification decision. Steps 1 to 5 never touch the predictor-outcome relationship; only Step 6 does. --- ## Expected results (design-stage, not executed) This is a design-stage plan. The directions below are expectations that define the test, not measured results. - **Under H1:** heritage shows a moderate association in Model A, then attenuates and loses stability in Model B, while parts-class and test-fidelity carry the larger, stable coefficients. - The infant-mortality outcome is expected to load on **test-fidelity**, because early failure is an uncaught latent defect, which screening and system test exist to catch. - The estimates may contradict every expectation, in which case the contribution is falsified toward H0. --- ## The falsification rule (fixed in advance) - **H1 supported** if, across both outcomes and within the survivorship bounds, standardized parts-class and test-fidelity coefficients exceed the standardized heritage coefficient, and heritage's Model B interval includes negligible effect. - **H0 (contribution falsified)** if heritage remains the largest standardized coefficient with an interval excluding negligible effect after parts-class and test-fidelity enter. - **Unidentified** if the overlap region is too thin to support the comparison. Symmetric, decidable, and frozen before the data are seen. --- ## Confidence and uncertainty - **High confidence:** the estimator matches the documented hazard structure; the design is outcome-blind; the contribution is valuable under all three readings. - **Moderate confidence:** that the selection-on-observables assumption holds on this population; held deliberately, not assumed. - **Low / undetermined:** whether overlap is adequate, whether the sample clears the power floor, and which way the coefficients fall. - An underpowered non-result is reported as underpowered, never as confirmation of H0. --- ## Rival readings and how the design distinguishes them - **Bundling (unidentified):** heritage and rigor inseparable. Tested by the overlap diagnostic; a no-overlap finding is itself decision-relevant. - **Mediation vs confounding:** does heritage cause rigor, or merely co-occur with it? Tested by whether heritage predicts parts-class and test-fidelity, reporting both total and conditional associations. - **Program competence (unobserved common cause):** proxied by prominence and the documentation model; residual confounding bounded, not eliminated. - **Secular technology trend:** tested with finer epoch dummies and a stricter same-environment heritage re-coding. --- ## The argument in brief **The contribution:** a pre-registered, falsifiable specification competing heritage against parts-class and test-fidelity; the deliverable is the design and its falsification conditions. - **The problem is real:** heritage substitutes for rigor while infant mortality persists. - **The problem is material:** failure concentrates in few subsystems; test/parts are large budget lines. - **The mechanism is addressed:** nested A-to-B survival design with overlap and bad-controls discipline. - **It beats the alternatives:** descriptive curves and naive regressions cannot decide the test; design-based identification can. - **The residual risk is acceptable:** conditional association, survivorship corrected, underpowered/unidentified results reported honestly. --- ## Residual risk and scope - **Estimand:** a comparison of conditional associations, not a single causal contrast. - **Population:** JPL-class deep-space and science missions with documented heritage and parts records. - **Residual risk:** unobserved competence confounding (proxied, not removed); thin overlap (reported as unidentified if so); modest power (reported, not assumed away). - **Unit of analysis:** the unit is a statistical cell, so decision-relevance is carried through mission-assurance policy rather than an architectural mapping. --- ## Implications: where the next assurance dollar goes - **If H1:** move assurance attention from provenance review toward verification of the delivered article; grant the heritage discount only when parts-class is matched and a test-fidelity floor is met in the new environment; weight delta-qualification above design-lineage provenance. - **If H0:** keep current practice, now on a defended empirical warrant, and record heritage depth so shallow claims are not granted the discount deep claims earned. - **If unidentified:** require the rigor regardless of the heritage claim. --- ## Contribution, restated - A pre-registered, falsifiable cross-mission survival specification that decides whether claimed heritage or delivered-article properties predict JPL-class subsystem reliability. - Design-based identification, an overlap gate reported as a first-class result, survivorship corrected at the frame and the weight, and a falsification rule fixed before the data are seen. - The contribution is the specification and its falsification conditions, not estimated coefficients. - Next step: complete the JPL heritage and parts record assembly, then execute the frozen specification and report the falsification decision exactly as defined. --- ## Defense questions anticipated - Is there enough independent variation between heritage and rigor to identify the comparison, or are they bundled? - Is attenuation of heritage in Model B confounding or mediation, and how do you tell them apart? - How sensitive is the conclusion to the documentation-probability model? - Could unobserved program competence drive both heritage retention and low failure, and what bounds it? - What is the minimum detectable coefficient difference given the assembled sample? --- ## References (anchors) - Castet and Saleh, satellite and subsystem reliability program (2009 to 2013): nonparametric and mixture-Weibull reliability, subsystem concentration, mass and orbit dependence. - Tafazoli (2009); Saleh and Castet (2011): on-orbit failure review and platform health scorecard. - Dubos, Saleh, and Braun (2008): technology readiness, schedule risk, and slippage. - Brandhoff (2021); Dijks/Akay (2025): COTS radiation risk and radiation-hardness-assurance review. - Angrist and Pischke (2009); Rosenbaum and Rubin (1983); Rubin (2008); Imbens and Rubin (2015): design-based and potential-outcomes identification. Full IEEE bibliography with clickable DOIs in the dissertation References section.