--- marp: true theme: default paginate: true --- # Science Productivity of Earth-Observation Data Policy A difference-in-differences study of open-data release on mission citation yield **Candidate:** JPL_ASTRO_EARTH_09 **COLLEGIUM 1st Battalion** North Star / JPL category: Earth Science Missions Hall-of-Shoulders anchors: North, Kuznets, Callaway and Sant'Anna 2026-06-15 Data gathered about the Earth are held in public trust; their terms of release should rest on measured benefit, not assertion. Dissertation defense brief. Design-stage; no result is executed. --- # The contribution This dissertation contributes the first mission-level, matched, staggered-adoption difference-in-differences design for estimating the downstream scientific yield of free-and-open Earth-observation data release. - **H0 (null):** open-data adoption has no effect on a mission's downstream publication rate or dataset-citation rate, relative to matched restricted-access missions. - **H1 (alternative):** open-data adoption produces a measurable upward break in those rates, in the periods after adoption. The deliverable is the pre-registered design and its falsification conditions, not estimated coefficients. Falsifiable in both directions. --- # The problem - NASA funds Earth-science missions to produce knowledge; the return is realized only when researchers obtain, analyze, and publish from the data. - Data-access policy sits on the path between the mission and its scientific return. - Restricted or fee-based access raises the cost a user faces; free-and-open access lowers it. - The scientific return on open release is today **asserted** at the directorate level and shown only on single before-after episodes, not **estimated** at the mission level. Policy question: does lowering access cost raise a mission's downstream scientific yield, and by how much? --- # The gap in the literature - Open-access and open-data citation advantages are well studied at the **article** level (Piwowar and Vision 2013; Colavizza et al. 2020; Eysenbach 2006). - A systematic review finds the magnitude contested and selection pervasive (Langham-Putrow et al. 2021). Two joint gaps: 1. Almost no work treats the **mission** as the unit and **mission open-data adoption** as the treatment. 2. Most evidence is associational and cannot remove author self-selection. The staggered timing of adoption across NASA Earth missions is an unexploited natural experiment. --- # Theoretical framework: two anchors **North (institutions, the mechanism).** An open-data policy is a rule change that lowers the transaction cost of access. Prediction: lower access cost raises impersonal downstream use; the response is path-dependent and gradual. FAIR separates nominal from functional openness. **Kuznets (measurement, the proxy stance).** Citation and publication counts are constructed proxies, not productivity itself. Discipline: state the proxy's error structure, and separate real new output from improved counting (the McMillan-Rodrik relabeling distinction). --- # The conceptual model the design tests ``` Free-and-open release (institutional rule change, North) -> transaction cost of obtaining and reusing data falls toward a download -> distribution / download volume rises first (Earthdata + DAAC logs) -> impersonal publication by non-team researchers rises -> formal dataset citation rises in the later window -> NASA / JPL weigh open-release cost against a measured yield ``` The mechanism's predicted ordering (distribution before publication) doubles as the measurement check: a publication rise with no distribution rise is read as relabeling, not new use. --- # Data: four real, named sources - **NASA ADS and Web of Science** - mission-linked publication and citation counts; the dual source guards against single-database indexing artifacts. - **NASA Earthdata and DAAC access logs** - distribution volume as an early-response outcome and mechanism check. - **Hand-coded adoption-date register** - open-data transition date, prior access regime, and FAIR functional-openness status per mission; double-entered and adjudicated. Unit: the **mission-period**. The panel is unbalanced; treatment timing is staggered; never-adopters and not-yet-adopters supply comparison information. --- # Outcomes and the frozen counting rule - **Outcome 1, publication rate:** peer-reviewed articles using the mission's data, by mission/instrument, dataset-identifier, and acknowledgment-text matching, with the rule **frozen across periods**. A no-prior-affiliation version isolates impersonal use. - **Outcome 2, dataset-citation rate:** formal citations to the mission's datasets; later window only, with explicit power caveats. - **Outcome 3, distribution volume:** access and download events; early-response and mechanism check. Freezing the rule is the operational form of the Kuznets within-versus-relabeling distinction. --- # Research design and identification - **Primary estimator:** Callaway and Sant'Anna (2021) group-time ATT, aggregated to a dynamic event-study path. - Naive two-way fixed effects is unreliable under staggered timing and heterogeneity (Goodman-Bacon 2021); it can place negative weights on already-treated controls. - **Matching** on sensor class and mission age (Rosenbaum-Rubin 1983; Stuart 2010) makes parallel trends plausible within strata; balance reported (Austin 2009). - **Identification:** conditional parallel trends within matched strata; not-yet-treated missions supply comparison information at every adoption date. --- # The estimand, in fixed notation For adoption cohort g and period t: `ATT(g, t) = E[ Y_t(g) - Y_t(0) | G = g ]` aggregated to a dynamic event-study path indexed by event time e = t - g: `theta(e) = sum_g w_g * ATT(g, g + e)` - Last pre-adoption period (e = -1) normalized to zero. - Leads (e < 0) test pre-trends; lags (e >= 0) trace the dynamic effect. - Count-appropriate (Poisson / log-link) with structural zeros; mission-clustered errors with wild-cluster bootstrap. --- # Threats to validity, each with a mitigation - **Internal:** parallel-trends violation (maturation, selection on trends). Defenses: lead tests, Roth power diagnostics, Rambachan-Roth sensitivity, mission age as covariate and control. - **Construct:** the outcome is a proxy; nominal vs functional openness. Defenses: frozen rule, no-prior-affiliation specification, distribution-log check, FAIR coding, dual database. - **External:** estimate is internal to NASA Earth missions; heterogeneity reported by sensor class, not averaged away. - **Statistical:** skewed counts, modest mission count; wild-cluster bootstrap, heterogeneity-robust estimators. --- # Why the naive estimator is rejected - Adoption is staggered; the North mechanism predicts heterogeneous, dynamic effects that build over event time. - Under exactly these conditions, the two-way fixed-effects coefficient is a weighted average that includes forbidden already-treated-as-control comparisons and can carry the wrong sign (Goodman-Bacon 2021; de Chaisemartin and D'Haultfoeuille 2020; Sun and Abraham 2021). - Callaway and Sant'Anna never uses an already-treated mission as a control and aggregates with transparent, non-negative weights. Honest qualifier: no estimator in this family relaxes parallel trends; the choice makes the aggregation explicit, not the assumption weaker. --- # Analysis plan (pre-registered, fixed order) 1. Assemble the mission-period panel; link ADS, Web of Science, DAAC logs, adoption dates. 2. Construct outcomes and covariates under the **frozen counting rule**. 3. Match and check covariate balance (proceed-or-revise gate). 4. Estimate the event-study path (Callaway and Sant'Anna). 5. Test pre-trend leads with Roth power diagnostics. 6. Robustness: Sun-Abraham, Borusyak et al., de Chaisemartin-D'Haultfoeuille. 7. Rambachan-Roth sensitivity; report the breakdown region. 8. Sensor-class heterogeneity; distribution and dataset-citation repeats. --- # The fixed decision rule Support for H1 requires **all three** jointly: 1. **Flat leads** - pre-adoption coefficients jointly indistinguishable from zero. 2. **Positive lags** - post-adoption path positive and the aggregated overall effect positive and distinguishable from zero. 3. **Robustness** - the effect survives the Rambachan-Roth sensitivity analysis at the stated breakdown threshold. Specific falsifiers, checked first: positive trending pre-leads void identification; a publication rise with no distribution rise is read as relabeling. Either falsifies the contribution. --- # Expected results (design-stage, illustrative) Numbers below are illustrative placeholders, **not** estimated on the full dataset. Result tables are specified but unpopulated by design. - Pre-adoption leads: near zero, intervals spanning zero (no pre-trend). - Post-adoption lags: a rising path consistent with gradual, path-dependent adjustment. - Distribution volume: an earlier, sharper break than publications. - Sensor-class heterogeneity: larger effect for optical imagers, consistent with the documented Landsat episode (Wulder et al. 2019, 2022). A flat lead-and-lag path would be a clean null and would falsify the contribution. --- # The Landsat precedent and its limit - Landsat's 2008 free-and-open switch was followed by distribution rising from tens of thousands to tens of millions of scenes per year, and a sharp expansion in publications (Wulder et al. 2019; Zhu et al. 2019). - Images used per change-detection study rose from about 10 to about 100,000 (Hemati et al. 2021). But Landsat is one mission, observed before and after one date, with **no contemporaneous control**. The contribution embeds the Landsat-type episode in a matched, staggered, multi-mission design that supplies the missing control. --- # Confidence and uncertainty The design supports a **conditional, moderate-confidence** reading at its strongest, capped by three structural features: - Small cohort (tens of missions, not hundreds); thin pools for some sensor classes. - The outcome is a biased proxy for productivity, not productivity itself. - Identification rests on an untestable conditional-parallel-trends assumption, probed by leads and bounded by sensitivity. Confidence **rises** with flat well-powered leads, a wide surviving sensitivity region, cross-estimator agreement, and distribution breaking before publication. It **falls** with trending leads, a narrow region, estimator disagreement, or a publication rise without a distribution rise. --- # Per-arm confidence - **Distribution volume:** best powered, earliest to respond; carries the mechanism reading. - **Publication rate:** the headline; adequately powered in larger sensor classes, marginal in thin ones. - **Dataset citation:** weakest; formal data citation is sparse early. Reported as exploratory; a null on this arm is **not** read as evidence of no effect. The per-arm calibration is the honest alternative to a single confidence figure. --- # How the argument holds together **The aim:** the design is fit to produce a credible, falsifiable, mission-level yield estimate, with both H1 and H0 reportable. | Part of the argument | Where it is carried | |---|---| | The question is a live one | Ch 1, 3: Piwowar-Vision; Colavizza; Landsat record | | The stakes are large | Ch 1, 3: Wulder; Zhu; Hemati; Turner; Apicella | | The design reaches the mechanism, not just the correlation | Ch 2, 4, 5, 6: Callaway-Sant'Anna; North; matching | | It earns its place over simpler estimators | Ch 3, 5: Goodman-Bacon; de Chaisemartin; Langham | | What it cannot rule out is bounded and stated | Ch 4, 5, 7: Kuznets proxy; Roth; Rambachan-Roth | **Residual risk:** a mission-specific, adoption-synchronized confound cannot be proven absent; it is bounded, not closed, by the sensitivity analysis. --- # Rival explanations and defenses - **Maturation** - age as matching covariate and control; flat-leads requirement. - **Selection on trends** - lead test, Roth power diagnostics, Rambachan-Roth breakdown region. - **Counting change (relabeling)** - frozen rule, no-prior-affiliation specification, distribution-log check. - **Concurrent policy shock** - era-contemporaneous matched comparison; cohort-specific ATT separates event time from calendar time. The rival the design cannot fully eliminate, a mission-specific adoption-synchronized confound, is named and bounded rather than assumed away. --- # Policy and mission significance - Moves the open-data-advantage question from the article level to the **mission level**, where the treatment is a policy event, not an author choice. - Gives NASA and JPL an identified, mission-level yield estimate, located in event time and decomposed by sensor class. - Converts an asserted benefit into a measured one, usable at mission formulation. - A **null is equally informative**: it would redirect the marginal dollar toward analysis funding, product maturity, or community size. The link between objective and decision is stated in plain language; this is an econometric policy-evaluation study and does not produce a systems or capability architecture. --- # What stands even if H1 is not confirmed Three assets are contributions regardless of the sign of the effect: 1. **The mission-level design** - a unit at which the policy lever NASA actually pulls can be evaluated. 2. **The frozen measurement discipline** - a transferable protocol for separating new output from relabeling. 3. **The assembled adoption register** - a structured, double-entered, FAIR-coded cross-mission asset that does not exist in the prior literature. A null from this design is a more informative null than the article-level literature can produce, because it is a null on an identified estimand. --- # Defense questions anticipated 1. How do you rule out that missions go open precisely when output is already rising? (Lead tests; Roth diagnostics; Rambachan-Roth.) 2. How do you separate real new research from improved counting? (Frozen rule; no-prior-affiliation; distribution logs.) 3. Why mission-level rather than article-level? (The policy lever and the funded unit are the mission.) 4. What if the matching pool is thin for some sensor classes? (Stated limitation; heterogeneity reported, not forced; MDE analysis.) 5. What single result would falsify your claim? (Zero overall effect, failed sensitivity, trending pre-leads, or publication without distribution.) --- # Contribution restated - One falsifiable claim, on the mission as the unit: open-data adoption raises mission publication and dataset-citation yield; the null is no effect. - Method: matched, staggered-adoption difference-in-differences event study with heterogeneity-robust estimators and a pre-registered decision rule. - Real named data: ADS, Web of Science, Earthdata/DAAC logs, hand-coded adoption dates. - North supplies the mechanism; Kuznets supplies the measurement discipline; Callaway and Sant'Anna supply the estimator. - Design and pre-registered analysis plan complete; results not yet executed; reproducibility is itself an application of the open-data principle under test. --- # Selected references - Callaway, B., and Sant'Anna, P. H. C. (2021). Difference-in-differences with multiple time periods. *J. Econometrics*. doi:10.1016/j.jeconom.2020.12.001 - Goodman-Bacon, A. (2021). Difference-in-differences with variation in treatment timing. *J. Econometrics*. doi:10.1016/j.jeconom.2021.03.014 - Rambachan, A., and Roth, J. (2023). A more credible approach to parallel trends. *Rev. Econ. Stud.* doi:10.1093/restud/rdad018 - North, D. C. (1990). *Institutions, Institutional Change and Economic Performance.* doi:10.1017/CBO9780511808678 - Piwowar, H. A., and Vision, T. J. (2013). Data reuse and the open data citation advantage. *PeerJ*. doi:10.7717/peerj.175 - Wulder, M. A., et al. (2019). Current status of Landsat program, science, and applications. *Remote Sens. Environ.* doi:10.1016/j.rse.2019.02.016 Full 148-entry reference list in the dissertation back matter.