# ASTRA: evidence-network pilot and acceptance gates

Status: prospective plan, not a completed study or evidence of effectiveness.

## What is implemented

- Claim-level evidence, methods, uncertainty and reproducibility assessments, with reviewer expertise, relationship and conflict declarations. A decision author cannot submit a separate-account peer assessment on their own decision.
- Immutable protocols, structured outcomes, revisions and source-change reassessments. Supporting, non-supporting, inconclusive and technical-failure outcomes use the same required fields and appear in the invited outcome registry.
- Direct and downstream source-impact lookup, dependency paths and named reassessment owners. Warnings appear when records open or refresh. Conclusions require explicit human revision.
- On-demand Crossref update-notice checks, with timestamps, deduplication and explicit errors. An empty metadata response is not evidence that a source is unchanged.
- Connected JSON export. Hosting, membership and administration are centralized; this is not a federated network.

## Recruit a small accountable pilot

Appoint one lead investigator and a coordinator. Seek university systematic-review or evidence-synthesis researchers in psychology, education or health-services research; an academic evidence-synthesis librarian; and a research-methods specialist or statistician. Select people with an actual bounded review question and inspectable source material. Recruit two reviewers outside the participating team's reporting line; verify expertise and disclose relationships manually. Distinct accounts alone are insufficient.

Invite three small teams to use ASTRA on existing, non-sensitive research material. Participation, publication permission and any institutional ethics requirements must be resolved by the investigators. Do not invite people or publish their work automatically. Credit critical review, reproducible artifacts and well-documented null outcomes equally; never reward a particular scientific conclusion or a positive product review.

## Freeze the validation protocol before data collection

1. Define one primary task per team: inspect a conclusion, trace a changed source, or decide whether an attempted experiment merits repetition.
2. Freeze an adjudicated reference set of 20 source/decision cases: direct and multi-hop links, branched chains, unrelated decisions, corrections, retractions, missing metadata and technical failures. Separate synthetic routing fixtures from real publisher notices. A reference set prepared after seeing ASTRA's answers is exploratory, not confirmatory.
3. Two qualified assessors independently identify affected claims and justified conclusions; record disagreement and adjudicate it with a third reviewer. Keep this reference work separate from product implementation.
4. Compare each team's existing workflow with ASTRA on matched tasks. Randomize order, prevent reuse of the same answer, and record setup, training, task and rework time. Predefine correctness criteria and a noninferiority margin with a methods specialist before recruiting the final sample.
5. Preserve all assignments, incomplete tasks, failures, corrections and reviewer scores. Report paired time differences and uncertainty alongside errors, missed dependencies and inappropriate warnings. A small convenience pilot establishes feasibility, not general effectiveness.

## Proposed release gates, to be agreed before the pilot

- Every seeded dependency in the frozen routing set is reached; unrelated decisions are not flagged; original records remain unchanged. Any missed seeded dependency blocks the monitoring release until explained and fixed.
- Real source notices are linked to the correct DOI and direction. Measure notice recall only against a defined reference set and time window; never infer exhaustive coverage from Crossref alone.
- No participant mistakes an inconclusive outcome or technical failure for proof that a hypothesis is false. Observe this in tasks, rather than accepting a checkbox as evidence.
- An independent assessment of the full investigation, including disagreements and raw artifacts, is completed before calling a case independently reviewed.
- Claim time savings only if the preregistered comparison supports it without unacceptable quality loss. Publish adverse and inconclusive product findings too.

## Before unattended monitoring can be enabled

The current deployment has no authenticated unattended updater. Provision a least-privilege service identity and supported scheduler with access to the invited records; do not reuse a user's browser session or expose credentials. Select an explicit DOI watch list and responsible owner for each decision. Add persistent last-success/checkpoint state, bounded batches, provider backoff, retry/idempotency tests and a visible stale-monitor warning. Test with a fake clock and controlled notice feed before a limited real-source run. Email or other notifications require explicit opt-in and a connected delivery service.

Acceptance evidence: recorded successful scheduled runs, an injected update reaching all expected decisions, failure and recovery logs, and a stale check visibly marked overdue. A schedule that exists but has not successfully run does not meet this gate.

## Before calling the network decentralized

Publish a versioned interoperable record schema and governance policy; implement validated import as well as export. Agree on identity verification, authorship, conflicts, appeals, abuse handling, privacy and correction rules with participating institutions. Demonstrate an independent installation exchanging a record and preserving provenance, permissions and revisions. JSON export alone is portability, not federation.

## Publish the first real case

Use the existing pilot guide and Validation Lab to record the protocol and evaluation. Publish only after authors and reviewers authorize the exact artifacts. Include the question, search log, source passages, conflicting evidence, protocol, every outcome, revisions, review declarations, unresolved disputes and limitations. Link the data and code with a durable repository identifier where feasible. Do not replace the fictional demonstration with fabricated participation or a demonstration labeled as a completed study.

## Extension: living claim review

The next pilot should include new contradictory claim links, author responses, reviewer resolution/reopening and delayed external notice checks. Measure whether participants distinguish a workflow label (Contested) from a scientific verdict. Verify the visible-page watcher updates indicators without losing a draft, and that it reports failed checks. It does not search new papers or continue after the page closes.

Claims now have stable library citation links and recorded supports/challenges/qualifies relationships. Test reuse across two real investigations, including repeated DOIs and shared datasets; distinct source identifiers are not proof of independent evidence. Claim-linked protocols make next tests and replication outcomes inspectable. The gap profile reports underlying counts without inventing a calibrated score.

Methodological references: [Cochrane on living systematic reviews](https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-22) describes ongoing surveillance and explicit update protocols. [GRADE Working Group](https://www.gradeworkinggroup.org/) describes multidimensional evidence-certainty assessment. ASTRA's workflow counts do not implement or replace GRADE, a statistical fragility index, or expert appraisal.