49.1% of applicable items were fully reported
Each of the 189 eligible papers was assessed against 32 reporting items derived from Princeton's REFORMS. Of 5,913 applicable item judgments, 2,904 were fully reported, 1,667 were partly reported, and 1,342 were absent or unclear. The secondary score gives half credit to partial reporting.
The 49.1% figure is an item-level reporting rate; papers were not assigned pass/fail results. Every eligible paper fully reported at least 2 items.
189 candidates met the full scoring criteria
The candidate frame includes papers whose titles or abstracts use explicit ML language. Screening then asks whether model performance or output is used as evidence for a scientific claim, and whether a complete original paper is available.
Eligibility outcome
| Outcome | Papers | Share |
|---|---|---|
| Eligible full ML-based-science papers | 189 | 29.7% |
| Not ML-based science | 435 | 68.4% |
| ML-based science, but no complete paper body | 8 | 1.3% |
| Unresolved | 4 | 0.6% |
Scientific-use classification
| Class | Papers | Share |
|---|---|---|
| ML-based science | 197 | 31.0% |
| ML methods research | 141 | 22.2% |
| ML mentioned, but not used as evidence | 126 | 19.8% |
| Predictive analytics | 105 | 16.5% |
| Non-ML quantitative analysis | 62 | 9.7% |
| Unresolved | 5 | 0.8% |
The 197 ML-based-science candidates comprise 189 complete original papers and 8 records without a complete paper body, which could not be scored against the full reporting standard.
Publication form in the candidate set
Publication form is recorded separately from scientific use. This keeps reviews, editorials, responses, and conference abstracts visible as sample outcomes without treating them as complete empirical papers.
| Publication form | Papers | Share |
|---|---|---|
| Original Research | 488 | 76.7% |
| Review | 63 | 9.9% |
| Systematic Review | 19 | 3.0% |
| Editorial Opinion | 18 | 2.8% |
| Conference Abstract | 16 | 2.5% |
| Other | 12 | 1.9% |
| Bibliometric Review | 7 | 1.1% |
| Commentary Response | 6 | 0.9% |
| Meta Analysis | 3 | 0.5% |
| Protocol | 3 | 0.5% |
| Uncertain | 1 | 0.2% |
Pre/post comparison
The currently eligible papers include 90 before REFORMS and 99 after it. Full reporting was 49.0% before and 49.2% after, a change of +0.2 percentage points. The half-credit score changed by -0.5 points. These figures remain descriptive: the comparison contains 189 papers, and human validation has not started.
| Period | Papers | Applicable items | Present | Partial | Absent or unclear | Full reporting | Half-credit score |
|---|---|---|---|---|---|---|---|
| Before REFORMS | 90 | 2,815 | 1,379 | 814 | 622 | 49.0% | 63.4% |
| After REFORMS | 99 | 3,098 | 1,525 | 853 | 720 | 49.2% | 63.0% |
Paper-level reporting varied substantially
Every eligible paper reported at least some REFORMS-aligned information. The table treats each paper equally; the preceding comparison treats each applicable item equally.
| Period | Papers | Mean paper score | Median | Range | Papers with zero present items |
|---|---|---|---|---|---|
| Before REFORMS | 90 | 49.0% | 48.4% | 9.7% to 93.8% | 0 |
| After REFORMS | 99 | 49.2% | 50.0% | 6.2% to 90.3% | 0 |
Run instructions and reproduction workflows were rarely reported
Across the 189 papers, run instructions and end-to-end reproduction workflows were almost never supplied. Code versions, missing-data frequencies, uncertainty estimates, external validity, exclusions, and safeguards against leakage were also often absent or incomplete.
| Item | Reporting requirement | Present | Partial | Absent | Full reporting |
|---|---|---|---|---|---|
R2d | README or equivalent run instructions supplied | 1 | 8 | 180 | 0.5% |
R2e | Reproduction script or workflow supplied | 12 | 28 | 149 | 6.3% |
R2b | Code and code version identified | 7 | 40 | 142 | 3.7% |
R3f | Missing data frequency reported | 15 | 45 | 122 | 8.2% |
R6b | Dependencies or duplicates across train/test partitions addressed | 21 | 37 | 118 | 11.9% |
R7b | Uncertainty estimates reported with calculation method | 40 | 43 | 106 | 21.2% |
R4b | Impossible or corrupt sample handling described | 49 | 49 | 88 | 26.3% |
R8a | External validity evidence reported | 39 | 67 | 83 | 20.6% |
R6a | Preprocessing and modeling restricted to training information where required | 33 | 72 | 69 | 18.8% |
R4a | Sample exclusions identified and justified | 77 | 53 | 57 | 41.2% |
R5e | Hyperparameter tuning reported | 52 | 88 | 42 | 28.6% |
R5f | Baseline comparisons justified | 96 | 51 | 42 | 50.8% |
Shown: the 12 items with the largest absent counts. The downloadable item table contains all 32 items, including unclear and not-applicable counts.
Study goals were reported more consistently than reproducibility details
Module-level scores show the contrast. Authors generally explained their scientific goals and models, but often did not provide the artifacts needed to rerun the analysis or the evidence needed to judge transportability.
| Module | Present | Partial | Absent | Full reporting | Half-credit score |
|---|---|---|---|---|---|
| Computational Reproducibility | 101 | 315 | 528 | 10.7% | 27.4% |
| Data Leakage | 159 | 189 | 191 | 29.3% | 46.8% |
| Metrics Uncertainty | 194 | 181 | 113 | 39.8% | 58.3% |
| Generalizability Limitations | 189 | 97 | 92 | 50.0% | 62.8% |
| Data Preprocessing | 282 | 132 | 148 | 50.2% | 61.9% |
| Data Quality | 745 | 413 | 155 | 56.7% | 72.5% |
| Modeling | 701 | 307 | 111 | 62.6% | 76.4% |
| Study Goals | 533 | 33 | 1 | 94.0% | 96.9% |
Explicit REFORMS adoption
None of the 189 eligible papers cited REFORMS, and none supplied a REFORMS checklist. That is unsurprising for a small early sample: publication lags, indirect adoption through journal policy, and broader reproducibility norms can all affect reporting without producing a citation.
Human validation has not started
Codex performed the screening and item-level assessment from the retrieved paper texts. Human validation checks eligibility decisions and individual item scores so that the project can estimate AI error and place uncertainty bounds around later results. The current 636 screening decisions come from a frozen 1,200-candidate sample.