How I'll test it

Each guess I'm making, by phase, and how I plan to check it.

Grouped by phase. For phase 1 the order is conversations with lab heads I know first, then the assistant test, then conversations with journals and funders with both results in hand. Phases 2 and 3 are tested as the earlier phases come into use.

Phase	Question	My guess	How I'll test it	What counts as a yes
Phase 1	Are these the pain points that matter most?	Yes, with hard-to-read new signals and missed related work at the top.	Ask lab heads to rank them and say what's missing.	They rank them without rewriting the list.
Phase 1	What would make their lives easier?	Methods and figures written for them, nothing lost when students leave, faster revisions.	Ask where they lose time today.	A benefit they name without prompting.
Phase 1	Does the assistant raise the lookalikes that turned out to be right?	Yes where the library is seeded, such as topological transport; unknown elsewhere.	Run it on Frolov's four 2026 cases, and on cases published after the model's training cutoff so it can't have memorized the answers.	It raises the explanations the field settled on, before being told.
Phase 1	Can the assistant rerun condensed matter analyses from figure data to figures?	Yes for clean code and data, harder for lab formats.	Run it on 10 to 20 condensed matter papers that already share data and code.	Most reproduce, and failures have clear causes.
Phase 1	Will referees trust the assistant's report and the researcher's answers?	Yes, if the researcher must answer their questions or decline with a reason they see, and the editor settles disagreements.	Show referees a mock report and conversation for a real paper.	They can sign off, or know exactly what to ask.
Phase 1	What does it cost per paper, in money and people's time?	Compute is cheap. The real cost is people's time, answering the assistant's questions and packaging the data; the goal is minutes.	Time it on a few real papers before submission, note every step that still needs a person, and compare the cost with what economics journals spend on checking.	Researchers call the time acceptable, and the cost is small next to current fees.
Phase 1	Who adopts first: would a journal or funder offer this?	One journal or one funder, in whichever subfield is most open to it, which others then copy. Journals that already require data statements are closest.	Conversations with a journal editor and a foundation program officer, ideally through introductions.	A first mover willing to discuss a pilot.
Phase 2	Would labs write short notes on what they tried?	Yes, if a note takes minutes and can be cited.	Ask 5 labs for one note each on something they tried and dropped, written with the assistant.	Most write one, in under an hour.
Phase 2	Do the notes help other labs?	Yes, when they're tagged well enough to find and say how much effort went in.	Show researchers notes from their subfield and ask whether any would have changed what they tried.	Some would have.
Phase 2	Do labs already keep their data in a central or cloud system?	Many do in some form, which would make the optional run log easier.	Ask two lab heads I know, then 3 to 5 more labs: which tools, where data lives, who can see it.	Most store data centrally already.
Phase 2	Is automatic logging light enough?	Yes, if it adds minutes, not hours.	Show a one-page mockup of the workflow and ask what they'd drop or fight.	They'd try it on one real project.
Phase 2	Can logging plug into the tools labs already use?	Yes, through their existing interfaces and open-source connectors.	List two labs' measurement and analysis tools and prototype one connector.	One lab's data flows in with no manual step.
Phase 2	Will labs trust a third party with their raw data?	Worries about being scooped and about security; encryption and an assistant-only check ease them.	Ask labs directly.	Access rules answer the worries.
Phase 3	Can AI flag selection by comparing final data with the full log?	Partly. It can show what share was used; judging whether that share is typical is harder.	Try it on a case with full public data, such as one of the published topological replications.	It flags the known problem.
Phase 3	Does pooled data show whether an artifact is common?	Yes, for well-documented device types with enough shared runs.	Pool shared data from one device type across a few labs and check whether a known artifact shows up across them.	The known artifact shows up where expected.
Phase 3	Would the field make the run log the default?	Only after labs that opt in find it worth the effort.	Track how many labs that try the optional log still use it a year later.	Most keep it.

More in the appendix