Why now: what AI makes cheap

My hypothesis is that AI now makes the work behind shared, reproducible research cheap. The evidence so far comes from social science and machine learning, not lab data, so testing the assistant on condensed matter papers is the clear next step.

Reproducible research has always cost time: documenting each step, packaging data, rerunning someone else's analysis, checking the literature. Those are the tasks AI is getting good at.

Task	What AI does now
Repairing broken replication code	Agent workflows fixed deliberately broken analyses in 69 to 96% of cases, across five social science studies.¹
Reproducing published results, with people	AI-assisted teams reproduced 91% of results and human-only teams 94%, across 103 teams, and human-only teams found more major coding errors. AI didn't help here.²
Reproducing a machine learning experiment	About $1.87 per experiment, on a benchmark of 13 AI papers.³

1 Shah, Hopfgartner and Bleier 2026, arXiv 2602.08561 AI only

On five fully reproducible R-based social science studies with deliberately injected failures, agent-based repair workflows succeeded in 69 to 96 percent of cases, against 31 to 79 percent for prompt-based workflows. Scope: Five R-based social science studies; a preprint. Open the source Confidence: confident.

2 Brodeur et al. 2026, PNAS 123, e2524747123 AI only

In an experiment that randomly assigned 288 researchers to 103 teams reproducing results from leading social science journals, human-only teams reproduced 94% of results and AI-assisted teams 91%, though human-only teams found significantly more major coding errors. Scope: Quantitative social science; ChatGPT models of the GPT-4 era. Open the source Confidence: confident.

3 Zhao et al. 2025, AutoReproduce, arXiv 2505.20662 AI only

On a benchmark of 13 AI research papers, the average cost for AutoReproduce to reproduce one experiment was $1.87 with o3-mini as the model. Scope: 13 machine learning papers; a preprint. Open the source Confidence: confident.

Some lab software is ready for it. Many quantum device labs run measurements through open-source software that saves each measurement with its settings, though tuning done live often isn't saved, and AI models can now work with those instruments through the same software.⁴^,5

In one quantum device lab I've talked with, AI already writes instrument drivers and runs measurement sweeps. An experiment run this way logs every setting and every failed run as a side effect.⁶

The limits

The evidence is from social science and machine learning code. Nobody has tested an assistant like this on condensed matter papers and lab data, which are large and often in instrument-specific formats.
AI makes checking an analysis, capturing data and packaging it cheap. It doesn't make building a device or growing a sample cheap.
Open-source measurement software is common in quantum device labs. Many other labs run LabVIEW, MATLAB or vendor software for STM, PPMS and ARPES, and each would need its own connector for the run log.
It can't tell whether the instruments and samples are right, and control experiments stay expensive.
On its own, the phase 1 assistant can only ask about the devices that weren't shown; it can't see them. That's what the optional run log adds.

How I plan to test the assistant on condensed matter papers is in How I'll test it.

How I'll test it →

The limits

More in the appendix