The pain points in detail
Five pain points, each labeled with how well it's documented, and how anyone could measure how big the problem is.
These are the pain points as I see them, from the literature and from conversations with physicists.
New phenomena are hard to interpret
A new signature often has ordinary lookalikes. In hybrid nanowires, zero-energy modes that resemble the Majorana signal can emerge without any topological transition. Four more 'smoking gun' signatures turned out most likely not topological once more of the data was examined (the Majorana example in this appendix).1,2
Documented for topological signatures. How often it happens across the field hasn't been measured.
Nobody can know all the related work
arXiv listed about 17,000 new condensed matter papers in 2024, more than 45 a day, and what didn't work mostly never becomes a paper. A lab head put it to me plainly: we usually don't know all the literature related to our experiment, and that is a common root of an honest misreading.3,4
The volume is measured. How often missed work leads to a misreading is a working physicist's judgment.
What labs learn stays in the lab
In a sample of 2014 articles from top journals in four fields, physics shared original data at the lowest rate, and asking authors directly works only part of the time: 39% of requests to Nature and Science authors succeeded. Physical Review journals require a statement saying where the data can be found, but public sharing is optional. Failed runs are seldom written up, and replications struggle to get published: journals turned down one group's replications as not novel.5,6,7,8
Sharing rates and reasons are measured in physics overall and in other fields. How often condensed matter data is lost hasn't been measured.
The exciting reading gets rewarded
Publishing high matters for funding and reputation, so it's natural to prefer the more exciting interpretation and to spend less time on the ordinary effects that could produce the same observation. It sits on a spectrum, and mostly comes down to where the time goes. The Pittsburgh report adds that reputation and authority often stand in for checking when readers decide which results to trust.4,9
The judgment of physicists in the field: a lab head I've talked with, and the report's authors.
Ruling out the simple explanation is expensive
Sometimes the only test of a trivial explanation is another experiment, on a modified device where the exciting effect shouldn't appear but an artifact would. That can take months. Samples also vary: maps of magic-angle graphene show the local twist angle varying by close to 0.1 degree within a single device, so one device seldom settles a question on its own, and confirmation from other devices often comes later.4,10
The variation is measured. The cost of control experiments is a working physicist's judgment.
How big is the problem?
Nobody knows yet. I found no base rate for how often headline claims in condensed matter fail, and no systematic replication study of the field. The evidence is a run of costly corrections in the hottest subfields, plus structural signs: little data shared and few replications published.
Physical sciences publish fewer positive results than social sciences, so publication bias may be smaller here than elsewhere. That count can't include null results that were never written up: 'we looked and saw nothing' leaves no paper.13
Three cheap ways to measure it:
First, read published referee reports. Nature Communications publishes them for papers submitted since November 2022, so you can count how often referees asked for data and whether they got it.14
Second, the test of the phase 1 assistant doubles as a reproduction study of the field, like the Institute for Replication's in economics.
Third, pick one subfield, list its headline claims since 2010, and record which still stand.