AI agents struggle to independently replicate scientific research despite their ability to extract data and perform calculations. A study titled Reconstructing Implicit Scientific Knowledge evaluated LLM agents across 14 astronomy papers, including 13 from Nature and one from The Astrophysical Journal.
In 11 of the 13 Nature articles, researchers identified methodological or interpretative ambiguities that prevented a clear sequence of actions for reproduction. The team tested 12 different analysis variants for a single dataset, which yielded distance estimates ranging from 2.16 to 3.53 kiloparsecs, compared to the original 2.70 kiloparsecs.
The discrepancy often stems from an inability to link specific parameters to the underlying research logic. For instance, AI agents failed to recognize the causal necessity of a plus 0.02 milliarcsecond parallax correction until the analysis demonstrated its impact, even though the value was explicitly stated in the source text.
The authors of the study argue that matching a final number does not prove that an AI has reconstructed the researchers' reasoning. They propose evaluating scientific AI systems based on the entire reproduction process, including data selection and configuration, rather than focusing solely on the final output.
