Symbolic regression with a learned concept library

This NeurIPS 2024 paper introduces LaSR, a symbolic-regression method in which a large language model finds reusable concepts in strong candidate equations. On the Feynman benchmark it recovered more target equations than the methods it was compared with.

Published in Advances in Neural Information Processing Systems 37 and presented at NeurIPS 2024 in Vancouver, the paper by Arya Grayeli, Atharva Sehgal, Omar Costilla-Reyes, Miles Cranmer, and Swarat Chaudhuri introduces LaSR. The research was led by Sehgal and Grayeli in collaboration with Chaudhuri and Cranmer over a year-long effort, with Dr. Costilla Reyes contributing as a research scientist at MIT CSAIL, and was supported by the National Science Foundation under grant 1918839, Understanding the World Through Code.

Symbolic regression is the task of finding a compact mathematical expression that explains a dataset, rather than fitting the numbers inside a fixed model. Evolutionary methods keep a population of candidate expressions and improve them by random mutation and recombination. The paper’s starting point is that a human scientist does not mutate explanations at random, and LaSR is built on that observation. It extends PySR, an existing symbolic-regression system, by asking a large language model (LLM), without task-specific training, to read the best candidates found so far and write down abstract ideas they share, such as “exponential growth or decay.” These textual concepts form a library that is refined over successive rounds and used to steer new candidates, which combine standard evolutionary steps with steps proposed by the LLM. Because every candidate is still an explicit equation, the result stays readable.

On the Feynman equations, a benchmark of formulas from the Feynman Lectures on Physics, the paper reports that LaSR recovered more of the target equations than the other methods it compared, and that it also outperformed a state-of-the-art genetic method on synthetic tasks designed to rule out test-set leakage. The authors also used LaSR to propose a new scaling law for language models, showing that the method can suggest candidate laws for a current research question as well as rediscover known physics. Because the concept step is carried out by a language model, the approach benefits directly from advances in language models. The paper points to a project website for its released materials.

Dr. Costilla Reyes described the motivation as developing interpretability methods for critical settings such as healthcare, where a domain expert can check a readable equation in a way that is not possible with a neural network. The method sits alongside InceptionSR and the body-fat estimation study on the site, and its concept-library idea connects to LEMMA and MathDSL, which learn reusable abstractions for mathematical reasoning.