Research shared from the NeurSym and MIT CSAIL community
On May 30, 2023, Dr. Costilla Reyes shared a study of whether a language model trained only on program text can learn what the programs mean. He shared it as a research scientist and research project manager at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL), where he worked on the National Science Foundation expedition Understanding the World Through Code, the multi-institution project behind the Neurosymbolic Programming for Science (NeurSym) community. Charles Jin and Martin Rinard authored the paper, “Emergent Representations of Program Semantics in Language Models Trained on Programs,” which appeared at ICML 2024; Dr. Costilla Reyes shared it as research from the wider NeurSym and MIT CSAIL community.
The question matters for the kind of research he coordinated there. Neurosymbolic programming combines learned models with symbolic structure such as programs, so evidence about what a language model internally represents when trained on code bears on how far such models can be trusted as components of systems that reason over programs.
How the study probes program meaning
The researchers trained a Transformer, a neural network architecture, only to predict the next token, or word-like unit, in a synthetic set of programs for navigating grid worlds. Each program was paired with a partial specification of input and output grid states. They then used probing classifiers, small models trained to read the network’s hidden states, to test whether those states encoded the intermediate grid states that the program text never states. According to the paper’s abstract, the probes recovered these unobserved states more accurately as training progressed, which the authors interpret as the model acquiring an ability to interpret the programs. They also introduced an intervention baseline to help separate what the language model represents from what the probe itself learns.
Why it matters
The paper offers an experimental framework for studying how meaning is acquired. Using a synthetic programming language gives the researchers full control over what the model sees and what its programs mean, so each internal representation can be checked against the true program behavior. The finding that program meaning emerges in the model’s hidden states from next-token training alone gives the NeurSym community a concrete, measurable way to ask how language models understand code, and it informs the design of neurosymbolic systems that combine learned models with programs.