Skip to main content
Article

Emergent Representations of Program Semantics in Language Models Trained on Programs — research context

Dr. Costilla Reyes shared research by Charles Jin and Martin Rinard on whether language models trained on programs learn what programs mean. He shared it in May 2023 as part of the NeurSym and MIT CSAIL community.

Written by Dr. Omar Costilla Reyes
Diagram shared with the May 2023 post, from the study by Charles Jin and Martin Rinard, showing a probe reading a language model’s transcript of programs to test whether it encodes meaning. Credit: Image courtesy of Dr. Omar Costilla Reyes.

Research shared from the NeurSym and MIT CSAIL community

On May 30, 2023, Dr. Costilla Reyes shared a study of whether a language model trained only on program text can learn what the programs mean. He shared it as a research scientist and research project manager at the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL), where he worked on the National Science Foundation expedition Understanding the World Through Code, the multi-institution project behind the Neurosymbolic Programming for Science (NeurSym) community. Charles Jin and Martin Rinard authored the paper, “Emergent Representations of Program Semantics in Language Models Trained on Programs,” which appeared at ICML 2024; Dr. Costilla Reyes shared it as research from the wider NeurSym and MIT CSAIL community.

The question matters for the kind of research he coordinated there. Neurosymbolic programming combines learned models with symbolic structure such as programs, so evidence about what a language model internally represents when trained on code bears on how far such models can be trusted as components of systems that reason over programs.

How the study probes program meaning

The researchers trained a Transformer, a neural network architecture, only to predict the next token, or word-like unit, in a synthetic set of programs for navigating grid worlds. Each program was paired with a partial specification of input and output grid states. They then used probing classifiers, small models trained to read the network’s hidden states, to test whether those states encoded the intermediate grid states that the program text never states. According to the paper’s abstract, the probes recovered these unobserved states more accurately as training progressed, which the authors interpret as the model acquiring an ability to interpret the programs. They also introduced an intervention baseline to help separate what the language model represents from what the probe itself learns.

Why it matters

The paper offers an experimental framework for studying how meaning is acquired. Using a synthetic programming language gives the researchers full control over what the model sees and what its programs mean, so each internal representation can be checked against the true program behavior. The finding that program meaning emerges in the model’s hidden states from next-token training alone gives the NeurSym community a concrete, measurable way to ask how language models understand code, and it informs the design of neurosymbolic systems that combine learned models with programs.

Topics