Essay · Methodology
The Substrate Problem
A behavioural claim about “LLM agents,” measured on a single model, is mostly a claim about that model.
Key Takeaway
Evaluate LLM agent architectures across at least three heterogeneous model substrates to distinguish model-specific quirks from generalizable agent behaviors.
Open most papers on LLM multi-agent systems and you will find a confident, model-agnostic headline: agents cooperate, agents propagate false beliefs, agents form coalitions, agents are robust. Read the methods and you will usually find a single model behind all of it. The substrate is treated as an invisible constant — a neutral medium through which the real phenomenon is observed.
It is not a neutral medium. In one experiment I ran the same multi-agent protocol across four different models. The variance explained by which model was running the agents dwarfed the variance explained by the experimental treatment, by a factor of roughly 173. The thing we were nominally studying was a rounding error next to the thing we were holding constant.
Rails and regimes
Two of the four models sat on behavioural rails. One never moved off its default no matter how it was manipulated; the other could not hold the task together under any condition. Only two were in a regime where the treatment effect was even measurable.
This is the trap. A study run on the first model reports “no effect.” A study on the second reports “total collapse.” A study on either of the middle two reports a clean, publishable treatment effect. Same protocol, three incompatible abstracts — chosen, in practice, by whichever model was convenient that month.
Why it keeps happening
Cost and momentum. You pick one capable model, run the condition matrix, and write the paper. Running the full matrix across several heterogeneous substrates multiplies the spend and the engineering, and the field does not yet demand it. So the substrate stays invisible — until the next lab uses a different model and discovers a different “law” of agent behaviour.
A modest standard
Behavioural claims about LLM agents should be run on at least three heterogeneous substrates, with at least one verified to be in an active regime — that is, a model the treatment can actually move. If a finding only appears on one substrate, that is itself the finding, and the title should say so. Until then, the honest version of most agent results has a model name in it.