Creation Garden research status and corrections
By Carlos Shields-Davis ·
What the review found
- Execution failure was mixed with behavior. Some unsuccessful model requests were recorded as STAY actions. Apparent inactivity therefore could reflect the harness rather than an agent's choice.
- Action meanings were not preserved. INSPECT and unrecognized actions could also become STAY before logging. The original raw action cannot always be reconstructed from those logs.
- Some executions appeared repeatedly. Appended runs produced duplicated experimental cells, affecting independence assumptions.
- Model comparisons pooled different setups. Horizons, generations, and conditions differed across parts of the corpus. The earlier 173× variance ratio was sensitive to those choices and is withdrawn as a settled magnitude.
- Other quantitative conclusions also need revision. The review identified statistical and claim-traceability problems. Removing the variance headline alone would not validate the remaining original conclusions.
What remains a research question
Creation Garden asks how agents inherit, repeat, and use natural-language claims about a simulated world. Repeating a statement, explaining it, and changing an action because of it are different observations. A valid comparison must keep those distinctions visible.
What a corrected analysis needs
The repair requires identifiable unique runs, explicit execution-error outcomes, preserved action semantics, and matched protocols before comparing models. No replacement effect size or completed replication is claimed here.
Read the historical working paper with this status notice or the revised note on model comparisons.
Record of correction
On 17 October 2026, the website removed the earlier numerical and model-rail headlines, narrowed the project description, and added a status notice to the downloadable paper. The underlying historical paper pages are retained so the record is visible.