Limits of Procedural Solvers

Our procedural floorplan generator is a work of mathematical beauty. It uses recursive backtracking and constraint satisfaction to layout hallways, place bedrooms adjacent to bathrooms, and align doors perfectly. It outputs thousands of architecturally correct layouts a second. But when we used these layouts to benchmark vision-language models, the models scored near-perfectly. The worlds were too clean.

Real architecture is messy. It has misplaced columns, awkward half-bathrooms under stairs, and closets that block hallways. Models that successfully navigated our "clean" procedurally generated floorplans failed completely when confronted with real-world blueprints.

We had to rewrite our generator to seed "structural chaos." We introduced rules that occasionally break constraints—creating dead-ends, misplaced doors, and non-standard room layouts. Testing VLMs on these "broken" floorplans revealed deep reasoning gaps that our clean solver had masked. If your test harness doesn't contain errors, you aren't testing capability; you're testing compliance.