PlatBench: LLMs can't generate CAD from survey plats

Today we're sharing the results of PlatBench. We evaluated 8 LLMs on tasks a CAD Technician might do for a client to better understand the gap between GIS-adjacent professionals and today's AI models.

When given real-world documents, today's models can't reliably follow surveyors' diagrams and generate CAD files. We asked LLMs to recreate property boundaries in CAD of typical deeds, easements, and plats — but instead they filled their context window with poorly executed GDAL calls (ogrinfo foot gun), failed to determine the relevant text in a sea of markings, and submitted work with preventable errors (much of this task is verifiable).

Claude Opus 5, then Kimi K3, then OpenAI 5.6 Sol

PlatBench separates frontier models significantly, likely attributable to visual reasoning over engineering diagrams and CAD familiarity.

Task difficulty was calibrated such that pass@8 was high while pass@1 was low for a typical frontier model. Claude Opus 5's high score should be understood in this context: the median difficulty of PlatBench tasks is significantly easier than the typical, poorly defined cadastral mapping task. CAD Technicians work with large sets of partially relevant documents and figure out which subset satisfies their standard of accuracy.

We also find that the strongest models separate themselves by accomplishing more per dollar. This is especially clear when considering Claude Sonnet 5 and Gemini 3.6 Flash, which are considerably more expensive per task than OpenAI 5.6 Luna despite achieving comparable accuracy.

Models continue to make silly mistakes

We wrote about how difficult it is to digitize a plat automatically four years ago, but as models have developed better problem solving skills, their weaknesses are now qualitatively different.

I analyzed the reasoning traces of models as they succeeded and failed and found why frontier models failed at some tasks while others succeeded.

OpenAI 5.6 Sol doesn't consider discrepancies in its inputs

This simple diagram showing a property boundary near Barbados is more complex than it appears: despite having a straightforward legal description in the top in red text, the text is truncated and disagrees with the metes and bounds accompanying the diagram.

For example, 24 DEGREES 08 MINUTES 51.51 METRES in the diagram is annotated with 69° 08', which the model must realize is a printing mistake for 249 DEGREES ... — 180° off the 69° 08' label.

Most big models realize the description text doesn't match the diagram and proceed carefully, and our task prompt even warns of this (removing that hint significantly hurts models' ability to complete the task). Moonshot's Kimi K3 spots it, reasoning (bold is mine):

Red (untruncated guesses): [2]49°08', 339°04', [2]29°44'?? hmm ‘29 DE?’ probably ‘29 DEGREES 44 MINUTES’ full. Or ‘229’? Then 80°24' or [2]80°24', 10°54' or [x]10, 31°13' or [x]31, ??°59' with degrees fully missing, 154°33'.

Candidate from brute force mask (1,0,1,1,1,1,1,0): 249.13, 339.07, 29.74, 80.40, 350.90, 31.22, 68.98, 154.55.

Red guesses: 249°08' ✓ (‘24 DEGREES’ = ‘249’ truncated!), 339 ✓, 29°44' ✓, 80°24' ✓, leg5: red ‘10 DEGREES 54’ vs mask B 350°54'...

OpenAI 5.6 Sol's hidden reasoning summary doesn't ultimately connect the typo with its final .dxf:

There's an inconsistency between bullet 69 and transcript 24 — maybe a typo from OCR?

I need to correct a measurement for ‘69’ based on some truncation notes. The angle ‘24 degrees 08’ seems to relate to the same point.

Claude Opus 5 one-shots a GeoParquet reader in a fluke

In 30 of the US states (mostly the west, excluding the 13 original colonies, Texas, Louisiana etc. with more complicated land grant systems), surveyed land is based on the Public Land Survey System, shorthand PLSS. Many boundaries establish a point of commencement by referring to a particular corner of some PLSS section.

For one task, I'd provided a state PLSS GeoParquet and instructed the model to treat it as monuments with coordinates.

But while I had written GDAL into the Dockerfile, I omitted the Apache Arrow library which lets you read/write GeoParquet files. Oops.

Most models, like Grok 4.6, complained:

Need to figure out how to read parquet without pyarrow...

I need to parse the parquet file myself since GDAL doesn't have parquet support. Let me look at the parquet format and parse it with Python.

Before failing. But Claude Opus 5 one-shot a Parquet reader in Python before completing the task with an average distance of 2.1 mm (within a typical 1:10,000 accuracy standard common in the industry). Fantastically impressive.

Gemini 3.7 consistently fails while 3.6 succeeds

One of the more disappointing results of this benchmark was yet another provider error preventing us from properly evaluating Gemini 3.7 Flash. Through OpenRouter, hitting both Vertex and AI Studio, we consistently got MALFORMED_FUNCTION_CALL while all other Chat Completions-compatible endpoints succeeded, including 3.6 Flash.

Once this bug is fixed upstream, we'll update the results of PlatBench with 3.7 Flash, which has shown lots of promise in visual reasoning.

Releasing the tasks

We are not releasing these tasks on the public internet to prevent benchmark contamination.

If you are a researcher looking to improve performance on professional tasks relevant to architecture, engineering, and construction (AEC), email me at brendan@buntinglabs.com.

PlatBench score table