PlatBench: LLMs can't generate CAD from land surveys

Today we're sharing the results of PlatBench. We evaluated 8 LLMs across 14 task configurations drawn from five real survey sets. These are tasks a CAD Technician might do for a client, and they help us better understand the gap between GIS-adjacent professionals and today's AI models.

When given real-world documents, today's models can't reliably follow surveyors' diagrams and generate CAD files. We asked LLMs to recreate property boundaries in CAD of typical deeds, easements, and plats — but instead they filled their context window with poorly executed GDAL calls (ogrinfo foot gun), failed to determine the relevant text in a sea of markings, and submitted work with preventable errors (much of this task is verifiable).

Land rights are surveyed: they're not GPS coordinates

In many countries, land rights are defined by surveying a property boundary by measuring very precise distances from known points. These form legal descriptions of land that are passed between people and corporations, like deeds and easements.

For example, an electric utility might want to install a new power line pole. The easement PDF says you can install power lines in a the corner of Alice's backyard— and a utility employee must determine where precisely the project can proceed, according to the legal description.

To do this, a CAD Technician would map the easement by finding a verified monument (the county sometimes gives GPS coordinates) that can be traced to the legal description, map the boundary, and compare the location with the construction project. This sort of work happens regularly for anyone dealing with land rights.

Claude Opus 5, then Kimi K3, then 5.6 Sol

PlatBench separates frontier models significantly, likely attributable to differences in visual reasoning over engineering diagrams and CAD familiarity.

Task difficulty was calibrated such that pass@8 was high while pass@1 was low for a typical frontier model. Claude Opus 5's high score should be understood in this context: the median difficulty of PlatBench tasks is significantly easier than the typical, poorly defined cadastral mapping task. CAD Technicians work with large sets of partially relevant documents and figure out which subset satisfies their standard of accuracy.

The latest sweep adds three Southern California easement tasks from Quail Hill. Six models completed the fully transcribed level, but no model cleared the point-of-commencement gate at levels 2 or 3. The result is a meaningful drop in every aggregate score and a sharper separation between the benchmark's easiest and image-dependent tasks.

Performance and price do not move together. OpenAI 5.6 Sol now delivers the strongest score per dollar, while Claude Opus 5 retains the highest absolute score. Claude Sonnet 5 and Gemini 3.6 Flash remain considerably more expensive per task than OpenAI 5.6 Luna despite achieving comparable accuracy.

Models continue to make silly mistakes

We wrote about how difficult it is to digitize a plat automatically four years ago, but as models have developed better problem solving skills, their weaknesses are now qualitatively different.

I analyzed the reasoning traces of models as they succeeded and failed and found why frontier models failed at some tasks while others succeeded.

OpenAI 5.6 Sol doesn't consider discrepancies in its inputs

This property-boundary diagram is more complex than it appears: despite having a straightforward legal description in the top in red text, the text is truncated and disagrees with the metes and bounds accompanying the diagram.

For example, 24 DEGREES 08 MINUTES 51.51 METRES in the diagram is annotated with 69° 08', which the model must realize is a printing mistake for 249 DEGREES ... — 180° off the 69° 08' label.

Most big models realize the description text doesn't match the diagram and proceed carefully, and our task prompt even warns of this (removing that hint significantly hurts models' ability to complete the task). Moonshot's Kimi K3 spots it, reasoning (bold is mine):

Red (untruncated guesses): [2]49°08', 339°04', [2]29°44'?? hmm ‘29 DE?’ probably ‘29 DEGREES 44 MINUTES’ full. Or ‘229’? Then 80°24' or [2]80°24', 10°54' or [x]10, 31°13' or [x]31, ??°59' with degrees fully missing, 154°33'.

Candidate from brute force mask (1,0,1,1,1,1,1,0): 249.13, 339.07, 29.74, 80.40, 350.90, 31.22, 68.98, 154.55.

Red guesses: 249°08' ✓ (‘24 DEGREES’ = ‘249’ truncated!), 339 ✓, 29°44' ✓, 80°24' ✓, leg5: red ‘10 DEGREES 54’ vs mask B 350°54'...

OpenAI 5.6 Sol's hidden reasoning summary doesn't ultimately connect the typo with its final .dxf:

There's an inconsistency between bullet 69 and transcript 24 — maybe a typo from OCR?

I need to correct a measurement for ‘69’ based on some truncation notes. The angle ‘24 degrees 08’ seems to relate to the same point.

Claude Opus 5 one-shots a GeoParquet reader in a fluke

In 30 of the US states (mostly the west, excluding the 13 original colonies, Texas, Louisiana etc. with more complicated land grant systems), surveyed land is based on the Public Land Survey System, shorthand PLSS. Many boundaries establish a point of commencement by referring to a particular corner of some PLSS section.

For one task, I'd provided a state PLSS GeoParquet and instructed the model to treat it as monuments with coordinates.

But while I had written GDAL into the Dockerfile, I omitted the Apache Arrow library which lets you read/write GeoParquet files. Oops.

Most models, like Grok 4.6, complained:

Need to figure out how to read parquet without pyarrow...

I need to parse the parquet file myself since GDAL doesn't have parquet support. Let me look at the parquet format and parse it with Python.

Before failing. But Claude Opus 5 one-shot a Parquet reader in Python before completing the task with an average distance of 2.1 mm (within a typical 1:10,000 accuracy standard common in the industry). Fantastically impressive.

Gemini 3.7 consistently fails while 3.6 succeeds

One of the more disappointing results of this benchmark was yet another provider error preventing us from properly evaluating Gemini 3.7 Flash. Through OpenRouter, hitting both Vertex and AI Studio, we consistently got MALFORMED_FUNCTION_CALL while all other Chat Completions-compatible endpoints succeeded, including 3.6 Flash.

Once this bug is fixed upstream, we'll update the results of PlatBench with 3.7 Flash, which has shown lots of promise in visual reasoning.

Releasing the tasks

We are not releasing these tasks on the public internet to prevent benchmark contamination.

If you are a researcher looking to improve performance on professional tasks relevant to architecture, engineering, and construction (AEC), email me at brendan@buntinglabs.com.

PlatBench score table