We introduce SurveyorBench, an eval focusing on multimodal tasks CAD Technicians solve during both construction and property transactions in the United States.
These tasks involve multimodal reasoning through surveyors' maps, legal descriptions, and easements to create an exact map output as a CAD file.
SurveyorBench builds on our years of experience building AI mapping products for customers including the U.S. Geological Survey and the State of California.
Civil engineers don't use AI
Surveying is an essential step in nearly every civil engineering project, forming the backbone of how CAD designs are mapped to the real world.
150k+ civil engineers manage $1T in infrastructure spend, yet only 0.8% of civil engineering work is done with AI.
SurveyorBench shows that LLMs fail at the real-world multimodal reasoning tasks civil engineers need to accomplish their daily work. From our experience working with civil engineering firms, there’s a desire to use AI to accelerate work, but low-quality outputs prevent reliance on it.
What is a SurveyorBench task?
As an example of one of our tasks, the City of Ventura gave a utility the right to bury electric lines underground in this park corner:
SurveyorBench task #78 asks for a polygon as a .dxf (Autodesk's CAD file) representing where the buried electric lines
can be legally installed and maintained to 1-meter accuracy.
Accomplishing the task requires taking the documents, identifying relevant information across them, resolving conflicting evidence, correctly connecting the information in an exact order, and generating a valid CAD output.
This GPT-5.6 Sol rollout investigates 7 documents in our medium difficulty environment, reasoning through the diagrams and tables in 48 steps:
The model then uses Python tools to generate the .dxf before submitting it.
In this rollout, GPT-5.6 Sol got a 100% on our medium difficulty level, judged by its distance to the survey-grade output; scroll down to see its failure on the same task in a different rollout.
SurveyorBench tasks vary in difficulty and complexity
SurveyorBench mixes easy, medium, and hard tasks. Truly representative tasks, involving fetching relevant corner records and plat maps from the assessor/recorder's office, reading through the entire documents for the relevant parts, and then locating the exactly-relevant info, would be harder than "hard".
Experts solve tasks in SurveyorBench-Hard in 2-3 hours, using desktop software like Autodesk Civil 3D or Carlson. The strongest models we evaluated have pass rates between 0-4% on our hardest tasks because they often require identifying human errors in the documents or aligning basis of bearings between surveys completed decades ago.
Opus 5 leads 5.6 Sol, Grok 4.6, and Kimi K3
Gaps in SurveyorBench scores are likely attributable to differences in visual reasoning over engineering diagrams and openmindedness in the face of conflicting evidence.
Scores are measured as pass@1 with a 30 minute timeout. Scores include partial rewards when output is below acceptable accuracy. All tasks are graded against survey-grade data. pass@k analysis is available
at the bottom.
Models continue to make amateur mistakes
When given real-world documents,
today's models can't reliably identify the right information, reason between documents, and generate CAD files. Instead,
in our evaluation,
they failed to determine the relevant text in a sea of markings,
filled their context window with poorly executed GDAL calls (Grok 4.6 fell trap to a ogrinfo foot gun), and submitted work with
preventable errors (many of these tasks are verifiable).
GPT-5.6 Sol travels the wrong direction along surveyed lines
When recreating a particular parcel via COGO,
a technician might decide to reverse a survey call to arrive at the right destination. Conveniently when using quadrant bearings
you can simply flip N 34° 52' W to S 34° 52' E to go along the back bearing.
This is second nature to anyone with plat experience, but GPT-5.6 Sol fails one task because it refuses to flip the bearing:
I’m looking at the intersection and heading east along Saticoy CL, noting its direction as N34W. The map’s north arrow seems a bit odd… if Saticoy runs northwest to the right, I think I’m on the right track here.
The correct point of beginning for the easement is south along the orange pictured highway, and adjusting for the north arrow
would make anyone realize the correct traversal direction is southeast: you must flip N34W to S34E.
Note: this rollout failed while another rollout earlier succeeded on the same task #78
Opus 5 doesn't identify human errors in source documents
Real-world documents have human mistakes: this survey from 2001 reports the metes and bounds
of L38 as both N 16° 51' 00" E and N 16° 51' 00" W. To correctly determine the easement boundary and write a .dxf CAD file,
the agent must determine which survey call is correct.
Opus 5 reads both bearings in its reasoning trace without backtracking and spotting the mistake:
L38 runs N16°51'00"E for 117.74 feet to P4...
the westerly line keeps the N16°51'00"W bearing...
Then submits a .dxf having incorrectly chosen N 16° 51' 00" E, in addition to 2 other fatal errors
that make the digitized boundary inaccurate for determining the utility's legal rights.
Gemini 3.7 Flash loops on image crop tools
One standout agent instance was a 23.6M token, $5.44 run where Gemini spent 92 tool calls over 15 minutes rotating, cropping, and OCR'ing the maps before failing:
When a typical rollout costs $0.10 to $0.50, this 10x cost was surprising and rollouts like it are what push Gemini 3.7 Flash to be the cost-adjusted worst performer in our evaluation.
SurveyorBench tasks are agentic image understanding tasks, so models where this is out of distribution require the most attempts.
Claude Opus 5 one-shots a GeoParquet reader in a fluke
In 30 of the US states (mostly the west, excluding the 13 original colonies, Texas, Louisiana etc. with more complicated land grant systems), surveyed land is based on the Public Land Survey System, shorthand PLSS. Many boundaries establish a point of commencement by referring to a particular corner of some PLSS section.
For one task, I'd provided a state PLSS GeoParquet and instructed the model to treat it as monuments with coordinates.
But while I had written GDAL into the Dockerfile, in my first draft I omitted the Apache Arrow library
which lets you read/write GeoParquet files.
Most models, like Grok 4.6, complained before failing:
Need to figure out how to read parquet without pyarrow...
I need to parse the parquet file myself since GDAL doesn't have parquet support. Let me look at the parquet format and parse it with Python.
But Claude Opus 5 one-shot a Parquet reader in Python before completing the task with an average distance of 2.1 mm (within a typical 1:10,000 accuracy standard common in the industry). Fantastically impressive.
GPT-5.6 Sol has exceptional agentic efficiency
One standout result from SurveyorBench is the cost efficiency of GPT-5.6 Sol. Most benchmarks allow formation of a Pareto frontier between cost and accuracy, which is trivially formed with GPT-5.6 Sol and Opus 5 here.
But a more interesting axis is understanding the cost of failed attempts. Conditional on task failure, GPT-5.6 Sol is 4.1x cheaper than the average evaluated model, mostly because it uses 30% fewer steps per failed task:
Multimodal reasoning tasks like SurveyorBench separate models strongly along this axis: Gemini 3.7 Flash costs nearly 10x what GPT-5.6 Sol did, mostly because Gemini filled its context window with duplicated images in crop loops:
Generating more tasks like SurveyorBench
If you are a researcher looking to improve performance on long horizon CAD tasks requiring visual understanding, please email me at brendan@buntinglabs.com.
