We introduce SurveyorBench, an eval focusing on multimodal tasks CAD Technicians solve during both construction and property transactions in the United States.
These tasks involve multimodal reasoning through surveyors' maps, legal descriptions, and easements to create an accurate map output as a CAD file.
SurveyorBench builds on our years of experience building AI mapping products for customers including the U.S. Geological Survey and the State of California.
Civil engineers don't use AI
Surveying is an essential step in nearly every civil engineering project, forming the backbone of how CAD designs are mapped to the real world.
150k+ civil engineers manage $1T in infrastructure spend, yet only 0.8% of civil engineering work is done with AI.
SurveyorBench shows that LLMs fail at the real-world multimodal reasoning tasks civil engineers need to accomplish their daily work. From our experience working with civil engineering firms, there’s a desire to use AI to accelerate work, but low-quality outputs prevent reliance on it.
What is a SurveyorBench task?
As an example of one of our tasks, the City of Ventura gave a utility the right to bury electric lines underground in this park corner:
SurveyorBench task #78 asks for a polygon as a .dxf (CAD file format) representing where the buried electric lines
can be legally installed and maintained to 1-meter accuracy.
Accomplishing the task requires taking the documents, identifying relevant information across them, resolving conflicting evidence, correctly connecting the information in an exact order, and generating a valid CAD output.
This GPT-5.6 Sol rollout investigates 7 documents in our medium difficulty environment, reasoning through the diagrams and tables in 48 steps:
The model then uses Python tools to generate the .dxf before submitting it.
In this rollout, GPT-5.6 Sol scored 100% on this medium-difficulty task, as judged by its IoU to the expert boundary. Scroll down to see its failure on the same task in a different rollout.
SurveyorBench tasks vary in difficulty and complexity
SurveyorBench mixes easy, medium, and hard tasks. Truly representative tasks, involving fetching relevant corner records and plat maps from the assessor/recorder's office, reading through the entire documents for the relevant parts, and then locating the exactly-relevant info, would be harder than "hard".
Experts solve tasks in SurveyorBench-Hard in 2-3 hours, using desktop software like Autodesk Civil 3D or Carlson. The strongest models we evaluated have pass rates between 0% and 4% on our hardest tasks because they often require identifying human errors in the documents or aligning basis of bearings between surveys completed decades ago.
Opus 5 leads 5.6 Sol, Grok 4.6, and Gemini 3.7 Flash
Gaps in SurveyorBench scores are likely attributable to differences in visual reasoning over engineering diagrams and open-mindedness in the face of conflicting evidence.
Scores include partial rewards when output is below acceptable accuracy. Tasks use a 30 minute timeout. All tasks are graded against expert-generated, survey-grade data. Model is reported with best performing effort level, e.g. (H) for high.
Gemini 3.7 Flash's low cost per token (75% discount until New Years) compensates for its proclivity to fill its context window with image crops, putting it on the SurveyorBench Pareto frontier, while GPT-5.6 Sol leads in token efficiency.
Models continue to make amateur mistakes
When given real-world documents, today's models can't reliably identify the right information, reason between documents, and generate CAD files. Instead, in our evaluation, they failed to determine the relevant text in a sea of markings, filled their context window with unnecessary tool calls, and submitted work with preventable errors (many of these tasks are verifiable).
GPT-5.6 Sol travels the wrong direction along surveyed lines
When recreating a particular parcel via COGO,
a technician might decide to reverse a survey call to arrive at the right destination. Conveniently when using quadrant bearings
you can simply flip N 34° 52' W to S 34° 52' E to go along the back bearing.
This is second nature to anyone with plat experience, but GPT-5.6 Sol fails one task because it refuses to flip the bearing:
I’m looking at the intersection and heading east along Saticoy CL, noting its direction as N34W. The map’s north arrow seems a bit odd… if Saticoy runs northwest to the right, I think I’m on the right track here.
The correct point of beginning for the easement is south along the orange pictured highway, and adjusting for the north arrow
would make anyone realize the correct traversal direction is southeast: you must flip N34W to S34E.
Note: this rollout failed while another rollout earlier succeeded on the same task #78
Opus 5 doesn't identify human errors in source documents
Real-world documents have human mistakes: this survey from 2001 reports the metes and bounds
of L38 as both N 16° 51' 00" E and N 16° 51' 00" W. To correctly determine the easement boundary and write a .dxf CAD file,
the agent must determine which survey call is correct.
Opus 5 reads both bearings in its reasoning trace without backtracking and spotting the mistake:
L38 runs N16°51'00"E for 117.74 feet to P4...
the westerly line keeps the N16°51'00"W bearing...
Then submits a .dxf having incorrectly chosen N 16° 51' 00" E, in addition to 2 other fatal errors
that make the digitized boundary inaccurate for determining the utility's legal rights.
Grok 4.6 falls prey to a ogrinfo footgun
Using tools efficiently is key to agentic performance and polluting the context window leads to unnecessary costs. In one task, we provided the agent with the PLSS data as a GeoPackage and provided it GDAL/OGR CLI tools.
Grok 4.6 in a single tool call used ogrinfo -al ... to list the coordinates of every geometry in
the file, instantly feeding ~131k tokens into its input context, filling up 26% of its context window
with random digits:
2328173.21244843,6686107.65980683
2326861.56936161,6686104.16450327
2325549.92704435,6686100.66818433
2324238.28587821,6686097.45439223
2322926.2833035,6684773.74508507
...
That step alone cost $0.26.
Grok 4.6 medium and Opus 5 high reasoning use fewer tool calls
One of the more interesting results is that Grok 4.6 medium underperforms the expected improvement in
scores you see as you crank reasoning higher. In our n=1 rollout, both Grok 4.6 and Opus 5 (high) models chose to use fewer tool calls
when solving the same task. In Grok's case, likely letting errors slip through due to less double checking.
Multimodal reasoning tasks like SurveyorBench separate models strongly along tool call count: Gemini 3.7 Flash uses 3x the steps as Sol/Kimi/Grok, yet costs one-third of the price due to cheap tokens.
Muse Spark 1.2 and Sonnet 5 are far from the cost frontier
Both Claude Sonnet 5 and Meta's Muse Spark 1.2 were far more expensive than their performance should indicate.
Muse Spark 1.2 was beat by Gemini 3.7 Flash on low reasoning at one fifth the cost per task.
Sonnet 5 had the same cost per task as Opus 5 on low reasoning (the above cost chart shows cost of medium reasoning) yet 65% of the score.
Generating more tasks like SurveyorBench
If you are a researcher looking to improve performance on multimodal reasoning tasks that can accelerate real-world engineering projects, please email me at brendan@buntinglabs.com.
