Spatial-IQ#

Spatial-IQ: Deconstructing Spatial Intelligence
via Hierarchical Capability Tests

A diagnostic framework that decomposes 3D object counting into nine hierarchical spatial perception and cognition sub-tasks.

Patrick Rim^1,2 Tom Long¹ Ekta Prashnani¹ Ruth Rosenholtz¹ Ben Boudaoud¹ Peter Xenopoulos¹ Alex Wong² Joohwan Kim¹ Jae-Hyun Jung¹

¹NVIDIA Research ²Yale University

📄Paper (arXiv — coming soon) 💾Dataset </>Code

_images/demo.gif — **DEMO:** One scene, two reasoners. The human (top) walks through each sub-task correctly and the components compose into the total. Gemini 3 Pro (bottom) hits *visible count* but its underlying hierarchy is broken and the intermediates don’t compose.#

_images/teaser.png — **Figure 1.** Humans count stacked 3D objects by hierarchically decomposing the structure into columns, layers, and visible vs. hidden blocks (left). Current MLLMs miscount; fine-tuning on Spatial-IQ with chain-of-thought supervision over our proposed hierarchy teaches them to do the same (right).#

Motivation#

In Piaget & Inhelder (1956) and the subsequent human developmental-psychology tradition, human spatial cognition is not a single monolithic capability. It develops hierarchically: simpler perceptual operations — object individuation, perceptual grouping, parsing scenes into columns and layers — accumulate into more complex ones (visible counting, hidden-count inference from physical support), which in turn compose into target tasks like 3D object counting. Crucially, the relation runs both ways: solving the higher target task presupposes mastery of its prerequisites, and mastery of the prerequisites enables the higher task. Headline accuracy and sub-capability accuracy are coupled in humans because they are facets of the same compositional process.

Does the same hierarchy hold for current MLLM spatial reasoning? Do models that succeed on Object Counting also succeed on the prerequisite sub-capabilities? Do strong sub-capability scores translate to strong target-task scores? Or are hierarchy adherence and target-task performance independent properties in current models — such that two models with the same headline accuracy can be doing fundamentally different things underneath?

Existing benchmarks cannot answer this. They evaluate models as black boxes, scoring only the target task. When a model fails, it is unclear whether the failure is perceptual (recognizing object boundaries) or cognitive (reasoning about occlusion to infer hidden geometry). To answer the question, we needed a benchmark that opens the box.

TL;DR. In human developmental psychology, spatial cognition is hierarchical — simpler perceptual sub-capabilities compose into complex ones, and the relation runs both ways. To ask whether current MLLM spatial reasoning shares this structure, we built Spatial-IQ: a hierarchical benchmark of ~80,000 procedurally generated Isaac Sim scenes, decomposing 3D object counting into 9 sub-tasks + 2 target tasks, evaluated across 3 response modalities (free-response text, image-option MCQ, image editing) against a human baseline collected in both text and MCQ. Comparing 8 frontier text models and 3 image-editing models, we find that top MLLMs frequently hit the target task without preserving the underlying hierarchy — and that two models at the same headline score can have very different mechanisms. Applying the same hierarchy as a training signal (CoT supervision over the decomposition + RL with verifiable rewards) lifts a 32B Qwen2.5-VL from 2.9% → 62.6% on Object Counting while preserving the full hierarchical chain.

Abstract#

Multimodal large language models (MLLMs) excel at visual interpretation but fail on spatial reasoning tasks that humans solve reliably. Existing benchmarks evaluate these models as black boxes, limiting their ability to identify the underlying causes of lower performance: when a model fails a spatial reasoning task, it remains difficult to ascertain whether the hurdle is perceptual (recognizing object boundaries) or cognitive (reasoning about occlusion to infer hidden geometry).

We introduce Spatial-IQ, a hierarchical diagnostic framework that decomposes object counting in stacked 3D structures into 9 perceptual and cognitive sub-tasks organized by the developmental stages of human spatial cognition, with mental rotation as an additional target probe. Using NVIDIA Isaac Sim, we procedurally generated a diverse dataset of roughly 80,000 stacked 3D structures with per-task ground truth. We evaluate models across three output formats — free-response text, multiple-choice images, and image editing — alongside a human baseline.

The Spatial-IQ framework shows that top-performing models often succeed at the target task (object counting) without succeeding on the lower-level sub-tasks intended to support it, and that models differ in how much of these hierarchical chains they preserve, often revealing shortcut behavior that raw target-task accuracy alone would obscure. Finally, we demonstrate that training models with chain-of-thought (CoT) supervision over our hierarchical sub-tasks, combined with reinforcement learning with verifiable rewards, significantly improves both spatial consistency across sub-tasks and target-task accuracy.

MLLMs pass the target without the prerequisites

Top-performing MLLMs frequently succeed at Object Counting without succeeding at the sub-tasks the count ostensibly requires. The hierarchy that couples target and prerequisites in humans is largely broken in current models — the gap is hierarchical, not just numeric.

Same headline score, different mechanism

Qwen and Gemini reach comparable headline Object Counting accuracy, but Qwen approximates the human Summation Mechanism (88.9% both-correct, 64.2% exact match) while Gemini reaches similar accuracy through a much weaker mechanism (48.0% / 18.6%). Hierarchy adherence and target accuracy are partially independent.

Humans win by composition

Humans reach 82.1% on Object Counting; the best MLLM reaches 17.7%. Human accuracy is concentrated in the both-component-correct cell (90.6%) with a 69.1% exact-match Summation Mechanism — the empirical signature of composition that current models miss.

The hierarchy is also a training signal

Chain-of-thought supervision over the decomposition + RL with verifiable rewards lifts a 32B Qwen2.5-VL from 2.9% → 62.6% on Object Counting while preserving the full hierarchical chain.

The Spatial-IQ Framework#

Spatial-IQ treats spatial intelligence as a hierarchy of dependencies. Following the classical human developmental psychology of spatial cognition (Piaget & Inhelder, 1956), we view spatial competence as developing through a progression of more elementary sub-capabilities that compose into higher target tasks: column, layer, and cluster understanding for structural reasoning, and support-object and column identification for the gravitational reasoning needed to infer hidden supporting blocks. Successfully solving the higher target task therefore presupposes mastery of these lower-level capabilities — in humans, headline performance and sub-capability performance are coupled because they are facets of the same compositional process.

Within this framework, we adopt a target task drawn from standardized cognitive assessment: the Block Counting subtest of the Kaufman Assessment Battery for Children (KABC), in which the responder reports the total number of objects in a stacked 3D structure, including those occluded but logically required for physical support. The Spatial-IQ hierarchy is the explicit decomposition this target task recruits — visible counting, hidden inference, and the support reasoning that links them.

Spatial-IQ therefore evaluates models on both the target task and its sub-capabilities, in order to test (i) whether models exhibit a human-like hierarchical structure in spatial understanding, and (ii) whether such hierarchical competence translates to better performance on the target task.

_images/tasks.png — **Figure 2.** [Left] The nine sub-tasks (S1–S9) and two target tasks (T1, T2) of Spatial-IQ. S1–S4 probe primitive perceptual grouping (object individuation, clustering, columns, layers). S5 counts visible objects. S6–S8 test structural reasoning (top layer, direct support, support columns). S9 counts hidden objects. T1 is the total *Object Count*; T2 is *Mental Rotation*. [Right] An example scene with per-task ground-truth masks for the image-editing format.#

_images/chains.png — **Figure 3.** We pre-specify four sub-task relations for hierarchy-dependency analysis: the **Internal Referential Chain** (top layer → direct support → support column), the **Visible Support Hierarchy**, the **Hidden Support Hierarchy**, and the **Summation Mechanism** (visible count + hidden count → total).#

Dataset. We procedurally generate scenes in NVIDIA Isaac Sim 5.1, each consisting of a foreground structure of multiple identical objects (textured cubes, factory boxes, soup cans, or potted meat) constrained to a 4×4×4 voxel grid. Hidden objects are identified through a pixel-level depth-buffer occlusion test that retains an object only if it is camera-visible or required to support a retained object above it — guaranteeing every hidden object in the scene is logically required by physical support and never merely concealed behind a visible neighbor at an unrecoverable depth. The full dataset comprises ~80,000 scenes; our primary benchmark uses a 3,000-sample evaluation set, with a disjoint 68,000-sample set reserved for training.

Validating the Benchmark#

We evaluate frontier proprietary text-output models (Gemini 3 Pro, GPT 5.4, Claude Opus 4.6), open-weight models (Qwen3.5-27B, Kimi K2.5, GLM 4.6), the small-model anchors Qwen3.5-3B and VLA-0, and three image-editing models (Gemini 3.1 Flash Image, Qwen-Image-Edit, HunyuanImage-Instruct).

_images/fig_c1.png — **Figure 4.** Per-task accuracy in the text free-response modality, ordered according to our task taxonomy. Humans (black) perform well above all current models on *Object Counting*, while model performance varies by an order of magnitude. Models are shown in order of *Object Counting* performance.#

The target task is solvable by humans and meaningfully ranks models. Humans reach 82.1% on Object Counting, demonstrating that the task is solvable yet not at ceiling. The performance spread distinguishes humans from models without saturating: humans sit well above the best model (Qwen, 17.7%), which in turn outperforms the worst model by an order of magnitude (Qwen2.5-3B, 2.1%). All consecutive differences between adjacent models are statistically significant under paired McNemar tests with multiple-comparisons correction.

_images/wrong_answer_mcp.png — **Figure 5.** Wrong-answer preferences in the five-choice MCQ condition. Each column denotes a wrong-answer type; each cell reports the raw percentage of responses assigned to that type. Only cells significant under a one-sided binomial test against the 20% chance baseline are shown. Models exhibit task-dependent wrong-answer preferences — *phantom block above top*, *merge two columns*, *drop visible blocks* — whereas humans show no significant preference.#

_images/fig_c2.png — **Figure 6.** *Object Counting* accuracy as a function of five difficulty controls in the text modality. Total objects, hidden objects, layers, columns, and fill ratio all affect difficulty in the expected direction (more is harder, except fill ratio, where denser is easier), for both humans and models.#

Decomposing Model Performance#

Three columns of the table below report raw task accuracies (Obj. Count, Visible, Hidden). The remaining four columns capture pre-specified hierarchy relations: the Internal Referential Chain asks whether models preserve object reference along Top Layer → Direct Support → Support Column; the Visible and Hidden Support Hierarchies ask whether models preserve higher-level support capability within each branch; the Summation Mechanism asks whether the model integrates correct visible and hidden counts into a correct total. The latter four columns are conditional accuracies of the lower-level task given correctness on the designated higher-level tasks.

Text modality#

Table 1. Text modality evaluative summary (% accuracy). Per-column model rankings (humans excluded): first, second, third.

Responder	Obj. Count	Int. Ref. Chain	Visible	Vis. Support	Hidden	Hid. Support	Sum. Mech.
Human	82.1	90.1	76.8	84.5	79.9	78.5	69.1
Qwen	17.7	50.7	19.2	78.2	49.2	47.9	64.2
Gemini	14.8	64.5	17.7	56.7	43.6	34.6	18.6
Claude	14.1	52.4	15.7	48.5	51.4	32.5	43.7
Kimi	13.6	47.2	11.5	52.9	33.1	24.5	25.0
GPT	8.3	54.3	8.7	56.2	47.4	28.4	32.8
VLA-0	4.2	16.7	1.2	0.0	19.0	7.4	19.9
GLM	3.5	39.9	2.9	32.8	16.4	9.5	8.2
Qwen3B	2.1	17.1	0.2	0.0	19.0	6.4	24.7

_images/fig_e1a.png — **Figure 7.** *Object Counting* accuracy, conditioned on correct performance on *Visible Object Count* (V) and *Hidden Object Count* (H), text free-response modality. Rows and columns mark correct (✓) or incorrect (×) performance on the corresponding sub-task; cell color and label indicate conditional *Object Counting* performance over n scenes.#

Three observations emerge:

The human baseline shows the empirical signature of the Summation Mechanism. Object Counting accuracy is concentrated in the both-correct cell (90.6%) and decays monotonically as either component count is missed, with a Summation Mechanism exact-match rate of 69.1%.
Model recovery of the Summation Mechanism is uneven and partially independent of headline accuracy. Qwen most closely approximates the human signature (88.9% in both-correct, 64.2% exact match). Gemini reaches comparable headline accuracy through a markedly weaker mechanism (48.0% / 18.6%). Object Counting accuracy and Summation Mechanism strength are not the same property.
The pre-specified hierarchy is largely preserved. Model rows preserve the qualitative ordering observed in humans: the Internal Referential Chain is consistently cleaner than the Visible and Hidden Support Hierarchies. VLA-0 outperforms its Qwen3B backbone across the support relations despite being trained only on robot trajectories — hinting that action-grounded training transfers positively to spatial reasoning.

Multiple-choice modality#

Under five-choice MCQ, every model collapses to chance on Object Counting while humans hold 86.2% accuracy. Reducing to four and three choices partially restores the expected Summation Mechanism, but recovery is uneven across models.

Table 2. MCQ modality evaluative summary (chance-adjusted % accuracy, adj_acc = (acc−1/k)/(1−1/k)). 3CQ / 4CQ / 5CQ shown side-by-side. Humans measured at 5CQ only.

Responder	Obj. Count			Int. Ref. Chain			Visible			Vis. Support			Hidden			Hid. Support			Sum. Mech.
Responder	3CQ	4CQ	5CQ	3CQ	4CQ	5CQ	3CQ	4CQ	5CQ	3CQ	4CQ	5CQ	3CQ	4CQ	5CQ	3CQ	4CQ	5CQ	3CQ	4CQ	5CQ
Human	—	—	82.7	—	—	97.0	—	—	99.3	—	—	99.3	—	—	89.3	—	—	89.6	—	—	83.6
Claude	23.7	17.8	0.0	40.7	33.3	25.0	44.6	43.6	40.7	44.7	43.8	40.4	8.0	10.7	11.7	5.3	10.6	12.7	26.8	26.1	−2.2
Gemini	14.0	12.8	−0.6	38.0	24.2	21.3	45.9	35.6	32.6	47.3	36.6	37.8	21.9	17.7	14.1	22.9	18.8	17.0	21.4	20.0	−2.9
GPT	7.9	5.9	0.9	36.7	34.6	35.4	42.1	24.3	24.0	42.0	25.0	23.6	15.0	10.4	12.6	13.9	11.1	12.3	12.9	16.9	2.2
Qwen	20.5	14.8	0.2	32.0	21.6	10.2	36.5	17.3	8.0	34.4	16.7	7.3	−20.9	−12.6	−2.0	−20.6	−13.1	−1.6	7.4	5.7	1.0

Image-editing modality#

In the image-editing format, the model directly edits the reference image to color the objects specified by the sub-task (e.g., highlighting the top layer or the visible objects). The qualitative pattern from text and MCQ replicates: the Internal Referential Chain remains the most format-stable backbone, and target-task success and local hierarchy adherence are partially dissociable. Qwen-Image-Edit gives the clearest example — strongest Internal Referential Chain integrity in the modality (0.676), but raw Visible Object Count of only 3.3%.

Table 3. Image-editing modality evaluative summary (% accuracy). Columns parallel Table 1.

Responder	Obj. Count	Int. Ref. Chain	Visible	Vis. Support	Hidden	Hid. Support	Sum. Mech.
Gemini Flash Image	14.7	53.7	36.1	37.0	46.5	36.5	29.5
Qwen-Image-Edit	12.0	67.6	3.3	58.7	38.7	11.2	23.8
HunyuanImage-Instruct	5.9	50.1	10.8	32.2	30.5	19.0	9.4

Training on Spatial-IQ Sub-Tasks#

The structure of the benchmark suggests a natural training signal: a model trained to walk through the sub-task decomposition before committing to a final count should learn the prerequisite competencies. We test this on Qwen2.5-VL-7B-Instruct and Qwen2.5-VL-32B-Instruct under three conditions:

SFT-plain — supervised fine-tuning on the integer total alone, no decomposition.
SFT-CoT — SFT on a chain-of-thought target listing sub-task answers in the order required for additive composition.
DAPO-tight — brief SFT warmup on 10% of the training data, followed by RL with verifiable rewards (GRPO under the DAPO recipe) on the remaining 90%.

Table 4. Trained Qwen2.5-VL checkpoints, text modality (% accuracy). Values in parentheses are absolute point gains over the corresponding zero-shot backbone. Per-column rankings within each scale group (humans excluded; zero-shot baselines included): first, second.

Responder	Obj. Count	Int. Ref. Chain	Visible	Vis. Support	Hidden	Hid. Support	Sum. Mech.
Human	82.1	90.1	76.8	84.5	79.9	78.5	69.1
DAPO-tight 7B	50.7 (+46.4)	93.7 (+45.6)	51.7 (+48.3)	66.9 (+65.6)	76.0 (+30.7)	85.5 (+28.9)	90.0 (+45.3)
SFT-CoT 7B	40.8 (+36.5)	93.7 (+45.6)	41.9 (+38.5)	60.5 (+59.2)	76.7 (+31.4)	85.2 (+28.6)	89.6 (+44.9)
SFT-plain 7B	39.7 (+35.4)	0.0 (−48.1)	29.2 (+25.8)	80.3 (+79.0)	0.0 (−45.3)	0.0 (−56.6)	0.0 (−44.7)
Qwen2.5-VL-7B (zero-shot)	4.3	48.1	3.4	1.3	45.3	56.6	44.7
DAPO-tight 32B	62.6 (+59.7)	95.0 (+53.9)	64.5 (+62.6)	73.7 (+62.3)	78.2 (+27.8)	86.1 (+5.8)	93.3 (+42.3)
SFT-CoT 32B	46.7 (+43.8)	95.5 (+54.4)	48.9 (+47.0)	61.3 (+49.9)	79.2 (+28.8)	85.7 (+5.4)	92.9 (+41.9)
SFT-plain 32B	40.1 (+37.2)	0.0 (−41.1)	30.9 (+29.0)	85.0 (+73.6)	0.0 (−50.4)	0.0 (−80.3)	0.0 (−51.0)
Qwen2.5-VL-32B (zero-shot)	2.9	41.1	1.9	11.4	50.4	80.3	51.0

Sub-task decomposition is a real training signal, and the gain compounds with scale. At 7B, SFT-CoT and SFT-plain are nearly tied on Object Counting (1.1-pt gap), but at 32B the decomposition produces a 6.6-pt gap in its favor. More importantly, the two diverge sharply on the hierarchy columns: SFT-plain collapses to zero on Internal Referential Chain, Hidden Support Hierarchy, and Summation Mechanism — indicating it has learned a route to the right total that bypasses the decomposition entirely. SFT-CoT preserves all four hierarchy relations.

RL with verifiable rewards adds substantial gain on top of SFT-CoT. DAPO-tight beats SFT-CoT by 9.9 absolute points at 7B and 15.9 at 32B, with the 32B DAPO-tight model reaching 62.6% Object Counting accuracy where the zero-shot backbone achieved 2.9%. The reward signal is on the integer total only, so these gains come from sharpening the policy onto chain-of-thought trajectories that lead to correct totals — not from new sub-task supervision.

Mental Rotation as a Complementary Probe#

Our second target task addresses a distinct facet of spatial cognition: viewpoint transformation. Empirically, the proposed hierarchy aligns much more strongly with Object Counting than with Mental Rotation, while the two rotation tasks ($90^\circ$ and $180^\circ$) align primarily with each other. Humans perform worse on Mental Rotation (46.4% / 45.8%) than on Object Counting, which limits the headroom available for distinguishing models. The Spatial-IQ hierarchy is therefore a faithful decomposition of Object Counting but not of Mental Rotation — they probe largely separable competencies.

Putting It Together: Hierarchy as Both Diagnostic and Training Signal#

The motivation, benchmark design, evaluation results, and training results above converge on a single observation: for the compositional spatial tasks Spatial-IQ probes, what distinguishes humans from current MLLMs is not better counting but the ability to compose the answer from prerequisite sub-capabilities — and the same hierarchy that surfaces this gap is also the training signal that closes it.

Humans show the empirical signature that the hierarchical view predicts. Object Counting accuracy concentrates in the both-correct cell (90.6%) and decays monotonically as either component count is missed; the Summation Mechanism exact-match rate is 69.1%. Humans reach the right total because visible count + hidden count → total actually runs, exactly as the developmental hierarchy says it should.
Current MLLMs do not automatically inherit this structure — target-task performance and hierarchy adherence dissociate. Qwen and Gemini sit within a few points of each other on raw Object Counting (17.7% vs 14.8%), yet their Summation Mechanism exact-match rates differ by more than 3× (64.2% vs 18.6%). A model can hit the integer by shortcut.
Forcing the hierarchy during training installs the missing structure; bypassing it caps the gain. At 32B, SFT-plain reaches 40.1% but collapses to 0% on the Internal Referential Chain / Hidden Support / Summation Mechanism. SFT-CoT preserves all four hierarchy relations and reaches 46.7%; DAPO-tight reaches 62.6% with the chain intact (Internal Referential Chain 95.0%, Summation Mechanism 93.3%). The reward never names the sub-tasks — RL sharpens the policy onto chain-of-thought trajectories that lead to correct totals.

In humans, headline accuracy implies hierarchy adherence; in current MLLMs by default, it does not. The diagnostic dissociation (#2) and the causal training result (#3) tell the same story: teach a model the composition and the final integer follows; let it skip the composition and target-task accuracy caps at the level of a shortcut.

Discussion#

Spatial-IQ reframes spatial-intelligence evaluation as a hierarchical decomposition and shows that the same hierarchy serves both as a diagnostic instrument and as a training signal. Top models often succeed on Object Counting while failing the underlying sub-tasks. Chain-of-thought supervision over the hierarchy, combined with verifiable-reward reinforcement learning, closes much of the gap to the human ceiling while preserving the intended chain. The finding is task-specific: the Spatial-IQ hierarchy faithfully decomposes Object Counting but not Mental Rotation, suggesting that compositional spatial tasks and transformational spatial tasks recruit separable competencies — and that future spatial-intelligence benchmarks should pair each target task with its own task-appropriate hierarchy rather than treating “spatial intelligence” as a single scalar. Future work includes extending this framework to other classical spatial-intelligence tasks and a broader investigation of whether action-grounded training in vision-language-action models systematically benefits spatial reasoning.

BibTeX#

@misc{rim2026spatialiq,
  title         = {Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests},
  author        = {Rim, Patrick and Long, Tom and Prashnani, Ekta and Rosenholtz, Ruth and
                   Boudaoud, Ben and Xenopoulos, Peter and Wong, Alex and Kim, Joohwan and
                   Jung, Jae-Hyun},
  year          = {2026},
  note          = {Preprint. Under review.},
}