The Prestige: Benchmarking Cognitive Visual Reasoning using Magic Tricks

Shuo Wen^1,2,* Beixi Du^3, Edwin Meriaux^1,2,4, Junming Shi^1,2, Chloe Si^1,
David Meger^1,2 Doina Precup^1,2 Gregory Dudek^1,2

¹McGill University ²Mila – Quebec AI Institute ³Cornell University ⁴Université Paris-Saclay, CentraleSupélec

Equal contribution, listed alphabetically by last name. ^* Correspondence to swen@utexas.edu.

40th Conference on Neural Information Processing Systems (NeurIPS 2026) · Evaluations & Datasets Track

A magic trick video shown to three VLMs, with excerpts of their responses explaining the surprise. — An example prompt to challenge representative VLMs to explain the elements of surprise in a video sequence. Results highlight current limitations in VLM visual reasoning, including partial event detection and false identification. All models are evaluated using their latest web versions at the time of testing. Responses are excerpted, and images are cropped for brevity.

Abstract

We introduce The Prestige, a video understanding benchmark that probes cognitive visual reasoning through professional magic performances, where deliberate misdirection challenges models to track events and explain unexpected outcomes. The benchmark comprises open-ended questions ranging from temporal and causal understanding to robustness against adversarial questions. Evaluation of leading open-weight and proprietary VLMs reveals a substantial gap between even the strongest model and human performance. We also find systematic discrepancies between human and automated grading and identify recurring model failure modes. Ablations further show that models can select correct multiple-choice answers even when they fail to generate correct explanations independently, motivating further investigations into question formats. Together, these results highlight limitations in both VLM reasoning and its evaluation, informing the development of more reliable video understanding models and benchmarks.

Word cloud of question keywords printed on a fan of four playing cards, with terms such as magic, card, ring, hand, coin, rope, and dice. — Word cloud of the question keywords

The Benchmark

Selected videos are professional, ethical, and high-resolution. They must be entirely comprehensible through visual cues alone, independent of audio or text, and not captioned, and the performances must demonstrate intentional causal coherence and high contextual density.

The benchmark contains 180 videos and 2,177 open-ended questions, drawn from unique routines performed by over 150 magicians across multiple platforms. The questions fall into six categories, each targeting distinct aspects of visual, temporal, and common-sense reasoning in multi-step and long-dependency settings.

Pie chart of question types: Contextual Reasoning 36.2%, Paradoxical Reasoning 34.4%, False Implication 10.7%, Instant Reasoning 9.4%, Temporal Summarization 5.1%, and False Statement 4.2%. — Question type distribution

Type	Abbr.	Share	What it tests
Instant Reasoning	IR	9.4%	Describing the complete, question-relevant visual state at a single moment, using only what is visible in the current frame.
Contextual Reasoning	CR	36.2%	Using memory and relational reasoning to answer questions about a specific moment from earlier video evidence, such as tracking an object through occlusion or remembering which card was signed. The current frame alone is insufficient.
Paradoxical Reasoning	PR	34.4%	Explaining why an outcome is surprising by tracking the earlier setup, reasoning about what should happen under common-sense expectations, and identifying how the observed outcome differs.
Adversarial Questions—False Implication	AQ.FI	10.7%	Recognizing a false assumption implied by the question, checking it against the video, and rejecting the ungrounded framing rather than answering within it.
Adversarial Questions—False Statement	AQ.FS	4.2%	Recognizing an explicit claim that contradicts the video and rejecting it rather than accepting it or inventing supporting details.
Temporal Summarization	TS	5.1%	Producing a compact, high-level summary of a specified time interval that captures the essential details while suppressing both low-level details and overgeneralizations.

The six question types and their share of the 2,177 questions.

Six example panels, one per question type, each with a film strip, a question, and the ground-truth answer. — Examples of the six distinct categories of video reasoning tasks evaluated: Instant Reasoning (IR), Contextual Reasoning (CR), Temporal Summarization (TS), Adversarial Questions—False Implication (AQ.FI) and False Statement (AQ.FS), and Paradoxical Reasoning (PR). Each panel displays a sample question, the ground-truth human answer, and a film strip of the relevant video clip. Images are zoomed in for clarity. Representative frames are only selected for demonstration purposes. In actual evaluations, models receive frames sampled uniformly at 1 FPS over the available temporal interval, up to a maximum duration of four minutes.

Comparison with Existing Benchmarks

Dataset	# videos	# QA	Format	Common-sense	Cognitive Flow	Multi-step	Long-context	Annotation
FunQA	4,365	311,950	QA	✓	△	✗	✗	Manual+LLM
Video-MME-v2	800	3,200	8-way MC	✓	✗	✓	✓	Manual
Causal-VidQA	26,900	107,600	MC	✓	✗	✗	✗	Manual
VRBench	960	8,243	MC&QA	✗	✗	✓	✓	Manual+LLM
Video-Holmes	270	1,837	MC	✓	✗	✓	✗	Manual+LLM
CG-Bench	1,219	12,129	MC&QA	✗	✗	✓	✓	Manual
MMR-V	317	1,257	MC	✓	✗	✓	✓	Manual
LongVideoBench	3,763	6,678	MC	✗	✗	✗	✓	Manual
LVBench	103	1,549	MC	✗	✗	✓	✓	Manual+LLM
Ours	180	2,177	Open QA	✓	✓	✓	✓	Manual

Comparison of different video understanding benchmarks. MC stands for multiple choice questions, QA stands for open Question-Answer questions. Cognitive Flow denotes whether the benchmark centrally evaluates reasoning over videos designed to examine the cognitive visual reasoning of the models. △ indicates partial coverage.

Results

Gemini3.1-Pro performs best overall at 47.08%, but remains 45.71 percentage points below the human baseline (92.79%). No model leads every category. The strongest open-source models, Kimi-K2.6 (39.37% overall) and Qwen3.6 (38.26%), both outperform GPT-5.2 (37.67%).

Model	Size	IR	CR	PR	AQ.FI	AQ.FS	TS	Overall
Open-source models
Qwen3VL-Thinking	8B	40.00	34.69	9.76	24.46	15.22	17.86	23.84
Qwen3VL-Instruct	8B	46.34	39.90	17.25	18.88	20.65	29.46	29.12
InternVideo2.5-Chat	8B	30.24	27.57	3.74	3.86	5.43	8.93	15.20
InternVL3.5	8B	25.37	26.81	4.14	19.31	9.78	12.50	16.63
MiMo-VL-SFT-2508	7B	22.44	22.11	1.87	21.89	15.22	8.04	14.15
MOSS-VL-Instruct	11B	36.59	33.16	12.43	30.47	7.61	28.57	24.76
Qwen3.6	35A3B	51.71	48.16	28.34	31.76	21.74	37.50	38.26
Cambrian-S	7B	24.88	22.36	4.55	20.17	8.70	10.71	15.07
Keye-VL-1.5	8B	30.73	27.83	5.61	23.18	14.13	10.71	18.51
Kimi-K2.6	1.1TA32B	54.15	47.14	31.28	30.47	27.17	40.18	39.37
Proprietary models
GPT-5.2	UKN	53.17	46.12	29.41	30.47	15.22	38.39	37.67
Gemini3.1-Flash-Lite	UKN	55.12	49.94	34.89	41.63	18.48	41.96	42.63
Qwen3.6-Plus	UKN	49.76	51.46	35.16	28.33	16.30	39.29	41.11
Gemini3.1-Pro	UKN	52.68	50.44	43.32	45.06	55.43	35.71	47.08
Human baseline	NA	93.17	93.01	94.39	91.42	83.70	90.18	92.79

Model accuracy by question type across 14 models over the full dataset. All model results are scored by human evaluation. Bold marks the best model per column. IR: Instant Reasoning; CR: Contextual Reasoning; PR: Paradoxical Reasoning; AQ.FI: Adversarial Questions—False Implication; AQ.FS: Adversarial Questions—False Statement; TS: Temporal Summarization.

Human vs. Machine Grading

Machine grades are determined by majority vote among three LLM graders, each evaluating every model response with the corresponding video frames. Human grading yields higher overall scores for every model, with a mean gap of 4.56 percentage points (SD 1.23). TS has the largest mean difference (7.40 points) and CR the smallest (3.39 points).

Model	IR	CR	PR	AQ.FI	AQ.FS	TS	Overall
Open-source models
Qwen3VL-Thinking-8B	7.80	3.81	6.42	3.86	10.87	8.04	5.60
Qwen3VL-Instruct-8B	8.78	5.47	8.96	4.29	9.78	13.39	7.44
InternVideo2.5-Chat-8B	6.83	4.32	2.54	0.86	4.34	7.14	3.72
InternVL3.5-8B	7.81	3.18	2.94	3.86	3.26	10.71	4.00
MiMo-VL-SFT-2508-7B	8.29	2.80	1.07	7.30	8.70	3.58	3.49
MOSS-VL-Instruct-11B	4.39	2.66	5.21	5.58	6.52	16.96	4.92
Qwen3.6-35A3B	6.83	4.96	9.22	−2.57	6.52	11.61	6.20
Cambrian-S-7B	6.83	3.17	3.21	8.15	8.70	6.25	4.46
Keye-VL-1.5-8B	5.85	2.93	2.27	2.58	6.52	2.67	3.08
Kimi-K2.6-1.1TA32B	1.95	1.90	6.95	−3.44	1.08	12.50	3.59
Proprietary models
GPT-5.2	5.37	3.17	1.74	15.45	6.52	−0.90	4.14
Gemini3.1-Flash-Lite	7.80	3.69	5.61	3.86	1.09	6.25	4.78
Qwen3.6-Plus	5.86	3.43	9.09	−0.85	2.17	4.47	5.14
Gemini3.1-Pro	2.44	2.03	5.62	3.86	−1.09	0.89	3.30
AVG SD	6.20 2.08	3.39 1.01	5.06 2.83	3.77 4.76	5.36 3.63	7.40 5.13	4.56 1.23

Signed percentage-point difference between human and machine grading scores, computed as human score minus machine score. The final row reports the mean for each column, with the corresponding standard deviation.

Full machine-graded results

Model	Size	IR	CR	PR	AQ.FI	AQ.FS	TS	Overall
Open-source models
Qwen3VL-Thinking	8B	32.20	30.88	3.34	20.60	4.35	9.82	18.24
Qwen3VL-Instruct	8B	37.56	34.43	8.29	14.59	10.87	16.07	21.68
InternVideo2.5-Chat	8B	23.41	23.25	1.20	3.00	1.09	1.79	11.48
InternVL3.5	8B	17.56	23.63	1.20	15.45	6.52	1.79	12.63
MiMo-VL-SFT-2508	7B	14.15	19.31	0.80	14.59	6.52	4.46	10.66
MOSS-VL-Instruct	11B	32.20	30.50	7.22	24.89	1.09	11.61	19.84
Qwen3.6	35A3B	44.88	43.20	19.12	34.33	15.22	25.89	32.06
Cambrian-S	7B	18.05	19.19	1.34	12.02	0.00	4.46	10.61
Keye-VL-1.5	8B	24.88	24.90	3.34	20.60	7.61	8.04	15.43
Kimi-K2.6	1.1TA32B	52.20	45.24	24.33	33.91	26.09	27.68	35.78
Proprietary models
GPT-5.2	UKN	47.80	42.95	27.67	15.02	8.70	39.29	33.53
Gemini3.1-Flash-Lite	UKN	47.32	46.25	29.28	37.77	17.39	35.71	37.85
Qwen3.6-Plus	UKN	43.90	48.03	26.07	29.18	14.13	34.82	35.97
Gemini3.1-Pro	UKN	50.24	48.41	37.70	41.20	56.52	34.82	43.78

Model accuracy by question type across 14 models over the full dataset. All model results are scored by machine evaluation. The final machine grade is determined by majority voting. A response passes if at least two of the three graders assign a score of 1.

Error Analysis

We randomly sample 100 incorrectly answered questions from each of Gemini3.1-Pro, GPT-5.2, and Kimi-K2.6, and manually assign each sampled error to one of seven error categories.

Error type	Abbr.	Description
Incorrect Visual Identification	IVI	Incorrect recognition of evidence directly visible at the queried timestamp, such as misidentifying objects, misreading card ranks or suits, or miscounting items.
Context Referral Deficiency	CRD	Failure to retrieve information established earlier, such as a previously shown object identity or a hidden object location.
Lack of Cognitive Flow	LCF	Failure to connect the setup, misdirection, and reveal of a routine, or reuse of an earlier surprise to explain a later, distinct one.
Lack of Robustness	LR	Accepting a misleading or false premise in the question instead of rejecting it and grounding the answer in visual evidence.
Incorrect Output Formatting	IOF	Failure to follow the required response format, such as never-ending responses, refusals to answer, or missing required components.
Human Intention Mismatch	HIM	Plausible or partially correct answers that fail to match the intended specificity or reasoning target of the question.
Hallucination	HAL	Unsupported visual or causal claims, including invented objects, nonexistent actions, or incorrect surprises that substantially diverge from the ground truth.

Error categories.

LCF, CRD, and IVI together account for 67%, 66%, and 65% of sampled errors in Gemini3.1-Pro, GPT-5.2, and Kimi-K2.6, respectively, identifying cognitive-flow reasoning, long-context reference, and local visual identification as the dominant failure modes.

Radar chart of the distribution of the seven error types for Gemini3.1-Pro, GPT-5.2, and Kimi-K2.6. — Error type distribution

Bar chart of the question types of the sampled errors for each model. — Question type distribution of sampled errors

Open-ended vs. Multiple-choice

We compare open-ended answering with 8-way and 4-way multiple-choice answering on Qwen3.6-Plus, Kimi-K2.6, and Gemini3.1-Flash-Lite. On 200 randomly sampled questions, mean accuracy rises from 37.5% for open-ended answering to 67.2% for 8-way and 71.5% for 4-way multiple choice. On 50 questions that all three models failed in the open-ended setting, the mean 4-way gain is 73.3 percentage points.

Line plots of accuracy in the open-ended, 8-way, and 4-way settings for three models, with stacked bars of question types. — Ablation study on the effects of multiple-choice questions for (a) 50 failure-selected and (b) 200 randomly sampled questions. Stacked bars show the question-type distributions.

BibTeX

Cartoon of the authors performing the sawing-a-person-in-half trick on a picnic table. — Reviewers requested additional ablations, so one of us volunteered.