VTCBench: Can Vision-Language Models Understand Long Contexts with Vision-Text Compression?

VTCBench is the first comprehensive benchmark specifically designed to evaluate the long-context understanding capabilities of Vision-Language Models (VLMs) within the Vision-Text Compression (VTC) paradigm.

VTC is an emerging framework that converts long texts into dense 2D visual representations (images), achieving token compression ratios of 2-10x compared to standard text tokenization. VTCBench rigorously assesses whether VLMs can actually understand this compressed information or if they are merely performing surface-level OCR.

📈 Main Findings

Performance of 16 VLMs/LLMs on VTCBench-Wild with (hatched) and without (solid) vision-text compression. Scores are averaged over three tasks: retrieval, associative reasoning, and long-term dialogue memory. Gemini-3-Pro leads on both pure-text and VTC, while other top-tier models maintain high text performance but exhibit lower visual-text capability.

Perception ≠ Comprehension: While many VLMs excel at OCR and simple retrieval, their performance collapses on reasoning and memory tasks compared to text-only LLMs.
Length Fragility: VLM performance degrades significantly as the context length increases (e.g., from 1k up to 32k tokens).
Parameter Sensitivity: VTC performance is highly sensitive to font size and the spatial positioning of information.

🚀 Key Features

Three Core Tasks: Retrieval, Reasoning, and Memory

VTC-Retrieval: A visual “Needle-In-A-Haystack” (NIAH) test. Requires locating “needles” (key-value pairs) embedded within a large “haystack” of distractors.
VTC-Reasoning: Tests associative reasoning with minimized literal overlap between query and key, requiring inference of latent associations.
VTC-Memory: Multi-turn conversations testing long-term memory retention.

VTCBench-Wild

A wild-version designed to simulate real-world visual diversity (e.g., varying fonts, backgrounds, and layouts)

Two Evaluation Settings

Predefined VTC Ratio: Predetermines the compression ratio (e.g., $r_\text{VTC}=2.0$) to compare model intelligence at a standardized information density.
Predefined Rendering: Uses a fixed document format (12-pt Helvetica, 96 DPI) to simulate realistic document processing.

Extensive Model Coverage

Benchmarks 10+ leading models including GPT-5, Gemini-3-Pro, Gemma3, Glyph, Qwen2.5 & Qwen3 & InternVL3.5 series, and more.

Easily extensible to new models via our server-client evaluation framework.

📊 Benchmark Explained by Tasks

Task	Task Categories	Context Example	Evaluation Example
VTC-Retrieval (NIAH)	Lexical Matching, Multi-Hop Tracing, Aggregation	^{Dynamic query/key-value with types: word-word, word-number, uuid-number. visual example} (essays...) One of the special magic numbers for long-context is: 2026. ...One of the special magic numbers for distracting-information is: 2025.	QA Variant: Q: What's the special magic number for long-context? A: 2026. Completion Variant: Prompt: one of the special magic number for long-context is: Completion: 2026.
VTC-Reasoning (NIAH)	Associative Reasoning, Question-Answering	^{Dynamic query/key-value with types: event/action-person. visual example.} (books...) There was a vegan guest, named Katie.	One-Hop Reasoning: Q: Which character cannot eat fish-based meals? A: Katie. Two-Hop Reasoning: Q: Which character cannot eat Brandade meals? A: Katie.
VTC-Memory (QA)	Memory, Question-Answering	^{No dynamic query/key-value, fully static. visual example.} (conversations...) Caroline: Researching adoption agencies—it's been a dream to have a family and give a loving home to kids who need it.	Q: What did Caroline research? A: Adoption agencies.
VTCBench-Wild	All of the above	A more challenging variant of the above tasks, introducing visual diversity to simulate real-world document conditions.

🛠 Usage & Data

Please refer to the Usage Guide for instructions on how to use VTCBench.

📄 Citation

@inproceedings{zhao-etal-2026-vtcbench,
    title = "{VTCB}ench: Can Vision-Language Models Understand Long Contexts with Vision-Text Compression?",
    author = "Zhao, Hongbo  and
      Wang, Meng  and
      Zhu, Fei  and
      Liu, Wenzhuo  and
      Ni, Bolin  and
      Zeng, Fanhu  and
      Meng, Gaofeng  and
      Zhang, Zhaoxiang  and
      Liu, Cheng-Lin",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Jessy  and
      Elliott, Desmond",
    booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.emnlp-main.145/",
    doi = "10.18653/2026.emnlp-main.145",
    pages = "3349--3379",
    ISBN = "979-8-89176-459-0"
}