From Retrieved Context to Evidence-Seeking Trajectories: A Survey of Agentic Multimodal RAG

1. Problem setting

Retrieval-Augmented Generation grounds language models in external knowledge, but the standard retrieve-then-read formulation remains mostly text-centric and pipeline-based. The assumption becomes insufficient when the required support is distributed across long documents, layouts, tables, charts, screenshots, webpages, frames, clips, and videos.

This survey uses evidence seeking as the central problem. In this setting, a system must decide where to look, which tool to use, which observation to trust, whether the accumulated evidence is sufficient, and how the final answer is grounded. The object of analysis is therefore not only a retrieved chunk or final answer, but the sequence of decisions that leads to the answer.

The survey reconstructs the flow from LLM agents and static RAG to agentic RAG, multimodal RAG, visual document RAG, document agents, visual web/GUI agents, and video agentic RAG. It uses the evidence-seeking step as the unit of analysis: action, observation, memory update, verification, and stopping.

2. Research questions

The survey is organized around five research questions. These questions turn the topic from a broad catalog of agentic and multimodal systems into a focused study of how systems search, inspect, verify, remember, stop, and cite evidence.

3. Terminology and scope

Agentic Multimodal RAG combines terms that are often used loosely. This survey uses the following terminology to keep the scope precise.

4. Survey protocol and coding axes

A trajectory-level survey should make its corpus construction and coding criteria explicit. The protocol below is designed for a paper version of this blog and can be used to support a systematic screening log or appendix.

5. Positioning against nearby surveys

The survey distinguishes its scope from several adjacent survey families. LLM-agent surveys emphasize planning, memory, tools, reflection, and collaboration, but usually do not center exact evidence units or citation. RAG surveys emphasize indexing, retrieval, augmentation, generation, reranking, and evaluation, but often treat retrieval as a module rather than a trajectory decision. Multimodal RAG surveys cover cross-modal retrieval and fusion, while document-oriented MRAG surveys focus on document pipelines, retrieval granularity, graph structures, datasets, and applications.

The main distinction is trajectory-level evidence seeking over heterogeneous evidence units. The selected unit can be a passage, page, region, table cell, chart mark, screenshot element, frame, clip, event, or UI state. The evaluation target also changes: answer correctness is separated from search, inspection, grounding, stopping, citation, and cost.

6. Agentic Multimodal Evidence Seeking

The survey defines Agentic Multimodal Evidence Seeking as an iterative process where a system gathers and verifies evidence across a multimodal information space before answering. The evidence space E can contain passages, pages, regions, tables, charts, frames, clips, webpages, interface states, or videos.

Here, q is the query, M_t is evidence memory, o_<t are previous observations, and b_t is the remaining budget. The operation τ_t can be retrieve, browse, crop, zoom, OCR, parse, localize, calculate, summarize, compare, or verify. The memory update stores content, provenance, confidence, and possibly rejected evidence. The process stops when a verifier estimates that the accumulated evidence is sufficient, grounded, and worth the cost of answering.

The formulation separates five design decisions:

• Evidence unit: where to search.
• Tool/action: how to inspect the selected evidence.
• Memory/provenance: what to retain across steps.
• Verification: whether the evidence supports the claim.
• Stopping: when to answer under a budget.

7. Research flow and paper line

The paper line is organized along three axes: control primitives, retrieval policy, and evidence granularity. Agentic Multimodal RAG appears where these axes converge.

From generation to action

ReAct turns reasoning into an interaction loop by interleaving reasoning traces, actions, and observations. Toolformer, PAL, and HuggingGPT expand the action space through APIs, executable programs, and model orchestration. Planning, reflection, and multi-agent systems add deliberation, self-correction, and role specialization. These mechanisms provide the control layer used later for multimodal evidence seeking.

From static RAG to retrieval as policy

RAG and DPR establish external text memory for generation, but static retrieval is limited for multi-hop or long-form tasks. IRCoT interleaves retrieval with chain-of-thought reasoning, FLARE retrieves actively during generation, Self-RAG learns to retrieve and critique through self-reflection, Adaptive-RAG routes questions by complexity, and CRAG evaluates retrieval quality before corrective actions. CogPlanner brings this transition into multimodal RAG by treating retrieval strategy selection and query reformulation as a planning problem.

From text chunks to multimodal evidence units

MuRAG, RA-CM3, and REVEAL expand retrieval augmentation from text passages to image-text or multimodal memory. ColPali, VisRAG, and VDocRAG retrieve page images or visually rich document units rather than text chunks. RAG-Anything and related graph-oriented approaches motivate representing multimodal content as interconnected entities rather than isolated modality-specific stores. The evidence unit can become a page, region, table cell, chart mark, figure, frame, clip, or event.

From document input to document environment

Document agents move beyond page retrieval by deciding where to look, how to inspect, what to remember, and when to verify. ViDoRAG frames visual document RAG as multi-agent exploration with retrieval, summarization, and reflection. DocAgent treats long-context document understanding as an agentic reading process over structured document evidence. MDocAgent is treated as a preprint and is described as exploring multi-agent specialization over text and image evidence.

Beyond documents: webpages, GUIs, and video

SeeAct, VisualWebArena, and OSWorld show that screenshots and interfaces can be treated as visual environments with actions and grounding requirements. Their objective is often task completion rather than question answering, but the same action-observation structure appears. VideoAgent, Video-RAG, and VideoRAG extend evidence seeking temporally: page retrieval becomes frame/clip retrieval, document memory becomes temporal memory, and page citation becomes temporal citation.

8. Taxonomy

The taxonomy classifies systems by how they seek evidence rather than by model family alone. Changing from passage retrieval to page retrieval changes the available actions. Changing from page to region changes the grounding target. Changing from document to video changes memory and citation format.

Levels of agenticity

A common weakness in Agentic RAG discussions is that any iterative retrieval system may be called agentic. This survey separates weak, moderate, and strong forms of agenticity.

System classes

• Text-only agentic RAG: passage or chunk evidence, rewrite/decompose/retrieve/critique/stop actions, passage citation as grounding target.
• Multimodal RAG: passage, image, or table evidence; cross-modal retrieval followed by VLM/LLM reading.
• Visual document RAG: page or visual document unit retrieval, with page citation as a common grounding target.
• Single-document agents: section, page, region, table, and figure units with browse/crop/OCR/inspect/summarize/verify actions.
• Multi-document agents: document, page, region, and source units with search/compare/consolidate/verify actions.
• Visual web/GUI agents: screenshot, DOM element, and UI state evidence with plan/click/type/observe/verify actions.
• Video agentic RAG: frame, clip, scene, and event evidence with localize/retrieve/summarize/compare/verify actions.
• General multimodal agents: heterogeneous units across mixed sources with search, code, visual tools, retrieval, and verification.

9. Architectural patterns and components

The taxonomy identifies recurring patterns: reason-act-observe loops, retrieval as action, planning and routing, planner-reader-verifier separation, hierarchical navigation, multi-agent specialization, and grounded citation.

10. Benchmarks and evaluation

The benchmark section separates answer quality from evidence-seeking behavior. A system can obtain a correct final answer while using irrelevant evidence, excessive tool calls, or unsupported citations. Conversely, it can retrieve the correct page or clip but fail at visual reading, numerical reasoning, memory update, verification, or citation.

Across existing benchmark families, answer quality is usually measured more directly than evidence quality, exact grounding, trajectory quality, or cost-aware utility. This creates an evaluation gap for systems whose main behavior is not only answering, but deciding which multimodal evidence to inspect and cite.

Evaluation layers

• Answer quality: whether the final answer is correct and task-appropriate.
• Evidence quality: whether gathered evidence is relevant, diverse, nonredundant, and sufficient.
• Grounding quality: whether claims align with exact passages, pages, regions, table cells, frames, clips, events, or UI states.
• Trajectory quality: whether the selected tools and evidence units are appropriate under a realistic budget.
• Cost-aware utility: whether improvements are worth additional tokens, retrieval calls, VLM calls, latency, and monetary cost.

Trajectory score decomposition

A useful trajectory score should be decomposable. Otherwise, the benchmark may only report that an agent failed, without showing where the failure happened.

Metric definitions

The evaluation protocol uses explicit process metrics. Evidence precision measures the fraction of retrieved or inspected units that support the answer. Evidence recall measures the fraction of gold support units found by the trajectory. Support sufficiency checks whether cited evidence entails the answer claim. Localization quality can be measured by exact page/cell match, spatial IoU for regions, or temporal IoU for clips/events. Tool-use precision measures necessary actions over total actions. Unnecessary-action rate captures wasted calls. Stopping accuracy checks whether the agent stops only when evidence is sufficient.

Failure diagnosis

• Search failure: the relevant unit is never selected.
• Inspection failure: the right unit is found but the wrong crop, tool, or parser is used.
• Reading failure: the model misinterprets visible evidence.
• Memory failure: relevant evidence is overwritten or summarized away.
• Verification failure: unsupported claims pass the critic.
• Citation failure: the final answer points to evidence that is relevant but insufficient.

11. Reporting protocol

The reporting protocol lists the information needed to compare evidence-seeking policies. It separates system capability from budget usage and separates retrieval granularity from reader strength.

12. Open problems

Unified benchmarks for multimodal agentic behavior. Benchmarks should jointly evaluate final answer quality, evidence relevance, exact visual/temporal grounding, tool-use decisions, trajectory quality, and cost. Existing benchmarks cover fragments of this objective.

Trajectory supervision without overfitting. Step-wise labels provide supervision, but multiple evidence-seeking paths may be valid for the same question. This distinction separates support-equivalent trajectories from incorrect ones and emphasizes concise sufficient evidence rather than a single canonical chain.

Region-level and temporal grounding. Page-level or video-level evidence is often too coarse. Exact regions, table cells, chart marks, frames, clips, and events are grounding targets that require annotation and evaluation beyond answer accuracy.

Visual hallucination and evidence verification. VLMs can produce plausible but unsupported interpretations of charts, figures, screenshots, or frames. Multimodal verification can be decomposed across textual entailment, table consistency, chart value extraction, visual grounding, UI-state validation, and temporal localization.

Long-context memory and cost-aware inference. Iterative multimodal inference can involve VLM calls, OCR, cropping, ASR, retrieval, and video processing. Cost-normalized performance and provenance-aware memory are used to track source identifiers, coordinates, timestamps, confidence, and rejected evidence.

Multi-document and multi-source conflict resolution. Real tasks can involve conflicting reports, websites, papers, or videos. Claim-level evidence graphs, contradiction detection, and source-reliability modeling are required mechanisms.

Security and adversarial evidence. Retrieved webpages, screenshots, document images, or video frames may contain prompt-injection text, poisoned captions, misleading visual evidence, or adversarial layout. Evidence sanitization, source isolation, provenance tracking, and verifiers that distinguish task instructions from retrieved content are required directions.

Standardized trajectory logging. Different systems expose different levels of detail in tool calls, observations, intermediate notes, and evidence provenance. Without a shared logging schema, trajectory-level comparison remains difficult.

13. Conclusion

The survey's central shift is from retrieved context to evidence seeking. LLM-agent work provides action and tool-use mechanisms; agentic RAG turns retrieval into policy; multimodal RAG expands evidence beyond passages; visual document RAG moves retrieval to pages and regions; web and GUI agents expose action grounding; document agents make documents navigable environments; and video agentic RAG extends the same principle to temporal evidence.

Under this framing, future RAG systems are evaluated not only by final-answer accuracy, but also by how they retrieve, inspect, verify, remember, stop, and cite multimodal evidence under cost and grounding constraints.