Multimodal Augmented Reasoning and Search (MARS)

Research Areas

Efficient Multimodal Retrieval

How do we search over massive, multilingual video collections quickly without giving up quality? We build late-interaction retrievers like Video-ColBERT, compress multi-vector indexes in any modality, and fuse evidence across modalities and languages with MMMORRF. Since many information needs are too complex for a single retrieval pass, we also build rerankers that reason over video, audio, and text, such as RankVideo.

Reasoning & Calibration

Multimodal evidence is often partial, noisy, or contradictory. We study models that ground events in videos and text, and that perform uncertain inference across modalities with confidence that reflects what the evidence actually supports.

Collaborative Article Generation

We build agents that write and maintain grounded articles from many videos and documents, from WikiVideo and video RAG pipelines like MARQUIS to Knowledge Pull Requests for keeping documents up to date with people in the loop.

Evaluation

Generated text needs evaluation that works no matter what modality it came from. We develop metrics like MiRAGE for multimodal RAG and study what retrieval metrics actually tell us about downstream generation. We build benchmarks and shared tasks, including MultiVENT 2.0, MultiVENT-Raw, WikiVideo, and MAGMaR 2026, and we probe model capabilities and their flaws, such as attention sinks in VLMs.

Join Us

If you’re a Johns Hopkins student interested in any of these projects, send me an email. Search for Alexander Martin in the Hopkins Outlook directory and I’ll come up. I especially love it when students bring their own ideas, so if you have something you want to work on, tell me about it!

If you’re interested in working with Ben Van Durme, Xiang Xiang, or Reno Kriz, feel free to email them directly too.

People

Collaborators

• Benjamin Van Durme Professor, Johns Hopkins University
• Xiang Xiang Visiting Associate Professor, Johns Hopkins University
• Reno Kriz Research Scientist, HLTCOE
• Mahsa Yarmohammadi Associate Research Scientist, CLSP

Students

• Katherine Guerrerio B.S. 2027, Johns Hopkins University

Alumni

• Charles Weng Initial placement: Research Scientist, TikTok
• Dengjia Zhang Initial placement: Ph.D. Student, University of Pittsburgh
• Hanxiang Qin Initial placement: Machine Learning Engineer, Kuaishou