Sheng Zhang

I am a Principal Research Lead at Microsoft Research, where I build foundation models and frontier AI systems for multimodal reasoning and real-world applications.

My work includes foundation models used by millions of people (BiomedCLIP, Curiosity, GigaPath); test-time scaling methods for OpenAI frontier models (MedPrompt); and agent harnesses that extend LLMs to new modalities (Be My Eyes).

I also develop new post-training paradigms and data recipes (LLaVA-Med, OctoMed), and post-train models that address real-world problems frontier LLMs cannot yet solve (UniRG). I am fortunate to work with talented students and collaborators on a range of exciting research directions.

Publications [ Selected (9) | Full (69) | Google Scholar ]

2024BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs NEJM AI
Sheng Zhang^*, Yanbo Xu^*, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Angela Crabtree, Brian Piening, Carlo Bifulco, Matthew P. Lungren, Tristan Naumann, Sheng Wang, Hoifung Poon (^*equal contribution)
[ Model | Data ]
2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day NeurIPS 2023 (Spotlight)
Chunyuan Li^*, Cliff Wong^*, Sheng Zhang^*, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, Jianfeng Gao (^*equal contribution)
[ Project page ]
2026OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning CVPR 2026
Timothy Ossowski^*, Sheng Zhang^*, Qianchu Liu, Guanghui Qin, Reuben Tan, Tristan Naumann, Junjie Hu, Hoifung Poon (^*equal contribution)
[ Model | Blog | Data ]
2026Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaboration
James Y. Huang^*, Sheng Zhang^*, Qianchu Liu, Guanghui Qin, Tinghui Zhu, Tristan Naumann, Muhao Chen, Hoifung Poon (^*equal contribution)
[ Blog ]
2023Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine
Harsha Nori^*, Yin Tat Lee^*, Sheng Zhang^*, Dean Carignan, Richard Edgar, Nicolo Fusi, Nicholas King, Jonathan Larson, Yuanzhi Li, Weishung Liu, Renqian Luo, Scott Mayer McKinney, Robert Osazuwa Ness, Hoifung Poon, Tao Qin, Naoto Usuyama, Chris White, Eric Horvitz (^*equal contribution)
[ Blog | Code ]
2024A whole-slide foundation model for digital pathology from real-world data Nature
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, Yanbo Xu, Mu Wei, Wenhui Wang, Shuming Ma, Furu Wei, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Jaylen Rosemon, Tucker Bower, Soohee Lee, Roshanthi Weerasinghe, Bill J. Wright, Ari Robicsek, Brian Piening, Carlo Bifulco, Sheng Wang, Hoifung Poon
[ Model | Data ]
2026Multimodal AI generates virtual population for tumor microenvironment modeling Cell
Jeya Maria Jose Valanarasu, Hanwen Xu, Naoto Usuyama, Chanwoo Kim, Cliff Wong, Peniel Argaw, Racheli Ben Shimol, Angela Crabtree, Kevin Matlock, Alexandra Q. Bartlett, Jaspreet Bagga, Yu Gu, Sheng Zhang, Tristan Naumann, Bernard A. Fox, Bill Wright, Ari Robicsek, Brian Piening, Carlo Bifulco, Sheng Wang, Hoifung Poon
2024UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition ICLR 2024
Wenxuan Zhou^*, Sheng Zhang^*, Yu Gu, Muhao Chen, Hoifung Poon (^*equal contribution)
[ Demo | Model | MSR Podcast ]
2019AMR Parsing as Sequence-to-Graph Transduction ACL 2019 Best Paper Nominee
Sheng Zhang, Xutai Ma, Kevin Duh, Benjamin Van Durme
[ Code | BibTex ]

Sheng Zhang

Publications [ Selected (9) | Full (69) | Google Scholar ]

Tutorials

Service