Le Zhuo

About Me

I am currently working as a young researcher at Shanghai AI Lab after receiving my B.E. degree in 2024, supervised by Dr. Peng Gao and Prof. Hongsheng Li. Before this, I was a research intern at Mila - Québec AI Institute supervised by Prof. Jian Tang. During my undergraduate years, I was supervised by Prof. Si Liu and I was fortunate to work with Prof. Shuicheng Yan and Dr. Jie Fu at Beijing Academy of Artificial Intelligence.

Research Interests

Generative Modeling: Broad families of generative models and their potential applications, e.g., diffusion models, flow matching, etc.
Multimodal Learning: Developing models that seamlessly integrate and bridge the gap between various modalities, such as text, image, and audio.

News

[Feb. 2025] One paper about building efficient video LLM is accepted to CVPR 2025!
[Jan. 2025] Three papers are accepted to ICLR 2025, including one spotlight paper for our Lumina-T2X!
[Sep. 2024] Our paper “Lumina-Next” for improving text-to-image generation is accepted to NeurIPS 2024!
[May 2024] Our paper “ProtLLM” about protein understanding using LLMs is accepted to ACL 2024!
[May 2024] Check out our Lumina series of papers, the next generation of generative models including flow-based models, autoregressive models, and more on the way!
[Dec. 2023] I am honored to receive the 2023 SenseTime Scholarship, which is awarded to 30 Chinese undergraduate students in the field of AI!
[Sep. 2023] Our benchmark paper about music representation learning is accepted to NeurIPS 2023!
[Jul. 2023] Our paper about music driven dance generation is accepted to ACM MM 2023 as an oral presentation!
[Jul. 2023] Our paper about video background music generation is accepted to ICCV 2023!
[Jun. 2023] Our paper about automatic lyrics transcription is accepted to ISMIR 2023!

Publications

ICLR

LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation

Fangxun Shu*, Yue Liao*, Le Zhuo, Chenning Xu, Lei Zhang, Guanghao Zhang, Haonan Shi, Long Chen, Tao Zhong, Wanggui He, Siming Fu, Haoyuan Li, Bolin Li, Zhelun Yu, Si Liu, Hongsheng Li, Hao Jiang

International Conference on Learning Representations (ICLR), 2025.

PDF Code

ICLR

PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions

Weifeng Lin*, Xinyu Wei*, Renrui Zhang*, Le Zhuo, Shitian Zhao, Siyuan Huang, Junlin Xie, Yu Qiao, Peng Gao, Hongsheng Li

International Conference on Learning Representations (ICLR), 2025.

PDF Code

Preprint

Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining

Dongyang Liu*, Shitian Zhao*, Le Zhuo*, Weifeng Lin*, Yu Qiao, Hongsheng Li, Peng Gao

Technical Report.

PDF Code

NeurIPS

Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT

Le Zhuo*, Ruoyi Du*, Han Xiao*, Yangguang Li*, Dongyang Liu*, Rongjie Huang*, Wenze Liu*, Lirui Zhao, Fu-Yun Wang, Zhanyu Ma, Xu Luo, Zehan Wang, Kaipeng Zhang, Xiangyang Zhu, Si Liu, Xiangyu Yue, Dingning Liu, Wanli Ouyang, Ziwei Liu, Yu Qiao, Hongsheng Li, Peng Gao

Neural Information Processing Systems (NeurIPS), 2024.

PDF Code

ICLR

Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers

Peng Gao*, Le Zhuo*, Dongyang Liu*, Ruoyi Du*, Xu Luo*, Longtian Qiu, Yuhang Zhang, Chen Lin, Rongjie Huang, Shijie Geng, Renrui Zhang, Junlin Xi, Wenqi Shao, Zhengkai Jiang, Tianshuo Yang, Weicai Ye, He Tong, Jingwen He, Yu Qiao, Hongsheng Li

International Conference on Learning Representations (ICLR), 2025.

PDF Code Spotlight

ACL

ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training

Le Zhuo*, Zewen Chi*, Minghao Xu*, Heyan Huang, Jianan Zhao, Heqi Zheng, Conghui He, Xian-Ling Mao, Wentao Zhang

Association for Computational Linguistics, (ACL), 2024.

PDF Code Project Page

ICLR

LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions

Songhao Han*, Le Zhuo*, Yue Liao, Si Liu

ICLR 2025 Science for Foundation Models (SCI-FM) Workshop

PDF Code

NeurIPS

GraphText: Graph Reasoning in Text Space

Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, Jian Tang

NeurIPS 2024 Workshop on Adaptive Foundation Models.

PDF Code

ACM MM

DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation

Qiaosong Qi*, Le Zhuo*, Aixi Zhang, Yue Liao, Fei Fang, Si Liu, Shuicheng Yan

ACM International Conference on Multimedia (ACM MM), 2023.

PDF Oral Presentation

ICCV

Video Background Music Generation: Dataset, Method and Evaluation

Le Zhuo*, Zhaokai Wang*, Baisen Wang*, Yue Liao, Chenxi Bao, Stanley Peng, Songhao Han, Aixi Zhang, Fei Fang, Si Liu

IEEE/CVF International Conference on Computer Vision (ICCV), 2023.

PDF Code

NeurIPS

MARBLE: Music Audio Representation Benchmark for Universal Evaluation

Ruibin Yuan*, Yinghao Ma*, Yizhi Li*, Ge Zhang*, Xingran Chen, Hanzhi Yin, Le Zhuo, Yiqi Liu, Jiawen Huang, Zeyue Tian, Binyue Deng, Ningzhi Wang, Chenghua Lin, Emmanouil Benetos, Anton Ragni, Norbert Gyenge, Roger Dannenbert, Wenhu Chen, Gus Xia, Wei Xue, Si Liu, Shi Wang, Ruibo Liu, Yike Guo, Jie Fu

Neural Information Processing Systems (NeurIPS), 2023.

PDF Code Project Page

ISMIR

LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT

Le Zhuo, Ruibin Yuan, Jiahao Pan, Yizhi Li, Ge Zhang, Jiawen Huang, Si Liu, Roger Dannenberg, Jie Fu, Chenghua Lin, Emmanouil Benetos, Wenhu Chen, Wei Xue, Yike Guo

International Society for Music Information Retrieval (ISMIR), 2023.

PDF Code

About Me

Research Interests

News

Publications

Services

Conference Reviewers