Le Zhuo

About Me

I am currently working as a young researcher at Shanghai AI Lab after receiving my B.E. degree in 2024, supervised by Dr. Peng Gao and Prof. Hongsheng Li. Before this, I was a research intern at Mila - Québec AI Institute supervised by Prof. Jian Tang. During my undergraduate years, I was supervised by Prof. Si Liu and I was fortunate to work with Prof. Shuicheng Yan and Dr. Jie Fu at Beijing Academy of Artificial Intelligence.

Research Interests

Generative Modeling: Broad families of generative models and their potential applications, e.g., diffusion models, flow matching, etc.
Multimodal Learning: Developing models that seamlessly integrate and bridge the gap between various modalities, such as text, image, and audio.
AI4Science: Exploring ways to make deep learning algorithms more robust to common challenges in drug discovery, e.g., protein understanding, molecular generation, etc.

News

[Sep. 2024] Our paper “Lumina-Next” for improving text-to-image generation is accepted to NeurIPS 2024!
[May 2024] Our paper “ProtLLM” about protein understanding using LLMs is accepted to ACL 2024!
[May 2024] Check out our Lumina series of papers, the next generation of generative models including flow-based models, autoregressive models, and more on the way!
[Dec. 2023] I am honored to receive the 2023 SenseTime Scholarship, which is awarded to 30 Chinese undergraduate students in the field of AI!
[Sep. 2023] Our benchmark paper about music representation learning is accepted to NeurIPS 2023!
[Jul. 2023] Our paper about music driven dance generation is accepted to ACM MM 2023 as an oral presentation!
[Jul. 2023] Our paper about video background music generation is accepted to ICCV 2023!
[Jun. 2023] Our paper about automatic lyrics transcription is accepted to ISMIR 2023!

Publications

Preprint

Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining

Dongyang Liu*, Shitian Zhao*, Le Zhuo*, Weifeng Lin*, Yu Qiao, Hongsheng Li, Peng Gao

Technical Report.

PDF Code

NeurIPS

Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT

Le Zhuo*, Ruoyi Du*, Han Xiao*, Yangguang Li*, Dongyang Liu*, Rongjie Huang*, Wenze Liu*, Lirui Zhao, Fu-Yun Wang, Zhanyu Ma, Xu Luo, Zehan Wang, Kaipeng Zhang, Xiangyang Zhu, Si Liu, Xiangyu Yue, Dingning Liu, Wanli Ouyang, Ziwei Liu, Yu Qiao, Hongsheng Li, Peng Gao

Neural Information Processing Systems (NeurIPS), 2024.

PDF Code

Preprint

Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers

Peng Gao*, Le Zhuo*, Ziyi Lin*, Chris Liu*, Junsong Chen*, Ruoyi Du*, Enze Xie*, Xu Luo*, Longtian Qiu*, Yuhang Zhang, Chen Lin, Rongjie Huang, Shijie Geng, Renrui Zhang, Junlin Xi, Wenqi Shao, Zhengkai Jiang, Tianshuo Yang, Weicai Ye, He Tong, Jingwen He, Yu Qiao, Hongsheng Li

Technical Report.

PDF Code

ACL

ProtLLM: An Interleaved Protein-Language LLM with Protein-as-Word Pre-Training

Le Zhuo*, Zewen Chi*, Minghao Xu*, Heyan Huang, Jianan Zhao, Heqi Zheng, Conghui He, Xian-Ling Mao, Wentao Zhang

Association for Computational Linguistics, (ACL), 2024.

PDF Code Project Page

Preprint

LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions

Songhao Han*, Le Zhuo*, Yue Liao, Si Liu

Under review.

PDF Code

Preprint

GraphText: Graph Reasoning in Text Space

Jianan Zhao, Le Zhuo, Yikang Shen, Meng Qu, Kai Liu, Michael Bronstein, Zhaocheng Zhu, Jian Tang

Under review.

PDF

ACM MM

DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation

Qiaosong Qi*, Le Zhuo*, Aixi Zhang, Yue Liao, Fei Fang, Si Liu, Shuicheng Yan

ACM International Conference on Multimedia (ACM MM), 2023.

PDF Oral Presentation

ICCV

Video Background Music Generation: Dataset, Method and Evaluation

Le Zhuo*, Zhaokai Wang*, Baisen Wang*, Yue Liao, Chenxi Bao, Stanley Peng, Songhao Han, Aixi Zhang, Fei Fang, Si Liu

IEEE/CVF International Conference on Computer Vision (ICCV), 2023.

PDF Code

NeurIPS

MARBLE: Music Audio Representation Benchmark for Universal Evaluation

Ruibin Yuan*, Yinghao Ma*, Yizhi Li*, Ge Zhang*, Xingran Chen, Hanzhi Yin, Le Zhuo, Yiqi Liu, Jiawen Huang, Zeyue Tian, Binyue Deng, Ningzhi Wang, Chenghua Lin, Emmanouil Benetos, Anton Ragni, Norbert Gyenge, Roger Dannenbert, Wenhu Chen, Gus Xia, Wei Xue, Si Liu, Shi Wang, Ruibo Liu, Yike Guo, Jie Fu

Neural Information Processing Systems (NeurIPS), 2023.

PDF Code Project Page

ISMIR

LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT

Le Zhuo, Ruibin Yuan, Jiahao Pan, Yizhi Li, Ge Zhang, Jiawen Huang, Si Liu, Roger Dannenberg, Jie Fu, Chenghua Lin, Emmanouil Benetos, Wenhu Chen, Wei Xue, Yike Guo

International Society for Music Information Retrieval (ISMIR), 2023.

PDF Code

About Me

Research Interests

News

Publications

Services

Conference Reviewers