Zeyu Huang

I am a PhD student (2024.01 - Now) at the University of Edinburgh. I am very lucky to be supervised by Ivan Titov and Edoardo M. Ponti. I am honored to be named a 2026 Apple Scholar in AIML (1 of 20 globally).

I am broadly interested in the learning of large language models, including the learning paradigms we use, the objectives we optimize, and the model architectures we build.

Learning paradigms. How LLMs learn from corrections, demonstrations and exploration, or context enriched with information from external search (Transformer-Patcher, Prefix-RFT, Context Training).
Learning objectives. How rewards and training objectives shape model behavior and expert specialization (Reward Calibration, Cancellation Hypothesis, Load Balancing Loss).
Model architectures. How modularity, routing, and attention shape learning and model behavior (Emergent Modularity, Recurrent Router, A Closer Look into MoE, Gated Attention, Attention and Residual Sinks).

I am currently seeking full-time opportunities. Please feel free to reach out if you think there may be a good fit.

Email / CV / Google Scholar / GitHub / Twitter / LinkedIn

Selected Awards

2026 Apple PhD Scholar in AI/ML (1 of 20 globally)
NeurIPS 2025 Best Paper Award (4 / 21575 submissions), co-first-author
NAACL 2024 Outstanding Paper Award (6 / 2604 submissions), co-first-author
EPSRC DTA Scholarship (international flexibility), 2024-2027
National Scholarship for Graduate Students Award, 2021-2022 (3 / 291)
Outstanding Graduate Student of Beihang University, 2020-2021

Internships

Qwen Pretraining Team - AliStar Research Intern (Jul. 2026 - Present), Beijing
Advisor: Dr. Bo Zheng and Zihan Qiu
ByteDance Seed - TopSeed Research Intern (Mar. 2026 - Jul. 2026), Shanghai
Advisors: Dr. Wenhao Zhu and Shanbo Cheng
Google DeepMind - Student Researcher (Aug. 2025 - Feb. 2026), London
Advisor: Dr. Marc'Aurelio Ranzato and Dr. Adhiguna Kuncoro
INF Technology - Research Intern (Mar. 2024 - Aug. 2024), Remote
Advisor: Zili Wang
BAAI - Research Intern (Sep. 2022 - Jun. 2023), Beijing, China
Advisor: Dr. Jie Fu
WeChat AI - Research Intern (Nov. 2021 - May 2022), Beijing, China
Advisor: Dr. Yikang Shen

Selected Preprints

Context Training with Active Information Seeking
Zeyu Huang, Adhiguna Kuncoro, Qixuan Feng, Jiajun Shen, Lucio Dery, Arthur Szlam, Marc'Aurelio Ranzato.
arXiv, 2026

The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits
Tianhao Cheng*, Zeyu Huang*, Zihan Qiu, Yu Cheng, Edoardo Ponti, Yinghui Xu, Ivan Titov, Zenglin Xu.
arXiv, 2026

Selected Publications

A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
Zihan Qiu*, Zeyu Huang*, Kaiyue Wen*, Peng Jin*, Bo Zheng*, ... , Dayiheng Liu, Jingren Zhou, Junyang Lin.
COLM 2026

Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M Ponti, Ivan Titov.
ICML 2026 | code

A Controllable Examination for Long-Context Language Models
Yijun Yang*, Zeyu Huang*, Wenhao Zhu, Zihan Qiu, Fei Yuan, Jeff Z Pan, Ivan Titov.
NeurIPS 2025 DB Track, 🏆 Spotlight (56 / 1995 submissions) | code

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
Zihan Qiu*, Zekun Wang*, Bo Zheng*, Zeyu Huang*, Kaiyue Wen*, ... , Dayiheng Liu, Jingren Zhou, Junyang Lin
NeurIPS 2025, 🏆 Oral and Best Paper Award (4 / 21575 submissions) | code

Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
Zihan Qiu*, Zeyu Huang*, Bo Zheng*, Kaiyue Wen, Zekun Wang, Rui Men, Ivan Titov, Dayiheng Liu, Jingren Zhou, Junyang Lin
ACL 2025 Main

A Closer Look into Mixture-of-Experts in Large Language Models
Ka Man Lo*, Zeyu Huang*, Zihan Qiu*, Zili Wang, Jie Fu.
NAACL 2025 Findings

Post-hoc Reward Calibration: A Case Study on Length Bias
Zeyu Huang, Zihan Qiu, Zili Wang, Edoardo M. Ponti, Ivan Titov
ICLR 2025 | code

Layerwise Recurrent Router for Mixture-of-Experts
Zihan Qiu*, Zeyu Huang*, Shuang Cheng, Yizhi Zhou, Zili Wang, Ivan Titov, Jie Fu
ICLR 2025 | code

Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training
Wenyu Du*, Tongxu Luo*, Zihan Qiu, Zeyu Huang, Yikang Shen, Reynold Cheng, Yike Guo, Jie Fu
NeurIPS 2024, 🏆 Spotlight (325 / 15671 submissions) | code

Unlocking Continual Learning Abilities in Language Models
Wenyu Du, Shuang Cheng, Tongxu Luo, Zihan Qiu, Zeyu Huang, Ka Chun Cheung, Reynold Cheng, Jie Fu.
EMNLP 2024 Findings

Unlocking Emergent Modularity in Large Language Models
Zihan Qiu*, Zeyu Huang*, Jie Fu
NAACL 2024, 🏆 Outstanding Paper (6 / 2604 submissions) | code

Transformer-Patcher: One Mistake worth One Neuron
Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, Zhang Xiong
ICLR 2023 | code

Mixture of Attention Heads: Selecting Attention Heads Per Token
Xiaofeng Zhang, Yikang Shen, Zeyu Huang, Jie Zhou, Wenge Rong, Zhang Xiong.
EMNLP 2022 Main

website credits