Recent advancements in open-source pretrained large language models (LLMs) have created new opportunities for exploring exciting post-pretraining innovations. This talk shares some of our recent works. The first part of my talk focuses on context length generalization in LLMs. I will begin with a theoretical analysis that identifies major factors contributing to the failures of several commonly-used techniques, which leads to the development of a simple yet effective algorithm that enables pretrained LLMs to generalize to extreme context lengths without any parameter update. I will then shift focus to continual pretraining for length generalization, and share our recent findings highlighting the importance of training data mixture—a crucial yet previously overlooked factor. The second part of my talk will be about Eurus, our recently-released suite of open LLMs. On diverse benchmarks covering challenging math, coding, and reasoning problems, Eurus achieves state-of-the-art performance among all open-source models and outperforms GPT-3.5 Turbo. On two established reward modeling benchmarks, our 7B reward model achieves better correlation with human judgment than all existing models including GPT-4. I will especially highlight UltraInteract, our newly-curated alignment dataset that enables Eurus’s strong performance.

Bio: Hao Peng is an Assistant Professor in the Department of Computer Science of the University of Illinois at Urbana-Champaign (UIUC). He received his PhD from the University of Washington and his bachelor’s degree from Peking University. Before joining UIUC, he spent one year at the Allen Institute for Artificial Intelligence as a Young Investigator, and time at Microsoft Research, Google, and DeepMind as an intern. His research interest broadly spans natural language processing and machine learning. Hao’s research has been recognized by spotlights and award nominations at ICLR and ACL, and a Google PhD fellowship.