Abstract: Reinforcement learning has gone from being a largely academic field to an established engine of the AI boom in a span of 2 years. This talk will describe this “industrialization of reinforcement learning” happening in private AI labs, from technical, organisational, and economic perspectives. What are RL practitioners doing in AI labs, and how different is it from more traditional RL? We’ll cover research questions, scalable systems, and infrastructure hiccups encountered along the way. We’ll also discuss the organisational challenges of modern RL, where thousands of concurrent agents may interact with sandboxes, external services, reward models, and distributed inference systems. A particular focus will be on crafting useful RL datasets and environments. How do we know if an RL sample is really useful? How can we generate such samples synthetically at scale? We’ll also discuss the growing RL ecosystem: new economic actors, data vendors, human annotation, and how they combine to support successful RL runs. Finally, many important questions remain open, some of which are very hard to investigate at scale. We will highlight several of the open problems that still stand in the way of the growing RL industry.

Bio: Nathan Grinsztajn works on agents at Mistral AI. Previously, he was a Member of Technical Staff at Cohere, where he worked on RL for LLMs and reasoning, and before that a research scientist at InstaDeep, where he focused on transformers for combinatorial optimization and discrete problems.